MAJOR ENHANCEMENTS:
IP Enrichment Service (hyperglass/external/ip_enrichment.py):
- Increase IXP data cache duration from 24 hours to 7 days (604800s) for better performance
- Fix critical cache refresh logic: ensure_data_loaded() now properly checks expiry before using existing pickle files
- Remove 'force' refresh parameters from public APIs and admin endpoints to prevent potential abuse/DDOS
- Implement automatic refresh based on file timestamps and cache duration
- Add comprehensive debug logging gated by Settings.debug throughout the module
- Clean up verbose comments and improve code readability
- Update configuration model to enforce 7-day minimum cache timeout
MikroTik Traceroute Processing:
- Refactor trace_route_mikrotik plugin to use garbage cleaner before structured parsing
- Only log raw router output when Settings.debug is enabled to reduce log verbosity
- Simplify MikrotikTracerouteTable parser to expect pre-cleaned input from garbage cleaner
- Remove complex multi-table detection, format detection, and deduplication logic (handled by cleaner)
- Add concise debug messages for processing decisions and configuration states
Traceroute IP Enrichment (traceroute_ip_enrichment.py):
- Implement concurrent reverse DNS lookups using asyncio.to_thread and asyncio.gather
- Add async wrapper for reverse DNS with proper error handling and fallbacks
- Significant performance improvement for multi-hop traceroutes (parallel vs sequential DNS)
- Proper debug logging gates: only detailed logs when Settings.debug=True
- Upgrade operational messages to log.info level (start/completion status)
- Maintain compatibility with different event loop contexts and runtime environments
Configuration Updates:
- Update structured.ip_enrichment.cache_timeout default to 604800 seconds
- Update documentation to reflect new cache defaults and behavior
- Remove force refresh options from admin API endpoints
MIGRATION NOTES:
- Operators should ensure /etc/hyperglass/ip_enrichment directory is writable
- Any code relying on force refresh parameters must be updated
- Monitor logs for automatic refresh behavior and performance improvements
- The 7-day cache significantly reduces PeeringDB API load
PERFORMANCE BENEFITS:
- Faster traceroute enrichment due to concurrent DNS lookups
- Reduced external API calls with longer IXP cache duration
- More reliable refresh logic prevents stale cache usage
- Cleaner, more focused debug output when debug mode is disabled
TECHNICAL DETAILS:
- Uses asyncio.to_thread for non-blocking DNS operations
- Implements process-wide file locking for safe concurrent cache updates
- Robust fallbacks for various asyncio execution contexts
- Maintains backward compatibility while improving performance
FILES MODIFIED:
- hyperglass/external/ip_enrichment.py
- hyperglass/models/config/structured.py
- hyperglass/api/routes.py
- hyperglass/plugins/_builtin/trace_route_mikrotik.py
- hyperglass/models/parsing/mikrotik.py
- hyperglass/plugins/_builtin/traceroute_ip_enrichment.py
- docs/pages/configuration/config/structured-output.mdx
Summary:
- Add structured traceroute support with comprehensive IP enrichment (ASN/org/RDNS).
- Improve MikroTik traceroute cleaning and aggregation; collapse repeated tables into a single representative table.
- Enhance traceroute logging for visibility and add traceroute-specific cleaning helpers.
- Add/adjust IP enrichment plugins and BGP/traceroute enrichment integrations.
- UI updates for traceroute output and path visualization; update docs and configuration for structured output.
This commit squashes changes from 'structured-dev' into a single release commit.
MAJOR NEW ARCHITECTURE - STRUCTURED TRACEROUTE:
- Complete rewrite of traceroute data processing with structured output
- Dedicated TracerouteResult and TracerouteHop data models
- Platform-specific parsers with unified output format
- Rich metadata including ASN, organization, country, and prefix information
- AS path visualization with organization names in React Flow charts
SUPPORTED PLATFORMS:
- TraceroutePluginMikrotik: Handles MikroTik's complex multi-table format
* Progressive statistics parsing with deduplication
* Timeout hop handling and continuation line processing
* Loss percentage and RTT statistics extraction
- TraceroutePluginHuawei: Unix-style traceroute format parser
* Standard hop_number ip_address rtt format support
* Timeout hop detection with * notation
* Automatic cleanup of excessive trailing timeouts
COMPREHENSIVE IP ENRICHMENT SYSTEM:
- Offline enrichment using BGP.tools bulk data (1.3M+ CIDR entries)
- PeeringDB integration for IXP detection and ASN organization data
- Ultra-fast pickle cache system with combined data files
- Integer-based bitwise IP matching for maximum performance
- Bulk ASN organization lookup capabilities
- Private/reserved IP handling with AS0 fallbacks
- Country code mapping from ASN database
- Graceful fallbacks for missing enrichment data
FRONTEND ENHANCEMENTS:
- New traceroute table components with consistent formatting
- Enhanced AS path visualization with organization names
- Improved copy-to-clipboard functionality with structured data
- Unified table styling across BGP and traceroute results
- Better error handling and loading states
CONCURRENT PROCESSING INFRASTRUCTURE:
- Thread executor implementation for blocking I/O operations
- Query deduplication system to prevent resource conflicts
- Non-blocking Redis cache operations using asyncio executors
- Event coordination for waiting requests
- Background cleanup for completed operations
- Prevents website hangs during long-running queries
PLUGIN ARCHITECTURE IMPROVEMENTS:
- Platform-aware plugin system with proper execution restrictions
- Enhanced MikroTik garbage output cleaning
- IP enrichment plugins for both BGP routes and traceroute
- Conditional plugin execution based on platform detection
- Proper async/sync plugin method handling
CRITICAL BUG FIXES:
- Fixed double AS prefix bug (ASAS123456 → AS123456)
- Resolved TracerouteHop avg_rtt field/property conflicts
- Corrected Huawei traceroute source field validation
- Fixed plugin platform restriction enforcement
- Eliminated blocking I/O causing UI freezes
- Proper timeout and empty response caching prevention
- Enhanced private IP range detection and handling
PERFORMANCE OPTIMIZATIONS:
- Pickle cache system reduces startup time from seconds to milliseconds
- Bulk processing for ASN organization lookups
- Simplified IXP detection using single PeeringDB API call
- Efficient CIDR network sorting and integer-based lookups
- Reduced external API calls by 90%+
- Optimized memory usage for large datasets
API & ROUTING ENHANCEMENTS:
- Enhanced API routes with proper error handling
- Improved middleware for concurrent request processing
- Better state management and event handling
- Enhanced task processing with thread pool execution
This represents a complete transformation of hyperglass traceroute capabilities,
moving from basic text output to rich, structured data with comprehensive
network intelligence and concurrent processing support.
- Add BGPRoute import and export in hyperglass.models.data.__init__.py
- This allows BGP parsers to properly import BGPRoute for validation
- Resolves ImportError when starting hyperglass application
- Add BGPRoute import to Juniper, Arista, and FRR parsers
- Change all parsers to instantiate BGPRoute objects instead of raw dictionaries
- Harmonize variable naming across all BGP parsers:
- Use 'route_data' for dictionary variable
- Use 'routes' for output list
- Use 'route' for loop variable
- Ensure consistent validation behavior across MikroTik, Huawei, Juniper, Arista, and FRR
- All parsers now properly invoke BGP table class validation
- Remove custom __init__ from HuaweiBGPRouteTable class
- Use standard BGPRouteTable inheritance like MikroTik does
- Add BGPRoute import and instantiate BGPRoute objects in bgp_table()
- Enable proper Pydantic validation for both table and routes
- Ensure consistent behavior across all BGP parsers
- Update .gitignore to exclude dev-build.sh and dev-docker/
- Fix MED field showing N/A when value is 0 (now correctly shows 0)
- Hide Originator field when empty instead of showing N/A
- Hide Age field when not available instead of showing N/A
- Add HideableField component for fields that should be hidden when empty
- Update string output formatting to skip hidden fields in text export
Resolves display issues where valid zero values and unavailable fields
were showing as N/A instead of proper formatting or being hidden.
- Add IPv6 address pattern matching alongside IPv4
- Extract SENT count from traceroute lines to prioritize complete data
- Keep lines with highest SENT count (final results) instead of first occurrence
- Maintain original hop order while using best data for each IP
- Prefer non-timeout results when SENT counts are equal
Fixes issues where IPv4 showed early results (SENT=1) instead of final (SENT=3)
and IPv6 showed all intermediate steps instead of final consolidated results.
- Split type checking into two separate conditions for clarity
- Add early return when output contains non-string objects
- Ensure structured output like MikrotikBGPRouteTable passes through unchanged
- Prevent AttributeError on .strip() for tuple containing structured data
Resolves remaining crashes when processing already-structured BGP route data.
- Add type checking to handle already-processed structured output
- Skip processing when output is not raw strings (e.g., BGPRouteTable objects)
- Prevents AttributeError when trying to call .strip() on tuple/structured data
Fixes crash when MikroTik garbage plugin runs after structured parsing plugins.
- Change MikrotikGarbageOutput to apply to ALL MikroTik platform commands using common=True
- Add specialized traceroute output cleaning to handle progressive output and paging prompts
- Remove interactive paging prompts (-- [Q quit|C-z pause])
- Deduplicate traceroute hops to show only final results
- Clean up command echoes and empty progress lines
- Apply to custom directives automatically without manual configuration
Fixes messy traceroute output with repeated progress updates and interactive prompts.
- Add complete configuration reference (complete-config.mdx) with detailed parameter documentation, examples, and descriptions for all config.yaml options
- Add user guide (user-guide.mdx) explaining hostname resolution, 'My IP' feature, query types, and troubleshooting for end users
- Add quick-start configuration examples (quick-start.mdx) with 8 real-world scenarios: minimal, corporate, privacy-focused, high-performance, ISP, enterprise, development, and monitoring setups
- Update navigation structure (_meta.tsx files) to integrate new documentation sections
- Provide comprehensive coverage of DNS-over-HTTPS hostname resolution, myip.wtf IP detection, web UI customization, logging/webhooks, structured output, and theme configuration
- Include practical examples, use cases, troubleshooting tips, and best practices throughout
This addresses documentation gaps by providing both technical reference material for administrators and user-focused guidance for end users, with extensive real-world configuration examples.
- Update MikroTik parser to return -1 for unavailable age data instead of "0"
- Change age property return type from string to int for consistency
- Enhance frontend Age component to display "N/A" when age is -1
- Improve MonoField component to handle empty/undefined values gracefully
- Add appropriate tooltips and styling for missing data indicators
This addresses the issue where MikroTik devices don't provide route age
and source RID information, ensuring the UI displays user-friendly
indicators instead of potentially misleading placeholder values.
- Add new 'name' mode for BGP communities to append friendly names
- New configuration option `structured.communities.mode: name`
- Community mappings via `structured.communities.names` dictionary
- Communities display as "65000:1000 - Upstream Any" in UI
- Backward compatible with existing permit/deny modes
- Enhance RPKI configuration documentation
- Document both Cloudflare and Routinator backend options
- Add `structured.rpki.backend` and `structured.rpki.rpki_server_url` parameters
- Clarify Routinator web API endpoint usage vs RTR port
- Add comprehensive configuration examples
- Update structured output platform support
- Document all supported platforms: Arista EOS, FRRouting, Huawei VRP, Juniper Junos, Mikrotik RouterOS/SwitchOS
- Frontend enhancements
- Parse comma-separated community format in UI components
- Display friendly names alongside community codes
- Maintain existing functionality for communities without names
- Add validation and examples
- Validate that 'name' mode has community mappings configured
- Include example configuration and test cases
- Generic examples using ASN 65000 instead of specific networks
Expanded params config for rpki server URL and backend type for query switching and lookup handling.
Initial support for routinator server API endpoint.
