Summary:
- Add structured traceroute support with comprehensive IP enrichment (ASN/org/RDNS).
- Improve MikroTik traceroute cleaning and aggregation; collapse repeated tables into a single representative table.
- Enhance traceroute logging for visibility and add traceroute-specific cleaning helpers.
- Add/adjust IP enrichment plugins and BGP/traceroute enrichment integrations.
- UI updates for traceroute output and path visualization; update docs and configuration for structured output.
This commit squashes changes from 'structured-dev' into a single release commit.
MAJOR NEW ARCHITECTURE - STRUCTURED TRACEROUTE:
- Complete rewrite of traceroute data processing with structured output
- Dedicated TracerouteResult and TracerouteHop data models
- Platform-specific parsers with unified output format
- Rich metadata including ASN, organization, country, and prefix information
- AS path visualization with organization names in React Flow charts
SUPPORTED PLATFORMS:
- TraceroutePluginMikrotik: Handles MikroTik's complex multi-table format
* Progressive statistics parsing with deduplication
* Timeout hop handling and continuation line processing
* Loss percentage and RTT statistics extraction
- TraceroutePluginHuawei: Unix-style traceroute format parser
* Standard hop_number ip_address rtt format support
* Timeout hop detection with * notation
* Automatic cleanup of excessive trailing timeouts
COMPREHENSIVE IP ENRICHMENT SYSTEM:
- Offline enrichment using BGP.tools bulk data (1.3M+ CIDR entries)
- PeeringDB integration for IXP detection and ASN organization data
- Ultra-fast pickle cache system with combined data files
- Integer-based bitwise IP matching for maximum performance
- Bulk ASN organization lookup capabilities
- Private/reserved IP handling with AS0 fallbacks
- Country code mapping from ASN database
- Graceful fallbacks for missing enrichment data
FRONTEND ENHANCEMENTS:
- New traceroute table components with consistent formatting
- Enhanced AS path visualization with organization names
- Improved copy-to-clipboard functionality with structured data
- Unified table styling across BGP and traceroute results
- Better error handling and loading states
CONCURRENT PROCESSING INFRASTRUCTURE:
- Thread executor implementation for blocking I/O operations
- Query deduplication system to prevent resource conflicts
- Non-blocking Redis cache operations using asyncio executors
- Event coordination for waiting requests
- Background cleanup for completed operations
- Prevents website hangs during long-running queries
PLUGIN ARCHITECTURE IMPROVEMENTS:
- Platform-aware plugin system with proper execution restrictions
- Enhanced MikroTik garbage output cleaning
- IP enrichment plugins for both BGP routes and traceroute
- Conditional plugin execution based on platform detection
- Proper async/sync plugin method handling
CRITICAL BUG FIXES:
- Fixed double AS prefix bug (ASAS123456 → AS123456)
- Resolved TracerouteHop avg_rtt field/property conflicts
- Corrected Huawei traceroute source field validation
- Fixed plugin platform restriction enforcement
- Eliminated blocking I/O causing UI freezes
- Proper timeout and empty response caching prevention
- Enhanced private IP range detection and handling
PERFORMANCE OPTIMIZATIONS:
- Pickle cache system reduces startup time from seconds to milliseconds
- Bulk processing for ASN organization lookups
- Simplified IXP detection using single PeeringDB API call
- Efficient CIDR network sorting and integer-based lookups
- Reduced external API calls by 90%+
- Optimized memory usage for large datasets
API & ROUTING ENHANCEMENTS:
- Enhanced API routes with proper error handling
- Improved middleware for concurrent request processing
- Better state management and event handling
- Enhanced task processing with thread pool execution
This represents a complete transformation of hyperglass traceroute capabilities,
moving from basic text output to rich, structured data with comprehensive
network intelligence and concurrent processing support.
- Add BGPRoute import and export in hyperglass.models.data.__init__.py
- This allows BGP parsers to properly import BGPRoute for validation
- Resolves ImportError when starting hyperglass application
- Add BGPRoute import to Juniper, Arista, and FRR parsers
- Change all parsers to instantiate BGPRoute objects instead of raw dictionaries
- Harmonize variable naming across all BGP parsers:
- Use 'route_data' for dictionary variable
- Use 'routes' for output list
- Use 'route' for loop variable
- Ensure consistent validation behavior across MikroTik, Huawei, Juniper, Arista, and FRR
- All parsers now properly invoke BGP table class validation
- Remove custom __init__ from HuaweiBGPRouteTable class
- Use standard BGPRouteTable inheritance like MikroTik does
- Add BGPRoute import and instantiate BGPRoute objects in bgp_table()
- Enable proper Pydantic validation for both table and routes
- Ensure consistent behavior across all BGP parsers
- Update .gitignore to exclude dev-build.sh and dev-docker/
- Fix MED field showing N/A when value is 0 (now correctly shows 0)
- Hide Originator field when empty instead of showing N/A
- Hide Age field when not available instead of showing N/A
- Add HideableField component for fields that should be hidden when empty
- Update string output formatting to skip hidden fields in text export
Resolves display issues where valid zero values and unavailable fields
were showing as N/A instead of proper formatting or being hidden.
- Add IPv6 address pattern matching alongside IPv4
- Extract SENT count from traceroute lines to prioritize complete data
- Keep lines with highest SENT count (final results) instead of first occurrence
- Maintain original hop order while using best data for each IP
- Prefer non-timeout results when SENT counts are equal
Fixes issues where IPv4 showed early results (SENT=1) instead of final (SENT=3)
and IPv6 showed all intermediate steps instead of final consolidated results.
- Split type checking into two separate conditions for clarity
- Add early return when output contains non-string objects
- Ensure structured output like MikrotikBGPRouteTable passes through unchanged
- Prevent AttributeError on .strip() for tuple containing structured data
Resolves remaining crashes when processing already-structured BGP route data.
- Add type checking to handle already-processed structured output
- Skip processing when output is not raw strings (e.g., BGPRouteTable objects)
- Prevents AttributeError when trying to call .strip() on tuple/structured data
Fixes crash when MikroTik garbage plugin runs after structured parsing plugins.
- Change MikrotikGarbageOutput to apply to ALL MikroTik platform commands using common=True
- Add specialized traceroute output cleaning to handle progressive output and paging prompts
- Remove interactive paging prompts (-- [Q quit|C-z pause])
- Deduplicate traceroute hops to show only final results
- Clean up command echoes and empty progress lines
- Apply to custom directives automatically without manual configuration
Fixes messy traceroute output with repeated progress updates and interactive prompts.
- Add complete configuration reference (complete-config.mdx) with detailed parameter documentation, examples, and descriptions for all config.yaml options
- Add user guide (user-guide.mdx) explaining hostname resolution, 'My IP' feature, query types, and troubleshooting for end users
- Add quick-start configuration examples (quick-start.mdx) with 8 real-world scenarios: minimal, corporate, privacy-focused, high-performance, ISP, enterprise, development, and monitoring setups
- Update navigation structure (_meta.tsx files) to integrate new documentation sections
- Provide comprehensive coverage of DNS-over-HTTPS hostname resolution, myip.wtf IP detection, web UI customization, logging/webhooks, structured output, and theme configuration
- Include practical examples, use cases, troubleshooting tips, and best practices throughout
This addresses documentation gaps by providing both technical reference material for administrators and user-focused guidance for end users, with extensive real-world configuration examples.
- Update MikroTik parser to return -1 for unavailable age data instead of "0"
- Change age property return type from string to int for consistency
- Enhance frontend Age component to display "N/A" when age is -1
- Improve MonoField component to handle empty/undefined values gracefully
- Add appropriate tooltips and styling for missing data indicators
This addresses the issue where MikroTik devices don't provide route age
and source RID information, ensuring the UI displays user-friendly
indicators instead of potentially misleading placeholder values.
- Add new 'name' mode for BGP communities to append friendly names
- New configuration option `structured.communities.mode: name`
- Community mappings via `structured.communities.names` dictionary
- Communities display as "65000:1000 - Upstream Any" in UI
- Backward compatible with existing permit/deny modes
- Enhance RPKI configuration documentation
- Document both Cloudflare and Routinator backend options
- Add `structured.rpki.backend` and `structured.rpki.rpki_server_url` parameters
- Clarify Routinator web API endpoint usage vs RTR port
- Add comprehensive configuration examples
- Update structured output platform support
- Document all supported platforms: Arista EOS, FRRouting, Huawei VRP, Juniper Junos, Mikrotik RouterOS/SwitchOS
- Frontend enhancements
- Parse comma-separated community format in UI components
- Display friendly names alongside community codes
- Maintain existing functionality for communities without names
- Add validation and examples
- Validate that 'name' mode has community mappings configured
- Include example configuration and test cases
- Generic examples using ASN 65000 instead of specific networks
Expanded params config for rpki server URL and backend type for query switching and lookup handling.
Initial support for routinator server API endpoint.
