MAJOR ENHANCEMENTS:
IP Enrichment Service (hyperglass/external/ip_enrichment.py):
- Increase IXP data cache duration from 24 hours to 7 days (604800s) for better performance
- Fix critical cache refresh logic: ensure_data_loaded() now properly checks expiry before using existing pickle files
- Remove 'force' refresh parameters from public APIs and admin endpoints to prevent potential abuse/DDOS
- Implement automatic refresh based on file timestamps and cache duration
- Add comprehensive debug logging gated by Settings.debug throughout the module
- Clean up verbose comments and improve code readability
- Update configuration model to enforce 7-day minimum cache timeout
MikroTik Traceroute Processing:
- Refactor trace_route_mikrotik plugin to use garbage cleaner before structured parsing
- Only log raw router output when Settings.debug is enabled to reduce log verbosity
- Simplify MikrotikTracerouteTable parser to expect pre-cleaned input from garbage cleaner
- Remove complex multi-table detection, format detection, and deduplication logic (handled by cleaner)
- Add concise debug messages for processing decisions and configuration states
Traceroute IP Enrichment (traceroute_ip_enrichment.py):
- Implement concurrent reverse DNS lookups using asyncio.to_thread and asyncio.gather
- Add async wrapper for reverse DNS with proper error handling and fallbacks
- Significant performance improvement for multi-hop traceroutes (parallel vs sequential DNS)
- Proper debug logging gates: only detailed logs when Settings.debug=True
- Upgrade operational messages to log.info level (start/completion status)
- Maintain compatibility with different event loop contexts and runtime environments
Configuration Updates:
- Update structured.ip_enrichment.cache_timeout default to 604800 seconds
- Update documentation to reflect new cache defaults and behavior
- Remove force refresh options from admin API endpoints
MIGRATION NOTES:
- Operators should ensure /etc/hyperglass/ip_enrichment directory is writable
- Any code relying on force refresh parameters must be updated
- Monitor logs for automatic refresh behavior and performance improvements
- The 7-day cache significantly reduces PeeringDB API load
PERFORMANCE BENEFITS:
- Faster traceroute enrichment due to concurrent DNS lookups
- Reduced external API calls with longer IXP cache duration
- More reliable refresh logic prevents stale cache usage
- Cleaner, more focused debug output when debug mode is disabled
TECHNICAL DETAILS:
- Uses asyncio.to_thread for non-blocking DNS operations
- Implements process-wide file locking for safe concurrent cache updates
- Robust fallbacks for various asyncio execution contexts
- Maintains backward compatibility while improving performance
FILES MODIFIED:
- hyperglass/external/ip_enrichment.py
- hyperglass/models/config/structured.py
- hyperglass/api/routes.py
- hyperglass/plugins/_builtin/trace_route_mikrotik.py
- hyperglass/models/parsing/mikrotik.py
- hyperglass/plugins/_builtin/traceroute_ip_enrichment.py
- docs/pages/configuration/config/structured-output.mdx
Summary:
- Add structured traceroute support with comprehensive IP enrichment (ASN/org/RDNS).
- Improve MikroTik traceroute cleaning and aggregation; collapse repeated tables into a single representative table.
- Enhance traceroute logging for visibility and add traceroute-specific cleaning helpers.
- Add/adjust IP enrichment plugins and BGP/traceroute enrichment integrations.
- UI updates for traceroute output and path visualization; update docs and configuration for structured output.
This commit squashes changes from 'structured-dev' into a single release commit.
- Add complete configuration reference (complete-config.mdx) with detailed parameter documentation, examples, and descriptions for all config.yaml options
- Add user guide (user-guide.mdx) explaining hostname resolution, 'My IP' feature, query types, and troubleshooting for end users
- Add quick-start configuration examples (quick-start.mdx) with 8 real-world scenarios: minimal, corporate, privacy-focused, high-performance, ISP, enterprise, development, and monitoring setups
- Update navigation structure (_meta.tsx files) to integrate new documentation sections
- Provide comprehensive coverage of DNS-over-HTTPS hostname resolution, myip.wtf IP detection, web UI customization, logging/webhooks, structured output, and theme configuration
- Include practical examples, use cases, troubleshooting tips, and best practices throughout
This addresses documentation gaps by providing both technical reference material for administrators and user-focused guidance for end users, with extensive real-world configuration examples.
- Add new 'name' mode for BGP communities to append friendly names
- New configuration option `structured.communities.mode: name`
- Community mappings via `structured.communities.names` dictionary
- Communities display as "65000:1000 - Upstream Any" in UI
- Backward compatible with existing permit/deny modes
- Enhance RPKI configuration documentation
- Document both Cloudflare and Routinator backend options
- Add `structured.rpki.backend` and `structured.rpki.rpki_server_url` parameters
- Clarify Routinator web API endpoint usage vs RTR port
- Add comprehensive configuration examples
- Update structured output platform support
- Document all supported platforms: Arista EOS, FRRouting, Huawei VRP, Juniper Junos, Mikrotik RouterOS/SwitchOS
- Frontend enhancements
- Parse comma-separated community format in UI components
- Display friendly names alongside community codes
- Maintain existing functionality for communities without names
- Add validation and examples
- Validate that 'name' mode has community mappings configured
- Include example configuration and test cases
- Generic examples using ASN 65000 instead of specific networks
