1
0
Fork 1
mirror of https://github.com/thatmattlove/hyperglass.git synced 2026-04-17 21:38:27 +00:00
thatmattlove-hyperglass/hyperglass/models/config/structured.py
Wilhelm Schonfeldt 0398966062
feat: Add structured traceroute support with comprehensive IP enrichment
MAJOR NEW ARCHITECTURE - STRUCTURED TRACEROUTE:
- Complete rewrite of traceroute data processing with structured output
- Dedicated TracerouteResult and TracerouteHop data models
- Platform-specific parsers with unified output format
- Rich metadata including ASN, organization, country, and prefix information
- AS path visualization with organization names in React Flow charts

SUPPORTED PLATFORMS:
- TraceroutePluginMikrotik: Handles MikroTik's complex multi-table format
  * Progressive statistics parsing with deduplication
  * Timeout hop handling and continuation line processing
  * Loss percentage and RTT statistics extraction
- TraceroutePluginHuawei: Unix-style traceroute format parser
  * Standard hop_number ip_address rtt format support
  * Timeout hop detection with * notation
  * Automatic cleanup of excessive trailing timeouts

COMPREHENSIVE IP ENRICHMENT SYSTEM:
- Offline enrichment using BGP.tools bulk data (1.3M+ CIDR entries)
- PeeringDB integration for IXP detection and ASN organization data
- Ultra-fast pickle cache system with combined data files
- Integer-based bitwise IP matching for maximum performance
- Bulk ASN organization lookup capabilities
- Private/reserved IP handling with AS0 fallbacks
- Country code mapping from ASN database
- Graceful fallbacks for missing enrichment data

FRONTEND ENHANCEMENTS:
- New traceroute table components with consistent formatting
- Enhanced AS path visualization with organization names
- Improved copy-to-clipboard functionality with structured data
- Unified table styling across BGP and traceroute results
- Better error handling and loading states

CONCURRENT PROCESSING INFRASTRUCTURE:
- Thread executor implementation for blocking I/O operations
- Query deduplication system to prevent resource conflicts
- Non-blocking Redis cache operations using asyncio executors
- Event coordination for waiting requests
- Background cleanup for completed operations
- Prevents website hangs during long-running queries

PLUGIN ARCHITECTURE IMPROVEMENTS:
- Platform-aware plugin system with proper execution restrictions
- Enhanced MikroTik garbage output cleaning
- IP enrichment plugins for both BGP routes and traceroute
- Conditional plugin execution based on platform detection
- Proper async/sync plugin method handling

CRITICAL BUG FIXES:
- Fixed double AS prefix bug (ASAS123456 → AS123456)
- Resolved TracerouteHop avg_rtt field/property conflicts
- Corrected Huawei traceroute source field validation
- Fixed plugin platform restriction enforcement
- Eliminated blocking I/O causing UI freezes
- Proper timeout and empty response caching prevention
- Enhanced private IP range detection and handling

PERFORMANCE OPTIMIZATIONS:
- Pickle cache system reduces startup time from seconds to milliseconds
- Bulk processing for ASN organization lookups
- Simplified IXP detection using single PeeringDB API call
- Efficient CIDR network sorting and integer-based lookups
- Reduced external API calls by 90%+
- Optimized memory usage for large datasets

API & ROUTING ENHANCEMENTS:
- Enhanced API routes with proper error handling
- Improved middleware for concurrent request processing
- Better state management and event handling
- Enhanced task processing with thread pool execution

This represents a complete transformation of hyperglass traceroute capabilities,
moving from basic text output to rich, structured data with comprehensive
network intelligence and concurrent processing support.
2025-09-28 13:48:04 +02:00

62 lines
1.9 KiB
Python

"""Structured data configuration variables."""
# Standard Library
import typing as t
# Third Party
from pydantic import field_validator, ValidationInfo
# Local
from ..main import HyperglassModel
StructuredCommunityMode = t.Literal["permit", "deny", "name"]
StructuredRPKIMode = t.Literal["router", "external"]
class StructuredCommunities(HyperglassModel):
"""Control structured data response for BGP communities."""
mode: StructuredCommunityMode = "deny"
items: t.List[str] = []
names: t.Dict[str, str] = {}
@field_validator("names")
def validate_names(cls, value: t.Dict[str, str], info: ValidationInfo) -> t.Dict[str, str]:
"""Validate that names are provided when mode is 'name'."""
if info.data and info.data.get("mode") == "name" and not value:
raise ValueError(
"When using mode 'name', at least one community mapping must be provided in 'names'"
)
return value
class StructuredRpki(HyperglassModel):
"""Control structured data response for RPKI state."""
mode: StructuredRPKIMode = "router"
backend: str = "cloudflare"
rpki_server_url: str = ""
class StructuredIpEnrichment(HyperglassModel):
"""Control IP enrichment for structured data responses."""
enabled: bool = False
cache_timeout: int = 86400 # 24 hours in seconds (minimum)
enrich_next_hop: bool = False
enrich_traceroute: bool = True
@field_validator("cache_timeout")
def validate_cache_timeout(cls, value: int) -> int:
"""Ensure cache timeout is at least 24 hours (86400 seconds)."""
if value < 86400:
return 86400
return value
class Structured(HyperglassModel):
"""Control structured data responses."""
communities: StructuredCommunities = StructuredCommunities()
rpki: StructuredRpki = StructuredRpki()
ip_enrichment: StructuredIpEnrichment = StructuredIpEnrichment()