1
0
Fork 1
mirror of https://github.com/thatmattlove/hyperglass.git synced 2026-02-01 23:09:30 +00:00
hyperglass is the network looking glass that tries to make the internet better.
Find a file
Wilhelm Schonfeldt 0398966062
feat: Add structured traceroute support with comprehensive IP enrichment
MAJOR NEW ARCHITECTURE - STRUCTURED TRACEROUTE:
- Complete rewrite of traceroute data processing with structured output
- Dedicated TracerouteResult and TracerouteHop data models
- Platform-specific parsers with unified output format
- Rich metadata including ASN, organization, country, and prefix information
- AS path visualization with organization names in React Flow charts

SUPPORTED PLATFORMS:
- TraceroutePluginMikrotik: Handles MikroTik's complex multi-table format
  * Progressive statistics parsing with deduplication
  * Timeout hop handling and continuation line processing
  * Loss percentage and RTT statistics extraction
- TraceroutePluginHuawei: Unix-style traceroute format parser
  * Standard hop_number ip_address rtt format support
  * Timeout hop detection with * notation
  * Automatic cleanup of excessive trailing timeouts

COMPREHENSIVE IP ENRICHMENT SYSTEM:
- Offline enrichment using BGP.tools bulk data (1.3M+ CIDR entries)
- PeeringDB integration for IXP detection and ASN organization data
- Ultra-fast pickle cache system with combined data files
- Integer-based bitwise IP matching for maximum performance
- Bulk ASN organization lookup capabilities
- Private/reserved IP handling with AS0 fallbacks
- Country code mapping from ASN database
- Graceful fallbacks for missing enrichment data

FRONTEND ENHANCEMENTS:
- New traceroute table components with consistent formatting
- Enhanced AS path visualization with organization names
- Improved copy-to-clipboard functionality with structured data
- Unified table styling across BGP and traceroute results
- Better error handling and loading states

CONCURRENT PROCESSING INFRASTRUCTURE:
- Thread executor implementation for blocking I/O operations
- Query deduplication system to prevent resource conflicts
- Non-blocking Redis cache operations using asyncio executors
- Event coordination for waiting requests
- Background cleanup for completed operations
- Prevents website hangs during long-running queries

PLUGIN ARCHITECTURE IMPROVEMENTS:
- Platform-aware plugin system with proper execution restrictions
- Enhanced MikroTik garbage output cleaning
- IP enrichment plugins for both BGP routes and traceroute
- Conditional plugin execution based on platform detection
- Proper async/sync plugin method handling

CRITICAL BUG FIXES:
- Fixed double AS prefix bug (ASAS123456 → AS123456)
- Resolved TracerouteHop avg_rtt field/property conflicts
- Corrected Huawei traceroute source field validation
- Fixed plugin platform restriction enforcement
- Eliminated blocking I/O causing UI freezes
- Proper timeout and empty response caching prevention
- Enhanced private IP range detection and handling

PERFORMANCE OPTIMIZATIONS:
- Pickle cache system reduces startup time from seconds to milliseconds
- Bulk processing for ASN organization lookups
- Simplified IXP detection using single PeeringDB API call
- Efficient CIDR network sorting and integer-based lookups
- Reduced external API calls by 90%+
- Optimized memory usage for large datasets

API & ROUTING ENHANCEMENTS:
- Enhanced API routes with proper error handling
- Improved middleware for concurrent request processing
- Better state management and event handling
- Enhanced task processing with thread pool execution

This represents a complete transformation of hyperglass traceroute capabilities,
moving from basic text output to rich, structured data with comprehensive
network intelligence and concurrent processing support.
2025-09-28 13:48:04 +02:00
.github bump pnpm version in docs 2025-06-14 22:01:41 -04:00
.samples Rename sample_config.yaml to sample_config 2025-09-13 05:09:34 -03:00
.tests remove legacy tooling 2024-03-17 16:02:58 -04:00
.vscode tooling cleanup 2024-02-27 22:55:32 -05:00
docs docs: Add comprehensive configuration documentation and user guide 2025-09-26 12:12:27 +02:00
hyperglass feat: Add structured traceroute support with comprehensive IP enrichment 2025-09-28 13:48:04 +02:00
.dockerignore fix docker build issue 2024-05-27 14:21:30 -04:00
.gitattributes clean up git; add actions 2020-02-13 18:27:33 -07:00
.gitignore Fix: Make Huawei BGP parser conform to MikroTik approach 2025-09-26 16:37:51 +02:00
biome.json fix biome ignore 2024-06-16 17:20:14 -04:00
CHANGELOG.md feat: Add structured traceroute support with comprehensive IP enrichment 2025-09-28 13:48:04 +02:00
CODE_OF_CONDUCT.md Migrate to hyperglass.dev domain 2021-05-31 18:31:14 -07:00
compose.yaml Update compose.yaml 2025-09-13 02:56:27 -03:00
CONTRIBUTING.md housekeeping 2024-06-01 23:23:08 -04:00
debug_mikrotik_minimal.py feat: Add structured traceroute support with comprehensive IP enrichment 2025-09-28 13:48:04 +02:00
Dockerfile Update Dockerfile update CLICK ERRO typer 2025-09-25 19:57:22 -03:00
example_community_names_config.yaml feat: Add BGP community friendly names and enhance RPKI configuration 2025-09-26 11:32:29 +02:00
hooks.sh example & example validation cleanup 2020-03-21 15:00:37 -07:00
LICENSE housekeeping 2024-06-01 23:23:08 -04:00
poetry.lock Update poetry.lock update erro typer click 2025-09-25 19:59:55 -03:00
pyproject.toml Add requests dependency to pyproject.toml 2025-09-26 09:37:58 +02:00
README.md Update README.md 2025-09-13 05:08:19 -03:00
requirements-dev.lock Update requirements-dev.lock update typer erro click 2025-09-25 19:58:45 -03:00
requirements.lock Merge remote-tracking branch 'structured/main' into structured 2025-09-26 09:37:06 +02:00
test_community_names.py fix: Improve error message for community name validation and clean up test formatting 2025-09-26 11:49:27 +02:00
TRACEROUTE_RESTRUCTURE_SUMMARY.md feat: Add structured traceroute support with comprehensive IP enrichment 2025-09-28 13:48:04 +02:00
version.py housekeeping 2024-06-01 23:23:08 -04:00

"### Install https://hyperglass.dev/installation/docker"

mkdir -p /etc/hyperglass/svg

cd /opt

git clone https://github.com/CarlosSuporteISP/hyperglass_structured.git --depth=1

mv hyperglass_structured hyperglass

cd /opt/hyperglass

"### https://hyperglass.dev/configuration/overview"

"### https://hyperglass.dev/configuration/config Change the files in the /etc/hyperglass folder after copying with your information or add something following the official doc"

cp /opt/hyperglass/.samples/sample_config /etc/hyperglass/config.yaml

cp /opt/hyperglass/.samples/sample_terms-and-conditions /etc/hyperglass/terms-and-conditions.md

"### https://hyperglass.dev/configuration/devices Change the files in the /etc/hyperglass folder after copying with your information or add something following the official doc"

cp /opt/hyperglass/.samples/sample_devices2 /etc/hyperglass/devices.yaml

"### https://hyperglass.dev/configuration/directives Change the files in the /etc/hyperglass folder after copying with your information or add something following the official doc"

cp /opt/hyperglass/.samples/sample_directives_huawei /etc/hyperglass/directives.yaml

cp /opt/hyperglass/.samples/sample_directives_juniper /etc/hyperglass/directives.yaml

cp /opt/hyperglass/.samples/sample_directives_mikrotik /etc/hyperglass/directives.yaml

"### Environment Variables https://hyperglass.dev/installation/environment-variables"

cp /opt/hyperglass/.samples/sample_hyperglass /etc/hyperglass/hyperglass.env

"###"

You also need to add your AS prefixes to deny queries if you don't want others to look up your own prefixes from your hyperglass instance.

In the directives file, there is a field that is usually commented out. This configuration is meant for devices like Huawei or MikroTik, but it is currently still using the default option from the directives. From what I've tested, putting the rules in the configuration folder (/etc/hyperglass/...) didn't work. If it works later, we can do everything within the directives file in /etc/hyperglass, but for now, it's okay to use the default.

It's possible to create or use the ENTRYPOINT in the Dockerfile to change this at build time when starting the service, but I don't have time right now to stop and implement this.

/opt/hyperglass/hyperglass/defaults/directives/huawei.py | /opt/hyperglass_structured/hyperglass/defaults/directives/mikrotik.py

The code snippet, originally commented, should be modified to something like this:

     # DENY RULE FOR AS PREFIX - IPv4
     RuleWithIPv4(
        condition="172.16.0.0/22",
        ge="22",
        le="32",
        action="deny",
        command="",
    ),

    # DENY RULE FOR AS PREFIX - IPv6
    RuleWithIPv6(
        condition="fd00:2::/32",
        ge="32",
        le="128",
        action="deny",
        command="",
    ),

mikrotik v6

command="ip route print detail without-paging where {target} in dst-address bgp and dst-address !=0.0.0.0/0",

command="ipv6 route print detail without-paging where {target} in dst-address bgp and dst-address !=::/0",

mikrotik v7

command="routing route print detail without-paging where {target} in dst-address bgp and dst-address !=0.0.0.0/0",

command="routing route print detail without-paging where {target} in dst-address bgp and dst-address !=::/0",

"###"

"### Optional: Quickstart"

cd /opt/hyperglass

docker compose up

"### Create a systemd service"

cp /opt/hyperglass/.samples/hyperglass-docker.service /etc/hyperglass/hyperglass.service

ln -s /etc/hyperglass/hyperglass.service /etc/systemd/system/hyperglass.service

systemctl daemon-reload

systemctl enable hyperglass

systemctl start hyperglass

"###"

Acknowledgments:

To thatmatt for this incredible project that I really like. Nothing against other Looking Glass (LG) projects. https://github.com/thatmattlove/hyperglass

To remotti for the tips on Telegram, his attention, and for his fork https://github.com/remontti/hyperglass/tree/main, https://blog.remontti.com.br/7201, which is already quite deprecated due to its age (Node 14, etc.) and not being in Docker. This is why I decided to move to the official version.

To the user \邪萬教教我/ @Yukaphoenix572 好呆. Thanks to a message from him in the Telegram group, my mind was opened to the solution after I searched through the conversations.

To issue https://github.com/thatmattlove/hyperglass/issues/318 for the solution to queries that also weren't working on Tik-Tik (for those who use Claro).

And of course, last but not least: to AIs. My apologies to those who don't like the "code vibe," but they help a lot. I used many of the six main AIs on the market, but only Manus truly managed to help me, contributing about 45% of the development, testing, adjustments, and descriptions.

The total development time took over three weeks to get everything adjusted. Yes, I know I'm not that great at development, but I'm studying and improving. As I always say, in life and professionally, we always have something to learn; we never know everything.

I also adjusted the official plugin (which wasn't working) for Huawei.

The issue was the format in which the prefix was being passed to the device. Huawei expects the format 192.0.2.0 24 (with a space), but the official plugin was sending it in the 192.0.2.0/24 format (with a slash).

The fix was made to adapt to the format that Huawei accepts for queries.

"###"



The network looking glass that tries to make the internet better.


A looking glass is implemented by network operators as a way of providing customers, peers, or the general public with a way to easily view elements of, or run tests from the provider's network.

Documentation   |   Live Demo

Frontend Tests Backend Tests


hyperglass is intended to make implementing a looking glass too easy not to do, with the lofty goal of improving the internet community at large by making looking glasses more common across autonomous systems of any size.

Changelog

Features

  • BGP Route, BGP Community, BGP AS Path, Ping, & Traceroute, or add your own commands.
  • Full IPv6 support
  • Customizable everything: features, theme, UI/API text, error messages, commands
  • Built-in support for:
    • Arista EOS
    • BIRD
    • Cisco IOS
    • Cisco NX-OS
    • Cisco IOS-XR
    • FRRouting
    • Huawei VRP
    • Juniper Junos
    • Mikrotik
    • Nokia SR OS
    • OpenBGPD
    • TNSR
    • VyOS
  • Configurable support for any other supported platform
  • Optionally access devices via an SSH proxy/jump server
  • Access-list/prefix-list style query control to whitelist or blacklist query targets
  • REST API with automatic, configurable OpenAPI documentation
  • Modern, responsive UI built on ReactJS, with NextJS & Chakra UI, written in TypeScript
  • Query multiple devices simultaneously
  • Browser-based DNS-over-HTTPS resolution of FQDN queries

To request support for a specific platform, please submit a Github Issue with the feature label.

Get Started →

Community

Any users, potential users, or contributors of hyperglass are welcome to join and discuss usage, feature requests, bugs, and other things.

hyperglass is developed with the express intention of being free to the networking community.

However, if you're feeling particularly helpful or generous, small donations are welcome.

Donate

Acknowledgements

hyperglass is built entirely on open-source software. Here are some of the awesome libraries used, check them out too!

GitHub