mirrors/thatmattlove-hyperglass
1
0
Fork 1
mirror of https://github.com/thatmattlove/hyperglass.git synced 2026-05-07 12:43:05 +00:00
Code Issues Activity

refactor: update Dockerfile and compose.yaml, remove unused files and streamline configurations

Browse source
This commit is contained in:
Wilhelm Schonfeldt 2025-10-07 21:08:44 +02:00
parent 4a1057651f
commit 9d7cf8c823
No known key found for this signature in database
GPG key ID: 9A15BF796D5C3F1E
6 changed files with 7 additions and 636 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

27
Dockerfile
View file

@ -6,32 +6,17 @@ ENV HYPERGLASS_PORT=8001
ENV HYPERGLASS_DEBUG=false
ENV HYPERGLASS_DEV_MODE=false
ENV HYPERGLASS_REDIS_HOST=redis
ENV HYPEGLASS_DISABLE_UI=false
ENV HYPEGLASS_DISABLE_UI=true
ENV HYPERGLASS_CONTAINER=true
COPY . .
FROM base AS ui
FROM base as ui
WORKDIR /opt/hyperglass/hyperglass/ui
RUN apk add --no-cache build-base pkgconfig cairo-dev nodejs npm \
gcc \
g++ \
musl-dev \
python3-dev \
libffi-dev \
openssl-dev \
jpeg-dev \
zlib-dev \
freetype-dev \
lcms2-dev \
openjpeg-dev \
tiff-dev \
tk-dev \
tcl-dev \
harfbuzz-dev \
fribidi-dev \
curl && sleep 2 && npm install -g pnpm && pnpm install -P
RUN apk add build-base pkgconfig cairo-dev nodejs npm
RUN npm install -g pnpm
RUN pnpm install -P
FROM ui AS hyperglass
FROM ui as hyperglass
WORKDIR /opt/hyperglass
RUN pip3 install -e .

134
TRACEROUTE_RESTRUCTURE_SUMMARY.md
View file

@ -1,134 +0,0 @@
# MikroTik Traceroute Enhancement - Restructured Implementation
## Overview
Restructured the MikroTik traceroute implementation to follow consistent naming conventions and architectural patterns used throughout the hyperglass codebase, specifically matching the BGP route plugin structure.
## Key Changes Made
### 1. Consistent Naming Convention ✅
- **OLD**: `mikrotik_traceroute_structured.py`
- **NEW**: `trace_route_mikrotik.py` (matches `bgp_route_mikrotik.py` pattern)
This follows the established pattern:
- `bgp_route_{platform}.py` for BGP parsing
- `trace_route_{platform}.py` for traceroute parsing
### 2. Platform-Specific Parsing in models/parsing/ ✅
- **Added**: `MikrotikTracerouteTable` and `MikrotikTracerouteHop` classes in `models/parsing/mikrotik.py`
- **Removed**: `MikroTikTracerouteParser` from generic `models/parsing/traceroute.py`
- Follows the same pattern as BGP routes where platform-specific parsing is in `models/parsing/{platform}.py`
### 3. Structured Data Model Enhancements ✅
Enhanced `TracerouteHop` model in `models/data/traceroute.py` with MikroTik-specific statistics:
```python
# MikroTik-specific statistics
loss_pct: Optional[int] = None # Packet loss percentage
sent_count: Optional[int] = None # Number of probes sent
last_rtt: Optional[float] = None # Last RTT measurement
avg_rtt: Optional[float] = None # Average RTT
best_rtt: Optional[float] = None # Best (minimum) RTT
worst_rtt: Optional[float] = None # Worst (maximum) RTT
```
### 4. BGP.tools Enrichment - Structured Only ✅
- **BEFORE**: Applied enrichment to text-based traceroute output
- **NOW**: Only applies to structured `TracerouteResult` objects
- Added reverse DNS lookup using Python's socket library
- Cleaner separation of concerns
### 5. UI Table Component Structure ✅
Created complete table structure for displaying traceroute data:
- `TracerouteTable` component following BGP table patterns
- `TracerouteCell` component for cell rendering
- `traceroute-fields.tsx` for field-specific formatting
- TypeScript types in `globals.d.ts`
## File Structure
```
hyperglass/
├── models/
│ ├── data/
│ │ └── traceroute.py # Enhanced TracerouteResult/TracerouteHop
│ └── parsing/
│ ├── traceroute.py # Generic traceroute parsers (removed MikroTik)
│ └── mikrotik.py # MikroTik-specific parsing + MikrotikTracerouteTable
├── plugins/_builtin/
│ ├── trace_route_mikrotik.py # NEW: MikroTik traceroute plugin (consistent naming)
│ └── bgptools_traceroute_enrichment.py # Updated: structured data only
└── ui/
├── components/output/
│ ├── traceroute-table.tsx # Table component
│ ├── traceroute-cell.tsx # Cell rendering
│ └── traceroute-fields.tsx # Field formatters
└── types/
└── globals.d.ts # TracerouteResult/TracerouteHop types
```
## Benefits of Restructuring
### 1. Consistency ✅
- Matches established BGP route plugin patterns
- Predictable file locations and naming
- Easier for developers to understand and maintain
### 2. Separation of Concerns ✅
- Platform-specific parsing isolated to `models/parsing/{platform}.py`
- Text-based vs structured output clearly separated
- Enrichment only applies where it makes sense (structured data)
### 3. Enhanced Data Model ✅
- Full MikroTik statistics preserved (Loss, Sent, Last, AVG, Best, Worst)
- Ready for BGP.tools ASN/organization enrichment
- Reverse DNS lookup integration
- JSON serializable for API responses
### 4. UI Table Ready ✅
- Complete table component structure
- Proper cell formatting for latency, loss, ASN
- Color coding for performance indicators
- Responsive design following existing patterns
## Table Display Format
```
Hop | IP Address | Hostname | ASN | Loss | Sent | Last | AVG | Best | Worst
1 | 192.168.1.1 | gateway.local | AS65001 (MyISP) | 0% | 3 | 1.2ms | 1.1ms | 0.9ms | 1.3ms
2 | 10.0.0.1 | core1.isp.com | AS1234 (BigISP) | 0% | 3 | 15.2ms | 14.8ms | 14.2ms | 15.5ms
3 | — | — | — | 100% | 3 | * | * | * | *
4 | 203.0.113.1 | transit.net | AS5678 (Transit) | 0% | 3 | 25.4ms | 26.1ms | 25.1ms | 27.8ms
```
## Testing Results ✅
Standalone parser test confirms:
- ✅ Correct parsing of MikroTik traceroute format
- ✅ Proper handling of timeouts and timeout aggregation
- ✅ MikroTik-specific statistics extraction
- ✅ Ready for structured data enrichment
## Next Steps
1. **DNS Tools Integration**: Could integrate dedicated DNS tools library for more robust reverse DNS lookups
2. **Additional Platforms**: Apply same pattern to other platforms (Cisco, Juniper, etc.)
3. **Performance Optimization**: Bulk BGP.tools queries for multiple IPs
4. **Caching**: Cache BGP.tools and DNS results to avoid repeated lookups
## Migration Notes
### Plugin Registration
Updated `plugins/_builtin/__init__.py`:
```python
from .trace_route_mikrotik import TraceroutePluginMikrotik # New
__all__ = (
# ... existing plugins ...
"TraceroutePluginMikrotik", # Added
)
```
### Execution Order
1. `trace_route_mikrotik.py` - Parse raw output to structured format
2. `bgptools_traceroute_enrichment.py` - Enrich structured data (common phase)
3. UI renders structured data in table format
This restructuring makes the traceroute functionality consistent, maintainable, and feature-rich while following established hyperglass patterns.

14
compose.yaml
View file

@ -1,21 +1,9 @@
networks:
Rede-LG-Hyperglass:
name: Rede-LG-Hyperglass
driver: bridge
ipam:
config:
- subnet: "172.29.1.24/29"
gateway: "172.29.1.25"
services:
redis:
image: "redis:alpine"
networks:
Rede-LG-Hyperglass:
hyperglass:
depends_on:
- redis
networks:
Rede-LG-Hyperglass:
environment:
- HYPERGLASS_APP_PATH=/etc/hyperglass
- HYPERGLASS_HOST=${HYPERGLASS_HOST-0.0.0.0}
@ -28,6 +16,6 @@ services:
- HYPERGLASS_ORIGINAL_APP_PATH=${HYPERGLASS_APP_PATH}
build: .
ports:
- "${IP_HOST_VM_VPS_DOCKER-0.0.0.0}:${HYPERGLASS_PORT-8001}:${HYPERGLASS_PORT-8001}"
- "${HYPERGLASS_PORT-8001}:${HYPERGLASS_PORT-8001}"
volumes:
- ${HYPERGLASS_APP_PATH-/etc/hyperglass}:/etc/hyperglass

313
debug_mikrotik_minimal.py
View file

@ -1,313 +0,0 @@
#!/usr/bin/env python3
"""Minimal debug script for MikroTik traceroute parsing without full hyperglass deps."""
import re
import typing as t
from dataclasses import dataclass
# Simulate just the parsing logic without all the hyperglass imports
@dataclass
class MikrotikTracerouteHop:
"""Individual MikroTik traceroute hop."""
hop_number: int
ip_address: t.Optional[str] = None
hostname: t.Optional[str] = None
loss_pct: t.Optional[int] = None
sent_count: t.Optional[int] = None
last_rtt: t.Optional[float] = None
avg_rtt: t.Optional[float] = None
best_rtt: t.Optional[float] = None
worst_rtt: t.Optional[float] = None
@property
def is_timeout(self) -> bool:
"""Check if this hop is a timeout."""
return self.ip_address is None or self.loss_pct == 100
@dataclass
class MikrotikTracerouteTable:
"""MikroTik Traceroute Table."""
target: str
source: str
hops: t.List[MikrotikTracerouteHop]
max_hops: int = 30
packet_size: int = 60
@classmethod
def parse_text(cls, text: str, target: str, source: str) -> "MikrotikTracerouteTable":
"""Parse MikroTik traceroute output with detailed debugging."""
# DEBUG: Log the raw input
print(f"=== RAW MIKROTIK TRACEROUTE INPUT ===")
print(f"Target: {target}, Source: {source}")
print(f"Raw text length: {len(text)} characters")
print(f"Raw text:\n{repr(text)}")
print(f"=== END RAW INPUT ===")
lines = text.strip().split("\n")
print(f"Split into {len(lines)} lines")
# DEBUG: Log each line with line numbers
for i, line in enumerate(lines):
print(f"Line {i:2d}: {repr(line)}")
# Find all table starts
table_starts = []
for i, line in enumerate(lines):
if ("Columns:" in line and "ADDRESS" in line) or (
"ADDRESS" in line
and "LOSS" in line
and "SENT" in line
and not line.strip().startswith(("1", "2", "3", "4", "5", "6", "7", "8", "9"))
):
table_starts.append(i)
print(f"Found table start at line {i}: {repr(line)}")
if not table_starts:
print("WARNING: No traceroute table headers found in output")
return MikrotikTracerouteTable(target=target, source=source, hops=[])
# Take the LAST table (newest/final results)
last_table_start = table_starts[-1]
print(
f"Found {len(table_starts)} tables, using the last one starting at line {last_table_start}"
)
# Determine format by checking the header line
header_line = lines[last_table_start].strip()
is_columnar_format = "Columns:" in header_line
print(f"Header line: {repr(header_line)}")
print(f"Is columnar format: {is_columnar_format}")
# Parse only the last table
hops = []
in_data_section = False
hop_counter = 1 # For old format without hop numbers
# Start from the last table header
for i in range(last_table_start, len(lines)):
line = lines[i].strip()
# Skip empty lines
if not line:
print(f"Line {i}: EMPTY - skipping")
continue
# Skip the column header lines
if (
("Columns:" in line)
or ("ADDRESS" in line and "LOSS" in line and "SENT" in line)
or line.startswith("#")
):
in_data_section = True
print(f"Line {i}: HEADER - entering data section: {repr(line)}")
continue
# Skip paging prompts
if "-- [Q quit|C-z pause]" in line:
print(f"Line {i}: PAGING PROMPT - breaking: {repr(line)}")
break # End of this table
if in_data_section and line:
print(f"Line {i}: PROCESSING DATA LINE: {repr(line)}")
try:
if is_columnar_format:
# New format: "1 10.0.0.41 0% 1 0.5ms 0.5 0.5 0.5 0"
parts = line.split()
print(f"Line {i}: Columnar format, parts: {parts}")
if len(parts) < 3:
print(f"Line {i}: Too few parts ({len(parts)}), skipping")
continue
hop_number = int(parts[0])
# Check if there's an IP address or if it's empty (timeout hop)
if len(parts) >= 8 and not parts[1].endswith("%"):
# Normal hop with IP address
ip_address = parts[1] if parts[1] else None
loss_pct = int(parts[2].rstrip("%"))
sent_count = int(parts[3])
last_rtt_str = parts[4]
avg_rtt_str = parts[5]
best_rtt_str = parts[6]
worst_rtt_str = parts[7]
elif len(parts) >= 4 and parts[1].endswith("%"):
# Timeout hop without IP address
ip_address = None
loss_pct = int(parts[1].rstrip("%"))
sent_count = int(parts[2])
last_rtt_str = parts[3] if len(parts) > 3 else "timeout"
avg_rtt_str = "timeout"
best_rtt_str = "timeout"
worst_rtt_str = "timeout"
else:
print(f"Line {i}: Doesn't match columnar patterns, skipping")
continue
else:
# Old format: "196.60.8.198 0% 1 17.1ms 17.1 17.1 17.1 0"
parts = line.split()
print(f"Line {i}: Old format, parts: {parts}")
if len(parts) < 6:
print(f"Line {i}: Too few parts ({len(parts)}), skipping")
continue
ip_address = parts[0] if not parts[0].endswith("%") else None
# Handle truncated IPv6 addresses that end with "..."
if ip_address and ip_address.endswith("..."):
print(
f"Line {i}: Truncated IPv6 address detected: {ip_address}, setting to None"
)
ip_address = None
if ip_address:
loss_pct = int(parts[1].rstrip("%"))
sent_count = int(parts[2])
last_rtt_str = parts[3]
avg_rtt_str = parts[4]
best_rtt_str = parts[5]
worst_rtt_str = parts[6] if len(parts) > 6 else parts[5]
else:
# Timeout line
loss_pct = int(parts[0].rstrip("%"))
sent_count = int(parts[1])
last_rtt_str = "timeout"
avg_rtt_str = "timeout"
best_rtt_str = "timeout"
worst_rtt_str = "timeout"
# Convert timing values
def parse_rtt(rtt_str: str) -> t.Optional[float]:
if rtt_str in ("timeout", "-", "0ms"):
return None
# Remove 'ms' suffix and convert to float
rtt_clean = re.sub(r"ms$", "", rtt_str)
try:
return float(rtt_clean)
except ValueError:
return None
if is_columnar_format:
# Use hop number from the data
final_hop_number = hop_number
else:
# Use sequential numbering for old format
final_hop_number = hop_counter
hop_counter += 1
hop_obj = MikrotikTracerouteHop(
hop_number=final_hop_number,
ip_address=ip_address,
hostname=None, # MikroTik doesn't do reverse DNS by default
loss_pct=loss_pct,
sent_count=sent_count,
last_rtt=parse_rtt(last_rtt_str),
avg_rtt=parse_rtt(avg_rtt_str),
best_rtt=parse_rtt(best_rtt_str),
worst_rtt=parse_rtt(worst_rtt_str),
)
hops.append(hop_obj)
print(
f"Line {i}: Created hop {final_hop_number}: {ip_address} - {loss_pct}% - {sent_count} sent"
)
except (ValueError, IndexError) as e:
print(f"Failed to parse line '{line}': {e}")
continue
print(f"Before deduplication: {len(hops)} hops")
# For old format, we need to deduplicate by IP and take only final stats
if not is_columnar_format and hops:
# For old format, we need to deduplicate by IP and take only final stats
print(f"Old format detected - deduplicating {len(hops)} total entries")
# Group by IP address and take the HIGHEST SENT count (final stats)
ip_to_final_hop = {}
ip_to_max_sent = {}
hop_order = []
for hop in hops:
# Use IP address if available, otherwise use hop position for truncated addresses
if hop.ip_address:
ip_key = hop.ip_address
elif hop.ip_address is None:
ip_key = f"truncated_hop_{hop.hop_number}"
else:
ip_key = f"timeout_{hop.hop_number}"
# Track first appearance order
if ip_key not in hop_order:
hop_order.append(ip_key)
ip_to_max_sent[ip_key] = 0
print(f"New IP discovered: {ip_key}")
# Keep hop with highest SENT count (most recent/final stats)
if hop.sent_count and hop.sent_count >= ip_to_max_sent[ip_key]:
ip_to_max_sent[ip_key] = hop.sent_count
ip_to_final_hop[ip_key] = hop
print(f"Updated {ip_key}: SENT={hop.sent_count} (final stats)")
print(f"IP order: {hop_order}")
print(f"Final IP stats: {[(ip, ip_to_max_sent[ip]) for ip in hop_order]}")
# Rebuild hops list with final stats and correct hop numbers
final_hops = []
for i, ip_key in enumerate(hop_order, 1):
final_hop = ip_to_final_hop[ip_key]
final_hop.hop_number = i # Correct hop numbering
final_hops.append(final_hop)
print(
f"Final hop {i}: {ip_key} - Loss: {final_hop.loss_pct}% - Sent: {final_hop.sent_count}"
)
hops = final_hops
print(f"Deduplication complete: {len(hops)} unique hops with final stats")
print(f"After processing: {len(hops)} final hops")
for hop in hops:
print(
f"Final hop {hop.hop_number}: {hop.ip_address} - {hop.loss_pct}% loss - {hop.sent_count} sent"
)
return MikrotikTracerouteTable(target=target, source=source, hops=hops)
if __name__ == "__main__":
# Test with the actual IPv6 traceroute output that has truncated addresses
mikrotik_output = """ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS
2001:43f8:6d1::71:114 0% 1 20ms 20 20 20 0
2620:0:1cff:dead:beef::5e0 0% 1 0.1ms 0.1 0.1 0.1 0
2620:0:1cff:dead:beef::30e3 0% 1 0.1ms 0.1 0.1 0.1 0
2a03:2880:f066:ffff::7 0% 1 0.2ms 0.2 0.2 0.2 0
2a03:2880:f163:81:face:b00c:0... 0% 1 0.1ms 0.1 0.1 0.1 0
2001:43f8:6d1::71:114 0% 2 0.9ms 10.5 0.9 20 9.6
2620:0:1cff:dead:beef::5e0 0% 2 0.1ms 0.1 0.1 0.1 0
2620:0:1cff:dead:beef::30e3 0% 2 0.2ms 0.2 0.1 0.2 0.1
2a03:2880:f066:ffff::7 0% 2 0.1ms 0.2 0.1 0.2 0.1
2a03:2880:f163:81:face:b00c:0... 0% 2 0ms 0.1 0 0.1 0.1
2001:43f8:6d1::71:114 0% 3 0.8ms 7.2 0.8 20 9
2620:0:1cff:dead:beef::5e0 0% 3 0.1ms 0.1 0.1 0.1 0
2620:0:1cff:dead:beef::30e3 0% 3 0.2ms 0.2 0.1 0.2 0
2a03:2880:f066:ffff::7 0% 3 0.1ms 0.1 0.1 0.2 0
2a03:2880:f163:81:face:b00c:0... 0% 3 0.1ms 0.1 0 0.1 0"""
print("Testing MikroTik IPv6 traceroute parser with truncated address...")
result = MikrotikTracerouteTable.parse_text(
mikrotik_output, "2a03:2880:f163:81:face:b00c:0:25de", "CAPETOWN_ZA"
)
print(f"\n=== FINAL RESULTS ===")
print(f"Target: {result.target}")
print(f"Source: {result.source}")
print(f"Number of hops: {len(result.hops)}")
for hop in result.hops:
print(
f" Hop {hop.hop_number}: {hop.ip_address or '<truncated>'} - {hop.loss_pct}% loss - {hop.sent_count} sent - {hop.avg_rtt}ms avg"
)

21
example_community_names_config.yaml
View file

@ -1,21 +0,0 @@
# Example configuration using the new 'name' mode for BGP communities
# This would typically go in your main config.yaml file
structured:
communities:
mode: name
names:
"65000:1000:0": "Upstream Any"
"65000:1001:0": "Upstream A (all locations)"
"65000:1101:0": "Upstream A Location 1"
"65000:1201:0": "Upstream A Location 2"
"65000:1002:0": "Upstream B (all locations)"
"65000:1102:0": "Upstream B Location 1"
"65000:2000:0": "IXP Any"
# With this configuration:
# - BGP communities that appear in the output and match the keys in 'names'
# will be displayed with friendly names appended
# - For example: "65000:1000:0" becomes "65000:1000:0,Upstream Any"
# - Communities without mappings remain unchanged
# - The frontend will display them as "65000:1000:0 - Upstream Any"

134
test_community_names.py
View file

@ -1,134 +0,0 @@
"""Test the new name mode for community validation."""
# Standard Library
import typing as t
from unittest.mock import Mock
# Third Party
from pydantic import ValidationError
# Project
from hyperglass.models.data.bgp_route import BGPRoute
from hyperglass.models.config.structured import StructuredCommunities
def test_community_validation_name_mode():
"""Test that name mode correctly appends friendly names to communities."""
# Mock the state to return our test configuration
from hyperglass import state
# Create a mock structured config with name mode
mock_structured = Mock()
mock_structured.communities = StructuredCommunities(
mode="name",
names={
"65000:1000:0": "Upstream Any",
"65000:1001:0": "Upstream A (all locations)",
"65000:1": "Test Community",
},
)
# Mock the params with our structured config
mock_params = Mock()
mock_params.structured = mock_structured
# Mock the use_state function to return our mock params
original_use_state = getattr(state, "use_state", None)
state.use_state = Mock(return_value=mock_params)
try:
# Test data for BGP route
test_data = {
"prefix": "192.0.2.0/24",
"active": True,
"age": 3600,
"weight": 100,
"med": 0,
"local_preference": 100,
"as_path": [65000, 65001],
"communities": [
"65000:1000:0", # Should get friendly name
"65000:1001:0", # Should get friendly name
"65000:9999:0", # Should remain unchanged (no mapping)
"65000:1", # Should get friendly name
],
"next_hop": "192.0.2.1",
"source_as": 65001,
"source_rid": "192.0.2.1",
"peer_rid": "192.0.2.2",
"rpki_state": 1,
}
# Create BGPRoute instance
route = BGPRoute(**test_data)
# Check that communities have been transformed correctly
expected_communities = [
"65000:1000:0,Upstream Any",
"65000:1001:0,Upstream A (all locations)",
"65000:9999:0", # No friendly name, stays unchanged
"65000:1,Test Community",
]
assert route.communities == expected_communities
finally:
# Restore original use_state function
if original_use_state:
state.use_state = original_use_state
def test_community_validation_permit_mode_unchanged():
"""Test that permit mode still works as before."""
from hyperglass import state
# Create a mock structured config with permit mode
mock_structured = Mock()
mock_structured.communities = StructuredCommunities(
mode="permit", items=["^65000:.*$", "1234:1"]
)
mock_params = Mock()
mock_params.structured = mock_structured
original_use_state = getattr(state, "use_state", None)
state.use_state = Mock(return_value=mock_params)
try:
test_data = {
"prefix": "192.0.2.0/24",
"active": True,
"age": 3600,
"weight": 100,
"med": 0,
"local_preference": 100,
"as_path": [65000, 65001],
"communities": [
"65000:100", # Should be permitted (matches ^65000:.*$)
"65001:200", # Should be denied (doesn't match patterns)
"1234:1", # Should be permitted (exact match)
],
"next_hop": "192.0.2.1",
"source_as": 65001,
"source_rid": "192.0.2.1",
"peer_rid": "192.0.2.2",
"rpki_state": 1,
}
route = BGPRoute(**test_data)
# Should only include permitted communities
expected_communities = ["65000:100", "1234:1"]
assert route.communities == expected_communities
finally:
if original_use_state:
state.use_state = original_use_state
if __name__ == "__main__":
test_community_validation_name_mode()
test_community_validation_permit_mode_unchanged()
print("All tests passed!")