OBSERVABILITY, METRICS & DISTRIBUTED TRACING

Operating production edge security infrastructure requires real-time visibility. Hackergate emits granular Prometheus metrics, exports OpenTelemetry (OTel) traces across microservice boundaries, and formats structured JSON audit events adhering to modern observability standards.

Prometheus Metrics Engine (:9090/metrics)

Hackergate runs a dedicated, lightweight HTTP metrics server on port 9090 completely isolated from public dataplane traffic.

Metric IdentifierTypeLabelsDescription
hackergate_requests_totalCounterhost, status, protocol, vhostTotal count of processed requests partitioned by HTTP response code.
hackergate_request_duration_secondsHistogramvhost, backend_driverLatency distribution from TLS handshake receipt to final byte delivery.
hackergate_waf_blocked_totalCountervhost, rule_id, category, paranoiaCount of blocked requests partitioned by OWASP CRS rule ID and severity.
hackergate_gossip_events_totalCounterorigin_node, action, statusDecentralized threat gossip event volume sent and received across mesh.
hackergate_cache_hit_ratioGaugeshard_idThreat intelligence cache hit ratio across the 64 lock-striped shards.
hackergate_pow_challenges_totalCountervhost, result (issued/solved/failed)Cryptographic browser proof-of-work challenge state transitions.
hackergate_active_connectionsGaugeprotocol (http1, http2, http3)Current live client connections open on edge sockets.
PROMETHEUS SCRAPE CONFIGURATION (prometheus.yml)
scrape_configs: - job_name: 'hackergate_edge' scrape_interval: 10s scrape_timeout: 5s metrics_path: '/metrics' static_configs: - targets: - 'edge-fra-01.internal:9090' - 'edge-iad-01.internal:9090' - 'edge-sin-01.internal:9090' relabel_configs: - source_labels: [__address__] regex: '([^:]+):.*' target_label: instance

OpenTelemetry (OTel) Distributed Tracing

When downstream microservices use Jaeger, Grafana Tempo, or Datadog, Hackergate injects and propagates standard W3C traceparent headers. The gateway generates granular child spans measuring:

TLS Handshake Span

Measures crypto negotiation latency (TLS 1.3 0-RTT vs 1-RTT, ALPN protocol selection, and client cipher suite parsing).

WAF Inspection Span

Records execution duration for Coraza CRS Phase 1 (headers) and Phase 2 (body stream), tagged with evaluated rule IDs.

Upstream Proxy Span

Measures time spent establishing connections to backend origins, Time to First Byte (TTFB), and full response streaming.

/etc/hackergate/hackergate.toml [telemetry]
[telemetry] prometheus_enabled = true prometheus_addr = "0.0.0.0:9090" [telemetry.otlp] enabled = true endpoint = "otel-collector.internal:4317" protocol = "grpc" # grpc | http/protobuf sample_rate = 1.0 # Sample 100% of WAF blocks and 5% of clean traffic service_name = "hackergate-gateway" [logging] level = "info" # debug | info | warn | error format = "json" # json | text output = "/var/log/hackergate/access.log"

Structured JSON Access & Audit Logs

Hackergate produces machine-readable JSON logs ready for ingestion by Vector, FluentBit, Logstash, or OpenSearch:

AUDIT LOG RECORD EXAMPLE (POST /api/v1/auth/login)
{ "timestamp": "2026-11-09T10:14:22.842Z", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7", "client_ip": "198.51.100.42", "client_country": "DE", "host": "api.example.com", "method": "POST", "uri": "/api/v1/auth/login", "status": 403, "duration_ms": 1.24, "waf": { "action": "blocked", "paranoia_level": 2, "anomaly_score": 15, "matched_rules": [ { "id": 942100, "message": "SQL Injection Attack Detected via libinjection" }, { "id": 920270, "message": "Invalid character in request" } ] }, "bot_score": 12, "bytes_received": 528, "bytes_sent": 242 }

Container Health & Readiness Probes

For Kubernetes and cloud load balancer integration, Hackergate exposes unauthenticated probes on port 8080:

GET /healthz (Liveness)

Returns HTTP 200 immediately if the Go event loop and socket listeners are functional. Kubelet restarts container if this fails.

GET /readyz (Readiness)

Verifies that virtual host definitions are loaded, TLS certificates are valid, and at least one upstream origin is healthy.

← PREV: BOLA & GENAI LLM SHIELDNEXT: OPERATOR CLI REFERENCE (HGCTL) →