Hermes

Hermes centralizes my homelab on one self-hosted Ubuntu box: databases (mixed SQL/Parquet), compute (web apps and jobs in rootless containers), and network security monitoring, all behind one telemetry pipeline.

01Architecture

Platform: rootless Podman, systemd quadlets, three network zones. Monitoring lives in the core zone; only telemetry crosses into it.

        SENSORS                          PIPELINE                SINK

  host & containers ── node/podman/blackbox exporters ─┐
                                                       ├─ Prometheus ─┐
  runtime security ─── Tetragon (eBPF) ──┐             │              │
  typed commands ───── bpftrace readline ┼─ Promtail ──┼─ Loki ───────┼─ Grafana ── Telegram
  auth events ──────── journald tap ─────┘             │              │(dashboards
  network IDS ──────── Suricata + Zeek ──── Vector ────┘              │+alerting)
  traces ───────────── OTLP ──────────────────────────── Tempo ───────┘

Three sensing layers:

• Health. Prometheus: node-exporter, podman-exporter, blackbox-exporter. Process alive vs. service answering 200 are different facts.
• Host runtime. Tetragon traces execs and TCP connects in-kernel. One enforcement rule: SIGKILL on writes to /etc/ld.so.preload (rootkit persistence path). bpftrace on bash readline() catches shell builtins exec tracing misses.
• Wire. Suricata + Zeek on the NIC: protocol metadata, signature hits, no process attribution. Covers Tetragon's blind spot; Tetragon covers theirs.

Six dashboards, four alert rules (egress, SSH login, brute force, fail2ban ban) to Telegram. All config declarative YAML/JSON in git, one idempotent deploy script.

02Decisions

• Rootless by default. Only Tetragon and the IDS engines run rootful (eBPF, raw sockets need it). Rule: rootful writes files, rootless reads and ships them. No privileged process holds a network path into the log store.
• Loki over ELK. Elasticsearch doesn't fit a small ARM box. Loki indexes labels only, parses bodies at query time: low cardinality, structure extracted at query time.
• Filter at the deepest layer that can see the noise. Internal-mesh chatter dropped in-kernel via Tetragon selectors; the stack's own execs dropped at the shipper; dashboard panels carry presentation only, no per-panel regex tails. Exception: dual-use binaries (systemctl) are never dropped — a noisy line beats a blind query for systemctl enable persistence.service.
• Alerts and dashboards use different noise budgets. Dashboard shows LAN-peer connections. Alert path suppresses them: not exfiltration-worthy.
• Images pinned by digest. tempo:latest jumped a major version with a breaking schema on a routine redeploy. Updates are now deliberate.

03Failures

• 40% CPU / 72% iowait was a monitoring agent crash-looping on a bad config field, flooding the journal. Check "is something restart-looping" before any performance theory.
• One stuck eBPF policy silently blocked every policy load after it; the security layer degraded while the agent looked healthy. Sensor state is itself a monitoring target.
• Rootless networking blocks port publishing outright, so one reverse proxy is the only entry point — a constraint, not a workaround.

04Next

Promtail deprecated: fold it and Vector into Grafana Alloy. Add a dead-man's switch (today a dead alert path fails silent). Test the detections: synthetic failed logins, an Atomic Red Team trigger for the enforcement policy. Map coverage to MITRE ATT&CK. Add a switch mirror port so the IDS sees the whole LAN, not one host.

05Further reading

• Google SRE Book, ch. 6, *Monitoring Distributed Systems*
• Rob Ewaschuk, *My Philosophy on Alerting*
• Liz Rice, *Learning eBPF*
• Brendan Gregg, *BPF Performance Tools*
• Richard Bejtlich, *The Practice of Network Security Monitoring*
• Grafana docs, *Loki label best practices*
• MITRE ATT&CK & Atomic Red Team