Hermes

Hermes is the self-hosted Ubuntu server that runs my infrastructure: web apps and services in rootless containers, datasets behind a mixed SQL/Parquet storage layer, automation jobs, and a personal cloud, all hardened and monitored like a small production environment.

Everything I build eventually needs a home: web apps to host, datasets to keep, jobs that should run without me, files that should outlive my laptop. Hermes is that home, one small always-on box acting as my private cloud. Owning it shifts the question from the one every homelab guide answers, "is everything up?", to the one I actually cared about: is anyone on this box who shouldn't be, and what are they doing? My bet was that security signal is just telemetry. A process exec, a TCP connect and a failed SSH login are events with timestamps, and the same pipeline that carries application logs can carry them. So I built one pipeline and pointed everything at it. This post covers that pipeline: metrics, logs, eBPF and network IDS on a single small server, and what building it taught me.

01The architecture

The platform is rootless Podman with systemd quadlets, segmented into three network zones. The monitoring stack lives in the core zone, and telemetry is the only traffic allowed to flow into it.

        SENSORS                          PIPELINE                SINK

  host & containers ── node/podman/blackbox exporters ─┐
                                                       ├─ Prometheus ─┐
  runtime security ─── Tetragon (eBPF) ──┐             │              │
  typed commands ───── bpftrace readline ┼─ Promtail ──┼─ Loki ───────┼─ Grafana ── Telegram
  auth events ──────── journald tap ─────┘             │              │(dashboards
  network IDS ──────── Suricata + Zeek ──── Vector ────┘              │+alerting)
  traces ───────────── OTLP ──────────────────────────── Tempo ───────┘

Three layers of sensing, because they answer different questions:

• Health. Prometheus scrapes node-exporter (host), podman-exporter (container state), and blackbox-exporter, because "the process runs" and "the service answers HTTP 200" are different facts, and the gap between them is where outages hide.
• Host runtime. Tetragon traces process execs and TCP connections in the kernel, with one deliberate enforcement policy: SIGKILL on writes to /etc/ld.so.preload, the classic rootkit persistence path. Narrow enough to have zero false positives, so I never learn to ignore it. A small bpftrace probe on bash's readline() captures what humans actually type (exec tracing misses shell builtins entirely, a blind spot an intruder can sit in).
• Wire. Suricata and Zeek sniff the NIC. They see protocol metadata and signature hits but have no idea which process made the traffic; Tetragon knows the PID but only sees this host. Each covers the other's blind spot.

Everything lands in Grafana: six dashboards, four alert rules (high egress, SSH login, brute force, fail2ban ban) pushed to Telegram. All of it (quadlets, tracing policies, dashboards, alerts) is declarative YAML/JSON in git, deployed by one idempotent script.

02The decisions that mattered

Rootless by default, rootful only where the kernel demands it. Tetragon and the IDS engines need root (eBPF, raw sockets); everything else runs unprivileged. The rule that fell out of this became the design's backbone: rootful components write files, rootless components read and ship them. No privileged process ever holds a network path into the log store.

Loki over ELK. On a small ARM box, Elasticsearch was never an option. Loki indexes only labels and parses log bodies at query time, which fits the hardware, and forces a discipline that's valuable anyway: keep cardinality low, extract structure when you query.

Filter at the deepest layer that can see the noise. An observability stack that watches its own box mostly observes itself. The naive fix, regex tails on every dashboard panel, decays into unmaintainable copies. Instead each noise rule lives exactly once: internal-mesh chatter is dropped in the kernel by Tetragon selectors, the stack's own execs are dropped at the shipper, and only presentation concerns live in panels. One counter-rule: dual-use binaries like systemctl are never dropped. Tolerating a rare noisy line beats blinding every future query to systemctl enable persistence.service.

Alerts and dashboards have different noise budgets. The dashboard shows LAN-peer connections: I want to see them. The alert path suppresses them: a LAN neighbor is not an exfiltration suspect worth a phone notification. Same data, two intentionally different filters.

Pin images by digest. Not principle: a scar. tempo:latest silently jumped a major version with a breaking config schema on a routine redeploy. Updates are now a deliberate, reviewable act.

03What it taught me

Crash loops impersonate performance problems. The box showed 40% CPU and 72% iowait; the cause was a monitoring agent fatally exiting on an unsupported config field, restarting forever, and flooding the journal on a slow disk. "Is something restart-looping?" now comes before any performance theory.

The watcher needs watching. A single stuck eBPF policy silently blocked every policy loading after it, degrading the whole security layer while the agent's process looked healthy. Sensor state is a monitoring target like any other, and experimental probes belong in standalone tools where their blast radius is themselves.

Constraints produce better designs than conveniences. Rootless networking made publishing ports painful, so nothing publishes ports, and one reverse proxy is the single entry point. What started as a workaround is now the part of the architecture I'd defend first.

04What I'd do differently today

Promtail is deprecated; I'd consolidate it and Vector into Grafana Alloy. I'd add a dead-man's switch, because today a dead alerting path fails silent. Most importantly, I'd test the detections (synthetic failed logins, an Atomic Red Team-style trigger for the enforcement policy) and map coverage to MITRE ATT&CK so the gaps are chosen instead of accidental. And a switch mirror port would let the IDS layer see the whole LAN instead of one host.

05Further reading

• Google SRE Book, ch. 6, *Monitoring Distributed Systems*: the symptoms-vs-causes framing behind keeping only four alert rules.
• Rob Ewaschuk, *My Philosophy on Alerting*: why alert and dashboard filters diverge.
• Liz Rice, *Learning eBPF*: the best on-ramp to what Tetragon does under the hood.
• Brendan Gregg, *BPF Performance Tools*: origin of the bash-readline capture technique.
• Richard Bejtlich, *The Practice of Network Security Monitoring*: the collection-first mindset behind running Zeek alongside Suricata.
• Grafana docs, *Loki label best practices*: why low cardinality is the entire game.
• MITRE ATT&CK & Atomic Red Team: the coverage map and tests I'd retrofit next.