Keryx
The evidence-first Kubernetes incident investigator that never touches your cluster — every claim in Slack cited to the exact tool call that proves it, and a replayable query to check it yourself.
The problem
Every EKS incident starts the same way: a pager fires and someone half-awake
copy-pastes kubectl into a terminal, hunting for the one pod that's
crash-looping. Context is scattered across dashboards, stale runbooks, and
tribal memory. The slow part was never the fix — it was finding what to fix.
An LLM pointed at that problem makes it worse in a specific way: it produces a confident narrative you cannot check. So the bar isn't "can it diagnose" — it's can you verify the diagnosis at 2 a.m. without trusting it.
How it works
An Alertmanager webhook opens an investigation. A deterministic sweep snapshots
evidence before any model runs — evidence decays, and the sweep's named
checks (kube-crashes, recent-deploys, …) read as reproducible checks rather
than opaque agent steps. Then a hypothesis loop runs over read-only MCP servers
(Kubernetes, Prometheus, Grafana/Loki/Tempo, Flux, GitHub) and posts findings
into the alert's Slack thread.
Everything hangs off one rule: Keryx never writes to your cluster. Fixes are
proposed as GitOps pull requests against the Flux repo, never applied. Every
evidence item carries the tool call that produced it and a reproduce_hint —
the exact query you can re-run yourself.
Findings land on a conviction ladder — speculation → pattern-match → context-supported → validated-by-system-state → alternatives-ruled-out — computed in code from properties of the evidence. The model may argue a rung down, never up, and "insufficient evidence" is a valid verdict. Ruled-out hypotheses are shown with their reasons; empty or truncated tool output is recorded as evidence of failure rather than passed off as success.
Architecturally it's two services and one database: a TypeScript core-api (Hono, Zod schemas that generate the OpenAPI contract, SSE trace streaming) as the sole writer to Postgres + pgvector, and a stateless Python investigator on Strands that owns the agent loop and never touches the database. The Python client is generated from the OpenAPI spec — that codegen is what keeps two languages honest with each other. Models are bring-your-own-key with no Bedrock lock-in; keys stay customer-side, and Claude is the published-eval reference config, not a requirement.
What I'd change
M1–M6 are done and acceptance passed live on EKS against a purpose-built fake
production org (shopfront-hq) — its own repos, Helm charts, GitOps wiring, and
ephemeral Terraform-managed cluster, built solely to give Keryx something real
to be wrong about. Remaining before the Phase-1 gate: live sweep acceptance, the
demo slice, and hardening.
The differentiator I'd defend hardest is the one that's cheapest to skip: a published eval harness with chaos-injected scenarios scored per entity and fault type. Incumbents ship the tagline; almost nobody ships the numbers.