Infrastructure / Go / Applied AI

Brent Atchison

Infrastructure engineer working the full vertical slice - kernel TCP stats up to LLM control planes. I build Go systems that fail honestly, agent infrastructure with real security boundaries, and a private home cloud substantial enough to have opinions.

See the stack Projects
Austin, TX · Homelab online

About

Infrastructure engineer building operators, distributed services, and storage and network control planes. Go is primary; Python handles agent tools. I start with failure modes - stale state, partial outages, bounded queues, and rollback - then build declarative, observable operational systems that stay honest when dependencies fail.

Current focus is agent infrastructure: MCP servers, sandboxed execution, portable open-source skills, and research tooling where the agent judges and stdlib code verifies. Security lives in code, not the prompt. Schemas, typed events, and read-only boundaries decide what an agent can actually do.

My homelab is the proving ground: dual-WAN and 10G networking, 69 TB of usable storage, private observability, media automation, and RTX 5090 inference that wakes on demand. Hermes reaches those systems through bounded APIs; private services stay behind Cloudflare Access and Tailscale, with secrets in Bitwarden and configuration in Git and OpenTofu.

  • Go
  • Python
  • Kubernetes
  • Operators
  • CEL
  • ConnectRPC
  • Prometheus
  • VictoriaMetrics
  • ClickHouse
  • Envoy
  • TCP_INFO
  • SWIM
  • HRW hashing
  • RDMA
  • GitHub Actions
  • Test sharding
  • Kargo
  • Helm
  • MCP
  • Sandboxed eval
  • Model routing
  • Context engineering
  • Local inference
  • LM Studio
  • Cloudflare Access
  • Tailscale
  • OpenTofu
  • Bitwarden Secrets
  • Beszel
  • UniFi

Stack

What I reach for, by category.

Go & Kubernetes

  • Gooperators, ConnectRPC, pgx/v5
  • kubebuilderCRD design, CEL validation, admission webhooks
  • controller-runtimeenvtest, kind e2e, reconcile loops

Observability

  • Prometheusrecording rules, z-score alerting
  • VictoriaMetricsfleet-scale TSDB, global queries
  • Envoy + ClickHouseaccess logs to metrics, TTFB

Agents & MCP

  • Agent Skillsone skill, four agents - pinned releases
  • MCP serversvendor APIs, observability, UniFi
  • Safe-evalAST allowlists, HMAC webhooks, side-channels

CI/CD

  • GitHub Actionsdependency-aware matrix generation
  • Test shardingtiming-fed LPT bin-packing
  • Argo / Kargorelease-please, DevSpace, Helm

Data & Services

  • DatabasesCockroachDB, ClickHouse, SQLite
  • gRPC / ConnectRPCservice APIs, Envoy, h2c
  • SecretsVault, Doppler, External Secrets

Python & Local AI

  • Hermes Agenttools, memory, cron, WebUI + Discord surfaces
  • LM StudioOpenAI-compatible serving, wake-on-demand relay
  • Model operationscatalogs, routing, GGUF, quantization

Projects

Things I've built. Hit shuffle for a random tour.

1 / 1
Python / JavaScript / Maps

Passage

  • Explore shipping activity around the world on an interactive globe
  • Look back at disruptions in places like the Suez Canal and Red Sea
  • Compare ports over time, with daily records going back to 2019
  • Share a view or download the numbers to dig into them yourself
Explore → Source →
Python / Go / AI

Agent harness

  • Safe-eval sidecar - AST allowlists block imports, dunders, open, subprocess
  • Side-channels keep result sets out of the token window - CSVs go to Slack
  • Model routing: failover on health classes only, fallback sticks per session
  • PR review verb is a compile-time COMMENT - the model can't approve
Python / AI / Open source

Deep research framework

  • Agent judges sources and confidence; stdlib Python enforces the contract
  • Quoted text verified against source extracts - stable IDs, rendered citations
  • Claim/evidence graph checked for consistency and confidence guardrails
  • Pinned releases install the same skill into Hermes, Claude Code, Codex, OpenCode
Source →
Python / Security / Homelab

Homelab alert relay

  • Appliances send static tokens, Hermes wants HMAC - the relay is the boundary
  • Producer prose is discarded - only typed source and incident class survive
  • fsync to an owner-only outbox before ack; the relay owns retry durability
  • Triage gets one read-only probe and no credentials; native alarms stay live
Go / Distributed

Read cache mesh

  • Every request clears the gateway, bytes go backend-to-node
  • SWIM + HRW hashing - placement identity survives rolling restarts
  • Flapping nodes self-isolate until restart - flapping is worse than missing
  • HTTP 507 when the cache can't hold the working set; RDMA when enabled
Go / Storage

S3-compatible gateway

  • Body probe before committing status - honest 5xx, not a truncated 200
  • Stale-tolerant metadata cache - warm rides out outages, cold fails honestly
  • Keyset pagination - common prefixes skipped in a single DB hop
  • Versions are microsecond wall-clock - same-microsecond PUTs collide as 503
Go / Networking

Kernel TCP telemetry

  • TCP_INFO via getsockopt - RTT, retransmits, cwnd, delivery rate
  • Three modes: client probes, server listener, peer mesh via DNS discovery
  • Bounded allowed-unreachable peers - one blipping node can't page anyone
  • Jittered probe loops desynchronize pods - no self-inflicted thundering herd
Go / K8s

Kubernetes operators

  • Cache backing (hostPath / emptyDir / PVC) as a first-class CRD field
  • Observe-only mode watches chart-owned deployments, writes nothing
  • CEL at admission - path regexes, one volume source per directory
  • Control plane only - the operator never sits in the data path
Python / Go / MCP

MCP servers

  • 21 observability tools in Go, 5 storage tools in Python - all read-only
  • Concurrent fan-out - one bad cluster can't poison the whole batch
  • Optional startup policy - a degraded backend can't block boot
  • Schemas from Swagger with live-sample fallback, refreshed in the background
Python / CV / Open source

SampleMatch

  • Local-first CLI - find one known subject across terabytes of footage
  • YOLO detection to ResNet embeddings on MPS or CUDA; pure-stdlib fallback
  • Calibrate on a 500-file --limit with --auto-tune before the hours-long scan
  • SQLite state, HTML/CSV review reports, symlink exports - originals untouched
Source →
Go / CI

Test matrix sharding

  • Walks the reverse-dependency closure across go.work, stdlib included
  • LPT bin-packing shards tests by historical timing from main
  • 586s mega-suite split into six drivers - module ceiling 15.2m to 11m
  • Docs-only commits run zero test jobs - was all 31 modules per push
Go / Infra

Storage controller

  • Solo RFC - tie tenants to nodes, kill VIP pools and the 32-tenant cap
  • Reconciles bare-metal node CRs - node IP binds to its cluster's tenant
  • Single-concurrency reconciles with debounce - ordering safety over throughput
  • 30s deadline on storage API calls - fail fast, never hang the queue
Go / AI

Skills system

  • Three tiers: ~100-token metadata, bounded body, resources on demand
  • allowed-tools checked before rendering - rejected activations leak nothing
  • Immutable catalog behind atomic.Pointer - hot-swap without locks
  • Bundle generator emits a spec-compliant Claude Code plugin
Go / DevOps

Deployment system

  • Per-zone Helm values rendered from typed Go structs - no template hell
  • Declarative membership - one line per zone fans out ~31 rendered files
  • Promotion DAG errors out if staging or prod lacks a canary tier
  • Rendered plain YAML lives in git - SHAs are the versions
Go / Observability

Observability pipeline

  • Envoy writes the access log, not the app - metadata carries op, bucket, TTFB
  • Vector to ClickHouse to pre-aggregated Prometheus gauges
  • No _bucket series - threshold-on-gauge alerting instead
  • Sub-100KiB reads only - 7s peak while clients saw 30s. Know what a metric hides.
Infra / Security / IaC

Homelab control plane

  • Exact-host Cloudflare Access guards every published private service
  • Tailnet-only origins reject the same probes over the LAN
  • OpenTofu owns tunnel, DNS, Access, and R2-backed state
  • Bitwarden writes file-backed secrets; Git and Docker metadata stay clean
Containers / Observability

Private operations portal

  • Homepage presents; Beszel keeps host telemetry and history
  • Digest-pinned containers run read-only with no capabilities
  • GET-only Docker proxy and isolated SMART cache avoid root authority
  • Cloudflare SSO maps roles; password auth stays available for recovery
AI / Networking

Wake-on-demand local inference

  • Sleep-safe catalog queries do not wake the RTX 5090 workstation
  • Inference sends Wake-on-LAN, waits for LM Studio, then streams through
  • Loopback relay stores no API credential and exposes no LAN listener
  • One OpenAI-compatible endpoint hides power state, not startup latency
Static

batchison.dev

  • Plain HTML/CSS/JS - no framework, no build step, no dependencies
  • Cloudflare Pages at the edge - immutable CSS, hook-enforced cache keys
  • Scroll-driven reveals, view-transition shuffle, light-dark() theming
  • Hero glyph physics, dot-grid spotlight, no-JS and reduced-motion complete
Source →

Off-Screen

The rest of it. Where the hours go when the laptop closes.

currently 
batchison@lab:~$ cat current_state.yml
homelab:
  os: Unraid
  compute: Intel i5 / 32GB / 1TB NVMe
  storage: Synology DS1621+ / 69 TB usable
  observability: [Homepage, Beszel, UniFi]
  agent_host: M2 Mac Mini / Hermes Agent
  local_ai: RTX 5090 / LM Studio / wake relay
network:
  isp: AT&T Fiber
  gateway: UCG Fiber
  switching:
    - USW Pro XG 8 PoE
    - 3x USW Flex 2.5G 8 PoE
    - USW Flex 2.5G 8
  aps:
    - 2x U7 Pro Wall
    - U7 Lite
  backup_wan: U5G Max / T-Mobile
  dns: NextDNS
  ingress: Cloudflare Access + Tunnels
desk:
  case: Jonsbo Z20 (SFF)
  cpu: Ryzen 9800X3D
  gpu: RTX 5090
  display: LG C3 42" OLED
batchison@lab:~$ _

The Garage - Hill Country Runs

When the weekend clears, it's backroads and elevation.

Now
2023 BMW M2
Gearbox
Six-speed manual
Path
RAV4 → Mazda3 → MX-5 → M2
Planned
Tune and exhaust

Travel - Vacation Favorites

  • Japan
  • South Korea
  • Vietnam
  • Caribbean

Trips get planned around food first, sights second.

Kitchen - Always Bulking

Overnight dry brines, reverse sears, wagyu tallow on everything.

High score
8.85lb beef plate ribs
Kit
Oven and stainless
Next
Grill

Games - Off the Clock

Destiny and Halo above all.

Rotation
Cyberpunk, Civ VI, Black Flag
Nintendo
Zelda, Pokemon, Mario
Long haul
Destiny, Halo, OSRS

Reading - Far Futures

Big civilizations, strange machines, and enough dry wit to survive both.

Current
The Culture
Author
Iain M. Banks
Rabbit hole
Minds, orbitals, Special Circumstances
Writing
fiction-workshop, a continuity-aware story toolchain

Listening - On Rotation

Heavy bass
  • Ganja White Night
  • ATLiens
  • Apashe
  • Zeds Dead
  • KOAN Sound
Melodic bass
  • AVELLO
  • Trivecta
  • ILLENIUM
  • Said The Sky
  • Dabin
Funk / soul
  • GRiZ
  • Defunk
  • Maddy O'Neal
  • Elephant Heart
On repeat
  • Apashe - Lost In Mumbai
  • Zeds Dead - One Of These Mornings
  • ATLiens - FLOWERS
Scores
  • Martin O'Donnell
  • Michael Salvatori
  • Hans Zimmer
Remixes
  • NGHTMRE
  • Champagne Drip
  • ODESZA
  • Paper Skies
  • COASTR
Under the radar
  • Desren
  • Salvo
  • Dreamers Delight
  • Know Good
  • bardz
Festivals
  • Lost Lands
  • EDC
  • Electric Forest
  • Illfest

Acoustics - 3.2 Listening Room

A wall of sound without bass-shaker theater. Current system, not the archaeological record:

L/R
Ascend Sierra Towers, RAAL ribbons
Center
Ascend ELX Horizon
Subs
2x PSA S21 IPAL
Control
Denon X3500H, Audyssey XT32
Second zone
Wharfedale karaoke zone

Contact

LinkedIn is the best way to reach me. GitHub works too.

LinkedIn in/brent-atchison GitHub github.com/atchisonbrent