Infrastructure / Go / Applied AI

Brent Atchison

Infrastructure engineer working the full vertical slice - kernel TCP stats up to LLM prompt architecture. Go operators, distributed caches, agent infrastructure. Off the clock: local LLMs and a homelab that keeps growing.

See the stack Projects
Austin, TX · Homelab online

About

Infrastructure engineer in Austin. Mostly Go - operators, microservices, distributed systems - with Python for agent tooling. The pattern that keeps repeating: take a fragile, manually-operated system and turn it into a declarative, observable surface someone else can run.

Lately that means agent infrastructure. Built a production agent harness twice - Python first, then ported onto a Go platform. Custom MCP servers, safe-eval sandboxes with AST allowlists, model routing with health-class failover, side-channels that keep full result sets out of the LLM context. Security lives in code, not in the prompt.

Before that: observability that catches real signal - Prometheus recording rules, z-score alerting, dashboards as code, access logs flowing through ClickHouse into pre-aggregated gauges. CI that respects developer time - dependency-aware test matrices, timing-fed sharding.

This site is static, served from Cloudflare's edge. Dynamic stuff lives on subdomains pointing back to the homelab through a Cloudflare Tunnel.

  • Go
  • Python
  • Kubernetes
  • kubebuilder
  • Operators
  • CRDs
  • CEL
  • envtest
  • gRPC
  • ConnectRPC
  • Prometheus
  • VictoriaMetrics
  • Grafana
  • ClickHouse
  • Vector
  • CockroachDB
  • Ceph
  • Envoy
  • SWIM
  • HRW hashing
  • RDMA
  • TCP_INFO
  • Kafka
  • Temporal
  • Vault
  • Doppler
  • GitHub Actions
  • ArgoCD
  • Kargo
  • Helm
  • MCP
  • FastMCP
  • Model routing
  • Local LLMs
  • Docker
  • Cloudflare Tunnel

Stack

What I reach for, by category.

Go & Kubernetes

  • Gooperators, ConnectRPC, pgx/v5
  • kubebuilderCRD design, CEL validation, admission webhooks
  • controller-runtimeenvtest, kind e2e, reconcile loops

Observability

  • Prometheusrecording rules, z-score alerting
  • VictoriaMetricsfleet-scale TSDB, global queries
  • Envoy + ClickHouseaccess logs to metrics, TTFB

Agents & MCP

  • Agent harnessmodel routing, health-class failover
  • MCP serversvendor APIs, observability, files
  • Safe-evalAST allowlists, side-channels

CI/CD

  • GitHub Actionsdependency-aware matrix generation
  • Test shardingtiming-fed LPT bin-packing
  • Argo / Kargorelease-please, DevSpace, Helm

Data & Services

  • DatabasesCockroachDB, ClickHouse, SQLite
  • gRPC / ConnectRPCservice APIs, Envoy, h2c
  • SecretsVault, Doppler, External Secrets

Python & Local AI

  • Pythonagent tooling, FastMCP
  • Local LLMsOllama, LM Studio, local serving
  • Model formatsMoE, GGUF, quantization

Projects

Things I've built. Hit shuffle for a random tour.

1 / 1
Python / Go / AI

Agent harness

  • Safe-eval sidecar - AST allowlists block imports, dunders, open, subprocess
  • Side-channels keep result sets out of the token window - CSVs go to Slack
  • Model routing: failover on health classes only, fallback sticks per session
  • PR review verb is a compile-time COMMENT - the model can't approve
Go / Distributed

Read cache mesh

  • Every request clears the gateway, bytes go backend-to-node
  • SWIM + HRW hashing - placement identity survives rolling restarts
  • Flapping nodes self-isolate until restart - flapping is worse than missing
  • HTTP 507 when the cache can't hold the working set; RDMA when enabled
Go / Storage

S3-compatible gateway

  • Body probe before committing status - honest 5xx, not a truncated 200
  • Stale-tolerant metadata cache - warm rides out outages, cold fails honestly
  • Keyset pagination - common prefixes skipped in a single DB hop
  • Versions are microsecond wall-clock - same-microsecond PUTs collide as 503
Go / Networking

Kernel TCP telemetry

  • TCP_INFO via getsockopt - RTT, retransmits, cwnd, delivery rate
  • Three modes: client probes, server listener, peer mesh via DNS discovery
  • Bounded allowed-unreachable peers - one blipping node can't page anyone
  • Jittered probe loops desynchronize pods - no self-inflicted thundering herd
Go / K8s

Kubernetes operators

  • Cache backing (hostPath / emptyDir / PVC) as a first-class CRD field
  • Observe-only mode watches chart-owned deployments, writes nothing
  • CEL at admission - path regexes, one volume source per directory
  • Control plane only - the operator never sits in the data path
Python / Go / MCP

MCP servers

  • 21 observability tools in Go, 5 storage tools in Python - all read-only
  • Concurrent fan-out - one bad cluster can't poison the whole batch
  • Optional startup policy - a degraded backend can't block boot
  • Schemas from Swagger with live-sample fallback, refreshed in the background
Go / CI

Test matrix sharding

  • Walks the reverse-dependency closure across go.work, stdlib included
  • LPT bin-packing shards tests by historical timing from main
  • 586s mega-suite split into six drivers - module ceiling 15.2m to 11m
  • Docs-only commits run zero test jobs - was all 31 modules per push
Go / Infra

Storage controller

  • Solo RFC - tie tenants to nodes, kill VIP pools and the 32-tenant cap
  • Reconciles bare-metal node CRs - node IP binds to its cluster's tenant
  • Single-concurrency reconciles with debounce - ordering safety over throughput
  • 30s deadline on storage API calls - fail fast, never hang the queue
Go / AI

Skills system

  • Three tiers: ~100-token metadata, bounded body, resources on demand
  • allowed-tools checked before rendering - rejected activations leak nothing
  • Immutable catalog behind atomic.Pointer - hot-swap without locks
  • Bundle generator emits a spec-compliant Claude Code plugin
Go / DevOps

Deployment system

  • Per-zone Helm values rendered from typed Go structs - no template hell
  • Declarative membership - one line per zone fans out ~31 rendered files
  • Promotion DAG errors out if staging or prod lacks a canary tier
  • Rendered plain YAML lives in git - SHAs are the versions
Go / Observability

Observability pipeline

  • Envoy writes the access log, not the app - metadata carries op, bucket, TTFB
  • Vector to ClickHouse to pre-aggregated Prometheus gauges
  • No _bucket series - threshold-on-gauge alerting instead
  • Sub-100KiB reads only - 7s peak while clients saw 30s. Know what a metric hides.
Static

batchison.dev

  • Plain HTML/CSS/JS - no framework, no build step, no dependencies
  • Cloudflare Pages with auto TLS, served from the edge
  • Rainbow dot-grid spotlight, magnetic chips, FLIP shuffle
  • Scroll restoration and theme persistence across reloads
Source →

Off-Screen

The rest of it. Where the hours go when the laptop closes.

currently 
batchison@lab:~$ cat current_state.yml
homelab:
  os: Unraid
  compute: 8th gen Intel i5 NUC (32GB, 1TB SSD, USB 2.0 drive)
  storage: Synology DS1621+ (96TB raw)
  llm: [LM Studio, Ollama]
network:
  isp: AT&T Fiber
  gateway: UCG Fiber
  switching:
    - USW Pro XG 8 PoE
    - 3x USW Flex 2.5G 8 PoE
    - USW Flex 2.5G 8
  aps:
    - 2x U7 Pro Wall
    - U7 Lite
  backup_wan: [U5G Max, C4P Shield]
  dns: NextDNS
  ingress: Cloudflare Tunnels
desk:
  case: Jonsbo Z20 (SFF)
  cpu: Ryzen 9800X3D
  gpu: RTX 5090
  display: LG C3 42" OLED
batchison@lab:~$ _

The Lab - Self-Hosted

Someone else's cloud is just a computer you can't touch. No open ports, no exposed IP.

Media
Plex, QuickSync hardware transcode
Series
Sonarr split by TV / 4K / anime
Films
Radarr split by HDR / standard
Front end
Organizr, Ombi, Tautulli
Remote
Tailscale / WireGuard
Wired
10G to every room

The Garage - Hill Country Runs

When the weekend clears, it's backroads and elevation.

Now
2023 BMW M2
Gearbox
Six-speed manual
Path
RAV4 → Mazda3 → MX-5 → M2
Planned
Tune and exhaust

Travel - Vacation Favorites

  • Japan
  • South Korea
  • Vietnam
  • Caribbean

Trips get planned around food first, sights second.

Kitchen - Always Bulking

Overnight dry brines, reverse sears, wagyu tallow on everything.

High score
8.85lb beef plate ribs
Kit
Oven and stainless
Next
Grill

Games - Off the Clock

Destiny and Halo above all.

Rotation
Cyberpunk, Civ VI, Black Flag
Nintendo
Zelda, Pokemon, Mario
Grind
OSRS
Hosted
Terraria server on the lab

Listening - On Rotation

Heavy bass
  • Ganja White Night
  • ATLiens
  • Apashe
  • Zeds Dead
  • KOAN Sound
Melodic bass
  • AVELLO
  • Trivecta
  • ILLENIUM
  • Said The Sky
  • Dabin
Funk / soul
  • GRiZ
  • Defunk
  • Maddy O'Neal
  • Elephant Heart
On repeat
  • Apashe - Lost In Mumbai
  • Zeds Dead - One Of These Mornings
  • ATLiens - FLOWERS
Scores
  • Martin O'Donnell
  • Michael Salvatori
  • Hans Zimmer
Remixes
  • NGHTMRE
  • Champagne Drip
  • ODESZA
  • Paper Skies
  • COASTR
Under the radar
  • Desren
  • Salvo
  • Dreamers Delight
  • Know Good
  • bardz
Festivals
  • Lost Lands
  • EDC
  • Electric Forest
  • Illfest

Acoustics - 3.2 Listening Room

Chasing a wall of sound. Skip the bass shakers - just make actual bass. Every slot has been upgraded at least once:

Towers
Polk Monitor 75T → Ascend Sierra Towers
Bookshelves
Polk → Wharfedale Diamond 10.1
Center
Polk → Ascend ELX Horizon
Subs
Polk → SVS PB-2000 → 2x PSA S21 IPAL
Receiver
Marantz NR1403 → Denon X3500H

Contact

LinkedIn is the best way to reach me. GitHub works too.

LinkedIn in/brent-atchison GitHub github.com/atchisonbrent