Infrastructure / Go / Applied AI

Brent Atchison

Infrastructure engineer working the full vertical slice - kernel TCP stats up to LLM prompt architecture. Go operators, distributed caches, agent infrastructure. Off the clock: local LLMs and a homelab that keeps growing.

See the stack Projects
Austin, TX · Homelab online

About

Infrastructure engineer in Austin. Mostly Go - operators, microservices, distributed systems - with Python for agent tooling. The pattern that keeps repeating: take a fragile, manually-operated system and turn it into a declarative, observable surface someone else can run.

Lately that means agent infrastructure. Built a production agent harness twice - Python first, then ported onto a Go platform. Custom MCP servers, safe-eval sandboxes with AST allowlists, model routing with health-class failover, side-channels that keep full result sets out of the LLM context. Security lives in code, not in the prompt.

Before that: observability that catches real signal - Prometheus recording rules, z-score alerting, dashboards as code, access logs flowing through ClickHouse into pre-aggregated gauges. CI that respects developer time - dependency-aware test matrices, timing-fed sharding.

This site is static, served from Cloudflare's edge. Dynamic stuff lives on subdomains pointing back to the homelab through a Cloudflare Tunnel.

  • Go
  • Python
  • Kubernetes
  • kubebuilder
  • Operators
  • CRDs
  • gRPC
  • ConnectRPC
  • Prometheus
  • VictoriaMetrics
  • Grafana
  • ClickHouse
  • Vector
  • CockroachDB
  • Envoy
  • SWIM
  • HRW hashing
  • RDMA
  • TCP_INFO
  • Kafka
  • Vault
  • Doppler
  • GitHub Actions
  • ArgoCD
  • Kargo
  • Helm
  • MCP
  • FastMCP
  • Model routing
  • Local LLMs
  • Docker
  • Cloudflare Tunnel

Stack

What I reach for, by category.

Go & Kubernetes

  • Gooperators, ConnectRPC, pgx/v5
  • kubebuilderCRD design, CEL validation, admission webhooks
  • controller-runtimeenvtest, kind e2e, reconcile loops

Observability

  • Prometheusrecording rules, z-score alerting
  • VictoriaMetricsfleet-scale TSDB, global queries
  • Envoy + ClickHouseaccess logs to metrics, TTFB

Agents & MCP

  • Agent harnessmodel routing, health-class failover
  • MCP serversvendor APIs, observability, files
  • Safe-evalAST allowlists, side-channels

CI/CD

  • GitHub Actionsdependency-aware matrix generation
  • Test shardingtiming-fed LPT bin-packing
  • Argo / Kargorelease-please, DevSpace, Helm

Data & Services

  • DatabasesCockroachDB, ClickHouse, SQLite
  • gRPC / ConnectRPCservice APIs, Envoy, h2c
  • SecretsVault, Doppler, External Secrets

Python & Local AI

  • Pythonagent tooling, FastMCP
  • Local LLMsOllama, LM Studio, local serving
  • Model formatsMoE, GGUF, quantization

Projects

Things I've built. Hit shuffle for a random tour.

Python / Go / AI

Agent harness

  • Safe-eval sidecar - AST allowlists block imports, dunders, open, subprocess
  • Side-channels keep result sets out of the token window - CSVs go to Slack
  • Model routing: failover on health classes only, fallback sticks per session
  • PR review verb is a compile-time COMMENT - the model can't approve
Go / Distributed

Read cache mesh

  • Every request clears the gateway, bytes go backend-to-node
  • SWIM + HRW hashing - placement identity survives rolling restarts
  • Flapping nodes self-isolate until restart - flapping is worse than missing
  • HTTP 507 when the cache can't hold the working set; RDMA when enabled
Go / Storage

S3-compatible gateway

  • Body probe before committing status - honest 5xx, not a truncated 200
  • Stale-tolerant metadata cache - warm rides out outages, cold fails honestly
  • Keyset pagination - common prefixes skipped in a single DB hop
  • Versions are microsecond wall-clock - same-microsecond PUTs collide as 503
Go / Networking

Kernel TCP telemetry

  • TCP_INFO via getsockopt - RTT, retransmits, cwnd, delivery rate
  • Three modes: client probes, server listener, peer mesh via DNS discovery
  • Bounded allowed-unreachable peers - one blipping node can't page anyone
  • Jittered probe loops desynchronize pods - no self-inflicted thundering herd
Go / K8s

Kubernetes operators

  • Cache backing (hostPath / emptyDir / PVC) as a first-class CRD field
  • Observe-only mode watches chart-owned deployments, writes nothing
  • CEL at admission - path regexes, one volume source per directory
  • Control plane only - the operator never sits in the data path
Python / Go / MCP

MCP servers

  • 21 observability tools in Go, 5 storage tools in Python - all read-only
  • Concurrent fan-out - one bad cluster can't poison the whole batch
  • Optional startup policy - a degraded backend can't block boot
  • Schemas from Swagger with live-sample fallback, refreshed in the background
Go / CI

Test matrix sharding

  • Walks the reverse-dependency closure across go.work, stdlib included
  • LPT bin-packing shards tests by historical timing from main
  • 586s mega-suite split into six drivers - module ceiling 15.2m to 11m
  • Docs-only commits run zero test jobs - was all 31 modules per push
Go / Infra

Storage controller

  • Solo RFC - tie tenants to nodes, kill VIP pools and the 32-tenant cap
  • Reconciles bare-metal node CRs - node IP binds to its cluster's tenant
  • Single-concurrency reconciles with debounce - ordering safety over throughput
  • 30s deadline on storage API calls - fail fast, never hang the queue
Go / AI

Skills system

  • Three tiers: ~100-token metadata, bounded body, resources on demand
  • allowed-tools checked before rendering - rejected activations leak nothing
  • Immutable catalog behind atomic.Pointer - hot-swap without locks
  • Bundle generator emits a spec-compliant Claude Code plugin
Go / DevOps

Deployment system

  • Per-zone Helm values rendered from typed Go structs - no template hell
  • Declarative membership - one line per zone fans out ~31 rendered files
  • Promotion DAG errors out if staging or prod lacks a canary tier
  • Rendered plain YAML lives in git - SHAs are the versions
Go / Observability

Observability pipeline

  • Envoy writes the access log, not the app - metadata carries op, bucket, TTFB
  • Vector to ClickHouse to pre-aggregated Prometheus gauges
  • No _bucket series - threshold-on-gauge alerting instead
  • Sub-100KiB reads only - 7s peak while clients saw 30s. Know what a metric hides.
Static

batchison.dev

  • Plain HTML/CSS/JS - no framework, no build step, no dependencies
  • Cloudflare Pages with auto TLS, served from the edge
  • Rainbow dot-grid spotlight, magnetic chips, FLIP shuffle
  • Scroll restoration and theme persistence across reloads
Source →

Off-Screen

The rest of it. Where the hours go when the laptop closes.

currently 
batchison@lab:~$ cat current_state.yml
homelab:
  os: Unraid
  compute: 8th gen Intel i5 NUC (32GB, 1TB SSD, USB 2.0 drive)
  storage: Synology DS1621+ (96TB raw)
  llm: [LM Studio, Ollama]
network:
  isp: AT&T Fiber
  gateway: UCG Fiber
  switching:
    - USW Pro XG 8 PoE
    - 3x USW Flex 2.5G 8 PoE
    - USW Flex 2.5G 8
  aps:
    - 2x U7 Pro Wall
    - U7 Lite
  backup_wan: [U5G Max, C4P Shield]
  dns: NextDNS
  ingress: Cloudflare Tunnels
desk:
  case: Jonsbo Z20 (SFF)
  cpu: Ryzen 9800X3D
  gpu: RTX 5090
  display: LG C3 42" OLED
batchison@lab:~$ _
Acoustics

3.2 listening room

Chasing a wall of sound. Skip the bass shakers - just make actual bass. Every slot has been upgraded at least once:

Towers
Polk Monitor 75T → Ascend Sierra Towers
Bookshelves
Polk → Wharfedale Diamond 10.1
Center
Polk → Ascend ELX Horizon
Subs
Polk → SVS PB-2000 → 2x PSA S21 IPAL
The Lab

Self-hosted by default

Someone else's cloud is just a computer you can't touch. 96TB humming in the corner, local models on tap, ingress through Cloudflare Tunnels - no open ports, no exposed IP.

The Garage

Hill country runs

2023 BMW M2, six-speed manual - worked up through an MX-5, a Mazda3, and a RAV4 to get here. Tune and exhaust on the list. When the weekend clears, it's backroads and elevation.

Travel

Vacation favorites

  • Japan
  • South Korea
  • Vietnam
  • Caribbean

Trips get planned around food first, sights second.

Games

Off the clock

Destiny and Halo above all. Cyberpunk, Civ VI, Black Flag on rotation. Nintendo everything - Zelda, Pokemon, Mario. OSRS when the grind itch hits.

Listening

On rotation

Bass / EDM

Ganja White Night, LSDREAM, Of The Trees, ATLIENS, Illenium, Excision

Soundtracks

Martin O'Donnell / Michael Salvatori - Halo, Destiny

Deep work

Hans Zimmer, Hania Rani

Rock / pop

Phil Collins, Coldplay

Festivals

Lost Lands, Illfest

Kitchen

Always bulking

Overnight dry brines, reverse sears, wagyu tallow on everything. Current high score: 8.85lb of beef plate ribs. Oven and stainless for now - grill is next.

Contact

LinkedIn is the best way to reach me. GitHub works too.

LinkedIn in/brent-atchison GitHub github.com/atchisonbrent