Infrastructure / Go / Applied AI

Brent Atchison

Infrastructure engineer. Full vertical slice - kernel TCP stats up to LLM prompt architecture. Go operators, distributed cache mesh, agent infrastructure. Off the clock: local LLMs and a homelab that keeps growing.

See the stack Projects
Austin, TX · Homelab online

About

Infrastructure engineer in Austin. Mostly Go: operators, microservices, distributed systems. Full vertical slice - from kernel TCP stats to LLM prompt architecture. Built an agent harness twice - Python first, then Go. Custom MCP servers, safe-eval sandboxes with AST allowlists, model routing with health-class failover. Side-channels keep full result sets out of the LLM context. Security in code, not prompt.

Observability that catches real signal - Prometheus recording rules, Grafana dashboards as code, Envoy access logs through ClickHouse to Prometheus gauges. CI that respects developer time - dependency-aware test matrices, timing-fed sharding. The goal is always the same: make the system legible enough that someone else can run it.

This site is static, served from Cloudflare's edge. Dynamic stuff lives on subdomains pointing back to the homelab through a Cloudflare Tunnel.

  • Go
  • Python
  • Kubernetes
  • kubebuilder
  • Operators
  • CRDs
  • gRPC
  • ConnectRPC
  • Prometheus
  • VictoriaMetrics
  • Grafana
  • ClickHouse
  • Vector
  • CockroachDB
  • Envoy
  • SWIM
  • HRW hashing
  • RDMA
  • TCP_INFO
  • Kafka
  • Vault
  • Doppler
  • GitHub Actions
  • ArgoCD
  • Kargo
  • Helm
  • MCP
  • FastMCP
  • Model routing
  • Local LLMs
  • Docker
  • Cloudflare Tunnel

Stack

What I reach for, by category.

Go & Kubernetes

  • Gooperators, ConnectRPC, pgx/v5
  • kubebuilderCRD design, CEL validation, admission webhooks
  • controller-runtimeenvtest, kind e2e, reconcile loops

Observability

  • Prometheusrecording rules, alerting
  • VictoriaMetricsfleet-scale, global query layer
  • Envoy + ClickHouseaccess logs to metrics, TTFB

Agents & MCP

  • Agent harnessPython -> Go, model routing, failover
  • MCP serverscustom for vendor APIs, Grafana, FS
  • Safe-evalAST allowlists, side-channels

CI/CD

  • GitHub Actionsdependency-aware matrix generation
  • Test shardingtiming-fed LPT bin-packing
  • Argo / Kargorelease-please, DevSpace, Helm

Data & Services

  • CockroachDBdistributed SQL, ClickHouse
  • gRPC / ConnectRPCservice APIs, Envoy, h2c
  • SecretsVault, Doppler, External Secrets

Python & Local AI

  • Pythonagent tooling, FastMCP
  • Local LLMsollama, LM Studio, GGUF
  • Model formatsMoE, quantization, local serving

Projects

Things I've built. Hit shuffle for a random tour.

Python / Go / AI

Agent harness

  • Sidecar runs python -I -S with AST allowlists - imports, dunders, open, subprocess all blocked. Model sees analyze_result, never run_analysis
  • Side-channels keep result sets out of context - CSVs to Slack, not the token window
  • Model routing: failover on health/capacity classes, streaming buffered per attempt, fallback sticks for the session
  • SQLite on NFS PVC: exclusive WAL, 64MiB cache - every page fault is a network hop
  • PR review: COMMENT is compile-time constant via stdin. Model structurally cannot approve.
Go / Distributed

Read cache mesh

  • Metadata always hits gateway first - bytes go backend-to-node. Authorization never bypassed.
  • SWIM + HRW rendezvous hashing. PlacementID decoupled from dial address - rolling restarts are non-events.
  • HTTP 507 when cache can't hold the working set
  • Auto-isolation: flapping node self-removes. Rejoin needs restart - flapping is worse than missing.
  • RDMA transport via NVIDIA cuObject when enabled. Prefetch = HEAD with Range: bytes=0-0.
Go / Storage

S3-compatible gateway

  • 100-byte body probe before committing status - honest 5xx, not truncated 200
  • Stale-tolerant metadata cache: warm hides outages, cold fails honestly
  • Keyset pagination with base64 continuation tokens. Common-prefix skipping in one DB hop.
  • Two DB layers: gorm serializable for writes, pgx codegen for reads
  • version_num = microsecond wall-clock epoch. Two PUTs same microsecond = 503.
Go / Networking

Kernel TCP telemetry

  • Reads TCP_INFO via getsockopt: RTT, retrans, cwnd, delivery rate, pacing rate
  • Three modes: client probes, server listener, internal peer mesh via DNS discovery
  • Peer-failure tolerance: bounded allowed-unreachable prevents false-positive alerts
  • Jittered probe loops desynchronize pods - prevents self-inflicted thundering herd
  • Linux-only. Per-node attribution via Prometheus labels.
Go / K8s

Kubernetes operators

  • Cache backing as first-class CRD field (hostPath / emptyDir / PVC) - not a Helm value
  • Observe-only mode: watch chart-owned deployments without writing child resources
  • CEL validation: hostPath regex, one volume source per dir, hostNetwork requires HostIP
  • Control plane only - operator declares config, never sits in the data path
  • Single-concurrency reconciles for ordering safety
Python / Go / MCP

MCP servers

  • Two read-only servers: 21 observability tools (Go/Gin, Streamable HTTP/SSE) + 5 storage tools (Python/FastMCP)
  • Concurrent fanout with per-cluster failure attribution - one bad cluster doesn't poison results
  • startup_policy: optional - degraded backend cannot block agent startup
  • Schema from Swagger with live sample fallback, cached in background so liveness probes don't kill the pod
  • Server-side credentials, writes disabled, read-only by design
Go / CI

Test matrix sharding

  • Dependency-aware: walks reverse-dependency closure across go.work, including stdlib
  • LPT bin-packing shards by historical timing from main branch metrics
  • Cut 586s suite into 6 drivers, module ceiling 15.2m to 11m
  • YAML/docs-only commits: 0 Go test jobs (was running all 31 modules every push)
Go / Infra

Storage controller

  • 2024 RFC authored solo: tie tenants to nodes, kill per-tenant VIP pools and the 32-tenant limit
  • Reconciles BareMetalNode CRs, binds node IP to cluster storage tenant
  • MaxConcurrent:1 for ordering safety, debounced 1s/60s
  • 30s deadline on storage API calls - fails fast, doesn't hang the queue
Go / AI

Skills system

  • Three-tier progressive disclosure: metadata ~100 tokens, body <5000, resources on demand
  • allowed-tools check before rendering - rejected activation leaks nothing
  • Immutable catalog behind atomic.Pointer - hot-swap without locks
  • Merges filesystem builtins, remote syncs, agent-authored skills
  • Bundle generator emits spec-compliant Claude Code plugin with auto-generated Related footer
Go / DevOps

Deployment system

  • Go CLI renders per-zone Helm values from typed structs - no template string hell, DO NOT EDIT headers
  • Cluster membership is declarative: one line per zone, regeneration emits ~31 files
  • Kargo DAG: Beta to Staging canary to Staging to Production canary to Production. Errors if no canary.
  • Expression filters skip commits that break Kargo's diff parser (60k-file rename)
Go / Observability

Observability pipeline

  • Envoy sidecar writes the access log, not the app. Dynamic metadata: S3 op, bucket, TTFB.
  • Vector to ClickHouse to chexporter to pre-aggregated Prometheus gauges
  • No _bucket series - histogram_quantile returns nothing. Threshold-on-gauge alerting instead.
  • Metric undershoots client latency: counts sub-100KiB reads only, peaked 7s while clients saw 29-30s
Static

batchison.dev

  • Plain HTML/CSS/JS, no build step
  • Cloudflare Pages with auto TLS
  • Rainbow dot-grid spotlight on hover
  • Magnetic chips, scroll reveal, FLIP shuffle
Source →

Off-Screen

The rest of it. Where the hours go when the laptop closes.

currently 
batchison@lab:~$ cat current_state.yml
homelab:
  os: Unraid
  compute: 8th gen Intel i5 NUC (32GB, 1TB SSD, USB 2.0 drive)
  storage: Synology DS1621+ (96TB raw)
  llm: [LM Studio, Ollama]
network:
  isp: AT&T Fiber
  gateway: UCG Fiber
  switching:
    - USW Pro XG 8 PoE
    - 3x USW Flex 2.5G 8 PoE
    - USW Flex 2.5G 8
  aps:
    - 2x U7 Pro Wall
    - U7 Lite
  backup_wan: [U5G Max, C4P Shield]
  dns: NextDNS
  ingress: Cloudflare Tunnels
desk:
  case: Jonsbo Z20 (SFF)
  cpu: Ryzen 9800X3D
  gpu: RTX 5090
  display: LG C3 42" OLED
batchison@lab:~$ _
Acoustics

3.2 listening room

I want to feel a wall of sound. Why use bass shakers when you can just make BASS?

Towers
Polk Monitor 75T → Ascend Sierra Towers
Bookshelves
Polk → Wharfedale Diamond 10.1
Center
Polk → Ascend ELX Horizon
Subs
Polk → SVS PB-2000 → 2x PSA S21 IPAL
The Lab

Self-hosted by default

Someone else's cloud is just a computer you can't touch. The rack hums in the corner, 96TB on the Synology, ingress through Cloudflare Tunnels.

The Garage

Hill country runs

2023 BMW M2 (6MT) after an MX-5, Mazda3, and RAV4. Performance first - CS tune and midpipe / valve controller on the list. Weekend clears, find the good roads.

Travel

Vacation favorites

  • Japan
  • South Korea
  • Vietnam
  • Caribbean

Chasing the next flight and the next bowl.

Games

Off the clock

Destiny, Halo, Cyberpunk, Civ VI, AC Black Flag. Zelda, Pokemon, Mario, DK - Nintendo in general. OSRS for the grind, Overwatch and MapleStory in the mix.

Listening

On rotation

Bass / EDM

Ganja White Night, LSDREAM, Of The Trees, ATLIENS, Illenium, Excision

Soundtracks

Martin O'Donnell / Michael Salvatori - Halo, Destiny

Deep work

Hans Zimmer, Hania Rani

Rock / pop

Phil Collins, Coldplay

Festivals

Lost Lands, Illfest

Kitchen

Always bulking

Sushi, steak, chicken wings, 8.85lb beef plate ribs, Yonezawa Wagyu. AllClad pans, overnight dry-brine, reverse sear with wagyu tallow. Oven and stainless steel - grill to come.

Contact

LinkedIn is the best way to reach me. GitHub works too.

LinkedIn in/brent-atchison GitHub github.com/atchisonbrent