Development Update — July 23

Two threads today, both about stripping dependencies out from under a running system. The VPN router’s control plane started moving off shell-outs to ip and iptables and onto native Go — netlink and nftables spoken directly to the kernel — which is the first real step toward shipping Skywire as a dependency-free gokrazy appliance that carries no iproute2 or userland at all. Separately, the long-running TPD reward outage got its first concrete fix: the CXO aggregator that receives every visor’s Roots was never pruning the superseded ones, and on the production TPD host that was leaking toward OOM at roughly a gigabyte a day.

Skywire: A Go-Native VPN-Router Control Plane

3556 feat(vpn): Go-native control plane — netlink (default) + nftables firewall (tag-gated) replaces the router’s ip and sysctl shell-outs with a netlink-plus-/proc backend, pkg/vpn/netctl. Talking netlink directly is exactly what ip does under the hood — identical kernel behavior — but with no iproute2 dependency, no PATH assumptions, and no per-command pkexec/sudo, which kills the polkit prompt flood on non-root hosts. More to the point, it lets the router run on a userland-less appliance: gokrazy ships no ip or sysctl binaries at all. The data plane was already Go-native (tun via songgao/water, DHCP and DNS via the vendored gokrazy router7); this converts the control plane, mapping netctl one-to-one to every ip/sysctl site — address add/flush/del, link up and MTU, route add/replace/del, default-dev routes, iif policy rules, and IPv4/IPv6 forwarding toggles — and reproducing the old stderr handling (File exists → ok, Operation not permitted → denied) via EEXIST/EPERM errno checks. The nftables firewall backend rides along tag-gated, validated against a real kernel in a rootless user-plus-net namespace (unshare -rn) so it exercises actual netlink with no root and no host impact. 3558 feat(vpnrouter): firewall backend seam (iptables | nftables) — completes Phase 0b then puts a backend seam under the firewall rules themselves. The router’s masquerade, FORWARD-accept, TCPMSS-clamp, and mesh-gateway REDIRECT rules now dispatch through a small firewall interface with two build-tagged backends: the default (!nftfw) path is byte-identical iptables — same rule specs, same reversed -D teardown, so the fleet is untouched — while the nftfw path delegates to the pure-Go nftables backend from #3556 (one skywire table, Teardown drops it), which is what makes the arm64 gokrazy image’s firewall actually functional since the appliance ships no iptables. The Router now holds one firewall and lets the backend own its teardown, replacing the scattered forwardRules/mssClamped/meshRedirect/masquerading state; the nft backend it delegates to is already validated end-to-end on an Orange Pi Prime (aarch64, kernel 5.15). It’s fleet-safe by construction — the default build still compiles the iptables path.

Skywire: The First TPD Leak Fix

3561 fix(tpd): prune superseded CXO roots in the aggregator (memory leak) is the first concrete fix in the TPD reward-outage investigation. On the production TPD host, svc tpd had climbed to roughly 3.8 GB RSS with 8.5M live objects after 13 hours — the host 88% full — growing about 5 MB/minute toward OOM, and heap profiling put almost all of it in accumulated received CXO Roots: receiveMsg → DeserializeRawToValue, the in-memory CXDS store, and the wanted-object cache. The root cause is that TPD’s CXO aggregator subscribes to every visor’s feed and fills every Root they publish, dispatching each filled Root to redis — but its loop only ran reconcile; it never pruned superseded Roots. Every visor republishes a new Root on each transport change, and TPD kept them all, plus every object, pinned in its in-memory store forever. The three TPD publishers already self-clean on a ticker, but the aggregator builds a separate CXO node for the receive side and simply lacked that loop. The fix gives it the same cleanup on a CleanupInterval ticker (default 2m): keep only the newest Root per feed and sweep the freed objects. It’s safe by construction — the store is ephemeral (redis is authoritative, each Root is dispatched on fill), and the prune primitive only deletes Roots older than the latest, so an actively-filling newest Root is never touched. Once the deployment picks it up, TPD reclaims the ~3 GB and stops re-accumulating. As it turned out, this was necessary but not sufficient — the residual leak and the reward outage itself are the next day’s story.