Reliability
The first question every team asks about putting a third party in their egress path: what happens when that third party has a bad day? Outbound’s answer is architectural, not aspirational — the forwarding path has zero runtime dependency on Outbound’s backend.
The data plane is self-contained
Section titled “The data plane is self-contained”Everything needed to move your packets lives in your account, on the gateway and in AWS itself:
- Routing is in your VPC route tables — AWS infrastructure, not ours.
- NAT and forwarding are the Linux kernel’s iptables and conntrack, configured locally on the gateway and re-applied automatically on every boot.
- The eBPF sensor runs locally and, by construction, cannot affect forwarding.
The gateway makes outbound HTTPS calls to report telemetry — but forwarding never waits on them, and never needs them.
What a control-plane outage looks like
Section titled “What a control-plane outage looks like”If Outbound’s control plane is unreachable — our outage, a network partition, anything:
| During the outage | |
|---|---|
| Your egress traffic | Flows normally. Nothing on the forwarding path involves our backend. |
| Existing gateways | Keep running; systemd restarts local services if they ever fail. |
| Flow telemetry | Buffered/degraded; gaps possible in the dashboard, traffic unaffected. |
| Configuration changes | Pause. Onboarding new subnets, resizes, and replacements wait until the control plane returns, then reconcile. |
The failure mode of losing the control plane is “the dashboard is stale and I can’t make changes”, never “my traffic stopped.”
Telemetry is fail-open
Section titled “Telemetry is fail-open”Every observability component degrades in favor of your traffic:
- The XDP program passes every packet unconditionally; if its buffers fill, records are skipped, packets are not.
- If the sensor can’t reach the ingest API or fetch its API key, it logs, keeps retrying, and the gateway keeps forwarding.
- Attribution enrichment (mapping IPs to application names) is optional at runtime — if it’s unavailable, flows are recorded without it rather than dropped.
Static egress IPs
Section titled “Static egress IPs”Outbound maintains a small pool of pre-allocated Elastic IPs in your account (tagged do-not-delete-cloudphilos-gateway). When a gateway is replaced, its public identity comes from this pool — and the control plane treats losing a static IP as a hard failure: an operation that can’t preserve your egress IP is aborted rather than completed with a new IP. Your partners’ IP allowlists survive every upgrade.
This is also why pooled EIPs must never be managed or deleted by your IaC — see Terraform & IaC.
Gateway replacement
Section titled “Gateway replacement”Upgrades, resizes, and AMI refreshes all use the same make-before-break sequence:
- Launch the new gateway instance and wait until it’s fully running.
- Attach the reserved Elastic IP / assign from the spare pool.
- Re-route: update the private subnets’ default routes to the new instance. New connections cut over immediately.
- Drain and delete the old instance.
The default route always targets a live gateway; there is no window where private subnets route to nothing.
Instance failure
Section titled “Instance failure”Gateways are single EC2 instances per public subnet (this is what makes their behavior simple and auditable). If a gateway instance fails:
- On-host service failures are handled by systemd auto-restart, and the NAT configuration reasserts itself on reboot.
- An unrecoverable instance failure is handled by the control plane as a replacement (the same sequence as above), preserving the egress IP.
- During the replacement window, new outbound connections from the affected subnets are interrupted.
For workloads where even that window matters, ask us about multi-AZ layouts — separate gateways per Availability Zone bound the blast radius of any single instance or AZ event to that zone’s subnets.
Our own blast radius, honestly stated
Section titled “Our own blast radius, honestly stated”Putting any device in the traffic path adds risk; our design goal is that the managed parts (control plane, analytics, telemetry) can fail without consequence, and the critical part (forwarding) is reduced to boring, battle-tested Linux primitives plus AWS routing — the same stack that has run NAT instances for over a decade.