Echofront logo

Rust ยท resilient layer 7 proxy

Break a backend,
watch it heal.

Echofront is a resilience-focused layer 7 reverse proxy and load balancer built from the Rust standard library only, with zero external dependencies. Its whole job is the hard part of fronting a pool of backends: spreading load well, noticing when a backend goes bad, taking it out of rotation, retrying somewhere healthy without amplifying an incident, and letting it back in when it recovers. This playground runs that same logic live, so you can flip a node to failing and watch its breaker open, the outlier get ejected, retries fail over, and recovery ramp back through half-open to closed.

Open the playground View on GitHub
How to use this playground

Everything below is live. The clock advances on its own and every request flows through the same rules the real proxy uses. Here is what each control does.

  • Strategy picks how the proxy chooses a node: round robin, weighted (smooth weighted round robin), least connections, or sticky (consistent hashing). Choosing sticky reveals a key field.
  • Sticky key is the value hashed onto the ring. The same key always lands on the same node until that node leaves the pool. Leave it empty and the stream auto cycles through user-1 to user-8.
  • Send 1 request and Send 10 fire traffic instantly. Send 10 is the fastest way to see the spread across nodes.
  • Start stream sends one request roughly every 400 milliseconds of simulated time. Stop stream halts it.
  • Reset clears all counters, breakers, ejections, and the log.
  • Each node card has a Healthy or Failing toggle. Flip a node to Failing and its responses start returning 500 and its health probe drops it.
  • The breaker badge shows Closed (green), Half Open (amber), or Open (red). The Ejected badge appears during passive outlier ejection.

Scenarios to try

  • See the spread: pick a strategy, press Send 10, and compare the traffic share bars.
  • Watch an ejection: mark a node Failing, start a stream, and watch five consecutive failures eject it for five seconds.
  • Watch a breaker open: the same failing node opens its breaker after three consecutive failures, then stops receiving traffic.
  • Watch failover: in the log, a failed attempt retries onto a different node using the retry budget.
  • Watch recovery: flip the node back to Healthy, wait for the cooldown, and see the breaker go Half Open then Closed after two trial successes.

Guided tour

  1. Leave the strategy on round robin and press Send 10. Notice traffic spread evenly across all four nodes.
  2. Switch to weighted and press Send 10 again. Node C carries about twice the share because its weight is 2.
  3. Switch to least connections, press Start stream, and watch in flight counts stay balanced.
  4. While streaming, flip node C to Failing. Watch its breaker count failures and open, and each failed attempt fail over to another node.
  5. Keep watching. After five straight failures node C shows the Ejected badge and leaves the pool entirely.
  6. Flip node C back to Healthy. After the ejection window and the breaker cooldown it becomes Half Open, takes trial requests, and returns to Closed.
  7. Switch to sticky, type a key like user-3, and press Send 1 a few times. It sticks to one node. Fail that node and only its keys remap.

Controls

Live stats

0
Requests
0
Successes
0
Failures
0
Retries
0
Failovers
0
Retry tokens
0
Clock (ms)

Backend pool

Request log

Never selects an unavailable backend

Because all timing goes through a Clock trait and all backends through an Upstream trait, the resilience logic itself is the product, and its correctness is proven by seeded, deterministic gates in tests/gates.rs rather than hoped for in production. The same seed produces the same timeline every run.

# tests/gates.rs, seeded and deterministic
load balancing correctness    round robin exact cycle, weighted within tolerance,
                              consistent hash remaps only ~1/N on a membership change
circuit breaker state machine agrees with an independent reference model; Open is never callable
health over an injected clock  ejected within the window, reinstated on recovery
core invariant                never selects unhealthy, ejected, or open-circuit
EWMA latency scoring          slowest node ejected, reinstated at a ramping weight

# change the seed, replay the whole timeline:
ECHOFRONT_FUZZ_SEED=7 cargo test

There is not a single real socket in the tests. Retries fail over to a healthy backend, but a token bucket caps the amplification, so a small incident never becomes a retry storm. A backend at weight 0 receives no traffic, an all-zero-weight pool honestly reports no capacity, and a Half-Open breaker admits only its configured number of trial calls so the trial gets a clean signal.

Break a backend, watch it heal

The demo subcommand runs a scripted incident on a manual clock. A node starts failing, gets ejected as an outlier, its breaker trips Open, then it recovers back through Half-Open to Closed. This is a real transcript, trimmed.

$ cargo run --release -- demo

# 2) C starts failing. After 3 consecutive failures it is ejected as an
#    outlier and traffic fails over to A and B (breaker still Closed):
  200 /checkout?o=5 -> shop  [C(server error 503 FAIL) retry->A(200 ok)]
  200 /checkout?o=7 -> shop  [C(server error 503 FAIL) retry->B(200 ok)]
  node    healthy   in_fl   breaker    ejected   served
  C          true       0    Closed        yes        6

# 3) C reinstated, still failing, so its circuit breaker trips Open:
  node    healthy   in_fl   breaker    ejected   served
  C          true       0      Open         no        8

# 4) Cooldown, Half-Open trial, C has recovered, two successes Close it:
  200 /catalog?r=0 -> shop  [C(200 ok)]

How it compares

Echofront is not trying to replace a production proxy. It isolates just the resilience layer and makes it readable and provable.

Envoy, HAProxy, nginx

Production proxies that run real traffic at scale, with rich routing, TLS termination, and observability. Their resilience logic is battle tested, and it lives inside large C or C++ codebases that you exercise against real servers and real clocks.

Echofront

The readable, from scratch version of only the resilience layer: load balancing, circuit breaking, outlier ejection, EWMA scoring, and capped retries, in dependency free Rust. All timing runs over an injected clock and every backend over a trait, so the same seed replays the same incident and the correctness gates give a clear pass or fail.

The primitives it gets right

Each of these is a real mechanism with a load bearing gate behind it in tests/gates.rs.

balancing Smooth weighted round robin

Weighted selection spreads traffic in an even interleaved cycle rather than in bursts. A node at weight 0 receives nothing, and an all zero weight pool honestly reports no capacity.

balancing Consistent hash sticky

An FNV-1a ring pins each key to the same node until that node leaves the pool, and a membership change remaps only about 1/N of keys.

breaker Circuit breaker

A per upstream Closed, Open, Half-Open machine. It trips Open after consecutive failures and admits only its configured number of trial calls in Half-Open, so the trial gets a clean signal.

health Outlier ejection

A backend that fails repeatedly is taken out of rotation for a fixed window and reinstated on schedule, with a continuous streak counted exactly once.

health EWMA latency scoring

An exponentially weighted moving average of measured latency ejects a sustained slow node, and a reinstated node ramps back at partial weight in smooth proportion.

retries Retry budget

A token bucket caps retry amplification, so failing over to a healthy backend never turns a small incident into a retry storm.

Use it

# a scripted incident on a manual clock: spread, eject, breaker, recover
cargo run --release -- demo

# distribution report for any strategy
cargo run --release -- spread weighted
cargo run --release -- spread round-robin
cargo run --release -- spread least-conn
cargo run --release -- spread sticky

# the correctness gates and the lints
cargo test
cargo clippy --all-targets -- -D warnings