3 hours ago
At Avetrix Labs, I’m building Neural Void, an app that sends requests to different AI providers. Here’s a change I’d like to evaluate before shipping: suppose we shorten the time the router waits for provider A before trying B. More users might get a fast answer, but we could quietly send more traffic to a pricier model. A green healthcheck wouldn’t tell me whether that trade-off was worth it.
Could we build a small, reusable Railway example to help teams make that decision? My starting point would be a PR environment running the candidate router, with a test runner that sends the same synthetic requests through both the current and proposed routing policies. Two mock providers would follow a fixed script: A is normally fast, sometimes slow, and occasionally returns a rate limit; B answers reliably at a higher simulated cost. No production prompts or provider credentials would be needed to reproduce the comparison.
The case I’d test first: A slows down for 30 seconds, enough requests move to B that B also starts rate-limiting, then A recovers. For each policy I’d compare completed requests, time to first token at the 95th percentile, provider attempts per completed request, and simulated cost per completed request. I’d also check how long it takes traffic to return to A.
If the candidate improves latency for successful replies but completes fewer requests during that burst, would you ship it? What result would make you block the PR, and what would you still want to verify with a small, budget-capped live test?
I’d love to build the example around cases other teams have actually seen. A missed failure mode, a short provider timing trace with synthetic data, or a simpler Railway service layout would give the next person something concrete to test.
1 Reply
3 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 3 hours ago
5 minutes ago
No, I wouldn't ship it as-is. I ran your scenario through a quick simulation, and the result depended almost entirely on two details that are easy to get wrong in the test setup.
Setup: 5 req/s, A's time to first token jumps to 6–8 s for 30 s, B costs 5× and is capped at 120 req/min, 10 seeds. Current policy waits 8 s for A; the candidate waits 2 s, moves to B, and goes back to A if B rate-limits.
During the slow window:
- Current (8 s): 100% completed, p95 7.9 s, 1.00 attempts per completed, cost 1.0×
- Candidate, B limit per minute: 100%, p95 3.1 s, 2.00 attempts, cost 5–6×
- Candidate, B limit per second, 9 s client deadline: 73%, p95 8.8 s, 3.55 attempts, cost 3.2–5×
- Candidate + 10% failover budget: 99%, p95 7.9 s, 1.15 attempts, cost 1.3–1.5×
1. How B enforces its limit. If B lets you spend a full minute's quota at once, it absorbs the whole burst and the candidate looks like a clean win: faster, just pricier. Real providers usually don't work that way. Anthropic's docs say a 60 RPM limit "might be enforced as 1 request per second," plus a separate ramp limit when traffic jumps suddenly, which is exactly what happens when everything moves to B. Model that and the picture flips: B rejects, requests go back to slow A, and the candidate ends up slower and completes fewer. A mock B with a simple per-minute counter will tell you to ship something that falls over in production.
2. The client deadline. The drop only appears when "2 s wait + rejected by B + slow A" runs past the deadline. At 10 s everything completed; at 9 s about a quarter failed. Make the deadline part of the scenario, and measure p95 over every request sent, counting failures as the deadline. Measuring only successful replies flatters the candidate.
Would I ship it? If it's faster on successful replies but completes fewer during the burst, I'd block it. Block if, across seeds:
- burst completion drops more than ~1 percentage point;
- attempts per completed go above ~1.3 (the router is multiplying its own load);
- B starts rejecting earlier or for longer than under the current policy;
- normal-traffic cost goes over budget.
What worked was keeping the short timeout but capping how much traffic can fail over (Google's SRE book uses a 10% retry ratio per client; Envoy defaults to 20%). Once the cap is hit, requests keep waiting on A instead of piling onto B. Completion went back to 99% and cost to ~1.3–1.5×, but most of the latency gain during the slowdown went away too. That's the real trade-off. Respecting Retry-After from B helps as well.
Returning to A: if the router tries A first on every request, it recovers the moment A does. It only matters once you add a circuit breaker; then probe A with a little traffic and ramp back gradually so you don't knock it over again.
Failure modes worth adding:
- Billing for abandoned requests. Whether A still charges after the router gives up varies by provider and isn't clearly documented. It moved cost from 3.2× to 5× in my run, so simulate both and check it live.
- Fast first token, then a stall. You need a between-token timeout too, and you can't switch providers once tokens reached the user.
- Different 429s. Rate limit, ramp limit, and spend cap differ; some lack
Retry-After, and a spend cap won't clear on retry. - Token limits. Long prompts can hit tokens-per-minute before the request limit.
Railway layout, three services in the PR environment:
- Router built from the PR, with both policies inside and a header choosing one per request, so both run on the same build under the same noise.
- One mock provider serving A and B from a scenario file and seed, with separate rate-limit counters per policy so both can run at once.
- Runner that sends the same schedule through both policies and writes a pass/fail report. Set restart policy to Never.
They talk over private networking (mock-provider.railway.internal); each environment has its own network, available at runtime only, not during build. With Focused PR Environments, untouched mocks may be skipped, so deploy them explicitly. Drop a warm-up window and run several seeds.
The budget-capped live test should only check what mocks can't: real billing for abandoned requests, real rate-limit behavior, and real token timings to tune the mock. Run it on 1–5% of traffic with a hard spend cap at the provider and the router, and stop automatically if a block criterion trips.