Notes from a 2 AM On-Call: Why Our vLLM Semantic Router Dropped 40% of Requests Under Load

Dev.to•Wed, Oct 21, 2026, 08:01 PM•2 min read

Notes from a 2 AM On-Call: Why Our vLLM Semantic Router Dropped 40% of Requests Under Load The Failure Symptom 2024-03-15 02:17:43 WARNING: Route decision timeout after 5000ms 2024-03-15 02:17:48 ERROR: 127 requests dropped (no route assigned) 2024-03-15 02:18:12 INFO: QPS: 240 → 140 (41% degrada...

Source: [Dev.to](https://dev.to/dave_mille/notes-from-a-2-am-on-call-why-our-vllm-semantic-router-dropped-40-of-requests-under-load-345)

📰 Read Full Story

This is an aggregated headline summary. For the complete report, visit the original publisher.

Continue Reading at Dev.to ↗

#tech #route #router #requests #async #prompt #asyncio #semantic #import

More Headlines

TechnologyDev.to• 5m ago

I'm making a one-bullet puzzle game on my own. Here's what changed after players got their hands on it.

I'm the only developer on Bloop. Design, code, UI, levels and the store pages are all me. It's a mobile puzzle game where you get one bullet per level: pull back, bank it off the walls, and pop every Bloop on the screen with that one shot.

TechnologyHacker News• 5m ago

Ford to City: Drop Dead

1 points, 0 comments on Hacker News

TechnologyDev.to• 6m ago

3.2 Million Redis and 1.9 Million MongoDB Services: The Steady Background of Exposed Data Stores

3. 2 Million Redis and 1. 9 Million MongoDB Services: The Steady Background of Exposed Data Stores Month after month, the same services appear in internet scan data at the same order of magnitude: Redis in the millions, MongoDB in the millions, and a long tail of databases below them.

TechnologyHacker News• 6m ago

Notes from a 2 AM On-Call: Why Our vLLM Semantic Router Dropped 40% of Requests Under Load

More Headlines

I'm making a one-bullet puzzle game on my own. Here's what changed after players got their hands on it.

Ford to City: Drop Dead

3.2 Million Redis and 1.9 Million MongoDB Services: The Steady Background of Exposed Data Stores

4.7% of Open Source Projects Have One or Two Maintainers

AI Early adopters share their views on AI status

Sega Genesis Controllers on the Philips CD-i