Chaos mode
Turn on chaos to have some calls fail, so you can test retries, backoff and error handling. By default a failing call is answered with an HTTP error in place of a reply.
Fail every 3rd call with a 429
In the preset in .llmockrc.json:
{
"chaos": { "enabled": true, "frequency": 3, "mode": "every", "status": 429 }
}Or for one run, from the command line:
llmock start --chaos=true --chaosFrequency=3 --chaosStatus=429Calls 3, 6, 9 and so on now get a 429 with retry-after: 1, in the error format of the preset's provider, so your client's own error handling runs. To fail one particular prompt instead, see Failing one prompt. To switch chaos on and off from a test, see Using LLMock in tests.
Settings
| Setting | CLI flag | Meaning | Default |
|---|---|---|---|
enabled | --chaos | Chaos on or off | false |
frequency | --chaosFrequency | Fail 1 in this many calls. 1 fails every call | 1 |
mode | --chaosMode | "every" fails calls X, 2X, 3X and so on, which is repeatable. "random" gives each call a 1 in X chance of failing | "every" |
status | --chaosStatus | HTTP status of the error, 400 to 599 | 500 |
kind | --chaosKind | What a failing call gets: "http", "stream-error", "stream-drop" or "stream-stall" (see Stream failures) | "http" |
afterChunks | --chaosAfterChunks | How many content deltas a failing stream sends before it fails | 2 |
What a failing call gets
The error body has the shape the preset's provider uses:
Preset name | Error body |
|---|---|
openai (and custom templates) | { "error": { "message", "type", "param", "code" } } |
claude | { "type": "error", "error": { "type", "message" } } with rate_limit_error (429), overloaded_error (529) or api_error |
gemini | { "error": { "code", "message", "status" } } with RESOURCE_EXHAUSTED (429), UNAVAILABLE (503) or INTERNAL |
/v1/embeddings always uses the openai shape.
- A
429,503or529carriesretry-after: 1. Every chaos error carriesx-llmock-chaos: true, so a test can tell it from a real failure. - The response delay runs first, then the error is sent. A streamed call that fails gets the same JSON error, with no stream (unless
kindsays otherwise, see Stream failures).
Stream failures
An HTTP error is the easy failure to handle. The one that catches streaming apps out is a reply that starts and then dies. Set kind to have a failing call do that:
{
"chaos": { "enabled": true, "frequency": 2, "kind": "stream-error", "afterChunks": 2 }
}llmock start --model=claude --chaos=true --chaosFrequency=2 --chaosKind=stream-errorkind | What a failing streamed call gets |
|---|---|
http | The HTTP error described above, with no stream. This is the default |
stream-error | The stream starts, sends afterChunks content deltas, then sends the provider's in-stream error and ends |
stream-drop | The stream starts, sends afterChunks content deltas, then the connection is cut with nothing more sent |
stream-stall | The stream starts, sends afterChunks content deltas, then goes silent. The connection stays open until your client gives up |
With the settings above, every 2nd streamed call to the claude preset looks like this:
event: message_start
data: {"type":"message_start","message":{...}}
event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Lorem ipsum "}}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"dolor sit "}}
event: error
data: {"type":"error","error":{"type":"api_error","message":"llmock chaos: simulated 500 error"}}With the Anthropic SDK, a stream-error call makes the stream throw:
const stream = client.messages.stream({
model: 'claude-opus-5-5',
max_tokens: 256,
messages: [{ role: 'user', content: 'Hello' }],
});
try {
await stream.finalMessage();
} catch (error) {
// The text received before the failure is incomplete: discard it and retry
}Stream failure details
frequencyandmodestill decide which calls fail, and the dashboard's count includes these failures.- The stream has status
200, because its headers have gone before it fails, and carriesx-llmock-chaos: true. None of the closing events are sent: nomessage_stopforclaude, nodata: [DONE]for an OpenAI-style stream. afterChunkscounts content deltas: text deltas, and theinput_json_deltaevents of a Claude tool-call reply, across all its blocks.0fails straight after the opening events. If the reply has fewer deltas thanafterChunks, the stream fails after the last one. A reply is split into about 5 deltas.- For
claudethe in-stream error is anevent: errorwhose errortypefollowsstatus(rate_limit_errorfor 429,overloaded_errorfor 529, otherwiseapi_error). An OpenAI-style stream (openai,geminiand custom templates) gets adata:line holding theopenaierror body. - A failing call that did not ask for a stream has no stream to cut. With
stream-errorit gets the HTTP error. Withstream-dropthe connection is cut without an answer, and withstream-stallit is held open without one. This includes/v1/embeddings. stream-stallnever times out on the server, so give the client you are testing a timeout. Stalled connections are dropped when the server stops.
Details
- Chaos applies to the chat endpoint and
/v1/embeddings, which share one call count. The dashboard,/pingand the/ui-*routes never fail. - Only a call that would have succeeded is counted. An invalid request still gets its
400. - A
frequency,mode,status,kindorafterChunksin the config file that is not valid stops the server at startup. - The Chaos box on the dashboard's Settings page shows the settings and how many calls have been failed since the server started.