Build with the AI

Streaming

Stream tokens as they are generated with Server-Sent Events.

By default chat completions return one JSON object once the whole answer is ready. Set "stream": true and Krallo instead streams the response back over Server-Sent Events (SSE) as a sequence of OpenAI chat.completion.chunk frames, so you can render tokens the moment they arrive. The wire format is byte-for-byte compatible with the OpenAI Chat Completions streaming API, so the official OpenAI SDKs and editor integrations work unchanged.

Streaming uses the same endpoint, the same bearer API key, and the same ai:invoke scope as a buffered completion — it is one request flag, not a separate route.

Enable streaming

Add "stream": true to an otherwise ordinary chat request. Omitting it or sending false falls back to a single buffered JSON response (the string "true" is also accepted, but send a real boolean).

curl -N https://exe.krallo.com/api/v1/chat/completions \
  -H "Authorization: Bearer krallo_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "krallo-code",
    "messages": [{"role": "user", "content": "Count to three."}],
    "stream": true
  }'

The event stream

A streamed response is sent with HTTP 200 and the headers below. The body is chunk-transferred: each event is a line beginning with data:, followed by a blank line.

Frame sequence

For a text-only turn the frames always arrive in this order:

1. One role frame — delta is {"role":"assistant"}.
2. Zero or more content frames — each delta is {"content":"…"} carrying the next piece of text.
3. One terminal frame — an empty delta ({}) with finish_reason set to "stop".
4. The literal terminator line data: [DONE], then the connection closes.

Content is emitted in whatever granularity the upstream model produces — usually a few characters to a word per frame, not one frame per token. Concatenate every delta.content to reconstruct the full message text.

Chunk object

Every non-terminator frame carries one chat.completion.chunk object with this shape:

Example stream

The raw bytes for "Count to three." look like this (blank lines between frames are significant):

data: {"id":"chatcmpl-9f2c1a...","object":"chat.completion.chunk","created":1752566400,"model":"krallo-code","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-9f2c1a...","object":"chat.completion.chunk","created":1752566400,"model":"krallo-code","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl-9f2c1a...","object":"chat.completion.chunk","created":1752566400,"model":"krallo-code","choices":[{"index":0,"delta":{"content":", two"},"finish_reason":null}]}

data: {"id":"chatcmpl-9f2c1a...","object":"chat.completion.chunk","created":1752566400,"model":"krallo-code","choices":[{"index":0,"delta":{"content":", three."},"finish_reason":null}]}

data: {"id":"chatcmpl-9f2c1a...","object":"chat.completion.chunk","created":1752566400,"model":"krallo-code","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Consuming the stream

With an OpenAI SDK

Because the wire format is identical, the official OpenAI SDKs stream from Krallo with no changes beyond the base URL and key. This is the recommended path.

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.KRALLO_API_KEY,
  baseURL: "https://exe.krallo.com/api/v1",
});

const stream = await client.chat.completions.create({
  model: "krallo-code",
  messages: [{ role: "user", content: "Write a haiku about streams." }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

Parsing SSE by hand

If you are not using an SDK, split the body on blank lines, strip the data: prefix, stop on the literal [DONE] sentinel, and JSON.parse everything else. Buffer partial frames — a single read() may split a frame across chunk boundaries.

const res = await fetch("https://exe.krallo.com/api/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.KRALLO_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "krallo-code",
    messages: [{ role: "user", content: "Count to five." }],
    stream: true,
  }),
});

const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  let nl;
  while ((nl = buffer.indexOf("\n\n")) !== -1) {
    const frame = buffer.slice(0, nl).trim();
    buffer = buffer.slice(nl + 2);
    if (!frame.startsWith("data:")) continue;

    const payload = frame.slice(5).trim();
    if (payload === "[DONE]") break;

    const chunk = JSON.parse(payload);
    const delta = chunk.choices?.[0]?.delta?.content;
    if (delta) process.stdout.write(delta);
  }
}

import json, httpx

payload = {
    "model": "krallo-code",
    "messages": [{"role": "user", "content": "Count to five."}],
    "stream": True,
}
headers = {"Authorization": "Bearer krallo_live_...", "Content-Type": "application/json"}

with httpx.stream("POST", "https://exe.krallo.com/api/v1/chat/completions",
                  json=payload, headers=headers, timeout=None) as r:
    for line in r.iter_lines():
        if not line.startswith("data:"):
            continue
        data = line[5:].strip()
        if data == "[DONE]":
            break
        delta = json.loads(data)["choices"][0]["delta"]
        print(delta.get("content", ""), end="", flush=True)

Streaming with tools

When your request includes a tools array, the turn cannot be streamed incrementally — the model gateway has no combined streaming-plus-tools path. Krallo runs the completion to completion first, then replays the finished result as a valid SSE stream. Your client still receives well-formed chat.completion.chunk frames (including any tool_calls); they just arrive together at the end rather than token-by-token.

The replayed sequence is:

1. The role frame — {"role":"assistant"}.
2. If the model produced text, one content frame carrying the entire content string at once (not split).
3. If the model requested tools, one tool_calls frame carrying every call — each with a complete function.name and fully-formed function.arguments string, plus an index.
4. The terminal frame with an empty delta. Its finish_reason is "tool_calls" when tools were requested, otherwise "stop".
5. data: [DONE].

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1752566400,"model":"krallo-code","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1752566400,"model":"krallo-code","choices":[{"index":0,"delta":{"tool_calls":[{"id":"call_abc123","type":"function","function":{"name":"get_weather","arguments":"{\"city\":\"Paris\"}"},"index":0}]},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1752566400,"model":"krallo-code","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]

Errors and interruptions

There are two distinct failure regimes, split by whether the 200 response headers have been sent.

Before the stream starts

Authentication, scope, rate-limit, request validation, and the pre-flight credit reservation all run before any bytes are written. A failure here is a normal HTTP error with the correct status code and a JSON error body — the response is not an event stream at all. These are the same errors as a buffered completion:

After the stream starts

Once headers are flushed the HTTP status is committed at 200 and cannot change. If the upstream model stream then fails mid-flight, Krallo emits a single in-band error frame and closes the connection without a [DONE] terminator:

data: {"error":{"message":"The model provider stream failed.","type":"upstream_error"}}

Your parser must treat a data: frame that contains an error object as a failure, and must treat a stream that ends without [DONE] as incomplete. When this happens the credit reservation is rolled back — you are not charged for the failed call.

Token usage and billing

Streaming is billed exactly like a buffered completion, against your owner wallet. Credits are reserved before inference and settled after the stream finishes. Because most providers omit a usage block on streamed turns, Krallo settles from a character-based token estimate of the prompt and generated text when no provider count is available — so the meter stays proportional to real token volume rather than collapsing to a flat per-call charge. A failed or client-cancelled stream is rolled back and not charged.

Since no usage is delivered in the stream itself, read authoritative counts from the Usage endpoint, or issue a non-streaming request when you need the usage object inline. Full pricing and quota mechanics are in Billing & quotas.

Next steps

• Chat completions — the full request and response reference.
• Tools & function calling — drive an agent loop from your client.
• Use with opencode — a streaming editor integration with zero config.
• Errors — the platform-wide error model.