The key data structure driving both phases is the KV cache.

What the KV Cache Is

In the attention mechanism, each token attends to every previous token in the sequence. To compute attention for token N, you need the key and value vectors for tokens 1 through N-1. Recomputing those from scratch at each step would mean re-running the full model for all previous tokens every time you generate a new one – O(N^2) compute per sequence.

The KV cache avoids this. After the prefill phase, the key and value vectors for every input token are stored in GPU memory. During the decode phase, each new token appended to the sequence just needs to compute its own K and V vectors and append them to the cache. Attending to all previous tokens is then a lookup into the cached K and V tensors, not a recomputation.

The cost of this: GPU memory. For a 7B parameter model at FP16, the KV cache for a single sequence can be hundreds of megabytes depending on context length, number of layers, and number of attention heads. For larger models serving many concurrent users, the KV cache dominates memory usage.

The Fragmentation Problem

Naive serving allocates KV cache memory statically: reserve a maximum-size block for each sequence upfront. This has the same problem as a badly written memory allocator.

Say you pre-allocate 4096 tokens per sequence and your GPU can hold 20 such blocks. A new request comes in and needs 10 blocks. 18 are free but they’re fragmented across physical memory – no contiguous run of 10. Request rejected, GPU sitting at 60% capacity.

Even worse: you don’t know the output length when a request arrives. If you allocate for the max context length immediately, most of that memory is wasted for short outputs. If you allocate conservatively, long outputs fail mid-generation.

PagedAttention

vLLM’s solution, introduced in the 2023 paper, borrows the virtual memory abstraction from operating systems.

Physical GPU memory is divided into fixed-size blocks (pages), typically 16 tokens each. Each sequence gets a logical block table that maps logical block indices to physical blocks. The blocks for one sequence don’t need to be contiguous in physical memory – the block table handles the translation.

Sequence A: logical [0,1,2] → physical [block_7, block_2, block_15]
Sequence B: logical [0,1]   → physical [block_3, block_11]

Blocks are allocated one at a time as tokens are generated, not all upfront. A short output uses a few blocks and returns them when done. A long output gradually accumulates more. Fragmentation is eliminated because the allocation granularity matches the actual growth pattern of sequences.

The result: GPU memory utilization goes from 20-40% with naive block allocation to 70-90% with PagedAttention. More sequences fit in memory simultaneously, which means higher throughput.

Continuous Batching

PagedAttention works with continuous batching to make the GPU utilization gains practical.

Traditional batching waits until a batch of N requests is assembled, runs inference for the full batch, returns all results. Any sequence that finishes early holds its slot until the full batch is done. Expensive for variable-length outputs.

Continuous batching (also called iteration-level scheduling) processes one decode step at a time across all in-flight sequences. After each step, finished sequences are removed and new requests are inserted into the freed slots. The batch is always full – there’s no waiting.

Combined with PagedAttention, the memory freed by completed sequences is immediately available for new ones. The GPU is busy at every step, not idling through early completions.

RadixAttention

SGLang introduced RadixAttention for workloads with shared prompt prefixes, common in agentic systems and structured generation.

Consider an agent that runs many LLM calls with the same system prompt and tool definitions – often 1000+ tokens. With standard PagedAttention, each request recomputes the KV vectors for those shared tokens during prefill. If you’re making 100 calls with the same 1000-token preamble, that’s 100x the prefill compute for tokens that never change.

RadixAttention maintains a radix tree of cached KV blocks keyed by token sequence prefix. When a new request arrives, it walks the tree to find the longest matching prefix, reuses those cached blocks, and only computes KV vectors for the novel suffix.

For agentic workloads where the same system prompt and tool list appear in every call, the first request pays the full prefill cost. Every subsequent request with the same prefix pays only for its unique portion. At the scale of thousands of agent steps, this is a substantial throughput difference.

The tradeoff: RadixAttention adds cache management overhead. The tree needs to be pruned when memory pressure is high. The eviction policy matters – evicting a heavily shared prefix that gets reused on the next request undoes the benefit. SGLang uses LRU eviction, which works well for stable prefixes but less well for highly dynamic ones.

Choosing Between vLLM and SGLang

The choice comes down to workload:

vLLM is the default for general inference, chat APIs, and throughput benchmarks. PagedAttention and continuous batching are well-tested across a wide range of models and hardware.

SGLang is better for agentic workflows, structured output (constrained decoding for JSON, function calling), and any case where many requests share large prompt prefixes. The RadixAttention benefit only materializes when your prefix reuse rate is high.

Both expose an OpenAI-compatible API and support the same quantization formats. Running both and profiling your actual traffic pattern is the right answer if you’re unsure – the throughput difference on real workloads is usually decisive.

The KV cache is why LLM inference is memory-bound rather than compute-bound during decode. Understanding it makes the rest of the optimization landscape – quantization to shrink the model, PagedAttention to use memory efficiently, RadixAttention to reuse computation – follow naturally from first principles.

The problem was naive serving. Standard HuggingFace pipeline() processes one request at a time, sequentially. The GPU sits idle between requests, waiting. You’re paying for hardware that’s doing nothing most of the time.

The fix is a set of four techniques, each with a different ROI profile. Here’s what they are, how they work, and when to use each one.

Quantization

This is the first thing to do because it’s a one-time change with immediate impact. AWQ 4-bit quantization cuts VRAM usage by roughly 75%. FP8 cuts it by 50% but requires H100s.

That 75% reduction matters because it changes which GPU class you need. A model that required an A100 might now fit on an A6000. That’s not a minor cost reduction – it’s a different tier of hardware entirely.

The tooling is straightforward. Use AutoAWQ to quantize yourself, or just pull a pre-quantized checkpoint from HuggingFace. Most popular models have AWQ variants already uploaded.

Serving Engine

Once you’re quantized, the next question is which serving engine to use. The two serious options are vLLM and SGLang, and they’re not interchangeable.

vLLM is the right choice for chat and general inference. Two mechanisms make the difference:

Continuous batching inserts new requests into the batch as slots open up rather than waiting for a full batch to complete. Naive serving sits at around 20% GPU utilization. vLLM pushes that to 60-85%.

PagedAttention treats VRAM like virtual memory for the KV cache. Instead of pre-allocating contiguous blocks, it pages memory on demand. More sequences can coexist in memory simultaneously, which means more throughput per dollar.

The key vLLM flags worth knowing in production:

1--max-num-seqs 64        # tune for your context length + available VRAM
2--disable-log-stats      # remove logging overhead

SGLang is the right choice for agentic workflows and structured output. Its main advantage is RadixAttention – it automatically detects shared prompt prefixes and reuses their KV cache. If every request in your system starts with the same 1000-token system prompt and tool definitions, SGLang computes that prefix once and caches it. LMSYS benchmarks show higher throughput than vLLM on structured generation workloads.

The rule of thumb: SGLang for agents and JSON, vLLM for chat.

Speculative Decoding

This one is less obvious but gets you 2-3x speedup on long-form generation. The mechanism is a draft-then-verify loop:

1. A small draft model (1-7B parameters) generates 3-12 candidate tokens per step
2. The large target model verifies all candidates in a single parallel forward pass
3. When the draft guesses correctly – which happens 70-90% of the time – you get multiple tokens for the cost of one forward pass

The canonical pairing is Llama-3-8B-Instruct-AWQ as the draft model against Llama-3-70B-Instruct-AWQ as the target.

A full deployment looks like this:

 1import runpod
 2runpod.api_key = "your_api_key"
 3pod = runpod.create_pod(
 4    name="llama3-70b-speculative",
 5    image_name="vllm/vllm-openai:latest",
 6    gpu_type_id="NVIDIA RTX A6000",
 7    gpu_count=2,
 8    docker_args=(
 9        "--model TechxGenus/Meta-Llama-3-70B-Instruct-AWQ "
10        "--quantization awq "
11        "--tensor-parallel-size 2 "
12        "--speculative-model TechxGenus/Meta-Llama-3-8B-Instruct-AWQ "
13        "--num-speculative-tokens 5 "
14        "--max-model-len 8192"
15    )
16)

The critical thing to watch: draft model acceptance rate. Monitor it via Prometheus:

vllm:spec_decode_draft_acceptance_length

If acceptance drops below 0.5 tokens per step, you have the wrong draft model. It adds overhead rather than saving it. The draft model must match the target architecture – vLLM won’t warn you about this at startup.

Deployment Mode

The last decision is serverless vs. pods, and it’s purely a traffic pattern question.

Spiky or unpredictable traffic belongs on serverless – you pay only for actual inference time. Sustained, predictable load is cheaper on pods because you’re not paying the serverless premium on every call.

Neither is universally better. The mistake is running pods for workloads that get 10 requests an hour or paying for serverless when you have a steady 200 RPS.

Why Decode is the Bottleneck

LLM generation has two phases and they’re bound by different things.

Prefill is compute-bound – processing all the input tokens in one parallel forward pass. Decode is memory-bound – generating one token at a time, loading the entire model’s KV cache from VRAM on every single step.

This means VRAM bandwidth determines inter-token latency, not FLOPs. Two GPUs with similar compute specs can have very different inference latency:

That’s a ~4x difference on long-form generation. If you’re choosing hardware for an inference workload and only looking at FLOPs, you’re optimizing the wrong thing.

The practical checklist once you understand this:

• Quantize with AWQ first (or FP8 on H100s)
• Pick SGLang for agents and JSON, vLLM for chat
• Enable speculative decoding for long-form endpoints, monitor acceptance rate
• Wire Prometheus to vllm:gpu_cache_usage_perc to know when you’re VRAM-constrained
• Match deployment mode to your actual traffic shape

The hardware question comes last, not first.

This is the story of setting that up in one session, and the parts that went sideways.

The Stack

Four moving parts:

• Bifrost – LLM gateway already running on localhost:24242, routing to Ollama cloud models
• Hermes Agent – open-source AI agent framework by Nous Research, pointed at Bifrost
• Ollama – cloud relay for gemma4 and qwen3.5 cloud models via ollama.com
• Camoufox – Firefox fork with TLS fingerprint spoofing for scraping tasks

The goal was to have Hermes run as a Telegram bot so I could assign it tasks from my phone, with all model traffic flowing through Bifrost so routing stays centralized.

Bifrost First

Bifrost was already up on port 24242 with Ollama configured. The only addition needed was an ollama-cloud provider pointing at https://ollama.com rather than localhost. This is a non-obvious distinction.

The trap: Ollama’s OpenAI-compatible endpoint (/v1/chat/completions) does not work for cloud models. It hangs. Cloud models only work through Ollama’s native /api/chat endpoint. Pointing Bifrost directly at https://ollama.com instead of localhost:11434 bypasses the broken compat layer and hits the right path.

Once the provider was registered and OLLAMA_API_KEY was set in .zshrc, both cloud models confirmed working:

1ollama pull gemma4:31b-cloud
2ollama pull qwen3.5:cloud

Installing Hermes

1curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

That puts everything under ~/.hermes/. The setup wizard runs next:

1hermes setup

It walks through provider selection interactively. I pointed it at the Bifrost custom endpoint (http://localhost:24242/v1) and picked ollama-cloud/gemma4:31b-cloud as the default model.

Key config paths:

~/.hermes/config.yaml    -- main settings
~/.hermes/.env           -- secrets and API keys
~/.hermes/skills/        -- reusable agent procedures
~/.hermes/sessions/      -- full conversation history (searchable)

After setup, a quick smoke test confirmed the agent was up and routing through Bifrost correctly:

1hermes chat -q "what model are you using and where does traffic go"

Context Window

The default context length was under 20k tokens. Ollama cloud models support much larger windows. One config set:

1hermes config set model.context_length 200000

Takes effect on the next session restart.

Telegram Bot

This is where Hermes gets genuinely useful. The gateway turns it into a bot that accepts tasks from any platform you connect.

Create a bot via BotFather, get the token, then add it to ~/.hermes/.env:

1TELEGRAM_BOT_TOKEN=<your-bot-token>
2TELEGRAM_ALLOWED_USERS=<your-numeric-telegram-id>

TELEGRAM_ALLOWED_USERS takes the numeric user ID, not the username. Lock it down. Anyone who finds the bot URL can send it commands if this is not set.

Start the gateway:

1hermes gateway run

For persistent background operation, install it as a service:

1hermes gateway install
2hermes gateway start

A set of aliases in .zshrc makes it easier to manage day to day:

1alias hermes-gateway-start="hermes gateway start"
2alias hermes-gateway-stop="hermes gateway stop"
3alias hermes-gateway-log="tail -f ~/.hermes/logs/gateway.log"

First run hit a missing Python dependency and failed silently. The fix was running hermes gateway run in the foreground once to see the real error, installing the missing package, then moving back to the service.

Camoufox

Camoufox is a Firefox fork built for scraping. It spoofs TLS fingerprints so sites see a legitimate browser rather than a Playwright or CDP signature. It exposes a CDP endpoint that Hermes’s browser tool connects to.

1git clone https://github.com/daijro/camoufox
2cd camoufox && npm install
3python -m camoufox fetch   # ~300MB download

The camoufox fetch step downloads the actual browser binary and does not run automatically via postinstall. If you skip it, the first start fails silently. Run it manually and wait for the download to finish before starting the server.

Start it on port 9377:

1python -m camoufox server --port 9377 &

Smoke test via Hermes:

1hermes chat -q "use the browser to fetch the title of example.com"

Aliases for day-to-day:

1alias camofox-start="python -m camoufox server --port 9377 &"
2alias camofox-stop="pkill -f 'camoufox server'"

What Actually Went Wrong

Ollama native vs compat endpoint. Already covered above, but it cost a couple of hours. If a cloud model hangs at the provider level, check whether your gateway is talking to the right API path.

Camoufox silent failure. The binary fetch is not in postinstall. No error, just no browser. Always run python -m camoufox fetch explicitly before the first start.

Hermes gateway dependency. First start printed nothing useful. Run in foreground before daemonizing so you can see the actual error.

Bifrost and empty API keys. Ollama initially rejected non-empty dummy keys in the Authorization header. Setting the key to an empty string before switching to the real OLLAMA_API_KEY got things unstuck. The order mattered.

Session corruption in OpenCode. An OpenCode session used to orchestrate this setup got a zombie assistant message from an interrupted stream. The message had an empty string for content. Bifrost converted that empty string to null before forwarding, and the upstream provider rejected the entire next request with messages.315.member.content must not be null. Every retry included the same zombie in history, so the error was permanent until manually fixed.

The fix is to delete the zombie row directly from the OpenCode SQLite database:

1sqlite3 ~/.local/share/opencode/opencode.db \
2  "DELETE FROM message WHERE id = '<zombie-message-id>' AND session_id = '<session-id>';"

There is no built-in repair command. Back up the database before editing it.

Final State

Services running when done:

bifrost    :24242  -- Ollama cloud routes
ollama     :11434  -- cloud relay (gemma4, qwen3.5)
camoufox   :9377   -- CDP browser endpoint
hermes     gateway -- Telegram bot (restricted to single user)

The agent receives messages on Telegram, runs tool calls locally (terminal, browser, file ops, web search), and routes all model traffic through Bifrost.

What’s Left

The original plan included running a local Qwen3.5-27B model via llama.cpp with Metal acceleration. That is a 13.5GB download and a build-from-source step, so it was deferred to the next session. The cloud routing setup works fine in the meantime.

References

• Hermes Agent: https://github.com/NousResearch/hermes-agent
• Hermes docs: https://hermes-agent.nousresearch.com/docs/
• Bifrost: https://github.com/maximhq/bifrost
• Camoufox: https://github.com/daijro/camoufox
• OpenCode: https://opencode.ai

So I built freeseek-proxy: a local proxy that sits between OpenAI-compatible clients (opencode, Claude Code, any SDK) and DeepSeek’s web chat backend. You send standard /v1/chat/completions requests; it handles sessions, PoW, TLS fingerprinting, and SSE translation behind the scenes.

This post covers the four hardest problems I had to solve.

The architecture (it’s two processes, not one)

The proxy is split into two halves that talk over localhost:

                  ┌──────────────────────────┐
                  │   Client                 │
                  │   (opencode / curl / SDK)│
                  └────────────┬─────────────┘
                               │
                  POST /v1/chat/completions
                       (OpenAI format)
                               │
                  ┌────────────▼─────────────┐
                  │   TypeScript Proxy       │
                  │   Hono on Bun :9123      │
                  │                          │
                  │   · Request normalization│
                  │   · Tool-call compat     │
                  │   · SSE reformatting     │
                  │   · Session management   │
                  └────────────┬─────────────┘
                               │
                    HTTP localhost:8081
                               │
                  ┌────────────▼─────────────┐
                  │   Python Backend         │
                  │   FastAPI + uvicorn      │
                  │                          │
                  │   · curl_cffi (TLS spoof)│
                  │   · WASM PoW solver      │
                  │   · Cookie management    │
                  │   · Auth (token / login) │
                  └────────────┬─────────────┘
                               │
                   HTTPS chat.deepseek.com
                               │
                  ┌────────────▼─────────────┐
                  │      DeepSeek API        │
                  └──────────────────────────┘

Why two languages? DeepSeek does browser fingerprinting. Their servers check TLS fingerprints, and if yours looks like python-requests or node-fetch, you get blocked. The Python side uses curl_cffi with impersonate="chrome120" to produce a TLS handshake indistinguishable from Chrome. The TypeScript side handles everything else — request translation, streaming, tool calls — where Bun’s performance and type safety matter more.

Problem 1: Proof-of-Work with a WASM SHA3 solver

DeepSeek doesn’t just check your auth token. Before every chat completion, you must solve a Proof-of-Work challenge. The flow is:

1. Request a challenge from /api/v0/chat/create_pow_challenge
2. The server returns a challenge string, salt, difficulty, and expiry
3. You run a SHA3-based hash computation until you find a valid answer
4. Encode the answer as base64 JSON and send it as x-ds-pow-response header

The catch: DeepSeek’s PoW algorithm is implemented in a WebAssembly module. In the browser, this runs natively. Outside the browser, you need a WASM runtime.

The solver uses wasmtime (a standalone WASM runtime for Python) to load DeepSeek’s own sha3_wasm_bg.wasm binary and call its exported wasm_solve function:

 1class DeepSeekHash:
 2    def init(self, wasm_path: str):
 3        engine = wasmtime.Engine()
 4        with open(wasm_path, "rb") as f:
 5            wasm_bytes = f.read()
 6        module = wasmtime.Module(engine, wasm_bytes)
 7        self.store = wasmtime.Store(engine)
 8        linker = wasmtime.Linker(engine)
 9        linker.define_wasi()
10        self.instance = linker.instantiate(self.store, module)
11        self.memory = self.instance.exports(self.store)["memory"]
12        return self
13
14    def calculate_hash(self, algorithm, challenge, salt, difficulty, expire_at):
15        prefix = f"{salt}_{expire_at}_"
16        # Allocate WASM memory, write challenge + prefix
17        challenge_ptr, challenge_len = self._write_to_memory(challenge)
18        prefix_ptr, prefix_len = self._write_to_memory(prefix)
19        # Call the exported solver
20        self.instance.exports(self.store)["wasm_solve"](
21            self.store, retptr,
22            challenge_ptr, challenge_len,
23            prefix_ptr, prefix_len,
24            float(difficulty),
25        )
26        # Read the answer from WASM memory
27        value_bytes = bytes(memory_view[retptr + 8 : retptr + 16])
28        value = np.frombuffer(value_bytes, dtype=np.float64)[0]
29        return int(value)

The memory management is manual — you allocate via __wbindgen_export_0, write UTF-8 bytes one at a time, call the solver, then read the result from a return pointer. Stack pointer cleanup happens in a finally block. It’s the kind of code where one off-by-one error means silent corruption.

Why Python 3.11 specifically? The wasmtime + numpy + curl_cffi stack was verified working on 3.11. Python 3.14 caused silent failures in the WASM path during testing — the process would get killed mid-computation with no error message. This isn’t a “should work” situation; it’s a “was tested and doesn’t” situation.

Problem 2: TLS fingerprinting and Cloudflare

DeepSeek sits behind Cloudflare. Standard HTTP clients get 403s or “Just a moment…” challenge pages because their TLS Client Hello doesn’t match what a real browser sends.

The fix is curl_cffi with browser impersonation:

1response = requests.post(
2    f"{BASE_URL}{path}",
3    headers=headers,
4    json=body,
5    cookies=cookies,
6    impersonate="chrome120",  # TLS fingerprint matches Chrome 120
7    stream=stream,
8)

impersonate="chrome120" makes the TLS handshake (cipher suites, extensions, ALPN) identical to Chrome 120. This is why the backend is Python, not Node — there’s no equivalent library in the JS ecosystem that matches real browser TLS fingerprints this precisely.

The cookie management layer adds another defense: if a request hits Cloudflare anyway, the proxy can automatically refresh cookies via a configurable shell command:

1def refresh_cookies(self) -> dict[str, str]:
2    if not self.refresh_command:
3        return self.load_cookies()
4    subprocess.run(self.refresh_command, shell=True, check=True)
5    cookies = self.load_cookies()
6    self.save_cookies(cookies)
7    return cookies

This lets you plug in whatever cookie extraction tool you prefer — a headless browser script, a browser extension export, etc.

Problem 3: Translating SSE formats

DeepSeek’s streaming format is not OpenAI’s streaming format. Both use Server-Sent Events, but the payload structure is completely different.

OpenAI format (what clients expect):

1data: {"choices":[{"delta":{"content":"Hello"},"index":0}],"model":"gpt-4"}

DeepSeek format (what the server sends):

1data: {"v":{"response":{"fragments":[{"type":"text","content":"Hello"}]}},"o":"APPEND"}

DeepSeek uses a nested structure where content lives inside response.fragments, operations can be APPEND, SET, or BATCH, and thinking content is separated from regular content by fragment type. There’s also metadata like response/status and response/accumulated_token_usage mixed into the stream that clients don’t want.

The proxy handles this translation in parseSSELine and extractDeepSeekContent:

 1function extractDeepSeekContent(
 2  data: DeepSeekSSEData,
 3  includeThinking: boolean
 4): string {
 5  // DeepSeek nests content in multiple possible locations
 6  const target = data.v?.response ?? data.response ?? data;
 7  const fragments = target?.fragments;
 8
 9  if (Array.isArray(fragments)) {
10    return fragments
11      .filter(f => includeThinking || f.type !== "thinking")
12      .map(f => f.content)
13      .join("");
14  }
15
16  // Fallback: direct content field
17  if (typeof data.v === "string") return data.v;
18  if (typeof data.content === "string") return data.content;
19  return "";
20}

For streaming responses, each DeepSeek SSE chunk gets translated into an OpenAI chat.completion.chunk on the fly. For non-streaming, the proxy accumulates all chunks, then builds a single chat.completion response.

Problem 4: Tool calls on a model that doesn’t support them

This is the most interesting problem. DeepSeek’s web chat API doesn’t support OpenAI-style tool calls. There’s no tools parameter, no tool_choice, no tool_calls in the response. But clients like opencode and Claude Code rely heavily on tool use — without it, the agent can’t read files, run commands, or interact with your codebase.

The solution is a prompt-engineering compatibility layer. When the proxy detects tool-related fields in the request, it:

1. Injects a system prompt describing available tools and the expected JSON format
2. Flattens the conversation into a single prompt (including tool results from previous turns)
3. Parses the model’s response for JSON tool call envelopes
4. Translates back into OpenAI’s tool_calls format

The injected system prompt instructs the model to output tool calls as JSON:

 1function buildToolSystemPrompt(tools, toolChoice) {
 2  return [
 3    "You are replying through an OpenAI-compatible proxy that supports tool use.",
 4    toolChoiceRule, // varies by mode: auto/required/none/function
 5    `If you decide to call a tool, output only valid JSON with this exact shape:`,
 6    `{"opencode_tool_call": {"name": "tool_name", "arguments": { ... }}}.`,
 7    `For multiple calls: {"opencode_tool_calls": [...]}.`,
 8    "Do not wrap the JSON in markdown.",
 9    "Available tools:",
10    availableTools, // name, description, JSON schema for each
11  ].join("\n\n");
12}

The conversation history gets flattened because DeepSeek’s chat API takes a single prompt string, not an array of messages:

 1function formatConversationForDeepSeek(messages) {
 2  return messages.map(message => {
 3    if (message.role === "system") return `System: ${content}`;
 4    if (message.role === "user") return `User: ${content}`;
 5    if (message.role === "assistant") {
 6      if (message.tool_calls?.length)
 7        return `Assistant tool request: ${formatted calls}`;
 8      return `Assistant: ${content}`;
 9    }
10    if (message.role === "tool")
11      return `Tool (${message.tool_call_id}) result: ${content}`;
12  }).join("\n\n");
13}

On the response side, parseToolEnvelope scans the model output for JSON objects containing opencode_tool_call or opencode_tool_calls keys, validates that the function names match known tools, and converts them into proper OpenAI tool_calls with generated IDs:

 1function parseToolEnvelope(text, tools) {
 2  for (const candidate of extractJsonObjectCandidates(text.trim())) {
 3    const payload = JSON.parse(candidate);
 4    const rawCalls = payload.opencode_tool_calls ?? [payload.opencode_tool_call];
 5    const toolCalls = rawCalls
 6      .filter(call => tools.some(t => t.function.name === call.name))
 7      .map(call => ({
 8        id: `call_${uuidv4()}`,
 9        type: "function",
10        function: { name: call.name, arguments: JSON.stringify(call.arguments) },
11      }));
12    if (toolCalls.length) return { toolCalls };
13  }
14  return null;
15}

The JSON extraction is deliberately robust — it uses a bracket-depth parser to find JSON objects embedded in natural language, handles escaped strings, and tries multiple candidates. This matters because the model sometimes wraps tool calls in explanatory text despite being told not to.

For streaming with tools, there’s an extra step. The proxy can’t stream tool calls token-by-token (the client needs the complete tool call object). So when tools are present, the proxy collects the full response first, parses tool calls, then re-streams the result as a synthetic SSE sequence. This adds ~latency but is the only correct approach.

What the client sees

After all this translation, what comes out the other end is a standard OpenAI response:

1curl http://localhost:9123/v1/chat/completions \
2  -H "Content-Type: application/json" \
3  -d '{
4    "model": "deepseek-chat",
5    "messages": [{"role": "user", "content": "Hello"}],
6    "stream": true
7  }'

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0}]}
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"},"index":0}]}
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"!"},"index":0}]}
data: [DONE]

Indistinguishable from OpenAI. Any client that speaks the OpenAI protocol works without modification.

What I learned

WASM outside the browser is viable but fragile

The wasmtime runtime works, but debugging WASM memory issues from Python is miserable. There’s no stack trace when you read the wrong offset — you just get garbage bytes. The debug flags in pow.py exist because I needed them.

TLS fingerprinting is the real gate, not auth tokens

I initially thought the hard part would be authentication. It wasn’t. The hard part was making HTTP requests that don’t get blocked. curl_cffi with browser impersonation is the only reliable solution I found for Python. If you’re building any kind of web scraper or API proxy in 2026, TLS fingerprinting is the first problem you’ll hit.

Tool call compatibility via prompting is surprisingly reliable

I expected the prompt-engineering approach to tool calls to be flaky. It’s not. DeepSeek’s models are good at following structured output instructions, and the JSON extraction parser handles edge cases well. The key is being very explicit about the expected format and validating against the known tool list.

Two-process architecture was the right call

Splitting into TypeScript + Python felt like overengineering at first. It wasn’t. Each language does what it’s best at: Python for the “pretend to be a browser” parts (TLS, WASM), TypeScript for the “translate between API formats” parts (streaming, types, request handling). Trying to do both in one language would have meant worse tooling for at least half the problem.

Links

• Repo: freeseek-proxy
• curl_cffi: docs — Python HTTP client with browser TLS impersonation
• wasmtime-py: GitHub — Python bindings for the Wasmtime WASM runtime
• Hono: hono.dev — Lightweight web framework for Bun/Node/Deno

Except it doesn’t. It concatenates them all into one call: search_documentsget_usersearch_documents. The agent hallucinates, retries, and loops. I stared at the telemetry for twenty minutes before I realized the bug wasn’t in the agent — it was in the gateway’s streaming accumulator.

This is the story of how I found and fixed a parallel tool call merging bug in Bifrost, a 2.6k-star open-source AI gateway.

What is Bifrost?

Bifrost is a high-performance AI gateway written in Go. It sits between your application and LLM providers (OpenAI, Anthropic, Bedrock, Vertex, etc.), providing a unified OpenAI-compatible API with automatic failover, load balancing, semantic caching, and streaming support. Think of it as an intelligent reverse proxy for AI — one API, fifteen providers, ~11µs overhead at 5k RPS.

The relevant piece here is the streaming accumulator. When you make a streaming chat completion request, the LLM sends back a sequence of Server-Sent Events (SSE), each containing a small delta — a few tokens of text, or a fragment of a tool call’s arguments. The accumulator’s job is to reassemble these deltas into complete messages.

The bug: recency-based routing

Here’s the scenario. You ask Claude or GPT to call two tools in parallel:

User: "Search for compliance circulars about RBI and also get the user's profile"

The model decides to call both search_circulars and get_user_profile simultaneously. During streaming, the provider sends interleaved deltas:

Delta 1: tool_call[0] name="search" (index=0, id=call_abc)
Delta 2: tool_call[1] name="get_us" (index=1, id=call_xyz)
Delta 3: tool_call[0] args='{"query": "RBI'   (index=0)
Delta 4: tool_call[1] args='{"user_id": "u'   (index=1)
Delta 5: tool_call[0] args=' circulars"}'      (index=0)
Delta 6: tool_call[1] args='123"}'             (index=1)

Each delta carries an index (and optionally an id) that identifies which tool call it belongs to. The expected behavior: two distinct tool calls with their own names and arguments.

The actual behavior in Bifrost’s accumulator before the fix:

1// Old logic (simplified)
2if deltaToolCall.Function.Name != nil {
3    // New tool call — create it
4    existingToolCalls = append(existingToolCalls, newToolCall)
5} else {
6    // Argument delta — append to the LAST tool call
7    toolCallToModify = &existingToolCalls[len(existingToolCalls)-1]
8    toolCallToModify.Function.Arguments += deltaToolCall.Function.Arguments
9}

The routing logic was based on recency: if a delta has a Name, it’s a new tool call; otherwise, append arguments to the most recent one. This works perfectly when tool calls arrive sequentially. It breaks completely when they’re interleaved.

With the deltas above:

• Delta 3 (index=0 args) arrives → appended to the last tool call, which is get_user_profile (index=1). Wrong target.
• Delta 5 (index=0 args) arrives → again appended to get_user_profile. Now get_user_profile has arguments from two different calls mashed together.

The result: concatenated tool names, merged arguments, and an agent stuck in a retry loop trying to invoke search_circularsget_user_profile — a tool that doesn’t exist.

Finding the bug

I hit this in production. Our compliance platform uses an AI agent that calls multiple tools in parallel — searching circulars, fetching regulatory data, looking up user context. The agent kept failing with tool resolution errors, and the telemetry showed impossible tool names:

search_circularsearch_circularsearch_circular

At first I suspected the agent’s tool-calling logic. Then I looked at the raw SSE stream from the provider — the deltas were correct. Each one had the right index and id. The corruption was happening between the provider response and the final accumulated message.

I traced it to accumulateToolCallsInMessage in framework/streaming/accumulator.go. The function had no concept of tool call identity. It used the presence of a Name field as a heuristic for “new vs. continuation,” which is fragile and incorrect when deltas arrive out of order.

The fix

The fix is conceptually simple: route deltas by identity, not by recency.

Each tool call delta carries two potential identifiers:

1. tool_call.id — a string like call_abc123 (set by OpenAI/Anthropic)
2. tool_call.index — an integer position (0, 1, 2, …)

The new logic builds lookup maps on existing tool calls and uses them to route each incoming delta to the correct target:

1// Build lookup maps for identity-based routing
2idToIndex := make(map[string]int, len(existingToolCalls))
3indexToIndex := make(map[uint16]int, len(existingToolCalls))
4for i, toolCall := range existingToolCalls {
5    if toolCall.ID != nil && *toolCall.ID != "" {
6        idToIndex[*toolCall.ID] = i
7    }
8    indexToIndex[toolCall.Index] = i
9}

For each incoming delta, routing follows a priority chain:

 1for _, deltaToolCall := range deltaToolCalls {
 2    index := -1
 3
 4    // Priority 1: match by tool_call.id
 5    if deltaToolCall.ID != nil && *deltaToolCall.ID != "" {
 6        if existingIndex, ok := idToIndex[*deltaToolCall.ID]; ok {
 7            index = existingIndex
 8        }
 9    }
10
11    // Priority 2: match by tool_call.index
12    if index == -1 {
13        if existingIndex, ok := indexToIndex[deltaToolCall.Index]; ok {
14            index = existingIndex
15        }
16    }
17
18    // Priority 3: no match → create new tool call (if name present)
19    if index == -1 {
20        if deltaToolCall.Function.Name == nil {
21            // Can't create a tool call without a name
22            continue
23        }
24        newToolCall := schemas.ChatAssistantMessageToolCall{
25            Index:    deltaToolCall.Index,
26            ID:       deltaToolCall.ID,
27            Function: /* ... */,
28        }
29        existingToolCalls = append(existingToolCalls, newToolCall)
30        // Update maps for future deltas
31        indexToIndex[newToolCall.Index] = len(existingToolCalls) - 1
32        if newToolCall.ID != nil {
33            idToIndex[*newToolCall.ID] = len(existingToolCalls) - 1
34        }
35        continue
36    }
37
38    // Append arguments to the correct tool call
39    existingToolCalls[index].Function.Arguments += deltaToolCall.Function.Arguments
40}

The key insight is that the maps are updated as new tool calls are created, so subsequent deltas for the same call always find their target — even if they arrive many chunks later.

The same bug, twice

During code review, a reviewer (Pratham-Mishra04) pointed out that the same recency-based routing existed in a second code path: the Responses API streaming handler in responses.go.

The Responses API is OpenAI’s newer streaming format (the /v1/responses endpoint). It uses ItemID instead of tool_call.index, but the accumulation logic had the same flaw — FunctionCallArgumentsDelta events were always appended to the last message:

1// Old responses.go logic
2if resp.Delta != nil && len(messages) > 0 {
3    // Always targets the last message — wrong for parallel calls
4    a.appendFunctionArgumentsDeltaToResponsesMessage(
5        &messages[len(messages)-1], *resp.Delta)
6}

The fix follows the same pattern: search for a message matching resp.ItemID, and only fall back to the last message when no ItemID is present. If an ItemID exists but no message matches, create a stub message to hold the delta rather than merging it into the wrong call:

 1if resp.Delta != nil && len(messages) > 0 {
 2    targetIdx := len(messages) - 1
 3    if resp.ItemID != nil {
 4        targetIdx = -1
 5        for i := len(messages) - 1; i >= 0; i-- {
 6            if messages[i].ID != nil && *messages[i].ID == *resp.ItemID {
 7                targetIdx = i
 8                break
 9            }
10        }
11        if targetIdx == -1 {
12            // Create stub to avoid merging into wrong message
13            id := *resp.ItemID
14            messages = append(messages, schemas.ResponsesMessage{ID: &id})
15            targetIdx = len(messages) - 1
16        }
17    }
18    a.appendFunctionArgumentsDeltaToResponsesMessage(
19        &messages[targetIdx], *resp.Delta)
20}

Testing parallel accumulation

The regression test simulates the exact interleaving pattern that triggers the bug:

 1func TestAccumulateToolCallsInterleavedParallel(t *testing.T) {
 2    // ...
 3    toolCallID0 := "call_0"
 4    toolCallID1 := "call_1"
 5    toolNameAdd := "add"
 6    toolNameMultiply := "multiply"
 7
 8    // Simulate interleaved deltas
 9    accumulate(makeDelta(0, &toolCallID0, &toolNameAdd, ""))       // create call 0
10    accumulate(makeDelta(1, &toolCallID1, &toolNameMultiply, ""))  // create call 1
11    accumulate(makeDelta(0, nil, nil, `{"a": 1`))                  // args for call 0
12    accumulate(makeDelta(1, nil, nil, `{"a": 2`))                  // args for call 1
13    accumulate(makeDelta(0, nil, nil, `, "b": 3}`))                // more args for call 0
14    accumulate(makeDelta(1, nil, nil, `, "b": 4}`))                // more args for call 1
15
16    // Assert: two distinct tool calls with correct arguments
17    assert(addCall.Arguments == `{"a": 1, "b": 3}`)
18    assert(multiplyCall.Arguments == `{"a": 2, "b": 4}`)
19}

Without the fix, add would end up with multiply’s arguments concatenated in. With the fix, each call accumulates only its own deltas.

A second test covers the Responses API path with ItemID-based routing, using the same interleaving pattern but with OutputItemAdded + FunctionCallArgumentsDelta events.

The PR lifecycle

I filed Issue #1829 with reproduction details from production telemetry, then opened PR #1830 with the fix and tests.

The review process was smooth:

1. First commit: fix the Chat Completions accumulator + regression test
2. Reviewer feedback: same bug exists in Responses API — please fix there too + add changelog entries
3. Second commit: fix Responses API routing + regression test + changelogs
4. Merged by akshaydeo (Bifrost maintainer)

Total: 3 commits, ~280 lines of Go across 5 files. Same-day turnaround from issue to merge.

What I learned

1. Recency is not identity

The original code used position (last element) as a proxy for identity. This is a common pattern in sequential streaming — and it works, until the stream isn’t sequential. The fix is to use the actual identity field that the protocol already provides.

This is a general principle: if the protocol gives you an ID field, use it. Don’t rely on ordering assumptions that hold today but break when concurrency enters.

2. The same bug pattern repeats across API surfaces

Bifrost supports two streaming formats: Chat Completions (the classic tool_calls array) and Responses API (the newer ItemID-based format). Both had the same routing flaw because the accumulation logic was duplicated rather than shared. The reviewer catching this in a different file was the most valuable part of the review.

3. OSS contributions are about context, not complexity

The fix itself is straightforward — build a map, use it for lookup. The hard part was:

• Finding the bug in a codebase I’d never seen before
• Understanding why the existing approach was wrong
• Testing with realistic interleaved deltas
• Responding to review feedback about a parallel code path I hadn’t initially noticed

The code change was ~100 lines. The context-building was hours.

4. Production telemetry is the best bug report

The issue report included the exact malformed tool name from production: search_circularsearch_circularsearch_circular. That single string made the bug immediately reproducible and the fix trivially verifiable. If you’re filing a bug, include the malformed output — it’s worth more than a paragraph of description.

Links

• PR: fix: preserve parallel tool call deltas — #1830
• Issue: Streaming tool call deltas merge in parallel tool calling — #1829
• Project: Bifrost AI Gateway — 2.6k ⭐, fastest enterprise AI gateway

I recently dove deep into how LLMs like GPT-3, GPT-4, and open-source models generate text, and I was fascinated by the sheer number of knobs and dials available to control their behavior. These aren’t just abstract concepts,they’re the tools that make the difference between getting a coherent essay or a rambling mess, between creative poetry or repetitive drivel.

In this post, I’ll break down the key generation-time hyperparameters with practical examples, mathematical foundations, and real-world implications. We’ll go beyond the surface-level explanations and understand why each parameter matters and how to tune them effectively.

Temperature: The Creativity Knob

Temperature is arguably the most important parameter you’ll encounter when using LLMs. It’s like adjusting the “personality” of the model.

What it does: Temperature scales the logits (raw prediction scores) before they’re converted to probabilities via softmax. Mathematically:

where z_i is the logit for token i, and T is temperature.

Why it works: At lower temperatures (T < 1), the probability distribution becomes sharper, high-probability tokens become even more likely, leading to more predictable, conservative outputs. At higher temperatures (T > 1), the distribution flattens, giving low-probability tokens a better chance.

Real-world example: When generating code, you want T=0.1 for deterministic, correct syntax. For creative writing, T=0.8-1.2 gives you that spark of originality. I’ve seen T=2.0 produce genuinely surprising and creative outputs, though sometimes bordering on incoherent.

Typical range: 0.0 (completely deterministic) to 2.0 (highly random). Most applications use 0.7-1.0.

Top-k Sampling: Quality Control

Top-k is like telling the model, “Only consider your top k most confident predictions.”

What it does: After computing all token probabilities, sort them and keep only the top k tokens, then renormalize their probabilities.

Why it works: It prevents the model from picking extremely unlikely tokens that might derail the generation. For example, if the model is writing about “coffee,” top-k=50 ensures it doesn’t suddenly switch to discussing quantum physics unless that was already in its top 50 predictions.

Practical tip: For factual writing, use lower k (10-40) to stay on topic. For creative tasks, higher k (50+) allows more flexibility. I’ve found k=1 essentially gives you greedy decoding,always pick the most likely token.

Top-p (Nucleus Sampling): Dynamic Quality Control

Top-p takes top-k’s idea but makes it adaptive.

What it does: Instead of a fixed number of tokens, select the smallest set whose cumulative probability exceeds p. So if p=0.9, it might use 10 tokens one time and 50 tokens another time, depending on the probability distribution.

Why it works: In scenarios where the model is very confident (sharp distribution), it uses fewer tokens. When uncertain (flat distribution), it considers more options. This is particularly useful for maintaining coherence while allowing creativity.

Example: Writing a technical explanation,early tokens are predictable, so top-p might use 5-10 tokens. When getting creative, it expands to 20-30 tokens. I’ve seen this outperform top-k in many creative writing tasks because it adapts to the model’s confidence level.

Typical range: 0.8-1.0. Values below 0.5 can be too restrictive.

Logit Bias: Manual Steering

This is the parameter for when you want to force the model’s hand.

What it does: Directly add or subtract values from specific token logits before sampling:

Why it works: Want to ban certain words? Set their bias to -100. Want to encourage specific terminology? Add positive bias. I once used this to prevent an LLM from using certain brand names in generated content.

Practical applications: Content moderation, style enforcement, or domain-specific vocabulary control. Particularly useful in enterprise settings where you need to avoid certain terms.

Repetition Penalty: Fighting the Echo Chamber

Ever noticed how LLMs can get stuck repeating phrases? This parameter fights that.

What it does: For tokens that have appeared recently, divide their logits by a penalty factor r > 1:

Why it works: It makes previously used tokens less attractive, encouraging the model to explore new vocabulary. Without this, models can get into loops like “The best coffee is the best coffee is the best coffee…”

Real-world use: Essential for long-form content generation. I use values around 1.1-1.2 for most tasks, but for very creative work, sometimes 1.05 or lower to allow some intentional repetition for emphasis.

Stop Sequences: The Off Switch

Sometimes you need the model to know when to stop talking.

What it does: Define specific strings that, when generated, halt the generation process.

Why it works: Prevents runaway generation or ensures structured output. For example, when generating code, you might stop at the next function definition.

Examples: "\n\n", "END", "</response>". I once used this to generate structured JSON by setting the stop sequence to "}".

Max Tokens: Setting Boundaries

The word limit for your AI writer.

What it does: Caps the total number of tokens generated in a single response.

Why it works: Prevents excessive output and manages costs/compute time. In API contexts, this also affects billing.

Considerations: Different models have different limits (GPT-4: 4096, Claude: 8192). Always set this lower than the model’s maximum to leave headroom.

Frequency Penalty: Discouraging Word Hoarders

Similar to repetition penalty, but based on overall frequency in the generated text.

What it does: Penalize tokens based on how often they’ve appeared:

Why it works: Prevents overuse of common words. If “the” appears 50 times, it gets increasingly penalized.

Use cases: Particularly effective for generating diverse content or when you want to avoid repetitive language patterns.

Presence Penalty: Topic Freshness

This encourages introducing new concepts rather than revisiting old ones.

What it does: Penalize tokens that have appeared at all in the current generation:

where presence(i) is 1 if the token appeared, 0 otherwise.

Why it works: Promotes topic exploration. Without it, models can get stuck discussing the same ideas repeatedly.

Difference from frequency penalty: Frequency counts occurrences, presence is binary. Use presence penalty for encouraging breadth of topics.

Num Keep: Context Continuity

For long conversations or documents, this maintains coherence.

What it does: When refreshing the context window, retain the last N tokens from the previous context.

Why it works: Prevents abrupt topic changes in long-form generation. Imagine writing a novel,the model needs to remember what happened in the previous chapter.

Typical range: 100-500 tokens, depending on the task complexity.

Seed: Reproducibility Control

For when you need consistent outputs.

What it does: Initializes the random number generator with a fixed value, ensuring identical outputs for identical inputs.

Why it works: Crucial for testing, debugging, and applications requiring deterministic behavior. Without seeding, the same prompt can produce different outputs.

Implementation note: Not all APIs expose seed control, but when available, it’s invaluable for quality assurance.

Repeat Last N: Short-Term Memory

A more targeted repetition control.

What it does: Prevents tokens from the most recent N tokens from being selected.

Why it works: Stops immediate repetition like “I like coffee. I like coffee. I like coffee.”

Typical range: 1-10 tokens. Higher values can prevent natural repetition that’s actually desirable.

Mirostat: Adaptive Temperature

This is for the advanced users who want the model to maintain consistent “surprise” levels.

What it does: Dynamically adjusts temperature to target a specific perplexity level using parameters τ (target perplexity) and η (adjustment rate).

Why it works: Maintains consistent creativity levels throughout generation. Traditional temperature can lead to increasingly erratic outputs as generation progresses.

Typical range: τ: 20-100, η: 0.05-0.2. Requires experimentation to find good values for your use case.

Putting It All Together

These parameters don’t exist in isolation, they interact in complex ways. For most applications, you’ll primarily tune temperature, top-p, and repetition penalties. But understanding all of them gives you the full toolkit for controlling LLM behavior.

Remember: there’s no “perfect” set of parameters. The best settings depend on your specific use case, model, and desired output characteristics. Experimentation and iteration are key.

What parameter have you found most useful in your LLM work? I’d love to hear your thoughts!

ʕ •ᴥ•ʔ