Kimi K2.6 — Capacity under realistic Fabric load

Capacity → subscriber-count derivation

1. Empirical bench (below) shows H200 node sustains ~100 simultaneous active users at "good to excellent" experience with real Fabric usage pattern.
2. Real data: peak hour in last 14 days saw 8 simultaneously-active users out of 30 subscribers → ~27% peak concurrency ratio.
3. Conservative pricing choice: assume 50% peak concurrency, so 2 subscribers per concurrent seat.
4. Per-node subscriber cap scales with hardware throughput at SLO floor. H200=200 (measured); B200≈700 (FP4 TP=4 × 2 replicas, Blackwell-native config); B300≈740 (modest gain over B200 since K2.6 already fits in B200 HBM).

Hardware throughput comparison (raw — no spec decode, no idle)

From SemiAnalysis InferenceX benchmark: Kimi K2.5, ISL=8192, OSL=1024, vLLM, INT4, TP=8, no speculative decoding. Per-user decode tok/s at fixed concurrency.

Every single user got 🟢 Excellent service. Min user 63.5 tok/s, median 83.9, max 104.7.

The bench measures two kinds of "failure":

• HTTP non-200 — model rejected the request. Count across all runs: 0.
• Transport disconnect (TCP reset / aiohttp "Server disconnected") before any HTTP response. 19 across all 4306 turns ≈ 0.44%.

Root cause of the H200-local failures: VLLM_HTTP_TIMEOUT_KEEP_ALIVE=5 (uvicorn default). Any aiohttp keep-alive connection idle for >5 s gets closed server-side. Empirical Fabric think-time has ~25% of gaps over 13 s, so this fires on long-idle users.

What this means for production: LiteLLM is configured with num_retries: 1 — any transport disconnect transparently retries on a fresh connection. Effective production failure rate ≈ 0.17% × 0.17% = ~0.0003%, well under the 0.1% target.

Hardening applied after bench completed:

1. ✅ VLLM_HTTP_TIMEOUT_KEEP_ALIVE=75 set on the vllm-kimi26 container — server-side keep-alive bumped from 5 s → 75 s, eliminates idle-disconnect for any reasonable think time.
2. ✅ -o TCPKeepAlive=yes added to the Buzz → H200 h200-tunnel.service systemd unit — extra resilience against middlebox idle timeouts on the tunnel itself.

Re-measurement of the failure rate with these fixes in place is pending (would require restarting the isolated bench, which would knock real users off Kimi again).

Workload

• Shared system prompt: Fabric's actual manual_chat.txt + coding_agent.txt concatenated (~11k tokens) — same prefix the real product sends.
• Random programming questions from a pool of 20 realistic asks (algorithms, refactors, language idioms, system design).
• Multi-turn: each "user" runs a closed-loop conversation; conversation resets every 6 turns. Each turn appends the prior assistant response to grow the context.
• Output cap: max_tokens=384 per turn.

Think-time distribution — empirically derived from real Fabric traffic

Pulled from LiteLLM_SpendLogs on Server A: 14 days of real Fabric traffic, 32 real users, 48,939 intra-session gaps, filtered to fabric-xlarge + fabric-large only (excludes title-gen, summarization, etc.).

Per-turn think time is sampled from this distribution (weighted bucket → uniform within bucket). The dominant mode (57% under 5 s) reflects Fabric's agentic loop firing sequential model calls — one human action triggers many sub-requests in seconds.

Test isolation

• Buzz → H200 SSH tunnel stopped (systemctl stop h200-tunnel.service) for the duration.
• Real-user requests for fabric-xlarge cascaded to qwen3.5-397b via the LiteLLM fallback chain (verified before bench started).
• N=5–40 ran from Mac via a separate SSH tunnel; N=60+ ran directly on the H200 box itself for cleanest numbers.

Runtime config

vllm/vllm-openai:latest
  --model /models/Kimi-K2.6 (INT4)
  --tensor-parallel-size 8
  --max-num-seqs 128
  --max-model-len 262144
  --gpu-memory-utilization 0.92
  --enable-prefix-caching
  --no-async-scheduling          # defense vs LMCache #2356 (LMCache no longer in stack)
  --speculative-config '{"model":"/models/kimi-k2.6-eagle3","method":"eagle3","num_speculative_tokens":3}'
  --enable-auto-tool-choice --tool-call-parser kimi_k2 --reasoning-parser kimi_k2
  --init   --restart unless-stopped  --label autoheal=true
  HEALTHCHECK curl /health every 30 s, 15 min start-period
  GPU KV cache pool: 734,480 tokens

• "User" = a closed-loop synthetic worker, not a real Fabric user. Real users have heterogeneous prompt sizes, varying tool-call depths, and bursts.
• Output capped at 384 tok/turn. Real conversations sometimes generate 1–2k tokens; longer outputs eat more compute per turn (TPS unchanged but latency stretches).
• Empirical distribution conflates two behaviors: human think time + agentic-loop inner-loop. The 57% <5 s bucket is mostly the latter.
• Cold prefix-cache numbers would be worse. The empirical workload shares prefix heavily; first request per session takes ~10× the TTFT.
• Numbers are "right-now" capacity — assumes no other models on the box. fabric-large stays on H100; fabric-medium on GH200; nothing competes with Kimi on H200.