Kinference Routes vLLM Requests Toward Work Already Done

The second request in a conversation does not arrive at an empty system. Somewhere in the GPU fleet, a worker may already have processed most of its history. Send the next turn elsewhere, and the cluster can repeat expensive work it has already done.

That is the practical problem behind Buoyant’s November 4 Kinference announcement. The project puts routing decisions above vLLM, the inference engine that actually runs the model. Instead of treating three healthy workers as interchangeable, it tries to choose the one likely to deliver the first token soonest for this particular prompt.

The distinction matters to coding sessions and other applications that repeatedly send a long common context. It also makes the release easier to misunderstand: a strong result for one traffic pattern is not a universal GPU speedup, and the product’s intended feature list is broader than the current alpha. NxCode reviewed the public materials; we have not installed Kinference, operated a GPU cluster or reproduced its benchmark.

The next turn has a location

A key-value cache, usually shortened to KV cache, holds intermediate attention calculations from tokens the model has already processed. Reusing a matching prefix can reduce prefill, the work of reading a prompt before generating its answer. This is not a cache of completed answers: a new continuation still has to be generated.

The cache-routing documentation says Kinference tracks each vLLM worker’s cache content and runtime load. vLLM must emit cache events through ZMQ, a messaging transport. The router uses those observations to predict time to first token, or TTFT, for a specific prompt at each endpoint.

Imagine two workers serving the same model. One has the shared project instructions cached but is busy; the other has a shorter queue and no matching prefix. Cache overlap alone does not settle the choice. Waiting behind the first worker may cost more than recomputing on the second. Kinference’s stated aim is to weigh both pieces of information rather than always prefer the warmest cache.

For an application developer, that changes where latency can come from. The model weights can be identical, and the prompt can be unchanged, yet a different worker assignment can change how much input computation is repeated. Buying a faster model is only one possible response to a slow first token.

Two candidate workers differ in cached context and queue state; the router predicts first-token delay rather than always choosing the warm worker.

Figure: Original NxCode explanation based on Kinference documentation. The workers are illustrative; this is neither a benchmark trace nor a measured routing decision.

The headline has a specific comparison

Buoyant’s benchmark compares two pools, each with three L40S GPUs running vLLM 0.31.0 and Qwen-3.8-27B-FP8, using GuideLLM 0.8.1. The baseline Kubernetes Service randomly selects a pod per new connection; it is not another cache-aware router.

The vendor reports median TTFT falling from 6.88 seconds to 0.31 seconds on shared-prefix traffic. End-to-end median latency falls from 20.5 to 11.3 seconds. With independent prompts, median TTFT is essentially unchanged. Peak shared-prefix goodput rises from 14 to 103 requests per minute, but these peaks occur at different concurrency levels. Goodput counts requests meeting the selected latency threshold, not every completed request. The shared-prefix SLO requires 90% of first tokens within eight seconds; Buoyant describes these limits as unsuitable for interactive chat.

Those conditions are part of the result. The evidence supports an advantage over that baseline on that workload. It does not establish a similar gain on a different GPU, a shorter prompt, a stricter latency target or an existing inference-aware dispatcher.

A reader evaluating the announcement should therefore start with the shape of their requests. Repeated context creates an opportunity for reuse. Mostly unrelated prompts create a different experiment. The useful question is how much redundant input work a router can avoid while keeping queues under control.

A gateway still carries the request

The pinned README, at commit be2a7ad, describes Kinference as a control plane paired with a proxy such as Buoyant Gateway. Its gRPC interface takes an inference request and returns a destination and potentially changed body. This is a division of responsibilities, not a replacement for the serving engine.

The request-flow document makes the sequence concrete. The gateway pushes a body, asks for a pick, retrieves the outgoing body and forwards it to the chosen worker. Later it reports the first token and final token. These reports connect a routing prediction to what actually happened.

That feedback is easy to overlook in a diagram with one arrow from client to model. A dispatch decision leaves temporary state behind: the router has predicted a wait and recorded work it sent. Completion and abort reporting let it release that state. The documentation also describes expiration sweeps for callers that never report back.

This explains why replacing a URL is only the application-facing part of the change. The platform also needs a working relationship among gateway, router and engine observations. A client can keep a familiar request format while the infrastructure underneath acquires new jobs.

The architecture document separates worker discovery, cache tracking, scoring and Kubernetes resource watches. In a code review, these boundaries help locate where a claim should be checked. A graph of GPU utilization cannot, by itself, establish that prompt matching is accurate or that the gateway reports a completed request correctly.

The alpha is smaller than the product description

The website presents a platform that can route, limit and classify inference traffic. The feature pages are more explicit about delivery status. Both token-based limiting and semantic classification say they will arrive in a future release.

The pinned README also warns that the software has not had sustained production exposure. It lists rate limiting, semantic classifiers, Anthropic-style /v1/messages calls and support for multiple Kinference replicas among the missing capabilities. That last limitation concerns the control-plane deployment; it should not be confused with the benchmark’s multiple vLLM workers.

There is another wrinkle: the pinned architecture already discusses replica leadership, while the README still lists multiple replicas as a limitation. That is documentation tension, not enough evidence to announce supported high availability. Code paths, written design and a supported operating mode can develop at different speeds.

Similarly, a policy example that refers to a prompt’s sensitivity does not establish that a shipping classifier reliably detects sensitive content. The example explains a desired decision once a tag exists. It does not measure the accuracy of producing that tag.

This is a useful release to explain without turning it into adoption advice. The routing mechanism is inspectable, but a team depending on quotas, a particular protocol or control-plane redundancy needs a version-specific answer for those requirements. The announcement alone does not provide it.

Tenant labels are not tenant identity

The tenant guide says Kinference reads a configured request header and trusts it unconditionally. If callers can choose that value, a usage chart can attribute traffic to the wrong tenant. Reading a label and authenticating the caller are different operations.

For an internal model service, the consequence is concrete. Team-level accounting is meaningful only when the surrounding request path supplies an identity the user cannot casually rewrite. Kinference’s header interpretation does not settle that surrounding trust question.

The documentation is not fully consistent about metrics defaults, either. The tenant guide describes tracking every tenant by default; the metrics reference says per-tenant labels are off by default and documents an allowlist. The reference estimates roughly 91 time series for each model, quality-of-service class and tenant combination. More detailed attribution can therefore carry a real monitoring cost.

We would not silently choose one default and present it as verified behavior. The conflict is a reason to check the exact deployed version and configuration. It also prevents a false inference: the presence of tenant metrics is not proof that tenant limits are currently enforced.

A successful request can still have degraded routing

The pinned request flow says that when tokenization fails through every available path, scoring can fall back to load alone instead of refusing the pick. The metrics reference exposes tokenization availability and degraded scoring, along with predicted latency. A request can therefore succeed while losing the cache-aware part of its placement.

That behavior gives operators something more useful than a single up-or-down signal. They can distinguish whether a model returned text from whether the router had the information needed for its intended decision. Missing cache-usage reports also deserve their own category; they are not evidence that no cached work was used.

For NxCode readers building products around inference, this is the release’s strongest idea: latency belongs to a request moving through a changing fleet, not just to a model name. Kinference makes some of that movement visible and configurable. Its alpha status and conflicting documentation remain part of the story. The next meaningful evidence is a clearly supported release and measurements on another real workload, with both successful answers and routing quality in view.

Kinference Routes vLLM Requests Toward Work Already Done