Unified Cache Manager¶
Unified Cache Manager (UCM) persists LLM KVCache to replace redundant computations. When integrated with vLLM, UCM achieves a 3-10x reduction in inference latency across various scenarios, including multi-turn dialogue and long-context reasoning tasks.
-
Prefix Cache
Persist KVCache across requests and reuse it to avoid redundant prefill for multi-turn dialogue and shared prefixes. Supports non-HBM storage media including DRAM, SSD, and remote storage with pipeline, NFS, DS3FS, Mooncake, and compress backends.
-
Observability
Export Prometheus metrics through the vLLM connector and visualize with Grafana to monitor key performance metrics like KVCache hit rate, latency, and throughput in real-time.
-
Trace Mode
Lightweight diagnostic and evaluation mode that records request traces without performing actual KV cache operations, used to simulate theoretical hit rates and validate UCM deployment effectiveness.
Get Started¶
-
Installation
Pick your UCM version, engine, device, OS, and install method, and get the exact command to deploy.
-
Getting Started
Integrate UCM with vLLM, vLLM Ascend, SGLang, and MindIE.
-
Compatibility Matrix
Supported models, platforms, and feature coverage at a glance.
-
KV Cache Calculator
Estimate KV cache memory usage for your model configuration.