MooncakeConnector Usage Guide

About Mooncake

Mooncake aims to enhance the inference efficiency of large language models (LLMs), especially in slow object storage environments, by constructing a multi-level caching pool on high-speed interconnected DRAM/SSD resources. Compared to traditional caching systems, Mooncake utilizes (GPUDirect) RDMA technology to transfer data directly in a zero-copy manner, while maximizing the use of multi-NIC resources on a single machine.

For more details about Mooncake, please refer to Mooncake project and Mooncake documents.

Prerequisites

Installation

Install mooncake through pip: uv pip install mooncake-transfer-engine-cuda13.

vLLM defaults to CUDA 13. On a CUDA 12 environment install mooncake-transfer-engine instead — the two are the same release built against different CUDA majors, and the wrong one fails to import with libcudart.so.<major>: cannot open shared object file.

Refer to Mooncake official repository for more installation instructions

Usage

Prefiller Node (192.168.0.2)

vllm serve Qwen/Qwen2.5-7B-Instruct --port 8010 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}'

Decoder Node (192.168.0.3)

vllm serve Qwen/Qwen2.5-7B-Instruct --port 8020 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}'

Proxy

python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --prefill http://192.168.0.2:8010 --decode http://192.168.0.3:8020

Now you can send requests to the proxy server through port 8000.

For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV cache), see ECMooncakeConnector Usage Guide.

Environment Variables

• VLLM_MOONCAKE_BOOTSTRAP_PORT: Port for Mooncake bootstrap server

  • Default: 8998
  • Required only for prefiller instances
  • For headless instances, must be the same as the master instance
  • Each instance needs a unique port on its host; using the same port number across different hosts is fine

• VLLM_MOONCAKE_CONNECTOR_TIMEOUT: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup and metadata queries. (Optional)

  • Default: 30.0
  • Global rank 0 hosts the bootstrap server in the same process that mounts the Mooncake transfer engine's host segment, so a large MOONCAKE_GLOBAL_SEGMENT_SIZE can delay responses for several seconds during startup
  • Raise this if bootstrap server timeout warnings appear on hosts with very large host-memory segments
  • Bootstrap registration uses up to 3 attempts internally, with exponential backoff capped at 10 seconds; connection errors and timeouts are both retried
  • Once registration attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
  • Metadata queries reuse the same timeout and 3-attempt limit, with a fixed 100ms delay between attempts. If all attempts fail, the error is logged and waiting queries are notified

• WITH_NVIDIA_PEERMEM: Selects how mooncake registers GPU memory for RDMA. Read by mooncake, not vLLM.

  • Default: 1, which uses ibv_reg_mr() and requires the nvidia-peermem kernel module to be loaded
  • Set to 0 to use the DMA-BUF path, which does not need that module. Required on hosts where nvidia-peermem is not loaded, such as GB200
  • With the container image, pass it at run time: docker run -e WITH_NVIDIA_PEERMEM=0 ...
  • Symptom when left unset on such a host: Failed to register memory <addr>: Bad address [14] from rdma_context.cpp, and KV transfers fail

• VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT: Timeout (in seconds) for automatically releasing the prefiller’s KV cache for a particular request. (Optional)

  • Default: 480
  • If a request is aborted and the decoder has not yet notified the prefiller, the prefill instance will release its KV-cache blocks after this timeout to avoid holding them indefinitely.
  • When kv_lease_duration is set, the lease decides when those blocks are released instead. This timeout still bounds how long a pull waits for a prefill that has not finished.

KV Transfer Config

KV Role Options

• kv_producer: For prefiller instances that generate KV caches
• kv_consumer: For decoder instances that consume KV caches from prefiller
• kv_both: Enables symmetric functionality where the connector can act as both producer and consumer. This provides flexibility for experimental setups and scenarios where the role distinction is not predetermined.

kv_connector_extra_config

• num_workers: Size of thread pool for one prefiller worker to transfer KV caches by mooncake. (default 10)
• mooncake_protocol: Mooncake connector protocol. (default "rdma")
• device_name: Comma-separated whitelist of RDMA devices (e.g. "mlx5_0,mlx5_1") to restrict topology discovery to. Empty discovers every device. Useful on hosts exposing a mix of InfiniBand and RoCE ports, where both peers must settle on the same link layer.
• kv_lease_duration: Seconds the prefiller holds a finished request's KV blocks before the decoder must renew them. (default unset)
  • Unset keeps the fixed VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT.
  • When set, the decoder renews the lease every kv_lease_duration / 6 seconds from the time the request arrives until its pull settles, and each renewal extends it to at least kv_lease_duration * 2 / 3 seconds from then, as in NixlConnector. Blocks of a decoder that crashed are freed within about kv_lease_duration seconds, while requests queued on a busy decoder keep their blocks.
  • Set the same value on prefill and decode instances. A decoder without it does not renew, so its requests that wait longer than the lease lose their prefilled KV; a prefiller without it rejects renewals, and the decoder logs a warning. Prefill instances from releases without this option log an error for each renewal they receive, so upgrade them before enabling it on decode instances.
• bootstrap_server_address: Optional "host:port" of an already running Mooncake bootstrap server. All prefiller ranks register with this endpoint instead of launching an embedded server. The caller owns the server lifetime and must configure the proxy to use the same endpoint. The example proxy assumes the bootstrap server is on the prefiller API host; a different host requires a proxy that supplies that address in remote_bootstrap_addr. Omit this option to use the embedded server and VLLM_MOONCAKE_BOOTSTRAP_PORT.

Prometheus metrics

The following Prometheus metrics are exported when MooncakeConnector is active, each with the standard model_name and engine labels:

Metric name	Type	Description
vllm:mooncake_xfer_time_seconds	Histogram	Per-transfer copy duration (seconds).
vllm:mooncake_bytes_transferred	Histogram	Bytes moved per transfer.
vllm:mooncake_num_descriptors	Histogram	Descriptor count per transfer.
vllm:mooncake_num_failed_transfers	Counter	Cumulative count of failed KV-block transfers.
vllm:mooncake_num_failed_recvs	Counter	Cumulative count of failed KV-block receives.
vllm:mooncake_num_kv_expired_reqs	Counter	Requests whose KV blocks expired on the prefiller before the decoder read them (tracked on the P instance).

Note

Mooncake is push-based: the prefiller (P) performs the transfer, so the three histograms are only populated on P instances, while the decoder (D) reports only the failures it observes.

Tip

High vllm:mooncake_num_kv_expired_reqs indicates that the prefiller's lease duration is too short for your network or workload. Increase VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT, or set kv_lease_duration on both instances so that queued requests keep their blocks.

Example Scripts/Code

Refer to these example scripts in the vLLM repository:

• run_mooncake_connector.sh
• mooncake_connector_proxy.py