Serving Macaron-V1 at Scale: Page-Level 2D KV Resharding

When we released Macaron-V1, the response was immediate. A model whose defining trait is long personal context (agents that carry weeks of real interaction history) also generates serving load of exactly the kind traditional LLM stacks are worst at: many concurrent, long-context, long-running sessions that refuse to be truncated or summarized. The launch traffic made this concrete. Within days the cluster was saturated not on compute, but on the thing long-context serving runs into first: decode-side KV memory. The wall was not hypothetical; it was the first thing the production cluster hit.

This post documents one of the deployment optimizations we built in response: page-level 2D resharding, which splits KV sharding into two orthogonal dimensions and reshapes during the RDMA transfer itself. It is the piece that let us stop throwing hardware at memory-bound concurrency and start serving the long-context load Macaron-V1 actually creates.

Background and contribution. Prefill–decode disaggregation is a well-studied architecture for LLM serving: DistServe [14] and Splitwise [15] established the benefits of separating compute-bound prefill from memory-bound decode, and production systems carry the KV cache between the two sides over RDMA using engines such as Mooncake, Moonshot AI's (Kimi's) open-source KVCache-centric transfer engine, which sglang integrates as its disaggregated-transfer backend [16]. These works treat the KV transfer as a whole-sequence move: they do not consider sharding KV along different axes on each side. Context parallelism is also established; sglang ships PCP (Prefill CP) [5,6] and DCP (Decode CP) [1,2], but the two are developed as separate features. What is missing is the combination: when Prefill shards KV by layer (CP layer-split) and Decode shards by page (DCP), the transfer must simultaneously rearrange both the layer and the sequence dimensions. To our knowledge, no prior work, academic or open-source, addresses this page-level 2D reshard, and in production practice, mixing CP + PD + DCP in one cluster is effectively unsupported. Our contribution is the transfer algorithm that makes this combination work correctly: filter page indices at the source, write directly into the destination's DCP layout, and keep every cross-rank collective safe under prefix divergence (size-broadcast, consensus prefix, per-request bootstrap). This is the gap Macaron-V1's traffic exposed in production: a personal agent that carries real usage history must serve the full interaction, not a truncated summary, and the serving stack had no transfer path that could keep CP layer-split and DCP page sharding consistent across nodes. Our contribution is the transfer algorithm that closes it.

1. Motivation

The memory wall. Loading 4 LoRA experts + GLM-5.2 FP8 weights simultaneously, with EAGLE speculative decoding enabled, leaves very little KV cache per GPU on the Decode side. GLM-5.2 ships a built-in Multi-Token Prediction (MTP) layer; enabling EAGLE-style speculative decoding reuses that layer and reserves additional memory for its CUDA graphs and a dedicated draft KV pool. On B300 (Blackwell Ultra, 288GB HBM3e), without sequence sharding (DCP=1) this leaves only ~1.17M tokens of KV cache per rank (~2.0M without EAGLE speculative decoding). This severely limits Decode concurrency: when multiple long-context requests (e.g., 500K–1M tokens) arrive simultaneously, insufficient KV cache capacity causes request queuing and TTFT degradation. The fix is Decode Context Parallelism: sharding the sequence across decode ranks by page scales KV capacity linearly with the number of ranks, and §5 details how DCP attention and transfer work.

Table 1: Measured KV cache capacity and TPOT on B300 (TP8, FP8, page size 64) across DCP settings. "conc" = number of concurrent requests; TPOT is measured at 1 / 16 / 32 concurrent requests to show the latency cost of concurrency. With DCP=1 and EAGLE speculative decoding enabled, KV capacity is only 1.17M tokens (the MTP draft reserves its own KV pool and CUDA graphs); DCP=4 scales to 4.67M (4×), DCP=8 to 9.34M (8×). TPOT overhead is only ~5–15% (DCP1→DCP8), while EAGLE at DCP8 reduces TPOT from 30.8ms to 8.6ms (3.6×).

The core value of DCP: with $N$ DCP ranks, KV cache capacity scales linearly by $N$×, while TPOT increases only ~5–15%. This means DCP=4 can serve 4× more concurrent long-context requests on the same hardware with minimal latency cost. EAGLE speculative decoding further reduces TPOT: at DCP=8, EAGLE brings TPOT from 30.8ms down to 8.6ms (3.6× speedup), achieving low latency while maintaining high concurrency.

Production impact is a cliff, not a slope. Measuring the concurrency gain of DCP is genuinely hard to do fairly: the numbers swing dramatically with the workload (input length distribution, prefix-cache hit rate, EAGLE accept length), so any single figure should be read as one data point, not a universal number. Still, in our production environment, which is predominantly agent workloads, the difference is stark. With DCP=1 and EAGLE enabled, the decode side has only ~1M tokens of KV; the cluster starts queueing at 20–40 concurrent requests. When that happens, TTFT spikes because decode is serialized behind the in-flight requests: a new request cannot start decoding until an earlier one finishes, and requests spend ~70% of their time waiting on the transfer, not computing. The degradation is discontinuous: as long as decode KV fits your concurrency demand, TTFT stays low at normal prefix-cache hit rates; the moment you exceed capacity by even a single request's KV, latency is dictated by how long the current decode batch takes to drain. Decode compute is actually idle during this regime; the bottleneck is purely memory, which is why adding more decode machines relieves the queue but cannot fix the underlying compute-underutilization caused by memory-bound concurrency.

After enabling DCP=4 on the same hardware and configuration, the production cluster comfortably serves 100+ concurrent requests with the same memory footprint. Measured on live traffic, peak decode throughput roughly doubled from ~800 token/s to ~1800 token/s, and can go higher still if we accept trading some TPOT for more parallelism.

2. Problem Statement

In PD disaggregated inference, Prefill and Decode run on separate nodes, with KV cache transferred via mooncake RDMA. Prefill-Decode disaggregation is the standard architecture for separating the compute-intensive prefill from the memory-bound decode phase [9,10]. The two sides shard KV along orthogonal dimensions, forming a 2D reshard problem:

Table 2: The two orthogonal KV sharding dimensions between Prefill and Decode.

The two sides shard along orthogonal axes: Prefill (CP layer-split) distributes the 78 layers across ranks, Decode (DCP) distributes the sequence across ranks by page. Transfer must therefore rearrange both dimensions at once. Let the Prefill side have $N_{cp}$ CP ranks and the Decode side have $N_{dcp}$ DCP ranks. Each Prefill CP rank $r$ holds layer set $\mathcal{L}_r \subset \{0, \ldots, 77\}$ and all pages $\mathcal{P}_{\text{full}}$. Each Decode DCP rank $d$ holds page subset $\mathcal{P}_d = \{p \mid p \bmod N_{dcp} = d\}$ and all layers. Transfer target: for each path $(r, d)$, transfer $\mathcal{L}_r \times \mathcal{P}_d$ KV from Prefill rank $r$ to Decode rank $d$.

$$2D\ Reshard\ Target: \quad \forall r \in [0, N_{cp}), \ \forall d \in [0, N_{dcp}): \quad \text{Transfer}(\mathcal{L}_r, \mathcal{P}_d, r \to d)$$

Technical Challenge: DCP shards the sequence by page across multiple ranks on the Decode side, while the Prefill side uses CP Layer-Split (by layer). The two sharding dimensions are orthogonal, requiring page-level 2D reshard for KV transfer: simultaneously rearranging both layer and sequence dimensions, with correctness guarantees for cross-rank collectives (rank-invariant gates + size-broadcast + consensus prefix).

3. Architecture Overview

Configuration Choice: CP=8 + DCP=4. After balancing memory vs speed, we adopt Prefill-side CP=8 (8 CP ranks, each holding $78/8 \approx 10$ layers) + Decode-side DCP=4 (4 DCP ranks, each holding $1/4$ pages). CP=8 maximizes Prefill-side memory savings from layer splitting, while DCP=4 provides $4\times$ KV capacity expansion on the Decode side with acceptable TPOT overhead (~5–15%). All examples in this paper use CP=8 + DCP=4, but the algorithm is fully general: $N_{cp}$ and $N_{dcp}$ can be independently configured to any value.

Transfer Path: each Prefill CP rank $r$ iterates over its owned layers $\mathcal{L}_r$, applies page-level DCP filtering to all pages per layer, and writes via mooncake RDMA to the target Decode DCP rank $d$'s local physical page slots. $N_{cp}$ CP ranks cover all 78 layers, $N_{dcp}$ DCP ranks cover all pages. Left: horizontal bands = CP by layer (per-rank holds all pages). Right: vertical bands = DCP by page (per-rank holds all layers). Highlighted 2D cross-block during transfer = current $(r, d)$ path's KV data.

4. CP Layer-Split (Prefill Side)

Prefill Context Parallelism with layer-split distributes transformer layers across CP ranks, with each rank computing only its owned layers' KV [5,6]. This section details the layer assignment and the prefix broadcast protocol that keeps all ranks consistent.

4.1 Layer Assignment

78 Transformer layers are contiguously partitioned across CP ranks. $78 \bmod 8 = 6$; the first 6 ranks get 10 layers, the last 2 get 9. This matches the implementation, where $\text{base} = \lfloor L / N_{cp} \rfloor$ and $\text{extra} = L \bmod N_{cp}$; each rank $r$ owns a contiguous layer range $\mathcal{L}_r$:

$$\mathcal{L}_r = \left[ r \cdot \left\lfloor \frac{L}{N_{cp}} \right\rfloor + \min\!\left(r,\ L \bmod N_{cp}\right), \ (r+1) \cdot \left\lfloor \frac{L}{N_{cp}} \right\rfloor + \min\!\left(r+1,\ L \bmod N_{cp}\right) \right)$$

# Layer assignment: 78 layers → 8 CP ranks (div + remainder, not ceil)
function cp_layersplit_layer_range(N_layers, N_cp, r):
    base ← N_layers div N_cp          # 78 / 8 = 9
    extra ← N_layers mod N_cp          # 78 % 8 = 6
    start ← r × base + min(r, extra)
    end ← start + base + (1 if r < extra else 0)
    return [start, end)                # half-open interval

# Result:
# Rank 0: L[0, 10)   → 10 layers
# Rank 1: L[10, 20)  → 10 layers
# Rank 2: L[20, 30)  → 10 layers
# Rank 3: L[30, 40)  → 10 layers
# Rank 4: L[40, 50)  → 10 layers
# Rank 5: L[50, 60)  → 10 layers
# Rank 6: L[60, 69)  → 9 layers (remainder)
# Rank 7: L[69, 78)  → 9 layers (remainder)

# KV pool: per+1 slots per rank (owned layers + 1 transient for broadcast)
function local_layer_count(N_layers, N_cp, r):
    [start, end) ← cp_layersplit_layer_range(N_layers, N_cp, r)
    return (end - start) + 1   # +1 = transient slot

Animation 2: 78 layers assigned to 8 CP ranks, highlighting the currently active rank's layers.

4.2 Prefix KV Broadcast: Size-Broadcast Protocol

Non-owner ranks need prefix KV from the owner rank. Under CP layer-split, each rank computes its own radix-tree prefix match locally; HiCache prefetch is local-only, so the matched prefix length can diverge across ranks [11]. A naive broadcast keyed on the local prefix would let some ranks skip the NCCL collective while others participate, permanently desynchronizing the call counts and deadlocking. The size-broadcast protocol fixes this by making every rank agree on the owner's element count before any data transfer:

$$n = |\text{slots}_{\text{owner}}| \quad \text{(broadcast } n \text{ to all ranks first)}$$ $$\text{if } n = 0: \ \forall r: \text{skip} \quad (\text{consistent})$$ $$\text{if } n > 0: \ \forall r: \text{broadcast } n \text{ KV elements} \quad (\text{all ranks participate})$$

The count itself is broadcast with a 1-element NCCL broadcast that all ranks must join. The decision to skip or transfer is then uniform: either every rank skips (owner has no prefix) or every rank participates in the same n-element broadcast. Non-owner ranks only write the received KV into their transient pool when their local slot count matches n; otherwise they consume the collective without committing; consistency comes from the shared count, not from per-rank tree state. This is exactly the fix landed for the HiCache prefetch divergence case (the previous if not prefix_slots_list: return path caused a permanent NCCL call-count mismatch).

function broadcast_owner_layer_prefix(layer_id, forward_batch, kind):
    owner ← cp_layersplit_owner_rank(layer_id, ...)
    prefix_lens ← forward_batch.extend_prefix_lens_cpu

    # None guard: normalize to zeros (prevent per-rank None divergence)
    if prefix_lens = None:
        prefix_lens ← zeros(len(req_pool_indices))

    # Step 1: compute element count
    count ← |prefix_slots|

    # Step 2: broadcast count to ALL ranks (1-element, mandatory)
    n_tensor ← tensor([count])
    broadcast(n_tensor, src=owner, group=cp_group)
    n ← n_tensor.item()

    # Step 3: consistent skip or consistent broadcast
    if n = 0: return   # ALL ranks skip
    # Step 4-5: broadcast n KV elements; non-owner writes to transient_pool

5. DCP (Decode Side)

5.1 Page-Level Round-Robin Sharding

Decode-side KV cache is allocated in pages (64 tokens each) via round-robin. The DCP allocator size is $M \times N_{dcp}$ (virtual slots), with each DCP rank holding $1/N_{dcp}$ of pages:

$$\text{owner}(p) = p \bmod N_{dcp}, \quad p \in \{0, \ldots, M \cdot N_{dcp} - 1\}$$ $$\text{slot}_{\text{local}}(p) = \left\lfloor \frac{p}{N_{dcp}} \right\rfloor \in \{0, \ldots, M - 1\}$$

Animation 3: 16 pages sharded to 4 DCP ranks via page_idx % 4 round-robin.

5.2 DCP Attention: All-Gather + LSE Merge

Each DCP rank holds only $1/N_{dcp}$ of KV pages; attention output requires all-gather + LSE merge to combine:

$$O_{\text{full}} = \text{AllGather}(O_{\text{local}}) \quad \text{(concat along seq dim)}$$ $$\text{LSE}_{\text{merged}} = \log\!\left(\sum_{d=0}^{N_{dcp}-1} \exp(\text{LSE}_d)\right) \quad \text{(merge across DCP ranks)}$$

5.3 Communication Overhead Analysis

DCP introduces two sources of communication overhead per decode step:

Table 3: Per-step communication overhead breakdown at DCP=4, TP=8, 78 layers, on GLM-5.2. Sizes use the model's real KV layout: DSA/MLA latent = $kv\_lora\_rank + qk\_rope\_head\_dim = 512 + 64 = 576$ bytes/token/layer (FP8), so a 32K-token sequence holds $78 \times 576 \times 32768 \approx 1.47$ GB of KV. The per-step all-gather moves the full 1.47 GB (each DCP rank pulls the other 3/4), which is the dominant overhead and grows linearly with $N_{dcp}$; it is amortized by EAGLE speculative decoding (sglang's recommended GLM-5.2 config: speculative_num_steps=5, speculative_eagle_topk=1, speculative_num_draft_tokens=6; accept length typically reaches 5+ tokens with GLM-5.2's strong MTP head). The 32 PD transfer paths execute in parallel: each Prefill CP rank independently sends to each Decode DCP rank via mooncake RDMA, so transfer latency ≈ single-path latency (not 32×); with 2D reshard each path carries only $1/N_{dcp}$ of the sequence (≈368 MB for 32K at DCP=4), not the full KV.

TTFT impact: PD disaggregation adds one RTT for KV transfer (mooncake RDMA, ~0.5-2ms for typical 4K-32K token sequences). The 2D reshard adds page-level filtering overhead (~0.1ms for 256 pages × 4 DCP ranks), negligible compared to prefill compute time. Measured TTFT on B300: 0.38s (DCP=4, 1K input) to 1.7s (DCP=4, 32K input), dominated by prefill compute, not transfer. For 500K+ token sequences, DCP=4's 4× KV capacity is essential: a single 500K request consumes ~7.8K pages, exceeding DCP=1's 1.17M token capacity per rank.

6. Page-Level 2D Resharding (Core)

6.1 Design Evolution: From Token-Level to Transfer-Time Sharding

The current design, page-level sharding during RDMA transfer, was not the initial approach. The design went through two iterations:

• Initial approach: decouple sharding from transfer. The original idea was to shard KV at the token level for maximum flexibility, and keep the sharding logic separate from the PD transfer. Concretely, Prefill would transfer the full, unsharded KV to Decode via RDMA, then Decode would run an inplace_shard_dcp pass to redistribute tokens into the DCP rank layout. This is simple to reason about (the transfer knows nothing about DCP) but wasteful: the full sequence is transferred to every DCP rank (each rank receives $N_{dcp}\times$ the data it needs), only to discard $\frac{N_{dcp}-1}{N_{dcp}}$ of it during the in-place reshard.
• Final design: shard during transfer. We realized that the transfer itself is the natural sharding point. Since mooncake RDMA operates at page granularity (the physical page_size = 64 tokens), and DCP already shards by page (owner(p) = p mod N_dcp), we can filter page indices before the RDMA write, sending only the $1/N_{dcp}$ pages each DCP rank actually needs. The RDMA write lands directly in the correct DCP-local slot, eliminating the post-transfer reshard entirely (dcp_kv_shard_pages = None).

Key insight: The transfer unit is the sharding unit. Because RDMA transfers at page granularity and DCP shards by page, filtering at the source (Prefill) naturally produces the correct destination (Decode) layout: no intermediate full-sequence copy, no post-processing. The sharding is not a separate step; it is the transfer itself.

Bandwidth comparison (naive vs. 2D reshard). On GLM-5.2, each token's DSA KV latent is 576 B per layer ($kv\_lora\_rank + qk\_rope\_head\_dim = 512 + 64$, FP8), so a 32K-token request holds $78 \times 576 \times 32768 \approx 1.47$ GB of KV. The naive approach sends the full sequence to every DCP rank: at DCP=4 that is $4 \times 1.47 = 5.88$ GB of RDMA traffic, of which 4.41 GB (75%) is immediately discarded by inplace_shard_dcp. Transfer-time sharding sends only the pages each rank owns: each of the 4 DCP ranks receives $1.47/4 \approx 368$ MB, for a total of 1.47 GB, a 4× reduction in transfer volume, and the data lands directly in its final layout so no post-transfer reshard is needed. For a 500K-token request the gap widens to $4 \times 23$ GB vs. $23$ GB.

6.2 Page Index Filter

This is the core of 2D reshard. Before RDMA send, the Prefill side applies DCP filtering to page indices per layer:

$$\text{mask}(p, d) = \mathbb{1}\!\left[p \bmod N_{dcp} = d\right]$$ $$\text{filtered}(p, d) = \left\lfloor \frac{p}{N_{dcp}} \right\rfloor \quad \text{(compress to local slot)}$$ $$\mathcal{P}_{r \to d} = \left\{ \left\lfloor \frac{p}{N_{dcp}} \right\rfloor \;\middle|\; p \in \mathcal{P}_{\text{full}}, \ p \bmod N_{dcp} = d \right\}$$

# Input:  kv_indices = prefill-side global logical page indices (full set)
# Output: only pages owned by DCP rank d, mapped to local physical slots
function filter_kv_indices_for_dcp_rank(kv_indices, N_dcp, d):
    if N_dcp ≤ 1: return kv_indices
    mask ← kv_indices mod N_dcp = d       # page ownership test
    return kv_indices[mask] div N_dcp    # compress to local slot

Correctness Guarantee:

① kv_indices are prefill-side global logical page indices (slot = pos, DCP allocator size = $M \times N_{dcp}$)

② mask = page_idx % N_dcp == d: keep only pages belonging to the target decode rank

③ // N_dcp: compress global page slot to decode-side local physical page slot

④ RDMA write lands directly in decode's DCP sharded layout, no post-processing needed

6.3 Transfer via Mooncake (sglang's RDMA Transfer Engine)

We use Mooncake, Moonshot AI's open-source Transfer Engine integrated into sglang [16], as the RDMA transport. In the connector's transfer worker loop, each TransferInfo carries the target decode rank's $N_{dcp}$ and $d$. The Prefill side applies DCP filtering to each layer's page indices before sending:

# mooncake/conn.py — transfer worker loop (per layer, per TransferInfo)
N_dcp ← req.dcp_size        # from bootstrap metadata
d     ← req.dcp_rank        # = decode tp_rank % N_dcp

if N_dcp > 1:
    # DCP reshard: filter by decode-side page VALUE parity
    mask ← dst_indices mod N_dcp = d
    transfer_indices ← transfer_indices[mask]
    dst_indices ← (dst_indices[mask] div N_dcp).astype(int32)
    # Log: [DCP-XFER] pages: 256→64 (1/4 of full sequence)

# RDMA write: src=prefill_kv[owned_layer][transfer_indices]
#            dst=decode_kv[local_slot][dst_indices]
send_kvcache(session_id, transfer_blocks)

6.4 Transfer Page Size

Under DCP, PD transfer uses physical page_size (64), not the allocator's virtual page_size ($64 \times N_{dcp} = 256$). Layer-pipelined KV transfer further overlaps RDMA with GPU compute to hide transfer latency [12]:

$$P_{\text{transfer}} = \begin{cases} P_{\text{phys}} = 64 & \text{if DCP} \\ P_{\text{virt}} = 64 \times N_{dcp} & \text{otherwise} \end{cases}$$

# decode.py — DCP transfer page size
if dcp_enabled():
    # Use physical page_size (64) for PD transfer
    # PD transfer writes full 64-token pages in contiguous layout
    kv_transfer_page_size ← token_to_kv_pool.page_size   # 64
    # PD transfer with conn.py rank-filtering + //dcp mapping
    # writes KV directly to DCP page layout. No inplace_shard_dcp needed.
    decode_req.dcp_kv_shard_pages ← None   # skip post-transfer reshard

6.5 2D Reshard Matrix

Animation 4: The 8×4 = 32 transfer paths of 2D reshard, highlighting the current (CP, DCP) path. Each path transfers the layers owned by that CP rank (first 6 ranks get $\lfloor 78/8 \rfloor + 1 = 10$ layers, last 2 get $\lfloor 78/8 \rfloor = 9$, matching §4.1's div + remainder assignment) times $1/N_{dcp}$ pages.

6.6 CP Layer-Split and DCP Transfer Interaction

Under CP layer-split, each Prefill CP rank independently sends its owned layers' KV; no cross-CP-rank coordination is needed for the layer dimension. CP index filtering is disabled in layer-split mode:

$$F_{cp} = \begin{cases} \text{False} & \text{if layer-split or } N_{cp} > 1 \\ \text{True} & \text{if pure TP (} N_{cp} = 1\text{)} \end{cases}$$

function should_filter_cp_indices():
    # Under cp-layersplit: each CP rank owns distinct layers
    # and sends its own pages unfiltered → filtering N/A
    if is_cp_layersplit: return False
    if attn_cp_size > 1: return False   # CP interleave already per-rank
    return enable_all_cp_ranks_for_transfer

6.7 Bootstrap: Decode → Prefill DCP Metadata

The Decode side passes its DCP configuration to Prefill via bootstrap:

$$d = r_{\text{tp}} \bmod N_{dcp} \quad \text{(decode side)}$$ $$\text{DstKVInfo}: (N_{dcp},\ d) \xrightarrow{\text{bootstrap}} \text{Prefill} \quad \text{(per-request)}$$

# Decode side: compute dcp_rank
dcp_rank ← attn_tp_rank mod dcp_size

# Prefill side: learns decode's DCP config from bootstrap metadata
N_dcp ← info.dcp_size        # from DstKVInfo
d     ← info.dcp_rank       # target decode rank's DCP rank
# Each prefill rank sends 1/N_dcp page-shard to each decode DCP rank

Per-request, not per-cluster: DstKVInfo carries (dcp_size, dcp_rank) on every request, not once at startup. Prefill applies DCP filtering per transfer using the target decode rank's metadata, so it never assumes a static cluster-wide DCP topology. Two practical consequences:

• Heterogeneous decode ranks in one cluster. A decode group running with DCP=4 and another running DCP=1 (or even a different DCP size) can coexist: each request tells prefill which (N_{dcp}, d) to reshard for. Prefill-side dcp_size/dcp_rank are only local fallbacks (attn_tp_rank % dcp_size); the authoritative values come from the per-request DstKVInfo.
• DCP configuration can change between requests. Because reshard parameters ride along with the request, a decode node's DCP setting can be reconfigured (e.g., scaling a decode group from DCP=2 to DCP=4) without restarting prefill or renegotiating a global topology; each transfer simply filters by the metadata of the request it is serving.

7. Correctness Guarantees

Table 4: Correctness guarantees of the 2D page-level reshard across all involved collectives.

Core Design: 2D reshard = layer dimension (each CP rank sends its own layers) × page dimension (DCP filter). Each Prefill CP rank independently applies filter_kv_indices_for_dcp_rank, sending $1/N_{dcp}$ of pages to the target Decode DCP rank. Transfer is page-level (64 tokens), with RDMA write landing directly in decode's DCP sharded layout: no post-processing (dcp_kv_shard_pages = None).

8. EAGLE + DCP Adaptation

GLM-5.2 ships a built-in Multi-Token Prediction (MTP) layer. In sglang, --speculative-algorithm EAGLE reuses this MTP layer as the draft head; there is no separate draft model to load, and the checkpoint itself contains the NextN layer. The MTP draft maintains its own KV pool and its own CUDA graphs. The tension with DCP: the target KV pool is DCP-sharded (each rank holds $1/N_{dcp}$ of pages), but the MTP draft needs the full sequence to generate draft tokens. DCP for DSA models such as GLM-5.x uses a slot-interleaved page layout [13] that must be reconciled with the draft's full-sequence view. This creates a fundamental tension:

• Target model: runs under DCP, sees only $1/N_{dcp}$ of KV pages, uses all-gather + LSE merge for attention output.
• MTP draft: must see the full KV to predict next tokens. If it ran under DCP, the draft tokens would be inconsistent across DCP ranks (each rank only sees $1/4$ of the context).

Solution: the @dcp_disabled() context manager temporarily disables DCP for the MTP draft's forward pass. The draft uses a separate, non-sharded KV pool with draft_pool_token_multiplier = dcp_size (4×) to allocate enough capacity for the full sequence. After draft generation, the target model verifies draft tokens under DCP mode: each DCP rank independently verifies its $1/N_{dcp}$ shard of the draft tokens, then all-gather merges the acceptance results.

Memory cost of @dcp_disabled() . The draft KV pool is sized with draft_pool_token_multiplier = dcp_size: the target pool's max_total_num_tokens is the per-rank shard (one rank holds $1/N_{dcp}$ of the sequence), and the draft pool is allocated as max_total_num_tokens × dcp_size, which equals the full sequence, so every DCP rank can hold the whole context locally during draft generation. On GLM-5.2 (DSA latent = 576 B/token/layer), a 32K-token context needs ≈1.47 GB of KV; at DCP=4 the target pool holds ≈0.37 GB per rank and the draft pool holds the full ≈1.47 GB per rank. This is the dominant extra-memory term behind the "1.17M vs 2.0M tokens" gap in Table 1: not a separate draft-model's weights (there is none), but the draft KV pool + CUDA graphs. The tradeoff is deliberate: the draft runs only a handful of steps per verification cycle, so a memory-heavy, un-sharded draft pool is cheaper than trying to shard the draft's attention, which would require the draft to be DCP-aware and break its full-sequence view.

Verification correctness. Because the draft generates tokens with DCP disabled, every DCP rank runs the MTP head over the same full-sequence KV (replicated via the draft pool), so all ranks produce identical draft token sequences: there is no per-rank divergence to reconcile at generation time. Verification then happens under DCP: each rank computes the target-model logits for the same draft tokens using only its own $1/N_{dcp}$ KV shard, so each rank's per-step LSE is computed over a different KV subset. The per-step attention output is completed by the standard DCP all-gather + LSE merge (§5.2): AllGather concatenates each rank's $O_{\text{local}}$ by rank index in a fixed order, and log(Σ_d exp(LSE_d)) is a deterministic, commutative-associative reduction, so every rank arrives at the same merged LSE and the same full $O_{\text{full}}$, and therefore the same acceptance decision for each draft token. The acceptance length is then reduced across ranks (the verify step agrees on the common prefix of accepted drafts), and only the accepted tokens are committed to the DCP-sharded target KV pool. This keeps the target KV pool's DCP layout untouched (no draft-visible holes) while ensuring the accepted prefix is identical on every rank.

# eagle_worker_v2.py
@dcp_disabled()   # temporarily disable DCP for the MTP draft forward pass
function forward_draft(...):
    # draft (MTP head) sees full sequence (replicated draft KV pool),
    # not affected by DCP sharding -> identical draft tokens on all ranks
    ...

# verify: target model under DCP, all-gather + LSE merge per step
# -> identical logits on all ranks -> identical acceptance decisions
# accepted tokens are committed to the DCP-sharded target KV pool

# draft pool token allocation accounts for DCP multiplier
draft_pool_token_multiplier ← server_args.dcp_size   # 4×

Conclusion

We set out to serve concurrent long-context requests on B300 with a memory-heavy model stack, and ended up rethinking where sharding happens in the KV transfer path. The resulting design, page-level 2D resharding, treats the RDMA transfer itself as the sharding operation: Prefill CP ranks own distinct layers, Decode DCP ranks own distinct pages, and filtering page indices at the source produces the correct DCP layout at the destination with no post-processing. The correctness story is backed by a set of collective-level guarantees: rank-invariant gates, the size-broadcast protocol, consensus prefix lengths, and padded all-reduces that keep every rank participating in the same collectives.

The main limitation is that the guarantees are tuned for the layer-split + DCP combination we run; other sharding geometries (e.g., token-level sharding, interleaved CP) would need their own adaptation of the size-broadcast and consensus logic. A natural next step is to fold the 2D reshard metadata into a reusable transfer-plan library rather than per-connector logic, so the same page-filtering mechanism can be dropped into other disaggregated serving stacks (vLLM, TensorRT-LLM) that lack a CP + PD + DCP transfer path.

For Macaron-V1, this was the optimization that turned a launch that saturated on memory into one that could keep up. The agents V1 is built to serve accumulate long, high-cardinality histories (conversations, tool calls, environment feedback), and serving those at full fidelity is what keeps the model's behavior grounded in the real experience rather than a compressed proxy. Expanding KV capacity through clean sharding, instead of through truncation, is what let the deployment absorb the load instead of degrading under it.

References

[1] Roadmap: Decode Context Parallelism & Helix Parallelism (sgl-project et al, 2026)

[2] Implement DCP for DeepSeek-V2 (sgl-project et al, 2025)

[3] Support triton backend decode context parallel for Qwen3.5 (sgl-project et al, 2026)

[4] Consolidate decode-context-parallel (DCP) helpers (sgl-project et al, 2026)

[5] Roadmap: Context Parallelism (sgl-project et al, 2026)

[6] DSA cache layer split under Prefill CP (sgl-project et al, 2026)

[7] Roadmap: Prefill Context Parallel Refactor (sgl-project et al, 2026)

[8] Prefill Context Parallel with Zigzag Ring Attention + Split KV Transfer (sgl-project et al, 2026)

[9] Roadmap: Prefill and Decoding Disaggregation (sgl-project et al, 2025)

[10] Roadmap: Prefill-Decode Disaggregation (sgl-project et al, 2026)

[11] Support HiCache prefetching and PD-incremental transfer on decode side (sgl-project et al, 2026)

[12] Layer-pipelined KV transfer: overlap RDMA with GPU compute (sgl-project et al, 2026)

[13] Decode context parallelism (DCP) for DSA models (sgl-project et al, 2026)

[14] DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (Zhong et al, 2024)

[15] Splitwise: Efficient Generative LLM Inference Using Phase Splitting (Patel et al, 2024)

[16] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (Qin et al, 2024)

Author

Mind Lab

Core Contributors

Smith Li, Jingwei Cao, Rio Yang, Kieran Liu, Song Cao, Atlas Zeng, Nolan Ho, Pony Ma, Andrew Chen

Team

Asher Cai, Song Cao, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Jun Gao, Pyke Han, Nolan Ho, Mutian Hong, Charles Huang, Fancy Kong, Andrew Lei, Lucian Li, Ray Li, Fan Lin, Kieran Liu, Logan Liu, Neo Liu, Xiang Liu, Yuxin Lu, Pony Ma, Cole Qiao, Vince Qu, Vincent Wang, Bo Wu, Chengdong Xu, Rio Yang, Regis Ye, Yihang Zeng, Di Zhang, Jiahao Zheng, Adrian Zhou, Yuhua Zhou, Murphy Zhuang and Mindverse Team

Names are listed alphabetically within team.

Citation

Please cite this work using the BibTeX citation:

@misc{smithli2026dcp2dreshard,
  author = {Smith Li and Jingwei Cao and Rio Yang and Kieran Liu and Song Cao and Atlas Zeng and Nolan Ho and Pony Ma and Andrew Chen and {Mind Lab}},
  title = {Serving Macaron-V1 at Scale: Page-Level 2D KV Resharding},
  year = {2026},
  howpublished = {Mind Lab: A Lab for Experiential Intelligence},
  note = {https://macaron.im/mindlab/research/serving-macaron-v1-at-scale-page-level-2d-kv-resharding}
}