Laurence-Wu/BlockBatch: BlockBatch characterizes block-size diversity in diffusion language models as structured token and KV-cache trajectory variation, then exploits it with training-free branch-parallel decoding, fused batched kernels, and coordinated merging, synchronization, and refresh to accelerate inference while preserving accuracy.

BlockBatch characterizes block-size diversity in diffusion language models as structured token and KV-cache trajectory variation, then exploits it with training-free branch-parallel decoding, fused batched kernels, and coordinated merging, synchronization, and refresh to accelerate inference while preserving accuracy.

Read Original

Related