Optimizing Token Throughput with Speculative Decoding & FP8 Quantization

Serving frontier 70B+ parameter models at low latency is one of the toughest challenges in production AI infrastructure. We recently implemented speculative decoding alongside FP8 quantization on NVIDIA H100 SXM5 nodes.

Speculative Decoding Architecture

By pairing a small draft model (e.g. Llama-3.2-1B) with the target model (Llama-3.3-70B-Instruct), the draft generates 4-6 candidate tokens per forward pass. The target model then verifies all candidates in a single batched step.

• Speedup: 2.3x throughput increase on typical reasoning tasks.
• VRAM reduction: FP8 KV cache slashed memory footprint by 48% with <0.2% perplexity divergence on MMLU.
• Framework: Implemented natively using vLLM v0.6+ and TensorRT-LLM.

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.3-70B-Instruct",
    speculative_model="meta-llama/Llama-3.2-1B-Instruct",
    num_speculative_tokens=5,
    quantization="fp8",
    max_model_len=8192,
)

Have you deployed speculative decoding in production, and how did you tune num_speculative_tokens?