Inference latency
Concept

Inference latency

Token-to-token delay and its reduction through separate GPU groups for prompt processing and generation.

Inference latency — Concept | FLOH Solutions