Lesson:taming throughput latency tradeoff in llm inference with sarathi serve cf395778
| 제목 | Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve |
|---|---|
| 궁금했던 점 | How can LLM serving raise throughput without prefill requests causing decode stalls and tail-latency violations? |
| 해본 것 | Sarathi-Serve uses chunked prefills and stall-free scheduling to build uniform hybrid batches and reduce pipeline bubbles. |
| 당시 조건 | Venue: OSDI. Year: 2024.
Prefill and decode have different compute profiles; naive batching creates head-of-line stalls and pipeline bubbles. Verification: full_text; confidence=high. |
| 실제 결과 | workloads=Mistral-7B on 1xA100; Yi-34B on 2xA100; LLaMA2-70B on 8xA40; Falcon-180B on 8xA100; arXiv summarization trace; baselines=vLLM; metrics=serving capacity; TTFT; TBT; pipeline bubbles; results=2.6x Mistral-7B capacity; 3.7x Yi-34B capacity; 5.6x Falcon-180B capacity |
| 왜 그랬는지 | Constraining prefill work per iteration makes latency predictable while retaining batch efficiency. |
| 다음에 기억할 것 | Split long, bursty stages into schedulable chunks and co-batch them with steady-state work. |
| 언제 맞는지 | Autoregressive LLM services, especially pipeline-parallel deployments.
Limits: Speedups depend on model, GPU topology, arrival trace, chunk size, and chosen TTFT/TBT SLOs. |
| 신뢰도 | 높음 |
| 관련 자료 | Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. |
| 자료 출처 | 우리 기록 |
| 작성자 | S3ResearchAgent |
| 처음 작성한 시각 (UTC) | 2026-07-16T14:56:22.183652Z |
| 마지막 수정 시각 (UTC) | 2026-07-18T15:00:33.327192Z |
근거 ev_faf03ea2af104ae9: Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024.
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T14:56:23.099019Z
Bibliographic paper record.
근거 verified-content-v1-0056: Amey Agrawal et al., "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve", OSDI 2024.
(원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T18:41:15.579172Z
Verification: full_text; confidence=high.
Question: How can LLM serving raise throughput without prefill requests causing decode stalls and tail-latency violations?
Context: Prefill and decode have different compute profiles; naive batching creates head-of-line stalls and pipeline bubbles.
Method: Sarathi-Serve uses chunked prefills and stall-free scheduling to build uniform hybrid batches and reduce pipeline bubbles.
Evaluation: workloads=Mistral-7B on 1xA100; Yi-34B on 2xA100; LLaMA2-70B on 8xA40; Falcon-180B on 8xA100; arXiv summarization trace; baselines=vLLM; metrics=serving capacity; TTFT; TBT; pipeline bubbles; results=2.6x Mistral-7B capacity; 3.7x Yi-34B capacity; 5.6x Falcon-180B capacity
Interpretation: Constraining prefill work per iteration makes latency predictable while retaining batch efficiency.
Reusable lesson: Split long, bursty stages into schedulable chunks and co-batch them with steady-state work.
Applicability: Autoregressive LLM services, especially pipeline-parallel deployments.
Limits: Speedups depend on model, GPU topology, arrival trace, chunk size, and chosen TTFT/TBT SLOs.
근거 canonical-paper-v2-cf395778: Amey Agrawal et al., "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve", OSDI 2024.
(원문 열기)
논문 · 확인 범위: 원문 확인 · S3ResearchAgent · 2026-07-18T05:39:05.989635Z
Verification: full_text; confidence=high.
Question: How can LLM serving raise throughput without prefill requests causing decode stalls and tail-latency violations?
Context: Prefill and decode have different compute profiles; naive batching creates head-of-line stalls and pipeline bubbles.
Method: Sarathi-Serve uses chunked prefills and stall-free scheduling to build uniform hybrid batches and reduce pipeline bubbles.
Evaluation: workloads=Mistral-7B on 1xA100; Yi-34B on 2xA100; LLaMA2-70B on 8xA40; Falcon-180B on 8xA100; arXiv summarization trace; baselines=vLLM; metrics=serving capacity; TTFT; TBT; pipeline bubbles; results=2.6x Mistral-7B capacity; 3.7x Yi-34B capacity; 5.6x Falcon-180B capacity
Interpretation: Constraining prefill work per iteration makes latency predictable while retaining batch efficiency.
Reusable lesson: Split long, bursty stages into schedulable chunks and co-batch them with steady-state work.
Applicability: Autoregressive LLM services, especially pipeline-parallel deployments.
Limits: Speedups depend on model, GPU topology, arrival trace, chunk size, and chosen TTFT/TBT SLOs.
자료 검증 verify_e2b54acb85e747777100:
ev_faf03ea2af104ae9 ·
판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T15:00:33.031766Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ccd533c5292ad14358c344c5c3fae1683f7b03183a7275ef9fbf90ca8fbc18ce / 위치: 보존 파일 objects/sha256/cc/ccd533c5292ad14358c344c5c3fae1683f7b03183a7275ef9fbf90ca8fbc18ce
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.
자료 검증 verify_882565139750f7adbe0c:
verified-content-v1-0056 ·
판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T15:00:33.181822Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ccd533c5292ad14358c344c5c3fae1683f7b03183a7275ef9fbf90ca8fbc18ce / 위치: 보존 파일 objects/sha256/cc/ccd533c5292ad14358c344c5c3fae1683f7b03183a7275ef9fbf90ca8fbc18ce
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.
자료 검증 verify_577c1ff141167a0f07bd:
canonical-paper-v2-cf395778 ·
지지함
확인 범위: 공식 초록 확인 · 주장: observation,interpretation,reusable_lesson · S3ResearchAgent · 2026-07-18T15:00:33.327192Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ccd533c5292ad14358c344c5c3fae1683f7b03183a7275ef9fbf90ca8fbc18ce; independently adjudicated claim-bearing primary source / 위치: Saved USENIX presentation page, official abstract.
observation=supported; interpretation=supported; reusable_lesson=supported