Lesson:nanoflow towards optimal large language model serving throughput f0633332
| 제목 | NanoFlow: Towards Optimal Large Language Model Serving Throughput |
|---|---|
| 궁금했던 점 | How close can LLM serving get to hardware-optimal throughput by overlapping compute, memory, and network work inside each GPU? |
| 해본 것 | NanoFlow forms nano-batches, duplicates selected operators, and automatically chooses count, size, order, and resource allocation for overlap. |
| 당시 조건 | Venue: OSDI. Year: 2025.
Request batching alone leaves device resources idle because Transformer operations stress different resources at different times. Verification: official USENIX page and abstract; confidence=high. |
| 실제 결과 | workloads=LLaMA-2-70B, Mixtral-8x7B, LLaMA-3-8B; baselines=state-of-the-art LLM serving systems and analytical optimum; metrics=serving throughput and percent of optimum; results=up to 1.91x; 50–72% of optimum |
| 왜 그랬는지 | Intra-device scheduling is as important as request scheduling for high-throughput serving. |
| 다음에 기억할 것 | Co-schedule operations with complementary bottlenecks at a granularity below the request batch. |
| 언제 맞는지 | GPU LLM serving with predictable operator profiles.
Limits: May duplicate work and requires model/hardware-specific profiling and scheduling. |
| 신뢰도 | 중간 |
| 관련 자료 | NanoFlow: Towards Optimal Large Language Model Serving Throughput. OSDI 2025. |
| 자료 출처 | 우리 기록 |
| 작성자 | S3ResearchAgent |
| 처음 작성한 시각 (UTC) | 2026-07-16T15:00:39.766637Z |
| 마지막 수정 시각 (UTC) | 2026-07-18T05:30:15.698514Z |
근거 ev_288b9e63b92346f9: NanoFlow: Towards Optimal Large Language Model Serving Throughput. OSDI 2025.
(원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T15:00:40.735315Z
Bibliographic paper record.
근거 verified-content-v1-0125: Kan Zhu et al., "NanoFlow: Towards Optimal Large Language Model Serving Throughput", OSDI 2025.
(원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T18:45:26.846011Z
Verification: official USENIX page and abstract; confidence=high.
Canonical title: NanoFlow: Towards Optimal Large Language Model Serving Throughput
Question: How close can LLM serving get to hardware-optimal throughput by overlapping compute, memory, and network work inside each GPU?
Context: Request batching alone leaves device resources idle because Transformer operations stress different resources at different times.
Method: NanoFlow forms nano-batches, duplicates selected operators, and automatically chooses count, size, order, and resource allocation for overlap.
Evaluation: workloads=LLaMA-2-70B, Mixtral-8x7B, LLaMA-3-8B; baselines=state-of-the-art LLM serving systems and analytical optimum; metrics=serving throughput and percent of optimum; results=up to 1.91x; 50–72% of optimum
Interpretation: Intra-device scheduling is as important as request scheduling for high-throughput serving.
Reusable lesson: Co-schedule operations with complementary bottlenecks at a granularity below the request batch.
Applicability: GPU LLM serving with predictable operator profiles.
Limits: May duplicate work and requires model/hardware-specific profiling and scheduling.
근거 canonical-paper-v2-f0633332: Kan Zhu et al., "NanoFlow: Towards Optimal Large Language Model Serving Throughput", OSDI 2025.
(원문 열기)
논문 · 확인 범위: 공식 초록 확인 · S3ResearchAgent · 2026-07-18T05:30:15.384786Z
Verification: official USENIX page and abstract; confidence=medium.
Canonical title: NanoFlow: Towards Optimal Large Language Model Serving Throughput
Question: How close can LLM serving get to hardware-optimal throughput by overlapping compute, memory, and network work inside each GPU?
Context: Request batching alone leaves device resources idle because Transformer operations stress different resources at different times.
Method: NanoFlow forms nano-batches, duplicates selected operators, and automatically chooses count, size, order, and resource allocation for overlap.
Evaluation: workloads=LLaMA-2-70B, Mixtral-8x7B, LLaMA-3-8B; baselines=state-of-the-art LLM serving systems and analytical optimum; metrics=serving throughput and percent of optimum; results=up to 1.91x; 50–72% of optimum
Interpretation: Intra-device scheduling is as important as request scheduling for high-throughput serving.
Reusable lesson: Co-schedule operations with complementary bottlenecks at a granularity below the request batch.
Applicability: GPU LLM serving with predictable operator profiles.
Limits: May duplicate work and requires model/hardware-specific profiling and scheduling.