]> 2026-09-08T16:34:35+09:00 Lesson:nanoflow towards optimal large language model serving throughput f0633332 3000 ko GPU LLM serving with predictable operator profiles. Limits: May duplicate work and requires model/hardware-specific profiling and scheduling. NanoFlow forms nano-batches, duplicates selected operators, and automatically chooses count, size, order, and resource allocation for overlap. S3ResearchAgent medium Venue: OSDI. Year: 2025. Request batching alone leaves device resources idle because Transformer operations stress different resources at different times. Verification: official USENIX page and abstract; confidence=high. 2026-07-16T15:00:39Z 2461238.1254514 NanoFlow: Towards Optimal Large Language Model Serving Throughput. OSDI 2025. Intra-device scheduling is as important as request scheduling for high-throughput serving. NanoFlow: Towards Optimal Large Language Model Serving Throughput workloads=LLaMA-2-70B, Mixtral-8x7B, LLaMA-3-8B; baselines=state-of-the-art LLM serving systems and analytical optimum; metrics=serving throughput and percent of optimum; results=up to 1.91x; 50–72% of optimum How close can LLM serving get to hardware-optimal throughput by overlapping compute, memory, and network work inside each GPU? lab Co-schedule operations with complementary bottlenecks at a granularity below the request batch. Draft 2026-07-18T14:58:47Z 2461240.1241551 2026-07-18T14:58:47Z 2461240.1241551 nanoflow towards optimal large language model serving throughput f0633332 3000 ev_288b9e63b92346f9 2026-07-16T15:00:40Z 2461238.125463 S3ResearchAgent NanoFlow: Towards Optimal Large Language Model Serving Throughput. OSDI 2025. paper Bibliographic paper record. not_recorded nanoflow towards optimal large language model serving throughput f0633332# 068e7742e7bd5b67f6450dd299f89d8d 3000 verified-content-v1-0125 2026-07-16T18:45:26Z 2461238.2815509 S3ResearchAgent Kan Zhu et al., "NanoFlow: Towards Optimal Large Language Model Serving Throughput", OSDI 2025. paper Verification: official USENIX page and abstract; confidence=high. Canonical title: NanoFlow: Towards Optimal Large Language Model Serving Throughput Question: How close can LLM serving get to hardware-optimal throughput by overlapping compute, memory, and network work inside each GPU? Context: Request batching alone leaves device resources idle because Transformer operations stress different resources at different times. Method: NanoFlow forms nano-batches, duplicates selected operators, and automatically chooses count, size, order, and resource allocation for overlap. Evaluation: workloads=LLaMA-2-70B, Mixtral-8x7B, LLaMA-3-8B; baselines=state-of-the-art LLM serving systems and analytical optimum; metrics=serving throughput and percent of optimum; results=up to 1.91x; 50–72% of optimum Interpretation: Intra-device scheduling is as important as request scheduling for high-throughput serving. Reusable lesson: Co-schedule operations with complementary bottlenecks at a granularity below the request batch. Applicability: GPU LLM serving with predictable operator profiles. Limits: May duplicate work and requires model/hardware-specific profiling and scheduling. not_recorded nanoflow towards optimal large language model serving throughput f0633332# 4bdd920368bbc245d06b8461056ec4bf 3000 canonical-paper-v2-f0633332 2026-07-18T05:30:15Z 2461239.7293403 S3ResearchAgent Kan Zhu et al., "NanoFlow: Towards Optimal Large Language Model Serving Throughput", OSDI 2025. paper Verification: official USENIX page and abstract; confidence=medium. Canonical title: NanoFlow: Towards Optimal Large Language Model Serving Throughput Question: How close can LLM serving get to hardware-optimal throughput by overlapping compute, memory, and network work inside each GPU? Context: Request batching alone leaves device resources idle because Transformer operations stress different resources at different times. Method: NanoFlow forms nano-batches, duplicates selected operators, and automatically chooses count, size, order, and resource allocation for overlap. Evaluation: workloads=LLaMA-2-70B, Mixtral-8x7B, LLaMA-3-8B; baselines=state-of-the-art LLM serving systems and analytical optimum; metrics=serving throughput and percent of optimum; results=up to 1.91x; 50–72% of optimum Interpretation: Intra-device scheduling is as important as request scheduling for high-throughput serving. Reusable lesson: Co-schedule operations with complementary bottlenecks at a granularity below the request batch. Applicability: GPU LLM serving with predictable operator profiles. Limits: May duplicate work and requires model/hardware-specific profiling and scheduling. official_abstract nanoflow towards optimal large language model serving throughput f0633332# 07c5a0b593d598763dc93e7a6fb631ad 3000 2026-07-18T14:58:46Z 2461240.1241435 S3ResearchAgent verify_46e5dc31aaa17075f442 partial_source context 보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함. inconclusive 7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e 보존 파일 objects/sha256/7c/7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e ev_288b9e63b92346f9 e93a265aacf664fd215b91e162dfb30e33e30fdb9ebc17bd5d1a3454e10ec70d nanoflow towards optimal large language model serving throughput f0633332# 7e4dca9ae36b8a01492f754a09c972ea 3000 2026-07-18T14:58:47Z 2461240.1241551 S3ResearchAgent verify_d1bee1e23ddb012940f3 partial_source context 보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함. inconclusive 7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e 보존 파일 objects/sha256/7c/7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e verified-content-v1-0125 12c33c70a0f6aafb23a30debaa755285079327ea2dd597e22564fb81ab6c37d7 nanoflow towards optimal large language model serving throughput f0633332# aeff565e4e8b6c94366f845427e070f8 3000 2026-07-18T14:58:47Z 2461240.1241551 S3ResearchAgent verify_9204528c8cbd2e341e7b partial_source context 보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함. inconclusive 7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e 보존 파일 objects/sha256/7c/7ca4384d8b6109799afa2e1b170e20ae2dbfd84d92fd89861efdef92c997701e canonical-paper-v2-f0633332 a1c5383ecfa9f0c18484dac70c4431e0e552eee512e5605cd516f936e9386e0a nanoflow towards optimal large language model serving throughput f0633332# 5cafb953d9cc7ae3a2baa5f433cf757b