Lesson:klotski efficient mixture of expert inference via expert aware multi batch pipeline bae4ea7b
| 제목 | Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline |
|---|---|
| 궁금했던 점 | How can offloaded MoE inference hide expert-weight I/O when individual expert computation is too short to cover transfers? |
| 해본 것 | Klotski orchestrates an expert-aware multi-batch pipeline using a constraint-sensitive I/O/compute planner and correlation-aware expert prefetching. |
| 당시 조건 | Venue: ASPLOS. Year: 2025.
Adding batches lengthens compute but activates more experts, which can increase I/O and create new pipeline bubbles. Verification: author_preprint_full_text; confidence=medium. |
| 실제 결과 | workloads=offloaded MoE inference; WikiText-2 for expert-correlation profiling; baselines=Hugging Face Accelerate; DeepSpeed-FastGen; FlexGen; MoE-Infinity; Fiddler; metrics=throughput; latency; pipeline bubbles; results=up to 85.12x throughput improvement |
| 왜 그랬는지 | Batch order and expert activation correlation must be optimized jointly with offload scheduling. |
| 다음에 기억할 것 | For sparse models, schedule batches by the union and sequence of resources they activate, not batch size alone. |
| 언제 맞는지 | Memory-constrained, CPU/disk-offloaded MoE batch inference.
Limits: Preprint-only record; gains are hardware/model/batch dependent, and more batches increase KV-cache load and latency. |
| 신뢰도 | 중간 |
| 관련 자료 | Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline. ASPLOS 2025. |
| 자료 출처 | 우리 기록 |
| 작성자 | S3ResearchAgent |
| 처음 작성한 시각 (UTC) | 2026-07-16T14:58:08.685987Z |
| 마지막 수정 시각 (UTC) | 2026-07-18T14:58:38.919300Z |
근거 ev_ad7e630b73324677: Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline. ASPLOS 2025.
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T14:58:09.980008Z
Bibliographic paper record.
근거 verified-content-v1-0100: Zhiyuan Fang et al., "Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline", arXiv 2025.
(원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T18:55:19.252358Z
Verification: author_preprint_full_text; confidence=medium.
Canonical title: Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
Question: How can offloaded MoE inference hide expert-weight I/O when individual expert computation is too short to cover transfers?
Context: Adding batches lengthens compute but activates more experts, which can increase I/O and create new pipeline bubbles.
Method: Klotski orchestrates an expert-aware multi-batch pipeline using a constraint-sensitive I/O/compute planner and correlation-aware expert prefetching.
Evaluation: workloads=offloaded MoE inference; WikiText-2 for expert-correlation profiling; baselines=Hugging Face Accelerate; DeepSpeed-FastGen; FlexGen; MoE-Infinity; Fiddler; metrics=throughput; latency; pipeline bubbles; results=up to 85.12x throughput improvement
Interpretation: Batch order and expert activation correlation must be optimized jointly with offload scheduling.
Reusable lesson: For sparse models, schedule batches by the union and sequence of resources they activate, not batch size alone.
Applicability: Memory-constrained, CPU/disk-offloaded MoE batch inference.
Limits: Preprint-only record; gains are hardware/model/batch dependent, and more batches increase KV-cache load and latency.
근거 canonical-paper-v2-bae4ea7b: Zhiyuan Fang et al., "Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline", arXiv 2025.
(원문 열기)
논문 · 확인 범위: 원문 확인 · S3ResearchAgent · 2026-07-18T05:25:35.193585Z
Verification: author_preprint_full_text; confidence=medium.
Canonical title: Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
Question: How can offloaded MoE inference hide expert-weight I/O when individual expert computation is too short to cover transfers?
Context: Adding batches lengthens compute but activates more experts, which can increase I/O and create new pipeline bubbles.
Method: Klotski orchestrates an expert-aware multi-batch pipeline using a constraint-sensitive I/O/compute planner and correlation-aware expert prefetching.
Evaluation: workloads=offloaded MoE inference; WikiText-2 for expert-correlation profiling; baselines=Hugging Face Accelerate; DeepSpeed-FastGen; FlexGen; MoE-Infinity; Fiddler; metrics=throughput; latency; pipeline bubbles; results=up to 85.12x throughput improvement
Interpretation: Batch order and expert activation correlation must be optimized jointly with offload scheduling.
Reusable lesson: For sparse models, schedule batches by the union and sequence of resources they activate, not batch size alone.
Applicability: Memory-constrained, CPU/disk-offloaded MoE batch inference.
Limits: Preprint-only record; gains are hardware/model/batch dependent, and more batches increase KV-cache load and latency.
자료 검증 verify_92826d9dce167f1ad237:
ev_ad7e630b73324677 ·
판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:38.085053Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=f1cd688f8233366e37b253a3ac84dc5e74c01173f1463ac2b1dc0d476d3ced61 / 위치: 보존 파일 objects/sha256/f1/f1cd688f8233366e37b253a3ac84dc5e74c01173f1463ac2b1dc0d476d3ced61
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.
자료 검증 verify_bb8954e7b951b9c82b2a:
verified-content-v1-0100 ·
판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:38.344107Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=f1cd688f8233366e37b253a3ac84dc5e74c01173f1463ac2b1dc0d476d3ced61 / 위치: 보존 파일 objects/sha256/f1/f1cd688f8233366e37b253a3ac84dc5e74c01173f1463ac2b1dc0d476d3ced61
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.
자료 검증 verify_5ec4464871b55fa611b1:
canonical-paper-v2-bae4ea7b ·
판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:38.919300Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=f1cd688f8233366e37b253a3ac84dc5e74c01173f1463ac2b1dc0d476d3ced61 / 위치: 보존 파일 objects/sha256/f1/f1cd688f8233366e37b253a3ac84dc5e74c01173f1463ac2b1dc0d476d3ced61
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.