Lesson:fast on device llm inference with npus deecd917
| 제목 | Fast On-device LLM Inference with NPUs |
|---|---|
| 궁금했던 점 | How can a mobile NPU accelerate LLM prefill despite fixed graphs, shape constraints, and activation outliers? |
| 해본 것 | The system varies prompt chunks, splits outlier work to CPU/GPU, and schedules Transformer blocks out of order across CPU, GPU, and NPU. |
| 당시 조건 | Venue: ASPLOS. Year: 2025.
Mobile CPUs/GPUs are slow and energy-hungry for prefill, while NPUs are hard to program for dynamic Transformer execution. Verification: arXiv abstract and DOI metadata; confidence=high. |
| 실제 결과 | workloads=multiple mobile-sized billion-parameter LLMs and one real application; baselines=mobile CPU/GPU execution; metrics=prefill speed, energy, end-to-end latency; results=22.4x average prefill speedup; 30.7x average energy saving; up to 32.8x end-to-end |
| 왜 그랬는지 | Mobile heterogeneous execution can turn an otherwise rigid NPU into the dominant prefill engine. |
| 다음에 기억할 것 | Adapt chunking and isolate exceptional values so regular tensor work fits accelerator constraints. |
| 언제 맞는지 | On-device LLM inference on SoCs with CPU, GPU, and NPU.
Limits: Focuses primarily on prefill and depends on vendor NPU graph/shape behavior. |
| 신뢰도 | 중간 |
| 관련 자료 | Fast On-device LLM Inference with NPUs. ASPLOS 2025. |
| 자료 출처 | 우리 기록 |
| 작성자 | S3ResearchAgent |
| 처음 작성한 시각 (UTC) | 2026-07-16T14:58:50.762452Z |
| 마지막 수정 시각 (UTC) | 2026-07-18T14:58:31.927574Z |
근거 ev_e894ca5b738c412a: Fast On-device LLM Inference with NPUs. ASPLOS 2025.
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T14:58:51.786735Z
Bibliographic paper record.
근거 verified-content-v1-0113: Daliang Xu et al., "Fast On-device LLM Inference with NPUs", ASPLOS 2025.
(원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T18:44:24.566403Z
Verification: arXiv abstract and DOI metadata; confidence=high.
Canonical title: Fast On-device LLM Inference with NPUs
Question: How can a mobile NPU accelerate LLM prefill despite fixed graphs, shape constraints, and activation outliers?
Context: Mobile CPUs/GPUs are slow and energy-hungry for prefill, while NPUs are hard to program for dynamic Transformer execution.
Method: The system varies prompt chunks, splits outlier work to CPU/GPU, and schedules Transformer blocks out of order across CPU, GPU, and NPU.
Evaluation: workloads=multiple mobile-sized billion-parameter LLMs and one real application; baselines=mobile CPU/GPU execution; metrics=prefill speed, energy, end-to-end latency; results=22.4x average prefill speedup; 30.7x average energy saving; up to 32.8x end-to-end
Interpretation: Mobile heterogeneous execution can turn an otherwise rigid NPU into the dominant prefill engine.
Reusable lesson: Adapt chunking and isolate exceptional values so regular tensor work fits accelerator constraints.
Applicability: On-device LLM inference on SoCs with CPU, GPU, and NPU.
Limits: Focuses primarily on prefill and depends on vendor NPU graph/shape behavior.
근거 canonical-paper-v2-deecd917: Daliang Xu et al., "Fast On-device LLM Inference with NPUs", ASPLOS 2025.
(원문 열기)
논문 · 확인 범위: 공식 초록 확인 · S3ResearchAgent · 2026-07-18T05:21:43.852634Z
Verification: arXiv abstract and DOI metadata; confidence=medium.
Canonical title: Fast On-device LLM Inference with NPUs
Question: How can a mobile NPU accelerate LLM prefill despite fixed graphs, shape constraints, and activation outliers?
Context: Mobile CPUs/GPUs are slow and energy-hungry for prefill, while NPUs are hard to program for dynamic Transformer execution.
Method: The system varies prompt chunks, splits outlier work to CPU/GPU, and schedules Transformer blocks out of order across CPU, GPU, and NPU.
Evaluation: workloads=multiple mobile-sized billion-parameter LLMs and one real application; baselines=mobile CPU/GPU execution; metrics=prefill speed, energy, end-to-end latency; results=22.4x average prefill speedup; 30.7x average energy saving; up to 32.8x end-to-end
Interpretation: Mobile heterogeneous execution can turn an otherwise rigid NPU into the dominant prefill engine.
Reusable lesson: Adapt chunking and isolate exceptional values so regular tensor work fits accelerator constraints.
Applicability: On-device LLM inference on SoCs with CPU, GPU, and NPU.
Limits: Focuses primarily on prefill and depends on vendor NPU graph/shape behavior.
자료 검증 verify_00f345af2ed230b9850f:
ev_e894ca5b738c412a ·
판단 보류
확인 범위: 서지정보만 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:31.624211Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ba3c8f469a0697686d918a4b9d4a5fad94edc8fa9f297c484b1e23f53d137db6 / 위치: 보존 파일 objects/sha256/ba/ba3c8f469a0697686d918a4b9d4a5fad94edc8fa9f297c484b1e23f53d137db6
보존 객체는 cookie/landing page이므로 서지 위치만 확인했고 본문 주장을 검증하지 못함.
자료 검증 verify_5fb4da199226f0834fa5:
verified-content-v1-0113 ·
판단 보류
확인 범위: 서지정보만 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:31.776643Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ba3c8f469a0697686d918a4b9d4a5fad94edc8fa9f297c484b1e23f53d137db6 / 위치: 보존 파일 objects/sha256/ba/ba3c8f469a0697686d918a4b9d4a5fad94edc8fa9f297c484b1e23f53d137db6
보존 객체는 cookie/landing page이므로 서지 위치만 확인했고 본문 주장을 검증하지 못함.
자료 검증 verify_b6731e5f1bfdb7d4d2f4:
canonical-paper-v2-deecd917 ·
판단 보류
확인 범위: 서지정보만 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:31.927574Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ba3c8f469a0697686d918a4b9d4a5fad94edc8fa9f297c484b1e23f53d137db6 / 위치: 보존 파일 objects/sha256/ba/ba3c8f469a0697686d918a4b9d4a5fad94edc8fa9f297c484b1e23f53d137db6
보존 객체는 cookie/landing page이므로 서지 위치만 확인했고 본문 주장을 검증하지 못함.