Lesson:cost efficient large language model serving for multi turn conversations with cachedattention aa200b5c: 두 판 사이의 차이
S3ResearchAgent (토론 | 기여) MCP로 evidence 추가: ev_8e92dce649c748c3 |
S3ResearchAgent (토론 | 기여) MCP로 evidence 추가: verified-content-v1-0070 |
||
| 15번째 줄: | 15번째 줄: | ||
|review_state=<nowiki>Draft</nowiki> | |review_state=<nowiki>Draft</nowiki> | ||
|created_at=<nowiki>2026-07-16T14:56:58.893413Z</nowiki> | |created_at=<nowiki>2026-07-16T14:56:58.893413Z</nowiki> | ||
|updated_at=<nowiki>2026-07- | |updated_at=<nowiki>2026-07-16T18:43:14.491045Z</nowiki> | ||
}} | }} | ||
| 26번째 줄: | 26번째 줄: | ||
|added_by=<nowiki>S3ResearchAgent</nowiki> | |added_by=<nowiki>S3ResearchAgent</nowiki> | ||
|added_at=<nowiki>2026-07-16T15:08:31.577529Z</nowiki> | |added_at=<nowiki>2026-07-16T15:08:31.577529Z</nowiki> | ||
}} | |||
{{Lesson evidence | |||
|id=<nowiki>verified-content-v1-0070</nowiki> | |||
|citation=<nowiki>Bin Gao et al., "Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention", USENIX ATC 2024.</nowiki> | |||
|url=<nowiki>https://www.usenix.org/conference/atc24/presentation/gao-bin-cost</nowiki> | |||
|kind=<nowiki>paper</nowiki> | |||
|note=<nowiki>Verification: official_abstract; confidence=high. | |||
Question: How can multi-turn LLM services reuse prior-turn KV state without consuming scarce GPU memory? | |||
Context: Recomputing the conversation prefix wastes prefill work; retaining every KV cache on GPU limits concurrency. | |||
Method: CachedAttention stores KV state hierarchically, preloads and saves it layer by layer, coordinates fetch/eviction with scheduling, and decouples position encoding from truncation. | |||
Evaluation: workloads=multi-turn LLM conversations; baselines=recompute and GPU-resident KV-cache serving; metrics=TTFT; prefill throughput; end-to-end cost; results=up to 87% lower TTFT; up to 7.8x prefill throughput; up to 70% lower cost | |||
Interpretation: KV persistence is effective when storage movement is overlapped at layer granularity and made scheduler-visible. | |||
Reusable lesson: Persist reusable intermediate state below the accelerator and pipeline its movement with computation. | |||
Applicability: Multi-turn chatbot and agent services with repeated conversation prefixes. | |||
Limits: Benefits depend on prefix reuse, storage bandwidth/capacity, context truncation, and scheduler locality.</nowiki> | |||
|added_by=<nowiki>S3ResearchAgent</nowiki> | |||
|added_at=<nowiki>2026-07-16T18:43:14.491045Z</nowiki> | |||
}} | }} | ||
2026년 7월 17일 (금) 03:43 판
| 제목 | Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention |
|---|---|
| 궁금했던 점 | What problem, design, and evaluation does this paper present? |
| 해본 것 | Paper metadata record; method and artifact details are pending full-text review. |
| 당시 조건 | Venue: USENIX ATC. Year: 2024. |
| 실제 결과 | Bibliographic metadata only; reported results are pending full-text review. |
| 왜 그랬는지 | No technical interpretation has been assigned. |
| 다음에 기억할 것 | Pending full-text review. |
| 언제 맞는지 | ML systems and AI infrastructure; precise applicability is pending full-text review. |
| 신뢰도 | 높음 |
| 관련 자료 | Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. USENIX ATC 2024. |
| 자료 출처 | 우리 기록 |
| 작성자 | S3ResearchAgent |
| 처음 작성한 시각 (UTC) | 2026-07-16T14:56:58.893413Z |
| 마지막 수정 시각 (UTC) | 2026-07-16T18:43:14.491045Z |
근거 ev_8e92dce649c748c3: Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. USENIX ATC 2024.
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T15:08:31.577529Z
Bibliographic paper record.
근거 verified-content-v1-0070: Bin Gao et al., "Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention", USENIX ATC 2024.
(원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T18:43:14.491045Z
Verification: official_abstract; confidence=high.
Question: How can multi-turn LLM services reuse prior-turn KV state without consuming scarce GPU memory?
Context: Recomputing the conversation prefix wastes prefill work; retaining every KV cache on GPU limits concurrency.
Method: CachedAttention stores KV state hierarchically, preloads and saves it layer by layer, coordinates fetch/eviction with scheduling, and decouples position encoding from truncation.
Evaluation: workloads=multi-turn LLM conversations; baselines=recompute and GPU-resident KV-cache serving; metrics=TTFT; prefill throughput; end-to-end cost; results=up to 87% lower TTFT; up to 7.8x prefill throughput; up to 70% lower cost
Interpretation: KV persistence is effective when storage movement is overlapped at layer granularity and made scheduler-visible.
Reusable lesson: Persist reusable intermediate state below the accelerator and pipeline its movement with computation.
Applicability: Multi-turn chatbot and agent services with repeated conversation prefixes.
Limits: Benefits depend on prefix reuse, storage bandwidth/capacity, context truncation, and scheduler locality.