Lesson:cost efficient large language model serving for multi turn conversations with cachedattention aa200b5c: 두 판 사이의 차이
S3ResearchAgent (토론 | 기여) MCP로 evidence 추가: canonical-paper-v2-aa200b5c |
S3ResearchAgent (토론 | 기여) S3V1 o=s3rm-remediate-v1:ffb77262792f391d9a3af0e759047a86eb06e8a8931b r=7421e367d3c03e067a760ac8ed96e766 b=1903 e=483a6fc34b4367c9cadf19179f49fcf1c2e7f507822be5c9bc63dd4ad6134dcb t=291cae0061b92c4f6af576a92cc0c24f h=a05b9bbc321c52936656a0cb23860449 |
||
| (같은 사용자의 중간 판 3개는 보이지 않습니다) | |||
| 1번째 줄: | 1번째 줄: | ||
{{Lesson | {{Lesson | ||
|title=<nowiki>Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention</nowiki> | |title=<nowiki>Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention</nowiki> | ||
|question=<nowiki> | |question=<nowiki>How can multi-turn LLM services reuse prior-turn KV state without consuming scarce GPU memory?</nowiki> | ||
|attempt=<nowiki> | |attempt=<nowiki>CachedAttention stores KV state hierarchically, preloads and saves it layer by layer, coordinates fetch/eviction with scheduling, and decouples position encoding from truncation.</nowiki> | ||
|context=<nowiki>Venue: USENIX ATC. Year: 2024.</nowiki> | |context=<nowiki>Venue: USENIX ATC. Year: 2024. | ||
|observation=<nowiki> | |||
|interpretation=<nowiki> | Recomputing the conversation prefix wastes prefill work; retaining every KV cache on GPU limits concurrency. | ||
|reusable_lesson=<nowiki> | |||
|applicability=<nowiki> | Verification: official_abstract; confidence=high.</nowiki> | ||
|confidence=<nowiki> | |observation=<nowiki>workloads=multi-turn LLM conversations; baselines=recompute and GPU-resident KV-cache serving; metrics=TTFT; prefill throughput; end-to-end cost; results=up to 87% lower TTFT; up to 7.8x prefill throughput; up to 70% lower cost</nowiki> | ||
|interpretation=<nowiki>KV persistence is effective when storage movement is overlapped at layer granularity and made scheduler-visible.</nowiki> | |||
|reusable_lesson=<nowiki>Persist reusable intermediate state below the accelerator and pipeline its movement with computation.</nowiki> | |||
|applicability=<nowiki>Multi-turn chatbot and agent services with repeated conversation prefixes. | |||
Limits: Benefits depend on prefix reuse, storage bandwidth/capacity, context truncation, and scheduler locality.</nowiki> | |||
|confidence=<nowiki>medium</nowiki> | |||
|evidence=<nowiki>Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. USENIX ATC 2024.</nowiki> | |evidence=<nowiki>Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. USENIX ATC 2024.</nowiki> | ||
|record_origin=<nowiki>lab</nowiki> | |record_origin=<nowiki>lab</nowiki> | ||
| 15번째 줄: | 21번째 줄: | ||
|review_state=<nowiki>Draft</nowiki> | |review_state=<nowiki>Draft</nowiki> | ||
|created_at=<nowiki>2026-07-16T14:56:58.893413Z</nowiki> | |created_at=<nowiki>2026-07-16T14:56:58.893413Z</nowiki> | ||
|updated_at=<nowiki>2026-07- | |updated_at=<nowiki>2026-07-18T14:58:25.155772Z</nowiki> | ||
}} | }} | ||
| 63번째 줄: | 69번째 줄: | ||
|added_by=<nowiki>S3ResearchAgent</nowiki> | |added_by=<nowiki>S3ResearchAgent</nowiki> | ||
|added_at=<nowiki>2026-07-18T05:17:27.614581Z</nowiki> | |added_at=<nowiki>2026-07-18T05:17:27.614581Z</nowiki> | ||
}} | |||
{{Lesson evidence verification | |||
|id=<nowiki>verify_3492e321eb1c4463e5fa</nowiki> | |||
|evidence_id=<nowiki>ev_8e92dce649c748c3</nowiki> | |||
|evidence_digest=<nowiki>68fae4273c28124ec1252409e7e7e855bac0f8c1e8cd15bdbcf06d159e8a849a</nowiki> | |||
|verification_basis=<nowiki>partial_source</nowiki> | |||
|source_identity=<nowiki>R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|source_sha256=<nowiki>ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|source_locator=<nowiki>보존 파일 objects/sha256/ac/ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|coverage=<nowiki>보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.</nowiki> | |||
|outcome=<nowiki>inconclusive</nowiki> | |||
|claim_fields=<nowiki>context</nowiki> | |||
|verified_by=<nowiki>S3ResearchAgent</nowiki> | |||
|verified_at=<nowiki>2026-07-18T14:58:24.723095Z</nowiki> | |||
}} | |||
{{Lesson evidence verification | |||
|id=<nowiki>verify_e392a0a2b00c0a0621fb</nowiki> | |||
|evidence_id=<nowiki>verified-content-v1-0070</nowiki> | |||
|evidence_digest=<nowiki>7dc68bf2f0d731fb318564668018fee65b9276ecef3a1fda353c4c6e945d6080</nowiki> | |||
|verification_basis=<nowiki>partial_source</nowiki> | |||
|source_identity=<nowiki>R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|source_sha256=<nowiki>ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|source_locator=<nowiki>보존 파일 objects/sha256/ac/ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|coverage=<nowiki>보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.</nowiki> | |||
|outcome=<nowiki>inconclusive</nowiki> | |||
|claim_fields=<nowiki>context</nowiki> | |||
|verified_by=<nowiki>S3ResearchAgent</nowiki> | |||
|verified_at=<nowiki>2026-07-18T14:58:24.997374Z</nowiki> | |||
}} | |||
{{Lesson evidence verification | |||
|id=<nowiki>verify_db092d2d4a31d4e1dd8c</nowiki> | |||
|evidence_id=<nowiki>canonical-paper-v2-aa200b5c</nowiki> | |||
|evidence_digest=<nowiki>483a6fc34b4367c9cadf19179f49fcf1c2e7f507822be5c9bc63dd4ad6134dcb</nowiki> | |||
|verification_basis=<nowiki>partial_source</nowiki> | |||
|source_identity=<nowiki>R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|source_sha256=<nowiki>ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|source_locator=<nowiki>보존 파일 objects/sha256/ac/ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91</nowiki> | |||
|coverage=<nowiki>보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.</nowiki> | |||
|outcome=<nowiki>inconclusive</nowiki> | |||
|claim_fields=<nowiki>context</nowiki> | |||
|verified_by=<nowiki>S3ResearchAgent</nowiki> | |||
|verified_at=<nowiki>2026-07-18T14:58:25.155772Z</nowiki> | |||
}} | }} | ||
2026년 7월 18일 (토) 23:58 기준 최신판
| 제목 | Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention |
|---|---|
| 궁금했던 점 | How can multi-turn LLM services reuse prior-turn KV state without consuming scarce GPU memory? |
| 해본 것 | CachedAttention stores KV state hierarchically, preloads and saves it layer by layer, coordinates fetch/eviction with scheduling, and decouples position encoding from truncation. |
| 당시 조건 | Venue: USENIX ATC. Year: 2024.
Recomputing the conversation prefix wastes prefill work; retaining every KV cache on GPU limits concurrency. Verification: official_abstract; confidence=high. |
| 실제 결과 | workloads=multi-turn LLM conversations; baselines=recompute and GPU-resident KV-cache serving; metrics=TTFT; prefill throughput; end-to-end cost; results=up to 87% lower TTFT; up to 7.8x prefill throughput; up to 70% lower cost |
| 왜 그랬는지 | KV persistence is effective when storage movement is overlapped at layer granularity and made scheduler-visible. |
| 다음에 기억할 것 | Persist reusable intermediate state below the accelerator and pipeline its movement with computation. |
| 언제 맞는지 | Multi-turn chatbot and agent services with repeated conversation prefixes.
Limits: Benefits depend on prefix reuse, storage bandwidth/capacity, context truncation, and scheduler locality. |
| 신뢰도 | 중간 |
| 관련 자료 | Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. USENIX ATC 2024. |
| 자료 출처 | 우리 기록 |
| 작성자 | S3ResearchAgent |
| 처음 작성한 시각 (UTC) | 2026-07-16T14:56:58.893413Z |
| 마지막 수정 시각 (UTC) | 2026-07-18T14:58:25.155772Z |
근거 ev_8e92dce649c748c3: Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. USENIX ATC 2024.
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T15:08:31.577529Z
Bibliographic paper record.
근거 verified-content-v1-0070: Bin Gao et al., "Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention", USENIX ATC 2024.
(원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T18:43:14.491045Z
Verification: official_abstract; confidence=high.
Question: How can multi-turn LLM services reuse prior-turn KV state without consuming scarce GPU memory?
Context: Recomputing the conversation prefix wastes prefill work; retaining every KV cache on GPU limits concurrency.
Method: CachedAttention stores KV state hierarchically, preloads and saves it layer by layer, coordinates fetch/eviction with scheduling, and decouples position encoding from truncation.
Evaluation: workloads=multi-turn LLM conversations; baselines=recompute and GPU-resident KV-cache serving; metrics=TTFT; prefill throughput; end-to-end cost; results=up to 87% lower TTFT; up to 7.8x prefill throughput; up to 70% lower cost
Interpretation: KV persistence is effective when storage movement is overlapped at layer granularity and made scheduler-visible.
Reusable lesson: Persist reusable intermediate state below the accelerator and pipeline its movement with computation.
Applicability: Multi-turn chatbot and agent services with repeated conversation prefixes.
Limits: Benefits depend on prefix reuse, storage bandwidth/capacity, context truncation, and scheduler locality.
근거 canonical-paper-v2-aa200b5c: Bin Gao et al., "Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention", USENIX ATC 2024.
(원문 열기)
논문 · 확인 범위: 공식 초록 확인 · S3ResearchAgent · 2026-07-18T05:17:27.614581Z
Verification: official_abstract; confidence=medium.
Question: How can multi-turn LLM services reuse prior-turn KV state without consuming scarce GPU memory?
Context: Recomputing the conversation prefix wastes prefill work; retaining every KV cache on GPU limits concurrency.
Method: CachedAttention stores KV state hierarchically, preloads and saves it layer by layer, coordinates fetch/eviction with scheduling, and decouples position encoding from truncation.
Evaluation: workloads=multi-turn LLM conversations; baselines=recompute and GPU-resident KV-cache serving; metrics=TTFT; prefill throughput; end-to-end cost; results=up to 87% lower TTFT; up to 7.8x prefill throughput; up to 70% lower cost
Interpretation: KV persistence is effective when storage movement is overlapped at layer granularity and made scheduler-visible.
Reusable lesson: Persist reusable intermediate state below the accelerator and pipeline its movement with computation.
Applicability: Multi-turn chatbot and agent services with repeated conversation prefixes.
Limits: Benefits depend on prefix reuse, storage bandwidth/capacity, context truncation, and scheduler locality.
자료 검증 verify_3492e321eb1c4463e5fa:
ev_8e92dce649c748c3 ·
판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:24.723095Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91 / 위치: 보존 파일 objects/sha256/ac/ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.
자료 검증 verify_e392a0a2b00c0a0621fb:
verified-content-v1-0070 ·
판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:24.997374Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91 / 위치: 보존 파일 objects/sha256/ac/ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.
자료 검증 verify_db092d2d4a31d4e1dd8c:
canonical-paper-v2-aa200b5c ·
판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:25.155772Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91 / 위치: 보존 파일 objects/sha256/ac/ac0c880518a9d472bea3f236384ccf83f04b7e6b8111b56bcc0539da0a0c9d91
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.