Lesson:kvcache cache in the wild characterizing and optimizing kvcache cache at a large cloud provider 0b39b53c

신뢰도 높음 마지막 수정: 2026-07-18T15:00:27.485427Z

메모 편집

제목	KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider
궁금했던 점	What reuse patterns actually determine KV-cache value in production LLM traffic?
해본 것	Aliyun Tongyi의 to-C와 to-B production trace에서 request type, turn, reuse time, spatial locality, KV-cache lifespan과 capacity requirement를 분석합니다. Request category별 reuse-probability distribution, spatial locality, lifespan을 함께 사용한 workload-aware eviction policy를 vLLM에 구현해 LRU/LFU와 비교합니다.
당시 조건	Venue: USENIX ATC. Year: 2025. KV-cache policies are often designed from synthetic multi-turn assumptions rather than cloud traces. Verification: official USENIX paper PDF and arXiv abstract; confidence=high.
실제 결과	workloads=2024년 12월과 2025년 2월에 수집한 Aliyun Tongyi production trace 두 세트(to-C Trace A, to-B Trace B; 논문은 각 trace의 대표 하루를 주로 분석)와 vLLM replay; baselines=무한-capacity ideal, LRU, LFU; metrics=KV-block hit ratio, reuse skew/time/lifespan, required cache capacity, mean response time; results=ideal hit ratio는 Trace A 62%, Trace B 54%; 상위 10% KV block이 reuse의 77%를 만들었고 to-B에서는 single-turn request가 cache hit의 97%를 만들었다. to-B KV의 P99 lifespan은 97초였으며 GPU HBM의 2배 cache로 common GQA model의 ideal hit rate에 근접했다. Workload-aware policy는 LRU/LFU 대비 cache hit를 3.9% 높이고 mean response time을 최대 41.4% 개선했다.
왜 그랬는지	Production KV-cache reuse는 multi-turn 여부 하나로 설명되지 않습니다. To-B API workload에서는 반복 system prompt 때문에 single-turn 요청이 reuse를 지배할 수 있고, request category별 시간/공간 locality와 짧은 lifespan을 함께 모델링해야 cache size와 eviction policy를 정할 수 있습니다.
다음에 기억할 것	실제 request category별 reuse probability, spatial locality, lifespan을 측정한 뒤 cache size와 eviction priority를 정합니다. Multi-turn 비율을 KV reuse의 대리변수로 사용하지 않습니다.
언제 맞는지	Cross-request prefix/KV reuse가 있는 shared LLM serving cluster에 적용합니다. Limits: 한 cloud provider의 두 production 기간을 분석했고 논문은 각 trace의 대표 하루 결과를 주로 제시합니다. Reasoning workload와 prefix 이외의 KV reuse, global scheduling, cache fairness는 평가 범위 밖입니다.
신뢰도	높음
관련 자료	KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider. USENIX ATC 2025.
자료 출처	우리 기록
작성자	S3ResearchAgent
처음 작성한 시각 (UTC)	2026-07-16T14:58:52.700260Z
마지막 수정 시각 (UTC)	2026-07-18T15:00:27.485427Z

근거 ev_4a14e72ef65a4587: KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider. USENIX ATC 2025. (원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T14:58:55.709915Z
Bibliographic paper record.

근거 verified-content-v1-0114: Jiahao Wang et al., "KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider", USENIX ATC 2025. (원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T18:44:27.944894Z
Verification: official USENIX paper PDF and arXiv abstract; confidence=high. Canonical title: KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider Question: What reuse patterns actually determine KV-cache value in production LLM traffic? Context: KV-cache policies are often designed from synthetic multi-turn assumptions rather than cloud traces. Method: The paper analyzes large-provider KV-cache traces and derives category-aware, workload-aware caching and eviction guidance. Evaluation: workloads=large cloud-provider KV-cache traces; baselines=generic cache sizing and eviction assumptions; metrics=reuse incidence, predictability, and ideal capacity; results=qualitative production reuse findings; no unambiguous primary headline number recorded Interpretation: Production cache value depends on semantic/request categories, not conversation length alone. Reusable lesson: Measure real reuse classes before choosing cache size or eviction policy. Applicability: Shared LLM serving clusters with cross-request prefix/KV reuse. Limits: Trace findings may reflect one provider and period; exact intervention gains were not extracted from the primary PDF.

근거 canonical-paper-v2-0b39b53c: Jiahao Wang et al., "KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider", USENIX ATC 2025. (원문 열기)
논문 · 확인 범위: 원문 확인 · S3ResearchAgent · 2026-07-18T05:26:03.636936Z
Verification: official USENIX paper PDF and arXiv abstract; confidence=high. Canonical title: KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider Question: What reuse patterns actually determine KV-cache value in production LLM traffic? Context: KV-cache policies are often designed from synthetic multi-turn assumptions rather than cloud traces. Method: The paper analyzes large-provider KV-cache traces and derives category-aware, workload-aware caching and eviction guidance. Evaluation: workloads=large cloud-provider KV-cache traces; baselines=generic cache sizing and eviction assumptions; metrics=reuse incidence, predictability, and ideal capacity; results=qualitative production reuse findings; no unambiguous primary headline number recorded Interpretation: Production cache value depends on semantic/request categories, not conversation length alone. Reusable lesson: Measure real reuse classes before choosing cache size or eviction policy. Applicability: Shared LLM serving clusters with cross-request prefix/KV reuse. Limits: Trace findings may reflect one provider and period; exact intervention gains were not extracted from the primary PDF.

자료 검증 verify_ba32038385846edb8fd9: ev_4a14e72ef65a4587 · 판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T15:00:26.254503Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=4c4e923bcc94006b316be64c9def8efffdf771786484e99ba1b820a46dd41bbf / 위치: 보존 파일 objects/sha256/4c/4c4e923bcc94006b316be64c9def8efffdf771786484e99ba1b820a46dd41bbf
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.

자료 검증 verify_da6fac4d9ba9feb04b9d: verified-content-v1-0114 · 판단 보류
확인 범위: 일부 자료 확인 · 주장: context · S3ResearchAgent · 2026-07-18T15:00:26.415232Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=4c4e923bcc94006b316be64c9def8efffdf771786484e99ba1b820a46dd41bbf / 위치: 보존 파일 objects/sha256/4c/4c4e923bcc94006b316be64c9def8efffdf771786484e99ba1b820a46dd41bbf
보존 원문 객체를 확보했으나 이 일괄 검증에서는 claim-bearing 범위를 재판정하지 않아 결론을 보류함.

자료 검증 verify_cf72562440238c930b0b: canonical-paper-v2-0b39b53c · 지지함
확인 범위: 원문 확인 · 주장: observation,interpretation,reusable_lesson · S3ResearchAgent · 2026-07-18T15:00:26.566142Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=4c4e923bcc94006b316be64c9def8efffdf771786484e99ba1b820a46dd41bbf; independently adjudicated claim-bearing primary source / 위치: PDF §§3–5, pp. 4–16: characterization, workload-aware eviction, and evaluation.
observation=supported; interpretation=supported; reusable_lesson=supported

자료 검증 verify_75b8c8a9513f44d2bdca: canonical-paper-v2-0b39b53c · 반박함
확인 범위: 원문 확인 · 주장: attempt,observation,interpretation,reusable_lesson,applicability · S3ResearchAgent · 2026-07-18T15:00:26.718520Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=4c4e923bcc94006b316be64c9def8efffdf771786484e99ba1b820a46dd41bbf; adjudicated detailed correction / 위치: DETAILED-CD-011 PDF §§3-5, pp.4-16 pre-revision source locator.
Current claims are contradicted or exceed the preserved source; revision is staged at low confidence.

자료 검증 verify_bf7551db1d38f4386b04: canonical-paper-v2-0b39b53c · 지지함
확인 범위: 원문 확인 · 주장: observation,interpretation,reusable_lesson · S3ResearchAgent · 2026-07-18T15:00:27.171733Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=4c4e923bcc94006b316be64c9def8efffdf771786484e99ba1b820a46dd41bbf; adjudicated detailed correction / 위치: DETAILED-CD-011 PDF §§3-5, pp.4-16 post-revision claim locator.
Post-revision corrected claims are supported at the stated source scope.