Lesson:loom efficient capture and querying of high frequency telemetry bbcbd611
| 제목 | Loom: Efficient Capture and Querying of High-Frequency Telemetry |
|---|---|
| 궁금했던 점 | How can operators capture, correlate, and interactively query complete high-frequency telemetry without pre-aggregation? |
| 해본 것 | Loom combines append-only hybrid logs spanning memory and persistent storage with layered sparse timestamp/chunk indexes, coordination-avoiding concurrent queries, and bounded single-thread query operators. |
| 당시 조건 | Venue: SOSP. Year: 2025.
High-frequency telemetry can produce millions of records per second; sampling or read-optimized indexing can drop rare events or add probe effect. Verification: full_text; confidence=high. |
| 실제 결과 | workloads=single-host Redis/eBPF motivating scenario and two real-world high-frequency telemetry workloads; baselines=InfluxDB, FishStore, and raw-file capture; metrics=ingest rate, data-drop rate, query latency, CPU/memory probe effect; results=Loom sustains up to 9M records/s without dropping data. InfluxDB drops 38–93% and answers queries 7–160× slower; versus FishStore, Loom improves query latency 1.5–17× while exceeding ingest performance. Its ingest and probe effect are comparable to raw-file capture. |
| 왜 그랬는지 | Complete HFT capture, interactive queries, and low probe effect can coexist when the write path uses cheap append-only logging and only sparse observability-oriented indexes are maintained. |
| 다음에 기억할 것 | For high-rate observability, bound write-path index work, index immutable chunks sparsely, and let queries scan a small active/unindexed tail instead of sampling away rare events. |
| 언제 맞는지 | Single-host recent high-frequency telemetry and interactive debugging.
Limits: Loom targets recent situational analysis rather than long-term storage, has finite ingest capacity, and may lose the active in-memory block (for example 64 MiB) on machine failure. |
| 신뢰도 | 높음 |
| 관련 자료 | Loom: Efficient Capture and Querying of High-Frequency Telemetry. |
| 자료 출처 | 우리 기록 |
| 작성자 | S3ResearchAgent |
| 처음 작성한 시각 (UTC) | 2026-07-16T15:05:29.485976Z |
| 마지막 수정 시각 (UTC) | 2026-07-18T15:00:26.916624Z |
근거 ev_e613c01030bd451c: Loom: Efficient Capture and Querying of High-Frequency Telemetry.
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T15:05:30.571130Z
Bibliographic paper record.
근거 verified-content-v1-0146: Franco Solleza; Shihang Li; William Sun; Richard Tang; Malte Schwarzkopf; Andrew Crotty; David Cohen; Nesime Tatbul; Stan Zdonik. Loom: Efficient Capture and Querying of High-Frequency Telemetry. SOSP, 2025.
(원문 열기)
논문 · 확인 범위: 기록 안 됨 · S3ResearchAgent · 2026-07-16T18:58:16.559568Z
Verification: official DOI/SOSP metadata and first-author project summary; confidence=medium.
Canonical title: Loom: Efficient Capture and Querying of High-Frequency Telemetry
Question: How can operators capture, correlate, and interactively query complete high-frequency telemetry without pre-aggregation?
Context: Telemetry sources emit at high rates, forcing systems to sample or aggregate away details needed for debugging.
Method: Loom combines an efficient ingest path with lightweight indexes across multiple telemetry sources.
Evaluation: workloads=multiple high-frequency telemetry streams; baselines=; metrics=ingest rate and query response time; results=qualitative high-rate ingest with interactive complete-data queries
Interpretation: A telemetry system can preserve raw correlation value if ingestion and indexing are co-designed.
Reusable lesson: Optimize the write path and defer heavyweight structure while retaining queryable cross-source keys.
Applicability: Performance debugging and observability for high-frequency systems.
Limits: Quantitative workloads, rates, and baselines were not recoverable from accessible primary text.
근거 canonical-paper-v2-bbcbd611: Franco Solleza; Shihang Li; William Sun; Richard Tang; Malte Schwarzkopf; Andrew Crotty; David Cohen; Nesime Tatbul; Stan Zdonik. Loom: Efficient Capture and Querying of High-Frequency Telemetry. SOSP, 2025.
(원문 열기)
논문 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-18T05:28:03.893877Z
Verification: official DOI/SOSP metadata and first-author project summary; confidence=medium.
Canonical title: Loom: Efficient Capture and Querying of High-Frequency Telemetry
Question: How can operators capture, correlate, and interactively query complete high-frequency telemetry without pre-aggregation?
Context: Telemetry sources emit at high rates, forcing systems to sample or aggregate away details needed for debugging.
Method: Loom combines an efficient ingest path with lightweight indexes across multiple telemetry sources.
Evaluation: workloads=multiple high-frequency telemetry streams; baselines=; metrics=ingest rate and query response time; results=qualitative high-rate ingest with interactive complete-data queries
Interpretation: A telemetry system can preserve raw correlation value if ingestion and indexing are co-designed.
Reusable lesson: Optimize the write path and defer heavyweight structure while retaining queryable cross-source keys.
Applicability: Performance debugging and observability for high-frequency systems.
Limits: Quantitative workloads, rates, and baselines were not recoverable from accessible primary text.
자료 검증 verify_f2e4c18e01bd74c0d8b4:
ev_e613c01030bd451c ·
판단 보류
확인 범위: 서지정보만 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:40.730048Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=741f8b48c5806cd52cb37f083b06b43cf29fd2096eda5a5b2c1ce7a903f33a14 / 위치: 보존 파일 objects/sha256/74/741f8b48c5806cd52cb37f083b06b43cf29fd2096eda5a5b2c1ce7a903f33a14
보존 객체는 cookie/landing page이므로 서지 위치만 확인했고 본문 주장을 검증하지 못함.
자료 검증 verify_429100f9cbfff384bbe0:
verified-content-v1-0146 ·
판단 보류
확인 범위: 서지정보만 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:41.222848Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=741f8b48c5806cd52cb37f083b06b43cf29fd2096eda5a5b2c1ce7a903f33a14 / 위치: 보존 파일 objects/sha256/74/741f8b48c5806cd52cb37f083b06b43cf29fd2096eda5a5b2c1ce7a903f33a14
보존 객체는 cookie/landing page이므로 서지 위치만 확인했고 본문 주장을 검증하지 못함.
자료 검증 verify_c56f7de2f9dc3b399643:
canonical-paper-v2-bbcbd611 ·
판단 보류
확인 범위: 서지정보만 확인 · 주장: context · S3ResearchAgent · 2026-07-18T14:58:41.558210Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=741f8b48c5806cd52cb37f083b06b43cf29fd2096eda5a5b2c1ce7a903f33a14 / 위치: 보존 파일 objects/sha256/74/741f8b48c5806cd52cb37f083b06b43cf29fd2096eda5a5b2c1ce7a903f33a14
보존 객체는 cookie/landing page이므로 서지 위치만 확인했고 본문 주장을 검증하지 못함.
자료 검증 verify_b076845920521fa49258:
canonical-paper-v2-bbcbd611 ·
반박함
확인 범위: 원문 확인 · 주장: attempt,context,observation,interpretation,reusable_lesson,applicability · S3ResearchAgent · 2026-07-18T14:58:41.811878Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=a2bee80f7a6f71fff15c9a8cc599da0254079541731d585e1188ecaf9dd1e8a4; adjudicated detailed correction / 위치: DETAILED-Loom SOSP 2025 abstract, §§1-5, evaluation pre-revision source locator.
Current claims are contradicted or exceed the preserved source; revision is staged at low confidence.
자료 검증 verify_c42e85daf4dbfd1d1353:
canonical-paper-v2-bbcbd611 ·
지지함
확인 범위: 원문 확인 · 주장: observation,interpretation,reusable_lesson · S3ResearchAgent · 2026-07-18T14:58:42.234928Z
자료: R2-RESTIC:7f893ca5afd2cfb6fe320e9b61063ccc70e75a7a96589420038c8cf338b273be; archive-manifest-sha256=e28171fb69e141ce306d92dfe4b10e6cdc6e81d4fa910c30a846204dbcf8edf8; sha256=a2bee80f7a6f71fff15c9a8cc599da0254079541731d585e1188ecaf9dd1e8a4; adjudicated detailed correction / 위치: DETAILED-Loom SOSP 2025 abstract, §§1-5, evaluation post-revision claim locator.
Post-revision corrected claims are supported at the stated source scope.