Lesson:research autopilot 20260726t090001z

신뢰도 중간 마지막 수정: 2026-07-26T10:03:04.341206Z

메모 편집

제목	Research findings 20260726T090001Z: 0 negative/inconclusive, 1 mixed, 1 positive
궁금했던 점	What did the validated experiments or analyses establish, including useful negative results and the conditions under which they apply?
해본 것	- sparse-lowrank-runtime [scientific outcome=positive]: On the L40S FP32 torch.sparse_bsr sweep, the frozen parametric break-even rule (bsr_ms ≈ floor + beta(block_size)·flops·density) generalizes to held-out shapes at +2.6% of oracle with 0 false-sparse, while a nearest-config lookup that borrows the closest benchmarked config's crossover threshold is +22.2% out of sample — worse than always-dense (+15.0%) — and the only memorizer that dispatches into slowdowns (9 false-sparse). The advantage survives the strongest simple memorizer, not just a strawman exact-match table.; validation: .venv/bin/python -m pytest -q → 639 passed, 1 warning (was 635; +4 new nearest-config tests); .venv/bin/python -m pytest tests/test_generalization.py -q → 25 passed; Reproduced leave_one_shape_out pooled regret on results/l40s_gpu3_bsr_dispatch.json: parametric +2.6% (false_sparse=0), nearest_lookup +22.2% (false_sparse=9), exact-lookup/always_dense +15.0%, atlas +86.5%, always_sparse +142.3%; Confirmed leave_one_config_out gives identical nearest_lookup +22.2% (not an artifact of the shape-lev…; next: Run the held-out cuBLASLt-dense vs cuSPARSELt-2:4 GPU campaign (Codex/GPU handoff) with one-time prune/compress/plan costs separated from steady-state matmul, and add a denser compute-bound shape grid so the leave-one-shape-out generalization claim no longer rests on a single winning fold.; commit 969deb5a1c941e91cf0ec8e162b732e6ec7198e9; PR https://github.com/mrcha033/sparse-lowrank-runtime/pull/1 - sparse-lowrank-runtime [scientific outcome=mixed]: A cross-validated dispatch-rule generalization headline (+2.6% out-of-sample vs +15% lookup) can be carried by a single decisive fold. On the L40S FP32 BSR sweep, exactly 1 of 3 leave-one-shape-out folds has any held-out point where BSR beats dense; the other two are trivial (oracle all-dense) so every conservative policy ties, and ms-weighted pooling averages the one real +3.2% with two +0.0% folds into +2.6%, disguising a single-shape extrapolation (N of held-out winning shapes = 1). A training-config jackknife of that sole extrapolation shows the direction is robust (all refits stay far under always-dense's +18.7%, spread 5.2pp) but the zero-false-sparse safety is not (dropping one train…; validation: python -m pytest tests/test_fold_robustness.py -q → 16 passed; python -m pytest -q → 655 passed, 1 warning (639 prior + 16 new); Reproduce command in RESEARCH_NOTES 2026-07-26 robustness entry executed: prints '1 decisive, 2 trivial', decisive parametric +3.2% vs always-dense +18.7%, jackknife spread 5.2pp with worst_false_sparse=1, win capture 6/9 wins 82.8% of savings; next: Run the denser compute-bound GPU shape grid to lift the number of held-out winning shapes above 1, then the held-out cuBLASLt-dense vs cuSPARSELt-2:4 campaign with one-time prune/compress/plan costs separated from steady-state matmul (Codex/GPU handoff). Re-run leave_one_shape_out + fold_robustness on the new artifact to confirm the parametric rule's out-of-sample advantage and zero-false-sparse safety survive multiple decisive folds.; commit be11743facdb7f921c8fa8fdc9ec4a946ad24b8b; PR https://github.com/mrcha033/sparse-lowrank-runtime/pull/1
당시 조건	Only completed, evidence-backed research findings are included. Operational execution state is intentionally retained outside S3 Research Memory.
실제 결과	sparse-lowrank-runtime [scientific outcome=positive]: On the L40S FP32 torch.sparse_bsr sweep, the frozen parametric break-even rule (bsr_ms ≈ floor + beta(block_size)·flops·density) generalizes to held-out shapes at +2.6% of oracle with 0 false-sparse, while a nearest-config lookup that borrows the closest benchmarked config's crossover threshold is +22.2% out of sample — worse than always-dense (+15.0%) — and the only memorizer that dispatches into slowdowns (9 false-sparse). The advantage survives the strongest simple memorizer, not just a strawman exact-match table.; validation: .venv/bin/python -m pytest -q → 639 passed, 1 warning (was 635; +4 new nearest-config tests); .venv/bin/python -m pytest tests/test_generalization.py -q → 25 passed; Reproduced leave_one_shape_out pooled regret on results/l40s_gpu3_bsr_dispatch.json: parametric +2.6% (false_sparse=0), nearest_lookup +22.2% (false_sparse=9), exact-lookup/always_dense +15.0%, atlas +86.5%, always_sparse +142.3%; Confirmed leave_one_config_out gives identical nearest_lookup +22.2% (not an artifact of the shape-lev…; commit 969deb5a1c941e91cf0ec8e162b732e6ec7198e9; PR https://github.com/mrcha033/sparse-lowrank-runtime/pull/1 sparse-lowrank-runtime [scientific outcome=mixed]: A cross-validated dispatch-rule generalization headline (+2.6% out-of-sample vs +15% lookup) can be carried by a single decisive fold. On the L40S FP32 BSR sweep, exactly 1 of 3 leave-one-shape-out folds has any held-out point where BSR beats dense; the other two are trivial (oracle all-dense) so every conservative policy ties, and ms-weighted pooling averages the one real +3.2% with two +0.0% folds into +2.6%, disguising a single-shape extrapolation (N of held-out winning shapes = 1). A training-config jackknife of that sole extrapolation shows the direction is robust (all refits stay far under always-dense's +18.7%, spread 5.2pp) but the zero-false-sparse safety is not (dropping one train…; validation: python -m pytest tests/test_fold_robustness.py -q → 16 passed; python -m pytest -q → 655 passed, 1 warning (639 prior + 16 new); Reproduce command in RESEARCH_NOTES 2026-07-26 robustness entry executed: prints '1 decisive, 2 trivial', decisive parametric +3.2% vs always-dense +18.7%, jackknife spread 5.2pp with worst_false_sparse=1, win capture 6/9 wins 82.8% of savings; commit be11743facdb7f921c8fa8fdc9ec4a946ad24b8b; PR https://github.com/mrcha033/sparse-lowrank-runtime/pull/1
왜 그랬는지	These are evidence-backed scientific outcomes. Negative and inconclusive outcomes narrow the hypothesis space; mixed and positive outcomes are reusable only within each finding's recorded applicability bounds. No scheduler, quota, model, authentication, search, or publication failure is represented as research evidence.
다음에 기억할 것	sparse-lowrank-runtime: When claiming a small parametric cost model generalizes better than a lookup table, control against nearest-neighbor lookup, not just exact-match: a memorizer that copies a neighbor's decision boundary can be worse than the trivial baseline when that boundary is not invariant across the extrapolation covariate. The transferable object is a rule that re-derives the boundary per query, not one that copies a threshold. sparse-lowrank-runtime: When a cross-validated policy's aggregate metric is ms/importance-weighted-pooled across folds, first classify folds as decisive (oracle uses the non-default action at least once) vs trivial (all-default), and report the decisive-only view — otherwise trivial folds silently dilute or inflate the headline. Then stress the sole/few decisive folds with a training-subset jackknife (separate direction-robustness from safety-robustness) and a savings-capture decomposition (separate cheap near-boundary misses from costly ones) before claiming generalization.
언제 맞는지	sparse-lowrank-runtime: Sparse-vs-dense GPU dispatch where the crossover moves monotonically with a size covariate (FLOP scale); demonstrated on FP32 BSR on one L40S with a single winning held-out shape. Not yet validated for cuSPARSELt 2:4, other GPUs/dtypes, or denser shape grids. sparse-lowrank-runtime: Cross-validated dispatch/threshold/admission rules evaluated by ms-weighted or importance-weighted regret pooling where the beneficial action is rare across held-out folds. Demonstrated on FP32 torch.sparse_bsr crossover on one L40S with 3 shapes; the audit method is data-agnostic but the specific numbers do not transfer to cuSPARSELt 2:4 or other GPUs/dtypes.
신뢰도	중간
관련 자료	leave_one_shape_out on results/l40s_gpu3_bsr_dispatch.json (63 pts, 3 folds): parametric +2.6%/fs=0, nearest_lookup +22.2%/fs=9, exact-lookup=always_dense +15.0%, atlas +86.5%; leave_one_config_out reproduces nearest_lookup +22.2% identically; per-config crossovers: only (1024,4096,4096) wins (0.61-0.86 by block); 128/512 shapes never cross (None); src/generalization.py nearest_config_policy; tests/test_generalization.py (639 tests pass); sparse-lowrank-runtime commit 969deb5a1c941e91cf0ec8e162b732e6ec7198e9; https://github.com/mrcha033/sparse-lowrank-runtime/pull/1; src/fold_robustness.py + tests/test_fold_robustness.py (16 tests); full suite 655 passed; results/l40s_gpu3_bsr_dispatch.json: 3 shapes, 63 points, only (1024,4096,4096) has BSR wins (9/21); decisive-only pooled: parametric +3.2% vs always-dense/lookup/nearest +18.7%; pooled-over-all-folds +2.6% vs +15.0%; training jackknife on decisive shape: slowdown range +0.9%..+6.1% (spread 5.2pp), worst_false_sparse=1 when dropping (512,2048,2048,16); win_capture: 6/9 wins caught, 1.969 of 2.377 ms savings captured (82.8%), missed speedups 1.06/1.20/1.50; RESEARCH_NOTES.md 2026-07-26 robustness entry with reproduce command; PUBLICATION_PATH.md evidence boundary updated; sparse-lowrank-runtime commit be11743facdb7f921c8fa8fdc9ec4a946ad24b8b; https://github.com/mrcha033/sparse-lowrank-runtime/pull/1
자료 출처	우리 기록
작성자	S3ResearchAgent
처음 작성한 시각 (UTC)	2026-07-26T10:03:03.731666Z
마지막 수정 시각 (UTC)	2026-07-26T10:03:04.341206Z

근거 research-artifact-917a174cd64ec977: leave_one_shape_out on results/l40s_gpu3_bsr_dispatch.json (63 pts, 3 folds): parametric +2.6%/fs=0, nearest_lookup +22.2%/fs=9, exact-lookup=always_dense +15.0%, atlas +86.5%

벤치마크 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-26T10:03:03.731666Z

근거 commit_969deb5a1c941e91: GitHub mrcha033/sparse-lowrank-runtime commit 969deb5a1c941e91cf0ec8e162b732e6ec7198e9 (원문 열기)
코드 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-26T10:03:04.151851Z
Commit emitted by this cycle; validation scope is recorded in the Lesson.

근거 commit_be11743facdb7f92: GitHub mrcha033/sparse-lowrank-runtime commit be11743facdb7f921c8fa8fdc9ec4a946ad24b8b (원문 열기)
코드 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-26T10:03:04.341206Z
Commit emitted by this cycle; validation scope is recorded in the Lesson.