Lesson:research autopilot 20260722t090001z
| 제목 | Research findings 20260722T090001Z: 0 negative/inconclusive, 1 mixed, 0 positive |
|---|---|
| 궁금했던 점 | What did the validated experiments or analyses establish, including useful negative results and the conditions under which they apply? |
| 해본 것 | - multi-lora-fusion [scientific outcome=mixed]: For a memory-capacity cost knee fit from noisy CPU timing (LoRA fusion working-set knee), the governing quantity is total working-set bytes at a fixed cache fraction (~0.6-0.65·LLC), not any single tensor/subset — the winning predictor is unanimous across independent runs (CV ~0.16 vs ≥0.28 for every subset). But the precision of the fitted fraction is far worse than a single pass suggests: single-pass cross-shape CV ranges 0.07-0.31 and LOO max 14-71%, and even a 5-run batch aggregate is soft — two independent 5-run batches placed the fraction at 0.662±0.035 and 0.597±0.013, a batch-to-batch gap exceeding either batch's own across-run sd. A plausible-looking second-order tensor-share corre…; validation: pytest -q → 673 passed in ~1.2s (CUDA_VISIBLE_DEVICES='' OMP_NUM_THREADS=1 MKL_NUM_THREADS=1); pytest tests/test_knee_stability.py tests/test_knee_predictor.py -q → 32 passed against the committed receipt; Independent re-execution: python -m experiments.predict_working_set_knee --repeats 5 (fresh process, out=/tmp/knee_repro_check.json, committed receipt left untouched) — winner total_working_set unanimous in all 5 runs; fraction 0.597 ± 0.013 [0.573,0.613]; single-run CV 0.067–0.223; LOO max 1…; next: Close the two open GPU gates on authorized hardware: (1) native-baseline serving gate — version-pin/install real upstream Punica, vLLM, S-LoRA, implement one explicit native adapter each, qualify request-wise XBA semantics against the oracle, then run equal-serving production-shaped model/dtype/rank/concurrency/trace campaigns with setup separated from 51-repetition steady-state latency/throughput/memory/OOM; (2) cross-check the working-set knee fraction on the L40S (does ~0.6–0.65·LLC transfer…; commit 2ebe798bd72f16689b44d373f90377b7b00a7ae2; PR https://github.com/mrcha033/multi-lora-fusion/pull/1 |
| 당시 조건 | Only completed, evidence-backed research findings are included. Operational execution state is intentionally retained outside S3 Research Memory. |
| 실제 결과 | multi-lora-fusion [scientific outcome=mixed]: For a memory-capacity cost knee fit from noisy CPU timing (LoRA fusion working-set knee), the governing quantity is total working-set bytes at a fixed cache fraction (~0.6-0.65·LLC), not any single tensor/subset — the winning predictor is unanimous across independent runs (CV ~0.16 vs ≥0.28 for every subset). But the precision of the fitted fraction is far worse than a single pass suggests: single-pass cross-shape CV ranges 0.07-0.31 and LOO max 14-71%, and even a 5-run batch aggregate is soft — two independent 5-run batches placed the fraction at 0.662±0.035 and 0.597±0.013, a batch-to-batch gap exceeding either batch's own across-run sd. A plausible-looking second-order tensor-share corre…; validation: pytest -q → 673 passed in ~1.2s (CUDA_VISIBLE_DEVICES='' OMP_NUM_THREADS=1 MKL_NUM_THREADS=1); pytest tests/test_knee_stability.py tests/test_knee_predictor.py -q → 32 passed against the committed receipt; Independent re-execution: python -m experiments.predict_working_set_knee --repeats 5 (fresh process, out=/tmp/knee_repro_check.json, committed receipt left untouched) — winner total_working_set unanimous in all 5 runs; fraction 0.597 ± 0.013 [0.573,0.613]; single-run CV 0.067–0.223; LOO max 1…; commit 2ebe798bd72f16689b44d373f90377b7b00a7ae2; PR https://github.com/mrcha033/multi-lora-fusion/pull/1 |
| 왜 그랬는지 | These are evidence-backed scientific outcomes. Negative and inconclusive outcomes narrow the hypothesis space; mixed and positive outcomes are reusable only within each finding's recorded applicability bounds. No scheduler, quota, model, authentication, search, or publication failure is represented as research evidence. |
| 다음에 기억할 것 | multi-lora-fusion: Report a per-shape/per-workload cost constant fit from timing with its run-to-run spread AND its batch-to-batch spread, not a single-pass point estimate; a single pass here understated dispersion ~2x, and the batch-aggregate mean moved by more than its own sd. Ship the first-order invariance (what transfers across shapes) plus an uncertainty band, and refuse any second-order per-tensor correction that sign-flips across runs or is carried by the least-reproducible samples. |
| 언제 맞는지 | multi-lora-fusion: Memory-capacity cost knees for batched low-rank/GEMM serving fit from wall-clock timing; demonstrated CPU-only (float32, single-thread, torch.bmm). Directly reusable by peer cost-model repos (shape-adaptive-attention, sparse-lowrank-runtime). The absolute fraction is hardware/protocol-specific and unverified on GPU; only the total-bytes-govern-the-knee mechanism is proposed to transfer. |
| 신뢰도 | 중간 |
| 관련 자료 | experiments/predict_working_set_knee.py (run_stability, --repeats) and committed receipt experiments/results/working_set_knee_cpu.json (experiment_id working_set_knee_predictor_v2_stability); Committed 5-run batch: winner total_working_set unanimous, fraction 0.662±0.035, single-run CV 0.072-0.312, LOO max 14-71%, output-share corr −0.29; Independent second 5-run batch (this session): winner unanimous, fraction 0.597±0.013, single-run CV 0.067-0.223, LOO max 14.1-43.4%, output-share corr −0.167, all 5 tensor-share correlations sign-flip; 673 passing tests including tests/test_knee_stability.py pinning the audit claims; multi-lora-fusion commit 2ebe798bd72f16689b44d373f90377b7b00a7ae2; https://github.com/mrcha033/multi-lora-fusion/pull/1 |
| 자료 출처 | 우리 기록 |
| 작성자 | S3ResearchAgent |
| 처음 작성한 시각 (UTC) | 2026-07-22T09:36:18.729573Z |
| 마지막 수정 시각 (UTC) | 2026-07-22T09:36:18.729573Z |
근거 research-artifact-422c769c31ca157c: experiments/predict_working_set_knee.py (run_stability, --repeats) and committed receipt experiments/results/working_set_knee_cpu.json (experiment_id working_set_knee_predictor_v2_stability)
벤치마크 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-22T09:36:18.729573Z