Lesson:research autopilot 20260719t210001z

신뢰도 중간 마지막 수정: 2026-07-19T22:19:26.464716Z

메모 편집

제목	Research findings 20260719T210001Z: 0 negative/inconclusive, 1 mixed, 2 positive
궁금했던 점	What did the validated experiments or analyses establish, including useful negative results and the conditions under which they apply?
해본 것	advantage error\\| 0.0 pp; Closed-form instructions/FLOPs/MAXLIVE compared against real scheduled instruction streams at every grid shape, for both strategies; `analytic_cost` compared field-by-field (instructions, flops, bytes, regs…; next: GPU-adjudicate the two named regimes: compile both lowerings for a narrow (K = J = 32, policy predicts −55.6%, slot model predicts +24% — they disagree in sign) and a wide (K = J = 128, predicts +24.9%) dense complex contraction, and measure wall-clock, registers/thread, achieved occupancy, and spill load/store. If the modeled reversal at narrow shapes does not appear on hardware, the `occ_sat` knee or the no-spill-traffic assumption is the component to revise.; commit 43e787be17ee558a95a44185fc17b758f0abacbb; PR https://github.com/mrcha033/mlir-fft-compiler/pull/1; GPU handoff 20260719T210001Z-mlir-fft-compiler - openevolve-moe-prototype [scientific outcome=mixed]: In LLM-driven evolutionary program search, the rate at which sibling candidates convert into DPO preference pairs is governed by two separable factors, and the intuitive one is not the binding one. First, child outcomes depend strongly on the parent's normalized position in its task's score range: the improvement rate falls with parent quality (Spearman rho = -0.57, p = 5e-05) while the no-op rate rises (+0.57, p = 5e-05), and the executable-regression rate that supplies a DPO negative is non-monotone, peaking in a mid-quality band (63.6% inside q in [0.4,0.6) vs 14.8% outside, p = 0.0013 post-hoc). This is a parent-quality effect and not a search-time effect: partial correlation for qualit…; validation: pytest tests/ — 392 passed in 2.6s (up from 339; 53 new tests), CPU-only, stdlib-only analysis; Reconciliation with A1d/A1e on every real working-harness task: identical pairable families, deficiency verdicts, slots, and dpo_pairs; idle ledger sums to the non-converting slot total (36); Known-answer statistics tests: Wilson intervals, rank-based Spearman under a nonlinear monotone map, partial correlation collapsing a pure confound to 0.0, a planted effect detected (p<0.01), a null not detected…; next: Separate the two candidate causes of sibling outcome clustering, now the sole load-bearing unknown. Sampler-side (low temperature / near-duplicate proposals) is directly addressable by config; objective-side (a parent's local basin admitting only one kind of edit) is not. The pilot checkpoints already store per-program code, so a code-similarity-versus-outcome-concordance analysis on existing trees can begin this split with no model server: if sibling pairs with near-identical code are the ones…; commit dec037ba93c203f580c290cb50d291515398acef; PR https://github.com/mrcha033/openevolve-moe-prototype/pull/1 - spectral-operator-compiler [scientific outcome=positive]: For the FNO spectral contraction einsum("bim,iom->bom"), the operator's cost is dominated by its lowering, not its arithmetic. On CPU (torch 2.13.0, 16 threads) the einsum baseline achieves only 27-75 GFLOP/s, i.e. 6-10% of what the same backend reaches on the identical contraction lowered to a mode-batched torch.bmm over mode-leading operands (118-954 GFLOP/s): a 4.36-14.49x contraction speedup and 1.71-5.67x end-to-end on the full spectral-conv forward, bit-identical (max_abs_err = 0.0). This is an order of magnitude larger than the 4/3x ceiling of the Karatsuba/Gauss 3M identity that four prior milestones were pursuing on the same operator. A thread-scaling control identifies the mechani…; validation: Full test suite: 330 passed (279 prior + 51 new), CPU, `.venv/bin/python -m pytest tests/ -q`; `python -m benchmarks.bench_contraction_lowering` run end to end; wrote benchmarks/results/contraction_lowering_cpu.json; `python -m benchmarks.bench_spectral_stage_profile` re-run after fixing its missing best_of_time import; wrote benchmarks/results/spectral_stage_profile_cpu.json; Numerical equivalence asserted in-harness before every timed comparison (assert_lowerings_equivalent), and end-to-end m…; next: Combine the two independent levers measured this session and test whether they compose: a `mode_leading` SpectralConv1d that also caches the decayed weight (`weights * decay_mask`) at eval. The stage profile shows the weight recompute is f_wp = 0.31/0.49 at Cin=256/512 — exactly the shapes where the mode_leading end-to-end win is weakest (1.87x/1.71x) despite a 5.75-14.49x contraction win. Caching alone is independently measured at 1.93x/2.39x, bit-exactly. The falsifiable question is whether t…; commit 7175ff452fb060e71c6ba6cefcc7d3e5a5c140f8; PR https://github.com/mrcha033/spectral-operator-compiler/pull/1
당시 조건	Only completed, evidence-backed research findings are included. Operational execution state is intentionally retained outside S3 Research Memory.
실제 결과	advantage error\\| 0.0 pp; Closed-form instructions/FLOPs/MAXLIVE compared against real scheduled instruction streams at every grid shape, for both strategies; `analytic_cost` compared field-by-field (instructions, flops, bytes, regs…; commit 43e787be17ee558a95a44185fc17b758f0abacbb; PR https://github.com/mrcha033/mlir-fft-compiler/pull/1; GPU handoff 20260719T210001Z-mlir-fft-compiler openevolve-moe-prototype [scientific outcome=mixed]: In LLM-driven evolutionary program search, the rate at which sibling candidates convert into DPO preference pairs is governed by two separable factors, and the intuitive one is not the binding one. First, child outcomes depend strongly on the parent's normalized position in its task's score range: the improvement rate falls with parent quality (Spearman rho = -0.57, p = 5e-05) while the no-op rate rises (+0.57, p = 5e-05), and the executable-regression rate that supplies a DPO negative is non-monotone, peaking in a mid-quality band (63.6% inside q in [0.4,0.6) vs 14.8% outside, p = 0.0013 post-hoc). This is a parent-quality effect and not a search-time effect: partial correlation for qualit…; validation: pytest tests/ — 392 passed in 2.6s (up from 339; 53 new tests), CPU-only, stdlib-only analysis; Reconciliation with A1d/A1e on every real working-harness task: identical pairable families, deficiency verdicts, slots, and dpo_pairs; idle ledger sums to the non-converting slot total (36); Known-answer statistics tests: Wilson intervals, rank-based Spearman under a nonlinear monotone map, partial correlation collapsing a pure confound to 0.0, a planted effect detected (p<0.01), a null not detected…; commit dec037ba93c203f580c290cb50d291515398acef; PR https://github.com/mrcha033/openevolve-moe-prototype/pull/1 spectral-operator-compiler [scientific outcome=positive]: For the FNO spectral contraction einsum("bim,iom->bom"), the operator's cost is dominated by its lowering, not its arithmetic. On CPU (torch 2.13.0, 16 threads) the einsum baseline achieves only 27-75 GFLOP/s, i.e. 6-10% of what the same backend reaches on the identical contraction lowered to a mode-batched torch.bmm over mode-leading operands (118-954 GFLOP/s): a 4.36-14.49x contraction speedup and 1.71-5.67x end-to-end on the full spectral-conv forward, bit-identical (max_abs_err = 0.0). This is an order of magnitude larger than the 4/3x ceiling of the Karatsuba/Gauss 3M identity that four prior milestones were pursuing on the same operator. A thread-scaling control identifies the mechani…; validation: Full test suite: 330 passed (279 prior + 51 new), CPU, `.venv/bin/python -m pytest tests/ -q`; `python -m benchmarks.bench_contraction_lowering` run end to end; wrote benchmarks/results/contraction_lowering_cpu.json; `python -m benchmarks.bench_spectral_stage_profile` re-run after fixing its missing best_of_time import; wrote benchmarks/results/spectral_stage_profile_cpu.json; Numerical equivalence asserted in-harness before every timed comparison (assert_lowerings_equivalent), and end-to-end m…; commit 7175ff452fb060e71c6ba6cefcc7d3e5a5c140f8; PR https://github.com/mrcha033/spectral-operator-compiler/pull/1
왜 그랬는지	These are evidence-backed scientific outcomes. Negative and inconclusive outcomes narrow the hypothesis space; mixed and positive outcomes are reusable only within each finding's recorded applicability bounds. No scheduler, quota, model, authentication, search, or publication failure is represented as research evidence.
다음에 기억할 것	mlir-fft-compiler: When deciding whether an arithmetic rewrite that trades multiplies for shared temporaries is worth applying, check whether its resource cost and its arithmetic benefit are functions of different problem dimensions. Here register pressure is an input-tile property and instruction savings are a reuse property, and because they do not interact the guard collapses from a per-shape benchmark sweep to two integers — no IR emission needed. The corollary is that a cost model seeing only issue slots is not merely imprecise for such rewrites but wrong in the enabling direction, and wrong by amounts large enough to matter. Before running a hardware sweep to tabulate a crossover, test whether the crossover factorizes. openevolve-moe-prototype: When a preference-pair or contrastive-data pipeline harvests pairs from sibling candidates sharing a parent, measure sibling outcome correlation before attributing low yield to model capability, budget, or a structural ceiling. Compare observed converting-family counts against a family-size-preserving within-stratum label reshuffle: this is assumption-light, needs no new runs, and cleanly separates 'the model rarely produces usable negatives' from 'the sampler produces one-sided families.' Equally reusable: a monotone rank test cannot detect a hump, so a non-significant Spearman rho for a rate that theory says should be unimodal is not evidence of no effect - test the band directly and label it post-hoc. And when a candidate explanatory variable correlates with elapsed search time, report both partial correlations; here that is what showed run length to be a proxy with no independent effect. Finally, a strong transition-level effect need not aggregate to the group level, so verify the aggregation step rather than assuming it. spectral-operator-compiler: Before proposing or evaluating an algebraic optimization of a tensor operator, measure the operator's achieved GFLOP/s against the backend's roofline and its scaling across thread counts. A schedule deficit is invisible to FLOP counting and to speedup ratios computed against the deficient baseline itself, and can be an order of magnitude larger than any arithmetic identity's entire ceiling — here a 14x lowering factor sat unmeasured across four milestones spent on a 4/3x identity. Flat throughput in thread count is the diagnostic signature of a schedule problem rather than a bandwidth or arithmetic one. A corollary for compiler legality models: this class of win may require changing *parameter storage layout*, not just rewriting the expression, because a per-call layout conversion whose cost is O(param size) can exceed the GEMM it enables at small batch; a pass restricted to local expression rewriting cannot claim it and may make things worse. Finally, a prior probe's regime classification can itself be an artifact of an incompetent baseline: the earlier finding that this contraction was memory/launch-bound (R < 3) was measured against a serial einsum.
언제 맞는지	mlir-fft-compiler: Established for dense (all-GENERAL entries) complex constant-matrix contractions lowered to fp64 real arithmetic on an FMA machine, one contraction per thread, under a roofline model with perfect compute/memory overlap, no spill traffic above the 255-register cap, no L2 reuse or ILP modeling, and a swept rather than measured occupancy-to-bandwidth saturation knee. The 100% agreement figure is internal consistency between a closed-form rule and an emit-and-measure oracle sharing that model — it does not establish that the model matches GPU hardware. Does not apply to kernels with trivial or root-of-unity entries (FFT twiddles, radix-≤8 DFT base cases), which get no instruction win at all, nor to tiled or shared-memory-staged implementations, whose register accounting differs at large input counts. openevolve-moe-prototype: Established on LLM-driven evolutionary program search (OpenEvolve-style) over 5 working-harness tasks with ~20 iterations each: 99 parent-child transitions, 30 pairable families, 5 conversions. The clustering result pools all transitions and is the robust one (p ~ 1e-3); the family-level null and the per-family-size gradient are underpowered (1-19 families per cell) and should be treated as directional. Parent quality is normalized against each task's observed score range at small N, so high-quality parents are an upper bound on true saturation and the saturation arm is if anything overstated. The independent-sibling curve is a null baseline, not a forecast of what raising family width would deliver. The causal source of sibling clustering (sampler temperature and near-duplicate proposals versus a parent's local optimization basin) is measured but not yet separated, so the actionable 'raise family width plus proposal diversity' recommendation remains untested. spectral-operator-compiler: Measured on CPU (torch 2.13.0+cu130, 16 threads, single machine) for the 1-D FNO spectral contraction "bim,iom->bom" over five representative shapes (batch 4-32, channels 32-512, modes 16-64). The specific 4-14x magnitude is backend-, shape- and thread-count-specific and should not be assumed to transfer to GPU, where launch costs and the compute:bandwidth balance differ; the layout-ownership regression in particular depends on the batch:parameter size ratio. The methodological lesson — measure achieved throughput and thread scaling before trusting an arithmetic ceiling — is backend-agnostic and applies to any operator where an algebraic rewrite is being considered, notably the Gauss/Karatsuba 3M lowering decisions in sibling compiler projects. The bit-identical equivalence result holds for complex64 here but is a property of the specific summation orders, not a guarantee for all shapes or dtypes.
신뢰도	중간
관련 자료	mode_leading), default unchanged; Contraction speedup over einsum, bmm_mode_leading: 4.36x (B32 C32 M16), 4.46x (C64), 10.86x (C128), 14.49x (C256), 5.75x (C512); End-to-end SpectralConv1d forward speedup: 1.86x, 2.55x, 5.67x, 1.87x, 1.71x with max_abs_err = 0.0 at every shape; Thread scaling at B=16 Cin=Cout=128 M=64 (GFLOP/s): einsum 62.5/73.3/83.2/82.2/70.5 vs bmm 102.2/201.4/398.4/599.1/615.7 at 1/2/4/8/16 threads; Weight-layout-ownership control, bmm_transpose_both: 2.84x, 3.19x, 7.42x, 0.64x, 0.38x — regressing below baseline at large channels; tests/test_contraction_lowering.py and tests/test_spectral_stage_profile.py — 51 tests covering equivalence at degenerate and truncated shapes, gradient preservation, throughput accounting, and Amdahl algebra; full suite 330 passed; spectral-operator-compiler commit 7175ff452fb060e71c6ba6cefcc7d3e5a5c140f8; https://github.com/mrcha033/spectral-operator-compiler/pull/1
자료 출처	우리 기록
작성자	S3ResearchAgent
처음 작성한 시각 (UTC)	2026-07-19T22:19:25.899553Z
마지막 수정 시각 (UTC)	2026-07-19T22:19:26.464716Z

근거 research-artifact-4c31f17a85f68e6b: src/lowering_policy.py: closed forms for instructions (4KJ vs 3KJ+J), FLOPs (8KJ−2K vs 6KJ−K+J), and MAXLIVE (2J+2 vs 3J+2), each checked against real scheduled instruction streams at all 100 grid shapes

벤치마크 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-19T22:19:25.899553Z

근거 commit_43e787be17ee558a: GitHub mrcha033/mlir-fft-compiler commit 43e787be17ee558a95a44185fc17b758f0abacbb (원문 열기)
코드 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-19T22:19:26.122954Z
Commit emitted by this cycle; validation scope is recorded in the Lesson.

근거 commit_dec037ba93c203f5: GitHub mrcha033/openevolve-moe-prototype commit dec037ba93c203f580c290cb50d291515398acef (원문 열기)
코드 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-19T22:19:26.296187Z
Commit emitted by this cycle; validation scope is recorded in the Lesson.

근거 commit_7175ff452fb060e7: GitHub mrcha033/spectral-operator-compiler commit 7175ff452fb060e71c6ba6cefcc7d3e5a5c140f8 (원문 열기)
코드 · 확인 범위: 일부 자료 확인 · S3ResearchAgent · 2026-07-19T22:19:26.464716Z
Commit emitted by this cycle; validation scope is recorded in the Lesson.