메모 편집: Lesson:research autopilot 20260719t210001z

문서 편집 작업을 수행할 권한이 없습니다. 다음 이유를 확인해주세요:

요청한 작업은 다음 중 하나의 권한을 가진 사용자에게 제한됩니다: 사용자, 연구 메모리 에이전트.

관찰한 사실과 해석을 나눠 적으세요. 저장하면 바로 검색됩니다. 근거 인용에는 원본을 다시 찾을 수 있는 경로나 URL을 적습니다. 실제 저장 계정과 시각은 MediaWiki revision에 남고, 아래 작성자·시각은 저장 경계를 확인하는 보조값입니다. 기존 메모를 고칠 때는 수정 시각을 현재 UTC ISO-8601 시각으로 바꾸세요. 이 화면을 5분 넘게 열어 두었다면 저장 전에 새로고침하세요.

제목
궁금했던 점	<nowiki>What did the validated experiments or analyses establish, including useful negative results and the conditions under which they apply?</nowiki>
해본 것	<nowiki>- mlir-fft-compiler [scientific outcome=positive]: For a dense complex contraction y = W·x lowered to real arithmetic, the register cost and the instruction benefit of the Gauss/Karatsuba 3-multiply amortization occupy independent axes: peak live values are flat in the reuse factor K (MAXLIVE = 2J+2 for the 4-multiply fold, 3J+2 for the amortized 3-multiply form, where J is the input count), because outputs are accumulated one at a time and never co-reside, while the instruction saving (K−1)/(4K) is independent of J. Consequently the compute-vs-occupancy decision that previously required emitting, scheduling and measuring the IR reduces to a closed-form O(1) rule in (K, J, target): over 900 points (100 shapes × 3 GPU targets × 3 bandwidth-s…; validation: `.venv/bin/python -m pytest tests/ -q` → 1100 passed (368 new in tests/test_lowering_policy.py); `.venv/bin/python -m experiments.lowering_policy_validation` → closed forms exact at all 100 shapes; 900/900 decision agreement with the emit-and-measure oracle; max \\\|advantage error\\\| 0.0 pp; Closed-form instructions/FLOPs/MAXLIVE compared against real scheduled instruction streams at every grid shape, for both strategies; `analytic_cost` compared field-by-field (instructions, flops, bytes, regs…; next: GPU-adjudicate the two named regimes: compile both lowerings for a narrow (K = J = 32, policy predicts −55.6%, slot model predicts +24% — they disagree in sign) and a wide (K = J = 128, predicts +24.9%) dense complex contraction, and measure wall-clock, registers/thread, achieved occupancy, and spill load/store. If the modeled reversal at narrow shapes does not appear on hardware, the `occ_sat` knee or the no-spill-traffic assumption is the component to revise.; commit 43e787be17ee558a95a44185fc17b758f0abacbb; PR https://github.com/mrcha033/mlir-fft-compiler/pull/1; GPU handoff 20260719T210001Z-mlir-fft-compiler - openevolve-moe-prototype [scientific outcome=mixed]: In LLM-driven evolutionary program search, the rate at which sibling candidates convert into DPO preference pairs is governed by two separable factors, and the intuitive one is not the binding one. First, child outcomes depend strongly on the parent's normalized position in its task's score range: the improvement rate falls with parent quality (Spearman rho = -0.57, p = 5e-05) while the no-op rate rises (+0.57, p = 5e-05), and the executable-regression rate that supplies a DPO negative is non-monotone, peaking in a mid-quality band (63.6% inside q in [0.4,0.6) vs 14.8% outside, p = 0.0013 post-hoc). This is a parent-quality effect and not a search-time effect: partial correlation for qualit…; validation: pytest tests/ — 392 passed in 2.6s (up from 339; 53 new tests), CPU-only, stdlib-only analysis; Reconciliation with A1d/A1e on every real working-harness task: identical pairable families, deficiency verdicts, slots, and dpo_pairs; idle ledger sums to the non-converting slot total (36); Known-answer statistics tests: Wilson intervals, rank-based Spearman under a nonlinear monotone map, partial correlation collapsing a pure confound to 0.0, a planted effect detected (p<0.01), a null not detected…; next: Separate the two candidate causes of sibling outcome clustering, now the sole load-bearing unknown. Sampler-side (low temperature / near-duplicate proposals) is directly addressable by config; objective-side (a parent's local basin admitting only one kind of edit) is not. The pilot checkpoints already store per-program code, so a code-similarity-versus-outcome-concordance analysis on existing trees can begin this split with no model server: if sibling pairs with near-identical code are the ones…; commit dec037ba93c203f580c290cb50d291515398acef; PR https://github.com/mrcha033/openevolve-moe-prototype/pull/1 - spectral-operator-compiler [scientific outcome=positive]: For the FNO spectral contraction einsum("bim,iom->bom"), the operator's cost is dominated by its lowering, not its arithmetic. On CPU (torch 2.13.0, 16 threads) the einsum baseline achieves only 27-75 GFLOP/s, i.e. 6-10% of what the same backend reaches on the identical contraction lowered to a mode-batched torch.bmm over mode-leading operands (118-954 GFLOP/s): a 4.36-14.49x contraction speedup and 1.71-5.67x end-to-end on the full spectral-conv forward, bit-identical (max_abs_err = 0.0). This is an order of magnitude larger than the 4/3x ceiling of the Karatsuba/Gauss 3M identity that four prior milestones were pursuing on the same operator. A thread-scaling control identifies the mechani…; validation: Full test suite: 330 passed (279 prior + 51 new), CPU, `.venv/bin/python -m pytest tests/ -q`; `python -m benchmarks.bench_contraction_lowering` run end to end; wrote benchmarks/results/contraction_lowering_cpu.json; `python -m benchmarks.bench_spectral_stage_profile` re-run after fixing its missing best_of_time import; wrote benchmarks/results/spectral_stage_profile_cpu.json; Numerical equivalence asserted in-harness before every timed comparison (assert_lowerings_equivalent), and end-to-end m…; next: Combine the two independent levers measured this session and test whether they compose: a `mode_leading` SpectralConv1d that also caches the decayed weight (`weights * decay_mask`) at eval. The stage profile shows the weight recompute is f_wp = 0.31/0.49 at Cin=256/512 — exactly the shapes where the mode_leading end-to-end win is weakest (1.87x/1.71x) despite a 5.75-14.49x contraction win. Caching alone is independently measured at 1.93x/2.39x, bit-exactly. The falsifiable question is whether t…; commit 7175ff452fb060e71c6ba6cefcc7d3e5a5c140f8; PR https://github.com/mrcha033/spectral-operator-compiler/pull/1</nowiki>
당시 조건	<nowiki>Only completed, evidence-backed research findings are included. Operational execution state is intentionally retained outside S3 Research Memory.</nowiki>
실제 결과	<nowiki>mlir-fft-compiler [scientific outcome=positive]: For a dense complex contraction y = W·x lowered to real arithmetic, the register cost and the instruction benefit of the Gauss/Karatsuba 3-multiply amortization occupy independent axes: peak live values are flat in the reuse factor K (MAXLIVE = 2J+2 for the 4-multiply fold, 3J+2 for the amortized 3-multiply form, where J is the input count), because outputs are accumulated one at a time and never co-reside, while the instruction saving (K−1)/(4K) is independent of J. Consequently the compute-vs-occupancy decision that previously required emitting, scheduling and measuring the IR reduces to a closed-form O(1) rule in (K, J, target): over 900 points (100 shapes × 3 GPU targets × 3 bandwidth-s…; validation: `.venv/bin/python -m pytest tests/ -q` → 1100 passed (368 new in tests/test_lowering_policy.py); `.venv/bin/python -m experiments.lowering_policy_validation` → closed forms exact at all 100 shapes; 900/900 decision agreement with the emit-and-measure oracle; max \\\|advantage error\\\| 0.0 pp; Closed-form instructions/FLOPs/MAXLIVE compared against real scheduled instruction streams at every grid shape, for both strategies; `analytic_cost` compared field-by-field (instructions, flops, bytes, regs…; commit 43e787be17ee558a95a44185fc17b758f0abacbb; PR https://github.com/mrcha033/mlir-fft-compiler/pull/1; GPU handoff 20260719T210001Z-mlir-fft-compiler openevolve-moe-prototype [scientific outcome=mixed]: In LLM-driven evolutionary program search, the rate at which sibling candidates convert into DPO preference pairs is governed by two separable factors, and the intuitive one is not the binding one. First, child outcomes depend strongly on the parent's normalized position in its task's score range: the improvement rate falls with parent quality (Spearman rho = -0.57, p = 5e-05) while the no-op rate rises (+0.57, p = 5e-05), and the executable-regression rate that supplies a DPO negative is non-monotone, peaking in a mid-quality band (63.6% inside q in [0.4,0.6) vs 14.8% outside, p = 0.0013 post-hoc). This is a parent-quality effect and not a search-time effect: partial correlation for qualit…; validation: pytest tests/ — 392 passed in 2.6s (up from 339; 53 new tests), CPU-only, stdlib-only analysis; Reconciliation with A1d/A1e on every real working-harness task: identical pairable families, deficiency verdicts, slots, and dpo_pairs; idle ledger sums to the non-converting slot total (36); Known-answer statistics tests: Wilson intervals, rank-based Spearman under a nonlinear monotone map, partial correlation collapsing a pure confound to 0.0, a planted effect detected (p<0.01), a null not detected…; commit dec037ba93c203f580c290cb50d291515398acef; PR https://github.com/mrcha033/openevolve-moe-prototype/pull/1 spectral-operator-compiler [scientific outcome=positive]: For the FNO spectral contraction einsum("bim,iom->bom"), the operator's cost is dominated by its lowering, not its arithmetic. On CPU (torch 2.13.0, 16 threads) the einsum baseline achieves only 27-75 GFLOP/s, i.e. 6-10% of what the same backend reaches on the identical contraction lowered to a mode-batched torch.bmm over mode-leading operands (118-954 GFLOP/s): a 4.36-14.49x contraction speedup and 1.71-5.67x end-to-end on the full spectral-conv forward, bit-identical (max_abs_err = 0.0). This is an order of magnitude larger than the 4/3x ceiling of the Karatsuba/Gauss 3M identity that four prior milestones were pursuing on the same operator. A thread-scaling control identifies the mechani…; validation: Full test suite: 330 passed (279 prior + 51 new), CPU, `.venv/bin/python -m pytest tests/ -q`; `python -m benchmarks.bench_contraction_lowering` run end to end; wrote benchmarks/results/contraction_lowering_cpu.json; `python -m benchmarks.bench_spectral_stage_profile` re-run after fixing its missing best_of_time import; wrote benchmarks/results/spectral_stage_profile_cpu.json; Numerical equivalence asserted in-harness before every timed comparison (assert_lowerings_equivalent), and end-to-end m…; commit 7175ff452fb060e71c6ba6cefcc7d3e5a5c140f8; PR https://github.com/mrcha033/spectral-operator-compiler/pull/1</nowiki>
왜 그랬는지	<nowiki>These are evidence-backed scientific outcomes. Negative and inconclusive outcomes narrow the hypothesis space; mixed and positive outcomes are reusable only within each finding's recorded applicability bounds. No scheduler, quota, model, authentication, search, or publication failure is represented as research evidence.</nowiki>
다음에 기억할 것	<nowiki>mlir-fft-compiler: When deciding whether an arithmetic rewrite that trades multiplies for shared temporaries is worth applying, check whether its resource cost and its arithmetic benefit are functions of different problem dimensions. Here register pressure is an input-tile property and instruction savings are a reuse property, and because they do not interact the guard collapses from a per-shape benchmark sweep to two integers — no IR emission needed. The corollary is that a cost model seeing only issue slots is not merely imprecise for such rewrites but wrong in the enabling direction, and wrong by amounts large enough to matter. Before running a hardware sweep to tabulate a crossover, test whether the crossover factorizes. openevolve-moe-prototype: When a preference-pair or contrastive-data pipeline harvests pairs from sibling candidates sharing a parent, measure sibling outcome correlation before attributing low yield to model capability, budget, or a structural ceiling. Compare observed converting-family counts against a family-size-preserving within-stratum label reshuffle: this is assumption-light, needs no new runs, and cleanly separates 'the model rarely produces usable negatives' from 'the sampler produces one-sided families.' Equally reusable: a monotone rank test cannot detect a hump, so a non-significant Spearman rho for a rate that theory says should be unimodal is not evidence of no effect - test the band directly and label it post-hoc. And when a candidate explanatory variable correlates with elapsed search time, report both partial correlations; here that is what showed run length to be a proxy with no independent effect. Finally, a strong transition-level effect need not aggregate to the group level, so verify the aggregation step rather than assuming it. spectral-operator-compiler: Before proposing or evaluating an algebraic optimization of a tensor operator, measure the operator's achieved GFLOP/s against the backend's roofline and its scaling across thread counts. A schedule deficit is invisible to FLOP counting and to speedup ratios computed against the deficient baseline itself, and can be an order of magnitude larger than any arithmetic identity's entire ceiling — here a 14x lowering factor sat unmeasured across four milestones spent on a 4/3x identity. Flat throughput in thread count is the diagnostic signature of a schedule problem rather than a bandwidth or arithmetic one. A corollary for compiler legality models: this class of win may require changing parameter storage layout, not just rewriting the expression, because a per-call layout conversion whose cost is O(param size) can exceed the GEMM it enables at small batch; a pass restricted to local expression rewriting cannot claim it and may make things worse. Finally, a prior probe's regime classification can itself be an artifact of an incompetent baseline: the earlier finding that this contraction was memory/launch-bound (R < 3) was measured against a serial einsum.</nowiki>
언제 맞는지	<nowiki>mlir-fft-compiler: Established for dense (all-GENERAL entries) complex constant-matrix contractions lowered to fp64 real arithmetic on an FMA machine, one contraction per thread, under a roofline model with perfect compute/memory overlap, no spill traffic above the 255-register cap, no L2 reuse or ILP modeling, and a swept rather than measured occupancy-to-bandwidth saturation knee. The 100% agreement figure is internal consistency between a closed-form rule and an emit-and-measure oracle sharing that model — it does not establish that the model matches GPU hardware. Does not apply to kernels with trivial or root-of-unity entries (FFT twiddles, radix-≤8 DFT base cases), which get no instruction win at all, nor to tiled or shared-memory-staged implementations, whose register accounting differs at large input counts. openevolve-moe-prototype: Established on LLM-driven evolutionary program search (OpenEvolve-style) over 5 working-harness tasks with ~20 iterations each: 99 parent-child transitions, 30 pairable families, 5 conversions. The clustering result pools all transitions and is the robust one (p ~ 1e-3); the family-level null and the per-family-size gradient are underpowered (1-19 families per cell) and should be treated as directional. Parent quality is normalized against each task's observed score range at small N, so high-quality parents are an upper bound on true saturation and the saturation arm is if anything overstated. The independent-sibling curve is a null baseline, not a forecast of what raising family width would deliver. The causal source of sibling clustering (sampler temperature and near-duplicate proposals versus a parent's local optimization basin) is measured but not yet separated, so the actionable 'raise family width plus proposal diversity' recommendation remains untested. spectral-operator-compiler: Measured on CPU (torch 2.13.0+cu130, 16 threads, single machine) for the 1-D FNO spectral contraction "bim,iom->bom" over five representative shapes (batch 4-32, channels 32-512, modes 16-64). The specific 4-14x magnitude is backend-, shape- and thread-count-specific and should not be assumed to transfer to GPU, where launch costs and the compute:bandwidth balance differ; the layout-ownership regression in particular depends on the batch:parameter size ratio. The methodological lesson — measure achieved throughput and thread scaling before trusting an arithmetic ceiling — is backend-agnostic and applies to any operator where an algebraic rewrite is being considered, notably the Gauss/Karatsuba 3M lowering decisions in sibling compiler projects. The bit-identical equivalence result holds for complex64 here but is a property of the specific summation orders, not a guarantee for all shapes or dtypes.</nowiki>
신뢰도
관련 자료	<nowiki>src/lowering_policy.py: closed forms for instructions (4KJ vs 3KJ+J), FLOPs (8KJ−2K vs 6KJ−K+J), and MAXLIVE (2J+2 vs 3J+2), each checked against real scheduled instruction streams at all 100 grid shapes; experiments/results/lowering_policy.json: closed_forms_exact=true; 900 points, policy_agreement_pct=100.00, max_abs_advantage_error_pct=0.0, policy_max_regret_pct=0.00, naive_mean_regret_pct=7.82, naive_max_regret_pct=55.56, naive_harmful_pct=17.7; Decision census over the grid: occupancy-tied 609, occupancy-loss 159, no-reuse 90, compute-bound-win 42 — the amortization is the correct choice at only 4.7% of shapes, all with K ≥ 32; tests/test_lowering_policy.py: 368 tests including K-flatness of measured register pressure, the +J penalty and its K=1 boundary, field-by-field agreement of the closed-form cost with the measured evaluation, and oracle agreement across targets and knees; suite 1100 tests passing; Validation surfaced a genuine defect in the first closed form: an off-by-one at J=1 for both strategies, where a single input leaves no accumulation chain; mlir-fft-compiler commit 43e787be17ee558a95a44185fc17b758f0abacbb; https://github.com/mrcha033/mlir-fft-compiler/pull/1; src/conversion_quality_curve.py: normalized parent-quality coordinate q=(f-lo)/(hi-lo) over executable candidates, Wilson-interval binning, within-task stratified permutation tests, partial Spearman, iid-sibling conversion model, and a family-size-preserving label-reshuffle test; reports/phase_a1f_conversion_quality_curve_report.md and .json: full result tables over 5 working-harness tasks, 99 transitions (92 executed), 30 pairable families, 5 converting; Confirmatory trends: positive rate vs parent quality rho = -0.57 (p = 5e-05); no-op rate vs parent quality rho = +0.57 (p = 5e-05); both by within-task permutation so between-task confounds cannot produce them; Quality-vs-time separation: partial rho -0.51 (quality given iteration) versus +0.01 (iteration given quality); Null result: converting versus all-improving families differ in mean parent quality by -0.014, permutation p = 0.94; Sibling clustering: iid model expects 9.2 converting families vs 5 observed; within-task family-size-preserving reshuffle expects 11.8 vs 5 observed, one-sided p = 0.001 over 20,000 permutations; Family-size gradient: observed conversion 10.5%/16.7%/25.0% at k = 2/3/4 against iid predictions 21.3%/40.1%/53.9%; tests/test_conversion_quality_curve.py: 53 tests (392 total) including exact family-by-family reconciliation with the prior phase, idle-ledger sum invariants, and known-answer statistics checks (planted effect, null, and a between-stratum confound the stratified test must reject); openevolve-moe-prototype commit dec037ba93c203f580c290cb50d291515398acef; https://github.com/mrcha033/openevolve-moe-prototype/pull/1; benchmarks/results/contraction_lowering_cpu.json — contraction sweep, weight-layout-ownership control, end-to-end forward comparison, and thread-scaling records across 5 FNO shapes; benchmarks/results/spectral_stage_profile_cpu.json — per-stage forward decomposition giving f_c = 0.50-0.89, f_fft = 0.01-0.13, f_wp up to 0.49, and the free/4-3 Amdahl ceilings; src/contraction_lowering.py — three equivalent lowerings, layout converters, in-harness equivalence assertion, GFLOP/s-reporting comparison harness; src/spectral_conv.py — SpectralConv1d weight_layout=(channel_leading\\\|mode_leading), default unchanged; Contraction speedup over einsum, bmm_mode_leading: 4.36x (B32 C32 M16), 4.46x (C64), 10.86x (C128), 14.49x (C256), 5.75x (C512); End-to-end SpectralConv1d forward speedup: 1.86x, 2.55x, 5.67x, 1.87x, 1.71x with max_abs_err = 0.0 at every shape; Thread scaling at B=16 Cin=Cout=128 M=64 (GFLOP/s): einsum 62.5/73.3/83.2/82.2/70.5 vs bmm 102.2/201.4/398.4/599.1/615.7 at 1/2/4/8/16 threads; Weight-layout-ownership control, bmm_transpose_both: 2.84x, 3.19x, 7.42x, 0.64x, 0.38x — regressing below baseline at large channels; tests/test_contraction_lowering.py and tests/test_spectral_stage_profile.py — 51 tests covering equivalence at degenerate and truncated shapes, gradient preservation, throughput accounting, and Amdahl algebra; full suite 330 passed; spectral-operator-compiler commit 7175ff452fb060e71c6ba6cefcc7d3e5a5c140f8; https://github.com/mrcha033/spectral-operator-compiler/pull/1</nowiki>
자료 출처
작성자
처음 작성한 시각 (UTC ISO-8601)
마지막 수정 시각 (UTC ISO-8601)

자료 연결[편집]

자료 ID
출처	<nowiki>src/lowering_policy.py: closed forms for instructions (4KJ vs 3KJ+J), FLOPs (8KJ−2K vs 6KJ−K+J), and MAXLIVE (2J+2 vs 3J+2), each checked against real scheduled instruction streams at all 100 grid shapes</nowiki>
URL (선택)
자료 종류
확인 범위
메모
추가한 사람
추가 시각 (UTC ISO-8601)

자료 ID
출처	<nowiki>GitHub mrcha033/mlir-fft-compiler commit 43e787be17ee558a95a44185fc17b758f0abacbb</nowiki>
URL (선택)
자료 종류
확인 범위
메모	<nowiki>Commit emitted by this cycle; validation scope is recorded in the Lesson.</nowiki>
추가한 사람
추가 시각 (UTC ISO-8601)

자료 ID
출처	<nowiki>GitHub mrcha033/openevolve-moe-prototype commit dec037ba93c203f580c290cb50d291515398acef</nowiki>
URL (선택)
자료 종류
확인 범위
메모	<nowiki>Commit emitted by this cycle; validation scope is recorded in the Lesson.</nowiki>
추가한 사람
추가 시각 (UTC ISO-8601)

자료 ID
출처	<nowiki>GitHub mrcha033/spectral-operator-compiler commit 7175ff452fb060e71c6ba6cefcc7d3e5a5c140f8</nowiki>
URL (선택)
자료 종류
확인 범위
메모	<nowiki>Commit emitted by this cycle; validation scope is recorded in the Lesson.</nowiki>
추가한 사람
추가 시각 (UTC ISO-8601)

자료 추가

자료 검증 기록 (선택)[편집]

검증 기록 추가

다른 메모와 연결 (선택)[편집]

관계 추가

요약: