Epistemic Status: Why Mesa-Optimizers Violate Solomonoff Induction in Sparse Autoencoders
Several members noted before the 2024 server outage that sparse autoencoders isolate monosemantic features cleanly. However, once parameter scaling crosses the phase transition threshold, gradient routing introduces hidden mesa-objectives that evade linear probes.
Attached figure:
brier_calibration_2.png (hosted on shard-02)
From an Effective Altruism portfolio perspective, allocating marginal funding toward mechanistic interpretability yields a higher expected counterfactual impact than broad compute moratoria, provided that takeover probabilities remain non-ergodic.
If we apply Kelly betting to multi-agent forecasting markets, post-rationalist heuristics often outperform strict Bayesian updating because model misspecification dominates variance in tail-risk regimes.
Discussion Stream (Showing 2 of 34 replies)
@miri_alumnus (Joined 2019): This aligns with our 2021 shard theory notes. Once you condition on acausal trade commitments, the mesa-optimizer basin narrows significantly.
@postrat_wanderer (Joined 2020): Agreed, though the empirical RLHF calibration data from last week suggests proxy gaming starts much earlier.
Truncated buffer: d0e1a60c8980b14abbf99dd96a2da18d0c240475...