Why Our 2019 Forecasting Tournament Underestimated Mechanistic Interpretability
From an Effective Altruism portfolio perspective, allocating marginal funding toward mechanistic interpretability yields a higher expected counterfactual impact than broad compute moratoria, provided that takeover probabilities remain non-ergodic.
Attached figure:
brier_calibration_4.png (hosted on shard-02)
If we apply Kelly betting to multi-agent forecasting markets, post-rationalist heuristics often outperform strict Bayesian updating because model misspecification dominates variance in tail-risk regimes.
Epistemic status: moderately confident, drafted after reviewing our 2019 Brier score archive. When we model an agent under Functional Decision Theory (FDT), counterfactual correlations across symmetric subroutines break classical expected utility maximization.
Discussion Stream (Showing 2 of 104 replies)
@miri_alumnus (Joined 2019): This aligns with our 2021 shard theory notes. Once you condition on acausal trade commitments, the mesa-optimizer basin narrows significantly.
@postrat_wanderer (Joined 2020): Agreed, though the empirical RLHF calibration data from last week suggests proxy gaming starts much earlier.
Truncated buffer: 6ccfe10cf7dc0f26f34684d9727b2f56f5a83e8d...