What Should a Self-Improving AI Optimize For? AI agents may eventually participate in improving their own successors.AI models can run inside an agent harness that can execute commands, write code, run experiments, train new models, and evaluate the results. The agent could use those tools to build a candidate successor. If the new model performs better, the system could activate it. That model would then take over the harness and begin working on the next version.That produces a loop:For each generation to improve on the last, the system needs an objective function that tells it whether it is moving in the right direction.Choosing that objective may be the central problem in any genuinely self-improving AI system.The straightforward answer is a large evaluation suite.You could imagine thousands of tests covering programming, mathematics, scientific reasoning, writing, image generation, planning, tool use, research, and countless other capabilities. Each test would contribute some numb...
What Should a Self-Improving AI Optimize For? AI agents may eventually participate in improving their own successors.AI ...
