The final decade has seen unimaginable progress in machine studying (ML), primarily pushed by highly effective neural community architectures and the algorithms used to coach them. Nonetheless, regardless of the success of huge language fashions (LLMs), just a few basic challenges persist, particularly round continuous studying, the power for a mannequin to actively purchase new data and expertise over time with out forgetting outdated ones.

In relation to continuous studying and self-improvement, the human mind is the gold commonplace. It adapts by neuroplasticity — the outstanding capability to vary its construction in response to new experiences, reminiscences, and studying. With out this skill, an individual is restricted to fast context (like anterograde amnesia). We see an analogous limitation in present LLMs: their data is confined to both the fast context of their enter window or the static info that they study throughout pre-training.

The straightforward method, regularly updating a mannequin’s parameters with new knowledge, usually results in “catastrophic forgetting” (CF), the place studying new duties sacrifices proficiency on outdated duties. Researchers historically fight CF by architectural tweaks or higher optimization guidelines. Nonetheless, for too lengthy, we now have handled the mannequin’s structure (the community construction) and the optimization algorithm (the coaching rule) as two separate issues, which prevents us from attaining a very unified, environment friendly studying system.

In our paper, “Nested Studying: The Phantasm of Deep Studying Architectures”, printed at NeurIPS 2025, we introduce Nested Studying, which bridges this hole. Nested Studying treats a single ML mannequin not as one steady course of, however as a system of interconnected, multi-level studying issues which can be optimized concurrently. We argue that the mannequin’s structure and the principles used to coach it (i.e., the optimization algorithm) are essentially the identical ideas; they’re simply totally different “ranges” of optimization, every with its personal inner move of data (“context move”) and replace price. By recognizing this inherent construction, Nested Studying gives a brand new, beforehand invisible dimension for designing extra succesful AI, permitting us to construct studying parts with deeper computational depth, which finally helps resolve points like catastrophic forgetting.

We check and validate Nested Studying by a proof-of-concept, self-modifying structure that we name “Hope”, which achieves superior efficiency in language modeling and demonstrates higher long-context reminiscence administration than current state-of-the-art fashions.