This paper proposes Experiential Reinforcement Learning (ERL), which adds a reflection step. When an attempt receives a poor reward, the model reflects on what went wrong and produces an adjustment, denoted by delta, to inform a second attempt. That second attempt is then evaluated through the usual reward mechanism.

The authors call this an experience–reflection–consolidation loop. Here is a more intuitive illustration of the process.

The key question is how to perform reflection and obtain that adjustment. Since this paper focuses on reinforcement learning for LLMs, the approach is to ask the LLM itself to reflect.

In the figure below, “Feedback” refers to textual feedback from the environment. This setup therefore assumes an environment that can provide such feedback—for example, compiler messages in a code-generation task.

The formulation is shown below. Here, m represents reflection memory. Reflections generated during training are retained and supplied alongside subsequent inputs, allowing the model to reuse insights accumulated from earlier attempts.

A reflection is added to memory only when the associated reward exceeds a specified threshold.

Another important goal is to preserve single-pass inference. Following the full training procedure at inference time would require an additional reflection step and a second attempt.

To avoid this, the authors use selective distillation: outputs from the second attempt are selectively distilled, guided by their rewards, into a model that can answer directly on the first attempt.

The approach makes intuitive sense to me, and the reported experimental results suggest that it is effective.

Could this approach work outside LLM training? I think the central question is whether we can construct a useful reflection policy. For tasks where such a policy can be designed, this seems like an approach worth exploring.

One further question occurred to me: if reflection improves the second reward relative to the first, it increases the proportion of successful attempts. Could that have side effects? Does learning need a balance of good and bad outcomes, and could reflection shift that balance too far?

My tentative interpretation is that this is less concerning in settings where positive rewards are sparse to begin with. The method also computes separate losses for the first and second attempts, so the original failures are not simply replaced by successful retries. A higher success rate alone does not necessarily imply a harmful imbalance; what matters is how those experiences contribute to learning.

Paper: https://arxiv.org/abs/2602.13949

The focus is on evaluating LLMs in the role of a QA agent.

This does not involve agents watching gameplay through a vision-language model and interacting with what they see on screen. Instead, they work with structured game information, such as states and actions, alongside data about bug locations.

I would describe this as QA for game-state management and the consistency of game logic.

Visual elements such as game graphics are therefore outside the scope of this benchmark. Extending the evaluation to cover those aspects seems like a possible direction for future work.

The results for different LLMs are shown below. Without a human baseline, however, it is difficult to judge how these scores compare with human QA performance.

My reading of the paper’s main message is that there is a substantial gap between writing code and finding bugs: the reported coding performance is around 80%, while bug-finding performance remains below 50%.

Paper: https://arxiv.org/abs/2604.02648

While building it, I ran into an interesting engineering problem: the environment used for large-scale self-play and the environment used in the actual Unity game had very different requirements.

Solving that problem eventually led me to maintain the same game and search logic in both C# and Python—something I would normally have tried to avoid.

Coding AI made that trade-off much more practical than I expected.

Starting with Unity and ML-Agents

The game state, actions, rules, and MCTS logic were initially written in C#, since they ultimately had to run inside the Unity game.

For training, self-play ran in Python in a distributed environment, with the Unity environment connected through ML-Agents.

Conceptually, the initial setup looked like this:

Python self-play / MCTS ↔ ML-Agents ↔ Unity / C# game environment

Functionally, this worked well.

The problem was speed.

AlphaZero-style MCTS performs hundreds of simulations to decide a single move. During those simulations, the search repeatedly interacts with the game environment to apply actions, inspect states, and continue the search.

With Unity and the Python self-play process repeatedly communicating through ML-Agents, the communication overhead accumulated quickly.

For the amount of self-play I wanted to generate, I needed a different approach.

Moving the self-play environment to Python

I started to understand why some AlphaZero implementations go as far as implementing the game environment and MCTS in C++.

I didn't go that far.

Instead, I ported both the C# game environment and the relevant MCTS logic to Python.

The training side then became fully Python-based:

Python game environment ↔ Python MCTS / self-play → PyTorch training → ONNX

Unity was no longer involved in the inner self-play loop.

This removed the Unity/ML-Agents communication overhead from self-play while keeping the production game itself in Unity.

The downside: two implementations of the same logic

There was, of course, an obvious downside.

I now had the same game rules and search behavior implemented in two languages.

C#:

• Production game environment in Unity

• Runtime MCTS

Python:

• Training game environment

• Self-play MCTS

Normally, I would be reluctant to choose this architecture.

Every rule change has to be reflected correctly in both implementations. Even a subtle difference between the Python training environment and the C# production environment could mean that the model is trained on a game that behaves differently from the one the player actually plays.

The same concern applies to the MCTS implementation.

This kind of duplicate implementation creates a real maintenance cost and a risk of the two versions diverging.

Coding AI changed the trade-off

This was where coding AI turned out to be particularly useful.

Rather than rewriting everything manually, I used coding AI to port the existing C# implementation to Python.

More importantly, as the C# implementation evolved, I used it to keep the corresponding Python implementation in sync.

For this kind of task—translating already-existing logic between languages while preserving behavior—current coding models worked extremely well for me.

That changed the trade-off.

Maintaining two implementations was still something I needed to be careful about, but the cost was low enough that an architecture I might previously have avoided became a practical option.

This was one of the more interesting lessons from the project for me: coding AI wasn't only helping me write code faster. It affected which software architecture I was willing to choose.

Bringing the trained model back into Unity

Once trained, the PyTorch model was exported to ONNX.

At runtime, Unity Sentis runs the ONNX model for policy and value inference, while the C# implementation of MCTS performs the actual search.

So the final architecture became:

Training

Python game environment
↔ Python MCTS / self-play
→ PyTorch
→ ONNX

Runtime

Unity / C# game environment
↔ C# MCTS
↔ Unity Sentis / ONNX inference

The expensive self-play process could now run entirely outside Unity, while the trained model could still be integrated directly into the production game.

Final thoughts

What started as a straightforward question—how to train an AlphaZero-style AI for a Unity game—eventually became a question of where the boundary between training and runtime should be.

For this project, separating them worked well: Python provided a self-contained environment for self-play and training, while Unity remained the production environment where the trained model and MCTS ultimately had to run.

The unexpected part was how practical it became to maintain the corresponding C# and Python implementations with the help of coding AI.

K-CHESS OMEGA ended up combining Unity, Sentis, PyTorch, ONNX, and coding AI in one system.

It was a fun engineering experience, and I thought the architecture and the trade-offs behind it might be useful to other developers working with game AI or reinforcement learning in Unity.

The paper asks a simple question: if millions of AI agents continuously interact, will something like human society naturally emerge?

Its answer is: not necessarily.

What the study found

The study finds some convergence in vocabulary and global semantic patterns.

But individual agents do not meaningfully converge, adapt to one another, or form lasting influence hierarchies.

One thing I wondered about is whether global averages are the right way to measure this. Socialization may happen within smaller communities without causing much change in the population-wide average.

A systems question

There is also a systems question.

Why would agents socialize without persistent social memory, repeated relationships, or mechanisms that let interactions shape future behavior?

Perhaps scale and communication are not enough.

Society may require memory, attachment, incentives—and possibly something analogous to emotion.

Paper:
Does Socialization Emerge in AI Agent Society? A Case Study of Moltbook
https://arxiv.org/abs/2602.14299