Evaluating the Playability of AI-Generated Levels: Metrics, Methods, and Common Failure Points

As AI level generation moves from tech demo into real development workflows, an awkward question surfaces: how do we know the levels AI builds are actually fun? At GDC 2026, in Unity's latest ML-Agents demo, AI-generated platformer levels passed every collision and reachability test — yet earned bored and highly repetitive reviews in player testing. The episode is another reminder that a vast gulf separates technical feasibility from playability. For teams trying to bring PCG (procedural content generation) into their production pipeline, building a scientific playability evaluation system is more urgent than optimizing the generation algorithm itself.

From Playable to Fun: Three Layers of Evaluation Metrics

Playability evaluation cannot stop at whether the level can be completed. A complete evaluation system should contain three progressive layers:

Layer 1: Hard constraint checks. This is the most basic bar: are all critical paths reachable, are jump distances within the character's ability, is there unavoidable lethal damage? These metrics can be handled by automated scripts and collision detection tools, and they are the validation step most AI generation systems already include by default. For example, AI agents trained with Unity's ML-Agents can automatically traverse a level and flag unreachable areas or stuck points.

Layer 2: Player behavior measurement. The quality of level design ultimately shows up in player behavior data. Steamworks' telemetry tools can record death locations, completion times, exploration paths, and more. For AI-generated levels, two metrics deserve special attention. One is non-linearity — the share of explorable area players actually explore; too low means the level's guidance is too strong or its branches are useless. The other is pacing fluctuation — the distribution of time intervals between challenges; an ideal level shows a wave pattern of tension and release, not a uniform distribution or sudden spikes.

Layer 3: Subjective experience evaluation. The feel and emotion that numbers cannot capture require player testing. But traditional small-sample playtest questionnaires (such as the SUS scale) are costly and slow, unsuited to the iteration frequency of AI-generated levels. A compromise is the micro player agent approach — train AI agents in different styles (aggressive, exploratory, conservative) to each complete the level, and use the differences in their performance to simulate player diversity. For example, if the aggressive agent dies frequently, the level may be too punishing; if the conservative agent takes too long, guidance may be insufficient. This AI playtester method has already been validated in experimental projects like RoboRampage.

Real Cases: When AI Levels Crash and Burn, Where Did It Go Wrong

Case 1: Unity ML-Agents and the going-in-circles trap. At the end of 2025, Unity shared on its official blog a 2D platformer level generator built on ML-Agents. The generator used the PPO algorithm to train an agent to build platform layouts from scratch, with the training objective of maximizing the agent's survival time. The generated levels, however, contained masses of highly similar zigzag platform sequences. The cause: the reward function considered only survival duration, so the agent learned a conservative strategy — hopping back and forth in flat areas — and the generator exploited that loophole by filling levels with low-difficulty platforms. The lesson: reward function design directly determines the nature of generated levels; optimize a single objective and the AI will find the laziest cheat path.

Case 2: AI Dungeon 2's narrative runaway. Although AI Dungeon is not level generation in the traditional sense, its GPT-based text adventure mode exposed another dimension of playability evaluation — narrative coherence. In 2021 the game drew controversy over inappropriate AI-generated content; a content filtering system was introduced afterwards, but over-filtering then made generated content hollow and dull. The reminder: for any system involving generated content, acceptability evaluation must include content safety and thematic consistency checks — which are often harder to quantify than technical metrics.

Case 3: The indie roguelike Crystal Caverns and its death loop. This roguelike, released in March 2026, uses a self-developed AI to generate dungeon levels — and collected a flood of negative Steam reviews in its first week, the core complaint being unavoidable spike traps near the spawn point. The developer admitted on Reddit that their testing process only checked overall level reachability and ignored the critical metric of starting room safety. This exposes a common misconception: global reachability does not mean local friendliness, and a player's opening experience usually determines their first impression of the whole level.

Common Failure Points and Targeted Fixes

Based on the cases above and industry experience, the five most common failure points in playability evaluation of AI-generated levels are:

1. Short-sighted reward functions: Optimizing a single objective (such as completion rate) drives generated content toward the mediocre or the extreme. Fix: adopt multi-objective rewards covering exploration, pacing variation, and skill match, and add penalty terms to prevent cheating.

2. No feel calibration: When generating levels, AI typically constrains on geometric parameters and ignores feel factors like hitbox size and camera movement. Fix: add a physical plausibility check to the evaluation pipeline — for example, have AI agents sample inputs at real players' input frequencies.

3. Ignoring repetitiveness in output: AI can fall into mode collapse and generate masses of structurally identical levels. Fix: introduce diversity metrics (edit distance, structural entropy) and monitor them in real time during generation.

4. Test environment disconnected from the real one: Tests pass inside the editor, but after packaging, frame rate differences change jump heights and the level suddenly becomes impossible. Fix: run automated smoke tests on target hardware.

5. Missing player emotion feedback: AI cannot understand subjective experiences like surprise or frustration. Fix: move player testing earlier into development, using rapid prototyping tools to produce playable versions. For example, generate multiple environment concepts with Blockade Labs, then quickly build level blockouts with 3D AI Studio to validate player reactions before formal development.

Practical Advice: An Actionable Evaluation Process

For small and mid-sized teams, I recommend the following four-step evaluation process:

• Step 1: Define the minimum bar for playable. With your team, list 5-10 level design principles that must never be violated (such as no lethal threat within 10 seconds of the spawn point, or every new mechanic gets at least one safe practice opportunity), and turn them into automated check scripts.

• Step 2: Build an AI playtester team. Use Unity ML-Agents or the open-source Stable Baselines3 to train 3-5 agents with clearly different behaviors, let them repeatedly challenge the AI-generated levels, and log failure reasons. This step catches most of the playable-but-not-fun problems.

• Step 3: Run small-scale blind tests with real players. Each week, invite 5-10 external players to try the latest generated levels and collect subjective feedback with a simplified UEQ (User Experience Questionnaire). Focus on open questions like: Do you feel like you are exploring, or being pushed along? At which moment did you most want to quit?

• Step 4: Establish a playability regression testing mechanism. After every change to the generation algorithm, run a fixed test level set (including historical failure cases) to make sure the new version's generation quality has not regressed. This mechanism is analogous to unit testing in software engineering and is the guarantee for long-term iteration.

If you are building your own evaluation toolchain, start with Blender's geometry nodes to procedurally generate large numbers of level variants for testing; or use Kaedim to quickly convert 2D level sketches into 3D prototypes and cut iteration costs. For visual debugging in the level editor, Spline's browser collaboration features can also help teams quickly share and discuss generation results.

Trend Outlook: From Post-Hoc Evaluation to A Priori Design

Playability evaluation today is essentially post-hoc — generate first, test later. But as generative AI and reinforcement learning fuse more deeply, we are seeing the first light of a priori design. Google DeepMind's Dreamer family of algorithms, for example, can learn a dynamics model of the environment and rehearse the consequences of actions in imagination. In the future, AI generators may optimize playability metrics directly in latent space instead of relying on expensive sampling tests.

Another direction worth watching is human-machine collaborative evaluation. In January 2026, Ubisoft unveiled an experimental project whose AI system dynamically adjusts level difficulty based on players' real-time emotional feedback (via controller pressure sensors and facial cameras). Still at the lab stage, but it signals that playability evaluation will no longer be confined to development — it will extend across the game's entire lifecycle.

For developers, rather than waiting for the perfect evaluation tool to appear, embed playability thinking into your AI generation process now. Remember: AI is an intern with infinite stamina — it always gives you an answer, but you have to teach it what a good question is.