LeMario

LeMario is a Joint-Embedding Predictive Architecture (JEPA) world model trained specifically on Super Mario Bros. The project was developed from scratch to explore how a JEPA can learn world dynamics from both pixel data and action inputs within the context of a video game environment. Its main focus is on action-conditioned prediction and reward-free planning, using Super Mario Bros. as a testbed for these capabilities.

The model was evaluated on its ability to generalize to new, unseen episodes and to use action information to predict five-step future game states more accurately than strong baseline methods. In practice, LeMario demonstrated that it could move the Mario character toward nearby image-based goals and finish within a small margin of the target location, measured in pixels. This suggests that the model effectively predicts the immediate consequences of actions within the game world, even in the absence of explicit reward signals.

Despite its strengths in short-term prediction and local planning, the tool revealed limitations when tasked with more complex navigation, such as moving Mario over major obstacles or toward distant goals. These findings provide insights into the challenges of using JEPA models for longer-term planning and progress in dynamic environments.

LeMario is intended for those interested in the technical aspects of world modeling, action-conditioned prediction, and reinforcement learning, particularly within the context of video games. The project is documented as a technical walkthrough and postmortem, detailing the architecture, testing methods, and experimental results.