Ghost OS

Ghost OS is an open-source framework that enables AI agents to interact with and operate any application on macOS by leveraging structured data from the system's accessibility tree. Designed to move beyond the limitations of agents confined to terminals or browser-based environments, Ghost OS allows agents to perceive and control native macOS apps, facilitating actions such as clicking buttons, filling forms, sending emails, organizing files, and executing cross-app workflows. The tool is intended for developers and teams building AI agents that require robust, reliable computer-use capabilities on Apple Silicon devices running macOS 14 or later.

The core functionality of Ghost OS centers on reading the macOS accessibility tree, which provides detailed, structured information about UI elements like buttons, text fields, and labels. This approach enables rapid and precise identification of interface components, avoiding the need for slow and fragile screenshot-based inference. When the accessibility tree does not provide sufficient detail, Ghost OS cascades through additional fallback methods: querying the Chrome DevTools Protocol for web app DOM data, invoking a local vision model (ShowUI-2B) running on Apple Silicon via MLX for visual grounding, and ultimately generating synthetic input events at the system level. All processing occurs locally on the user's Mac, ensuring privacy and data security.

A distinctive feature of Ghost OS is its self-learning recipe system. Users can demonstrate a workflow—such as sending an email—while Ghost OS records each action with full context via the accessibility API and CGEvent tap. An AI model (such as Claude) then synthesizes a parameterized, transparent recipe in JSON format, which can be verified, reused, and shared among team members. This eliminates repeated reasoning costs and enables efficient, repeatable automation of complex tasks across any supported application.

Ghost OS communicates through MCP, an open protocol for AI tool interfaces, and includes a toolkit of 29 MCP tools for perceiving, acting, navigating, learning, and replaying actions. Installation is streamlined, requiring only a Homebrew command and setup, with no need for Docker or virtual machines. 0, and all recipes and models are stored locally, further emphasizing user control and privacy.