PerceptAI aims to give agents vision beyond browsers

A Hacker News user has posted PerceptAI, a tool meant to give AI agents access to screens outside the browser. The creator, Neerajj04, said the project was built because most agent tools only work on websites through the DOM, or Document Object Model, which is the structured page data browsers expose to software.

In the post, the creator argued that this leaves out a large share of real computer work. “75% of real computer work happens in desktop apps, legacy software, and tools with zero APIs,” the post said. In that setup, agents are “completely blind” to the software people actually use on desktops and in older business systems.

PerceptAI is described as a way to fill that gap. According to the post, it combines EasyOCR, which reads text from images, with Groq Vision, a vision model that can interpret screen content, and PyAutoGUI, a Python library that can control mouse and keyboard input on a computer.

The creator said the goal is to let a user give one plain-English instruction and have the system execute it autonomously. The post also says the tool includes self-healing and memory, though the Hacker News listing does not explain those features in detail.

The project was shared as a demo on Vercel, with a GitHub repository linked in the post. The Hacker News item had one point and one comment at the time of posting.

One commenter noted that Claude can already do similar computer control through its computer-use documentation, pointing to Anthropic’s docs. That comment suggests PerceptAI is entering a space where larger AI tools are already experimenting with screen-based automation.

The post does not say whether PerceptAI is a research prototype, an open-source tool, or a production-ready product. It does, however, make a clear pitch: agents should not be limited to browser pages when much of office work still lives in desktop software, legacy systems, and tools without APIs.