Most desktop automation tools work by hooking into system APIs, accessibility trees, or browser DOM. They parse HTML, call CDP endpoints, read element selectors. It works.
Source: [Dev.to](https://dev.to/mininglamp/vision-only-vs-api-based-desktop-agents-same-task-two-approaches-different-results-33a0)