Anthropic’s New AI Model Takes Control of Your Computer

Anthropic says it is teaching its Claude AI model to perform common computing tasks based on information. In demonstration videos, the model is shown controlling a computer cursor to perform out-of-town research, searching the web for places to visit near the user’s home and adding trips to their desktop calendar.

The functionality is only available to developers today, and it’s not clear what the price looks like or how well the technology works. Anthropic says in a tweet about the new capabilities that while testing its model, Claude strayed from coding and started searching on Google for images of Yellowstone National Park. So, yes … there are still issues to be addressed.

From a technical standpoint, Anthropic says Claude can control the computer by taking screenshots and sending it back to the model, reading what’s on the screen, including the distance between the cursor position and the button he needs to press, and the commands that come back. to continue the work.

Anthropic, which is backed by the likes of Amazon and Google, says Claude is “the first AI model to offer computing in public beta.”

It is not yet clear which automated computing may be useful in practice. Anthropic suggests that it can be used for repetitive tasks or open research. If anyone figures out how to use this new function, the /r/overworked community on Reddit will likely be the first. At least it could be a new jiggler mouse for Wells Fargo employees. Or maybe you can use it to go through your social media accounts and delete all your old posts without needing a third-party tool to do it. Things that aren’t mission critical or require factual accuracy.

Although there has been a lot of hype in the AI ​​space, and companies have spent billions of dollars developing AI chatbots, most of the money in the space is still generated by companies like Nvidia that provide GPUs to these AI companies. Anthropic raised over $7 billion last year alone.

The latest buzzword tech companies want to sell is technology for “agents,” or autonomous bots that are said to be able to complete tasks on their own. Microsoft on Monday announced the ability to create autonomous agents with Copilot that can do “everything from expediting lead generation and processing sales orders to automating your supply chain.”

Salesforce CEO Marc Benioff carelessly incorrectly called Microsoft’s product “Clippy 2.0”—although, of course, he was saying this while promoting Salesforce’s competing AI products. Salesforce wants to enable its customers to create their own custom agents that can perform tasks such as answering customer support emails or prospecting for new customers.

White collar workers have yet to adopt chatbots like ChatGPT or Claude. Reception of Microsoft’s Copilot assistant has been lukewarm, rather than a small part of Microsoft 365 customers spend $30 per month to access AI tools. But Microsoft is reorienting its entire company around this AI explosion, and it needs to show investors a return on that investment. So, agents are a new thing.

The main problem, as always, is that AI chatbots like ChatGPT and Google’s Gemini produce a lot of output that is factually inaccurate, of poor quality, or reads like it was clearly not written by a human. The time it takes to edit and clean up the bot’s output almost certainly throws off any efficiency they produced in the first place. It’s fine to go down rabbit holes in your spare time, but in the workplace it’s unacceptable to produce work full of mistakes. I would be nervous about setting up Claude to use my email, only to have it send human jargon back in response, or wrap up some work that I have to go back and fix. The fact that OpenAI itself admits most of its active users are students says it all.

Anthropic in a tweet about the new functionality itself acknowledges that computer use should be limited to “low-risk activities.”