- The next time someone's supercharged AI agent attempts to make a phone call to negotiate a cable bill, it could be an actual human who haggles with customer service.
- Foundation models developed by companies like Meta and OpenAI still can't reliably take actions across the web, particularly those involving phone calls.
- Marissa Mayer's Dazzle AI and other personal AI agent startups like Duckbill and Wajo AI are all emphasizing the role human workers play in accomplishing tasks when software fails.
The next time someone's supercharged AI agent attempts to make a phone call to negotiate a cable bill, it could be an actual human who haggles with customer service rather than the software.
That's because AI agents, powered by the bleeding-edge foundation models developed by companies like Meta and OpenAI, still can't reliably take actions across the web, particularly those involving phone calls. Former Yahoo CEO Marissa Mayer, who now helms agent startup Dazzle AI, says current top-tier AI models excel at surfacing online home renovation services for consumers, but might blow it when they need to book and schedule a rooftop upgrade.
"We're relying on the frontier models to give us good suggestions and a good candidate list, but we're checking that over to make sure that we're really delivering the type of service that we can stand behind," Mayer said, referring to how Dazzle funnels certain tasks to human contractors.
Digital assistants driven by artificial intelligence are currently gaining renewed interest thanks to Meta, OpenAI and others that are pitching the technology as capable of tasks like scouting the internet for shoes.
VIDEO8:4808:48Watch CNBC's full interview with Dazzle AI CEO Marissa MayerMeta's Muse, OpenAI's Dots and a host of other agents can accomplish a multitude of web-browsing feats, but it's still early days for the technology, and it often hits digital roadblocks. Those barriers typically involve tasks that would have an AI agent make a phone call, schedule a doctor's appointment on someone's behalf, or reserve a couch scrubbing from a local cleaning service, said Meghan Joyce, CEO of digital agent startup Duckbill.
For tasks that require real-world judgment, Joyce said humans still trump AI, and companies building consumer-grade digital assistants will need to patch in real people to take over. That's why Duckbill recently debuted an application programming interface, or API, intended for companies building personal agents to connect with human contract workers.
"We work on a model that is not dissimilar from Uber, where we boil tasks down into atomic bite-sized pieces," Joyce said, referencing the ride-hailing company where she previously worked.
Companies making agents can use Duckbill's API to delegate the tasks that the agent is likely to fumble. Duckbill translates those tasks into bite-sized pieces that the average adult can do consistently and reliably.
"And then we track what happens," Joyce said.
Confused agents
Mayer said that AI models can still lack common sense. She said a Dazzle AI super user attempted to have the technology simultaneously handle a series of home renovation projects involving a roof replacement, a broken window and the purchase of a glass tabletop.
But the AI agent, Mayer said, confused window glass with tabletop glass, resulting in the digital assistant "trying to figure out if he could make the tabletop fit in the window."
The mishap didn't last long, Mayer said, because Dazzle's technology flagged the task and outsourced it to a person to complete.
"Within a few seconds, the human had things sorted out. Whereas an agent would have burned a ridiculous number of tokens," Mayer said. "Or they may never have gotten to a good solution."
Shivani Poddar, CEO of personal AI agent startup Wajo AI, is also incorporating human workers to handle phone calls and even CAPTCHA tests intended to fend off bots from sites.
"I think AI plus humans, collaborative AI, is what you need to actually get the job done," Poddar said, emphasizing that people will stop using the technology if agents can't complete tasks.
watch nowVIDEO4:5604:56Would like more color from Meta on Muse AI's performance, says Jefferies' Brent ThillClosing Bell: OvertimeChatbot and agent interactions can be ultra personal and users have good reason to be concerned about the humans stepping in, particularly if the company didn't tell them how it works.
Meta employees previously raised privacy concerns about the company's internal test using human contractors to handle phone calls on behalf of its Muse AI agent. A Meta spokesperson said in a statement at the time that this kind of testing is "core to the product development process."
Meta currently doesn't use humans to complete users' Muse requests.
Duckbill's Joyce said her startup does "full background checking" into the 1099 contractors it uses, comparing them to drivers for Uber and DoorDash, "people who live here in the United States and want a source of additional income." Duckbill also uses contract workers from businesses that typically offer contact center labor to companies.
Mayer said Dazzle conducts background checks and looks for contractors with the same qualities someone might look for in a human personal assistant.
"We generally are screening for people who we think are extremely discreet and have good taste and good judgment, both in terms of how they interact with our users, but just the ability to kind of forget and compartmentalize that information afterwards," Mayer said. "I understand it's a point of privilege, but I've had personal assistants for some time, and I know how I want them, and I know how I guide them."
Mayer said that while foundation models are still improving, the market is in a "transitionary period" until the technology can perform more involved web tasks and phone calls as efficiently as people.
"We're not even trying to hide it, we think the human in the loop is a benefit," Mayer said.
WATCH: Who's liable when AI agents break the rules?
watch nowVIDEO11:3411:34Who's liable when AI agents break the rules?CNBC WorkChoose CNBC as your preferred source on Google and never miss a moment from the most trusted name in business news.Chart: Anthropic's cash pile towers over nearly every U.S. companyThird-party evaluators found vulnerabilities in OpenAI monitoring system during test pilotApple and Meta team up to make Muse AI model run on Mac computers Read More
