- Amazon discouraged teams from relying solely on token usage or code generation as measurements of AI adoption, calling them "deeply flawed" metrics in an internal memo.
- Companies are grappling with rising AI costs and how to reliably measure whether the tools lead to a boost in productivity.
- Amazon in May shut down an internal leaderboard after discovering employees were tokenmaxxing.
In this article
Follow your favorite stocksCREATE FREE ACCOUNTAmazon CEO Andy Jassy speaks at a company event in New York on Feb. 26, 2025.Michael Nagle | Bloomberg | Getty ImagesAmazon laid out its AI usage guidelines for software engineers in a memo to staffers, and discouraged employees from turning to "tokenmaxxing" in their push to embrace artificial intelligence tools.
The memo, which was circulated internally in September and posted to the company's website on Tuesday, revises how the company defines the role of software development engineers and managers in the age of AI.
"We're just seeing the beginning of AI's effect on software engineering," the company wrote. "While we have a vision for what the future looks like, there's a lot that's still emerging. We are certain that whatever future best practices are, AI will be at the center of software engineering practice."
Under CEO Andy Jassy, Amazon has rapidly integrated AI across the company's sprawling business operations. Jassy has previously predicted that AI will "change the scope and speed" of day-to-day work, and potentially shrink its corporate workforce in the next few years through automation, though he's noted it will also create "other jobs."
As employees rush to adopt AI tools, companies have grappled with the technology's surging costs and how to reliably measure whether the technology leads to a productivity boost or other valuable returns on investment.
Companies have recently cracked down on tokenmaxxing, in which developers use AI as much as possible with little regard for output. Some companies have set limits on employees' usage of tokens, or the units of data that are processed and generated by AI models, in a bid to slow rising AI bills. Others have switched to cheaper, open-source models for certain tasks.
- A guide to some of the biggest risks in Anthropic's IPO prospectus
- Nvidia, AMD to invest $4 billion in Meta as tech giants deepen AI partnership
- Apple responds to user reports of broken iPhone Duo screens on new foldable phone
- Anthropic's IPO plans: 900% revenue retention, but $80 billion in first-half losses
An Amazon spokesperson said the company hasn't instituted caps on employees' token usage. Amazon in May shut down an internal leaderboard after discovering employees were tokenmaxxing to climb the ranks.
In the memo posted Tuesday, Amazon said metrics like token usage, the rate of adoption of AI tools, and code changes or reviews remain "deeply flawed" ways of measuring teams or organizations' AI usage, adding that they can lead to higher costs and too much complexity.
It stopped short of telling staffers to ignore those metrics altogether, however. Token usage and code generation should be considered "with care and judgment" alongside other data, Amazon said.
"We believe that it is best to measure, and be measured by, long-term outcomes," the company wrote.
Marc Brooker, a vice president and distinguished engineer at Amazon Web Services, told CNBC in an interview that the company has looked to metrics such as "feature velocity" to measure how AI is improving engineer productivity.
The metric looks at things like, "how many launches are we doing, how many successful changes are we making in production," he said.
"Broadly, we think about this as a return on investment," Brooker said. "If there is an important problem to solve for our customers, important problem to solve for our business, we're going to invest tools, time, tokens, human effort, etc. into those."
He added, "What we don't want to see is wasted effort where we are going and spending those tokens solving problems that don't matter and sometimes it's hard to know, so we do need to have some appetite for experimentation."
Amazon also discouraged staffers from using AI when it doesn't make sense for a given task.
"If your work does not benefit from current AI tooling, that is a finding, not a failing," Amazon said in the memo.
The company added, "An engineer who can explain where AI does not yet help their work is demonstrating the judgement we are asking for."
watch nowVIDEO4:5504:55Dan Ives on big tech's earning test: It all comes down to demand and what we see in CapExPower LunchChoose CNBC as your preferred source on Google and never miss a moment from the most trusted name in business news.