Growing Regulations, Skyrocketing Investment and “Seriously Lacking” Evaluations: Top Takeaways from Stanford University’s 2024 AI Index Report

AI has begun to surpass human performance on several benchmarks, including image classification, visual reasoning, and English understanding. 

However, standardised evaluations are lacking, and a growing awareness of AI’s potential impact is growing along with a mounting pile of strict regulations.

This is according to the Stanford University 2024 AI Index Report – an annual barometer for artificial intelligence. Recognised globally as one of the most credible and authoritative sources, it comes at a time when AI’s influence on society has never been bigger. 

“As AI rapidly evolves, the AI Index aims to help the AI community, policymakers, business leaders, journalists, and the general public navigate this complex landscape,” said Ray Perrault and Jack Clark, co-directors of the report.

“It provides ongoing, objective snapshots tracking several key areas: technical progress in AI capabilities, the community and investments driving AI development and deployment, public opinion on current and potential future impacts, and policy measures taken to stimulate AI innovation while managing its risks and challenges. By comprehensively monitoring the AI ecosystem, the Index serves as an important resource for understanding this transformative technological force.”

Some of the report’s top findings include: 

1. AI outperforms humans in several areas but not all
   Artificial Intelligence has exceeded human capabilities in a variety of tasks, such as image classification and understanding English, yet it falls short in more intricate challenges like high-level maths, visual commonsense reasoning, and strategic planning.
2. Industry remains at the forefront of pioneering AI research
   In 2023, the commercial sector released 51 noteworthy machine-learning models, compared to just 15 from academic institutions. Furthermore, collaborations between industry and academia produced 21 significant models, reaching a new peak.
3. Frontier models become significantly more costly
   The AI Index has reported that the costs for training cutting-edge AI models have soared to new heights. Notably, the estimated expenditure to train OpenAI’s GPT-4 was $78 million, whilst Google’s Gemini Ultra required $191 million in computing costs.
4. The United States continues to be the primary source of top AI models
   In 2023, the US led with 61 distinguished AI models, significantly more than the European Union’s 21 and China’s 15.

1. A glaring deficiency in standardised evaluations for LLM responsibility
   Research from the AI Index highlights a considerable gap in standardisation for responsible AI reporting. Major developers like OpenAI, Google, and Anthropic predominantly utilise different benchmarks for testing their models, which hinders the ability to uniformly assess risks and limitations.
2. Investment in generative AI soars
   Despite a downturn in overall private AI investments last year, funding for generative AI rocketed, increasing almost eightfold from 2022 to total $25.2 billion. Key industry players such as OpenAI, Anthropic, Hugging Face, and Inflection achieved notable investment rounds.
3. AI demonstrably enhances workforce productivity and quality of work
   Studies from 2023 suggest that AI boosts worker efficiency and enhances the quality of their work. These findings also indicate AI’s role in narrowing the skills gap between different levels of workers, although some research warns of reduced performance if AI is used without adequate oversight.
4. AI-driven acceleration in scientific progress
   AI’s role in propelling scientific innovation deepened in 2023 with the development of applications like AlphaDev for more efficient algorithmic sorting and GNoME, which aids in materials discovery.
5. A marked rise in AI regulations in the United StatesThe past year saw a substantial increase in AI-related regulations, with the count rising from a single regulation in 2016 to 25 in 2023, reflecting a growth of 56.3% over just one year.
6. Global awareness and apprehension regarding AI’s impact grow
   An Ipsos survey indicates that over the past year, awareness of AI’s significant potential impact in the near future has risen from 60% to 66%. Additionally, 52% of respondents expressed increased anxiety about AI, a rise of 13 percentage points from 2022. In the US, Pew data shows that 52% of Americans feel more concerned than excited about AI, up from 38% the previous year.

You can access the full report HERE.

—

Be a catalyst for salary transparency. Gain a better understanding of your value. Keep salaries competitive. Take part in Harnham’s 2024-2025 Data & AI Salary Survey HERE.