Get Involved With This Work
Executive Summary
AI investment isn’t slowing down and much of it is informed by performance on AI benchmarks. Benchmarks are metrics used to evaluate and compare the performance of AI systems that shape the incentives of the field by defining what counts as “progress.” They act as guides for AI developers on which capabilities are worth pursuing, and they are used by developers to measure improvements in their models.
Unfortunately, the improvements being measured are often disconnected from what people need and are being developed at a time where there is growing distrust in AI. Artists feel sidelined, educators worry about job security and classroom impacts, and activists raise concerns about cultural loss. While the contexts vary, the sentiment is consistent: many see AI as something being done to them, not with them or even for them.
If we want AI to serve the public good, we need new benchmarks that measure what actually matters: progress on issues like hunger, clean water, and sustainability.
Aspen Digital is leveraging AI benchmarking as a new way to include community voice in AI development. Many AI developers want their technologies to serve a social benefit, but are not versed in the needs and priorities of the public. Our approach is grounded in a simple idea: make it as easy as possible to do the right thing by translating public needs into the language that AI developers already understand. By defining new benchmarks that measure success in terms of impact on the United Nations Sustainable Development Goals (SDGs), we can realign incentives, encouraging researchers and engineers to build systems that tackle real-world problems and deliver tangible public value.
What we’ve found:
- The SDGs most prioritized by the global public are Zero Hunger, Clean Water & Sanitation, No Poverty, and Good Health & Well-being, with food security being the top priority.
- Unfortunately, food security is one of the most at-risk SDGs with many of its subgoals stagnant or in regression.
- On our current trajectory, a majority of experts do not believe AI will deliver progress on the SDGs anytime soon, with 42% of experts from lower- and lower-middle income countries, where food security is a more pressing public issue, saying it will never.
- But evidence suggests that there is opportunity for AI to be helpful on over half of the SDGs, including for food security.
- There is little alignment between where AI development today is focused and where it is needed, with only 3.6% of over thirteen thousand SDG-focused AI projects targeting food security.
Following a series of preliminary convenings, Aspen Digital will survey global food security experts to surface the biggest challenges in the field. Then, working with both community representatives and AI researchers, we’ll translate these challenges into benchmarks that reflect real-world needs and can guide future development.
This project is about moving beyond vague hopes that AI will help the world, toward concrete tools that will deliver the better AI futures we deserve.