No items found.

Humanity’s Last Exam

Center for AI Safety, Scale AI and HLE Contributors Consortium

Center for AI Safety, Scale AI and HLE Contributors Consortium

“Himalayas”, Eastern Himalaya, southwestern China
03

Abstract

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

03

Runs

Cite this
TextBibTeXCopy
Center for AI Safety, Scale AI and HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature 649(8099), 1139–1146 (2026).
@Article{Phan2026, author="{Center for AI Safety} and Phan, Long and Gatti, Alice and Li, Nathaniel and Khoja, Adam and Kim, Ryan and Ren, Richard and Hausenloy, Jason and Zhang, Oliver and Mazeika, Mantas and Hendrycks, Dan and {Scale AI} and Han, Ziwen and Hu, Josephina and Zhang, Hugh and Zhang, Chen Bo Calvin and Shaaban, Mohamed and Ling, John and Shi, Sean and Choi, Michael and Agrawal, Anish and Chopra, Arnav and Nattanmai, Aakaash and McKellips, Gordon and Cheraku, Anish and Suhail, Asim and Luo, Ethan and Deng, Marvin and Luo, Jason and Zhang, Ashley and Jindel, Kavin and Paek, Jay and Halevy, Kasper and Baranov, Allen and Liu, Michael and Avadhanam, Advaith and Zhang, David and Cheng, Vincent and Ma, Brad and Fu, Evan and Do, Liam and Lass, Joshua and Yang, Hubert and Sunkari, Surya and Bharath, Vishruth and Ai, Violet and Leung, James and Agrawal, Rishit and Zhou, Alan and Chen, Kevin and Kalpathi, Tejas and Xu, Ziqi and Wang, Gavin and Xiao, Tyler and Maung, Erik and Lee, Sam and Yang, Ryan and Yue, Roy and Zhao, Ben and Yoon, Julia and Sun, Xiangwan and Singh, Aryan and Peng, Clark and Osbey, Tyler and Wang, Taozhi and Echeazu, Daryl and Wu, Timothy and Patel, Spandan and Kulkarni, Vidhi and Sundarapandiyan, Vijaykaarti and Le, Andrew and Nasim, Zafir and Yalam, Srikar and Kasamsetty, Ritesh and Samal, Soham and Sun, David and Shah, Nihar and Saha, Abhijeet and Zhang, Alex and Nguyen, Leon and Nagumalli, Laasya and Wang, Kaixin and Wu, Aidan and Telluri, Anwith and Yue, Summer and Wang, Alexandr and {HLE Contributors Consortium} and Dodonov, Dmitry and Nguyen, Tung and Lee, Jaeho and Anderson, Daron and Doroshenko, Mikhail and Stokes, Alun Cennyth and Mahmood, Mobeen and Pokutnyi, Oleksandr and Iskra, Oleg and Wang, Jessica P. and Levin, John-Clark and Kazakov, Mstyslav and Feng, Fiona and Feng, Steven Y. and Zhao, Haoran and Yu, Michael and Gangal, Varun and Zou, Chelsea and Wang, Zihan and Popov, Serguei and Gerbicz, Robert and Galgon, Geoff and Schmitt, Johannes and Yeadon, Will and Lee, Yongki and Sauers, Scott and Sanchez, Alvaro and Giska, Fabian and Roth, Marc and Riis, S{\o}ren and Utpala, Saiteja and Burns, Noah and Goshu, Gashaw M. and Naiya, Mohinder Maheshbhai and Agu, Chidozie and Giboney, Zachary and Cheatom, Antrell and Fournier-Facio, Francesco and Crowson, Sarah-Jane and Finke, Lennart and Cheng, Zerui and Zampese, Jennifer and Hoerr, Ryan G. and Nandor, Mark and Park, Hyunwoo and Gehrunger, Tim and Cai, Jiaqi and McCarty, Ben and Garretson, Alexis C. and Taylor, Edwin and Sileo, Damien and Ren, Qiuyu and Qazi, Usman and Li, Lianghui and Nam, Jungbae and Wydallis, John B. and Arkhipov, Pavel and Shi, Jack Wei Lun and Bacho, Aras and Willcocks, Chris G. and Cao, Hangrui and Motwani, Sumeet and de Oliveira Santos, Emily and Veith, Johannes and Vendrow, Edward and Cojoc, Doru and Zenitani, Kengo and Robinson, Joshua and Tang, Longke and Li, Yuqi and Vendrow, Joshua and Fraga, Natanael Wildner and Kuchkin, Vladyslav and Maksimov, Andrey Pupasov and Marion, Pierre and Efremov, Denis and Lynch, Jayson and Liang, Kaiqu and Mikov, Aleksandar and Gritsevskiy, Andrew and Guillod, Julien and Demir, G{\"o}zdenur and Martinez, Dakotah and Pageler, Ben and Zhou, Kevin and Soori, Saeed and Press, Ori and Tang, Henry and Rissone, Paolo and Green, Sean R. and Br{\"u}ssel, Lina and Twayana, Moon and Dieuleveut, Aymeric and Imperial, Joseph Marvin and Prabhu, Ameya and Yang, Jinzhou and Crispino, Nick and Rao, Arun and Zvonkine, Dimitri and Loiseau, Gabriel and Kalinin, Mikhail and Lukas, Marco and Manolescu, Ciprian and Stambaugh, Nate and Mishra, Subrata and Hogg, Tad and Bosio, Carlo and Coppola, Brian P. and Salazar, Julian and Jin, Jaehyeok and Sayous, Rafael and Ivanov, Stefan and Schwaller, Philippe and Senthilkumar, Shaipranesh and Bran, Andres M. and Algaba, Andres and Van den Houte, Kelsey and Van Der Sypt, Lynn and Verbeken, Brecht and Noever, David and Kopylov, Alexei and Myklebust, Benjamin and Li, Bikun and Schut, Lisa and Zheltonozhskii, Evgenii and Yuan, Qiaochu and Lim, Derek and Stanley, Richard and Yang, Tong and Maar, John and Wykowski, Julian and Oller, Mart and Sahu, Anmol and Ardito, Cesare Giulio and Hu, Yuzheng and Kamdoum, Ariel Ghislain Kemogne and Jin, Alvin and Vilchis, Tobias Garcia and Zu, Yuexuan and Lackner, Martin and Koppel, James and Sun, Gongbo and Antonenko, Daniil S. and Chern, Steffi and Zhao, Bingchen and Arsene, Pierrot and Cavanagh, Joseph M. and Li, Daofeng and Shen, Jiawei and Crisostomi, Donato and Zhang, Wenjin and Dehghan, Ali and Ivanov, Sergey and Perrella, David and Kaparov, Nurdin and Zang, Allen and Sucholutsky, Ilia and Kharlamova, Arina and Orel, Daniil and Poritski, Vladislav and Ben-David, Shalev and Berger, Zachary and Whitfill, Parker and Foster, Michael and Munro, Daniel and Ho, Linh and Sivarajan, Shankar and Hava, Dan Bar and Kuchkin, Aleksey and Holmes, David and Rodriguez-Romero, Alexandra and Sommerhage, Frank and Zhang, Anji and Moat, Richard and Schneider, Keith and Kazibwe, Zakayo and Clarke, Don and Kim, Dae Hyun and Dias, Felipe Meneguitti and Fish, Sara and Elser, Veit and Kreiman, Tobias and Vilchis, Victor Efren Guadarrama and Klose, Immo and Anantheswaran, Ujjwala and Zweiger, Adam and Rawal, Kaivalya and Li, Jeffery and Nguyen, Jeremy and Daans, Nicolas and Heidinger, Haline and Radionov, Maksim and Rozho{\v{n}}, V{\'a}clav and Ginis, Vincent and Stump, Christian and Cohen, Niv and Po{\'{s}}wiata, Rafa{\l} and Tkadlec, Josef and Goldfarb, Alan and Wang, Chenguang and Padlewski, Piotr and Barzowski, Stanislaw and Montgomery, Kyle and Stendall, Ryan and Tucker-Foltz, Jamie and Stade, Jack and Rogers, T. Ryan and Goertzen, Tom and Grabb, Declan and Shukla, Abhishek and Givr{\'e}, Alan and Ambay, John Arnold and Sen, Archan", title="A benchmark of expert-level academic questions to assess AI capabilities", journal="Nature", year="2026", month="Jan", day="01", volume="649", number="8099", pages="1139--1146", abstract="Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90{\%} accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.", issn="1476-4687", doi="10.1038/s41586-025-09962-4", url="https://doi.org/10.1038/s41586-025-09962-4" }
Nature 2026
January 24, 2025
Nature
Main
Journal, vol. 649 (2026)
humanitys-last-exam
More research
All research →
“Ganges River Delta”, Bay of Bengal“Ganges River Delta”, Bay of Bengal
NeurIPS 2026 Workshop
meta

GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

October 15, 2026
NeurIPS 2026 workshop
Workshop
globebench
NeurIPS 2026 Workshop
“Earth’s Aquarium”, Bering Sea“Earth’s Aquarium”, Bering Sea
NeurIPS 2026
meta

Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

May 7, 2026
NeurIPS 2026
Main
world-modeling-for-agentic-rl
NeurIPS 2026
From the blog
All posts →
No items found.