Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.
Center for AI Safety, Scale AI and HLE Contributors Consortium. A benchmark of expert-level academic questions to assess AI capabilities. Nature 649(8099), 1139–1146 (2026).
@Article{Phan2026,
author="{Center for AI Safety}
and Phan, Long
and Gatti, Alice
and Li, Nathaniel
and Khoja, Adam
and Kim, Ryan
and Ren, Richard
and Hausenloy, Jason
and Zhang, Oliver
and Mazeika, Mantas
and Hendrycks, Dan
and {Scale AI}
and Han, Ziwen
and Hu, Josephina
and Zhang, Hugh
and Zhang, Chen Bo Calvin
and Shaaban, Mohamed
and Ling, John
and Shi, Sean
and Choi, Michael
and Agrawal, Anish
and Chopra, Arnav
and Nattanmai, Aakaash
and McKellips, Gordon
and Cheraku, Anish
and Suhail, Asim
and Luo, Ethan
and Deng, Marvin
and Luo, Jason
and Zhang, Ashley
and Jindel, Kavin
and Paek, Jay
and Halevy, Kasper
and Baranov, Allen
and Liu, Michael
and Avadhanam, Advaith
and Zhang, David
and Cheng, Vincent
and Ma, Brad
and Fu, Evan
and Do, Liam
and Lass, Joshua
and Yang, Hubert
and Sunkari, Surya
and Bharath, Vishruth
and Ai, Violet
and Leung, James
and Agrawal, Rishit
and Zhou, Alan
and Chen, Kevin
and Kalpathi, Tejas
and Xu, Ziqi
and Wang, Gavin
and Xiao, Tyler
and Maung, Erik
and Lee, Sam
and Yang, Ryan
and Yue, Roy
and Zhao, Ben
and Yoon, Julia
and Sun, Xiangwan
and Singh, Aryan
and Peng, Clark
and Osbey, Tyler
and Wang, Taozhi
and Echeazu, Daryl
and Wu, Timothy
and Patel, Spandan
and Kulkarni, Vidhi
and Sundarapandiyan, Vijaykaarti
and Le, Andrew
and Nasim, Zafir
and Yalam, Srikar
and Kasamsetty, Ritesh
and Samal, Soham
and Sun, David
and Shah, Nihar
and Saha, Abhijeet
and Zhang, Alex
and Nguyen, Leon
and Nagumalli, Laasya
and Wang, Kaixin
and Wu, Aidan
and Telluri, Anwith
and Yue, Summer
and Wang, Alexandr
and {HLE Contributors Consortium}
and Dodonov, Dmitry
and Nguyen, Tung
and Lee, Jaeho
and Anderson, Daron
and Doroshenko, Mikhail
and Stokes, Alun Cennyth
and Mahmood, Mobeen
and Pokutnyi, Oleksandr
and Iskra, Oleg
and Wang, Jessica P.
and Levin, John-Clark
and Kazakov, Mstyslav
and Feng, Fiona
and Feng, Steven Y.
and Zhao, Haoran
and Yu, Michael
and Gangal, Varun
and Zou, Chelsea
and Wang, Zihan
and Popov, Serguei
and Gerbicz, Robert
and Galgon, Geoff
and Schmitt, Johannes
and Yeadon, Will
and Lee, Yongki
and Sauers, Scott
and Sanchez, Alvaro
and Giska, Fabian
and Roth, Marc
and Riis, S{\o}ren
and Utpala, Saiteja
and Burns, Noah
and Goshu, Gashaw M.
and Naiya, Mohinder Maheshbhai
and Agu, Chidozie
and Giboney, Zachary
and Cheatom, Antrell
and Fournier-Facio, Francesco
and Crowson, Sarah-Jane
and Finke, Lennart
and Cheng, Zerui
and Zampese, Jennifer
and Hoerr, Ryan G.
and Nandor, Mark
and Park, Hyunwoo
and Gehrunger, Tim
and Cai, Jiaqi
and McCarty, Ben
and Garretson, Alexis C.
and Taylor, Edwin
and Sileo, Damien
and Ren, Qiuyu
and Qazi, Usman
and Li, Lianghui
and Nam, Jungbae
and Wydallis, John B.
and Arkhipov, Pavel
and Shi, Jack Wei Lun
and Bacho, Aras
and Willcocks, Chris G.
and Cao, Hangrui
and Motwani, Sumeet
and de Oliveira Santos, Emily
and Veith, Johannes
and Vendrow, Edward
and Cojoc, Doru
and Zenitani, Kengo
and Robinson, Joshua
and Tang, Longke
and Li, Yuqi
and Vendrow, Joshua
and Fraga, Natanael Wildner
and Kuchkin, Vladyslav
and Maksimov, Andrey Pupasov
and Marion, Pierre
and Efremov, Denis
and Lynch, Jayson
and Liang, Kaiqu
and Mikov, Aleksandar
and Gritsevskiy, Andrew
and Guillod, Julien
and Demir, G{\"o}zdenur
and Martinez, Dakotah
and Pageler, Ben
and Zhou, Kevin
and Soori, Saeed
and Press, Ori
and Tang, Henry
and Rissone, Paolo
and Green, Sean R.
and Br{\"u}ssel, Lina
and Twayana, Moon
and Dieuleveut, Aymeric
and Imperial, Joseph Marvin
and Prabhu, Ameya
and Yang, Jinzhou
and Crispino, Nick
and Rao, Arun
and Zvonkine, Dimitri
and Loiseau, Gabriel
and Kalinin, Mikhail
and Lukas, Marco
and Manolescu, Ciprian
and Stambaugh, Nate
and Mishra, Subrata
and Hogg, Tad
and Bosio, Carlo
and Coppola, Brian P.
and Salazar, Julian
and Jin, Jaehyeok
and Sayous, Rafael
and Ivanov, Stefan
and Schwaller, Philippe
and Senthilkumar, Shaipranesh
and Bran, Andres M.
and Algaba, Andres
and Van den Houte, Kelsey
and Van Der Sypt, Lynn
and Verbeken, Brecht
and Noever, David
and Kopylov, Alexei
and Myklebust, Benjamin
and Li, Bikun
and Schut, Lisa
and Zheltonozhskii, Evgenii
and Yuan, Qiaochu
and Lim, Derek
and Stanley, Richard
and Yang, Tong
and Maar, John
and Wykowski, Julian
and Oller, Mart
and Sahu, Anmol
and Ardito, Cesare Giulio
and Hu, Yuzheng
and Kamdoum, Ariel Ghislain Kemogne
and Jin, Alvin
and Vilchis, Tobias Garcia
and Zu, Yuexuan
and Lackner, Martin
and Koppel, James
and Sun, Gongbo
and Antonenko, Daniil S.
and Chern, Steffi
and Zhao, Bingchen
and Arsene, Pierrot
and Cavanagh, Joseph M.
and Li, Daofeng
and Shen, Jiawei
and Crisostomi, Donato
and Zhang, Wenjin
and Dehghan, Ali
and Ivanov, Sergey
and Perrella, David
and Kaparov, Nurdin
and Zang, Allen
and Sucholutsky, Ilia
and Kharlamova, Arina
and Orel, Daniil
and Poritski, Vladislav
and Ben-David, Shalev
and Berger, Zachary
and Whitfill, Parker
and Foster, Michael
and Munro, Daniel
and Ho, Linh
and Sivarajan, Shankar
and Hava, Dan Bar
and Kuchkin, Aleksey
and Holmes, David
and Rodriguez-Romero, Alexandra
and Sommerhage, Frank
and Zhang, Anji
and Moat, Richard
and Schneider, Keith
and Kazibwe, Zakayo
and Clarke, Don
and Kim, Dae Hyun
and Dias, Felipe Meneguitti
and Fish, Sara
and Elser, Veit
and Kreiman, Tobias
and Vilchis, Victor Efren Guadarrama
and Klose, Immo
and Anantheswaran, Ujjwala
and Zweiger, Adam
and Rawal, Kaivalya
and Li, Jeffery
and Nguyen, Jeremy
and Daans, Nicolas
and Heidinger, Haline
and Radionov, Maksim
and Rozho{\v{n}}, V{\'a}clav
and Ginis, Vincent
and Stump, Christian
and Cohen, Niv
and Po{\'{s}}wiata, Rafa{\l}
and Tkadlec, Josef
and Goldfarb, Alan
and Wang, Chenguang
and Padlewski, Piotr
and Barzowski, Stanislaw
and Montgomery, Kyle
and Stendall, Ryan
and Tucker-Foltz, Jamie
and Stade, Jack
and Rogers, T. Ryan
and Goertzen, Tom
and Grabb, Declan
and Shukla, Abhishek
and Givr{\'e}, Alan
and Ambay, John Arnold
and Sen, Archan",
title="A benchmark of expert-level academic questions to assess AI capabilities",
journal="Nature",
year="2026",
month="Jan",
day="01",
volume="649",
number="8099",
pages="1139--1146",
abstract="Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90{\%} accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.",
issn="1476-4687",
doi="10.1038/s41586-025-09962-4",
url="https://doi.org/10.1038/s41586-025-09962-4"
}
Nature 2026
January 24, 2025
Nature
Main
Journal, vol. 649 (2026)
humanitys-last-exam