How do you build an expert-grade research QA benchmark without hand-authoring a single question?Mine the survey articles. Their section structure already encodes what a field asks and what a complete answer must contain, so a new benchmark distills 21K queries and grading rubrics from surveys across 75 fields. It doubles as a stress test: even the best system tops out at 75% rubric coverage, fully addressing under 11% of needed citations.https://benjaminhan.net/posts/20260625-researchqa/?utm_source=mastodon&utm_medium=social#LLMs #Evaluation #AI
Related
#llm #ai #movies #tvI bet Credits software in movies and TV is getting vibe coded. It's almost the perfect case, use-onc...
#llm #ai #movies #tvI bet Credits software in movies and TV is getting vibe coded. It's almost the perfect case, use-once, disposable.
Hj em dia todo mundo fala de #LLM, de como programar, automatizar e resolver as coisas, como fazer o bom uso de IA no de...
Hj em dia todo mundo fala de #LLM, de como programar, automatizar e resolver as coisas, como fazer o bom uso de IA no desenvolvimento de software. Sabe qual a real? É papo de “sexo...
Lethal autonomous weapons powered by artificial intelligence are set to revolutionize warfare but there is no proper reg...
Lethal autonomous weapons powered by artificial intelligence are set to revolutionize warfare but there is no proper regulatory or legal framework governing the new technology. htt...