Germany's Soofi consortium withdrew benchmark scores after discovering its training data contained paraphrased questions from the GPQA evaluation set. The 11.1-point gain on GPQA-Diamond can no longer stand as evidence of model capability. A researcher spotted the contamination; the team confirmed it within a week. https://www.implicator.ai/soofi-gpqa-contamination-scores-withdrawn/ #ai #evaluation #openscience
Related
Agent Engineering using Claude by Venkatesh Tadinada is free with a Leanpub Reader membership! Or you can buy it for $29...
Agent Engineering using Claude by Venkatesh Tadinada is free with a Leanpub Reader membership! Or you can buy it for $29.95! https://leanpub.com/agent_engineering_using_claude #age...
The Next Big Influencer Is This 4-Foot-Tall Robot From ChinaThe Unitree G1 has found online fame as a relatively afforda...
The Next Big Influencer Is This 4-Foot-Tall Robot From ChinaThe Unitree G1 has found online fame as a relatively affordable robot that can charm a crowd. But can it ever hold down ...
https://itsfoss.com/news/proton-ai-paper-trail/Proton has created a tool called "AI Paper Trail" which allows you to exp...
https://itsfoss.com/news/proton-ai-paper-trail/Proton has created a tool called "AI Paper Trail" which allows you to export your conversation history from your chatbot of choice, t...