So relatable: (from 2019! But nothing changed)«A recurring recipe for Al research goes like this. Define a relatively na...

So relatable: (from 2019! But nothing changed)«A recurring recipe for Al research goes like this. Define a relatively narrow, though useful, task and collect a large data set for testing machine performance on this task. Perform a limited measure of human ability on this data set. Set up a competition in which Al systems can vie to outperform one another on this data set, until the human performance measure is met or exceeded. Report not only on the genuinely impressive and useful achievement, but also claim, falsely, that the winning AI systems have human-level performance on a more general task (for example, 'reading comprehension’).»#AI #benchmarks #MelanieMitchell