1677 articles tagged with Benchmark
We introduce ClosureBench, a constructive benchmark for compositional graph-relational reasoning with programmatically verified ground truth. Unlike fixed-test-set benchmarks vulne...
GulliBench is a benchmark that purports to measure the "gullibility" of AI models.Here's there most "gullible" top 10:1. Opus 5 - 492. Fable 5 - 483. Muse Spark 1.2 - 424. Gemini 3...
More artificial intelligence companies may join the Hang Seng Index to increase the benchmark’s weighting towards technology, as the index compiler is due to reveal the result of i...
Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific...
Why acc and acc_norm disagree in lm-eval-harness: log-likelihood length bias, byte-length normalization, and when each multiple-choice metric lies.
“the losses generated by Aschenbrenner’s leveraged #AI #stock #bets put him up at the top of the global leaderboard of #fundfacepalms. We know this because we found a cool table on...
📦 Fresh open weights on the leaderboard: Qwen3.8 27B (Alibaba)Context: 1M tokens · $0.45 in / $3.2 out per 1M · open weightsAll the latest models, tracked hourly:https://olud.ai/la...
Every AI code review tool ships a blog post full of precision numbers. "98% precision, 87% recall,...
oh-my-agent 12.4.0 just shipped with a cross-vendor refactor guard and isolated harness evaluation....
Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk of models being optimized for t...
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential ...
🤖 An AI agent just stacked blocks in a live physics simulation — building an open benchmark arena for embodied AISharing an early result from a project I'm building: an open, brows...
TRACES moves beyond static datasets and answer keys, testing whether AI can navigate real-world scientific environments, adapt to feedback and produce verifiable discoveries REDWOO...
📊 Mistral Large 2 (Nov '24) — the actual numbers GPQA: 48.6% MMLU-Pro: 69.7% Humanity's Last Exam: 3.3% Long Context Reasoning: 5.3%Measured independently, not self-reported →https...
Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the ...
Physical interaction quality is central to deformable-object manipulation, yet most benchmarks evaluate task success alone. A policy may complete the task while allowing slip or ca...
The mere fact that there's ANY change in #LLM behavior, ANY hint of improvement (if you cherry-pick the right #benchmark tests, for geeklords are positively addicted to meaningless...
Provael publishes a leaderboard of attack success rates against vision-language-action robot...
Large language models are increasingly deployed through agent harnesses that manage tools, extensions, persistent state, permissions, and external actions. Existing safety benchmar...
We introduce PTXBench, a benchmark for evaluating and adapting large language models (LLMs) to use architecture-specific PTX for GPU kernel optimization. PTXBench measures function...
Everyone measures whether the agent works. Almost nobody measures whether they'd know if it...
Introducing Ayeixa Sentience Eval: Catching Raw JSON Leaks & Robotic Clichés in Agent...
Z.ai has released GLM-5.3, claiming the top spot from Anthropic and OpenAI on the CyberGym benchmark.But before upgrading, look at the fine print on active exploitation: https://w...