What I learned trying to benchmark local LLMs honestly
I had eight models sitting on my laptop and no idea which one I should actually be using. I could...
#Benchmark
1677 articles tagged with Benchmark
The Hutter Prize measures intelligence better than any OpenAI benchmark
Winning a leaderboard doesn't hold any value for me now. Every week, a new model surpasses an...
Your eval suite passes. Does it actually check anything?
Here's a grader I've shipped. Maybe you have too: assert: - type: contains value:...
Moonshot AI's 2.8 Trillion Parameter Model Just Became the First From China to Top a Major Coding Benchmark
Moonshot AI's 2.8 Trillion Parameter Model Just Became the First From China to Top a Major Coding Benchmark On July 16, the Chinese AI start-up Moonshot AI released its Kimi K3 lar...
【VAKRAの内部構造：エージェントの推論、ツールの使用、および障害モード】https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis※AI生成の自動投稿（見出し＋リン...
【VAKRAの内部構造：エージェントの推論、ツールの使用、および障害モード】https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGenerated
Your Golden Dataset Is Rotting: The Eval Oracle Nobody Re-Validates
We talk about agents drifting. We almost never talk about the thing we measure them against drifting....
Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure
Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim. We document this concretely in two GPU-kernel-optimization...
360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents
We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of embodied agents within a photorealistic environment constructed from 360-degree videos. Ex...
Qwen3 VL 4B (Reasoning) hits 70% on MMLU-Pro but only 4.6% on Humanity's Last Exam—a clear gap between strong general kn...
Qwen3 VL 4B (Reasoning) hits 70% on MMLU-Pro but only 4.6% on Humanity's Last Exam—a clear gap between strong general knowledge and truly hard reasoning.https://olud.ai/leaderboard...
131 Tests, 4 Layers, $00.03/Run: Why I Built My AI Agent Eval Harness First
Originally published on AIdeazz — cross-posted here with canonical link. I shipped a silent failure....
DeepSeek V4 Flash Just Aced the Hardest AI Reasoning Benchmark — for $0.02 a Task
When DeepSeek released V4 Flash on July 31, 2026, it quietly accomplished something that would have...
DeepSeek undercuts OpenAI and Anthropic on benchmark running costs
A version of Chinese startup DeepSeek’s flagship AI model is by far the least expensive to run on benchmark tests among well-known models globally and more than 100 times cheaper ...
Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives
The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has larg...
📊 Qwen3 235B A22B (Reasoning) — the actual numbers GPQA: 70% MMLU-Pro: 82.8% Humanity's Last Exam: 11% Long Context Reas...
📊 Qwen3 235B A22B (Reasoning) — the actual numbers GPQA: 70% MMLU-Pro: 82.8% Humanity's Last Exam: 11% Long Context Reasoning: 0%💰 5.1 intelligence points per dollarMeasured indepe...
The Model Passed Your Benchmark. Now Stop Merging Its Code Blindly
A few weeks ago I wrote about building a reproducible test harness for comparing free AI coding...
Your text-to-SQL model isn't as wrong as your benchmark says. The gold SQL is.
We bucketed 238 BIRD-dev losses with a structural differ: 46 differ from gold only by a DISTINCT the model rightly added. Audit gold quality before writing prompt directives, or yo...
Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination
Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. Contamination mitigation evaluation intervenes in the decodi...
【FFASRリーダーボードのご紹介：実世界におけるASRのベンチマーク】https://huggingface.co/blog/ffasr-leaderboard※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGe...
【FFASRリーダーボードのご紹介：実世界におけるASRのベンチマーク】https://huggingface.co/blog/ffasr-leaderboard※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGenerated
agent-memory-leaderboard/leaderboard - AI Space/Demo
Hugging Face Space: agent-memory-leaderboard/leaderboard
WeatherNext 2 sets a new benchmark in cyclone prediction accuracy.Source: Google AI Bloghttps://blog.google/innovation-a...
WeatherNext 2 sets a new benchmark in cyclone prediction accuracy.Source: Google AI Bloghttps://blog.google/innovation-and-ai/models-and-research/google-deepmind/weathernext-2-cycl...
Qwen 3.8-Max and Claude Opus 5 show why raw benchmark scores don't predict the bill
Alibaba released Qwen 3.8-Max this week and marketed the preview as second only to Claude Fable 5 (their launch-day table was more equivocal: the model leads on one of 12 coding-ag...
Your model already knows the answer: how benchmark answers leak into LLMsArticle URL: https://elman.ai/news/your-model-a...
Your model already knows the answer: how benchmark answers leak into LLMsArticle URL: https://elman.ai/news/your-model-already-knows-the-answer/ Comments URL: https://news.ycombina...
New benchmark just dropped...https://www.felonybench.com/#AI #benchmark #lawandorder #AIregulation
New benchmark just dropped...https://www.felonybench.com/#AI #benchmark #lawandorder #AIregulation
MameLoshnLM: Yiddish Language Model and Evaluation Benchmark
We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish. Despite Yiddish's rich textual tradition, its limited digital presence and...