926 articles tagged with Benchmark
Benchmarks for systems that are optimized against the evaluation signal measure something different from what they claim. We document this concretely in two GPU-kernel-optimization...
We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of embodied agents within a photorealistic environment constructed from 360-degree videos. Ex...
Qwen3 VL 4B (Reasoning) hits 70% on MMLU-Pro but only 4.6% on Humanity's Last Exam—a clear gap between strong general knowledge and truly hard reasoning.https://olud.ai/leaderboard...
Originally published on AIdeazz — cross-posted here with canonical link. I shipped a silent failure....
When DeepSeek released V4 Flash on July 31, 2026, it quietly accomplished something that would have...
A version of Chinese startup DeepSeek’s flagship AI model is by far the least expensive to run on benchmark tests among well-known models globally and more than 100 times cheaper ...
The rapid advancement of Large Language Models (LLMs) is revolutionizing AI for Games by enabling open-ended and fluid interactive storytelling. However, existing research has larg...
📊 Qwen3 235B A22B (Reasoning) — the actual numbers GPQA: 70% MMLU-Pro: 82.8% Humanity's Last Exam: 11% Long Context Reasoning: 0%💰 5.1 intelligence points per dollarMeasured indepe...
A few weeks ago I wrote about building a reproducible test harness for comparing free AI coding...
We bucketed 238 BIRD-dev losses with a structural differ: 46 differ from gold only by a DISTINCT the model rightly added. Audit gold quality before writing prompt directives, or yo...
Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. Contamination mitigation evaluation intervenes in the decodi...
【FFASRリーダーボードのご紹介：実世界におけるASRのベンチマーク】https://huggingface.co/blog/ffasr-leaderboard※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGenerated
Hugging Face Space: agent-memory-leaderboard/leaderboard
WeatherNext 2 sets a new benchmark in cyclone prediction accuracy.Source: Google AI Bloghttps://blog.google/innovation-and-ai/models-and-research/google-deepmind/weathernext-2-cycl...
Alibaba released Qwen 3.8-Max this week and marketed the preview as second only to Claude Fable 5 (their launch-day table was more equivocal: the model leads on one of 12 coding-ag...
Your model already knows the answer: how benchmark answers leak into LLMsArticle URL: https://elman.ai/news/your-model-already-knows-the-answer/ Comments URL: https://news.ycombina...
New benchmark just dropped...https://www.felonybench.com/#AI #benchmark #lawandorder #AIregulation
We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish. Despite Yiddish's rich textual tradition, its limited digital presence and...
Your model already knows the answer: how benchmark answers leak into LLMshttps://elman.ai/news/your-model-already-knows-the-answer/#ai #llm #llms
📦 DeepSeek V4 Flash 0731 lands on the leaderboard with open weights: 1M context at $0.09 in / $0.18 out. A new option for long-context tasks without the high cost.https://olud.ai/l...
Cursor Grok 4.5 free desktop for Windows, macOS and Linux without X Premium. Grok 4.5 free on Cursor, grok 4.5 free API key and free tier, grok 4.5 vs Opus 5 vs GPT-5.6, grok 4.5 E...
Anthropic reviewed 141,006 eval runs. Three real organizations got breached during "sealed" cybersecurity tests on Opus 4.7 and Mythos 5. The environment had live internet. Models ...
When AI Benchmarks Plateau: A Systematic Study of Benchmark SaturationArticle URL: https://arxiv.org/abs/2602.16763 Comments URL: https://news.ycombinator.com/item?id=49170915 Poin...
We introduce NOLLI, a procedurally generated English-Korean puzzle benchmark designed to diagnose where Korean performance gaps arise. It comprises 15 puzzle types (25 tasks; 7,500...