1677 articles tagged with Benchmark
Your model already knows the answer: how benchmark answers leak into LLMshttps://elman.ai/news/your-model-already-knows-the-answer/#ai #llm #llms
📦 DeepSeek V4 Flash 0731 lands on the leaderboard with open weights: 1M context at $0.09 in / $0.18 out. A new option for long-context tasks without the high cost.https://olud.ai/l...
Cursor Grok 4.5 free desktop for Windows, macOS and Linux without X Premium. Grok 4.5 free on Cursor, grok 4.5 free API key and free tier, grok 4.5 vs Opus 5 vs GPT-5.6, grok 4.5 E...
Anthropic reviewed 141,006 eval runs. Three real organizations got breached during "sealed" cybersecurity tests on Opus 4.7 and Mythos 5. The environment had live internet. Models ...
When AI Benchmarks Plateau: A Systematic Study of Benchmark SaturationArticle URL: https://arxiv.org/abs/2602.16763 Comments URL: https://news.ycombinator.com/item?id=49170915 Poin...
We introduce NOLLI, a procedurally generated English-Korean puzzle benchmark designed to diagnose where Korean performance gaps arise. It comprises 15 puzzle types (25 tasks; 7,500...
Alibaba released benchmark scores for Qwen3.8-Max two weeks after claiming superiority without data. The 2.4T-parameter model scores come from internal testing only. Independent ve...
How local LLMs and specialized evaluation harnesses create a defense-in-depth strategy against the chaos of unmanaged AI agent supply chains.
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationhttps://arxiv.org/abs/2602.16763Comments: https://news.ycombinator.com/item?id=49170915#HackerNews #AI #Benchm...
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Clinical decision support is moving toward committees of language-model agents deliberating on a shared workspace. We ask whether such committees can be gamed by shortcuts, cues a ...
Homebench – Benchmark local LLMs for speed, memory, and qualityhttps://github.com/david-g-3654/homebench#github #llm #llms
Spirit AI, a Chinese physical AI startup, briefly overtook Nvidia to top the global RoboArena benchmark with its Spirit v1.6 model, raising questions about benchmark integrity in t...
Homebench – Benchmark local LLMs for speed, memory, and quality
📊 Nova 2.0 Lite (medium) — the actual numbers GPQA: 76.8% MMLU-Pro: 81.3% Humanity's Last Exam: 8.6% Long Context Reasoning: 58.3%⚡ 217.9 tokens/sec💰 22.4 intelligence points per d...
Recent advances in persistent personal-agent frameworks are making human-centered agent networks realistic deployment targets: each user can be served by an AI agent that acts on t...
Every eval you run against a public benchmark is a training signal you hand the next model. The leaderboard isn't measuring capability, it's leaking answers into the pretraining se...
Alibaba open-sources a Max-class Qwen model with coding benchmark gains, OpenAI's super PAC is caught running an AI-generated news site to shape policy debates, plus a language mod...
📊 Qwen3 Max — the actual numbers GPQA: 76.4% MMLU-Pro: 84.1% Humanity's Last Exam: 11.1% Long Context Reasoning: 46.7%💰 10 intelligence points per dollarMeasured independently, not...
Geospatial foundation models aim to learn representations that transfer across regions and sensors, yet evaluating them on specific tasks requires large, high-quality, multi-modal ...
Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument:...
My personal AI benchmark: "Generate an SVG of a frog with a Habsburg jaw."https://frogs.vaguespac.es/#ai
My personal AI benchmark: "Generate an SVG of a frog with a Habsburg jaw."
Meta AI tests a second agent as a memory coach to keep long tasks on track, improving benchmark scores by up to 8.3%.Source: The Decoder AIhttps://the-decoder.com/meta-ai-uses-a-se...