926 articles tagged with Benchmark
Alibaba released benchmark scores for Qwen3.8-Max two weeks after claiming superiority without data. The 2.4T-parameter model scores come from internal testing only. Independent ve...
How local LLMs and specialized evaluation harnesses create a defense-in-depth strategy against the chaos of unmanaged AI agent supply chains.
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationhttps://arxiv.org/abs/2602.16763Comments: https://news.ycombinator.com/item?id=49170915#HackerNews #AI #Benchm...
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Homebench – Benchmark local LLMs for speed, memory, and qualityhttps://github.com/david-g-3654/homebench#github #llm #llms
Spirit AI, a Chinese physical AI startup, briefly overtook Nvidia to top the global RoboArena benchmark with its Spirit v1.6 model, raising questions about benchmark integrity in t...
Homebench – Benchmark local LLMs for speed, memory, and quality
📊 Nova 2.0 Lite (medium) — the actual numbers GPQA: 76.8% MMLU-Pro: 81.3% Humanity's Last Exam: 8.6% Long Context Reasoning: 58.3%⚡ 217.9 tokens/sec💰 22.4 intelligence points per d...
Recent advances in persistent personal-agent frameworks are making human-centered agent networks realistic deployment targets: each user can be served by an AI agent that acts on t...
Every eval you run against a public benchmark is a training signal you hand the next model. The leaderboard isn't measuring capability, it's leaking answers into the pretraining se...
Alibaba open-sources a Max-class Qwen model with coding benchmark gains, OpenAI's super PAC is caught running an AI-generated news site to shape policy debates, plus a language mod...
📊 Qwen3 Max — the actual numbers GPQA: 76.4% MMLU-Pro: 84.1% Humanity's Last Exam: 11.1% Long Context Reasoning: 46.7%💰 10 intelligence points per dollarMeasured independently, not...
Geospatial foundation models aim to learn representations that transfer across regions and sensors, yet evaluating them on specific tasks requires large, high-quality, multi-modal ...
Two weeks ago, I published "Why Agent Evaluation Is Harder Than Model Evaluation." The core argument:...
My personal AI benchmark: "Generate an SVG of a frog with a Habsburg jaw."https://frogs.vaguespac.es/#ai
My personal AI benchmark: "Generate an SVG of a frog with a Habsburg jaw."
Meta AI tests a second agent as a memory coach to keep long tasks on track, improving benchmark scores by up to 8.3%.Source: The Decoder AIhttps://the-decoder.com/meta-ai-uses-a-se...
I don't work with chatbots, but this is a good article about how Airbnb used Eval Driven Development (EDD - kind of like Test Driven Development (TDD)) to get a handle on testing n...
Composio a benchmarké le coût par tâche des #agents harnesses, sur les prix listes de Kimi K3 : #Hermes Agent à 0,39 $, #Claude Code à 1,47 $. Soit 3,7x plus cher.Le coût médian co...
🧠 Researchers examine MUD (a benchmark environment) as a tool for evaluating AI systems and identify how LLM judges can become distorted in ways that standard aggregate metrics lik...
RT @__tinygrad__: Die Benchmark-Ergebnisse liegen vor: GLM-5.2 auf einer $160k Tinybox Pro V2 Black erreicht 119 Token pro Sekunde im Einzelbetrieb und 917 Token pro Sekunde im Agg...
Structured machine learning workflow for dataset preparation, model training, evaluation, and grid load forecasting, with organized notebooks and saved model artifacts to support r...
When Llama 3.1 Tulu3 405B hits 71.6% on MMLU-Pro but only 3.5% on Humanity's Last Exam, the gap shows even top open models struggle with truly hard reasoning—https://olud.ai/leader...
You wrote a new eval. It caught the failure you just saw in production. You shipped it as a gate....