1677 articles tagged with Benchmark
The benchmark you trust most is the one whose answers already leaked into the pretraining set. Every public eval is a training signal you handed the next model. Contamination isn't...
Qwen Audio 3.0 TTS Plus leads the Speech Arena leaderboard with 16-language support and natural style control. Output speed is its main drawback.#AI #AutomationSource: The Decoder ...
Claude Fable 5 debuts with benchmark gains and stricter guardrails, Apple reveals Siri AI at WWDC, Google cuts AI Plus pricing, and OpenAI files for IPO. A busy week in AI. 🤖#AI #A...
Radar's open source, single Go binary, MCP server built in.https://radarhq.io/blog/radar-mcp-vs-kubectl-benchmark#Kubernetes #AI #MCP #DevOps #OpenSource
📊 DBRX Instruct — the actual numbers GPQA: 33.1% MMLU-Pro: 39.7% Humanity's Last Exam: 6.6% LiveCodeBench: 9.3%Measured independently, not self-reported →https://opensourceai.tech/...
Two prompts on one eval set are paired data. The two-proportion SE said 'collect more'; McNemar said it was already decided.
Evaluating the factuality of long-form generations has focused predominantly on precision, measuring whether the claims a model makes are correct. The dominant decompose-search-ver...
Kimi K3 (Moonshot AI) beat GPT-5.6 and Fable 5 on the front-end coding benchmark, 76% vs 63%. Open weights, 2.8 trillion parameters.It's also half the price per token, but needs ro...
📊 Mistral Medium 3 — the actual numbers GPQA: 57.8% MMLU-Pro: 76% Humanity's Last Exam: 4.3% Long Context Reasoning: 28%⚡ 49.2 tokens/sec💰 15.6 intelligence points per dollarMeasur...
Why has Austria become Europe's hottest AI market this year?Austria’s benchmark ATX index has gained 21.3% since the beginning of January.The biggest driver of that performance is ...
LoRA Speedrun - a public wall-clock leaderboard for fine-tuning techniqueshttps://github.com/Saivineeth147/lora-speedrun#HackerNews #Tech #AI
Multi-agent systems routinely place one AI agent in authority over another. When a subordinate refuses a task, the manager chooses the outcome: it can renegotiate, report the failu...
World modeling is an unsettled field: architectures, training objectives, and state representations interact in complex ways, and no single recipe dominates across environments. Th...
Perplexity has released WANDR, an open benchmark with 500 evidence-heavy tasks for testing research agents. The benchmark evaluates whether agents can discover many qualifying enti...
There's a slide in every "LLM eval platform" pitch deck that says: score every response, catch every...
AI agent benchmark tests tool-switching when reliability shiftsA new arXiv paper uses cognitive psychology's set-shifting concept to test whether LLM agents adapt when reliable too...
Facts change. An earnings forecast is revised. A policy is amended. A medication dose is corrected....
【QIMMA قِمّة ⛰: 品質第一のアラビア語LLMリーダーボード】https://huggingface.co/blog/tiiuae/qimma-arabic-leaderboard※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGenerated
China's 2.8-trillion-parameter Kimi K3 beats Claude Fable 5 in Frontend Code Arena benchmark— Moonshot AI delivers largest open-weight AI model ever, as Ch…Beijing-based Moonshot A...
【Open ASR リーダーボードに Benchmaxxer Repellant を追加】https://huggingface.co/blog/open-asr-leaderboard-private-data※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGenerated
SWE-bench needs 90% of tasks for reliable agent benchmark resultsA replay analysis of three LLM agent benchmarks finds the safe partial-run fraction ranges from 15% to over 95%, wi...
LLM agent memory poisoning evades defenses, 1,227-case study findsMemPoison benchmark tests 1,227 attacks across 10 model families, finding write-time defenses miss sophisticated m...
LLM agents spot 88% of supply-chain failures but can't actNew STOCKTAKE benchmark: LLM agents detect up to 88% of hidden supply-chain failures but two of four models score below a ...
📊 K-EXAONE (Reasoning) — the actual numbers GPQA: 78.3% MMLU-Pro: 83.8% Humanity's Last Exam: 13.1% Long Context Reasoning: 55.7%Measured independently, not self-reported →https://...