Smart Routing Benchmark: Cost and Quality
A live Smart Routing benchmark compares the same prompts across routed and fixed-model requests, including classifier charges, answer checks, and latency. See where routing saved m...
#Benchmark
1676 articles tagged with Benchmark
98.7% better on seasonal, 9.8% worse on a random walk: the most useful benchmark I ran
Before pointing Google's TimesFM 3.0 at anything real, I ran a calibration baseline. Two synthetic...
One Hung API Call Used to Kill My 1,000-Run Benchmark. Here's the Fix.
The experiment was fine. The runner was the bug. A single hung API call discarded hours of...
A benchmark on whether LLMs check WHO said something before acting on it. Strip the source label off a rumor, phrase it ...
A benchmark on whether LLMs check WHO said something before acting on it. Strip the source label off a rumor, phrase it like policy, and a model that correctly refused will spend y...
MY Edit or Abstain: AI Code-Repair Decision Benchmark
This is a submission for the Kaggle Benchmarking Challenge For this challenge, I wanted to test a...
A Field Guide to AI Documentation: Model Cards, Eval Reports, Agent Cards, and More
Every developer learns the same documentation types. The README. The API reference. Code comments....
OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories
Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet benchmarks rarely capture these rea...
Golden Trace: The Eval Anchor Most Teams Skip
Cross-post of Insights #5 — canonical:...
Golden Trace: The Eval Anchor Most Teams Skip#ai #devops #architecture #testing #software #coding #development #engineer...
Golden Trace: The Eval Anchor Most Teams Skip#ai #devops #architecture #testing #software #coding #development #engineering #inclusive #communityGolden Trace: The Eval Anchor Most ...
Do LLMs Actually Check Their Tools? I Built a Benchmark That Lies to Them
This is a submission for the Kaggle Benchmarking Challenge. Every day, agents book invoices, check...
Try: "Qwen3 4B (Non-reasoning) posts 39.8% GPQA, 58.6% MMLU-Pro, 3.3% Humanity's Last Exam, 23.3% LiveCodeBench. These a...
Try: "Qwen3 4B (Non-reasoning) posts 39.8% GPQA, 58.6% MMLU-Pro, 3.3% Humanity's Last Exam, 23.3% LiveCodeBench. These aren't vendor claims — they're measuredhttps://olud.ai/leader...
Hook constraint benchmark kaggle-challenge DONE !!!
This is a submission for the Kaggle Benchmarking Challenge What I Benchmarked 😀: Shortform video...
Does an AI Trust Itself More Than It Trusts You? A Benchmark for Belief Attribution
This article is a submission for the Kaggle Benchmarking Challenge. "I told an AI something was...
I wrote five agents to cheat my own benchmark. They found three holes. Three more found me.
I recently published an RL environment — a reinforcement learning task that scores an agent on what...
Оценка AI‑агентов: 7 ошибок, из‑за которых тесты врутЗелёный статус eval‑тестов ещё не гарантирует, что AI‑агент работае...
Оценка AI‑агентов: 7 ошибок, из‑за которых тесты врутЗелёный статус eval‑тестов ещё не гарантирует, что AI‑агент работает корректно: он может выдавать правильный ответ, но выполнят...
The CAVEAT benchmark evaluates computer-use agents in marketplaces where platform incentives may conflict with user goal...
The CAVEAT benchmark evaluates computer-use agents in marketplaces where platform incentives may conflict with user goals.Source: arXivhttps://arxiv.org/abs/2609.27273#MachineLearn...
Clinical Intent Extraction recovers prospective clinical actions as structured records, introducing the CIRCA benchmark ...
Clinical Intent Extraction recovers prospective clinical actions as structured records, introducing the CIRCA benchmark and the Clinical Intent Representation.Source: arXiv cs.CLht...
Field note 009. A replication of a published agent-loop benchmark, and what it hides. ...#ai #agents #machinelearning #r...
Field note 009. A replication of a published agent-loop benchmark, and what it hides. ...#ai #agents #machinelearning #research #software #coding #development #engineering #inclusi...
IndicBankBench is a 799-case benchmark that evaluates safety, tool use, response adequacy and advisory quality of langua...
IndicBankBench is a 799-case benchmark that evaluates safety, tool use, response adequacy and advisory quality of language model assistants in Indian retail banking.Source: arXivht...
A benchmark that only checks the examples in the prompt is measuring the wrong thing. I ran into...#ai #machinelearning ...
A benchmark that only checks the examples in the prompt is measuring the wrong thing. I ran into...#ai #machinelearning #codereview #testing #software #coding #development #enginee...
Grouped Value Attention slashes transformer KV memory by roughly 45 % without hurting benchmark...#ai #machinelearning #...
Grouped Value Attention slashes transformer KV memory by roughly 45 % without hurting benchmark...#ai #machinelearning #abotwrotethis #software #coding #development #engineering #i...
SatNav: A Scalable Benchmark for Long-Horizon UAV Vision-Language Navigation from Satellite Imagery
Urban uncrewed aerial vehicle (UAV) vision-language navigation (VLN) requires agents to follow instructions across extended urban spaces, inherently demanding long-term memory and ...
4 Months of vox-bench in Production — What 14,000 Benchmark Runs Taught Us About Voice AI Latency
In June, we open-sourced vox-bench, the latency benchmarking tool we built after a TTS spike caused...
AI performance costs are falling faster than those of any previous technology. AI is hitting a fixed benchmark performan...
AI performance costs are falling faster than those of any previous technology. AI is hitting a fixed benchmark performance level at a rapidly falling cost.Source: The Decoderhttps:...