The Judge Reads Tokens: An Adversarial Evaluation of an Eval Toolkit's Own Scorers
The Judge Reads Tokens: An Adversarial Evaluation of an Eval Toolkit's Own...
#Benchmark
1658 articles tagged with Benchmark
"GDPx is a benchmark that covers the most valuable domains for frontier intelligence."The 11 domains are: chip design, c...
"GDPx is a benchmark that covers the most valuable domains for frontier intelligence."The 11 domains are: chip design, cybersecurity, biology, finance, legal, quant trading, recurs...
In my 23-28 October 2026 routing benchmark, 190 of 1,000 CLINC requests were out of scope.Mammoth-1 caught 152, with 100...
In my 23-28 October 2026 routing benchmark, 190 of 1,000 CLINC requests were out of scope.Mammoth-1 caught 152, with 100 false alarms. Kev-4B caught 136, with 239.I'd test the "non...
AI Inference Benchmark: Keln vs Fireworks, Together AI, Baseten, Modal, DeepInfra & Z.ai
We ran the same agent sessions across seven inference platforms to see how they compare on speed,...
Running LLMs on a CPU-Only Server: My Benchmark of Gemma, Qwen, Spark, and MiniCPM
We all know the default answer when someone asks about running local LLMs: "Get a GPU." Preferably an...
OpenAI Introduces BioBasicsBench for Wet-Lab AI Agent EvaluationOpenAI has introduced BioBasicsBench, a benchmark measur...
OpenAI Introduces BioBasicsBench for Wet-Lab AI Agent EvaluationOpenAI has introduced BioBasicsBench, a benchmark measuring how AI agents interpret raw experimental data across 100...
duartegroup/CHELATE: A post-docking filter for metalloprotein-ligand complexes: CHELATE predicts metal-binding-group RMSD and keeps only poses that place the metal coordination correctly. Includes the curated dataset, a six-program docking benchmark and trained models.
A post-docking filter for metalloprotein-ligand complexes: CHELATE predicts metal-binding-group RMSD and keeps only poses that place the metal coordination correctly. Includes the ...
The Only Coding-Agent Benchmark That Matters Is Your Own Revert Rate
Leaderboards score a model plus a harness on someone else's tasks. Tag landed commits with Agent: trailers and count reverts, fixes and red CI per agent.
China's stocks pared weekly gains and the benchmark 10-year bond hit a record low as the global artificial intelligence ...
China's stocks pared weekly gains and the benchmark 10-year bond hit a record low as the global artificial intelligence stock bubble burst following the US Democrats' landslide win...
My Claude Code router cost more than Opus. Here's what the benchmark taught me
I built AutoMode because most of my Claude Code sessions look the same. I ask a quick question, then...
Terminal-style Mathematical Modeling Bench: The First Agentic and Verifiable Benchmark for Mathematical Modeling
Mathematical modeling turns an open-ended real-world problem into assumptions, a model, a solution, and a defended recommendation, and its answers matter only if they are both clos...
DawidNezo/local-image-gen-benchmark: CLI for benchmarking text-to-image diffusion models locally with Hugging Face Diffusers: load and generation time, throughput, CPU/RAM/GPU usage.
CLI for benchmarking text-to-image diffusion models locally with Hugging Face Diffusers: load and generation time, throughput, CPU/RAM/GPU usage.
NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma
Current medical benchmarks typically break down clinical competence into isolated factual questions, single-image tasks, or static case vignettes. These tests score individual answ...
Single-prompt evals can't surface stateful failures. YANAMI is a new multimodal dataset and benchmark built for multitur...
Single-prompt evals can't surface stateful failures. YANAMI is a new multimodal dataset and benchmark built for multiturn, stateful interaction — the eval design that agentic model...
LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis
We introduce LongBanana, to our knowledge the first expert-designed and expert-verified benchmark for long-context multi-reference image synthesis in the age of agent-driven image ...
Alternative: "⚖️ Qwen3.8 Max (0902) scores 33.7; GPT-6 Astrahttps://olud.ai/leaderboard.html#OpenSource #AI #LLM
Alternative: "⚖️ Qwen3.8 Max (0902) scores 33.7; GPT-6 Astrahttps://olud.ai/leaderboard.html#OpenSource #AI #LLM
maerd610/MOSS-TE-Benchmark: MOSS-Image text encoder benchmark: int8 vs bf16, 69 paired cases (same seed / prompt / sampler, only clip_name differs)
MOSS-Image text encoder benchmark: int8 vs bf16, 69 paired cases (same seed / prompt / sampler, only clip_name differs)
📰 Benchmark chairJames Birdle described his experiment using AI to assist in chair design: What…Tags: chair, creativity,...
📰 Benchmark chairJames Birdle described his experiment using AI to assist in chair design: What…Tags: chair, creativity, Emergence Magazine, James Birdle, process📰 Source: FlowingD...
AuremHQ/weight-space-programming: wsp-kit by Aurem — an engineering tool for small models. Try to fix small open-weight models by editing weights directly, no retraining. An eval gate decides.
wsp-kit by Aurem — an engineering tool for small models. Try to fix small open-weight models by editing weights directly, no retraining. An eval gate decides.
my take: a benchmark at 90% tells you nothing. one at 12% shows you where the work still is.https://gdpx.semianalysis.co...
my take: a benchmark at 90% tells you nothing. one at 12% shows you where the work still is.https://gdpx.semianalysis.com/#AI #AIAgents #LLM #Benchmarks #BuildInPublic
📦 Fresh open weights on the leaderboard: Flint (free) (Recursive)Context: 262.1k tokens · free tier · open weightsAll th...
📦 Fresh open weights on the leaderboard: Flint (free) (Recursive)Context: 262.1k tokens · free tier · open weightsAll the latest models, tracked hourly:https://olud.ai/latest.html#...
Why we need actual Open Source #AI. Not just open weights...open source, open data, open eval, open alignment.#ML #MLsec...
Why we need actual Open Source #AI. Not just open weights...open source, open data, open eval, open alignment.#ML #MLsec #security #infosec #swsec #appsec https://berryvilleiml.com...
The eval set is the spec: how to build the gate that every model change has to pass
Three frontier models shipped in one week. Your provider can roll a version underneath you. The only thing standing between that churn and your...