930 articles tagged with Benchmark
I built an open benchmark to measure how much of the MCP (AI-agent) attack surface a defense actually covers — then used that number, one gap at a time, to drive a proxy from 9% to...
TL;DR. Our ticket-routing eval scored 0.94 for five weeks. The number was manufactured. We had built...
TL;DR. Almost every eval harness reports a pass rate with an error bar, and almost every one of those...
📊 Qwen3 VL 8B (Reasoning) — the actual numbers GPQA: 57.9% MMLU-Pro: 74.9% Humanity's Last Exam: 3.3% Long Context Reasoning: 31%⚡ 128.9 tokens/sec💰 16.1 intelligence points per do...
See the biggest startup yappers on X/Twitter Discussion | Link
Visually impaired individuals (VIIs) encounter significant daily challenges due to limited access to visual information. Although Multimodal Large Language Models (MLLMs) have achi...
📄 AI paper of the day:« SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding »▲ 47 upvotes on Hugging Facehttps://huggingface.co/papers/2607.10400#AI ...
Submission for DEV's Summer Bug Smash — Smash Stories track. There was a file in my repo called...
Imaging-101 benchmark exposes where AI coding agents fail at real sciencehttps://1ban.news/imaging-101-benchmark-llm-coding-scientific/#1ban #imaging #101 #benchmark #llm #tech
📊 Llama 3.3 Nemotron Super 49B v1 (Reasoning) — the actual numbers GPQA: 64.3% MMLU-Pro: 78.5% Humanity's Last Exam: 6.5% Long Context Reasoning: 17%Measured independently, not sel...
Perplexity launches WANDR benchmark to test AI research capabilities, revealing ongoing challenges in large-scale data discovery and evidence validation. The post Perplexity Launch...
Most AI chatbot templates you find online share the same problem: nobody ever tested them. They demo...
Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good...
One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed...
TL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B...
TL;DR. Most "top open-source LLM eval framework" roundups rank features. None of them ask the one...
Same query. Same documents. Same model. And the RAG eval can still hand back a different...
📊 Mi:dm K 2.5 Pro — the actual numbers GPQA: 70.1% MMLU-Pro: 80.9% Humanity's Last Exam: 7.7% Long Context Reasoning: 9%Measured independently, not self-reported →https://opensourc...
The Leaderboard Is Dead. Here's What I Actually Reach For. Let It Break — part 2 Tags: #ai...
Show HN: Benchmark your eng team's AI agent maturity in 5 minuteshttps://agent-benchmarks.com/software-factory/#ai
I Tested 300+ Models. Then I Killed the Benchmark. Let It Break — part 1 Tags: #ai #llm...
Best AI agents fail 50% of visual tool tasks, Apple benchmark showsA new open benchmark with 500+ tools reveals even frontier AI models can't reliably read an image and act on it, ...
AI music transcription scores 38% on new pop benchmarkA new 572-segment pop music benchmark shows the best AI transcription models still miss most notes in real recordings, affecti...
Evaluation pipeline for text-to-image and image-editing models using Soft-TIFA scoring and MLLM-as-judge