1692 articles tagged with Benchmark
📄 AI paper of the day:« SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding »▲ 47 upvotes on Hugging Facehttps://huggingface.co/papers/2607.10400#AI ...
Submission for DEV's Summer Bug Smash — Smash Stories track. There was a file in my repo called...
Imaging-101 benchmark exposes where AI coding agents fail at real sciencehttps://1ban.news/imaging-101-benchmark-llm-coding-scientific/#1ban #imaging #101 #benchmark #llm #tech
📊 Llama 3.3 Nemotron Super 49B v1 (Reasoning) — the actual numbers GPQA: 64.3% MMLU-Pro: 78.5% Humanity's Last Exam: 6.5% Long Context Reasoning: 17%Measured independently, not sel...
Perplexity launches WANDR benchmark to test AI research capabilities, revealing ongoing challenges in large-scale data discovery and evidence validation. The post Perplexity Launch...
Most AI chatbot templates you find online share the same problem: nobody ever tested them. They demo...
Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good...
One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed...
TL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B...
TL;DR. Most "top open-source LLM eval framework" roundups rank features. None of them ask the one...
Same query. Same documents. Same model. And the RAG eval can still hand back a different...
📊 Mi:dm K 2.5 Pro — the actual numbers GPQA: 70.1% MMLU-Pro: 80.9% Humanity's Last Exam: 7.7% Long Context Reasoning: 9%Measured independently, not self-reported →https://opensourc...
The Leaderboard Is Dead. Here's What I Actually Reach For. Let It Break — part 2 Tags: #ai...
Show HN: Benchmark your eng team's AI agent maturity in 5 minuteshttps://agent-benchmarks.com/software-factory/#ai
I Tested 300+ Models. Then I Killed the Benchmark. Let It Break — part 1 Tags: #ai #llm...
Best AI agents fail 50% of visual tool tasks, Apple benchmark showsA new open benchmark with 500+ tools reveals even frontier AI models can't reliably read an image and act on it, ...
AI music transcription scores 38% on new pop benchmarkA new 572-segment pop music benchmark shows the best AI transcription models still miss most notes in real recordings, affecti...
Evaluation pipeline for text-to-image and image-editing models using Soft-TIFA scoring and MLLM-as-judge
MedRealMM benchmark: AI matches doctors but fails on safetyNew 5,620-case benchmark finds frontier AI matches physicians on positive clinical steps but triggers more unsafe respons...
I've been running a small agentic eval harness against a local model and I'd like a sanity check on...
A paired workload protocol for validating latency, cost, correctness, and recovery before changing production models.
Clinical RAG can pass grounding checks while citing evidence about the wrong entity.
もうClaudeのモデル選びで迷わない — Anthropicが教える独自Eval（評価テスト）と「成功1件あたりのコスト」の測り方https://qiita.com/nogataka/items/6cd8c6ea583221e45def?utm_campaign=popular_items&utm_medium=feed&utm_source=popula...
LLM agents fail 70% of skill safety tests in SLBench studyA new 86-case benchmark finds Codex and Claude Code agents violate logical constraints in skill files up to 70% of the tim...