1677 articles tagged with Benchmark
This week also saw NVIDIA’s Nemotron 3 Embed claim the top spot on the Retrieval-Augmented Generation (RTEB) leaderboard, confirming that embedding optimization is the new battlegr...
Turn a coding-benchmark audit into a reproducible task-state model, sensitivity analysis, and acceptance rule.
🧠 Researchers introduce the Wandr Benchmark, a tool for evaluating AI agents that perform web search and information gathering tasks. The benchmark measures how well these agents c...
I built an open benchmark to measure how much of the MCP (AI-agent) attack surface a defense actually covers — then used that number, one gap at a time, to drive a proxy from 9% to...
TL;DR. Our ticket-routing eval scored 0.94 for five weeks. The number was manufactured. We had built...
TL;DR. Almost every eval harness reports a pass rate with an error bar, and almost every one of those...
📊 Qwen3 VL 8B (Reasoning) — the actual numbers GPQA: 57.9% MMLU-Pro: 74.9% Humanity's Last Exam: 3.3% Long Context Reasoning: 31%⚡ 128.9 tokens/sec💰 16.1 intelligence points per do...
See the biggest startup yappers on X/Twitter Discussion | Link
Visually impaired individuals (VIIs) encounter significant daily challenges due to limited access to visual information. Although Multimodal Large Language Models (MLLMs) have achi...
📄 AI paper of the day:« SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding »▲ 47 upvotes on Hugging Facehttps://huggingface.co/papers/2607.10400#AI ...
Submission for DEV's Summer Bug Smash — Smash Stories track. There was a file in my repo called...
Imaging-101 benchmark exposes where AI coding agents fail at real sciencehttps://1ban.news/imaging-101-benchmark-llm-coding-scientific/#1ban #imaging #101 #benchmark #llm #tech
📊 Llama 3.3 Nemotron Super 49B v1 (Reasoning) — the actual numbers GPQA: 64.3% MMLU-Pro: 78.5% Humanity's Last Exam: 6.5% Long Context Reasoning: 17%Measured independently, not sel...
Perplexity launches WANDR benchmark to test AI research capabilities, revealing ongoing challenges in large-scale data discovery and evidence validation. The post Perplexity Launch...
Most AI chatbot templates you find online share the same problem: nobody ever tested them. They demo...
Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good...
One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed...
TL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B...
TL;DR. Most "top open-source LLM eval framework" roundups rank features. None of them ask the one...
Same query. Same documents. Same model. And the RAG eval can still hand back a different...
📊 Mi:dm K 2.5 Pro — the actual numbers GPQA: 70.1% MMLU-Pro: 80.9% Humanity's Last Exam: 7.7% Long Context Reasoning: 9%Measured independently, not self-reported →https://opensourc...
The Leaderboard Is Dead. Here's What I Actually Reach For. Let It Break — part 2 Tags: #ai...
Show HN: Benchmark your eng team's AI agent maturity in 5 minuteshttps://agent-benchmarks.com/software-factory/#ai
I Tested 300+ Models. Then I Killed the Benchmark. Let It Break — part 1 Tags: #ai #llm...