1659 articles tagged with Benchmark
A passing agent eval can prove the wrong thing.Your agent may retrieve answers from the host machine.Do you review full trajectories before trusting a pass?#Bytecut #AI #AIAgents
📊 DeepSeek V3.2 (Reasoning) — the actual numbers GPQA: 84% MMLU-Pro: 86.2% Humanity's Last Exam: 24.6% Long Context Reasoning: 73.3%💰 68.3 intelligence points per dollarMeasured in...
A controlled comparison of Qwen3.8-Flash-Next and Claude Opus 5.5 on one LlamaStash feature, across Claude Code and Pi, at medium and xhigh effort
The OR-License benchmark evaluates large language model agents on real-world operations research tasks using original client materials.Source: arXiv cs.CLhttps://arxiv.org/abs/2610...
The study compares agentic AI systems with REST, SOAP, MCP and native function calls on a benchmark of 1,000 arithmetic expressions.Source: arXivhttps://arxiv.org/abs/2610.23921#Ma...
GardenBench is a new benchmark introduced with 11 test suites that evaluate LLM capabilities across various tasks.Source: arXivhttps://arxiv.org/abs/2610.23866#MachineLearning
Fully Remote | AI Benchmark & Red-Teaming Specialist (Llm)Contract, up to 40 hours weekly with 4 hours overlapping PST. Pay discussed during screening.✅ Building benchmark tasks th...
🤖 Yapay Zeka HaberiÇinli Yapay Zeka Firması Hakkında Çarpıcı İddia: Benchmark Sorularını mı Çaldı?MiniMax'in M3.1 Flash modelinin benchmark testlerinde hile yaptığı iddia ediliyor....
Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that ...
Decision models support routing, guardrails, and output verification in LLM systems. However, item accuracy alone cannot establish whether models make correct decisions across cont...
Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large langua...
Existing reasoning evaluations typically provide task-relevant information to models as available premises, rarely examining concurrently how models discover evidence autonomously ...
Laya returns choice, score and yes/no answers without generating text. Its own benchmark reports 0.362 accuracy zero-shot and 0.766 after fine-tuning: https://github.com/NandhaKish...
Microsoft AI’s MAI-Transcribe-2 achieved the lowest transcription error rate in 18 of 21 languages in...
Final post: "Moondream 3.1 (Moondream) enters the leaderboard withhttps://olud.ai/latest.html#AI #LLM #OpenSource
🧠Atlas is a benchmark that evaluates AI agents on search-intensive tasks. The framework assesses how well agents perform when they must retrieve and process information to complet...
Today's game: TightropeEvery day one AI-made browser game goes live for the whole world, with a single leaderboard. Top score at midnight UTC is the world champion.World record rig...
RealCLI introduces a benchmark that evaluates both task execution and human-agent collaboration for command-line interface agents.Source: arXiv cs.CLhttps://arxiv.org/abs/2610.2250...
Moondream 3 Preview (Moondream) lands on the leaderboard with open weights, 32k tokens, and $0.3 in / $2.5 out per 1M. That pricing makes open-weight inference genuinely cheap — an...
A benchmark score going up doesn't automatically mean a model "reasons" better. Our guide to reading AI papers critically: pin down what was actually measured, trace the inference ...
Video overlay removal aims to recover scenes obscured by subtitles, titles, and logos while preserving surrounding content. Existing methods use temporal propagation or source-cond...
General-purpose manipulation requires reliable behavior across embodiments, objects, and task demands. We introduce HandsOn, a composable simulation platform and benchmark for cros...
Robotic value models assess task execution from visual observations and instructions, providing feedback for policy learning and execution monitoring. However, existing evaluations...