1660 articles tagged with Benchmark
Robotic value models assess task execution from visual observations and instructions, providing feedback for policy learning and execution monitoring. However, existing evaluations...
Realistic market impact is essential for generative limit order book (LOB) models to serve as world models for trading agents. If a simulator misprices this mechanism, agents learn...
No agenda, no rush, no leaving San Francisco: Benchmark’s partners on investing in a frenzied marketBenchmark may have grown its funds and swapped out some partners, but onstage th...
📊 Molmo 7B-D — the actual numbers GPQA: 24% MMLU-Pro: 37.1% Humanity's Last Exam: 5.1% Long Context Reasoning: 0%Measured independently, not self-reported →https://olud.ai/leaderbo...
Krones expands its Taicang plant into its largest site outside Germany, boosting local smart manufacturing, aseptic lab testing, and service for China's beverage industry. The post...
RT @SJRonanMD: 1M-Token-Kontext auf einem DGX Station GB300: zwei Modelle, gemessen. mehr auf Arint.info #AI #Benchmark #DeepSeek #GLM #LLM #NeedleInAHaystack #arint_info https://x...
Most AI features fail quietly. A summary is a little off, a label is wrong, a suggestion gets...
Chinese researchers have built a benchmark testing whether AI-powered robots can remember past observations and actions. Zhejiang University EMem-Bench covers 2,554 tasks in simula...
"Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans"The authors introduce an expert-guided benc...
Bimanual manipulation with dexterous hands has attracted growing attention in research on robotic action models. However, few studies have developed benchmarks specifically tailore...
Game decompilation hub freezes leaderboard after AI turns progress into a hollow numbers game amid explosion of 'AI slop'decomp.dev has closed its public index of game decompilatio...
Decomp.dev is shutting down its public index and leaderboard because AI-driven progress has turned the community into a hollow numbers game.Source: Tom's Hardwarehttps://www.tomsha...
Show HN: Gemini 4 Argon plays StarCraftI've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI,...
A parent's Hindi voice note hides the ask at 0:15: "send ₹5,000 before Sunday". The note is megabytes...
Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful s...
Hot on Hacker News (350 pts, 301 comments): Mercor's Labor Index Benchmark Map estimates ~90% of US working hours go to tasks that AI benchmarks don't cover well — the coordination...
The reply was "No. The passage does not support Yes." Target: Yes. Two of Inspect AI's three deterministic scorers marked it correct. Ten BoolQ rows, one material.
Gemini 4 Argon erreicht 97,5 % im ARC-AGI-3, fällt im Standard-Harness aber auf 49,8 %. Benchmark-Ergebnisse bleiben stark testumgebungsabhängig und sind ohne Kontext wenig aussage...
OmniCompass, a grounding benchmark, hit 406 upvotes on Hugging Face. Rare: it tests omni-modal alignment - vision, audio, text - in one task, not single modality. Gap most ignore. ...
I work at a company that builds memory infrastructure, so I'm biased: we compete in the thing this...
OmicsBench is a long-horizon, multi-turn bioinformatics benchmark for AI agents. It links 97 tasks into a directed acyclic graph covering 662 subtasks.Source: arXivhttps://arxiv.or...
🧠 Researchers introduce RealtimeBench, a benchmark with 180 tasks that evaluates language model agents while simulated environments continue running in real time rather than pausin...
Blockchain forensics is adversarial and open-ended: money launderers split, swap, and bridge funds, cut their trail at mixers or custodial services, and change tactics from one inc...
DeepSeek V3 (Dec '24) — GPQA 55.7%, MMLU-Pro 75.2%, and 18.4 intelligence points per dollar. That efficiency at this accuracy is rare.https://olud.ai/leaderboard.html#LLM #Benchmar...