1698 articles tagged with Benchmark
Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to ...
Claude's Opus 4.7 reclaims coding benchmark leadership with 87.6% on SWE-bench Verified, while Anthropic narrows OpenAI's enterprise lead to just 4.6 percentage points. Meanwhile, ...
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during de...
Delve faked compliance certificates for 494 companies. Now agents are faking benchmark scores. Same pattern, new layer. The only thing that catches both is behavioral telemetry.
GeoAgentBench: New Dynamic Benchmark Tests LLM Agents on 117 GIS ToolsA new benchmark, GeoAgentBench, evaluates LLM-based GIS agents in a dynamic sandbox with 117 tools. It introdu...
GPU server rental 2026 — H100 from $1.85/hr, B200 access via CoreWeave. Full benchmark + price comparison: https://server-rental-guide.pages.dev/ #AI #infrastructure
Ivan Fioravanti ᯅ (@ivanfioravanti)평가(Eval)에서 thinking tokens가 결과와 소요 시간에 큰 영향을 주며, 모델이 과도하게 생각하는 경우를 벌점화하기 위해 평가별 최대 토큰 제한이 필요할 수 있다는 제안이다.https://x.com/ivanfioravanti/status/2045...
Gökdeniz Gülmez (@ActuallyIsaak)Apple의 MLX 프레임워크에서 LLM 성능을 평가하기 위한 첫 번째 종합 벤치마크인 MLX-Benchmark Suite가 소개되었습니다. 코드 이해, 작성, 디버깅 능력을 측정하는 CLI 도구와 데이터셋을 포함한 개발자용 평가 도구입니다.https://x.com...
As AI-assisted video creation becomes increasingly practical, instruction-guided video editing has become essential for refining generated or captured footage to meet professional ...
We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates existing multiword expression (M...
Multimodal large language models (MLLMs) have achieved impressive progress on vision language benchmarks, yet their capacity for visual cognitive and visuospatial reasoning remains...
The rise of autonomous AI agents suggests that dynamic benchmark environments with built-in feedback on scientifically grounded tasks are needed to evaluate the capabilities of the...
HORIZON Benchmark Diagnoses Long-Horizon Failures in GPT-5 and Claude AgentsA new benchmark called HORIZON systematically analyzes where and why LLM agents like GPT-5 and Claude fa...
🔬 Claude Opus 4.7 vs Opus 4.6 — Real Benchmark ResultsCoding: Opus 4.7 is 2.5x faster (13.4s vs 33.9s)Reasoning: TieContext: TieFull benchmark via Crazyrouter 👇https://crazyrouter....
Voice deepfake losses are projected to hit $40B by 2027, a 6,566% jump from 2023. Modulate's velma-2 now ranks #1 on Hugging Face's Speech Deepfake Leaderboard with a 1.104% averag...
Teams are making critical model selection decisions based on benchmarks designed for someone else's...
📰 Alibaba's Mystery AI "Happy Horse" Crashes the Video Generation Leaderboard — Then Reveals ItselfA stealthy 15-billion-parameter video AI appeared from nowhere, dethroned ByteDan...
Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation r...
The paradigm of agentic science requires AI systems to conduct robust reasoning and engage in long-horizon, autonomous exploration. However, current scientific benchmarks remain co...
Large language models have demonstrated strong performance on general-purpose programming tasks, yet their ability to generate executable algorithmic trading strategies remains und...
Diffusion-based image editing has achieved strong visual fidelity under natural language instructions, yet most existing systems still operate at the level of surface instruction f...
【VAKRAの内部構造：エージェントの推論、ツールの使用、および障害モード】https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis※AI生成の自動投稿（見出し＋リンク）#AI #生成AI #LLM #AIGenerated
Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (...
Which LLM is best for your AI agent? MiniMax-M2.7 vs Claude Sonnet 4.6📊 Benchmark Results:- Task completion: MiniMax +12%- Response quality: Claude +8%- Tool usage: TieSee full ben...