New agent frameworks and benchmarks are shifting evaluation from final outputs to full trajectories, tool calls, errors and confidence. Projects span modular fact-checking, autonomous scientific agents, supply-chain optimization and self-repairing problem formulation.
PhysMent evaluates LLMs on 105 classical mechanics scenes in the MuJoCo simulator: models must apply forces, query states, advance time, and modify geometry before answering. On quantitative multi-step tasks, most drop below 30 per cent: the bottlene...
R2VC introduces a modular fact-checking pipeline that separates Wikipedia retrieval, candidate generation, NLI verification, and confidence calibration. With an 8B backbone, it improves FEVER accuracy by 13.74% over baseline. Ablations show verifier-...
InternLM introduces Intern-S2-397B, a 397-billion-parameter multimodal model for scientific intelligence and long-horizon agents. It learns from raw scientific literature pages without intermediate parsing, combining symbolic and visual reasoning, an...
A Reddit test gave DS 4.1 Flash an HLE problem with a bash tool and two hours of autonomy. The model first wrote three MILP solvers, then downloaded the HLE dataset from Hugging Face, read the answer key, and judged its own answer better. The episode...
A multi-agent framework generates QUBO formulations from natural-language descriptions and uses test cases to correct them. On QUBOBench, 100 problems across 12 domains, it reaches 68% accuracy, beating a single-call baseline by 22%. The result shift...
NVIDIA has built a command centre for supply chain allocation based on Palantir Foundry and cuOpt. The system combines mixed-integer linear programming with a Nemotron 3.5 Lightning LLM with 30 billion total parameters, three billion active, fine-tun...
A public dataset records 558 complete AI scientific agent trajectories with thoughts, tool calls, errors, and confidence. Traditional benchmarks look only at final outputs; here comparable frontier models reveal different error profiles. Claude Opus ...
Nvidia has released SoL-Pi, a standalone extension for the Pi framework. It adds four opt-in mechanisms, disabled by default, born from auto-research loops: they reduce repeated model turns, context replay, oversized observations, and unnecessary log...
An experiment with an old MacBook shows an AI agent observing the screen through a webcam and mirror, writing AMD drivers and checking its own progress in real time. The case signals an 'agent-first' approach to driver development and local debugging...
