Browse
Log in / Register
Share
May 10, 2026 — General-purpose vision-language models have been handed the controls of a real car in what researchers describe as the first benchmark of its kind. DrivingBench put four leading model systems behind the wheel of a retrofitted 2022 Toyota Corolla navigating a low-speed cone course in an empty parking lot, with a safety driver ready to brake. Only one of the four completed the route, and it needed two attempts.
The test vehicle is a Toyota Corolla retrofitted with a comma four device and a modified version of openpilot. The model connects to the car through three tools: observe() pulls camera frames and vehicle state, set_motion() issues steering, target speed, duration, and a free-text reason, and stop_now() brakes immediately.
No conventional planner sits between model and machine. The vision-language model reads the camera feed and directly chooses motion commands; a controller translates those requests into vehicle actions. A new motion command replaces the one in progress rather than queuing, and when a command expires the car begins braking. Because the Corolla keeps moving while the model deliberates, inference latency is embedded in the task instead of being measured separately.
The course itself is a cone-marked 127-meter route in a parking lot, divided into sections A through G with defined start and finish zones. A human in the driver's seat stays ready to intervene at any moment.
The benchmark compared four model–harness pairings: GPT-6 Astra in Codex, Claude Fable 5.1 in Claude Code, GPT-5.6 Sol in Codex, and Grok 4.6 in Cursor. Each got one independent trial with up to three attempts inside the same conversation, so later attempts carried shared context and are not independent replicates.
Astra was the only system to finish. It completed the course on attempt two, entering the finish zone 4 minutes 44 seconds after its first accepted command, calling stop_now at 5:05, and closing the engagement at 5:22. Attempt one had covered 49% of the route. Claude Fable 5.1's best run reached 45% on its third attempt. GPT-5.6 Sol and Grok 4.6 never cleared the first corner. Across all eleven attempts, none other than Astra's second surpassed 45%. Two of the four systems improved materially across attempts, a sign that retained context helped — at least for some.
Progress is defined as the furthest point along the intended centerline the car reached while staying within 4 meters of that line, expressed as a share of the 127-meter course. The authors note the metric is sensitive to its assumptions: with the band reduced to 3 meters, Astra's first attempt would score 17% rather than 49%.
The data points to time as the decisive dimension. The median gap between the previous tool call and the next accepted set_motion command was 4.9 seconds for Astra, compared with 22.2 seconds for Fable. Those delays are not abstract. The car kept moving while each model thought, and between the last frame Astra had actually seen and the acceptance of its next command, the vehicle had typically traveled about 4 meters.
Physical actuation added to the lag. From a standstill, steering required a median of 4.0 seconds to reach 90% of the requested change; while rolling, just 1.9 seconds. The authors stop short of attributing any specific failure solely to latency, but they argue it made the task structurally harder for slower systems.
The traces also show a split between diagnosis and correction. Several written reflections from the models correctly identified the causes of earlier failures. Correct identification, however, did not reliably translate into different behavior on the following attempt. That failure mode — perceiving an error yet still failing to act on it under time pressure — is exactly what a closed-loop physical benchmark is built to expose.
The paper also details how the action interface influenced whether models drove at all. The tool output format and the framing of the task jointly determined whether a given model was willing to issue motion commands or refused outright. That insight extends beyond driving: as LLM agents gain tools that affect the physical world, interface decisions will shape both safety and engagement.
Prior work has kept language models close to the action without handing them the wheel. LaMPilot asks an LLM to write driving-policy code that calls established functions within a simulator; DiLu performs closed-loop simulated driving using reasoning, reflection, and stored experience. DrivingBench removes both the simulator and the classical planner boundary: the VLM itself picks steering and speed on a real car, at low speed but without a simulation safety net.
The team released the harness, prompts, course map, and traces, along with video and telemetry from every attempt. The paper is careful about its limits: the results compare deployed model–harness systems rather than isolated models, come from one trial per system, and were run on a low-speed private course with a vigilant safety driver. The authors frame the outcome as preliminary evidence of capability in one constrained configuration, not proof of general driving competence.
DrivingBench arrives amid growing interest in moving AI agents from text and image benchmarks into settings where timing cannot be ignored. Established autonomous-driving evaluation suites such as CARLA and nuScenes measure perception and planning in simulation or on recorded data, but they charge no penalty for slow deliberation. A real car does. Closed-loop evaluation on hardware adds a dimension that static accuracy scores omit: time.
For safety regulators, the setup offers a reasonable template. Low-speed testing on a private course with an attentive safety driver mirrors staged validation for automated vehicles; nothing here argues for unsupervised road deployment. At the same time, the fact that only one of four frontier systems could complete a simple cone course is a useful calibration for any near-term autonomous-driving claim.
For engineers building physical agents, the takeaways are practical: treat end-to-end latency as a first-class metric, model stale sensor frames as an expected input condition, and design tool interfaces that give models room to act decisively without stripping away their ability to refuse.
