TU Delft researchers have built a system that lets passengers adjust an autonomous vehicle's driving style using natural language. The system uses GPT-4o-mini to parse requests and translate them into parameter adjustments for a motion planner — the software component that selects safe paths through traffic. When a user says "I'm feeling dizzy," the LLM increases weighting on smooth steering and gentle acceleration. When they say "I'm running late," it prioritizes speed.
The system doesn't give the LLM direct control over driving decisions. It only adjusts the relative importance of predefined criteria — speed, steering smoothness, collision probability — that a deterministic controller uses to evaluate possible paths. All changes stay within engineer-defined safety bounds. Before implementing adjustments, the system describes its plan in plain language and asks the passenger to confirm, creating a human-in-the-loop safeguard against misinterpreted prompts or hallucinations.
Testing in the nuPlan simulator showed the system correctly tuned motion planner behavior across eight different prompts in highway merging scenarios. The work will be presented at the IEEE Intelligent Transportation Systems Conference in September. The preprint is available on arXiv.
This approach differs from recent efforts to use LLMs for direct autonomous driving decisions. ETH Zurich's Nicolas Baumann published similar work last year using LLMs to tune a racing car controller. TU Munich's Matthias Althoff has explored LLM-suggested driving decisions verified against traffic rules via formal methods. The Delft team argues their parameter-tuning approach avoids the slow response times and lack of performance guarantees that make LLMs unsuitable for real-time driving control.
The constraint-based architecture means even if the LLM hallucinates, it cannot issue unsafe commands — it can only adjust parameters within pre-validated ranges. The engineering challenge is defining those ranges correctly across diverse driving scenarios.
