Autonomous Vehicles Employing An End-to-End AI Approach

Of the many autonomous vehicle (AV) use cases, two continue to receive the most attention: robotaxis and autonomous trucks of various form factors. AI is one of autonomous mobility’s key enabling technologies. To achieve the determinism required in mobility, the AV software platforms in use today incorporated a combination of statistical and symbolic AI. Recently, a new crop of AV startups, led by Wayve and Waabi, received large financing rounds to support their “end-to-end” AI approach to autonomous mobility, which utilizes the same reinforcement learning AI technology as foundation models. Could these approaches result in vehicles that are cheaper to produce and operate than those utilizing the approaches used to date, adhere to regulations, and be accepted by autonomous vehicle users?

In my 2017 book The Big Data Opportunity In Our Driverless Future, “driverless” was used to denote any capability that alleviated the traveler from having to drive, be it using autonomous vehicles or various types of mobility services, such as ride-hailing. The book focuses on the key role of AI in achieving driverless mobility and the importance of data. Mobility service providers employing driven vehicles use AI technologies to match travelers to vehicles, optimize a trip’s route while considering the route’s risk, manage the vehicles operating within a geofence, and many others. Autonomous vehicle developers, like Waymo, utilize AI to make it possible for the vehicle to go from point A to point B by itself.

To ensure predictable and deterministic behavior, the AV stacks of Waymo, Cruise, Motional, MobileEye, Aurora, and others are based on modular architectures, their components (perception, localization, planning, control, and prediction) incorporate symbolic and statistical AI, their planning and control components utilize deterministic algorithms, and the localization component is coupled with painstakingly developed high-definition maps. For example, Waymo developed high-definition maps for twenty-five US cities. MobileEye has been collecting data for over ten years. Judging by the investments these companies made to date, we can safely conclude that their approach to autonomous mobility is expensive. Moreover, despite the approach’s guardrails, it doesn’t appear to overcome the regulators’ reservations and the users’ skepticism. High costs, regulator reservations, and user skepticism have been three of the major reasons that the progress of autonomous mobility has been slow, and the financial performance of the companies involved in it underwhelming.

More recently, two AV startups, Wayve and Waabi, received large funding rounds (and here) because they use a so-called “end-to-end” approach that involves deep neural networks combined with reinforcement learning. This approach, which is also embraced by Tesla and Nio, takes perception input from the vehicle’s various sensors and converts it directly into control of the vehicle. It is also the approach employed by generative AI foundation models, such as GPT-4, Gemini, and others. The neural networks at the heart of these systems are trained with huge data sets. Wayve and Waabi claimed that they use mostly synthetic data generated by simulators. Tesla and Nio use the data they collect from their vehicles globally and argue that it is of adequate volume and situational variety to train accurate models. The Wayve and Waabi investors assume that these companies could achieve similar technical performance as Waymo at a lower cost than that of the existing methods. The lower cost can be achieved because the companies that use an “end-to-end” AI approach:

1. Use synthetic data extensively, resulting in low-cost data collection and preprocessing.
2. Don’t rely on maps. The trained AI models can easily generalize to new previously unseen operating environments. This approach enables moving to new geofences faster and less expensive than it is for the companies using map-based approaches, performs better in complex environments, and significantly reduces the long tail of corner cases.
3. Use fewer and lower-cost sensor types to achieve autonomous driving compared to Waymo.

The companies using the “end-to-end” AI approach must first show that the cost of training and testing the deep neural network models and training them using synthetic data, as well as the cost of inferencing using the resulting models, will be lower than the approaches employed by companies to date. The work on various foundation and Large Language Models shows that despite the improvements achieved over the past two years, such models take a long time to train, are expensive to develop, maintain, and operate because of the resources they require, i.e., both training and inferencing are expensive, and their quality across versions can be inconsistent. This is consistent with the corresponding requirements of most generative AI systems. High power requirements by the vehicle’s computing resources negatively impact its range. It has already been reported that Tesla is one of NVIDIA’s top customers whose GPUs are used in Tesla’s proprietary Dojo supercomputers. Tesla equips its vehicles with its proprietary computing hardware, whereas many other automakers equip their vehicles with processors provided by NVIDIA. Next, we must validate that achieving lower costs is more important than guaranteeing a deterministic behavior. Without a guaranteed deterministic behavior, the vehicles using an “end-to-end” AI may be involved in failures that are more catastrophic when they occur than the failures resulting in autonomous vehicles using deterministic AI approaches. There is a trade-off between the “end-to-end” AI approach helping reduce the long tail of corner cases with the higher safety (and transparency) of the deterministic AI approach.

However, the technology approach’s potential cost advantage is only one of the considerations. Regulatory compliance, end-user acceptance, and overall fleet operating costs are three considerations these startups must address. It is acknowledged that AVs must be much safer than human drivers, though standards have not yet been established. US regulators may have issues with the “end-to-end” AI system’s lack of “guardrails” and problems of transparency, explainability, and traceability. When there is a problem, regulators want an understanding that the problem won’t happen again. Tracing the error in a complete “end-to-end” AI system is difficult.

As I wrote in Transportation Transformation, new mobility, including the automated and autonomous vehicles that enable it, will be adopted region by region. Regional population preferences and regulatory environment will impact which technology approach will be embraced even if the costs associated with one are significantly lower than those of alternatives. Technology costs are only one of the overall cost components. The costs to operate a fleet of robotaxis include insurance, vehicle maintenance, and many others. Studies have shown that to reduce such costs and regardless of the use case (robotaxis, or long-haul trucks) autonomous vehicle providers will achieve better economics if they partner with service providers, i.e., mobility services and logistics, who have a strong understanding of customer, operating environments, and business models instead of operating a fleet themselves.  When it comes to autonomous privately owned vehicles, as the automakers that offer vehicles with higher levels of driving automation are also finding out, they will need to properly educate their customers on the merits of such systems before they can drive meaningful adoption. The adoption of Tesla’s FSD remains low even though it has been in the market the longest. Ford’s BlueCruise is growing as is the usage of GM’s SuperCruise but we are still not certain whether it leads to better operating margins for these automakers.

It is encouraging to see venture investors re-energized about autonomous vehicles after a considerable hiatus. It is also intriguing to see the end-to-end deep neural network approach applied to autonomous mobility. Even if it doesn’t address all the issues associated with the existing approaches, we will still learn a lot and likely create hybrid solutions that make a reality affordable and safe autonomous mobility across many use cases.