Recently teaching my teenage son to drive, I couldn’t explain instinctive behavior I take for granted after 40-plus years behind the wheel. While things like following too closely and braking too late were easy to point out, anticipating the actions of other drivers and dealing with complex intersections only come with experience.
Autonomous vehicles (AVs) are like newbie drivers, except with better-developed brains and billions of dollars in tech to help shorten the learning curve. But even with all their sensors and software, AVs still have flaws to overcome before they drive with complete confidence and competence.
In the race to get robocars on the road, several under-the-radar tech trends are coalescing to help make true AVs a reality—and maybe make my son’s generation one of the last to learn to drive itself.
Teaching AVs the Rules of the Road
My new driver had to study the Oregon Driver Manual to learn the difference between, say, a stop and yield sign. AVs similarly learn the rules of the road, but through artificial intelligence (AI). Another requirement for my son is to log 50 hours with an adult in the car. AVs also acquire real-world experience by putting in hours on the road, but only in certain locations and conditions because of legal restrictions and weather.
AVs learn to interpret signs and other roadway info via a type of AI known as machine learning, which requires driving a route and humans verifying the data. Traffic-data company Inrix has a way for AVs to more quickly learn the rules even in places they’ve never driven. Inrix’s AV Road Rules platform lets cities digitize their traffic infrastructure and rules. This not only creates a shortcut for AVs to memorize traffic rules but also allows them to operate from accurate data.
“For 100 years, signs and lane markings have been the language of communicating traffic rules to drivers, and it’s worked pretty well,” says Avery Ash, head of autonomous mobility for Inrix. “But we’ve all been in situations where the signage is confusing or obscured or lane striping has been worn off, but we figure it out.”
Although machine learning can help AVs figure out such situations, Ash adds that “it’s a tedious, lengthy, and expensive process, and the results are not accurate enough for the sort of safety-critical operation required by AVs. AV Road Rules is an additional data layer that complements machine learning and HD maps.”
Once AVs know the rules, they need to apply them on roads. But just as I don’t have 50 extra hours to spend driving with my son, AV operators have limited time and resources when logging miles.
A 2016 RAND study estimated that AVs “would have to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their safety in terms of fatalities and injuries.” But developers have found ways to speed up the process through simulation software, and most are using simulation to accelerate AV deployment.
While Waymo leads the pack in terms of real-world miles traveled testing its AVs—more than 9 million since parent company Google first kicked off the project almost a decade ago—it’s able to achieve nearly the same number of miles every day using simulation software. “Having a proper simulation strategy is the only way that autonomous vehicles can train the sensors and decision-making functions for road testing and have the confidence that their technology is safe and ready,” says Danny Atsmon, CEO of Cognata, which is working with Audi and others.
Simulation also allows AV developers to test for “edge cases” such as pedestrians and cyclists suddenly crossing in front of the car or when the sun shines directly into an AV’s front-facing camera at sunset, temporarily blinding it.
“With simulation, we’re able to recreate blinding sun 24 hours a day,” says Danny Shapiro, senior director of automotive for chipmaker Nvidia. “And we can do it on every road and combine that with a rainstorm or any kind of weather. We can also simulate a car running a red light and evaluate if the AV is taking the correct action and detecting everything it should.”
Even with all their computing power and AI, situations remain that require AVs to call on human drivers for help. That’s why most AVs testing on public roads need a human behind the wheel to take over when a self-driving computer gets confused or can’t continue for some reason, such as when a road is closed or there’s a temporary construction zone.
But as AVs shed components such as steering wheels and pedals and humans become cargo, they’ll need to be remotely controlled; California’s regulations for testing AVs on public roads that took effect in April require so. Enter teleoperation, the industry term for remotely operated AVs.
“Think of teleoperation as an air traffic controller for autonomous vehicles,” says Jada Smith, VP of advanced engineering at Aptiv (formerly Delphi), which runs a robo-taxi program in Las Vegas in partnership with Lyft. But unlike air traffic control, AV teleoperators will be able to not only monitor but also operate self-driving cars when an autonomous vehicle encounters a situation it doesn’t know how to handle.
Most major AV players are either preparing for teleoperation of robo-taxis or testing it already. GM’s stable of Chevy Bolts being retrofitted by its Cruise Automation subsidiary to operate without a steering wheel or pedals have an “expert mode,” which relies on teleoperator assistance. Toyota has a patent for “remote operation of autonomous vehicle in unexpected environment,” while self-driving startup Zoox has one for a “teleoperation system and method for trajectory modification of autonomous vehicles.”
NASA remotely controlled a series of Martian rovers in the late 1990s and early 2000s. That’s why Nissan, meanwhile, has recruited former NASA scientists to apply a version of the space agency’s teleoperations expertise to the automaker’s autonomous vehicles and is testing remote control of a fleet of self-driving Leaf EVs at NASA’s Ames campus in Silicon Valley.
In the first demonstration of the technology on public roads (and on Earth), at the Consumer Electronics Show last January, Phantom Auto had a human operator 500 miles away in California control a car driving on the Las Vegas Strip. “AV technology may be about 97 percent of the way there, but that last 3 percent may take decades to solve,” says Elliot Katz, co-founder and chief strategy officer for Phantom Auto. “Teleoperation … serves as an essential technological bridge which enables AVs to be safely deployed now.”
Watch Me Now
Along with remote monitoring of self-driving cars, human drivers will also increasingly be under scrutiny, especially in the interim between SAE Levels 3 and 4 of autonomy when humans will need to be ready to take control at a moment’s notice. Although cameras are already used in some cars to monitor drivers, such as with Cadillac Super Cruise, a new generation of cameras will move beyond simply detecting whether the driver’s head is turned away from the road to include facial recognition and even be able to read emotions of passengers in fully autonomous cars.
Subaru introduced a feature on its all-new Forester, called DriverFocus, that uses facial recognition software to look for signs of driver distraction and fatigue. Part of the Subaru EyeSight suite of driver assists, DriverFocus can detect when someone is dozing off or looking away from the road for too long, and it will automatically stop the vehicle.
Another company, eyeSight (no relation to the Subaru option), has a camera that not only detects distraction and drowsiness but also includes what the company calls “contextual control” based on the direction of a driver’s gaze to highlight content in cockpit displays. A creepier profiling feature allows it to detect the age and gender of the driver and use the info for “connected car analytics.”
Renovo Auto is focused on creating a universal operating system for AVs called aWare. It incorporates sensor and software technologies and involves using cameras inside and outside a vehicle to capture emotions of passengers as well as pedestrians.
“During development, you need to monitor drivers to make sure they are engaged,” Renovo CEO Chris Heiser says. “And later you’ll need to interact with the passengers to help build trust and provide them all the services that a human driver does today.”
To provide this interaction, Renovo is working with AI startups Affectiva and Speak With Me to integrate their technology that analyzes the facial expressions and voices of passengers into the company’s fleet of AV test vehicles.
Now, if it could only make my teenage son more responsive and personal when I’m in the car with him for those 50 hours.