Synopsis: Ensuring ADS are safe is a major challenge for the broad deployment of automated driving systems. Emerging standards and regulations will help direct the efforts of ADS developers in testing, scenario description and more, but how can we collect viable safety metrics, expose the unknown unknowns and comply with new regulations that recommend a combination of physical and massive-scale virtual testing? This presentation will describe these challenges and offers pragmatic tools along with coverage-driven verification methodology as a way to supply quantitative safety metrics, expose unknown unknowns, describe scenarios in a reusable and manageable way, and more. It will demonstrate the use of coverage-driven verification methodology and the Foretify platform for the implementation of the new ALKS regulation package.

Synopsis: HEADSTART is an H2020 EU-funded project that aims to define testing and validation procedures of connected and automated driving functions including key technologies such as communications, cybersecurity and positioning. The tests will be in the simulation and real-world fields to validate safety and security performance according to the key users’ needs. The expected impact of the HEADSTART project is based on three main action pillars: testing and validation – potentiation of development strategies bringing time and cost reduction; assessment – creation of assessment protocols increasing vehicle safety awareness; certification – support of regulations ensuring the safe introduction of CAD technologies to the market.

Synopsis: One of the main challenges of validating ADAS and AV systems is analyzing the data mountains of real drives to identify the gaps in the data. When the gaps are identified, it is possible to synthetically generate the missing data to achieve full coverage for training and validation of ADAS and AV stacks. In this session we will explore how we ingest real data in order to find those gaps, and how we parameterize our simulation platform to generate meaningful synthetic data for perception, prediction and control stacks.

Synopsis: As autonomous vehicles (AVs) increase in maturity, the complexity in ensuring they are safe increases as well. The traditional automotive testing methodologies need to evolve to suit the ever-changing nature of AVs. This will bridge the gap between regulators and AV developers, and eventually lead to safe and effective implementation of AVs. In this presentation, CETRAN will present the key challenges it faces when testing Level 4 AVs, and will share its approach and the ongoing research/projects to tackle these challenges. The focus will be on the current unresolved issues in virtual and physical testing.

Synopsis: Development of AD/ADAS is a massively complex process, requiring the management of vast data sets. In turn, this leads to subsequent iterative development using HiL/SiL performance testing against defined test cases, and the development of real-time inference engines. A growing development ecosystem, including toolchains, algorithms, public training decks, annotation services and more is coming but requires carefully managed security and access to highly valuable, company-confidential information. An approach that offers access to multiple public cloud services at the lowest cost but with the greatest agility is essential to take a leading position in this highly competitive market. This webinar will discuss best-in-class methodologies for leveraging public cloud, shedding light on the tradeoffs and technical roadblocks introduced and how to avoid them.

Synopsis: Volvo Cars has developed cars with ADAS since the beginning of 2000. The company's ambition is and always has been to make safe cars. How does the SOTIF ISO/PAS 21448 affect active safety development? Are we perhaps fulfilling the SOTIF PAS already? How did we handle non-ISO hazards before SOTIF PAS existed? The presentation will explore some use cases/examples where SOTIF affects the product.

Synopsis: Due to the high degree of connectivity of highly automated and autonomous vehicles, they must function in a variety of scenarios during the utilization phase. Providing a realistic test environment that covers these usage scenarios is a major challenge for virtual engineering of autonomous driving functions. Especially for the virtual validation of the driving functions, this requires a large number of simulation models of external systems involved, to which not every developer has access. However, AI procedures can possibly be used in future to reduce the effort required to generate the necessary simulation models.

Synopsis: Adaptive driving beams incorporate technology that essentially removes the compromise between glare and illumination by eliminating the driver’s need to frequently switch between lower and upper beams. This technology automatically recognizes the lighting need and provides improved response time to pedestrians, animals and other unexpected objects in the road. Additionally, it is of great importance to the future of mobility by contributing to a more controlled optical environment for camera-based sensor technologies used for automated vehicles. But, as with all new technology, there are questions about how to adapt your testing program to accomplish the goals, as well as meet applicable standards. During this presentation, we will touch on a variety of topics, including adaptive driving beams and how they will affect the future of mobility, SAE J3069, alternative and upcoming regulations, and much more.

Synopsis: The Automotive Grade Linux (AGL) community consists of more than 150 companies across the automotive and tech industries who are working together to develop an open-source software platform for all in-vehicle applications from infotainment to autonomous driving. Sharing a single software platform across the industry decreases development times so OEMs and suppliers can focus on rapid innovation and bringing products to market faster. This talk provides an overview of AGL, production use cases including Toyota and Subaru, the project roadmap and how to get involved.

Synopsis: The presentation will outline the challenges, lessons and successes of Robopilot – a UK CCAV-funded project delivering a demonstration of Level 4 autonomous driving on mixed roads in the UK. It will focus on the testing and validation journey from research and simulation to on-road testing and live demos. Robopilot is a £12m consortium project based in the UK. Partners include UPS, Thales, Bristol Robotics Lab, Loughborough University, TVS and South Gloucestershire Council.

Synopsis: A fundamental challenge for the autonomous vehicle industry is how to deliver quality training and test data more quickly and cost-efficiently. rFpro brings the real world to simulation, where our highly accurate digital models provide the perfect environment for training, testing and validating AV and ADAS systems. What if you then need to expand your data production and deliver synchronous simulation across multiple sensors? One of our approaches, Data Farming, is being employed by Tier 1 suppliers and AV technology providers, including Denso and Ambarella, to achieve exactly that. Join us to hear more about how this approach is being successfully trialed with industry leaders.

Synopsis: Adaptive driving beams incorporate technology that essentially removes the compromise between glare and illumination by eliminating the driver’s need to frequently switch between lower and upper beams. This technology automatically recognizes the lighting need and provides improved response time to pedestrians, animals and other unexpected objects in the road. Additionally, it is of great importance to the future of mobility by contributing to a more controlled optical environment for camera-based sensor technologies used for automated vehicles. But, as with all new technology, there are questions about how to adapt your testing program to accomplish the goals, as well as meet applicable standards. During this presentation, we will touch on a variety of topics, including adaptive driving beams and how they will affect the future of mobility, SAE J3069, alternative and upcoming regulations, and much more.

Synopsis: Many research and innovation, testing and piloting activities are being carried out independently across Europe either through EU-funded consortia projects or at the national level driven by member states, industry or public-private partnerships. The lack of alignment and common approaches often results in duplications or overlaps, prevents comparability of results and the development of interoperable solutions, and hampers the harmonized deployment of CAD in Europe. In an effort to facilitate the exchange of lessons learned and best practices, as well as the identification of synergies and gaps between projects, the EU-funded ARCADE Coordination and Support Action has set up a comprehensive knowledge base on CAD-related activities in Europe and beyond, which is currently being populated in collaboration with the EU CCAM Single Platform. The project is also organizing concertation events to support consensus-building across stakeholders on methodologies or the identification of next R&I actions on CAD. This presentation will provide an overview of the CAD knowledge base and key outcomes from other harmonization-related activities of the project on methodologies or description of use cases.

Synopsis: In 2018 ASAM acquired its first simulation standards in the form of the OpenX portfolio (openDRIVE, openCRG and openSCENARIO), with the addition of the Open Simulation Interface in 2019. Since then, simulation experts worldwide have been working hard on the first official ASAM revisions, the first of which are to be released in Q1 of 2020. This presentation will give an overview of the ASAM activities to date, and provide some insight into where we see the road leading in 2020 and beyond. Our goal is standardization to facilitate the development of safe, regulated autonomous driving.

Synopsis: Testing of CAVs on public roads is a hot topic. Does it expose the public to unnecessary risk or is it essential for profound safety improvement on the world’s roads? Can we have the best of both worlds: on-road learning with enhanced safety? The presentation addresses these questions, using the £35m Midlands Future Mobility test and trialling ecosystem as a case study. There are exciting times ahead!

Synopsis: Testing is fundamental to the safety of automated driving software, but driving billions of miles to achieve sufficient scenario coverage is unfeasible and requires a better approach. Scenario sharing across the industry is gaining support as being the solution. With this aim, the standardization of scenario description is being worked on but we are still some way from a universal standard. This puts the system architect in a difficult position: how to design a scenario database that will support today’s standards as well as those of tomorrow? As part of the UK’s Midlands Future Mobility project, we answer this question.

Synopsis: Automated driving technology is developing at an unprecedented rate – a quick response is needed from regulators in order not to obstruct innovation and at the same time ensure market access to safe automated vehicles (AVs). The introduction of innovative certification approaches is required to assess and verify AVs safety, complementing conventional physical testing in order to ensure safe vehicle performance in real life. This presentation will introduce the proposed EU method for automated driving, including audit of the manufacturer design/development process, verification through track and on-road testing, and operational feedback from real-life experience after market introduction.

Synopsis: Most new vehicles sold in 2018 (56%) had, as either a standard or optional feature, advanced driver assistance systems (ADAS) equipped with pedestrian automatic emergency braking (PAEB). However, current testing protocols implemented by safety organizations and programs, such as Euro NCAP, NCAP and IIHS largely do not evaluate the performance of PAEB in dark conditions without streetlights. This is true despite data from the National Highway Traffic Safety Association (NHTSA) showing that more pedestrian fatalities occur in dark conditions without streetlights than in daylight conditions. Additionally, independent tests conducted by NHTSA and the American Automobile Association (AAA) reveal that in every scenario considered, PAEB systems frequently fail to protect pedestrians in dark conditions. Within this context, to encourage improvement in PAEB performance and increase night-time pedestrian safety, this presentation includes the following sections: 1) A proposal for the expansion, in future testing protocols, of PAEB tests conducted in dark, night-time conditions. More precisely, we propose expanded testing in less than 1 lux (lx) ambient illuminance, using low-beam headlights, and without streetlights; 2) Results of Velodyne’s night-time PAEB tests comparing a lidar-based system with one example of current camera and radar-based technology; 3) An overview of Velodyne’s PAEB solution.

Synopsis: We present a holistic approach that takes recorded traffic scenario instances and yields 'good' test scenarios for automated and autonomous driving systems. Such test scenarios are usually generated from scenario types, for which we present an approach that allows measuring both the test case quality and system behavior. Since this requires completeness of the list of scenario types, we provide both a statistical model and a methodological approach to assess completeness. To achieve the latter, we automatically derive scenario types from real data, which complements current manual scenario derivation. We show technical solutions for each of the steps presented.

Synopsis: The PEGASUS project developed and demonstrated a method for the validation of automated driving functions for the highway domain. Two projects currently elaborate on this approach and apply it to the far more complex urban environment. Simulation is supposed to provide the bulk of evidence for the homologation of the vehicles. For that, the simulation must be adaptable to various tasks in the verification and validation chain. And, of course, the simulation results must be validated. One project, SET Level 4-5, is developing simulation technology based on a modular architecture with standardized interfaces. The other, VVMethoden, covers the full development lifecycle and employs simulation technology. The talk will present the intended role of the simulation and the projects' approach to providing the technology with the desired features.

Synopsis: Many OEMs and other automated driving companies are collecting massive amounts of driving data to identify what scenarios the automated vehicle might have to deal with. Through scenario extraction, repeated driving patterns are categorized and turned into statistics essential for effective safety assessment. But when is the data collection enough? The TNO StreetWise scenario database includes completeness indicators at various steps in the scenario mining pipeline. We will introduce the meaning and application of these completeness indicators. In this way, OEMs can compare coverage of their data collection and quantify the completeness of the collected data.

Synopsis: The challenge of autonomous vehicle validation relies on massive simulation, due to the vast number of kilometers to run in various road, traffic and weather conditions. Nevertheless, acceptance of the proposed automation system will have a heavy impact on the use of the system and the efficiency of handling it when sharing or taking back vehicle control. Driver-in-the-loop simulation will play an essential role, using high-performance driving simulators or dedicated configurations, including virtual reality and web-based online solutions. These simulation tools will be essential to complete efficient autonomous vehicle engineering design for driver acceptance and comfort.

Synopsis: To prove that automated driving systems (ADS) are safer than human drivers, it is suggested that they need to be driven for over 11 billion miles. A number of miles is not an appropriate metric and doesn’t guarantee absolute safety. This highlights the question: “How safe is safe enough?” To answer, we suggest a departure from the world of absolute safety to informed safety. A key aspect of informed safety includes an accurate and standardized definition of the operational design domain for an ADS, including conveying it to users, regulators and other stakeholders. The first step in all safety standards: ODD definition.

Synopsis: A new concept for an autonomous driving system is presented, which in particular protects other road users such as pedestrians and cyclists. Furthermore, the system automatically ensures compliance with traffic regulations in a smart city equipped with it. It goes far beyond what we have seen so far with lane-keeping or automatic braking systems. The concept of geofencing will be further developed with unprecedented granularity. Detailed maps of the traffic areas of a city with 4in resolution will be used.

Synopsis: Homologation of automated driving functions presents a huge challenge for their market introduction. Existing regulatory safety frameworks applicable to conventional vehicles and their components are insufficient to fully assess the operational characteristics of current and future automated vehicle technologies. With increasing automation, vehicles transform into cyber-physical systems that no longer require a human driver; therefore, new safety challenges will have to be considered. This presentation discusses those challenges, provides an overview of the current regulatory and standardization work in progress and explains the possibilities for how to approve automated vehicles for public roads today.

Synopsis: To tackle the problem of verification and validation of autonomous vehicles, synthetic data generated in simulation environments can be employed to complement field testing, due to the fact that these environments are highly flexible and inexpensive. Although in recent years virtual environments have been employed by the community to develop and test ADAS/AD functionalities, such as AEB or ACC, there has been limited possibility to utilize synthetic data for sensing functions performance evaluation. This presentation will address the challenges and opportunities of simulation environments for verification and validation of camera-based sensing algorithms.

Synopsis: In an attempt to be as close to reality as possible, the most advanced AD and ADAS virtual test environments integrate real-world components. This HIL approach requires innovative new tools and methodologies to ensure smooth data flow between the virtual domain running synthetic scenarios, and the attached ECU real-world interfaces. This presentation will outline the challenges of quick and precise translations between physical and virtual domains within various HIL simulation setups for high-bandwidth video camera systems, and showcase how to overcome them with Xylon’s flexible and configurable single box solution.

Synopsis: ADAS/AD providers use artificial intelligence as one key component. Researchers and developers who can deliver insights faster while managing rapid infrastructure growth will be poised to be industry leaders. Following the process of data acquisition from test drives via ingest to the data center and processing of big data, many challenges occur. To handle this unstructured data, seamless scaling systems and toolsets are necessary from the sensor to the data center with storage and computing systems. This combined presentation provides a case study of how synchronized data capturing in test drives, big data computing, storage and archiving are integrated in today’s ADAS/AD development workflow.

Synopsis: The presentation will address the key challenges of safety validation of automated and connected vehicles from a methodological and standardization perspective. It will also deal with the need for and setup of an alliance between key standardization institutions and third-party technical service providers as well as technology and infrastructure developers, providers and operators. The latest outcomes as well as further focus areas and topics of the alliance will also be presented.

Synopsis: To pave the ground for scalable and cost-efficient test and validation of AD functions by real testing on public roads, Joanneum Research is producing Ultra High Definition Maps (UHDmaps) based on mobile mapping data in a scalable automated workflow. UHDmaps contain a digital copy of reality, which sets the benchmark for the digital assessment of automated driving functions. The depicted solution is already utilized by the Austrian Light Vehicle Proving Region for Automated Driving (ALP.Lab GmbH) and its partners. The presentation will give an overview of the technical solution and certain test use cases.

Synopsis: ADAS/AD software development needs to cope with complex sensor systems, plentiful corner cases still to be discovered and a cumbersome number of kilometers to test/certify. These tasks require high support from virtualization to be achievable under challenging deadlines and at a reasonable cost. The perfect tool that gathers all your requirements does not exist. However, many high-quality software solutions are able to simulate sensors, traffic, vehicle dynamics, driver behavior and realistic environments with the level of detail that every project needs. At Porsche Engineering, we bring together the best tools, data sources and our experience in automotive, to create a flexible simulation platform to support end-to-end ADAS development.

Synopsis: The adoption and acceptance of autonomous vehicles in society depends heavily on the ability to have confidence in the systems and features providing autonomous functionality. This requires a significant focus on test, including test strategy, test workflow and test assets. However, testing autonomous systems is extremely challenging and novel, because these systems under test are developed and based on machine-learning algorithms and operate in a near-infinite state-space. This presentation will identify some strategies that vehicle manufacturers and suppliers employ to tackle this challenge, from simulation-heavy test to deployable hardware test solutions. It will cover the intricacies of simulation and areas of investment that have not been adequately addressed today.

Synopsis: Use cases for Level 3 and 4 active safety and autonomous driving validation are skyrocketing, and the number of required certification scenarios is growing exponentially. Since vehicles are the anchor of these simulations, it is imperative to leverage a high-fidelity model that can scale into all simulation domains without changing simulation methodologies. This presentation will demonstrate how high-fidelity, OEM-validated solvers can be integrated into desktop simulation tools and easily be migrated to hardware-in-the-loop systems.

Synopsis: As autonomous vehicle programs prepare to deploy their systems on public roads, it is critical to ensure that the systems have been fully tested against the requirements, and that there’s complete traceability of the test results. Furthermore, testing and managing technical risks becomes more challenging as the programs work with multiple third-party vendors/partners. This discussion explores approaches to requirements testing, traceability, and using simulation to integrate multiple partners and make safety cases.

Synopsis: This paper will highlight the challenges of going from hardware-in-the-loop (HIL) to vehicle road testing as there are gaps today that can only be resolved through live road testing. By enabling the inclusion of actual sensor modules and wireless interfaces, Autonomous Drive Emulation (ADE) will be a way to compress the time and cost of vehicle road testing. As an example, this paper will discuss some of the details of how this can be accomplished from a cellular vehicle-to-everything (C-V2X) perspective.

Synopsis: ADAS/AD providers use artificial intelligence as one key component. Researchers and developers who can deliver insights faster while managing rapid infrastructure growth will be poised to be industry leaders. Following the process of data acquisition from test drives via ingest to the data center and processing of big data, many challenges occur. To handle this unstructured data, seamless scaling systems and toolsets are necessary from the sensor to the data center with storage and computing systems. This combined presentation provides a case study of how synchronized data capturing in test drives, big data computing, storage and archiving are integrated in today’s ADAS/AD development workflow.

Synopsis: As autonomous vehicle programs prepare to deploy their systems on public roads, it is critical to ensure that the systems have been fully tested against the requirements, and that there’s complete traceability of the test results. Furthermore, testing and managing technical risks becomes more challenging as the programs work with multiple third-party vendors/partners. This discussion explores approaches to requirements testing, traceability, and using simulation to integrate multiple partners and make safety cases.

Synopsis: Validating AV safety is a crucial part of ongoing research. The approach of separately track-testing sensors and driving algorithms is hardly sufficient to demonstrate AV safety. Scenario-based simulation approaches are necessary complements to the traditional approach, allowing computation of a controlled diversity of key variables in many iterations in a safe, fast and documented way. French car manufacturers Renault and PSA, together with academic researchers (VEDECOM, SystemX, Lab and Ceesar) and other partners (Valeo, AVS and Expleo), propose to address the challenge of demonstrating AV safety by taking into account an array of 'in-the-field' situations.

Synopsis: Execution of hundreds of thousands of test runs with a trustworthy virtual vehicle in a cluster or cloud environment is a challenge in itself. With Simcenter Prescan360, we introduced an off-the-shelf solution to generate scenario variants, orchestrate the massive numbers of test runs and post-process these results for easy interpretation. However, by adding additional smartness, the process can be optimized and automated much further. In this presentation we will highlight how design exploration and AI techniques are applied to create critical but unseen scenarios that might make the system fail. We will also introduce smart metrics to automate accurate safety assessments of the automated driving functional performance.

Synopsis: Automated driving solutions introduce new complexity into the development of embedded systems in cars. This complexity rises with each level of control and autonomy. The toolchain for such challenges is also complex and the integration of all the tools requires considerable effort without a real competitive advantage for the automated driving solution. Instead of solving these challenges alone, wasting lots of money along the way, Bosch's automated driving division has started an open-source community known as OpenADx. This talk will present the open-source approach, the current state of the community and the currently available solutions.

Synopsis: The verification and validation (V&V) of system safety are among the greatest challenges in the autonomous vehicle (AV) industry. For the associated homologation process, it is essential to bridge the gap between the industry's expertise and the regulators’ desire to define requirements and policies that safeguard AV development. Based on several years' experience that dSPACE has gained in the context of hardware- and software-in-the-loop simulation as well as release testing, a proposal for a comprehensive end-to-end V&V process is presented.

Synopsis: Automated driving systems (ADS) require solving a multitude of capabilities including perception, decision making and planning in real time. Each of them represents a challenge on its own, and researchers usually focus on one while abstracting away the rest of the system. In this presentation, we will introduce the AD-EYE platform, which has been developed under several EU projects and with the collaboration of multiple industrial partners. The goal of the project is to provide a simulation platform for ADS-related simulations with common base functionality that can be modified as per need. The aim is to provide better integration of the different projects by letting research groups work in a common environment. AD-EYE targets, in particular, dependability-related evaluation of architectures and algorithms for highly automated vehicles; as a prominent feature, it provides a configurable safety supervisor architecture. The talk will present the current status of the platform, illustrate its use and discuss its current roadmap.

Synopsis: New forms of cooperation are necessary to turn the vision of autonomous driving and fully connected mobility systems into reality. Virtual validation is an essential part of the development process. Standards for model and system interchange are vital for cross-company and cross-domain virtual integration and simulation of HAD functions. Standards like FMI, SSP and the OpenX at ASAM eV showcase current possibilities, challenges and future directions, as well as a vision of future collaborations. The foundations for the future ecosystem have been laid by the ENVITED ecosystem: standardized data for virtual test and validation; open and modular simulation architecture; traceability of standardized data and test results for a virtual proof of validation.

Synopsis: On October 8, 2020, Waymo opened its fully autonomous ride-hailing service to the general public in Phoenix. Right now, members of the public are hailing vehicles with no human driver controlling the car – either in the vehicle or remotely – to help them get to where they’re going as part of their everyday lives. This milestone is not just the output of many years’ experience in developing the Waymo Driver – it’s also the result of over a decade developing safety practices. Since its start in 2009 as the Google Self-Driving Car Project, Waymo has spent years refining a comprehensive set of robust methodologies that assess safety across its technology, operations and team culture, and ultimately guide the deployment and safe operations of the Waymo Driver. In this talk, Dr Trent Victor, Waymo's director of safety research and best practices will discuss the safety framework that guides Waymo's progress and is the first of its kind in the AV industry.

Synopsis: The technological challenges/opportunities of CAV/CAM and the roadmap to autonomy are maturing. However, the requirements to generate market pull and the skills required for the successful implementation and uptake of services are less well understood. For example, safe ongoing operation of vehicles needs new MOT tests addressing security, software and data privacy; new and disruptive ownership models present issues and opportunities regarding individual design and brand identity; trust and perception require more human-centered design for viable solutions. These and more questions need answering if economic, environmental and productivity projections are to be realized.

Synopsis: As the industry nears technical readiness in the development of automated vehicles, questions remain about how safe is safe enough. What does driving safely even mean? Many industry standards have been created that can assist in a safety assurance argument. But only one standard provides guidance on how we should understand what is reasonable to expect of others. This talk will explain the critical role that assumptions play in understanding what 'driving safely' means for an automated vehicle, and what work remains in order to fully realize a driverless future.