Simulation Supporting Certification

On the run-up to our World Congress in Stockholm, Sweden we ran a survey checking the simulation community's attitude to "Certification by Simulation". We wanted to know how far we were from having simulation replace test in the area of certification. It was a simple, blunt question designed to show the disparity of viewpoints we hear on this subject (Figure 1).

Here at NAFEMS, we get the chance to talk to organisations from across industry, and feel it is safe to say that in some areas simulation has already replaced test. Want an example? I recently talked to one regulator from the automotive industry who described how ray tracing was an acceptable way of clearing the passenger car rear
view mirror check. It is a pretty basic analysis. In this instance, numerical simulation can replace test as the technology is mature and the variability in the inputs that drive the simulation are low. The obvious reason for using simulation instead of test is that it reduces cost and time; by using simulation the fiddly, tedious, manual and time
consuming process of setting up a dummy and installing a camera to check the view field is eliminated. You can go on to make the case that the simulation is more useful than the test as the height of the driver can be adjusted easily without having to haul a dummy out of the car and spend time setting up a new one. Sure, errors can creep into a simulation but the same is also true with the test… was the dummy set up at exactly the right position? was the camera perfectly aligned? etc. There is variability on both sides of the simulation vs. test debate. There is some low hanging fruit, in some areas we can eliminate the need to test, and in these areas, I expect OEMS to be making the case to the regulator that simulation can replace test.

The aero industry is taking a long hard look at how Certification and Qualification by Analysis (CQbA) can be
put into practice. The European Union Aviation Safety Agency (EASA) recently closed public consultation on a
proposed certification memorandum that focused on providing guidance on the main certification aspects to
be considered when using modelling and simulation [1]. The EASA Certification Policy section includes the text;

"The use of Modelling & Simulation (M&S) techniques such as those based on computational Finite Methods, to support the showing of compliance with CS-25 structural certification specifications is becoming more and more widespread in the aerospace industry. In some cases these analytical techniques are proposed to replace (some of) the physical testing that otherwise would have taken place to support the showing of compliance. These M&S techniques have become more versatile and powerful in the last decade and have been accompanied by a significant increase in computational capabilities, to the extent that these techniques are now basically available to all applicants, to be applied to a range of physical phenomena at an affordable cost and effort."

But let us not confuse a promotion of certification by simulation with the retirement of testing. One of the
pillars on which CQBA stands is that the model needs to be validated! Testing is still required, but the focus is
shifting to what the testing is used for. Head over to the medical industry and in the US, you will see that the FDA
has been making great strides in addressing this topic. The evidence for medical device regulatory decision making comes from four different types of model; animal, bench (in essence a controlled setting), clinical trial and computer modelling. As with other industries there is the usual time and cost saving drivers, but here in the medical industry, there is a strong case to be made that simulation can allow situations to be assessed that simply are not possible with a clinical trial. Morrison et al [2] set out the FDA's perspective in the article "Advancing Regulatory Science with Computational Modeling for Medical Devices at the FDA's Office of Science and Engineering Laboratories". The article flags up the need for detailed reporting of the computational modelling as the regulatory staff are not
involved in the running of the models, and I'm sure the regulators in other industries will be in a similar position. Helpfully the FDA have addressed this issue by developing guidance on what information should be included in a report that documents a computational modelling study [3].

Waymo, the self-driving car project from Google, celebrated 10 Million miles of road testing in 2019. Hiding
in the shadows of this headline is the stat that 15 Billion virtual miles have also been logged via simulation. The
case for using simulation in the regulatory process for autonomous vehicles is simple; without simulation it isn't
possible to assess if the Automated Driver System (ADS) can operate effectively across all the potential scenarios it
will encounter. There is debate about the exact number of miles that need to be logged by a driverless car in the real
world before it can be considered safe. The RAND Corporation report "Driving to Safety – How Many Miles of
Driving Would it Take to Demonstrate Autonomous Vehicle Reliability?"[4] concludes that in order to demonstrate a
fatality rate of <1.09 fatalities per 100 million miles, it would take a fleet of 100 autonomous vehicles (driving for
24 hours a day, 365 days a year at an average speed of 25 miles per hour) 12.5 years! The 1.09 fatalities per 100
million miles is significant as this is the human performance equivalent in the US. The relationship between simulation and test isn't as clear cut here. We can't simply use simulation to identify the worst case scenario that should be tested. There are many situations which will be challenging to the ADS system and the
reason they are challenging, may at first glance not be obvious. It isn't just the multitude of road layouts and
potential traffic /pedestrian scenarios that need to be considered, all these different situations need to be
considered in conjunction with the infinitely variable combination of light and visibility (Rain, snow, dust etc.).
Developers and regulators are going to have to work together and in the US, the National Highway Traffic
Safety Administration has voluntary guidance that is intended to support the organisations working in this
area.

While it is fun to rehash the "when will simulation replace test" debate, the more productive conversation to have is
to look at how simulation and test can best work together. I see a lot of nice jargon for this, "Fusion of Analysis and
Test" [5], "Smarter Testing" [6,7,8]. In essence it comes down to:

1. Reducing the amount of testing that needs to be carried out. Do we need to test all the anchorage locations where the seat connects with the chassis of a bus or can we look, using simulation, to investigate how the load sheds throughout the structure and select the worst case scenario?
2. Getting more information, getting more value from your testing by performing a better test. One example of this is to use simulation to plan where the sensors should be placed to best capture the response.

So, here's my take on it all:

• We don't need simulation to replace test in the certification process. If simulation doesn't provide a robust alternative that can allow products to be developed more quickly and cheaply, that is great, we keep testing. In safety critical areas the big signoff test isn't going away. I can watch blade off signoff tests all day, who doesn't love the tension of seeing millions of pounds of kit deliberately written off (see the embedded video.
• At the most basic level, simulation supports certification by de-risking this final gate where a failure can cripple a company. For domains where the simulation is mature it is already replacing test. I know this comment will raise eyebrows but take a look at UN ECE Regulation 29 (Truck Cab Strength) [9]. In this document it states that certain tests can be eliminated if " .the manufacturer can show by computer simulation or calculations of the strength of the component parts of the cab or by other means to the satisfaction of the Technical Service that the cab will not undergo deformation dangerous to the occupants (penetration into the survival space) if subjected to the conditions of the tests."
• Across most industries I expect simulation to support certification by helping to identify the worst case which should be tested. This cuts down on the amount of costly testing and also improves the situation for the regulator by providing confidence that we are looking at the most onerous scenario.
• And finally I believe there are areas where simulation is required to support certification as there is no other game in town.

- Ian Symington, NAFEMS

References

Events

• NAFEMS World Congress 2021 incorporating the Int. SPDM Conference (25-29 October, Salzburg & Online)
• Visit the "Accepted Abstracts" page and search "Simulation Supporting Certification" for a listing of the 52 related presentation titles, presenters, and abstracts.