In this article, I’m going to explore the existing regulations and standards that form the foundation of trustworthy AI – and identify ways and means to enhance responsible AI. Many industries, for example, can learn from progress in the automotive sector. I’ll also reveal how testing options such as simulated environments can give confidence to ambitious companies as they widen their investment in automation.

But before I cut to the chase, let me provide some perspective by tracking the development of automation in the context of safety. Traffic lights provide a good example. Originally an operator manually switched between red and green, then human ingenuity created systems which changed colour as cars approached.

Developed in the pre-microchip days of the mid-20th century, they relied on a hardwired control system to serve simple logic loops. Nevertheless, these were automated systems – automatically performing a complex task without human interaction.

Roll forward a few decades and multiple controlled junctions with multiple traffic lights were connected to improve safety and traffic flow. The invention of microprocessors enabled more complex systems, with different sequences for morning and evening rush hour.

Again, this was an automated system, fully planned and defined by the human architect. Not a system you would consider ‘intelligent’ making its own decisions about how to manage traffic flow, but still one with the potential to do significant harm.

At today’s cutting edge, a whole city’s transport infrastructure can be connected to work together in harmony in an autonomous system. Now, the control algorithm need not be instructed directly by human, but rather a ‘neural network’ exposed to deep training and making its owns decisions about how to complete its task.

Here, the software’s structure mimics a cluster of biological neurones, or a very simple brain. It is self-taught to perform tasks through repetition over many thousands of examples, with the output used to continuously tweak the structure until it is considered fit for purpose. This is what we refer to as artificial intelligence.

Rogue AI is scary because we expect safety

AI is an immensely powerful tool able to tackle previously insurmountable problems. But the intelligence does have an anxiety inducing element, in that it ‘builds’ itself through the training process. Consequently, the inner decision trees cannot be easily followed like they can with traditional code.

This makes it harder to understand the logical process being followed and consequently track back to correct when mistakes or other undesirable emergent behaviour occurs. Why does rogue AI scare us so? I think it’s because we in our highly developed world take it for granted that the things we buy and the services we use are safe.

That intrinsic belief has been underscored over time by generally robust regulatory frameworks. Depending on legal variances around the world, the following generally applies:

• You are not allowed to sell your products unless they are certified fit for purpose
• State legislation lays down exactly what must be done to achieve certification, often in fairly broad terms when it comes to technical detail
• Published standards provide a more detailed technical requirement which your product should comply with
• A significant amount of paperwork will be required to demonstrate you meet the rigours of certification, covering every stage of the development process

Different people take different things from the above points. CEOs will think that if they don’t have certification, they don’t have a product, engineers will ask which standard they have to meet, and quality leads will worry that without good documentation from the start they’ll never get on top of the paperwork.

To manage this at Cambridge Consultants, we use a carefully managed development programme with risk management and documentation baked in. I’d like to tell you we do it because we’re great, and we are, but we also do it because we have to. It’s a requirement of the ISO 9001 standard which certifies us to deliver safe products.

ISO 9001 is one of more than 50,000 standards and guidance documents currently available from the International Organization for Standardization, ISO. Over the past 60 years the international framework of guidance has slowly evolved to encompass new developments and technologies.

Companies have developed their process to match them. Indeed, the same can be said for national and international directives, with the largest shakeup in that time probably being the introduction of the internet and mass dispersal of information.

The revolution in autonomous systems

Which brings me back to the main thread of my article. The status quo is now being questioned like never before, with the accelerated adoption of automation across multiple industries. Driverless cars are getting ever closer to everyday reality, pursued by a host of other automated products, in everything from food preparation and healthcare to agriculture.

Standards publishers are struggling to keep pace with this revolution. Many recognise their content is incomplete and are openly stating that further material will be added in coming versions. Leading the way here is automotive, probably as a result of the sector’s need for its products to be comprehensively insured against significant liabilities.

Where a published standard does not exist or cover a new product in development, we must take the relevant aspects of existing ones and apply and adapt as relevant. This is not a new process, but with increasingly complex systems it does mean drawing a larger web of references when compiling the safety case. Much of the good work done by the automotive sector can be taken and applied to other industries.

Just as products are changing, the way we develop them is being forced to change too. Long established processes are now finding themselves wanting. Take functional safety as an example. This is the act of designing safety into a system at its core to cover any anticipated failure.

Consider the air brakes on heavy commercial vehicles… if something goes wrong with the system a loss of pressure leads to the brakes coming on and bringing things to a halt. Similarly, for safety critical systems it is common to add redundant and tertiary backups.

This doesn’t work on modern autonomous systems though, where the final output behaviour can be the combination of multiple different feeds, all of which may be working nominally. Therefore, new approaches such as safety of the intended function (SOTIF) – which can account for sensor fusion and trained AI algorithms to mitigate risk – are required.

Similarly, conventional validation and verification techniques are becoming impractical. Just consider the logarithmic increase in scenarios which need to be evaluated with every small gain in autonomy and the way that small changes feeding into scenarios can affect a response.

Going back to the self-driving car, how many things affect a decision that it’s safe to pull out a junction? We don’t just consider the junction, but also lessons we’ve learnt on our way there, such as the behaviour of the driver next to us, the weight we are carrying in the boot or the amount of grip as a result of water on the surface.

Simulations for AI safety

To investigate all of these factors, multiple validation exercises are increasingly taking place inside computer-based simulations. Simulated testing allows the same scenario to be run multiple times with individual different variables adjusted and randomised, reducing the need for hazardous or costly physical hardware and speeding the process through running evaluations round the clock and in parallel.

If using this option, effort must be made to ensure and justify that the simulated test results are representative of real world equivalents.

Other techniques like Generative Adversarial Networks (GANS) can also be employed, to generate multiple training images for vision systems, for example. Using this simulated training data helps fill in real-world gaps and can be a tool in avoiding the inherent bias of conventional datasets.

It is important that as these new technologies develop, we consider not only the physical risks presented, but the less obvious societal ones.

These new simulated and virtual test techniques still need to be combined with traditional real-world testing as defined in the test plan. The ratios between real and simulated testing will vary depending upon the type of product being delivered and the level of confidence held in the aspects under review. As discussed, even with simulated environments it’s impossible to test modern systems against all the scenarios they may encounter. A risk-based approach must be used to identify key areas and select these for evaluation.

Use of a risk based approach is nothing new when assessing a product’s suitability for release, but when working with autonomous systems we must consider additional factors. They include events likely to trigger unexpected behaviour, our previous experience with similar systems and the likelihood of a malicious actor trying to disrupt the system. This disruption can come from someone simply walking out in front of a guided vehicle. But it might also be a more complex attack such as corrupting the training data used to build the AI so that it harbours a hidden fatal flaw which can be exploited when released.

So, in closing, I hope I’ve dispelled any simmering unease that the next instalment of the Terminator franchise will be played out in the real world. Yes, there are sophisticated challenges and potential obstacles ahead. But as I’ve revealed, there are ways through them for innovators across industries to develop trustworthy AI; responsible AI.

Existing safety regulations and standards are likely to have gaps, so we must pay them additional care. Testing is key and increasing options for using simulated environments are there to be investigated. Everything must be recorded in and presented in a well-structured safety case document.

Expert authors