While Artificial Intelligence, especially the generative type, is a highly disruptive form of IT innovation, its risk management still follows the same basic principles. We just need to extend those principles to new forms of data and software.

  • What are the assets?
  • What are the vulnerabilities and threats to it?
  • What is the damage that this can bring, realistically?

So let’s see how what the AI specifics are.

AI Assets

Let’s focus on the AI assets that are the product of deep learning. Deep learning neural networks have represented a shift in AI technology. Before that, symbolic AI was more common.

The reason to elaborate on this is that there is a fundamental difference between symbolic AI and neural network AI. Symbolic AI works by following programmatic rules, and is fairly deterministic. Neural networks are the result of training with large datasets. They have enabled generative AI which is much less deterministic.

Furthermore, and this is really significant from a security perspective, in its operation, GenAI mixes data and instruction to operate on that data deeply.

For example, anything an external user inputs to the system as data, might also be interpreted as an instruction. This represents a huge vulnerability for hackers to exploit.

The trained models represent a lot of information, and that information is often hard to identify, and therefore to see the classic information risks of.

Models are effectively software: you stick data in them, and data comes out. But because they also embody a lot of data (we are talking Gigabytes, Terabytes and beyond of training data) they combine the risks of software with the risks of data.

An AI system has a few more traditional and less traditional data assets associated with it. Base models, or foundational models, such as Llama, often get additional training or finetuning. The data used for that may well include proprietary or sensitive data.

Then there are system prompts and additional data sources that are fed into those models, together with user input.

AI Vulnerabilities

(this is a draft version)

The most fundamental vulnerability of Large Language Models is that they can’t be trusted to make a clear distinction between the data it works on and the instructions it gets to work on that data. As a result, all data that is unfiltered before being processed can be used to attack the systems that these LLMs are part of.

And this is one of the biggest reasons a lot of traditional IT risk controls fail to work. The principles remain sound, but controls can’t rely on old patterns anymore. For example, there is no such thing as a completely trustable AI gateway to protect from external risks.

(more text on actual damage will follow)