A machine learning model in production is three things at once: software, data and a statistical artefact whose behaviour was learned, not written. Ordinary application security covers the first. The other two add ways to attack a system that a web application firewall or a code scanner will not notice. This article lists those attacks and the controls that address them, in the order of a typical pipeline.
What can go wrong
Data poisoning
An attacker who can influence training data can influence the model. Poisoning may degrade accuracy in general, or it may plant a backdoor: the model behaves normally until it sees a specific trigger pattern. Pipelines that retrain automatically on user-submitted or web-scraped data are the most exposed.
Model supply chain
Pre-trained models are downloaded from public hubs and loaded with a single line of code. Some serialisation formats, notably Python pickle, can execute arbitrary code when a file is loaded. A model file therefore deserves the same suspicion as an executable from an unknown source.
Evasion with adversarial inputs
Small, deliberate changes to an input can flip a model's output while looking harmless to a person: a sticker on a road sign, a few altered bytes in a malware sample, a rephrased sentence. Any model that makes a security or safety decision should be assumed to face this.
Model extraction
By sending many queries to a prediction API and recording the answers, an attacker can train a copy of the model. The copy has commercial value, and it also gives the attacker a private test bench for crafting adversarial inputs against the original.
Privacy attacks
Models memorise some of their training data. Membership inference determines whether a particular record was in the training set. Model inversion and training-data extraction reconstruct parts of records. When the training data is personal, this is a data breach, and in India it falls under the Digital Personal Data Protection Act, 2023.
Prompt injection in language model applications
A language model cannot reliably tell its instructions apart from the content it is asked to process. Text inside a retrieved document, an email or a web page can redirect the model, make it reveal information or make it call tools with the attacker's parameters. The risk grows with every tool and data source the model is allowed to use.
Resource exhaustion
Inference is expensive. Long or specially crafted inputs can drive up latency and cost, which becomes either a denial of service or a very large bill.
Frameworks worth using
Four public references give structure to this work. MITRE ATLAS catalogues adversary techniques against machine learning systems in the style of ATT&CK. The OWASP Top 10 for Large Language Model Applications and the OWASP Machine Learning Security Top 10 list common weaknesses. The NIST AI Risk Management Framework and the NIST taxonomy of adversarial machine learning supply shared vocabulary for governance and for threat modelling. Mapping your controls to one of them makes gaps visible and gives auditors something they recognise.
Controls, stage by stage
Data
Record where every dataset came from and who can change it. Restrict write access to training data as tightly as write access to source code. Validate incoming data against a schema and against statistical expectations, and quarantine outliers for review before they enter training. Keep a trusted, access-controlled holdout set that never passes through the automated pipeline, and use it to detect a poisoned retrain.
Training
Run training in an isolated environment with pinned, scanned dependencies. Version the code, data and configuration together so that any model can be rebuilt and compared. Sign the resulting artefact, and store it in a registry that records who approved its promotion.
Model artefacts
Prefer formats that hold only weights, such as safetensors or ONNX, to formats that can carry code. Verify checksums and signatures of third-party models, scan them before loading, and load them first in a sandbox without network access.
Serving
Authenticate every caller and apply rate limits per identity. Return only what the client needs: a label or the top few classes, with rounded scores, reveals much less than a full probability vector. Validate input size and type before it reaches the model. Monitor query patterns for signs of extraction, such as high volumes of systematically varied inputs from one account.
Robustness
Where evasion is a realistic threat, test the model with adversarial examples during evaluation and consider adversarial training, accepting some loss of clean accuracy. Use calibrated uncertainty so that the system can abstain on unusual inputs and pass them to a person. Do not let a single model be the only barrier in a security decision.
Privacy
Minimise personal data in training sets, and de-identify what remains. For sensitive data, differentially private training limits what any single record can contribute to the model. Test for memorisation before release by checking whether the model reproduces known training records.
Language model applications
Treat model output as untrusted input to everything downstream: never pass it to a shell, a database query or a browser without validation. Give the model the least privilege that the task allows, with separate credentials for each tool. Require human approval for actions with consequences, such as sending email, moving money or deleting data. Keep retrieved content clearly separated from system instructions, filter both inputs and outputs, and log interactions for investigation.
| Threat | Stage | First control |
|---|---|---|
| Data poisoning | Data, training | Provenance, write-access control and a trusted holdout set |
| Malicious model file | Artefacts | Weights-only formats, signature checks, sandboxed loading |
| Adversarial inputs | Serving | Adversarial testing, abstention, a second independent check |
| Model extraction | Serving | Authentication, rate limits, reduced output detail |
| Membership inference | Training, serving | Data minimisation, differential privacy, rounded scores |
| Prompt injection | Application | Least-privilege tools, output validation, human approval |
| Resource exhaustion | Serving | Input limits, quotas and cost alerts |
Monitoring and incident response
Drift monitoring serves security as well as quality. A sudden change in input distribution may be seasonal, or it may be an attack in progress, and the same dashboards show both. Log inputs and outputs with appropriate privacy protection so that an incident can be reconstructed. Keep the previous model version ready for rollback.
Write a response playbook for model-specific incidents before one happens. It should say who decides to roll back, how a poisoned dataset is traced and cleaned, and when a model that leaked personal data must be retrained. If the incident involves a reportable breach, the CERT-In reporting timeline and the obligations of the Digital Personal Data Protection Act apply, as they would for any other system.
A sensible starting order
Teams that are new to this can make quick progress in four steps. First, put authentication and rate limiting in front of every model endpoint. Second, switch to weights-only model formats and verify everything downloaded. Third, lock down write access to training data and add a trusted holdout check to the retraining pipeline. Fourth, for language model features, remove any tool permission that is not strictly needed and add human approval to the rest. Threat modelling against MITRE ATLAS then shows what remains.
Our AI and machine learning practice builds these controls into the pipelines it delivers, and our cybersecurity practice assesses existing machine learning systems, including models built by other teams.