Pick the audit log fields a guardrail must persist to satisfy a typical AI act audit
Audit logs need reproducibility, attribution, and traceability, model and prompt hash, classifier verdicts, tool outcomes, and tokenised identity.
An audit log is like the flight recorder on a plane. It should let an investigator months later reconstruct what happened: which plane, which version of the software, which weather, which actions the pilot took. It is not a place for the pilot to write their feelings about why the flight went sideways, and it is definitely not a place to record the passengers' personal data in plaintext. Same with LLM audit logs. Record the facts that let a reviewer reconstruct the run, attribute it to a real (but tokenised) user, and trace every safety decision back to a rule and a classifier version. Speculation and raw PII both get the log thrown out by an auditor.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Audit logs are the part of the safety stack that engineers find boring and regulators find central. The boring framing is wrong. A good audit log is the difference between a clean regulatory review and a multi-quarter remediation project, and the difference between a defensible incident response and a litigation-discoverable hypothesis chain. The schema is not academic, it is the contract between your engineering team and the audit, security, and legal teams who will read these records cold months or years after the events.
The right way to design the schema is to imagine the auditor. They are reading one record, they do not know your codebase, they do not trust your good intentions, and they want to answer three questions: can I reconstruct what happened, can I tie this event to the safety rules that should have applied, and can I attribute the event to a real person without violating any data-protection rule. Every field in the schema either serves one of those three questions or it does not belong.
Reproducibility, making the model run rerunnable
The first job of the audit log is to let a reviewer reconstruct the model run. Without reproducibility, every safety decision in the log is dangling: you cannot tell whether the model behaved as expected, you cannot replay the case in a debugger, and you cannot test whether a later mitigation would have changed the outcome.
The minimal reproducibility manifest has: the model id and version (vendor model identifier plus the exact snapshot date for hosted models, or the model-weights hash for self-hosted), a hash of the exact system prompt used, the decoding parameters (temperature, top-p, max tokens, frequency-presence penalties, seed if supported), the prompt-template version, the tool-registry version, and the framework version. Each field is small. Together they form a manifest sufficient to rerun the request against an equivalent stack, modulo non-determinism inherent in the model, which the seed field mitigates where supported.
The system-prompt hash deserves attention. Production system prompts change weekly or more often in mature products. Logging the full prompt inline on every request multiplies storage cost and makes redaction harder (system prompts often contain configuration secrets). The pattern is to store the full prompt in a content-addressed object store and log only the hash; the auditor can resolve the hash to the prompt content via a documented lookup procedure. This also gives you a natural history of prompt versions over time, which is its own audit asset.
Reproducibility is also where vendor request ids come in. Anthropic and OpenAI both return a request id with every model call. Logging that id lets you cross-reference the vendor's own logs (which are subject to their retention and disclosure policies) with your local record, which is useful in incident response and required in some regulated industries.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- The EU AI Act Article 12 logging requirements for high-risk AI systems specifically expect reproducibility and traceability fields of the kind described.
- AWS Bedrock Guardrails and Azure AI Content Safety emit structured policy-decision events with classifier id, score, and rule fields that map directly to audit-log requirements.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design the retention policy difference between the main audit log and the PII vault?
Main log retention is set by audit and regulatory requirements (often years), holds only tokenised identifiers and structured event fields, and is read frequently by security and compliance teams; PII vault retention is set by GDPR data-minimisation and user-rights requirements (often shorter), holds the raw data behind the tokens, is read rarely and only under documented investigation workflows, and has its own access-audit log.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating the audit log as a debug log. Debug logs collect free text and full payloads for engineers. Audit logs are structured, redacted, and designed to survive cross-examination by a regulator.
60 second bullets to scan on the way to the call.
The three properties of an audit log: reproducibility, traceability, attribution
Why the system-prompt hash beats storing the full prompt inline
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.