
What four decades of aviation automation can teach us about AI governance, automation bias and building your own AI flight recorder.
Let me start with what may seem an unusual question. If AI gets it wrong, can you still fly the plane?
Hear me out. Aviation has spent more than four decades, and paid some very high prices, learning how to work with capable automation. One of its hardest lessons is this: automation reduces our workload, but as it becomes more reliable, something unexpected happens. We become less inclined to question it. We start by using these systems to support our judgement. Over time, we begin treating their answers as our judgement.
That shift, from "AI helped me decide" to "AI decided", is exactly where governance starts to matter more and more. In this article I'll use three aviation incidents, a couple of anonymised cyber incidents and a few lessons from ocean racing to show how you can build a "digital black box" for AI: an AI flight recorder that explains not just what AI recommended, but what people then decided, and what information is available when it all goes wrong. At the end you'll find a link to a one-page AI Governance Handbook you can put to work straight away.
A goal of good risk management is to remove or reduce bias. When it comes to automation and AI, two biases come to mind straight away: confirmation bias and automation bias. NASA has published extensive research on automation bias, which is using an automated recommendation as a substitute for independent verification. Aviation has researched this for decades; the rest of us are discovering it now that AI is in almost every tool we use.
Automation bias shows up in two dangerous ways. The first is a commission error: following wrong advice, such as accepting a convincing Copilot or agent recommendation without checking whether it's correct. The second is an omission error: missing something the AI didn't flag. A reviewer assumes there's no issue because the system found none. But what if the system simply wasn't looking for that problem?
Here's the encouraging part. Research found people made fewer errors when they were explicitly accountable for the accuracy of a decision, not just for following the process. And here's the paradox: the better a system performs normally, the more dangerous our reliance can become. Every correct recommendation reinforces trust, and over time we stop gathering independent evidence and simply assume the AI is getting it right.
Consider Asiana Flight 214 at San Francisco on 6 July 2013. A precision guidance system at the airport was unavailable, so the crew needed to manage the approach without that aid. The aircraft became too slow and too low, struck the seawall, and three passengers died. Investigators pointed to the crew's misunderstanding of the aircraft's automation logic, inadequate monitoring and delayed corrective action among the contributing factors.
So ask yourself: do your people have the authority, the time, the evidence and the expertise to challenge what your AI systems are telling them?
So how do capable professionals become so dependent on automation? I use six stages as a discussion tool, nothing more. It usually begins with "AI did it, it's probably right... it's so good now."

Stage one is assistance: "AI helped me analyse this." The person is engaged and thinking critically. Stage two is reliance: "I usually ask AI to analyse this." A dependency is forming. Stage three is bias: "The AI says this is the answer." Stage four is complacency: "I don't need to check, it's normally right." Stage five is loss of proficiency: I'm no longer confident I could recognise a wrong answer. Stage six is out-of-the-loop failure, where we can no longer explain what the system is doing or why.
In the late 1990s aviation had a name for this: "children of the magenta line". New pilots had become so dependent on the magenta course line on their navigation display that they stopped asking the basic questions. Where am I? Where am I going? Does this make sense? The earlier pilots who coined the phrase had built their own picture from charts, headings, time, position and conditions. The lesson is to know your domain, because you cannot challenge what you do not understand.
What behaviours should you look for in your own organisation? Are integrity checks quietly disappearing from IT or software review cycles? Can reviewers explain their approvals? Do your teams know how to intervene when the system is wrong?
There are two different questions here. The first: will we recognise when AI is actually wrong? The harder one: even if the system is working exactly as designed, are its assumptions still valid for this specific decision or use case?
A tool can process information correctly and still miss a change in context. Asiana 214 is a sobering example. The automation wasn't simply "broken", and that wasn't the root cause; the crew's understanding, monitoring and intervention didn't match the situation, and the results were devastating.
The comparison makes our responsibilities very clear. Autopilot assists the crew; AI assists the decision-maker. Instruments present information; AI presents analysis and recommendations. Pilots cross-check their instruments before take-off, and in the same way we must verify AI outputs against authoritative sources, in the context of each use case on our register. Humans in the loop, yes, but knowing what the automation does and how it works is just as critical, especially when it goes wrong.
Accountability never transfers to a machine. Aviation taught us that, and recent agentic misalignment incidents, where AI agents have reached sensitive data, keys and secrets they should never have touched, are teaching it again. Assessors, risk and security teams need to understand tooling limitations, retain the expertise to challenge outputs, keep a practical ability to intervene, and preserve records that explain both what happened and why. The pilots still need to fly the plane, even when it all goes wrong.
In an emergency, pilots work through three priorities in order: aviate, navigate, communicate. I find it one of the most useful frameworks for AI governance under pressure.

Aviate means retain control: keep flying the plane, no matter what. Stabilise operations, reduce harm and protect your critical services and crown jewels. Know who can stop or override an AI or automated action. Don't accept a recommendation just because the system sounds confident, and preserve the right evidence during containment in case you need to investigate.
Navigate means exercise independent judgement. Compare AI outputs with authoritative sources and, just as importantly, with the context in which they're being used. Check what data the system relied on, what remains uncertain and what contradicts it.
Communicate means explain the evidence and, equally, the uncertainty. State what is known, what is being investigated and what action is needed. Communication happens while control is being established, not only afterwards.
US Airways Flight 1549 is the famous example. On 15 January 2009 a bird strike disabled both engines shortly after take-off from LaGuardia. In a little over three minutes the crew retained control, assessed their options, chose the Hudson River and communicated calmly with air traffic control and the cabin. There wasn't a procedure for what Captain Sullenberger did; he described it not as a crash, but as a low-altitude river landing. All 155 people survived. Not every emergency ends that way, but it shows the value of cool, calm and closed-loop communication.
Throughout my career I've watched communication breakdowns impair incident response again and again. With AI now embedded in almost every technology, we need materially better communication discipline.
Outside of IT security, I've spent many hours and thousands of miles racing offshore, and ocean racing and aviation share the same principles. A capable crew needs a common picture. They communicate what they know and what they don't, clearly and calmly. They confirm who is doing what, and they close the loop once actions are complete.
In August 1997, Korean Air Flight 801 struck terrain three miles short of the runway in Guam; 228 of the 254 people aboard died. Investigators identified errors in the approach, fatigue, and inadequate crew coordination and cross-checking. Contrast that with Flight 1549. The lesson isn't merely to send messages; it's to build and maintain a shared operational picture.

The same loop works for AI governance. Observe: define the AI use case, its owner and its data in a couple of plain sentences. Interpret: assess the potential harm and your risk tolerance. Decide: select controls, performance measures, owners and escalation triggers. Act: give people the procedures, capability and authority to intervene, then confirm both completion and effect. Learn: review performance, incidents and overrides, and feed the results back into your assessments and your register.
During an incident, label every statement as a verified fact, an AI assessment or an untested assumption. Name the decision-maker, the action owner and the deadline. If nobody accepts responsibility, the action isn't assigned. And remember that acknowledgement is not the same as confirmation that the work is complete and effective. This is crew resource management for AI governance.
A communication loop depends on evidence people can find and trust. That's where the digital black box becomes critical. Flight data and cockpit voice recorders help investigators reconstruct both what the systems did and how the crew responded.

Evidence collection starts on the ground. During planning, record the use case, owner, data and contingencies. At take-off, preserve the model or service version, controls, permissions and response triggers. In normal operation, retain relevant outputs, checks and decisions. When conditions change, capture warnings, overrides, handoffs and escalations. In a mayday, record who knew what, who decided, who was affected and whether actions worked. Afterwards, retain the evidence, learning and corrective actions.
For agents that take actions, write code, approve software or review risk, ask what they can access, what decisions they make and how those activities are monitored. Then apply the traceability test: can you follow the source evidence to human approval, then to the instruction given to the action owner, and finally to the confirmed result? If not, your black box has gaps.
Two anonymised incidents show why evidence must survive the response. In one construction-sector incident, critical evidence was inadvertently deleted during the response. Operations stabilised, but the team had to reconstruct events from incomplete information, and straightforward stakeholder questions became hard to answer with confidence. In an aviation-sector incident involving AI-assisted information disclosure and a web server compromise, an attacker deliberately deleted artefacts.
Investigators could still reconstruct the sequence, because logging, retention and forensic readiness had left evidence elsewhere.
Confidence must follow evidence. If information is missing, say so and assign the next investigative action. Don't let an AI summary fill the gap with an unverified explanation.
All of this comes together in a simple, repeatable cycle. Start with an AI resources register, define each use case, assess the risks and impacts, and let that inform your controls. Then monitor, review, improve and update the register, so every cycle is re-baselined with fresh evidence. It's the same continual improvement thinking you'll recognise from ISO management systems, the NIST AI Risk Management Framework and the Australian Signals Directorate's guidance.

I've distilled this cycle, the Aviate, Navigate, Communicate framework and the pre-flight questions below into a one-page AI Governance Handbook: Your AI Pre-flight Checklist. It shows what good looks like at each of the nine steps, and the evidence to keep so your flight recorder has no gaps.
Download the one-page AI Governance Handbook
Think of one consequential AI use case in your organisation. Just one. Then ask these five questions.
1. Do we know the AI system, its uses and the data it accesses? Copilot may be able to read tender documents or internal pricing, or draft recommendations that influence resourcing. Is that use case visible in your register, or is it just listed as a productivity tool?
2. Who owns the system, the risks and the decisions, and who can stop the AI? Your policy may name a business owner, but can that person actually pause a workflow, or can only your software vendor or MSP intervene?
3. What evidence verifies the output? If AI says a supplier meets a security requirement because it found a policy statement, is that source current? And what evidence would exist if the control weren't working?
4. Can we spot errors and work without AI? Give an analyst a summary with an incorrect causal claim and an omitted indicator. Can they reconstruct events from the logs, find both problems and explain the next action? That's the magenta-line test, and tabletop exercises are a great way to run it.
5. Do our records explain decisions and overrides? "Approved by manager" isn't enough. Keep the original output, source evidence, model or service version, reviewers, overrides or escalations, the instruction to the action owner and the confirmed result.
If you can answer these confidently, you're well prepared for a safer journey. If you can't, pick that use case, exercise it with a plausible wrong output, and notice where understanding or authority breaks down.
Aviation has spent decades learning that automation improves safety while creating new human-factor risks. Automation bias swaps independent verification for automated recommendations, and human oversight becomes less meaningful when trust replaces challenge. A system can function exactly as designed and still support a dangerous decision when its assumptions don't reflect reality. That's why "human in the loop" alone isn't enough: the human needs authority, capability and accountability.
Speculation confuses decisions. Preparedness means maintaining control, validating achievable options, stating uncertainty clearly and confirming results. Build an independent picture, keep clear guardrails over your systems and data, and make sure your AI flight recorder captures what the technology did and what the people did with it.
Preparedness happens on the ground, before take-off, not on the fly.
Ready to start your pre-flight checks? Download the one-page AI Governance Handbook and work through it with one AI use case this month. If you'd like to talk through your AI governance, incident response or tabletop exercises, get in touch with us here.