Take a closer look at the staff Christmas party photo. Paper crowns, tinsel, and The Guest smiling in the middle of it all. Nobody remembers inviting him, because he was never there. One photo from a company social post and a free generative AI tool were enough to put him in it. Jason’s Christmas speech, filmed on someone’s phone, gave The Guest his voice too. He drops a voice note into the sales team chat: ‘Hi, it’s Jason. About to board, heading to the Sydney office. I need that customer’s phone number now, I’ll explain later.’ It works on anyone in a hurry.
Not long ago, that trick needed a gifted mimic. Today it takes one photo, a few minutes of audio and a free afternoon. Gen AI tools are now part of everyday work, and attackers use the same ones. So the useful question for IT leaders is a different one. When your people can’t spot a fake, do your processes still hold?
The con just got an upgrade
The most quoted example is still the most instructive. In 2024, a finance employee at a multinational firm in Hong Kong transferred around USD 25 million after joining a video call where every other participant, including the ‘CFO’, was an AI-generated impersonation. The employee had been suspicious of the original email. The video call is what convinced them.
That is the shift in one story. We have spent years telling people to be wary of emails and to pick up the phone to check. Attackers noticed, and now the phone call and the video call can be fake too.
Sophos is blunt about where this is heading. Its AI Security 2026 report found that AI-assisted social engineering and deepfakes are making scams ‘more scalable, more convincing across languages, and significantly cheaper to produce’. The same report tracked a single campaign in which roughly a dozen AI agents produced close to 80 attack modules in days, work that would have taken human operators weeks.
For defenders, that means fewer tell-tale signs. The clumsy grammar and odd formatting we trained people to look for are disappearing. Sophos’ own 2026 predictions put it plainly: ‘Deepfake audio and video will make business email compromise (BEC) campaigns more convincing and far more credible.’
Business email compromise is already the main event
This isn’t a future problem. According to the ASD’s Annual Cyber Threat Report 2024-25, the top cybercrimes reported by Australian businesses were email compromise (19%) and business email compromise fraud resulting in financial loss (15%). These attacks were already working without AI. A cloned voice on a follow-up call makes them much harder to resist.
AI is also creating a new kind of identity to protect. The same Sophos AI Security report found attackers increasingly going after OAuth tokens, AI service credentials, agents and developer tools. As your organisation adopts AI, each of those becomes another key to the building, and many of them sit outside the access reviews you already run.
Stop training people to be detectives
Asking staff to spot a good deepfake by looking for odd blinking or strange lip sync is a losing game. The technology improves every month, and your finance team has other things to do.
Sophos describes the shift needed as moving from trust by observation to trust by verification. It’s a useful frame. Instead of asking ‘does this look real?’, your people ask ‘has this been confirmed through a channel the attacker doesn’t control?’
That turns a judgement call into a process, and processes scale in a way that instinct doesn’t. In The Guest’s story, what finally stops the fake Jason isn’t a deepfake detector. It’s a colleague replying ‘Calling Jason’s actual mobile now.’ Stop, think, check, on a second channel.
Building a verification culture
The practical steps are refreshingly low-tech.
- Call back on a known number. Any request involving money, credentials or sensitive data gets confirmed on a number from your own directory, never one supplied in the message.
- Agree code words for high-value requests. A simple phrase known only to the finance team and senior leaders defeats a perfect voice clone.
- Build in a delay. Urgency is the attacker’s favourite tool. A mandatory cooling-off period for new payees or changed bank details takes the pressure out.
- Require two people for big decisions. Dual approval on payments over a set threshold means one convinced person isn’t enough.
- Give people permission to check the boss. If a junior staff member feels awkward calling the CEO to confirm a request, your process has a gap. Say it out loud: verifying is never rude.
- Bring AI identities into scope. Review the service accounts, tokens and integrations your AI tools rely on, with the same rigour you apply to people.
Critical thinking is still your team’s most valuable asset. Verification simply gives it somewhere to go.
Rehearse it before you need it
A verification process only works if people use it when they’re under pressure, and they’ll only do that if they’ve practised. Run tabletop scenarios with your finance and executive teams. Simulate a convincing request from a senior leader and see whether the callback happens. Include voice and SMS in your phishing simulations, not just email.
The first time someone uses your verification process shouldn’t be the day it matters.
Where ASI fits
We’ve been helping Australian organisations adapt to new technology for over 40 years, and deepfakes haven’t changed what we’ve learned. When a threat gets past one layer, it’s the other two that catch it.
That’s why we look at all three together:
- People. Building the habit of stopping, thinking and checking, so a convincing voice note gets a callback before it gets an answer.
- Process. Verification steps that hold up under pressure, such as callbacks to known numbers, dual approval on payments and a clear path for raising concerns.
- Technology. Phishing-resistant sign-in, monitoring that keeps watching after hours, and proper oversight of the AI tools and accounts your organisation now relies on.
Most organisations are strong in one or two of these. We help you find the gap and bring all three up to date.
Not sure how your team would handle a voice note from ‘Jason’? Find out before he sends one. The Guest Prevention Program starts with a free staff awareness assessment that tests how your people respond to AI voice scams, phishing and unexpected login requests. You’ll get a results summary and a walkthrough of what to fix first.
