You Built AI Automation for SOX. Who's Testing the AI?

AI is coming for SOX. If it hasn't shown up at your company yet, give it a quarter.

I don't say that as a warning. I say it as someone who spent twenty years inside compensation teams that would have given an arm for half of what these tools now do on their own. KPMG put out guidance in 2025 telling companies to hand AI agents real SOX work: pulling evidence, writing up walkthroughs, testing controls, even planning the audit calendar. Deloitte and EY have their own playbooks. The biggest internal audit shops in the world already run on this stuff. And honestly, good. I'm not going to defend the way we used to do it. If you've ever burned a Sunday reformatting a payroll export so it would line up against an HRIS dump, you don't get sentimental about manual evidence collection. Let the machine have it.

So this isn't a piece about AI being dangerous. It's about one question I keep waiting for someone to ask. It usually surfaces about ten minutes into the demo, right after everyone's done being impressed.

Who's testing the AI?

Picture the demo, then picture the audit

Here's how it tends to go. A company either buys an AI compliance platform or, more often than you'd think, builds their own. They wire it into the HRIS, the ICM, payroll, the CRM. It hums along, flags the weird stuff, drafts the documentation, and produces a stack of output that looks, for all the world, like control evidence.

Then audit season comes. They walk the external auditor through it, proud, expecting a nod. What they get instead is a question they didn't rehearse for: okay, but how was the AI itself controlled?

That's not the auditor being difficult. That's the auditor doing the job. PCAOB Auditing Standard 2201, the one that governs audits of internal control over financial reporting, says auditors have to evaluate the IT general controls around any system that touches financial data or runs a control. An AI tool touches financial data and runs a control. There's no carve-out for being new and shiny.

And if you built the thing in-house, the auditor doesn't get to take your word for any of it. They have to independently test whether the logic actually checks what you claim it checks, who's allowed to change its rules, how updates get documented and signed off, and whether the output is complete or quietly missed a third of what it should have caught.

Notice who can't run those tests. The team that built the tool. That's segregation of duties, and it isn't a technicality. You built a control, so somebody independent has to test the control. Bolting AI onto the process didn't make that requirement go away. It added a brand new control that now needs its own independent test. The tool you bought to shrink the audit just grew it.

I went and read what the frameworks actually say

I didn't want to be the guy asserting this from a stage, so I went and read the source documents. They're more direct than I expected.

COSO, February 2026. COSO is the body that wrote the framework basically every SOX program in America stands on. This year they published "Achieving Effective Internal Control Over Generative AI," and the line that stuck with me was blunt for a standards body: "GenAI cannot be controlled with a 'set it and forget it' mindset." They ran all five pieces of their control framework against AI use cases and kept landing in the same place. You have to decide, for each AI application, how much human oversight and separation of duties it needs, based on its risk and on how much its output actually drives decisions. Compensation feeds straight into the accrual on the financial statements. That's about as high-risk as it gets, which means the oversight isn't a nice-to-have.

PCAOB, December 2025. Their Technology-Assisted Analysis standard kicked in for audits starting December 15, 2025. It's written to be principles-based, which is a polite way of saying it's built to outlive whatever technology shows up next, AI included. The practical effect is simple. An automated control is only ever as trustworthy as the ITGC environment wrapped around it. Swap a manual review for an AI one and the auditor still has to ask whether that AI was properly controlled, same bar as any other automated control. And here's the part that should make people sit up. The PCAOB has already named "testing controls over the accuracy and completeness of data or reports used in the operation of controls" as one of the most common things firms get dinged for. AI output is a fresh, enormous new pile of exactly that kind of data.

KPMG, 2025. This one's almost funny. KPMG's "Seize the future: The agentic shift in SOX compliance" is a love letter to AI agents in SOX. And tucked inside it is this: "The focus is expected to shift toward employees carefully overseeing AI activities to verify quality and mitigate risk rather than performing these tasks manually." The same firm urging you to adopt AI is telling you what your people turn into once you do. The oversight layer. The job doesn't disappear. It moves up a level.

SEC Investor Advisory Committee, December 2025. In December the SEC's Investor Advisory Committee voted to recommend disclosure rules for how AI gets used in financial reporting and internal controls. Put plainly: audit committees should be watching the AI, and they should have to tell investors they're watching it. The question of who oversees the AI is quietly graduating from an internal process detail into something you disclose.

The part no tool can fix

Set the standards aside for a second, because underneath them there's something simpler, and it's the thing I can't get people to sit with.

When your AI runs a check and hands you a result, that result is yours. Your system, your rules, your infrastructure, your data. When the auditor evaluates it, they're evaluating something you made about yourself. We have a word for that, and it isn't "evidence." It's self-assessment.

SOX Section 404 split management's assessment from the auditor's attestation on purpose, back in 2002, for exactly this reason. Management can't credibly grade its own homework. A system management builds to grade that homework doesn't escape the logic. It inherits it.

So an internal AI can hand you data all day. Documentation, dashboards, tidy reports that look like proof. What it cannot hand you is independence, because independence isn't a feature you can ship. It's a property of who's doing the looking, and it has to come from someone who isn't the party being judged. That's the entire reason external auditors exist, and no amount of automation on your side of the line changes which side of the line you're standing on.

The firm that couldn't check its own AI

If that still sounds abstract, here's a real one, and it landed on a firm that sells AI assurance for a living.

In May 2026, EY published a cybersecurity report, "Points of Attack: Uncovering Cyber Threats and Fraud in Loyalty Systems." Thought leadership, from one of the four biggest professional services firms on earth. Then three researchers at GPTZero did the most boring possible thing. They checked the footnotes. All of them.

Out of 27 citations, more than 70 percent were hallucinated. Invented sources, statistics pinned to the wrong place, references to reports that don't exist in any archive anywhere. There was a McKinsey study cited as proof of a $200 billion loyalty market that has simply never been published. EY pulled the report the same day the findings went public.

The story isn't that EY made a mistake. Everyone makes mistakes. The story is which mistake, and who caught it. The firm that advises other companies on AI governance failed to check its own AI's output, and the catch came from three outsiders doing, by hand, the exact thing every framework I just quoted asks for. They traced the claims back to the source, from outside the process that produced them. That's independent verification. Unglamorous, manual, and still the only thing that actually works.

I keep this one close, because it isn't an anti-AI story. It's a demonstration of what the missing layer does, told in the negative.

Where companies talk themselves into trouble

The mistake I see most often isn't technical. It's a category error. Teams treat AI as a replacement for independent verification when it's a complement to it.

Give AI the grind and it shines. Collecting evidence, formatting, flagging anomalies across millions of rows, keeping a clean trail of the routine work. KPMG files these under their TACO framework, and the time savings are real, not marketing. I'd push you toward all of it.

The one thing it can't do is be independent. A system that checks whether your HRIS, your ICM, and your payroll agree, built and owned by the same company that owns all three, is not an independent check. It's the same team doing the same verification, only faster. Same blind spots, same incentives, same problem. I watched comp teams run clean against themselves for two decades. Good people, sharp checks, every box green. "We checked it ourselves, and quickly" was never the sentence the auditor wanted to hear.

The auditor's question is never "did you check?" It's "who checked, and how do we confirm, independently, that the check held?" For compensation, where a material weakness can drag you into a restatement and an executive clawback under SEC Rule 10D-1, that distinction has teeth.

Why every serious game has a referee

I keep coming back to officiating.

Every sport anyone cares about has referees, and not because the players are crooks. Most of them are honest. The refs exist because the game's credibility can't rest on the players' own account of what happened. Ask a team to self-report whether their guys were onside and you won't get a lie, exactly. You'll get a view from inside the play, shaped by what they were hoping to see. The ref's whole value is the angle. Same play, watched from outside the game.

AI in SOX is a hell of a player. Faster, more consistent, never tired, covering ground no human could. Worth every dollar. But more players, even great ones, never removed the need for a ref. Speed the game up and add complexity, and the ref matters more, not less.

The independent verification layer is the ref. Third-party, deterministic, producing evidence that neither side of the check wrote about itself. COSO wants it. PCAOB wants it. The split between management's assessment and the auditor's attestation has wanted it since the day SOX was signed. AI doesn't answer "who's testing the AI." Independence does. It always has.

The three questions you'll actually get asked

If you've put AI into your SOX program, or you're about to, these are the questions waiting in your next audit. Have answers ready.

What are the IT general controls around your AI tool, and who independently tested them? Access, change management, completeness and accuracy. The same ITGC domains you already apply to the HRIS and the payroll system apply here too. A new tool doesn't get a pass for being new.

Can you show that someone who didn't build or run the tool reviewed its output? If the honest answer is "our own team looked it over," that's a management review control, the one your auditor was already testing. The AI made it quicker. It didn't make it independent.

When the auditor asks for independent evidence that your comp data was right before the pay run, what's in your hand? If it's a report your AI generated, you've got documentation. Maybe even good documentation. But documentation and independent evidence are different things, and your auditor has known the difference for two decades.

Curious what independent compensation verification actually produces as evidence? View a sample Control Execution Record →

Or run a drift scan and see where your systems disagree, before your auditor does.