You're evaluating AI agencies. They all claim to "take security seriously." Their pitch decks have lock icons and compliance badges. But when your CISO asks pointed questions about data residency or model training opt-outs, the room goes quiet.

This is the gap that kills enterprise AI projects — not technical capability, but a fundamental misunderstanding of how regulated organizations actually operate. And it's widening: a Gartner survey of 175 employees (May–November 2025) found that over 57% use personal GenAI accounts for work and 33% admit inputting sensitive information into unapproved tools. Your agency needs to understand why that's terrifying, not just technically interesting.

Here's how to separate the agencies that actually get it from the ones performing security theater.

1. They Can Name the Frameworks That Apply to You

A competent agency doesn't wait for you to explain your compliance landscape. They ask what industry you're in and immediately start mapping obligations.

At minimum, they should be fluent in:

SOC 2 Type II — the table-stakes audit framework for any SaaS or cloud-based AI system handling customer data
HIPAA — if you're in healthcare, they need to understand Business Associate Agreements and PHI handling, not just spell the acronym
The EU AI Act — which began enforcing transparency and documentation requirements for general-purpose AI providers in 2025, with high-risk system obligations following in August 2026
NIST AI RMF — the AI Risk Management Framework that provides a voluntary but increasingly expected structure for governing AI risk across design, development, and deployment
Industry-specific regulations — GLBA and SEC guidelines for financial services, FedRAMP for government, FERPA for education

The red flag isn't that they don't hold every certification. It's that they don't know which ones matter for your use case.

2. They Have Opinions About Data Architecture — Strong Ones

Ask an agency: "Where does our data go when it enters your AI system?" If the answer is vague or they defer to "the model provider handles that," walk away.

An agency that understands enterprise data concerns will proactively address:

Data residency. Where is data stored and processed? Can it stay within a specific geographic region? This isn't optional for companies subject to GDPR, data sovereignty laws, or internal governance policies.
Training data isolation. Will your data be used to train or fine-tune models? Every major provider — OpenAI, Anthropic, Google — offers enterprise agreements with training opt-outs. A good agency has already negotiated these.
Data lifecycle management. How long is data retained? When and how is it purged? Can you get a data processing agreement (DPA) that specifies retention windows?
In-transit and at-rest encryption. TLS 1.2+ minimum in transit, AES-256 at rest. If they can't specify their encryption standards without looking it up, that tells you something.

The best agencies will sketch a data flow diagram on the whiteboard unprompted. They'll show you exactly which systems touch your data, what crosses network boundaries, and where the trust boundaries are.

3. They Talk About AI-Specific Risks, Not Just Generic InfoSec

Traditional security practices are necessary but insufficient for AI systems. An agency that truly understands AI compliance will raise risks that most IT security teams haven't encountered before:

Prompt injection. Can adversarial inputs manipulate the model into leaking data or performing unauthorized actions? What defenses are in place?
Model output validation. Are there guardrails preventing the AI from generating toxic, biased, or factually incorrect outputs that create liability?
Shadow AI proliferation. Gartner's research shows this is already endemic. Does the agency's architecture reduce the incentive for employees to use unsanctioned tools?
Supply chain risk. Which third-party models, embeddings, and vector databases are in the stack? What happens when a model provider changes their terms of service or data handling practices?

Forrester has formalized this thinking with their AEGIS framework (Agentic AI Enterprise Guardrails for Information Security), which addresses security across six critical domains for autonomous AI systems. An agency worth hiring should be familiar with frameworks like this — or have built their own equivalent.

4. They Can Show You an Audit Trail

Compliance isn't a one-time checkbox. It's an ongoing practice that requires evidence. Ask to see:

Logging and observability. Every AI interaction — prompts, completions, tool calls, decisions — should be logged to an immutable audit trail. Not just for debugging. For compliance, incident response, and regulatory examination.
Access controls. Role-based access to AI systems with the principle of least privilege. Who can access what data through the AI? How is that enforced?
Model cards or system documentation. What model is being used, what it was trained on, what its known limitations are. The EU AI Act and NIST AI RMF both emphasize this kind of transparency documentation.
Incident response plans. What happens when the AI produces a harmful output? When there's a data breach involving the AI pipeline? There should be a documented, tested playbook.

Organizations that deploy AI governance platforms are 3.4 times more likely to achieve high effectiveness in AI governance than those that don't, according to a 2025 Gartner survey of 360 organizations. Your agency should either use these tools or have built equivalent capabilities into their delivery process.

5. They've Done This Before — and Can Prove It

Theory is great. Shipping compliant AI systems in regulated industries is better. Ask for specifics:

Have they worked with healthcare organizations navigating HIPAA for AI-powered patient interactions?
Have they built systems for financial services companies where model explainability is a regulatory requirement?
Have they handled enterprise procurement processes that required security questionnaires, penetration testing, and vendor risk assessments?
Can they provide references from clients whose compliance and security teams were involved in the engagement?

An agency that has done real enterprise work won't flinch at these questions. They'll have war stories about BAA negotiations, security review boards, and the three-month vendor onboarding processes that large organizations require.

6. They Push Back on You (Respectfully)

This is the counterintuitive one. The best sign that an agency understands compliance is when they tell you something you're doing is risky.

"You want to feed customer support tickets directly into a fine-tuned model? Let's talk about what PII is in those tickets first."

"Your team is using a shared API key for the production LLM? That makes access auditing impossible."

"You're planning to deploy this without a human review step? For this use case, that's a liability exposure you should understand."

An agency that says yes to everything is an agency that hasn't thought through the consequences. Genuine expertise shows up as informed friction — they slow you down in the right places so you don't get burned later.

The Practical Checklist

Before you sign with any AI agency, get satisfactory answers to these questions:

Area	Question	Red Flag Answer
Data handling	Where does our data go, and who can access it?	"The cloud provider handles that"
Model training	Will our data be used to train models?	"I'd have to check"
Compliance	Which regulatory frameworks apply to this project?	"We're compliant with everything"
Audit trail	How do you log and monitor AI system behavior?	"We use standard application logging"
Incident response	What's your plan when the AI produces a harmful output?	"That won't happen with our system"
Access control	How is access to the AI system and its data governed?	"Everyone on the team has access"
Third-party risk	What model providers are in the stack, and what are their data terms?	"We use the best available model"
Experience	Show me a past project where compliance was a primary constraint.	Silence, or only startup/consumer examples

How We Think About This at Last Rev

We work with enterprise organizations that have real compliance requirements — not theoretical ones. That means every AI system we build starts with the constraints, not the capabilities. Data flow diagrams before prompt engineering. Security architecture reviews before sprint planning. DPAs and BAAs before code commits.

We've learned that compliance isn't a phase at the end of a project. It's an architectural decision that shapes every technical choice from day one. The agencies that bolt it on later are the ones whose projects stall in security review for months — or worse, ship and create liability.

The question isn't whether an AI agency can build something impressive. It's whether they can build something impressive that your CISO, your legal team, and your regulators will let you actually deploy.