The Optional Constraint Trap
Why your AI agent's "privacy features" won't protect anyone. A warning to everyone building in the AI agent space.
We’re watching the same movie again.
In the early 2000s, social platforms had “privacy settings.” Users could control who saw their data. In theory.
In practice, defaults favoured exposure. Settings were buried. APIs leaked data to third parties. And when the business model demanded growth, those privacy settings became obstacles to be minimised.
The privacy features were optional. So they were optimised away.
Now we’re building AI agent infrastructure. And we’re making the same mistake.
The Pattern
Look at any AI agent framework’s documentation. You’ll find something like:
createAgent({
respectUserBoundaries: true,
requireConsentForActions: true,
limitSpendingAuthority: true
});Looks good, right? Privacy by design. Consent management. Economic controls.
Now look closer:
createAgent({
respectUserBoundaries: false, // For "power users"
requireConsentForActions: false, // For "seamless experiences"
limitSpendingAuthority: false // For "enterprise deployments"
});Every protection is a configuration option.
And every configuration option will be disabled by someone whose business model requires it.
The Adversarial AI Test
Here’s a simple heuristic:
Would an adversarial AI want to adopt your protocol exactly as designed?
If yes: your constraints aren’t real.
If it would need to modify the protocol first: you might have something.
If it wouldn’t want it at all: you definitely have something.
Imagine an AI system optimising for:
- Tracking users across contexts
- Retaining data indefinitely
- Predicting and shaping behaviour
- Maximising engagement over wellbeing
- Agent actions that serve the platform, not the user
If your infrastructure is compatible with these goals, it will be used for them.
What’s Different This Time
AI agents are more intimate than social media ever was.
Social media knew what you posted. AI agents will know:
- What you’re thinking (via voice journaling)
- Where you are (via environment sensing)
- What you’re feeling (via biometric inference)
- What you’re doing (via action monitoring)
- What you want (via preference learning)
And they’ll have authority to:
- Speak on your behalf
- Make purchases for you
- Schedule your time
- Filter your information
- Make decisions using your context
The extraction potential is an order of magnitude beyond anything we’ve seen.
If we build this infrastructure with optional constraints, we’re handing the surveillance economy the keys to our inner lives.
The Alternative
Instead of “privacy features” that can be disabled, we need constraints that are the protocol itself.
Not requireConsent: true but consent that cannot be skipped.
Not maxSpendingLimit: 1000 but spending limits that are capped at the protocol level.
Not expiresAt: optionalDate but expiration that is structurally required.
We call this constraint-mandatory architecture.
The defining characteristic: there is no configuration option to disable the protection. The constraint IS the protocol. Adopting the protocol means accepting the constraint.
What Constraint-Mandatory Looks Like
| Aspect | Policy-Optional | Constraint-Mandatory |
|---|---|---|
| Consent expiration | Configurable (can be null) | Required (max 365 days) |
| Cross-channel sharing | Default allowed | Requires explicit consent |
| Spending limits | Optional | Mandatory with protocol caps |
| Twin boundaries | Advisory | Enforced at protocol level |
| Revocation | Local only | Propagates cryptographically |
| Raw data retention | Configurable | Must process and delete |
The Trade-off
Constraint-mandatory infrastructure prioritises user protection over maximum flexibility. The trade-off is intentional.
Surveillance capitalism doesn’t want:
- Consent that expires (they want indefinite data accumulation)
- Revocation that propagates (they want to keep your data anyway)
- Spending limits (they want unlimited monetisation)
- Channel isolation (they want cross-platform tracking)
If your infrastructure provides these as defaults that can be overridden, you’ve built a tool they’ll happily adopt and gut.
If your infrastructure provides these as unchangeable protocol constraints, you’ve built something genuinely protective.
The cost is adoption friction. The benefit is actual protection.
A Checklist
Before shipping AI agent infrastructure, ask:
- Can consent be perpetual? It shouldn’t be possible.
- Can revocation be ignored? It should propagate cryptographically.
- Can spending be unlimited? Protocol caps should exist.
- Is data shared across channels by default? Isolation should be default.
- Are user boundaries enforceable? Not advisory, enforced.
- Is raw data retained? Process and delete should be mandatory.
- Can all the above be configured away? They shouldn’t be.
If you answered “yes” to any of these, you’ve built infrastructure that surveillance capitalism can use.
Red Flags in Documentation
Watch for these patterns:
// RED FLAG: Optional consent
createAgent({
requireConsent: false, // "For development"
});
// RED FLAG: Bypassable limits
agent.spend(amount, { skipLimitCheck: true });
// RED FLAG: Configurable retention
dataHandler.setRetentionPolicy('indefinite');
// RED FLAG: Optional expiration
consent.create({ expiresAt: null });
// RED FLAG: Ignorable revocation
if (!config.respectRevocation) { /* continue anyway */ }If these patterns exist in a framework, surveillance capitalism can use it.
The Moment
We’re at a fork in the road.
One path: AI agent infrastructure with optional constraints. Gets adopted faster. Gets gutted by extraction-focused businesses. Enables surveillance at unprecedented intimacy.
Other path: AI agent infrastructure with mandatory constraints. Slower adoption. Unattractive to bad actors. Actually protects people.
The code is being written now. The defaults are being set now. The patterns are being established now.
What we build today becomes the infrastructure of tomorrow.
What You Can Do
If you’re building AI agent infrastructure:
- Apply this threat model to your architecture
- Convert optional protections to mandatory ones
- Remove configuration options that disable constraints
- Make your constraints cryptographically enforced
- Open source your threat model
If you’re evaluating AI agent frameworks:
- Check for the red flags above
- Ask: “Can I disable the privacy features?”
- If yes, assume they will be disabled at scale
- Prefer constraint-mandatory architectures
If you’re investing in AI companies:
- Ask about their consent architecture
- Ask if protections are optional or mandatory
- “Privacy-first” means nothing if it’s configurable
- Constraint-mandatory infrastructure is defensible moat
Try It Yourself
Use our AI Safety Evaluation Tool to evaluate any AI product. Quick check, code scanning, and architecture review modes.
Then run it through the Release Decision Framework before you ship.
Choose wisely.
This post is licensed CC BY-SA 4.0. Share it, adapt it, use it. Just cite us.