The confession
"Hey, we are building this new AI stuff. We have to review and secure it. I don't know if I can do a good enough job with this, as I know I can do with traditional AppSec."
When one of my senior engineers came to me with this confession, I knew we had crossed into new territory. This was a staff-level AppSec specialist recognizing that the ground had shifted beneath our feet.
At Sendbird, we went from a mature organization to essentially running like a seed-stage startup inside a mature company. We were pivoting to AI and building something new from the ground up very, very quickly. We were transforming from a chat API company into an AI agent platform, building what would become Delight, our AI customer service agent.
The attack paths are different. The types of security issues are different. The data security models are different.
That moment of vulnerability became the entry point for understanding something crucial: the transition to AI-powered systems isn't just a technical challenge. It's an architecture problem most organizations don't even know they have yet.
The false comfort of enterprise labels
The major blind spot was the notion of, yes, we're using GitHub Copilot. It's an enterprise tool backed by Microsoft, so it's secure. It's okay to use.
But here's what we discovered: the security posture of a beta model versus a GA model can be radically different. The terms of service vary. The data handling practices differ. The guarantees shift.
This pattern repeats across the AI landscape. AWS, Azure, OpenAI: all offer enterprise-grade AI services with different tiers, different models, different security implications. The enterprise badge has become a false signal, a shortcut that bypasses the detailed technical diligence these tools actually require.
We had to unlearn what worked in the past. Traditional enterprise procurement said: buy from the big vendor, trust their security certifications, move fast. AI security requires the opposite: dig into the details, verify every claim, understand that "enterprise-ready" often just means "productivity-optimized by default."
At Sendbird, we treat every AI service like we're the first customer to ask hard questions about it, because often, we are. And we think customers should be asking us those same hard questions, not letting our enterprise badge be the answer.
When incidents stop being binary
For decades, security incidents existed in a binary state: breach or no breach. AI demolishes that binary.
When an AI agent gives a wrong answer or a suboptimal answer, how do you classify that? It is an incident of sorts. It's not a breach. This is new.
Consider the airline chatbot that mistakenly offered a customer a deeply discounted ticket. No data was stolen. No systems were compromised. But real financial damage occurred.
The ambiguity multiplies when AI agents start taking actions in customer environments. Delight can make account changes, issue refunds, retrieve customer data from backend systems. Our applications have a path to make calls into the customer's environment. So that gets tricky because you then have to figure out authentication, authorization, logging, and visibility.
Traditional Security Incident: Binary event (breach or no breach)
AI Security Incident: Spectrum event (wrong answer, unauthorized action, data leakage, hallucination, financial impact)
The architecture nobody talks about
In the traditional, non-AI world, security is defined by a "Defense in Depth" strategy, a philosophy often referred to by the flashy marketing term "Zero Trust." Stripped of the buzzwords, the core idea is simple: assume that some of your security controls will fail. It's not a question of if they will fail, but when.
Having multiple layers of security work together has always been the approach we have taken at Sendbird. Here's what this looks like when an employee tries to access GitHub or AWS:
- Is the request from a company-issued device?
- Is it from a Chrome browser enrolled in Google Enterprise workspace?
- Okta performs device health checks
- CrowdStrike validates endpoint protection posture
- FIDO2-based MFA with YubiKey, Okta Verify, or fingerprint
- Passwords (still required in some countries)
This is like four or five layers that happen in the background that a user wouldn't even realize. Good security doesn't announce itself. It operates in the background, creating a safety net where if one thread breaks, four others hold strong.
For AI systems, the layers look different: vendor contracts defining data usage, model selection balancing capability with safety, input validation catching malicious prompts, output filtering detecting PII or hallucinations, human oversight enabling manual intervention, and logging creating audit trails across boundaries.
None of these layers alone creates security. Together, assuming some will fail, they create resilience.
The skills gap nobody wants to admit
A staff engineer, staff AppSec or project engineer is not automatically a staff AI security engineer. It does put a decent amount of stress on AppSec engineers where companies are moving really fast on AI and they're expected to help secure them.
When you go from traditional application testing (OWASP Top 10, XSS, SQL injection), that changes completely. The attack paths are different. It's a very different way of thinking.
Here's what I told my team: This is new. We will make mistakes. It's okay to do that. We'll eventually catch up to it.
When SQL injections came out in the very early days, it was a big deal and nobody knew exactly how they worked. Today, we are much ahead of that game. So we will eventually get to something similar on the AI front.
One of my engineers built an AI CTF over a weekend using Sendbird. It's a chatbot for a fictional store, deliberately made insecure with about ten flags hidden in it. Participating in this helped the entire company wrap their heads around how things can go wrong when all you're doing is asking questions that weren't expected.
It also helped us think through security considerations for Delight from multiple perspectives, not just as security engineers, but as potential attackers.
The culture equation
I've learned something counterintuitive: the more tools you provide, the more secure you can become (as long as you're building the right culture alongside them).
If we don't enable our people by providing them AI tools, they're going to find tools (either paying out of pocket or using free versions) to get their jobs done. So we started providing enterprise-level tools to the entire company: Google Gemini, ChatGPT Teams, Claude Code, Cursor Enterprise, GitHub Copilot.
At the same time, we're clear about what's blocked. All LLMs from China are blocked. Unofficial Chrome AI extensions are blocked. We communicate why.
But here's what makes this work: our engineers ping us saying, "Hey, we see this new ML model available in Copilot. Can we enable it?" They don't have to ask us. They have admin access. But they ask anyway.
That's not a technical control. That's culture.
When the rest of the organization realizes that the security team is coming in to help people do the right thing (and that they won't be penalized for taking shortcuts), you'll find that they often come to you.
It isn't going and saying, "You can or you cannot do this." It's saying, "Here's the problem. Here's the risk we see. How can we solve it together?"
The data question that changes everything
If you're not paying for the service with money, you're paying with your data.
If you're using an AI vendor, you need to ask:
- How are you using my data?
- Are you building your own LLM and using my data for training?
- What contracts do you have with commercial LLMs?
- How do you delete my data when I terminate?
At Sendbird, we have it in our contracts: our vendors will not use our customer's data for training. That's non-negotiable.
But the data question extends beyond B2B concerns. A lot of people are using ChatGPT as their personal doctor, uploading health symptoms. What they don't realize is how that data can be used to build a digital model of them.
The same applies to those free services that say, "Upload a photo. We're gonna make it into a professional photo." You're paying with your photo.
It doesn't mean you don't do it. It means you need to consciously make a risk decision: What am I giving to this service? Am I willing to take the risk for the output I'm getting?
Five hard-won lessons
1. Embed security in sprints, not reviews
We went from hour-long design reviews to embedding a security engineer directly into the teams. Be part of their sprints, their standups, their Slack channels. Security reviews are a bottleneck. Real-time security participation is an accelerant.
2. Build Trust OS, not just controls
We give customers the ability to see what Delight is doing. They can configure what it should and shouldn't do. They can build automated test cases. This isn't security theater. It's security architecture that acknowledges AI systems are probabilistic, not deterministic.
3. Acknowledge the skills gap
We give our teams the space to experiment. They're staff engineers for a reason: they can figure shit out. We need to create a culture where people can admit they don't know, where learning is expected and supported.
4. Bridge customer environments
AI agents that take actions in customer environments need authentication, authorization, visibility, and logging across organizational boundaries. We built security features directly into the platform: enforcing HTTPS, IP whitelisting, header-based authentication, PII masking.
5. Accept that layers will fail
You have to actually believe that layers will fail. It can't just be something you say. My incident response team is working with product teams to figure out how to detect incidents that span organizational boundaries. We don't have all the answers yet. But we're building the architecture that will let us discover the answers as we go.
The architecture we're building together
Let me bring this back to where we started: that engineer who said, "I don't know if I can do a good enough job with this."
The answer isn't that he needed to learn to do it perfectly. The answer is that we, as a team, would learn to build something new together.
AI security isn't about having all the answers. It's about building the architecture to discover answers together:
- Multiple layers that assume some will fail
- Real-time collaboration embedded in development workflows
- A culture of trust where people come to security instead of avoiding it
- Conscious risk decisions about data and tools
- Permission to experiment within clear boundaries
At Sendbird, we went from being a mature chat API company to building Delight, our AI customer service agent, essentially running a seed-stage AI company inside that mature organization. We had to acknowledge that staff engineers weren't automatically staff AI security engineers. We had to build new definitions of what an "incident" even means.
The companies that will win at AI security aren't the ones that have all the answers today. They're the ones building the culture, the architecture, and the mindset to adapt as the answers emerge.
This is new. We will make mistakes. It's okay to do that. We'll eventually catch up to it.
And in the meantime, we're building the hidden architecture (layer by layer, conversation by conversation) that will define what secure AI actually means.