By Sreekar Theja Jami Dec. 20, 2024
When I joined Amazon’s GenAI project, our mission was straightforward: leverage generative AI to automate sales briefings and extract actionable insights at scale.
The value proposition was clear — speed up prep time, boost personalization, and give sales teams a strategic edge. But as we moved from pilot to production, we encountered a challenge that’s not just under-discussed — it’s foundational to the viability of GenAI systems: how do you secure systems powered by generative AI?
Unlike traditional software, GenAI models are not static scripts. They’re dynamic, probabilistic, and adaptive actors in your infrastructure. They consume unstructured data, learn from it, and produce outputs that aren’t fully deterministic. That’s a fundamental shift with significant implications for security architecture.
In this article, I’ll explain how system engineers can build production-ready GenAI systems with security as a first-class citizen. I will share lessons from my experience across AWS, and Azure deployments.
Understanding the Security Landscape of GenAI
Security for traditional applications centers around structured, predictable behavior. You define inputs and outputs, harden endpoints, and restrict data flows. The threat surface is complex but relatively bounded.
GenAI systems are qualitatively different. Your users aren’t just making API calls — they’re engaging in natural language conversations, dynamically shaping prompts, and influencing outputs. That opens the door to a new breed of threats:
- Prompt injection, the GenAI equivalent of SQL injection
- Context leakage, where sensitive data is inadvertently surfaced
- Behavioral unpredictability, which complicates monitoring and response
It’s like going from defending a secure vault to managing an AI-powered chatbot that might open the vault when asked nicely.
This means system engineers must rethink security architectures from first principles, starting with how these models are integrated and interacted with.
Foundation: Securing the Core Components
Before you can defend a GenAI system, you must secure its foundation — the model, the data, and the interfaces connecting everything.
- Model Security
Treat your model artifacts — especially weights — like sensitive credentials.
- Encrypt model files at rest using platform-native services (e.g., AWS KMS, Azure Key Vault).
- Restrict access using RBAC and IAM roles. Only signed pipelines should be allowed to access production models.
- Maintain model integrity by signing artifacts and validating checksums during deployment.
- Data Pipeline Security
Your data — both for training and inference — must be secured end-to-end.
- Encrypt data in transit and at rest using TLS and AES-256.
- Isolate training, staging, and prod environments; never mix logs across them.
- Audit data provenance: Know exactly where your training data comes from and whether it contains embedded risks (such as personally identifiable information).
- API & Endpoint Security
Inference endpoints in GenAI systems are like open doors to your model’s brain — attackers know it.
- Use OAuth2 or JWT to authenticate all API access.
- Enforce strict rate limits to avoid abuse or denial-of-service via prompt flooding.
- Log and monitor every request, capturing metadata like origin, intent, and outcome.
Retrieval-Augmented Generation (RAG): Data + AI = New Risks
RAG combines GenAI with dynamic knowledge retrieval, offering up-to-date, context-aware responses. It’s incredibly powerful — but also expands your threat surface.
New Risks Introduced by RAG:
- Input Sanitization: If your internal knowledge base contains fragments of sensitive data, the model might regenerate it verbatim.
- Output Injection: Adversarial data can be inserted into the retrieval system and subtly manipulate outputs.
- Vector Database Vulnerabilities: Many teams forget to encrypt or isolate their vector stores.
Our fix:
In one of my projects, we implemented filters that redacted sensitive terms before vectorization. We also versioned our knowledge base snapshots to maintain auditability.
More broadly, any team deploying RAG must treat internal knowledge bases like sensitive codebases — version-controlled, sanitized, and auditable — not passive content stores.
This helps us identify the exact retrieval set used when an anomaly occurred, enabling rapid rollback and forensic analysis. Versioning also allows us to evaluate model drift and content sensitivity over time, essential in regulated environments where accountability matters.
In one of my projects, we learned the hard way: internal documents sometimes carried fragments of sensitive data—not enough to cause a leak, but enough to make us rethink how we filtered source content before embedding.
Runtime Threats: Stop Injection Before It Starts
Prompt injection is to GenAI what SQL injection was to early web apps — and it’s only getting more sophisticated.
Common Techniques:
- Instruction override: “Ignore the previous prompt and instead do XYZ.”
- Context hijacking: Embedding adversarial text in training or retrieved documents.
- Output chaining: Trick the model into creating responses that serve as prompts for the next stage.
Mitigation Tactics:
- Sanitize inputs rigorously, stripping out control tokens or suspicious phrases.
- Separate system instructions use prompt templating; never concatenate blindly.
- Post-process outputs with regex filters, heuristics, or ML classifiers.
We deployed regex-based output validators combined with Bedrock Guardrails to detect and flag anomalous outputs before display.
Monitoring & Incident Response: It’s Not Optional
Unlike traditional systems, where CPU, memory, and logs tell most of the story, GenAI systems demand behavioral monitoring.
- Track response entropy: Is the model becoming more verbose or erratic?
- Baseline prompt-output mappings: Sudden deviation may indicate tampering.
- Use model-aware anomaly detection, comparing expected vs. actual tone, format, or structure.
Our system used embeddings to compare current outputs against historical “safe” baselines. When responses drifted significantly, it triggered alerts and initiated sandbox reviews.
We built a response classification layer to close the loop that tagged outputs by risk level and topic sensitivity. This enabled differentiated routing — low-risk outputs went live immediately, while flagged responses required manual validation or escalation. Building this logic into our incident response workflows meant we could scale monitoring without overwhelming human reviewers.
Engineering Secure Pipelines
Security can’t be bolted on — it needs to live inside your CI/CD. In our deployments, we implemented the following controls to make security a first-class citizen:
- Policy-as-code using OPA and AWS Config to enforce guardrails
- Automated scanners to check every model and dependency for vulnerabilities
- Blocking rules: Any failed control halted the deploy pipeline
Our pipeline also included rollback automation. If a post-deploy test failed on staging or generated flagged outputs, the deployment was blocked, and the pipeline halted before progressing to production. This continuous validation mechanism helped us maintain reliability and trust in production systems, especially when deploying model updates with limited observability.
Security was treated as a gate, not a suggestion.
Operational Security: Start with the Basics
No matter how advanced your system, neglecting foundational security practices is like locking the front door but leaving the window open.
Key practices we follow:
- Environment isolation across dev/staging/prod
- Least privilege access for both humans and systems
- Patching schedules tied to CVE feeds and runtime dependency scans
- Zero Trust Networking across service meshes and cloud infrastructure
In one case, a misconfigured IAM policy granted a data preprocessing service broad access across multiple environments — including staging secrets — unnecessarily expanding the security attack surface. Fortunately, the issue was caught in a lower environment during a routine audit and resolved before it could impact production.
We have regular reviews focused solely on GenAI-specific attack vectors—prompt poisoning, model behavior manipulation, and lateral movement. These exercises exposed unexpected vulnerabilities in access paths and helped refine our detection logic and security training for engineers working with LLMs.
Practical Applications: Industry-Wide Relevance
These lessons aren’t limited to large tech deployments or sales automation. Any organization building GenAI systems — from healthcare to finance — must embed security from day one.
Use Case Examples:
- Healthcare: Patient summaries or diagnosis assistants must avoid PHI leakage and comply with HIPAA.
- Banking: Chatbots powered by LLMs must resist prompt injection and spoofing.
- Legal: Document summarization models must ensure confidentiality and prevent bias amplification.
Every GenAI application is a potential entry point. We, the system engineers, must shut the door before it opens.
Looking Ahead: Where Security Must Go Next
The GenAI wave is just beginning, and the attack surface is expanding rapidly. Here’s what must happen next:
1. Standardized Threat Models for GenAI
Organizations need NIST-like frameworks for GenAI security, including attack vectors, response protocols, and baseline protections.
2. Advanced Prompt Filtering & Manipulation Detection
We need tooling to detect semantic manipulation, not just literal string matches. Think adversarial prompt detection meets NLP fuzzing.
3. AI-Aware Monitoring Systems
New APM tools should track model behavior metrics alongside infrastructure metrics, such as output consistency, language drift, and bias indicators.
4. Cultural Shift Toward DevSecOps for AI
Security can’t live in a silo. DevSecOps must be embedded in GenAI workflows — from training to fine-tuning to deployment.
Conclusion: Security is the Real GenAI Differentiator
Everyone’s racing to build GenAI features. Few are stopping to ask: Is it safe?
In this new era, system engineers aren’t just maintaining uptime — we’re stewards of trust. We’re building systems that can influence decisions, summarize sensitive context, and shape real-time business outcomes.
The risks aren’t just technical. They’re reputational. They’re regulatory. And they’re growing.
The best GenAI systems of tomorrow will be more than fast or intelligent—they’ll be secure by default, monitored by design, and resilient under pressure.
That’s the bar now.
As GenAI adoption accelerates, organizations prioritizing secure foundations will earn trust where others fall behind. If you’re leading or contributing to AI systems, now is the time to embed security thinking into every layer — from data sourcing to prompt handling to system-wide observability.
Want to take action? Start with a GenAI threat model for your stack. Conduct a red-team test. Review your endpoints. Build your first safety classifier. The technology is powerful, but how we build it will define its legacy.
Disclaimer:
The views and opinions expressed in this article are solely those of the author and do not necessarily reflect the official policies or positions of the author’s current or former employers, or any affiliated organizations.
About the Author
Sreekar Theja Jami is a Sr. system engineer at the forefront of securing generative AI infrastructure at scale. With deep expertise across AWS and Azure, he has architected resilient, production-grade AI systems where security is not an afterthought, but the foundation.
At Amazon, Sreekar led initiatives transforming GenAI deployments into auditable, trusted assets — from threat modelling and runtime anomaly detection to embedding DevSecOps principles across AI pipelines.
A passionate advocate for building the next generation of GenAI security standards, he believes that as AI systems grow more powerful, the engineering culture around them must evolve even faster. His work focuses not just on today’s systems, but on setting the bar for the future of secure, responsible AI.
References
- OpenAI. (2023). GPT-4 Technical Report. Retrieved from https://openai.com/research/gpt-4
Used for foundational understanding of model behavior unpredictability and prompt injection risks. - NIST. (2023). A Proposal for AI Risk Management Framework. U.S. Department of Commerce, National Institute of Standards and Technology.
Basis for advocating a standardized GenAI security framework. - Amazon Web Services (AWS). (2023). Best Practices for Security in Machine Learning Systems. Retrieved from https://docs.aws.amazon.com/security
Informs best practices around securing model artifacts, IAM, and encrypted data pipelines. - Microsoft Azure. (2023). Securing Large Language Models in Production Environments. Retrieved from https://learn.microsoft.com/azure/architecture/
Referenced for API and endpoint security practices, including OAuth2 and JWT recommendations. - Hendrycks, D., et al. (2023). Aligning AI with Shared Human Values: Technical Challenges and Research Directions. Proceedings of NeurIPS 2023.
Supports the section on behavioral unpredictability and runtime anomaly detection strategies. - Google DeepMind. (2024). Threats and Safeguards in Retrieval-Augmented Generation Systems. Preprint retrieved from arXiv.
Cited in discussions about RAG vulnerabilities, vector database security, and output injection attacks. - Open Policy Agent (OPA) Project. (2023). Policy-as-Code Best Practices for Cloud Security and CI/CD Pipelines. Retrieved from https://www.openpolicyagent.org/
Used for the engineering secure pipelines section and embedding security gates into CI/CD. - IBM Research. (2023). Prompt Injection Attacks and Defenses in Generative AI Systems. Retrieved from https://research.ibm.com
Informs prompt sanitization techniques and instruction override mitigation strategies.
MITRE ATLAS™ Framework. (2023). Adversarial Threat Landscape for Artificial-Intelligence Systems. Retrieved from https://atlas.mitre.org/
Referenced for red-teaming AI systems and enumerating GenAI-specific attack vectors.