AI Threats Affecting Developers & AI Builders

This page documents AI security risks for developers and the LLM application security risks facing organizations building AI systems — including AI labs, open-source projects, platform providers, and companies deploying LLM-powered applications. It is intended for developers, security engineers, AI teams, engineering leaders, and open-source maintainers.

Developers and AI builders are classified under the Organizations category — groups where harm is experienced by institutional entities. This category distinguishes organizational-level impacts from individual harms (affecting natural persons) and systems-level harms (affecting societal structures like democracy or national security). Developers and AI builders are distinguished by their dual exposure: they are direct victims when their systems are attacked or their IP is stolen, and they bear responsibility when their systems cause downstream harm. When harm targets private sector operations more broadly (business organizations), public administration (government institutions), or essential services (critical infrastructure operators), those dedicated pages provide more targeted guidance.

This page summarizes recurring AI threat patterns, protective measures, and relevant regulatory context for developers and AI builders.

At a glance

• Primary threats: Supply chain attacks, prompt injection, model theft, data poisoning, downstream liability
• 43 documented incidents — the largest count of any organizational group, including the PyTorch supply chain compromise and indirect prompt injection attacks on LLM applications
• Key domains: Security & Cyber, Agentic Systems, Human-AI Control

---

How AI Threats Appear

The following are recurring patterns of AI-enabled harm documented across incidents affecting developers and AI builders. Each pattern reflects real-world events, not hypothetical risk.

Threat Pattern	Primary Domain	Key Indicator
Model theft and extraction	Security & Cyber	Systematic API queries indicating extraction attempts
Prompt injection and jailbreaking	Security & Cyber	Safety controls bypassed through prompt engineering
Data poisoning	Security & Cyber	Unexpected behavioral changes after training data updates
Supply chain attacks	Security & Cyber	Unvetted MCP servers, IDE integrations, or package dependencies
Downstream liability	Agentic Systems	AI agents with tool access lacking permission boundaries

• Model theft and extraction — Adversaries using systematic queries or side-channel attacks to replicate proprietary models
• Prompt injection and jailbreaking — Attacks that bypass safety controls, extract system prompts, or cause AI systems to behave in unintended ways
• Data poisoning — Manipulation of training data to introduce backdoors, biases, or vulnerabilities into models
• Supply chain attacks — Compromised dependencies, malicious model weights, or poisoned datasets introduced through the AI development pipeline
• Downstream liability — Harms caused by AI systems that generate legal, financial, or reputational consequences for the developers who built them

LLM application security risks

Applications built on large language models face a distinct threat surface:

• Indirect prompt injection — Malicious content in external data sources (web pages, emails, documents) that hijacks LLM behavior when the application processes those sources, enabling data exfiltration or unauthorized actions
• Tool and function call abuse — LLM-integrated applications that expose APIs, databases, or system commands through tool-use capabilities, creating paths for privilege escalation when safety boundaries are bypassed
• Context window poisoning — Attacks that manipulate the contents of an LLM’s context window to alter its behavior for subsequent interactions, persisting across conversation turns
• Hallucination exploitation — Adversaries who register domain names, create packages, or establish entities that match common LLM hallucinations, turning model errors into attack vectors (e.g., registering a package name that an LLM frequently recommends but does not exist)

AI SDK and development tool vulnerabilities

The development toolchain itself is an attack surface:

• MCP server compromises — Model Context Protocol servers that connect AI assistants to external tools can be compromised to execute arbitrary code in development environments
• IDE integration risks — AI coding assistants with file system, terminal, or network access that can be manipulated through crafted code comments or repository content
• Package manager poisoning — Malicious AI-related packages published to registries (PyPI, npm) with names similar to legitimate AI libraries, exploiting developers who install dependencies without verification
• Model weight tampering — Pretrained model files distributed through model hubs that contain embedded backdoors or malicious payloads activated under specific input conditions

---

Relevant AI Threat Domains

• Security & Cyber — Model extraction, adversarial attacks, and supply chain compromises targeting AI development infrastructure
• Agentic Systems — Autonomous AI agent failures, tool misuse, and privilege escalation in deployed applications
• Human-AI Control — Loss of control over deployed AI behavior, including jailbreaking and safety bypass
• Systemic Risk — Recursive self-improvement risks, capability overhang, and strategic misalignment in advanced AI systems

---

What to Watch For

Where the section above describes threat patterns, this section identifies concrete warning signs that developers, security engineers, and AI teams may encounter — and the immediate steps they can take in response.

• Model APIs accessible without rate limiting, query logging, or anomaly detection for extraction attempts — What developers can do: Implement query budgets, anomaly detection on API usage patterns, and watermarking on model outputs. Monitor for systematic probing patterns that indicate extraction attempts.

• Training pipelines that ingest uncurated web data without poisoning detection — What AI teams can do: Implement data provenance tracking and statistical anomaly detection for training datasets. Use held-out validation sets to detect unexpected behavioral changes after data updates.

• Deployed systems with safety controls that can be bypassed through prompt engineering — What developers can do: Assume prompt-level safety controls will be bypassed. Implement defense-in-depth with system-level guardrails, output filtering, and capability restrictions that do not depend on the model following instructions.

• Development tool dependencies (MCP servers, IDE integrations, package managers) with insufficient security vetting — What developers can do: Audit all AI development tool dependencies. Sandbox AI coding assistants and MCP servers with minimal required permissions. Verify package integrity before installation.

• Autonomous AI agents with tool access that lacks adequate sandboxing or permission boundaries — What developers can do: Apply principle of least privilege to all AI agent tool access. Implement human-in-the-loop approval for high-impact actions. Log all tool invocations for audit.

---

Protective Measures

These are practical steps developers, security engineers, and AI teams can take to secure AI systems throughout the development and deployment lifecycle.

• Defend against prompt injection — Prompt injection defense techniques protect deployed AI systems from manipulation and jailbreaking. The guide to preventing prompt injection covers implementation strategies.
• Detect data poisoning — Data poisoning detection tools identify compromised training data before it contaminates models. See the guide to detecting data poisoning for pipeline-level approaches.
• Secure the development supply chain — AI supply chain security practices address risks from dependencies, model weights, and datasets. The guide to securing AI supply chains provides structured frameworks.
• Test adversarial robustness — Adversarial input detection and red teaming for AI systems help identify vulnerabilities before deployment. See the guide to detecting adversarial inputs and AI red teaming guide.
• Implement operational safeguards — AI audit and logging systems maintain records of system behavior for debugging and accountability. The AI safety tools overview surveys the defensive tooling landscape.

Questions developers should ask before deploying LLM applications

Use these during security review and pre-deployment evaluation of AI-powered applications.

• “What happens if an adversary controls content in the model’s context window — can they trigger tool calls or data exfiltration?”
• “Are safety controls enforced at the system level, or do they depend on the model following prompt instructions?”
• “What is our response plan if a jailbreak bypasses safety controls in production?”
• “Have we tested this application with adversarial inputs crafted by someone outside the development team?”

Questions engineering leaders should ask AI vendors and open-source projects

Use these when evaluating AI components, SDKs, or model providers for integration into your stack.

• “What security testing has been conducted on this model, SDK, or tool against known attack patterns?”
• “How are model weights and training data verified for integrity before distribution?”
• “What is the vulnerability disclosure and patching process for this AI component?”
• “What are the liability boundaries if this AI component causes downstream harm in our application?”

---

Regulatory Context

• EU AI Act — Imposes obligations on AI providers (developers) for high-risk systems, including conformity assessments, post-market monitoring, and incident reporting
• ISO/IEC 42001 — Provides management system requirements for organizations developing AI, covering governance, risk management, and continuous improvement
• NIST AI RMF — Addresses AI risk management throughout the development lifecycle, with specific guidance on testing, evaluation, and monitoring

Regulatory frameworks are still catching up to the AI development ecosystem, and open-source AI development carries unique liability and governance questions that vary significantly across jurisdictions.

---

Documented Incidents

Based on incident analysis, developers and AI builders are most frequently affected by threats in the Security & Cyber and Agentic Systems domains, reflecting the convergence of supply chain attacks, prompt injection exploits, and autonomous agent failures.