AI in Application Penetration Testing: It’s Time to Go with the Flow - But Not Blindly

Artificial Intelligence is no longer a "good to have" in cybersecurity—it’s rapidly becoming a force multiplier. From solving complex challenges in CTFs to automating reconnaissance, exploitation, and even report generation, AI-driven penetration testing is demonstrating measurable promise.

But enterprise security is not a playground. It’s a controlled, high-stakes environment where assumptions can translate into real risk. As organizations begin to evaluate AI as a replacement—or augmentation—for human-led penetration testing, it’s critical to pause and ask the right questions.

• Is AI Penetration Testing Production Safe? - AI tools operate at speed and scale. Without strict guardrails, this introduces a real risk – unintended exploitation of live applications, service disruptions etc. Unlike human testers, AI does not inherently understand "safe boundaries" unless explicitly constrained.
• Is AI Only as Good as Its Prompter? – A prompt-orchestrated testing raises a fundamental dependency - the quality of findings is directly tied to the operator’s expertise. In effect, we may not be replacing human intelligence—we’re reshaping it.
• Can AI Replicate True Human Intelligence? - Some of the most critical vulnerabilities are not pattern-based—they are contextual (business logic flaws, privilege escalation chains etc.). These require situational awareness and integrated data flow understanding.
• Enterprise Reality: Integrated Application Ecosystems – In large enterprises, applications are interconnected - data flows across APIs, services, and third-party platforms. Security issues often emerge between systems—not within them. AI tools, unless specifically architected for this, may miss this integration.
• The False Positive Problem - AI can generate large volumes of findings quickly but is there still a need for manual triage? Are we shifting effort from "finding vulnerabilities" to "filtering noise"? Without a robust validation layer, organizations risk drowning in output with limited actionable intelligence.
• Data Privacy and Model Risk – AI thrives on data. By using AI penetration testers, are we risking data leakage and could this data be used to train third-party models? For many enterprises, this alone could be a blocker.
• Where Does the AI Pen Tester Sit in Your Network? - Deploying AI testing introduces architectural questions – does it require internet exposure? Is it deployed with full network access? What controls prevent lateral misuse if compromised?

The Way Forward: A Controlled, Measurable Approach

We are clearly at a turning point. AI in penetration testing is not a question of if—it’s a question of how and when. But premature adoption without structured evaluation can weaken, rather than strengthen, security posture. Organizations should resist binary thinking (AI vs Human) and instead focus on comparative validation:

• Conduct PoCs on real enterprise applications
• Benchmark AI-driven vs human-led testing
• Evaluate across:

• Depth of findings
• False positive rates
• Coverage of business logic vulnerabilities
• Time-to-deliver and cost efficiency 

AI is accelerating. Agent creation is becoming effortless. Automation is redefining scale. But security has never been about speed alone—it’s about precision, context, and judgment. We need to engineer the right balance between human intelligence and machine capability. 

We certainly should use AI in application security - it brings scale, speed, and the ability to uncover patterns that would otherwise take significant manual effort but the need of human intelligence cannot be completely written off. AI can accelerate discovery. Humans ensure relevance, accuracy, and real-world impact. Together, they create a security model that is not only efficient, but also resilient and trustworthy. Organizations that recognize this balance early will not just keep up with the shift—they will define it.

Article by Hemil Shah and Rishita Sarabhai 

[Case Study] Threat Simulation of AI Agents in Microsoft Copilot Studio

Executive Summary

Blueinfy performed a focused, time-bound security review of Microsoft Copilot Studio and its implementation at ACME to assess the potential risks introduced by AI agents.

The objective of the engagement was to evaluate how AI agents both legitimate and malicious could be misused, intentionally or unintentionally, to 

• Access sensitive enterprise data
• Expose user specific information
• Perform unauthorized actions
• Enable data exfiltration

The assessment combined configuration review with hands on threat simulation, where custom agents were built to replicate realistic attack scenarios as well as instructions were passed to exploit legitimate agents. The results demonstrated that even with platform level controls enabled, significant risks can persist due to configuration gaps, excessive permissions, and agent behavior manipulation.

The environments given for testing had pre-configured policies and controls applied prior to the assessment. The scope included - 

• AI agent configuration within Microsoft Copilot Studio
• Data access patterns through agents
• Connector usage and restrictions
• Guardrails and safety configurations
• Threat simulation using custom-built agents

Assessment Methodology

Blueinfy adopted a structured methodology combining configuration validation and adversarial testing.

1. AI Configuration Review

A focused configuration review was conducted to evaluate AI-specific settings. 

Areas Reviewed:
•    AI agent configuration settings
•    Connector policies and restrictions
•    Data access configurations
•    Prompt safety and guardrails
•    Logging and monitoring capabilities

Objective:
•    Identify risky configurations
•    Recommend controls to reduce exposure
•    Highlight configurations requiring governance before enablement

2. Agent Threat Simulation

Instead of testing existing agents, Blueinfy created custom agents within the allowed policy boundaries to simulate real world attack scenarios. This approach ensured:

• No disruption to production agents
• Realistic exploitation within permitted configurations
• Validation of platform controls under adversarial conditions

Threat Simulation Approach

Agents were built using only approved connectors and policies within the environment. Two categories of agents were designed:

1. Misuse of Legitimate Agents

• Agents behaving as intended but manipulated via inputs
• Exploiting trust in user prompts

2. Malicious Agent Design

• Agents intentionally designed to bypass safeguards
• Leveraging allowed configurations to simulate abuse

Key Attack Scenarios Tested

Blueinfy executed multiple scenarios to evaluate risk exposure:

• Prompt Injection and Instruction Override - Manipulating agent behavior using crafted inputs to override system instructions and cause unintended data access
• Data Exfiltration via Allowed Channels - Extracting sensitive data through email connectors, API responses and structured outputs
• Cross-Agent Interaction Risks - Simulating agent-to-agent communication and demonstrating potential lateral movement
• Rouge Agents – Malicious agents built with system instructions to exfiltrate data, phish users for credentials and send application/user data to unintended servers
• MCP Exposure – If MCP server and tools are accessible without correct authentication and authorization mechanisms

Key Observations

The assessment revealed several important findings:

• Misconfigurations Create Hidden Risks – Users of the agents are completely unaware of the data risks since once published/shared, end users have limited visibility in the agent configuration. We were able to send emails of agent users including email attachments and employee feedback responses etc. to our third-party servers.
• Agents Can Be Manipulated Through Inputs – Based on the guardrails, prompt injection enabled behavior override and agents could be influenced to exfiltrate data without changing configuration. This created a scenario where legitimate agents to summarize users emails, posting summary to Teams channels etc. could be exploited to share that summary data to third-party servers via malicious instructions received in email/submitted forms.
• Unauthenticated MCP Exposure – MCP Tools connect the LLM's to organization data sources like databases, knowledge sources etc. With this engagement, we were able to use the MCP tools without authentication and gain full access to client sensitive data like contract financials.
• Platform Controls Are Not Sufficient Alone - While Microsoft Copilot Studio provides robust built-in controls, their effectiveness depends heavily on configuration and usage. Sadly, they do not work without being configured per your needs.  

Conclusion

Blueinfy’s assessment demonstrated that while platforms like Microsoft Copilot Studio do provide strong foundational controls, they must be complemented with:

• Proper configuration
• Risk-aware governance
• Adversarial testing
• Monitoring and logging

By moving from assumption based security to evidence driven validation, ACME established a stronger foundation for secure AI adoption. Blueinfy team worked with ACME to create a robust agent threat simulation and security review process to protect against such risks with scaling agents in parallel. Please read this blog for the three-tier risk methodology for an agent review process.

Article by Hemil Shah and Rishita Sarabhai 

The Rise of AI Agents and the urgent need for an Agent Security Review Process

Organizations today are rapidly embracing AI-powered agents. Platforms like Microsoft Copilot Studio and Google Gemini are enabling business users, not just developers, to create powerful agents that automate workflows, access enterprise data, and make decisions. This democratization is transformative. But it also introduces a new, largely ungoverned attack surface.

The Explosion of Agents

In many enterprises, the number of agents being deployed is growing exponentially from hundreds, sometimes thousands, within a short span of time. These agents:

• Integrate with internal systems
• Access sensitive enterprise data
• Perform automated actions on behalf of users

Unlike traditional applications, these agents are often created outside formal development pipelines by business users, analysts, or developers. And that’s where the problem begins.

The Security Gap: No "AgentSec"

Organizations have matured practices for AppSec or InfraSec or Cloud Security but Agent Security (AgentSec) is still in its infancy.
There is typically:

• No formal review process before agent deployment
• Limited visibility into what agents are doing
• No standardized threat modeling for agent behavior
• Weak validation of platform-level security controls

This creates a dangerous blind spot.

Built-in Controls Are Not Enough

Platforms do provide security mechanisms at:

• Data access controls
• Authentication and authorization layers
• Prompt filtering and safety guardrails
• Activity monitoring

However, these controls are:

• Complex to configure correctly
• Highly dependent on implementation choices
• Difficult to validate in real-world scenarios

Misconfigurations or misunderstandings can easily render these protections ineffective.

Visualizing the Risk: Agent Attack Flow

The Missing Piece: A Scalable Agent Review Process

At first glance, the solution seems straightforward: introduce agent design reviews, configuration assessments, and threat modeling for every agent. But in reality, this approach does not scale.

In large enterprises with hundreds or thousands of agents built on platforms, performing deep security reviews on every agent would:

• Overwhelm security teams
• Slow down innovation
• Create operational bottlenecks

Instead, organizations must adopt a risk-based Agent Security (AgentSec) model. The three-tier risk model classifies agents based on their potential impact and exposure. 
 

• High-risk agents are typically misconfigured or intentionally malicious, capable of unsafe actions such as exfiltrating data to external emails, interacting with unauthorized external URLs, or executing harmful embedded instructions. 
• Medium-risk agents involve broader data interaction—often consuming sensitive or user-provided inputs through connectors, APIs, MCP integrations, or multi-agent communication—making them more prone to misuse or unintended data exposure. 
• Low-risk agents operate within a constrained scope, relying on public or read-only data sources such as web search, uploaded files, SharePoint, or Dataverse, with minimal ability to cause harm.

Automation enables scale by classifying the agents into risk buckets and a focused review can then be performed only for high-risk and medium-risk agents to assess the business impact by building abuse/exploit scenarios. This approach ensures that security teams invest effort where it truly matters - prioritizing depth and accuracy over volume.

Why This Model Works

This approach delivers both speed and security: fast approvals for low-risk agents, strong scrutiny for higher-risk ones, reduced burden on security teams, and scalable governance across thousands of agents. Most importantly, it aligns security effort with actual risk - not perceived risk.

The organizations that succeed will not be those attempting to review every agent, but those that automate the baseline, enforce non-negotiable security gates, and escalate only what truly matters. Because in a world of thousands of agents, scalability itself becomes security.

Article by Hemil Shah and Rishita Sarabhai 

Agentic AI Security - Threats and Attacks (Paper Review)

Agentic AI systems transform LLMs into autonomous operators that plan, call tools, use memory, and act across web, code, APIs, and even physical environments, which radically enlarges the attack surface beyond simple chatbots. The paper frames security for these systems around concrete threat families: prompt injection and jailbreaks; autonomous cyber‑exploitation with tool abuse; multi‑agent and protocol‑level attacks (including MCP and agent‑to‑agent ecosystems); and environment/interface issues such as unsafe action spaces and brittle web interaction. These systems must therefore be treated as distributed, partially trusted components that can both be attacked and weaponized as attackers themselves.

Prompt‑centric threats are broken down into direct and indirect prompt injection, intentional and unintentional attacks, multi‑modal and hybrid payloads (text, images, audio, code), propagation behaviors, and multilingual/obfuscated or split payloads that evade naive filters. Attackers can poison external content sources (web pages, PDFs, accessibility trees, APIs), craft adversarial code/SQL prompts, or hide instructions in non‑text modalities to hijack the agent’s plan and tool calls. The work also highlights that many proposed PI defenses are brittle, with adaptive IPI attacks able to bypass perplexity‑based and pattern‑based detectors in practice, which reinforces PI as a primary attack vector against agentic workflows.

On the offensive operations side, the paper shows that agents with code execution and network access can autonomously perform vulnerability discovery and exploitation, often outperforming traditional tools like OWASP ZAP or Metasploit on known‑vulnerable targets when given CVE descriptions and appropriate tools. Demonstrated capabilities include chaining XSS, CSRF, SSTI, and SQLi, navigating web apps in realistic sandboxes, and leveraging tools to iteratively refine exploits without human guidance. In multi‑agent and protocol‑driven settings (e.g., MCP or cross‑org agent meshes), they describe additional vectors such as fake or compromised agent registration, denial of service via recursive delegation, transitive prompt‑injection across agents, memory poisoning, and identity or role abuse that propagates through the agent network.

Reference: 

AGENTIC AI SECURITY:THREATS, DEFENSES, EVALUATION, AND OPEN CHALLENGES - https://arxiv.org/pdf/2510.23883 

 

Why Agentic Pentesting Can’t Fix the False Positive Problem

Agentic pentesting promises smarter orchestration of tools, but it does not magically eliminate false positives. At its core, an agent still leans on the same scanners, payload generators, and detection heuristics that produced noisy results in the first place. If the underlying tools misclassify behavior or lack application context, the agent simply becomes a faster, more automated way to generate and route those misclassifications. In other words, you risk “scaling the noise” as much as scaling the signal.

Another limitation is that most agentic systems still struggle with business context and intent, which is where many false positives are born. A finding that looks critical in HTTP traces might be benign in the real-world workflow because of compensating controls, domain‑specific logic, or risk acceptance decisions that only humans understand. Agents can replay exploits and correlate signals, but they cannot reliably answer questions like “Is this test user data or real PII?” or “Would exploiting this actually harm the business?” Without that judgment, they often cannot confidently close the loop on whether something is truly a vulnerability or just an academic issue.

Finally, agentic pentesting introduces its own new sources of error that can masquerade as false positives. Misconfigured prompts, overly broad goals, or aggressive automation can lead agents to test unsupported flows, mis-handle authentication, or misinterpret application responses. These mistakes can create “findings” that look real on paper but collapse under minimal human scrutiny. So while agentic approaches can help prioritize, group, and sometimes auto‑retest issues, they do not remove the need for human validation; they merely change where you spend your validation effort—from sifting through raw scanner output to scrutinizing AI‑curated results.

SSRF in Azure MCP Server Tools

In Microsoft's March 2026 Patch Tuesday release on March 10, an urgent high-severity vulnerability, CVE-2026-26118, emerged in Azure Model Context Protocol (MCP) Server Tools. This server-side request forgery (SSRF) flaw, scored at CVSS 8.8, allows low-privileged attackers to manipulate user-supplied inputs and force the server into making unauthorized outbound requests to attacker-controlled endpoints. MCP, designed to standardize AI model integrations with external data sources, unexpectedly became a vector for privilege escalation in AI-driven Azure environments, highlighting the growing risks in agentic AI architectures.

At its core, exploitation involves crafting malicious payloads that trick the MCP server—running versions prior to 2.0.0-beta.17—into leaking its managed identity token. Attackers can then impersonate the server's identity to access sensitive Azure resources like storage accounts, virtual machines, or databases, all without needing admin rights or user interaction. Public proof-of-concept exploits, such as those on GitHub, amplify the threat, enabling rapid weaponization in targeted attacks against organizations leveraging MCP for AI workflows. This vulnerability underscores a classic SSRF pattern (CWE-918) but tailored to cloud-native AI tools, where broad service principals often grant excessive permissions.

Organizations should prioritize patching via Microsoft's Security Update Guide, audit MCP deployments for over-privileged identities, and implement outbound request filtering to contain risks. As AI security evolves, this incident signals the need for runtime protections in MCP-based systems, including token rotation and anomaly detection for AI agent traffic. Application security teams, especially those testing AI integrations, can use tools like Burp Suite to validate fixes against SSRF payloads. Staying vigilant ensures AI innovation doesn't outpace defense in the cloud.

Reference - https://www.tenable.com/cve/CVE-2026-26118

Supply Chains and AI: Decoding OWASP Top 10 2026 Changes

OWASP’s 2026 Top 10 reflects how quickly modern application threats are evolving, especially with AI-heavy and highly distributed architectures. The list continues to emphasize long-standing problems like Broken Access Control and Cryptographic Failures, but the new edition elevates security misconfigurations and software supply chain issues as first-class risks. This shift acknowledges that complex CI/CD pipelines, third‑party services, and AI-powered components have dramatically expanded the attack surface beyond just your own code.

A key change in 2026 is the explicit spotlight on software supply chain failures and the mishandling of exceptional conditions. These categories capture real‑world issues such as compromised libraries, poisoned models, insecure infrastructure-as-code templates, and fragile error handling that leads to data leakage or privilege escalation. Rather than treating these as edge cases, OWASP now frames them as systemic risks that can undermine even well‑written business logic. For teams shipping fast, this is a wake‑up call that “secure by default” must include dependencies, pipelines, and runtime behavior—not just input validation and authentication.

The importance of the 2026 Top 10 lies in how it guides priorities for engineering, security architecture, and governance. It gives product and security leaders a shared vocabulary to justify investments in SBOMs, dependency scanning, secure AI integration patterns, and runtime protection. For practitioners, it acts as a practical roadmap: threat modeling features around these categories, aligning test cases and code reviews with them, and measuring progress over time. In a world where AI agents, APIs, and microservices are deeply interwoven, using the updated OWASP Top 10 as a baseline can be the difference between a resilient platform and one supply‑chain incident away from a major breach.