[Case Study] Building an AI Security Program for a Global Investment Firm

A multinational investment firm started adopting AI across the organization - through enterprise platforms like Google Gemini enterprise and independently within business units for vibe coding, data analytics, and customer facing use cases. This did help teams move faster but it also created a need to bring consistency, visibility, and security around how AI was being used.

Blueinfy was engaged to support the organization in setting up a structured AI Security Program that could scale with this adoption without slowing down innovation. The approach focused on creating a correct balance between governance and flexibility ensuring that AI could grow across the organization, but in a more controlled and visible manner.

Challenges

The key challenge was the way AI adoption had expanded in the organization - fast, scattered, and largely independent across teams. While enterprise tools provided scale, business units were along-side experimenting with different AI solutions, making it difficult to maintain a consistent security approach. 

There was limited visibility into how AI was being used, what kind of data was being shared, and which external tools were involved. At the same time, emerging risks such as overly permissive AI agents, unrestricted integrations, and unintended data exposure through prompts and workflows were becoming harder to track. 

From an execution standpoint, aligning multiple teams, ensuring the right access and prerequisites, and bringing everyone to a common approach required continuous coordination and validation. 

The organization’s AI adoption approach created distinct risk areas:

• AI usage was growing without a single view of where and how it was being used
• Different business units were following their own approaches, leading to inconsistency
• Sensitive user and enterprise data was being shared with AI systems without clear guardrails
• There was limited validation of AI use cases from a security standpoint
• Third-party AI tools as well as code generated by AI were not reviewed in detail

Overall, the challenge was less about lack of intent, and more about the absence of a structured approach.

Solution / Approach

Blueinfy aligned the overall approach around a single ownership model, supported by targeted and continuous activities.

 At a high level, a dedicated AI Security Program Lead was introduced to take comprehensive responsibility for AI security across the organization. This role acted as the central coordination point ensuring visibility, consistency, and alignment across security, IT, and business units.

For Business Units, the focus was on enablement. Teams were supported with clear guidance, practical do's and don'ts, and secure usage patterns. This allowed them to continue building and experimenting with AI without unnecessary resistance.

As part of this enablement, Blueinfy also helped define and roll out standardized documentation and guidelines, including:

• AI implementation guidelines covering architecture, integrations, and connectivity
• Access control and permission models for AI tools, agents, and APIs
• Guardrails for safe data usage, prompt handling, and output validation
• Responsible use of AI guidelines for end users (what can and cannot be shared with AI systems)
• Lightweight review and approval processes for new AI use cases

These documents provided a consistent baseline for teams, reducing ambiguity and improving adoption of secure practices.

For Enterprise AI platforms, a structured validation approach was followed. A threat simulation exercise was conducted to identify potential risks such as data exposure, misuse scenarios, and integration weaknesses.
Based on these insights, a continuous validation model was introduced:

• Agent security reviews to assess workflows, permissions, and integrations
• AI red teaming for new models and high-risk use cases
• Penetration testing for AI-driven customer-facing implementations

This ensured that AI security was not a one-time activity, but an ongoing process embedded into how new AI capabilities were introduced.

Outcome

With this model in place, the organization was able to bring more structure to its AI adoption without slowing down innovation.

• A clear ownership model improved coordination and decision-making
• Better visibility into AI use cases reduced unmanaged or "shadow" AI risks
• Business units were able to innovate with clearer guidance and fewer blockers
• Standardized guidelines helped teams follow consistent and secure practices
• Risks related to data exposure, integrations, and agent behavior were identified earlier
• Continuous reviews ensured that new AI implementations were assessed as they were introduced

Overall, the shift from one-time assessments to a continuous validation approach, supported by clear documentation and ownership, helped the organization stay aligned with the pace at which AI was evolving internally.

Conclusion

AI adoption in large organizations will naturally be fast and distributed. The real challenge is not controlling it completely, but making sure it grows in a structured and secure way.

This engagement shows that with clear ownership, practical guidance, and ongoing validation, organizations can build a sustainable AI security program that supports both innovation and risk management.

AI in Application Penetration Testing: It’s Time to Go with the Flow - But Not Blindly

Artificial Intelligence is no longer a "good to have" in cybersecurity—it’s rapidly becoming a force multiplier. From solving complex challenges in CTFs to automating reconnaissance, exploitation, and even report generation, AI-driven penetration testing is demonstrating measurable promise.

But enterprise security is not a playground. It’s a controlled, high-stakes environment where assumptions can translate into real risk. As organizations begin to evaluate AI as a replacement—or augmentation—for human-led penetration testing, it’s critical to pause and ask the right questions.

• Is AI Penetration Testing Production Safe? - AI tools operate at speed and scale. Without strict guardrails, this introduces a real risk – unintended exploitation of live applications, service disruptions etc. Unlike human testers, AI does not inherently understand "safe boundaries" unless explicitly constrained.
• Is AI Only as Good as Its Prompter? – A prompt-orchestrated testing raises a fundamental dependency - the quality of findings is directly tied to the operator’s expertise. In effect, we may not be replacing human intelligence—we’re reshaping it.
• Can AI Replicate True Human Intelligence? - Some of the most critical vulnerabilities are not pattern-based—they are contextual (business logic flaws, privilege escalation chains etc.). These require situational awareness and integrated data flow understanding.
• Enterprise Reality: Integrated Application Ecosystems – In large enterprises, applications are interconnected - data flows across APIs, services, and third-party platforms. Security issues often emerge between systems—not within them. AI tools, unless specifically architected for this, may miss this integration.
• The False Positive Problem - AI can generate large volumes of findings quickly but is there still a need for manual triage? Are we shifting effort from "finding vulnerabilities" to "filtering noise"? Without a robust validation layer, organizations risk drowning in output with limited actionable intelligence.
• Data Privacy and Model Risk – AI thrives on data. By using AI penetration testers, are we risking data leakage and could this data be used to train third-party models? For many enterprises, this alone could be a blocker.
• Where Does the AI Pen Tester Sit in Your Network? - Deploying AI testing introduces architectural questions – does it require internet exposure? Is it deployed with full network access? What controls prevent lateral misuse if compromised?

The Way Forward: A Controlled, Measurable Approach

We are clearly at a turning point. AI in penetration testing is not a question of if—it’s a question of how and when. But premature adoption without structured evaluation can weaken, rather than strengthen, security posture. Organizations should resist binary thinking (AI vs Human) and instead focus on comparative validation:

• Conduct PoCs on real enterprise applications
• Benchmark AI-driven vs human-led testing
• Evaluate across:

• Depth of findings
• False positive rates
• Coverage of business logic vulnerabilities
• Time-to-deliver and cost efficiency 

AI is accelerating. Agent creation is becoming effortless. Automation is redefining scale. But security has never been about speed alone—it’s about precision, context, and judgment. We need to engineer the right balance between human intelligence and machine capability. 

We certainly should use AI in application security - it brings scale, speed, and the ability to uncover patterns that would otherwise take significant manual effort but the need of human intelligence cannot be completely written off. AI can accelerate discovery. Humans ensure relevance, accuracy, and real-world impact. Together, they create a security model that is not only efficient, but also resilient and trustworthy. Organizations that recognize this balance early will not just keep up with the shift—they will define it.

Article by Hemil Shah and Rishita Sarabhai 

[Case Study] Threat Simulation of AI Agents in Microsoft Copilot Studio

Executive Summary

Blueinfy performed a focused, time-bound security review of Microsoft Copilot Studio and its implementation at ACME to assess the potential risks introduced by AI agents.

The objective of the engagement was to evaluate how AI agents both legitimate and malicious could be misused, intentionally or unintentionally, to 

• Access sensitive enterprise data
• Expose user specific information
• Perform unauthorized actions
• Enable data exfiltration

The assessment combined configuration review with hands on threat simulation, where custom agents were built to replicate realistic attack scenarios as well as instructions were passed to exploit legitimate agents. The results demonstrated that even with platform level controls enabled, significant risks can persist due to configuration gaps, excessive permissions, and agent behavior manipulation.

The environments given for testing had pre-configured policies and controls applied prior to the assessment. The scope included - 

• AI agent configuration within Microsoft Copilot Studio
• Data access patterns through agents
• Connector usage and restrictions
• Guardrails and safety configurations
• Threat simulation using custom-built agents

Assessment Methodology

Blueinfy adopted a structured methodology combining configuration validation and adversarial testing.

1. AI Configuration Review

A focused configuration review was conducted to evaluate AI-specific settings. 

Areas Reviewed:
•    AI agent configuration settings
•    Connector policies and restrictions
•    Data access configurations
•    Prompt safety and guardrails
•    Logging and monitoring capabilities

Objective:
•    Identify risky configurations
•    Recommend controls to reduce exposure
•    Highlight configurations requiring governance before enablement

2. Agent Threat Simulation

Instead of testing existing agents, Blueinfy created custom agents within the allowed policy boundaries to simulate real world attack scenarios. This approach ensured:

• No disruption to production agents
• Realistic exploitation within permitted configurations
• Validation of platform controls under adversarial conditions

Threat Simulation Approach

Agents were built using only approved connectors and policies within the environment. Two categories of agents were designed:

1. Misuse of Legitimate Agents

• Agents behaving as intended but manipulated via inputs
• Exploiting trust in user prompts

2. Malicious Agent Design

• Agents intentionally designed to bypass safeguards
• Leveraging allowed configurations to simulate abuse

Key Attack Scenarios Tested

Blueinfy executed multiple scenarios to evaluate risk exposure:

• Prompt Injection and Instruction Override - Manipulating agent behavior using crafted inputs to override system instructions and cause unintended data access
• Data Exfiltration via Allowed Channels - Extracting sensitive data through email connectors, API responses and structured outputs
• Cross-Agent Interaction Risks - Simulating agent-to-agent communication and demonstrating potential lateral movement
• Rouge Agents – Malicious agents built with system instructions to exfiltrate data, phish users for credentials and send application/user data to unintended servers
• MCP Exposure – If MCP server and tools are accessible without correct authentication and authorization mechanisms

Key Observations

The assessment revealed several important findings:

• Misconfigurations Create Hidden Risks – Users of the agents are completely unaware of the data risks since once published/shared, end users have limited visibility in the agent configuration. We were able to send emails of agent users including email attachments and employee feedback responses etc. to our third-party servers.
• Agents Can Be Manipulated Through Inputs – Based on the guardrails, prompt injection enabled behavior override and agents could be influenced to exfiltrate data without changing configuration. This created a scenario where legitimate agents to summarize users emails, posting summary to Teams channels etc. could be exploited to share that summary data to third-party servers via malicious instructions received in email/submitted forms.
• Unauthenticated MCP Exposure – MCP Tools connect the LLM's to organization data sources like databases, knowledge sources etc. With this engagement, we were able to use the MCP tools without authentication and gain full access to client sensitive data like contract financials.
• Platform Controls Are Not Sufficient Alone - While Microsoft Copilot Studio provides robust built-in controls, their effectiveness depends heavily on configuration and usage. Sadly, they do not work without being configured per your needs.  

Conclusion

Blueinfy’s assessment demonstrated that while platforms like Microsoft Copilot Studio do provide strong foundational controls, they must be complemented with:

• Proper configuration
• Risk-aware governance
• Adversarial testing
• Monitoring and logging

By moving from assumption based security to evidence driven validation, ACME established a stronger foundation for secure AI adoption. Blueinfy team worked with ACME to create a robust agent threat simulation and security review process to protect against such risks with scaling agents in parallel. Please read this blog for the three-tier risk methodology for an agent review process.

Article by Hemil Shah and Rishita Sarabhai 

The Rise of AI Agents and the urgent need for an Agent Security Review Process

Organizations today are rapidly embracing AI-powered agents. Platforms like Microsoft Copilot Studio and Google Gemini are enabling business users, not just developers, to create powerful agents that automate workflows, access enterprise data, and make decisions. This democratization is transformative. But it also introduces a new, largely ungoverned attack surface.

The Explosion of Agents

In many enterprises, the number of agents being deployed is growing exponentially from hundreds, sometimes thousands, within a short span of time. These agents:

• Integrate with internal systems
• Access sensitive enterprise data
• Perform automated actions on behalf of users

Unlike traditional applications, these agents are often created outside formal development pipelines by business users, analysts, or developers. And that’s where the problem begins.

The Security Gap: No "AgentSec"

Organizations have matured practices for AppSec or InfraSec or Cloud Security but Agent Security (AgentSec) is still in its infancy.
There is typically:

• No formal review process before agent deployment
• Limited visibility into what agents are doing
• No standardized threat modeling for agent behavior
• Weak validation of platform-level security controls

This creates a dangerous blind spot.

Built-in Controls Are Not Enough

Platforms do provide security mechanisms at:

• Data access controls
• Authentication and authorization layers
• Prompt filtering and safety guardrails
• Activity monitoring

However, these controls are:

• Complex to configure correctly
• Highly dependent on implementation choices
• Difficult to validate in real-world scenarios

Misconfigurations or misunderstandings can easily render these protections ineffective.

Visualizing the Risk: Agent Attack Flow

The Missing Piece: A Scalable Agent Review Process

At first glance, the solution seems straightforward: introduce agent design reviews, configuration assessments, and threat modeling for every agent. But in reality, this approach does not scale.

In large enterprises with hundreds or thousands of agents built on platforms, performing deep security reviews on every agent would:

• Overwhelm security teams
• Slow down innovation
• Create operational bottlenecks

Instead, organizations must adopt a risk-based Agent Security (AgentSec) model. The three-tier risk model classifies agents based on their potential impact and exposure. 
 

• High-risk agents are typically misconfigured or intentionally malicious, capable of unsafe actions such as exfiltrating data to external emails, interacting with unauthorized external URLs, or executing harmful embedded instructions. 
• Medium-risk agents involve broader data interaction—often consuming sensitive or user-provided inputs through connectors, APIs, MCP integrations, or multi-agent communication—making them more prone to misuse or unintended data exposure. 
• Low-risk agents operate within a constrained scope, relying on public or read-only data sources such as web search, uploaded files, SharePoint, or Dataverse, with minimal ability to cause harm.

Automation enables scale by classifying the agents into risk buckets and a focused review can then be performed only for high-risk and medium-risk agents to assess the business impact by building abuse/exploit scenarios. This approach ensures that security teams invest effort where it truly matters - prioritizing depth and accuracy over volume.

Why This Model Works

This approach delivers both speed and security: fast approvals for low-risk agents, strong scrutiny for higher-risk ones, reduced burden on security teams, and scalable governance across thousands of agents. Most importantly, it aligns security effort with actual risk - not perceived risk.

The organizations that succeed will not be those attempting to review every agent, but those that automate the baseline, enforce non-negotiable security gates, and escalate only what truly matters. Because in a world of thousands of agents, scalability itself becomes security.

Article by Hemil Shah and Rishita Sarabhai 

Agentic AI Security - Threats and Attacks (Paper Review)

Agentic AI systems transform LLMs into autonomous operators that plan, call tools, use memory, and act across web, code, APIs, and even physical environments, which radically enlarges the attack surface beyond simple chatbots. The paper frames security for these systems around concrete threat families: prompt injection and jailbreaks; autonomous cyber‑exploitation with tool abuse; multi‑agent and protocol‑level attacks (including MCP and agent‑to‑agent ecosystems); and environment/interface issues such as unsafe action spaces and brittle web interaction. These systems must therefore be treated as distributed, partially trusted components that can both be attacked and weaponized as attackers themselves.

Prompt‑centric threats are broken down into direct and indirect prompt injection, intentional and unintentional attacks, multi‑modal and hybrid payloads (text, images, audio, code), propagation behaviors, and multilingual/obfuscated or split payloads that evade naive filters. Attackers can poison external content sources (web pages, PDFs, accessibility trees, APIs), craft adversarial code/SQL prompts, or hide instructions in non‑text modalities to hijack the agent’s plan and tool calls. The work also highlights that many proposed PI defenses are brittle, with adaptive IPI attacks able to bypass perplexity‑based and pattern‑based detectors in practice, which reinforces PI as a primary attack vector against agentic workflows.

On the offensive operations side, the paper shows that agents with code execution and network access can autonomously perform vulnerability discovery and exploitation, often outperforming traditional tools like OWASP ZAP or Metasploit on known‑vulnerable targets when given CVE descriptions and appropriate tools. Demonstrated capabilities include chaining XSS, CSRF, SSTI, and SQLi, navigating web apps in realistic sandboxes, and leveraging tools to iteratively refine exploits without human guidance. In multi‑agent and protocol‑driven settings (e.g., MCP or cross‑org agent meshes), they describe additional vectors such as fake or compromised agent registration, denial of service via recursive delegation, transitive prompt‑injection across agents, memory poisoning, and identity or role abuse that propagates through the agent network.

Reference: 

AGENTIC AI SECURITY:THREATS, DEFENSES, EVALUATION, AND OPEN CHALLENGES - https://arxiv.org/pdf/2510.23883 

 

Why Agentic Pentesting Can’t Fix the False Positive Problem

Agentic pentesting promises smarter orchestration of tools, but it does not magically eliminate false positives. At its core, an agent still leans on the same scanners, payload generators, and detection heuristics that produced noisy results in the first place. If the underlying tools misclassify behavior or lack application context, the agent simply becomes a faster, more automated way to generate and route those misclassifications. In other words, you risk “scaling the noise” as much as scaling the signal.

Another limitation is that most agentic systems still struggle with business context and intent, which is where many false positives are born. A finding that looks critical in HTTP traces might be benign in the real-world workflow because of compensating controls, domain‑specific logic, or risk acceptance decisions that only humans understand. Agents can replay exploits and correlate signals, but they cannot reliably answer questions like “Is this test user data or real PII?” or “Would exploiting this actually harm the business?” Without that judgment, they often cannot confidently close the loop on whether something is truly a vulnerability or just an academic issue.

Finally, agentic pentesting introduces its own new sources of error that can masquerade as false positives. Misconfigured prompts, overly broad goals, or aggressive automation can lead agents to test unsupported flows, mis-handle authentication, or misinterpret application responses. These mistakes can create “findings” that look real on paper but collapse under minimal human scrutiny. So while agentic approaches can help prioritize, group, and sometimes auto‑retest issues, they do not remove the need for human validation; they merely change where you spend your validation effort—from sifting through raw scanner output to scrutinizing AI‑curated results.

SSRF in Azure MCP Server Tools

In Microsoft's March 2026 Patch Tuesday release on March 10, an urgent high-severity vulnerability, CVE-2026-26118, emerged in Azure Model Context Protocol (MCP) Server Tools. This server-side request forgery (SSRF) flaw, scored at CVSS 8.8, allows low-privileged attackers to manipulate user-supplied inputs and force the server into making unauthorized outbound requests to attacker-controlled endpoints. MCP, designed to standardize AI model integrations with external data sources, unexpectedly became a vector for privilege escalation in AI-driven Azure environments, highlighting the growing risks in agentic AI architectures.

At its core, exploitation involves crafting malicious payloads that trick the MCP server—running versions prior to 2.0.0-beta.17—into leaking its managed identity token. Attackers can then impersonate the server's identity to access sensitive Azure resources like storage accounts, virtual machines, or databases, all without needing admin rights or user interaction. Public proof-of-concept exploits, such as those on GitHub, amplify the threat, enabling rapid weaponization in targeted attacks against organizations leveraging MCP for AI workflows. This vulnerability underscores a classic SSRF pattern (CWE-918) but tailored to cloud-native AI tools, where broad service principals often grant excessive permissions.

Organizations should prioritize patching via Microsoft's Security Update Guide, audit MCP deployments for over-privileged identities, and implement outbound request filtering to contain risks. As AI security evolves, this incident signals the need for runtime protections in MCP-based systems, including token rotation and anomaly detection for AI agent traffic. Application security teams, especially those testing AI integrations, can use tools like Burp Suite to validate fixes against SSRF payloads. Staying vigilant ensures AI innovation doesn't outpace defense in the cloud.

Reference - https://www.tenable.com/cve/CVE-2026-26118