AI in Application Penetration Testing: It’s Time to Go with the Flow - But Not Blindly

Artificial Intelligence is no longer a "good to have" in cybersecurity—it’s rapidly becoming a force multiplier. From solving complex challenges in CTFs to automating reconnaissance, exploitation, and even report generation, AI-driven penetration testing is demonstrating measurable promise.

But enterprise security is not a playground. It’s a controlled, high-stakes environment where assumptions can translate into real risk. As organizations begin to evaluate AI as a replacement—or augmentation—for human-led penetration testing, it’s critical to pause and ask the right questions.

• Is AI Penetration Testing Production Safe? - AI tools operate at speed and scale. Without strict guardrails, this introduces a real risk – unintended exploitation of live applications, service disruptions etc. Unlike human testers, AI does not inherently understand "safe boundaries" unless explicitly constrained.
• Is AI Only as Good as Its Prompter? – A prompt-orchestrated testing raises a fundamental dependency - the quality of findings is directly tied to the operator’s expertise. In effect, we may not be replacing human intelligence—we’re reshaping it.
• Can AI Replicate True Human Intelligence? - Some of the most critical vulnerabilities are not pattern-based—they are contextual (business logic flaws, privilege escalation chains etc.). These require situational awareness and integrated data flow understanding.
• Enterprise Reality: Integrated Application Ecosystems – In large enterprises, applications are interconnected - data flows across APIs, services, and third-party platforms. Security issues often emerge between systems—not within them. AI tools, unless specifically architected for this, may miss this integration.
• The False Positive Problem - AI can generate large volumes of findings quickly but is there still a need for manual triage? Are we shifting effort from "finding vulnerabilities" to "filtering noise"? Without a robust validation layer, organizations risk drowning in output with limited actionable intelligence.
• Data Privacy and Model Risk – AI thrives on data. By using AI penetration testers, are we risking data leakage and could this data be used to train third-party models? For many enterprises, this alone could be a blocker.
• Where Does the AI Pen Tester Sit in Your Network? - Deploying AI testing introduces architectural questions – does it require internet exposure? Is it deployed with full network access? What controls prevent lateral misuse if compromised?

The Way Forward: A Controlled, Measurable Approach

We are clearly at a turning point. AI in penetration testing is not a question of if—it’s a question of how and when. But premature adoption without structured evaluation can weaken, rather than strengthen, security posture. Organizations should resist binary thinking (AI vs Human) and instead focus on comparative validation:

• Conduct PoCs on real enterprise applications
• Benchmark AI-driven vs human-led testing
• Evaluate across:

• Depth of findings
• False positive rates
• Coverage of business logic vulnerabilities
• Time-to-deliver and cost efficiency 

AI is accelerating. Agent creation is becoming effortless. Automation is redefining scale. But security has never been about speed alone—it’s about precision, context, and judgment. We need to engineer the right balance between human intelligence and machine capability. 

We certainly should use AI in application security - it brings scale, speed, and the ability to uncover patterns that would otherwise take significant manual effort but the need of human intelligence cannot be completely written off. AI can accelerate discovery. Humans ensure relevance, accuracy, and real-world impact. Together, they create a security model that is not only efficient, but also resilient and trustworthy. Organizations that recognize this balance early will not just keep up with the shift—they will define it.

Article by Hemil Shah and Rishita Sarabhai 

Understanding Where AI/LLM Vulnerabilities Originate — and How to Fix Them?

Most discussions around AI/LLM security focus on what the vulnerabilities are i.e. prompt injection, data leakage, bias, abuse, or excessive agency including our past blogs. However, during real world engagements, the most important question which needs to be addressed is 

Which part of the AI system is actually introducing these vulnerabilities, and where should those vulnerabilities needs to be fixed?

In non-agentic implementations, the most common vulnerability is prompt injection which results in either unintended data access or data exfiltration or bias, abuse, or content manipulation.  

In agentic implementations, the most common vulnerability is LLM excessive agency, where the system performs actions beyond its intended scope.

These vulnerabilities are rarely caused by the LLM alone. They originate from how the overall system is architected and integrated. In practice, AI systems are not a single component. They are composed of multiple layers, each with a specific responsibility. Vulnerabilities appear when responsibilities of each of these components are blurred or controls are missing. It is important to have defence in depth approach and implement security at all components.

Let’s take an example of a typical AI/LLM architecture and discuss the layers and protections that needs to be implemented at each component: -

The sequence of layers introduces the most impactful vulnerabilities and thus these underlying vulnerabilities at the core should be fixed on top priority. Below are the nature of vulnerabilities arising due to weaknesses at various layers and the type of fixes that can be applied: -

API Layer – Authorization and Access Control
A frequent root cause of unintended data access or excessive agency is due to LLM responses triggering APIs without validating user permissions and/or over-privileged service accounts used by AI workflows.
Fix:

• Enforce strict authorization checks on every API call
• Validate user context before returning data
• Do not rely on the LLM to decide access control
• Restrict tool access based on task and role
• Keep an approval-workflow for high-impact tools

The API layer will always be the final gatekeeper and thus this protection helps built secure AI systems. 

Code Layer – Prompt Encoding and Sanitization
Data exfiltration via prompts often succeeds because user inputs or model outputs are passed directly into HTML pages, logs, downstream systems or follow-up prompts as raw data without any encoding. 
Fix:

• Encode and sanitize all user inputs
• Encode and sanitize all LLM outputs
• Treat LLM output as untrusted data, similar to user input

This is a traditional application security control that still applies in all AI systems that are returning user injected data as output to the users.

LLM / Reasoning Layer – Using Provider Guardrails
Most cloud AI services provide security features (like Azure AI Content Safety, Checks Guardrails APIs (Google)), but they are often disabled or misconfigured.
Fix:

• Enable content filtering and safety controls
• Tune policies based on use case
• Do not treat default settings as sufficient

Provider guardrails are a baseline, not a complete solution. We will write a separate detailed blog entry in coming days on various configuration available for some of the LLM providers.

Application Layer – Custom Input and Output Controls
Relying only on LLM provider guardrails is insufficient. These guardrails can be bypassed by various prompt injection techniques and thus additional input/output validations are required to decrease the impact of prompt injection findings.
Introduce application-level validations such as: -

• Block special characters
• Input length restrictions
• Allowed language checks
• Encoding and decoding validation
• Custom blacklists or allow lists for specific keywords or characters
• Output validation before execution or display 

Prompt & LLM Integration Layer – Clear Instructions and Boundaries
Weak or ambiguous system prompts increase the likelihood of prompt injection and excessive agency. System prompts/instructions can be enhanced to clearly define what the model can do/not do, restrict response formats where possible, reinforce boundaries consistently across prompts. System prompts act as policy documents for the model and need continuous enhancements as and when bypasses are discovered – though they are not enough to block any attacks on AI systems. 

Conclusion
Just as with traditional applications, building secure AI systems is not really an afterthought, security has to be designed into the architecture from day one. As described above, AI security issues rarely originate from the LLM alone and are usually the result of missing controls across multiple layers. Thus, effective AI security requires: -

• Understanding the architecture
• Mapping risks to the correct layer
• Applying traditional security principles alongside AI-specific controls

The above details might help organizations move beyond identifying vulnerabilities and build and architect secure AI systems in a structured and sustainable way. In the coming posts, we will share practical configurations and patterns that can be applied across these layers to help teams design and deploy AI implementations with security in mind.

Article by Hemil Shah and Rishita Sarabhai

Defending AI applications - Security Controls

AI Security Controls
When implementing AI systems, covering both non-agentic and agentic use cases, there are certain security controls that should be considered for building secure systems. There are grouped into foundational areas such as data handling, model integrity, access control, context safety, action governance, and operational monitoring. These controls are drawn from established industry frameworks, including NIST AI RMF, OWASP LLM Top 10, SANS AI Security Guidelines, and emerging agentic protocols. They are intended to help development and platform teams integrate security into the design, build, and deployment of AI applications. These controls can be provided as a practical checklist with implementation guidance so they can be adopted consistently during development, validated during testing, and formally handed over to operations for ongoing governance. 

Deployment Strategies
Model and Dependency Security Scanning

• Scan all models before deployment.
• Ensure no malicious code, backdoors, supply-chain implants, biased data, PII or unsafe dependencies exist in the model artifacts.
• Regularly perform open-source model license compliance checks.
• Continuous model scanning after any retraining, fine-tuning, or indexing.

RAG, Embedding, and Vector Database Controls

• Validate and sanitize all documents before adding them for indexing.
• Prevent indexing of sensitive or classified documents (which can lead to breach of contracts, copyright issues or anything of that nature).
• Use embedding models from trusted, verified sources only.
• Enforce metadata-level access control to ensure retrieval respects user permissions.
• Periodically purge documents from the vector DB.

Data Protection & Inference Security 
Model Input Validation and Prompt Handling

• Validate all user inputs/prompts before passing them to the model. Reject or sanitize unexpected formats, extremely long inputs, and potentially harmful content.
• Prevent resource exhaustion be rate-limiting implementation.
• Do not allow raw user input/prompts to directly control system or developer prompts. Use controlled templates and parameterized instructions.
• Remove or neutralize prefixes, control tokens, or patterns that could trigger jailbreaks or prompt injection.
• Restrict model-access to sensitive business operations; ensure the model cannot perform unintended administrative actions via crafted prompts.

Model Output Controls and Response Filtering

• All model outputs should be validated and processed before rendering them to end users or any other downstream systems.
• Content filtering should be implemented for harmful content categories (e.g., violence, hate, self-harm, disallowed medical/legal advice, malware, phishing etc.) in the output.
• Ensure the system never executes model outputs as code, commands, or queries.
• Strict type-validation should be enforced on model-generated structured data such as JSON, SQL, or code.
• Variety of guardrails should be implemented to prevent hallucinated URLs, contacts, API calls, or misrepresentations.

Sensitive Data Controls

• Do not expose sensitive, proprietary, copyright or personal data to external users or third-party model APIs.
• Apply masking, tokenization, or pseudonymization before sending data to the model.
• Restrict the model from storing or self-learning based on user-provided sensitive data.
• Inspect prompts and output logs to ensure no leakage of secrets, credentials, API keys, or internal endpoints.
• Disable ingestion, training or fine-tuning capabilities without a pre-approved process flow.
• Prompt Injection, Jailbreak, and Abuse Prevention
• Implementation of layered guardrails - system prompts, content filters, and structured templates.
• Strict use of isolation prompts to prevent the model from modifying or revealing its system instructions.
• Prevent the model from operating on untrusted references such as user-uploaded documents without sanitization.
• Ensure the application ignores user attempts to override model identity, policies, or instructions.
• Continuous evaluation of jailbreaks using adversarial test suites and automated red-teaming pipelines.

Bias, Toxicity, and Hallucination Mitigation

• Evaluate model outputs for bias across protected categories; log and mitigate recurring patterns.
• Policy-based constraints that block discriminatory, abusive, or identity-targeted outputs.
• More focused data (RAG, retrieval checks) to reduce hallucinations.
• Ensure that the system provides citations or source references for high-risk outputs.
• Include human-in-the-loop for critical or decision-impacting outputs.

Model Behaviour Integrity and Drift Monitoring

• Track versioning for prompts, model weights, system templates, and embedding indices.
• Monitor outputs for accuracy degradation, bias drift, or changes in harmful content Behaviour.
• Ensure rollback mechanisms exist to revert to a previous safe model version.
• Implement real-time anomaly detection on output patterns using approved monitoring tools.
• Perform periodic re-evaluation of the model’s performance on business-critical tasks. 

Access Controls

Access Control and Authentication for AI Endpoints

• All model inference endpoints must require authentication; public inference endpoints are prohibited unless approved.
• Apply role-based access controls for administrative operations such as fine-tuning, dataset upload, vector store rebuild, or configuration updates.
• Apply record-level, data-level access controls for protecting against unintended data access.
• Enforce strong API key management standards; keys must not appear in client-side code.
• Restrict high-cost or high-impact model operations such as batch inference to privileged roles.
• Log access patterns, anomalies, and repeated misuse attempts.

Environment, Infrastructure, and API Security

• Isolate model workloads in dedicated inference environments; prevent lateral movement from model containers.
• Disable shell access and system command execution from within model pipelines.
• Encrypt all communications: model API calls, embedding store interactions, and dataset transfers.
• Apply strict resource quotas to prevent model abuse, cost spikes, or denial-of-service scenarios.
• Redact logs to avoid storing sensitive prompts and outputs in plaintext.

Monitoring & Governance, Risk, Compliance (GRC)

Model Explainability and Governance

• Maintain clear documentation for training datasets, model versions, decision boundaries, and fine-tuning sources.
• Provide audit logs for all inference requests/prompts tied to a user identity.
• Ensure regulatory alignment with data protection, algorithmic accountability, and sector-specific AI guidelines.
• Document known limitations or unsafe failure modes for the model.
• Integrate with internal AI governance workflows for approvals, reviews, and continuous compliance.

Human Feedback, Reinforcement, and Fine-tuning Controls

• Train or tune models only on authorized, clean, governance-approved datasets.
• Prevent user-generated adversarial prompts from polluting RLHF or fine-tuning datasets.
• Conduct manual review on all human-labelled datasets used for alignment.
• Track dataset lineage, consent, provenance, and ownership.
• Evaluate fine-tuned models again for safety, bias, and hallucination risk.

Additional Controls (Focused on Agentic AI) (Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent–User Interaction Protocol (AG-UI))
• Authentication & Identity Assurance

• Enforce strong authentication for all protocol endpoints, agents, tools, services, and UI sessions. 
• Each request or message must be tied to a verifiable identity. 
• No anonymous or implicitly trusted protocol participants.

• Authorization and Capability Restrictions

• Restrict the context provided to the agent/tool to the minimum required as per defined task. 
• Least privilege principle should be followed with proper permission checks against a strict capability list. 
• Require explicit policy checks before executing any agent-initiated action.

• Message Integrity & Transport Security

• Use TLS for all communication channels and enforce message signing or integrity hashes. 
• Reject tampered, malformed, or unsigned MCP/ACP/AG-UI messages.
• Reject duplicated messages, stale actions, or replayed session traffic.

• Context and State Isolation

• Segment context and shared state by user, agent, or task. 
• Prevent leakage across sessions, tools, or agents. Enforce TTL and expiry on all context and state artifacts.

• Tool and Action Execution Controls

• Use a sandbox or isolated environment for all external calls or tool executions.
• Prevent model outputs from directly executing commands without validation.
• Require human approval for high-risk actions defined by policy.

• Tool Discovery and Registration Security

• Restrict which tools can be discovered or registered. 
• Require authentication and integrity checks for tool manifests. 
• Prevent unapproved or malicious tool descriptors from being exposed to agents.
• Validate that agents cannot enumerate or access tools outside their assigned capability scope.

• Information and Session Leakage Prevention

• Ensure error responses do not disclose internal details, system state, or sensitive context.
• Use short-lived, scoped tokens for all protocol operations. 
• Store credentials securely and prevent leakage via logs or metadata. 
• Rotate tokens and invalidate them on session termination.

• Monitoring and Logging

• Log all agent requests, decisions, actions, and results in an immutable audit trail.
• Continuously monitor the agent for anomalous behaviour or unsafe action patterns.
• Maintain rollback mechanisms for any state-changing actions performed by the agent.
• Implement a kill-switch to immediately disable agent actions if unsafe conditions occur.

Article by Hemil Shah and Rishita Sarabhai

Deploying COTS Products In-House: Balancing Innovation with Security

In an era where AI based Commercial Off-The-Shelf (COTS) products are swarming the markets and organizations across industries are turning to such products to meet business needs quickly and efficiently, it is very key to pause and think about the risks associated to such implementations, some of these risk are inherited from classic problem of COTS and some are newly introduced with AI. There are quite a few risks, the major ones being data leakage and loss of intellectual property due to external hosting – to overcome this risk, many organizations consider in-house deployment (either on-premise or in private cloud). However, one needs to think of following responsibilities and risks arising out of those responsibilities before going that route: – 

• Responsibility of giving 100% up time (availability) 
• Responsibility of updating software (giving access to vendor to perform routine task) 
• Responsibility of patching underlying infrastructure (what if patch breaks the application or it’s supporting server)
• Responsibility of backing up data

Key Security Risks of In-House COTS AI Deployment

In one of our recent engagements, we evaluated the security of a SaaS AI platform for investment bank. The client decided to use a dedicated vendor platform by deploying it in their own private cloud. The SaaS product was leveraging AWS cloud however on a special request of a client, SaaS provider agreed to use Azure cloud which actually opened more challenge as SaaS provider had not in-depth expertise of Azure clod. The application leveraged a robust technology stack, including Next.js for client-side rendering and Node.js for backend processing. It leveraged components like Azure Key Vault, Blob Storage, PostgreSQL, Kubernetes, and OpenAI. At a high level, following was architecture -  

The investment bank hired Blueinfy to evaluate the security risks of the COTS AI product deployed in their environment before deploying it for internal use. The key focus of the review was : - 

• Unauthenticated Access to Client Data
• Unauthorized Access to Client Data 
• SaaS provider getting Access to Client Data

In order to assess the above, Blueinfy took an in-depth approach which combined a network layer assessment, a design review, penetration testing of the application including AI specific testing. Moreover, risk assessment in terms of business impact was the core focus of the assignment. This comprehensive review led to a list of observations which are also the most common and critical risks that organizations must consider: -  

Insecure Default Configuration

In order to provide ease of deployment, many COTS products come with default settings such as: -

• Open management interfaces
• Debugging enabled
• Hardcoded or default credentials
• Excessive file or database permissions

These default settings may provide easy access points to internal or external attackers if they are not examined and strengthened prior to go-live. We came across hidden URLs in responses which led to unintended back-end panel access on a specific port.

Inherited Vulnerabilities from the Vendor

The vendor controls the COTS software development lifecycle. If the vendor uses outdated third-party libraries, insecure configurations, or lacks a secure SDLC (Software Development Life Cycle), those flaws come bundled with the product. Such vulnerabilities will stay concealed and exploitable after deployment if they are not independently confirmed. There were multiple instances of use of components with published security vulnerabilities.

Inadequate Authentication/Authorization

Although COTS products generally come with built-in access control features, the security model of the company may not be compatible with them – for example Single Sign On (SSO). RBAC implementations that are not reviewed thoroughly can lead to: -

• Privilege escalation/LLM excessive agency
• Unauthorized access to sensitive data or functions
• Lack of separation between administrative and regular user functions

The impact of compromised accounts and the risks associated with insider threats are increased by inadequate access segregation. In our assessment of this implementation, this vulnerability is the most impactful in terms of risk to business as it violates the principle of least access.

Overlooked Data Flows and Outbound Communications

For tasks like license verification, model updates, product upgrades, telemetry, analytics, etc., COTS tools may initiate outbound connections by default. Firewalls may need to be opened for these activities, and if external calls are not monitored or controlled, they may unintentionally leak private information or violate regulations, particularly in regulated sectors. Furthermore, by default, data handling features like file export, email integration, or third-party API hooks may be activated, leaving room for data loss or abuse. We have come across scenarios where external service interaction is allowed to all domains instead of just white-listing the license server. This needs to be blocked at firewall level. 

Lack of Content Filtering & AI Guardrails

This can compromise the AI system, exposing proprietary prompts and enabling malicious inputs, which could lead to system manipulation or data misuse. Unfiltered content exposes systems to harmful, inappropriate, or irrelevant inputs, leading to mass phishing attacks when conversations are shared between users of the application. Due to such lack of guardrails, system prompt was leaked and direct & indirect prompt injection lead to data exfiltration. 
Incomplete or Inaccessible Security Documentation
Many vendors provide only high-level or marketing-friendly security collateral which does not have - 

• Detailed architecture diagrams
• Clear descriptions of data flows and storage
• Results from recent third-party security tests (DAST, SAST, penetration tests)

...you are left to evaluate on your own, making it even difficult to identify or prioritize risks accurately.

Conclusion

Bringing a COTS, be it AI product or a traditional product, into our own environment doesn’t mean the product now "inherits" the security posture of the company. Instead, it inherits all of the vendor’s decisions, good and bad, and must overlay controls to compensate. A secure in-house deployment of COTS software (AI based or traditional COTS) requires a deliberate and thorough review of configurations, privileges, dependencies, and operational behaviour. Every deployment should be scoped with advice on architecture, network, application & AI layer assessments to review which of these would suffice from a security standpoint. Skipping these steps can quickly turn a business enabler into a security liability. Thus before deployment, it is necessary to ask the hard questions and review independently. 

Article by Hemil Shah