In an era where AI based Commercial Off-The-Shelf (COTS) products are swarming the markets and organizations across industries are turning to such products to meet business needs quickly and efficiently, it is very key to pause and think about the risks associated to such implementations, some of these risk are inherited from classic problem of COTS and some are newly introduced with AI. There are quite a few risks, the major ones being data leakage and loss of intellectual property due to external hosting – to overcome this risk, many organizations consider in-house deployment (either on-premise or in private cloud). However, one needs to think of following responsibilities and risks arising out of those responsibilities before going that route: –
- Responsibility of giving 100% up time (availability)
- Responsibility of updating software (giving access to vendor to perform routine task)
- Responsibility of patching underlying infrastructure (what if patch breaks the application or it’s supporting server)
- Responsibility of backing up data
Key Security Risks of In-House COTS AI Deployment
In one of our recent engagements, we evaluated the security of a SaaS AI platform for investment bank. The client decided to use a dedicated vendor platform by deploying it in their own private cloud. The SaaS product was leveraging AWS cloud however on a special request of a client, SaaS provider agreed to use Azure cloud which actually opened more challenge as SaaS provider had not in-depth expertise of Azure clod. The application leveraged a robust technology stack, including Next.js for client-side rendering and Node.js for backend processing. It leveraged components like Azure Key Vault, Blob Storage, PostgreSQL, Kubernetes, and OpenAI. At a high level, following was architecture -
- Unauthenticated Access to Client Data
- Unauthorized Access to Client Data
- SaaS provider getting Access to Client Data
In order to assess the above, Blueinfy took an in-depth approach which combined a network layer assessment, a design review, penetration testing of the application including AI specific testing. Moreover, risk assessment in terms of business impact was the core focus of the assignment. This comprehensive review led to a list of observations which are also the most common and critical risks that organizations must consider: -
Insecure Default Configuration
In order to provide ease of deployment, many COTS products come with default settings such as: -
- Open management interfaces
- Debugging enabled
- Hardcoded or default credentials
- Excessive file or database permissions
These default settings may provide easy access points to internal or external attackers if they are not examined and strengthened prior to go-live. We came across hidden URLs in responses which led to unintended back-end panel access on a specific port.
Inherited Vulnerabilities from the Vendor
The vendor controls the COTS software development lifecycle. If the vendor uses outdated third-party libraries, insecure configurations, or lacks a secure SDLC (Software Development Life Cycle), those flaws come bundled with the product. Such vulnerabilities will stay concealed and exploitable after deployment if they are not independently confirmed. There were multiple instances of use of components with published security vulnerabilities.
Inadequate Authentication/Authorization
Although COTS products generally come with built-in access control features, the security model of the company may not be compatible with them – for example Single Sign On (SSO). RBAC implementations that are not reviewed thoroughly can lead to: -
- Privilege escalation/LLM excessive agency
- Unauthorized access to sensitive data or functions
- Lack of separation between administrative and regular user functions
The impact of compromised accounts and the risks associated with insider threats are increased by inadequate access segregation. In our assessment of this implementation, this vulnerability is the most impactful in terms of risk to business as it violates the principle of least access.
Overlooked Data Flows and Outbound Communications
For tasks like license verification, model updates, product upgrades, telemetry, analytics, etc., COTS tools may initiate outbound connections by default. Firewalls may need to be opened for these activities, and if external calls are not monitored or controlled, they may unintentionally leak private information or violate regulations, particularly in regulated sectors. Furthermore, by default, data handling features like file export, email integration, or third-party API hooks may be activated, leaving room for data loss or abuse. We have come across scenarios where external service interaction is allowed to all domains instead of just white-listing the license server. This needs to be blocked at firewall level.
Lack of Content Filtering & AI Guardrails
This can compromise the AI system, exposing proprietary prompts and enabling malicious inputs, which could lead to system manipulation or data misuse. Unfiltered content exposes systems to harmful, inappropriate, or irrelevant inputs, leading to mass phishing attacks when conversations are shared between users of the application. Due to such lack of guardrails, system prompt was leaked and direct & indirect prompt injection lead to data exfiltration.
Incomplete or Inaccessible Security Documentation
Many vendors provide only high-level or marketing-friendly security collateral which does not have -
- Detailed architecture diagrams
- Clear descriptions of data flows and storage
- Results from recent third-party security tests (DAST, SAST, penetration tests)
...you are left to evaluate on your own, making it even difficult to identify or prioritize risks accurately.
Conclusion
Bringing a COTS, be it AI product or a traditional product, into our own environment doesn’t mean the product now "inherits" the security posture of the company. Instead, it inherits all of the vendor’s decisions, good and bad, and must overlay controls to compensate. A secure in-house deployment of COTS software (AI based or traditional COTS) requires a deliberate and thorough review of configurations, privileges, dependencies, and operational behaviour. Every deployment should be scoped with advice on architecture, network, application & AI layer assessments to review which of these would suffice from a security standpoint. Skipping these steps can quickly turn a business enabler into a security liability. Thus before deployment, it is necessary to ask the hard questions and review independently.
Article by Hemil Shah