Generative AI Security

Are you planning or developing GenAI-powered solutions, or already deploying these integrations or custom solutions?

We can help you identify and address potential cyber risks every step of the way.


Embrace AI while Mitigating Security Risks

Artificial Intelligence is rapidly transforming industries, and organizations are increasingly integrating large language models (LLMs) into their services and products. Whether using off-the-shelf models, customizing pre-trained solutions, or developing proprietary AI, the transformative power of these technologies is undeniable.

While AI should be recognized and embraced as a game-changer for business innovation, it's essential to be aware of the potential cyber security risks beyond the hype.

We see the majority of cyber security risks stemming from how AI models are integrated into systems and workflows rather than from the models themselves.

Failing to address these risks can expose your organization to various threats, including data breaches, unauthorized access, and compliance violations. 

We can help you address the practical risks associated with integrating AI into enterprise systems and workflows. As a leading cyber security and pentesting company, we have extensive experience in helping organizations navigate the complexities of adopting new technologies such as GenAI and LLMs.

Common Pitfalls in the Use of GenAI

Practical risks associated with AI don't exist in isolation but are mostly related to the context in which the organization is using it. When building GenAI and LLM integrations, it's crucial to consider the potential security risks and implement robust safeguards from the outset.

We have identified that the most common pitfalls associated with the use of AI for businesses, especially from a security perspective, include:

Jailbreaking Prompt Injection Attacks

Malicious actors attempt to "jailbreak" the AI system by injecting carefully crafted prompts, tricking it into executing unauthorized actions or revealing sensitive information.

Excessive Agency and Malicious Intent

Coupled with jailbreaking and prompt injection, AI systems with excessive agency get manipulated by attackers causing the system to execute malicious actions, and posing significant security risks.

Insecure Tool/Plugin Design

Poorly designed or insecurely implemented AI tools, plugins, or integrations introduce vulnerabilities, enabling unauthorized access or data breaches.

Insufficient Monitoring, Logging, and Rate Limiting

Inadequate monitoring, logging, and rate-limiting mechanisms hinder the detection of malicious activity, making it challenging to identify and respond to security incidents promptly.

Lack of Output Validation

Failure to validate and sanitize the output from AI models leads to the disclosure of confidential information or the introduction of client-side vulnerabilities like Cross-Site Scripting (XSS).

Ensuring the Security of Your LLMs and GenAI Solutions

Whether your organization is in the early stages of planning or developing GenAI-powered solutions, or already deploying these integrations or custom solutions, our consultants can help you identify and address potential cyber risks every step of the way.

We can support your organization in adopting and integrating AI securely by assessing the potential security flaws of the GenAI/ LLM integrations and interaction to your systems and workflows, and providing recommendations on secure deployment.

Depending on your use case, the different assessment approaches may include any of the below. Contact us to discuss the best approach for your specific case.


Governance, Risk and Threat Modelling for AI

Our services to support you in the planning phase.

AI Governance

  • Defining the AI adoption objectives and acceptable use cases
  • Adapting or creating ad-hoc risk management frameworks based on your organization’s needs and regulatory requirements


AI Risk Modelling

  • Identifying and prioritizing security risks at an organizational and use case level
  • Creating a shared risk understanding between development teams, cyber security, and business units

AI Threat Modelling

  • Identifying the most relevant attack paths based on risk prioritization technical analysis
  • Identifying control gaps and prioritizing control implementations through cost/ benefit analysis

Implementation and Integration of AI solutions

Our services to support you in the implementation phase.

Pentesting for LLM Application

  • Identifying and addressing the cyber security weaknesses in your organization’s LLM applications and integrations
  • Understanding the exploit vulnerabilities of the LLM applications, the specific cyber risks they pose, and the attacker goals that will most likely lead to them being targeted


Pentesting for AI-Supporting Infrastructure

  • Identifying high risk attack paths leading to your AI-powered applications and offering recommendations to protect these
  • Ensuring secure hosting and AI-management, protecting AI data and access points

We Can Help

WithSecure™ is the trusted cyber security partner and industry-accredited, global provider of cyber security assurance services, with over 30 years of experience.

We understand the unique challenges that arise during the development and implementation of AI-powered solutions. That's why we offer comprehensive cyber security consulting services to support you every step of the way.

Our experienced and specialized team can help your organization leverage the full potential of AI technology while maintaining a resilient and secure infrastructure.

Contact us to find out how we can support your organization in the secure deployment of GenAI and LLMs.


Want to talk in more detail?

Complete the form, and we'll be in touch as soon as possible.

Case Studies

Securing an LLM-Powered Customer Support Agent

Securing an LLM-Powered Customer Support Agent for a Tech Start-up

Client's Challenge

A tech start-up was developing an LLM-powered virtual agent to automate customer support experience for organisations. The agent would have access to customer accounts and the ability to perform operations like updating address details. The client's primary concern was ensuring the security and privacy of customer data while maintaining the agent's functionality and effectiveness.



By providing expert guidance and recommendations, we helped the tech start-up implement robust security measures for their LLM-powered customer support agent. The redesigned API and guardrail pipelines, implemented by the client based on our advice, ensured strong access controls and protection against malicious prompt injection attempts.

Customer data remained secure, and the agent could function effectively without compromising privacy or exposing the client to potential data breaches or unauthorized access.

Our Solution

Our team conducted a comprehensive security assessment of the client's LLM integration, including a thorough evaluation of the agent's tools and APIs. We identified a critical vulnerability wherein the API allowed the LLM to specify the userID, opening the door for prompt injection or jailbreaking attacks. Malicious actors could potentially force the agent to invoke the API with a different userID, enabling unauthorized access and modification of confidential information across customer accounts.

To mitigate this risk, we advised the client on redesigning the API's access controls. We recommended removing the userID parameter from the API and supplementing it as part of a secure session management system. Additionally, we guided the client in integrating LLM guardrail pipelines that inspect untrusted input and limit the success rate of jailbreak or prompt injection attacks.

AI Security Strategy and Risk/Threat Modeling

AI Security Strategy and Risk/Threat Modeling for a Large Enterprise

Client's Challenge

A large multinational corporation aimed to enhance their workforce's capabilities by adopting GenAI solutions, both off-the-shelf productivity tools and integration of proprietary LLMs via access to external, 3rd party APIs. They sought guidance on evaluating the security implications of these GenAI implementations and establishing best practices for future GenAI projects. Given the dynamic nature of AI solutions, the client recognized the need for a tailored AI security strategy that complements traditional cybersecurity methods.



By engaging in this comprehensive AI security strategy, the client gained a deep understanding of the potential risks and threats associated with their AI implementations. Equipped with our risk and threat modeling insights, they could make informed decisions and implement appropriate security controls to mitigate these risks effectively.

Moreover, the AI security checklist provided a robust framework for future AI projects, ensuring a consistent and proactive approach to addressing security concerns from the outset. This empowered the client to leverage the transformative power of AI while maintaining the highest levels of security and protecting their organization's critical assets and data.

Our Solution

We engaged with the client through a multi-phased approach, combining risk modeling and threat modeling in the context of AI to create a holistic view of the risks associated with leveraging API-based AI solutions for the organization.

Phase 1: Risk Modeling Our team conducted a comprehensive risk assessment, identifying potential vulnerabilities and threats specific to the organization's AI implementation. We evaluated factors such as data privacy, model biases, transparency, and integration with existing systems.

Phase 2: Threat Modeling Building upon the risk assessment, we performed threat modeling exercises to understand the potential attack vectors and scenarios that malicious actors could exploit within the AI ecosystem. This included analyzing risks related to prompt injection, model hijacking, and insecure API integrations.

Phase 3: AI Security Checklist Based on our findings from the risk and threat modeling phases, we developed a tailored AI security checklist to guide the client in implementing robust security measures for future API-based AI projects. This checklist encompassed best practices for secure data handling, model validation, API access controls, monitoring, and incident response.

Further Resources


When your AI Assistant has an evil twin

This blog explores how attackers can use prompt injection to coerce Gemini into performing a social engineering attack against its users.

Read more

Webinar: Building Secure LLM Apps into Your Business

Gain practical understanding of the vulnerabilities of LLM agents and learn about essential tools and techniques to secure your LLM-based apps.

Watch on demand

Creatively malicious prompt engineering

The experiments demonstrated in our research proved that large language models can be used to craft email threads suitable for spear phishing attacks, "text deepfake” a person’s writing style, apply opinion to written content, write in a certain style, and craft convincing looking fake articles, even if relevant information wasn’t included in the model’s training data. 

Read more

Domain-specific prompt injection detection

This article focuses on the detection of potential adversarial prompts by leveraging machine learning models trained to identify signs of injection attempts. We detail our approach to constructing a domain-specific dataset and fine-tuning DistilBERT for this purpose. This technical exploration focuses on integrating this classifier within a sample LLM application, covering its effectiveness in realistic scenarios.

Read more

Should you let ChatGPT control your browser?

In this article, we expand our previous analysis, with a focus on autonomous browser agents - web browser extensions that allow LLMs a degree of control over the browser itself, such as acting on behalf of users to fetch information, fill forms, and execute web-based tasks.

Read more

Case study: Synthetic recollections

This blog post presents plausible scenarios where prompt injection techniques might be used to transform a ReACT-style LLM agent into a “Confused Deputy”. This involves two sub-categories of attacks. These attacks not only compromise the integrity of the agent's operations but can also lead to unintended outcomes that could benefit the attacker or harm legitimate users.

Read more


28 May 2024

10:45 - 11:10 EEST

Kaapelitehdas, Helsinki, Finland

Should you trust ChatGPT with your browser – or anything else you care about?

Donato Capitella, Principal Security Consultant at WithSecure Consulting, SPHERE24 Co-security Unconference


11 Apr 2024

60 min


Building Secure LLM Apps into Your Business

Gain practical understanding of the vulnerabilities of LLM agents and learn about essential tools and techniques to secure your LLM-based apps. Our host Janne Kauhanen is joined by Donato Capitella, Principal Security Consultant at WithSecure™.

The agenda for this 60-minute discussion includes:

  • Background and definitions of LLMs and LLM applications
  • Vulnerabilities of LLM applications (includes a demo)
  • Vulnerabilities of LLM agents (includes a demo)
  • Tools and techniques to secure LLM applications
  • Q&A session


  • Donato Capitella, Principal Security Consultant, WithSecure™ 
  • Janne Kauhanen, Cyber Host, WithSecure™   


27 Mar 2024

8:00 - 10:30 BST

The Langham Hotel, London, UK

AI and LLMs from a Cyber Security Perspective

TEISS Breakfast Briefing for CISOs


21 Sep 2023

60 min


AI: why now and what does it mean for your security?

Get set for a webinar that will redefine your approach to AI. Joining host Janne Kauhanen in the studio will be renowned cyber security experts Tomi Tuominen, VP of Security, Wolt, and Mikko Hyppönen, Chief Research Officer at WithSecure™.

The agenda for this 60-minute panel discussion includes:

  • Why we are experiencing an AI explosion right now
  • What vulnerabilities companies need to be aware of
  • The pros and cons of using AI in everyday company life
  • How we can protect ourselves against AI-based attacks – both now and in the future
  • Q&A session


  • Tomi Tuominen, VP of Security, Wolt
  • Mikko Hyppönen, Chief Research Officer, WithSecure™   
  • Janne Kauhanen, Cyber Host, WithSecure™   

Our accreditations and certificates