Developing software with AI: 7 security risks

With developing software with AI is fast and productive – but AI assistants do not automatically write secure code. In Veracode’s benchmark (2026), AI-generated code passed security tests only in around 56% of cases on average. The biggest risks: unreviewed code, fabricated software packages, credentials in code, missing authorization checks, unfiltered inputs, confidential data in cloud assistants, and AI agents with too many permissions.

This no longer affects software companies alone. Today, business units, founders, and agencies also use AI to build their own tools, customer portals, or automations – often without a developer who understands the code. That is exactly where the gaps arise.

1. Code that nobody has reviewed

AI delivers code that runs, and that looks convincing. Whether it is secure is not revealed by the first impression. Veracode found in 2025 that models chose the insecure rather than the secure variant in 45% of test tasks; they were particularly weak at cross-site scripting and log injection. The French authority ANSSI and the BSI therefore explicitly recommend having AI-generated code reviewed by humans.

2. Invented software packages

AI models sometimes suggest libraries that do not exist at all. In a study presented at the USENIX Security 2025 conference, around 19.7% of the generated code examples contained at least one invented package name. Attackers register such names and fill them with malicious code (“slopsquatting”). Anyone who installs the suggestion without checking brings the malware in themselves.

3. Credentials in the code

API keys, database passwords or tokens quickly end up directly in the code — and with them in the repository or even in the users’ browser. Credentials belong in a secure configuration, never in the source code.

4. Missing authorization check

A typical pattern: the interface hides functions from normal users, but the backend does not check who is making the request. Anyone who knows the address can retrieve other customers’ data. AI only builds in such checks if you explicitly ask for them.

5. Unfiltered inputs

Form inputs, file uploads and URL parameters must be validated and processed securely. If this is missing, classic issues such as SQL injection or cross-site scripting can occur — exactly the errors where AI-generated code failed particularly often in the Veracode test.

6. Confidential data in the cloud assistant

Anyone who copies customer data, trade secrets or access credentials into a cloud assistant is passing them on to a third party. ANSSI and BSI advise against giving sensitive data to cloud-based coding assistants and recommend carrying out a risk analysis before use. What is permitted in a company context is best governed by an AI policy.

7. AI agents with too many permissions

Programming agents can execute commands, modify files, and install software. With overly broad permissions — such as access to production data or servers — an error or a manipulated instruction can cause major damage. More on this in Prompt Injection.

How to develop more securely with AI

Risk	Countermeasure
Unverified code	Human review before every production release
Invented packages	Check every new dependency: Does it exist, who maintains it?
Credentials in code	Secrets management, automatic checks in the repository
Missing permission checks	Check permissions server-side, tests with unauthorized accounts
Unfiltered inputs	State security requirements in the prompt, static analysis
Sensitive data in the cloud	Policy, approved tools, if applicable local models
Agents with too many permissions	Minimum permissions, no production data, approvals for commands

You develop software with AI support – yourself, with your team, or with a service provider? We take a look. A human review of code, dependencies, access rights and configuration before your application goes live or processes customer data. Contact us.

Frequently asked questions

Is AI-generated code insecure?

Not inherently, but often. In Veracode benchmarks, AI code passed security tests in only about 56% of cases on average. What matters is whether the code is reviewed before it goes into production.

What is slopsquatting?

Attackers register software packages under names that AI models often invent, and fill them with malicious code. Anyone who installs such a suggestion without checking brings the malware into the project.

May I enter company data into an AI coding assistant?

Confidential data, customer data, and access credentials should not end up in cloud assistants. ANSSI and BSI recommend a risk assessment before use. Clear rules are set out in an AI policy.

As of October 2026. Sources: Veracode GenAI Code Security Report 2025 and 2026, Spracklen et al. (USENIX Security 2025), ANSSI/BSI: AI Coding Assistants.