AI tools, scored for your job
Get paid to train AI
🤝 You're in the Free & Open Source library — community-built tools you can use without paying. A different concept from our scored AI directory.
Home › Free & Open Source › Cybersecurity › gpt3_security_vulnerability_scanner
✦ Free & open sourceCybersecurity

gpt3_security_vulnerability_scanner

Experimental benchmark dataset and evaluation evaluating GPT-3's performance in automated static code analysis.

Benchmark repository for LLM-based static code analysis

Evaluate how legacy GPT-3 models perform against traditional commercial static application security tools.

Free forever · source-available · Licence not stated in the repository
✓ Light setup✓ Browser-based✓ Real-time collaboration
5.6
ZEKAI
Pioneering LLM security benchmark
Free & open source · Cybersecurity
Ease of setup3
Usefulness6
Maintenance7
Community6.5
★ 604
GITHUB
Jun 2026
UPDATED
Licence not stated in the repository
LICENSE
🛠 Don't want to self-host? We can set it up →
⚡ Quick answer

This repository serves as an early benchmark evaluating GPT-3's capability to detect security flaws across multi-language code snippets. You can use it to compare raw LLM vulnerability discovery against traditional static analysis software.

Runs on
Cross-platform
Hosting
Desktop
Languages
PHP
License
Licence not stated in the repository

What it is

The project provides a curated collection of 129 vulnerable code files organized by flaw types like log injection, format string attacks, and buffer overflows. It includes documented evaluation outputs showing how text-davinci-003 detected 213 security vulnerabilities across the codebase.

Who it's for

Security analysts, AppSec researchers, and developers building AI-driven auditing tools will find this repository useful for benchmarking. It is not suited for security teams looking for an out-of-the-box scanner binary to drop into a CI/CD pipeline.

Key features

Categorized Flaw Testbed

Organizes 129 vulnerable code samples into directory structures divided by specific security weaknesses.

Commercial Tool Comparisons

Includes comparative findings evaluating LLM detection rates against established commercial static analysis tools.

Multi-Language Samples

Provides realistic target code files written in C, C#, Java, and common modern web framework snippets.

Raw Model Outputs

Contains detailed markdown records detailing the exact text outputs generated during the GPT-3 evaluation.

How to start

Up and running in 3 steps

1
Clone the repository locally to view the categorized vulnerability folders.
2
Review the source files alongside the accompanying markdown analysis files.
3
Feed the sample code snippets into your own OpenAI API scripts to test newer LLM models.
Real talk

Our honest verdict

What's great

  • High vulnerability recall compared to legacy static application security testing tools.
  • Covers diverse security flaw categories including memory safety and unvalidated user inputs.
  • Provides real-world code snippets across C, C#, Java, and popular framework libraries.
  • Documents manual sample verification detailing verified bugs alongside specific false positives.

Where it falls short

  • Does not include an automated CLI execution wrapper or production-ready pipeline integration.
  • Requires you to supply your own OpenAI API keys and fund your token usage to re-run tests.
  • Limited by single-file context windows that cannot trace cross-file logic vulnerabilities.
Cost

Pricing & license

The dataset and repository files are free to access. Running evaluation scans yourself requires paying for OpenAI API token consumption.

Paid work

Open remote AI-training roles

873 open remote AI-training roles, updated November 15, 2026; most pay $20–$170 an hour. Zekai lists them free; the application goes to the employer’s platform.

See all open roles →
Compare

Alternatives

🎁 Free & open-source alternatives

These reviewed AI tools do the same job instantly — from our scored directory.

Get more

Use it, or have us set it up

📄 Free PDF: gpt3_security_vulnerability_scanner for Cybersecurity
A plain-language guide to getting your first result. Sent to your inbox.
🔔 64 people want a full video tutorial — add your vote
🛠 Don't want to set it up yourself?
We'll install, host & configure gpt3_security_vulnerability_scanner for you — and train your team.
Get it done for you ↗
Questions

FAQ

Is this a turnkey security scanner tool?

No, this repository is a research benchmark dataset and set of documented test results rather than an automated command-line scanner.

How does it handle multi-file context?

It scans individual files separately, relying on the model's internal training memory to recognize standard library functions.

Radar

Latest AI news

✉️ The weekly free toolkit
The best new free & open-source tools for Cybersecurity — one short email a week, in your language.
Free · 1 email/week · unsubscribe anytime
Get paid to train AI →

See Zekai first in Google