Experimental benchmark dataset and evaluation evaluating GPT-3's performance in automated static code analysis.
Evaluate how legacy GPT-3 models perform against traditional commercial static application security tools.
This repository serves as an early benchmark evaluating GPT-3's capability to detect security flaws across multi-language code snippets. You can use it to compare raw LLM vulnerability discovery against traditional static analysis software.
The project provides a curated collection of 129 vulnerable code files organized by flaw types like log injection, format string attacks, and buffer overflows. It includes documented evaluation outputs showing how text-davinci-003 detected 213 security vulnerabilities across the codebase.
Security analysts, AppSec researchers, and developers building AI-driven auditing tools will find this repository useful for benchmarking. It is not suited for security teams looking for an out-of-the-box scanner binary to drop into a CI/CD pipeline.
Organizes 129 vulnerable code samples into directory structures divided by specific security weaknesses.
Includes comparative findings evaluating LLM detection rates against established commercial static analysis tools.
Provides realistic target code files written in C, C#, Java, and common modern web framework snippets.
Contains detailed markdown records detailing the exact text outputs generated during the GPT-3 evaluation.
The dataset and repository files are free to access. Running evaluation scans yourself requires paying for OpenAI API token consumption.
873 open remote AI-training roles, updated November 15, 2026; most pay $20–$170 an hour. Zekai lists them free; the application goes to the employer’s platform.
These reviewed AI tools do the same job instantly — from our scored directory.
No, this repository is a research benchmark dataset and set of documented test results rather than an automated command-line scanner.
It scans individual files separately, relying on the model's internal training memory to recognize standard library functions.
See Zekai first in Google