// methodology
How the KQL benchmark works
An evaluation framework that tests AI models on generating effective cybersecurity detection rules from real-world attack scenarios.
Curated from Atomic Red Team tests covering real cybersecurity threats.
Frontier language models tested on natural language to KQL translation.
A three-step human review process keeps the benchmark reliable.
Test selection & question generation
Atomic Red Team foundation
The benchmark starts with 2,253 Atomic Red Team tests — real-world attack simulations covering the MITRE ATT&CK framework across platforms and techniques.
AI-generated questions
Language models generate realistic analyst-level questions from each scenario — the natural language a security professional would actually ask when investigating.
“A reconnaissance tool was executed on a Windows system. Identify the specific function of the tool executed…”
Log collection & environment setup
Controlled test environment
- Isolated Windows and Linux virtual machines
- Microsoft Defender for comprehensive logging
- Real-time protection disabled to avoid interference
Realistic data collection
When tests execute, we collect all logs from the environment — not just the malicious activity. This mirrors the noise levels of a real security operations center.
The “needle in the haystack” approach tests models under realistic conditions.
AI model evaluation process
Query generation
Models receive a natural language question and generate a KQL query.
Real-time execution
Generated queries run against real log data in Azure Log Analytics.
Iterative refinement
Models get up to 5 attempts to self-correct and find the answer.
// what we measure
- Accuracy: share of correct answers
- Attempts: how many tries to succeed
- Latency: speed of query generation
- Cost: API usage cost per model
Three-step human validation
// pass 01
Manual review of 38 representative questions to verify generated queries return correct results.
// pass 02
Examination of every question no model could solve, to remove poisoned or unsolvable tests.
// pass 03
A dashboard review to catch ambiguous questions with multiple valid answers.
// quality assurance
188 validated test cases in the final dataset
48 ambiguous or problematic questions removed
100% accuracy in spot-check validation
All validation steps documented for transparency
// results
Benchmark impact
A rigorous methodology gives the security community reliable, actionable insight into AI model capability for threat-detection automation.
// explore
Ready to read the results?
Open the interactive leaderboard to compare model accuracy, cost-effectiveness, and the full set of 188 benchmark scenarios.
View results