gpt-5-nano-medium vs o1-high KQL Benchmark
o1-high wins by 39.5%
Compared on 185 shared test questions
Overall Accuracy
gpt-5-nano-medium
23.8%
44 / 185 correct
o1-high
63.2%
117 / 185 correct
Average Cost per Query
gpt-5-nano-medium: $0.0069
o1-high: $0.5239
o1-high costs 7440.6% more
Average Execution Time
gpt-5-nano-medium: 65.07s
o1-high: 57.03s
gpt-5-nano-medium takes 14.1% longer
Question-by-Question Analysis
Question-by-Question Comparison
Detailed comparison showing where each model succeeded or failed
Showing 1 to 25 of 185 questions
Page 1 of 8
T1082 | On Windows systems, identify when the built-in Shadow Copy utility is used to enumerate existing snapshots. What was the full command executed? | ✓ | ✗ | gpt-5-nano-medium Wins |
T1082 | A Windows system shows a cmd.exe process spawn that appears to have been used for environment discovery. Review the process creation records to identify the exact command the adversary ran to enumerate environment variables. | ✓ | ✗ | gpt-5-nano-medium Wins |
T1112 | On Windows systems, disabling RDP via the registry generates registry write events. Investigate registry event logs for modifications under the Terminal Server configuration path. What is the name of the registry value that was changed to disable Remote Desktop Protocol? | ✓ | ✗ | gpt-5-nano-medium Wins |
T1217 | On Linux, review the process execution logs to uncover when Chromium’s bookmark JSON files were being located and the results persisted. Focus on shell commands that search under .config/chromium and write output to a file. What was the filename used to save the findings? | ✓ | ✗ | gpt-5-nano-medium Wins |
T1547.014 | Windows registry events show that a new key under the Active Setup Installed Components branch was added to launch a payload immediately via runonce.exe. Which component name was created? | ✓ | ✗ | gpt-5-nano-medium Wins |
T1562.004 | Investigate Linux process execution logs for any use of iptables that removes a rule blocking outbound FTP traffic. What full command was issued to delete this rule? | ✓ | ✗ | gpt-5-nano-medium Wins |
T1016.001 | An analyst notices a PowerShell process on a Windows host that appears to be checking SMB connectivity. Which PowerShell cmdlet was executed to perform this outbound port 445 test? | ✗ | ✓ | o1-high Wins |
T1016.001 | On a Linux host, a ping command was executed to test internet connectivity. Determine which IP address was used as the ping target. | ✗ | ✓ | o1-high Wins |
T1027 | On a Windows endpoint, look for evidence of a base64-encoded PowerShell payload execution. Which executable launched the encoded command? | ✗ | ✓ | o1-high Wins |
T1018 | A Windows host executed an ICMP-based network reconnaissance using a looping instruction in cmd.exe. Identify the exact command line that was used to perform the ping sweep. | ✗ | ✓ | o1-high Wins |
T1018 | Review Linux process execution records for any commands that list TCP metric cache entries and filter out loopback interfaces. Which utility was used? | ✗ | ✓ | o1-high Wins |
T1036.003 | A process is running under a familiar Windows host name but originates from a user's AppData folder rather than the System32 directory. Identify the filename used to masquerade the PowerShell binary on this Windows device. | ✗ | ✓ | o1-high Wins |
T1003 | On a Windows host, an attacker leveraged COMSVCS.DLL via rundll32.exe to dump the memory of svchost.exe into a file under the Temp directory. Review the file creation logs to determine the exact name of the dump file that was generated. | ✗ | ✓ | o1-high Wins |
T1053.003 | Linux hosts may log events when new files are added to /var/spool/cron/crontabs. Query those logs for a creation or write action in that directory and determine the file name that was added. | ✗ | ✓ | o1-high Wins |
T1049 | A user launched a Windows command prompt and executed a built-in utility to enumerate all active network connections. Using process creation logs, identify the exact tool that produced the list of current connections. | ✗ | ✓ | o1-high Wins |
T1048.003 | A Linux host briefly hosted an HTTP service under /tmp. Examine process creation logs to determine the exact python3 command that was used to start the server on port 9090. | ✗ | ✓ | o1-high Wins |
T1036.004 | A threat actor on a Windows system crafted and registered a service named almost identically to the standard time service, but redirecting execution to a custom script. Review the logging data to determine which native command-line tool was used to perform this action. What utility was invoked? | ✗ | ✓ | o1-high Wins |
T1049 | In a Windows log analytics workspace, search for PowerShell processes that were used to enumerate network connections. Determine which PowerShell cmdlet was executed to list active TCP connections. | ✗ | ✓ | o1-high Wins |
T1048.003 | Windows process creation logs show a PowerShell-driven file transfer to an FTP service. Which executable was leveraged to perform this exfiltration? | ✗ | ✓ | o1-high Wins |
T1036.004 | Analyze Windows process events for any schtasks.exe commands that created a new task invoking PowerShell. What is the name of the .ps1 script specified to run? | ✗ | ✓ | o1-high Wins |
T1053.005 | Investigate Windows process events for PowerShell activity that leverages WMI to register a scheduled task via XML import. What was the name of the XML file supplied to the RegisterByXml method? | ✗ | ✓ | o1-high Wins |
T1053.005 | On a Windows host, find any scheduled task that was registered using PowerShell native cmdlets instead of schtasks.exe. What was the name given to the new task? | ✗ | ✓ | o1-high Wins |
T1059.004 | On a Linux system, find any process creation record where awk is used with a BEGIN rule to launch a shell. What was the exact command invoked? | ✗ | ✓ | o1-high Wins |
T1057 | While reviewing Windows process events, you spot a PowerShell process executing a WMI enumeration cmdlet. What WMI class name did the attacker query? | ✗ | ✓ | o1-high Wins |
T1059.004 | On a Linux host, identify the process invocation that altered a user’s login shell. What was the full command used? | ✗ | ✓ | o1-high Wins |
Page 1 of 8
Explore individual model performance and detailed analysis