Question Analysis
On Windows systems, identify when the built-in Shadow Copy utility is used to enumerate existing snapshots. What was the full command executed?
7161b085-816a-491f-bab4-d68e974b7995
Question ID
12/16
Models Correct
1
Valid Answer
75%
Success Rate
Reference Answer
Human-validated correct answer for this question
vssadmin.exe list shadows
Model Performance Comparison
See how different AI models performed on this question
| Model | Answer | KQL Query | Status | Attempts | Time | Cost |
|---|---|---|---|---|---|---|
| gemini-2.5-flash-preview-04-17 | "vssadmin.exe" list shadows | DeviceProcessEvents
| where FileName =~ "vssadmin.exe"
| where ProcessCommandLine has "list shadows"
| project Timestamp, DeviceName, ProcessCommandLine | Correct | 1 | 6.68s | 0.0023 |
| gpt-35-turbo | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0109 |
| gpt-4-turbo-2024-04-09 | "vssadmin.exe" list shadows | DeviceProcessEvents
| where FileName == 'vssadmin.exe' and ProcessCommandLine contains 'list shadows'
| project TimeGenerated, DeviceName, ProcessCommandLine | Correct | 1 | 8.21s | 0.0503 |
| gpt-4.1 | "vssadmin.exe" list shadows | DeviceProcessEvents
| where FileName =~ "vssadmin.exe"
| where ProcessCommandLine has_any ("list shadows", "list shadowstorage")
| project TimeGenerated, DeviceName, AccountName, FileName, ProcessCommandLine | Correct | 1 | 5.42s | 0.0107 |
| gpt-4.1-finetuned | "vssadmin.exe" list shadows | DeviceProcessEvents
| where FileName =~ "vssadmin.exe" and (ProcessCommandLine contains "list" and ProcessCommandLine contains "shad") | Correct | 1 | 6.34s | 0.0141 |
| gpt-4.1-mini | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0071 |
| gpt-4.1-nano | vssadmin.exe list shadows | DeviceProcessEvents
| where ProcessCommandLine has_any ('vssadmin', 'vssadmin list shadows', 'shadowcopy', 'shadow copy')
| where ProcessCommandLine contains 'list shadows' or ProcessCommandLine contains 'list shadows'
| project TimeGenerated, DeviceName, InitiatingProcessCommandLine, InitiatingProcessFileName | Correct | 1 | 3.04s | 0.0006 |
| gpt-4o | vssadmin.exe list shadows | DeviceProcessEvents
| where FileName == "vssadmin.exe" and ProcessCommandLine contains "list shadows"
| project TimeGenerated, DeviceName, AccountName, FileName, ProcessCommandLine | Correct | 1 | 3.95s | 0.0145 |
| grok-3-beta | vssadmin.exe list shadows | DeviceProcessEvents
| where FileName == 'vssadmin.exe' and ProcessCommandLine contains 'list shadows'
| project Timestamp, DeviceName, AccountName, ProcessCommandLine, InitiatingProcessFileName, InitiatingProcessAccountName
| order by Timestamp desc | Correct | 1 | 8.48s | 0.0206 |
| grok-3-mini-beta | "vssadmin.exe" list shadows | DeviceProcessEvents
| where FileName == "vssadmin.exe"
| where tolower(ProcessCommandLine) contains "list shadows"
| project Timestamp, DeviceName, AccountName, ProcessCommandLine
| order by Timestamp desc | Correct | 1 | 16.17s | 0.0016 |
| o1-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.6543 |
| o1-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.6739 |
| o3-mini-high | vssadmin.exe list shadows | DeviceProcessEvents
| where tolower(FileName) == "vssadmin.exe" or tolower(InitiatingProcessFileName) == "vssadmin.exe"
| where tolower(ProcessCommandLine) has "list shadows" or tolower(InitiatingProcessCommandLine) has "list shadows"
| extend fullCommand = iif(isnotempty(ProcessCommandLine), ProcessCommandLine, InitiatingProcessCommandLine)
| project TimeGenerated, DeviceName, fullCommand
| order by TimeGenerated desc | Correct | 2 | 33.34s | 0.0201 |
| o3-mini-low | vssadmin.exe list shadows | DeviceProcessEvents
| where (InitiatingProcessFileName =~ "vssadmin.exe" or FileName =~ "vssadmin.exe")
| where tolower(iif(isnotempty(ProcessCommandLine), ProcessCommandLine, InitiatingProcessCommandLine)) contains "list shadows"
| project TimeGenerated, DeviceName, FullCommand = iif(isnotempty(ProcessCommandLine), ProcessCommandLine, InitiatingProcessCommandLine)
| order by TimeGenerated desc | Correct | 2 | 39.17s | 0.0175 |
| o4-mini-high | vssadmin.exe list shadows | let ShadowTools = dynamic(["vssadmin.exe","diskshadow.exe","wmic.exe"]);
DeviceProcessEvents
| where FileName in~ (ShadowTools)
| where ProcessCommandLine matches regex @"(?i)\b(list|get|enum)\b.*\b(shadow|shadowcopy)s?\b"
| project TimeGenerated, DeviceName, User = InitiatingProcessAccountName, Executable = FileName, CommandLine = ProcessCommandLine
| sort by TimeGenerated desc | Correct | 4 | 152.83s | 0.0418 |
| o4-mini-low | vssadmin.exe list shadows | DeviceProcessEvents
// Filter for built-in Shadow Copy utilities
| where FileName in ("vssadmin.exe", "diskshadow.exe")
// Look for commands that enumerate snapshots
| where tolower(ProcessCommandLine) has "list shadows"
// Project the relevant details
| project TimeGenerated, DeviceName, InitiatingProcessAccountName, FileName, ProcessCommandLine
| sort by TimeGenerated desc | Correct | 1 | 23.91s | 0.0100 |