Question Analysis
Investigate Windows registry modification events to find the name of the registry value that was changed under the WindowsFirewall policy path when someone turned the firewall off.
a4651931-ebbb-4cde-9363-ddf3d66214cb
Question ID
4/23
Models Correct
1
Valid Answer
17%
Success Rate
Reference Answer
Human-validated correct answer for this question
EnableFirewall
Model Performance Comparison
See how different AI models performed on this question
Model | Answer | KQL Query | Status | Attempts | Time | Cost |
---|---|---|---|---|---|---|
gemini-2.5-flash-preview-04-17 | EnableFirewall | DeviceRegistryEvents
| where ActionType == "RegistryValueSet"
| where RegistryKey contains "WindowsFirewall"
| project Timestamp, DeviceName, InitiatingProcessCommandLine, RegistryKey, RegistryValueName, PreviousRegistryValueData, RegistryValueData, ActionType | Correct | 1 | 13.20s | 0.0063 |
gpt-35-turbo | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0110 |
gpt-4-turbo-2024-04-09 | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.2353 |
gpt-4.1 | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0412 |
gpt-4.1-finetuned | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0434 |
gpt-4.1-mini | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0077 |
gpt-4.1-nano | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0015 |
gpt-4o | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0530 |
gpt-5-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.2364 |
gpt-5-mini-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0402 |
gpt-5-mini-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0107 |
gpt-5-mini-medium | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0141 |
gpt-5-nano-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0077 |
gpt-5-nano-medium | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0064 |
grok-3-beta | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0957 |
grok-3-mini-beta | EnableFirewall | DeviceRegistryEvents
| where ActionType == 'RegistryValueSet'
| where RegistryKey has 'WindowsFirewall'
| project RegistryValueName, RegistryKey, RegistryValueData, Timestamp
| summarize arg_max(Timestamp, *) by RegistryValueName, RegistryKey // This helps get the latest change per unique value name and key | Correct | 1 | 15.00s | 0.0016 |
o1-high | EnableFirewall | DeviceRegistryEvents
| where ActionType in ("RegistryValueSet", "RegistryValueModified")
| where RegistryKey has_any ("WindowsFirewall", "FirewallPolicy")
| project TimeGenerated,
DeviceName,
InitiatingProcessAccountName,
ActionType,
RegistryKey,
RegistryValueName,
PreviousRegistryValueData,
RegistryValueData
| order by TimeGenerated desc | Correct | 5 | 128.98s | 0.9753 |
o1-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 1.1727 |
o3-high | EnableFirewall | // Investigate registry modifications that disabled Windows Firewall and return the value name that was flipped off
// Time range is supplied by the hunting portal / API wrapper
DeviceRegistryEvents
// 1. Only registry–value changes (set / modify / delete)
| where ActionType in ("RegistryValueSet", // Defender event name
"RegValueModified", // legacy
"RegistryKeyValueModified", // Win-10/11
"RegistryValueDeleted") // value removed ‑ also disables FW
// 2. Limit to the Windows Firewall policy locations (GPO and local)
| where RegistryKey has "\\WindowsFirewall" // GPO based – HKLM\SOFTWARE\Policies\Microsoft\WindowsFirewall\*
or RegistryKey has "\\FirewallPolicy" // Local policy – HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\*
// 3. We are interested in the moment it was turned OFF – EnableFirewall == 0 or value removed
| extend NewValue = tostring(RegistryValueData),
OldValue = tostring(PreviousRegistryValueData)
| where RegistryValueName =~ "EnableFirewall" // the switch for every profile (Domain / Private / Public)
and (NewValue == "0" or isnull(NewValue) or NewValue == "")
// 4. (Optional) make sure it was previously on
| where isempty(OldValue) or OldValue !in ("0", "False", "Off")
// 5. Return a single event per device/profile with the data we care about
| summarize LastTimeDisabled = arg_max(TimeGenerated, *) by DeviceId, RegistryKey
| project LastTimeDisabled, // when it was switched off
DeviceName,
RegistryPath = RegistryKey,
DisabledBy = strcat(InitiatingProcessAccountDomain, "\\", InitiatingProcessAccountName),
Process = InitiatingProcessFileName,
ProcessCmd = InitiatingProcessCommandLine,
NewValue,
OldValue,
RegistryValueName
// 6. If you only need the value-name that was flipped, uncomment the next line
// | summarize by RegistryValueName | Correct | 2 | 67.22s | 0.0381 |
o3-mini-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0399 |
o3-mini-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0427 |
o4-mini-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0491 |
o4-mini-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0360 |