Investigate Windows registry modification events to find the name of the registry value that was changed under the WindowsFirewall policy path when someone turned the firewall off.
KQL answer reference and model-by-model benchmark comparison
a4651931-ebbb-4cde-9363-ddf3d66214cb
Question ID
4/23
Models Correct
1
Valid Answer
17%
Success Rate
Reference Answer
Human-validated correct answer for this question
EnableFirewall
Model Performance Comparison
See how different AI models performed on this question
| Model | Answer | KQL Query | Status | Attempts | Time | Cost |
|---|---|---|---|---|---|---|
| gemini-2.5-flash-preview-04-17 | EnableFirewall | DeviceRegistryEvents
| where ActionType == "RegistryValueSet"
| where RegistryKey contains "WindowsFirewall"
| project Timestamp, DeviceName, InitiatingProcessCommandLine, RegistryKey, RegistryValueName, PreviousRegistryValueData, RegistryValueData, ActionType | Correct | 1 | 13.20s | 0.0063 |
| gpt-35-turbo | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0110 |
| gpt-4-turbo-2024-04-09 | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.2353 |
| gpt-4.1 | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0412 |
| gpt-4.1-finetuned | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0434 |
| gpt-4.1-mini | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0077 |
| gpt-4.1-nano | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0015 |
| gpt-4o | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0530 |
| gpt-5-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.2364 |
| gpt-5-mini-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0402 |
| gpt-5-mini-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0107 |
| gpt-5-mini-medium | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0141 |
| gpt-5-nano-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0077 |
| gpt-5-nano-medium | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0064 |
| grok-3-beta | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0957 |
| grok-3-mini-beta | EnableFirewall | DeviceRegistryEvents
| where ActionType == 'RegistryValueSet'
| where RegistryKey has 'WindowsFirewall'
| project RegistryValueName, RegistryKey, RegistryValueData, Timestamp
| summarize arg_max(Timestamp, *) by RegistryValueName, RegistryKey // This helps get the latest change per unique value name and key | Correct | 1 | 15.00s | 0.0016 |
| o1-high | EnableFirewall | DeviceRegistryEvents
| where ActionType in ("RegistryValueSet", "RegistryValueModified")
| where RegistryKey has_any ("WindowsFirewall", "FirewallPolicy")
| project TimeGenerated,
DeviceName,
InitiatingProcessAccountName,
ActionType,
RegistryKey,
RegistryValueName,
PreviousRegistryValueData,
RegistryValueData
| order by TimeGenerated desc | Correct | 5 | 128.98s | 0.9753 |
| o1-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 1.1727 |
| o3-high | EnableFirewall | // Investigate registry modifications that disabled Windows Firewall and return the value name that was flipped off
// Time range is supplied by the hunting portal / API wrapper
DeviceRegistryEvents
// 1. Only registry–value changes (set / modify / delete)
| where ActionType in ("RegistryValueSet", // Defender event name
"RegValueModified", // legacy
"RegistryKeyValueModified", // Win-10/11
"RegistryValueDeleted") // value removed ‑ also disables FW
// 2. Limit to the Windows Firewall policy locations (GPO and local)
| where RegistryKey has "\\WindowsFirewall" // GPO based – HKLM\SOFTWARE\Policies\Microsoft\WindowsFirewall\*
or RegistryKey has "\\FirewallPolicy" // Local policy – HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\*
// 3. We are interested in the moment it was turned OFF – EnableFirewall == 0 or value removed
| extend NewValue = tostring(RegistryValueData),
OldValue = tostring(PreviousRegistryValueData)
| where RegistryValueName =~ "EnableFirewall" // the switch for every profile (Domain / Private / Public)
and (NewValue == "0" or isnull(NewValue) or NewValue == "")
// 4. (Optional) make sure it was previously on
| where isempty(OldValue) or OldValue !in ("0", "False", "Off")
// 5. Return a single event per device/profile with the data we care about
| summarize LastTimeDisabled = arg_max(TimeGenerated, *) by DeviceId, RegistryKey
| project LastTimeDisabled, // when it was switched off
DeviceName,
RegistryPath = RegistryKey,
DisabledBy = strcat(InitiatingProcessAccountDomain, "\\", InitiatingProcessAccountName),
Process = InitiatingProcessFileName,
ProcessCmd = InitiatingProcessCommandLine,
NewValue,
OldValue,
RegistryValueName
// 6. If you only need the value-name that was flipped, uncomment the next line
// | summarize by RegistryValueName | Correct | 2 | 67.22s | 0.0381 |
| o3-mini-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0399 |
| o3-mini-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0427 |
| o4-mini-high | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0491 |
| o4-mini-low | No answer provided | No query generated | Incorrect | 5 | 0.00s | 0.0360 |