// question
Investigate Windows registry modification events to find the name of the registry value that was changed under the WindowsFirewall policy path when someone turned the firewall off.
Reference KQL answer and model-by-model comparison.
a4651931-ebbb-4cde-9363-ddf3d66214cb
question id
4/23
models correct
1
valid answer
17%
success rate
// reference answer
Human-validated correct answer for this question.
EnableFirewall
// model comparison
| Model | Answer | KQL query | Status | Tries | Time | Cost |
|---|---|---|---|---|---|---|
| gemini-2.5-flash-preview-04-17 | EnableFirewall | DeviceRegistryEvents | where ActionType == "RegistryValueSet" | where RegistryKey contains "WindowsFirewall" | project Timestamp, DeviceName, InitiatingProcessCommandLine, RegistryKey, RegistryValueName, PreviousRegistryValueData, RegistryValueData, ActionType | pass | 1 | 13.20s | 0.0063 |
| gpt-35-turbo | no answer | no query generated | fail | 5 | 0.00s | 0.0110 |
| gpt-4-turbo-2024-04-09 | no answer | no query generated | fail | 5 | 0.00s | 0.2353 |
| gpt-4.1 | no answer | no query generated | fail | 5 | 0.00s | 0.0412 |
| gpt-4.1-finetuned | no answer | no query generated | fail | 5 | 0.00s | 0.0434 |
| gpt-4.1-mini | no answer | no query generated | fail | 5 | 0.00s | 0.0077 |
| gpt-4.1-nano | no answer | no query generated | fail | 5 | 0.00s | 0.0015 |
| gpt-4o | no answer | no query generated | fail | 5 | 0.00s | 0.0530 |
| gpt-5-high | no answer | no query generated | fail | 5 | 0.00s | 0.2364 |
| gpt-5-mini-high | no answer | no query generated | fail | 5 | 0.00s | 0.0402 |
| gpt-5-mini-low | no answer | no query generated | fail | 5 | 0.00s | 0.0107 |
| gpt-5-mini-medium | no answer | no query generated | fail | 5 | 0.00s | 0.0141 |
| gpt-5-nano-high | no answer | no query generated | fail | 5 | 0.00s | 0.0077 |
| gpt-5-nano-medium | no answer | no query generated | fail | 5 | 0.00s | 0.0064 |
| grok-3-beta | no answer | no query generated | fail | 5 | 0.00s | 0.0957 |
| grok-3-mini-beta | EnableFirewall | DeviceRegistryEvents | where ActionType == 'RegistryValueSet' | where RegistryKey has 'WindowsFirewall' | project RegistryValueName, RegistryKey, RegistryValueData, Timestamp | summarize arg_max(Timestamp, *) by RegistryValueName, RegistryKey // This helps get the latest change per unique value name and key | pass | 1 | 15.00s | 0.0016 |
| o1-high | EnableFirewall | DeviceRegistryEvents
| where ActionType in ("RegistryValueSet", "RegistryValueModified")
| where RegistryKey has_any ("WindowsFirewall", "FirewallPolicy")
| project TimeGenerated,
DeviceName,
InitiatingProcessAccountName,
ActionType,
RegistryKey,
RegistryValueName,
PreviousRegistryValueData,
RegistryValueData
| order by TimeGenerated desc | pass | 5 | 128.98s | 0.9753 |
| o1-low | no answer | no query generated | fail | 5 | 0.00s | 1.1727 |
| o3-high | EnableFirewall | // Investigate registry modifications that disabled Windows Firewall and return the value name that was flipped off
// Time range is supplied by the hunting portal / API wrapper
DeviceRegistryEvents
// 1. Only registry–value changes (set / modify / delete)
| where ActionType in ("RegistryValueSet", // Defender event name
"RegValueModified", // legacy
"RegistryKeyValueModified", // Win-10/11
"RegistryValueDeleted") // value removed ‑ also disables FW
// 2. Limit to the Windows Firewall policy locations (GPO and local)
| where RegistryKey has "\\WindowsFirewall" // GPO based – HKLM\SOFTWARE\Policies\Microsoft\WindowsFirewall\*
or RegistryKey has "\\FirewallPolicy" // Local policy – HKLM\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy\*
// 3. We are interested in the moment it was turned OFF – EnableFirewall == 0 or value removed
| extend NewValue = tostring(RegistryValueData),
OldValue = tostring(PreviousRegistryValueData)
| where RegistryValueName =~ "EnableFirewall" // the switch for every profile (Domain / Private / Public)
and (NewValue == "0" or isnull(NewValue) or NewValue == "")
// 4. (Optional) make sure it was previously on
| where isempty(OldValue) or OldValue !in ("0", "False", "Off")
// 5. Return a single event per device/profile with the data we care about
| summarize LastTimeDisabled = arg_max(TimeGenerated, *) by DeviceId, RegistryKey
| project LastTimeDisabled, // when it was switched off
DeviceName,
RegistryPath = RegistryKey,
DisabledBy = strcat(InitiatingProcessAccountDomain, "\\", InitiatingProcessAccountName),
Process = InitiatingProcessFileName,
ProcessCmd = InitiatingProcessCommandLine,
NewValue,
OldValue,
RegistryValueName
// 6. If you only need the value-name that was flipped, uncomment the next line
// | summarize by RegistryValueName | pass | 2 | 67.22s | 0.0381 |
| o3-mini-high | no answer | no query generated | fail | 5 | 0.00s | 0.0399 |
| o3-mini-low | no answer | no query generated | fail | 5 | 0.00s | 0.0427 |
| o4-mini-high | no answer | no query generated | fail | 5 | 0.00s | 0.0491 |
| o4-mini-low | no answer | no query generated | fail | 5 | 0.00s | 0.0360 |