Translated from Japanese

When evaluating AI alerts, I also want to compare the number of missed attacks

In tables comparing security AIs, I want to see the number of missed attacks placed right next to the F1 score. Even if the number of alerts is reduced, it is difficult to consider it an improvement if the number of unverified attacks increases.

The Alert-Bench paper includes a comparison using the same 300 alerts while varying the role configuration of Aida. Of those, 54 alerts were attack-related. The F1 score for all configurations was 0.952, and the recall—the rate at which attacks were caught—was 0.926. When the role responsible for counter-arguments was removed, the F1 was 0.946 and recall was 0.963. Calculated against the 54 alerts, the misses amount to 4 and 2, respectively.

The F1 score is a metric that combines the accuracy of alerts identified as attacks with the percentage of attacks successfully caught. In this comparison, the configuration with a slightly higher F1 actually resulted in more misses. However, the 95% confidence interval for the 0.006 F1 difference is -0.020 to 0.061, which includes zero. I do not intend to determine the superiority of the configuration based on this small comparison.

If it were up to me, I would keep the "total number of attacks, number caught, and number missed" in the comparison table and check how many alerts would be sent to humans under those conditions. This is an experiment replaying recorded logs, which is a classification of already-detected alerts. It cannot be used to measure performance in terms of how many actual attacks were prevented across an entire SOC.

Is it possible to reduce the number of alerts humans need to verify while keeping the miss rate at the same level?

Loading comments…