Measured on real traffic.

The WAF Comparison Project

open-appsec's WAF Comparison Project publishes two datasets: 1,040,242 legitimate requests recorded while browsing 692 real sites, and 73,924 attack payloads. Balanced accuracy is the average of the share of attacks blocked and the share of legitimate requests allowed.

strict's CSRF check is off in these runs, like bot checks and rate limits. It blocks POSTs from other sites by design, and 13% of this dataset's legitimate requests are such POSTs: beacons to analytics and ad hosts, and pages on www calling an API on a sibling subdomain. With it on, strict + CRS blocks 14.0% of the legitimate requests.

Published results for other products

For context, the project's 2026 report measured the products below. These are their numbers and their setup; we did not re-run them.

Balanced accuracy. Other products: open-appsec WAF Comparison Project 2026 report. LightMoon: our run of the same datasets, September 2026.

Read these numbers carefully

• The report ran its Python client over the network against each product. We ran the same requests in process, encoded the way that client encodes them, and counted a 403 as a block, as the project does.
• We used the legitimate set to find and fix false positives. To check that the fixes generalise, we tuned on half the sites and measured on the other half: on sites we never tuned with, LightMoon with CRS blocked 1.5% of legitimate requests, against 0.77% on the sites we tuned with. strict's tuning was done the same way: 1.59% blocked on the half it was chosen on, 1.69% on the other half. So were the later changes of 29 and 30 September: with them balanced + CRS blocks 0.41% of the tuning half's legitimate requests and 0.92% of the other half's (both near 1.2% before), strict + CRS 0.51% and 1.35%. The unseen half gains less, because the biggest fixes were for formats only the tuning half's sites send.
• open-appsec's engine is made by the authors of the benchmark.
• The benchmark measures signature detection only. Bot checks, rate limits, bans, response inspection and sink guards are outside it.

GoTestWAF

GoTestWAF v0.5.8 sends 675 attack requests and 141 legitimate ones. We ran it against LightMoon and against ModSecurity with CRS 4.29, both in front of the same origin on one machine, with rate limits and bans off on both.

On 26 September 2026 we added rules for NoSQL injection, mail (SMTP and IMAP) injection and Windows share paths. With balanced + CRS they took those GoTestWAF groups from 8%, 12.5% and 25% of attacks blocked to 48%, 42% and 50%, and legitimate requests blocked stayed at 3 of 141. On the WAF Comparison Project's 1,040,242 legitimate requests they fired on none, so the numbers above did not change. Most of what we still miss in those groups are values sent base64-encoded, which LightMoon doesn't decode.

OWASP CRS regression tests

Measured on 27 September 2026, before the libinjection port and strict's paranoia 2 tuning; not re-run since.

CRS ships its own regression tests: 5,060 requests, each written to show that one rule fires (an attack) or stays quiet (a legitimate request). We replay them through LightMoon and score attacks at the paranoia level of the rule they target.

These tests check CRS rules one by one, so they favour running CRS itself. The "legitimate" column overstates false positives: many of those tests are attacks that only assert one particular rule stays quiet. Most remaining misses are protocol checks we don't port and CRS rules we run as supporting evidence because they block real traffic.

HttpParamsDataset

Measured on 27 September 2026, before the libinjection port and strict's paranoia 2 tuning; not re-run since.

HttpParamsDataset holds 19,304 benign parameter values from the CSIC 2010 dataset and 11,763 attacks generated with sqlmap, XSSYA, Vega and FuzzDB. Each value is sent as a query parameter.

What changed on 29 September 2026

CRS's libinjection rules (942100 and 941100, and their paranoia 2 siblings) now run on a TypeScript port of libinjection that passes all 584 of libinjection's own tests and matched the C library on over a million inputs, and CRS rules marked multiMatch also test the value before and between their transformations. At full strength the libinjection rules caught 84 more of the WAF Comparison Project's attacks and blocked 17,222 more of its legitimate requests (1.16% to 2.82%), mostly consent and tracking cookies and JSON bodies, so they are supporting evidence. That way its SQL injection set goes from 86.7% to 89.3% blocked: 32 more attacks, and 129 more legitimate requests blocked (12,082 to 12,211, a few dozen of which depend on timing). GoTestWAF: 55.7% to 55.9% of attacks with balanced + CRS, 58.5% to 58.8% with strict + CRS, no change in legitimate requests blocked. The CRS regression tests and HttpParamsDataset above were measured on 27 September and not re-run.

POST bodies and fewer false positives. 1,806 of the WAF Comparison Project's path traversal payloads were blocked in a query string and missed when the same value came as a form body: CRS rule 930100 looks for encoded traversal in the raw query string, and ModSecurity gives it only the decoded form values. LightMoon now gives 930100 and 930110 the raw form body too. That took balanced + CRS from 94.3% to 96.8% of the attacks, with no change in legitimate requests blocked. Then, on the tuning half of the sites, we narrowed three CRS rules (a content type repeated in one header, a PHP directive name after a dot as in Google Analytics' ep.user_agent, a dotfile name right after a letter as in urls.addressBook) and made four rules supporting evidence (CRS 942190 and 933120, LightMoon's sqli-comment-terminator and rce-backtick): each was behind 100 or more blocked legitimate requests, mostly bot-manager telemetry, and added at most 10 attacks. Two more qualified, 941180 and 941130, but GoTestWAF's server-side include, XML and SOAP attacks depend on them, so they stay.

30 September: speed. Inspection with CRS got faster with the same verdicts. The 167 legitimate requests of the tuning half that ran past the 100 ms inspection budget (large beacon and form bodies, blocked when they run over) take a median of about 50 ms instead of 141 ms on one core; balanced + CRS now blocks 0.65% of the legitimate requests. A fix for Python's quoted cookies, which one rule read as NUL escapes, took strict + CRS to 0.91%. The attacks blocked didn't change.

The demotions cost 10 attacks with balanced + CRS, 8 of them SQL injection (89.3% to 88.4% of that set of 916). Most of what LightMoon + CRS still misses is path traversal: 2,096 of 2,384 missed attacks.

30 September: paths read like a file system. A new rule, lfi-path-resolved, decodes a value in up to three layers (also IIS %u, overlong UTF-8 and full-width look-alikes), resolves its . and .. segments with RFC 3986 section 5.2.4, and reports it if the result climbs out of the directory it started in. It follows the specifications and OWASP's guidance, not the benchmark's payloads, and reports only what lfi-path-traversal doesn't already see. Tuned on legitimate traffic only: its first version blocked 8 requests of half A and a second pass over both halves found 10 more, all bot-manager telemetry, so half B was not held out for the second fix. On the WAF Comparison Project it blocks 216 more attacks without the CRS rules (86.6% to 86.9% with balanced, 86.7% to 87.0% with strict) and 12 more with them, and the legitimate requests blocked didn't change beyond timing noise. On GoTestWAF balanced blocks 4 more of 675 attacks and strict 4 more, with the same 0 of 141 legitimate requests blocked.

strict at paranoia 2. Before this change strict + CRS blocked 95.9% of the WAF Comparison Project's attacks and 35.0% of its legitimate requests (43.3% with the CSRF check, which these runs now leave off). One CRS rule, 932271 (~ then a digit, a Unix shell expression), was behind 95,940 of the legitimate requests blocked on one half of the sites, matching the separators in bot-manager cookies. On that half, one rule at a time, we made each paranoia 2 rule that 100 or more blocked legitimate requests still depended on supporting evidence: 26 CRS rules and 6 of LightMoon's own. The list comes from legitimate requests only. strict + CRS now blocks 1.64% of legitimate requests (1.59% on the tuning half, 1.69% on the half it never saw) and 94.6% of attacks, 981 fewer than before but still 198 more than balanced + CRS. strict without CRS went from 0.97% to 0.39% of legitimate requests with the same 86.7% of attacks. GoTestWAF, strict + CRS: 58.8% to 57.0% of attacks, 17 to 6 of 141 legitimate requests blocked.

What changed on 27 September 2026

Eighteen CRS rules that JavaScript's regex engine couldn't run safely, among them the large command-injection rules, now run on a linear-time matcher, and CRS's Base64 step leaves text that isn't Base64 unchanged, as libmodsecurity does. At full strength the recovered rules doubled false positives on the WAF Comparison Project's legitimate requests (1.14% to 2.17%), so the 14 rules that fire on everyday traffic add evidence instead of blocking on their own. The result: CRS regression tests 74.6% to 76.1% (balanced) and 81.2% to 82.3% (strict), GoTestWAF 77.5% to 77.6%, and on the WAF Comparison Project 18 more attacks and 203 more legitimate requests blocked (1.14% to 1.16%). We picked those 14 rules using the full legitimate set, not a held-out half, so the 1.16% is measured on data the tuning has seen.

Reproduce it

All four harnesses are in the repository. The WAF Comparison Project's legitimate dataset is a 1.2 GB download.

npm run build
CRS=1 node bench/waf-comparison/run.mjs   # WAF Comparison Project
bench/gotestwaf/run.sh                     # GoTestWAF, needs Docker
node bench/crs-regression/run.mjs          # CRS regression tests (convert.py first)
node bench/http-params/run.mjs             # HttpParamsDataset

Performance

Per-request cost on one core of a desktop machine (Node 24):

Rules only run when they could match: each rule's regex is analysed for the literals any match must contain, and one Aho-Corasick pass per value decides which rules to run.