The 403 Economy: 1 in 14 Permitted Top-Site Homepages Returned 403 or a Stored Signature Flag

Research highlights

• The auditable union is 375 HTTP 403 responses + 244 stored signature flags - 212 overlaps = 407 outcomes.
• The policy gate permitted 5,601 of 10,000 domains; all 4,399 other frame members retain an explicit pre-request state.
• All 32 signature-only outcomes returned HTTP 200 with HTML; status-only classification would miss 7.86% of the 407-outcome union.
• A separate first-browser panel recorded an explicit wall on 19 of 477 initially clean pages, a measured client/time reversal.
• Leaving out any one rank band kept the measured rate between 7.15% and 7.39%, versus 7.27% in the full permitted panel.

ProxyWing Research analyzed a frozen frame of 10,000 Tranco domains using a declared research client and a conservative pre-request gate. The gate permitted 5,601 homepage requests. Of those requests, 407 returned HTTP 403, carried a stored signature flag, or did both.

Key statistic

407 of 5,601 permitted homepage requests met the recorded outcome definition: 7.27%, or about one in 14.

The result is more useful than a block-rate headline. It shows where an access measurement can fail if it ignores the policy gate, double-counts correlated signals, treats HTTP 200 as proof of content delivery, or assumes that one client predicts another.

Key findings

• The conservative gate permitted 5,601 of 10,000 homepages, or 56.01%. The other 4,399 were not evaluated at the homepage stage.
• The primary union contained 375 HTTP 403 responses and 244 stored signature flags, with 212 observations carrying both signals.
• The signals were highly overlapping. Of the 244 stored signature flags, 212, or 86.89%, co-occurred with HTTP 403.
• HTTP status alone missed 32 observations. Every one returned HTTP 200 with HTML while carrying a stored signature flag.
• Those 32 represented 0.65% of the 4,953 HTTP 200 HTML responses, or about one in 155.
• A separate paired browser panel reversed on 19 of 477 first Lighthouse attempts after every page had passed a same-domain 2xx HTML and no-stored-flag screen.
• Removing any one Tranco rank band left the primary observed rate between 7.15% and 7.39%. No single band drove the 7.27% result.

The denominator starts before the homepage request

The collector requested robots.txt before each homepage and proceeded only when its conservative policy permitted the request. That first decision removed 4,399 domains from the homepage denominator.

This is why 7.27% is conditional on the 5,601 permitted requests. The 4,399 excluded domains are not successes and are not failures. They are unobserved at the homepage stage.

The gate also changed coverage by popularity band. It permitted 42.00% of ranks 1-100, 48.78% of ranks 101-1,000, 53.73% of ranks 1,001-5,000 and 59.42% of ranks 5,001-10,000. A full-frame percentage that silently treated withheld domains as successful responses would mix access policy with homepage behavior.

Among the permitted requests, the recorded union rate was 388 of 5,085, or 7.63%, after a parsed allow decision and 19 of 516, or 3.68%, after a 404 or 410 allow decision. This is descriptive heterogeneity between gate pathways. It does not show that either robots outcome caused the later response.

The collector's policy was stricter than the latitude in the Robots Exclusion Protocol, RFC 9309, which allows access to other resources when robots.txt is unavailable through a 4xx status. The study keeps the original conservative decision instead of broadening the denominator after collection.

Union components

The primary metric is a union, not the sum of two independent counters.

The arithmetic is 375 + 244 - 212 = 407. Adding the two marginal counts would produce 619 and overstate the deduplicated union by 212 observations, or 52.09%.

The overlap also shows that the signals are dependent. A stored signature flag appeared on 56.53% of HTTP 403 responses, while 86.89% of all stored signature flags co-occurred with HTTP 403. The flag is therefore a complementary content check, not a second independent estimate.

The HTTP 200 trap

All 32 observations that carried a stored signature flag without HTTP 403 returned HTTP 200 with an HTML content type. A monitor that defined success as status == 200 would have classified every one as delivered.

There were 4,953 HTTP 200 HTML responses in the permitted panel. The 32 stored-flag observations account for 0.65% of them, approximately one in 155. They also account for 7.86% of the primary union.

This does not turn the fixed string detector into a CAPTCHA classifier. Response bodies were reduced to features and hashes rather than retained, so the 32 pages cannot now be manually relabeled. The operational conclusion is narrower: transport status and delivered content are different measurements. HTTP 200 should be followed by content-type, redirect and page-content checks.

A clean lightweight response did not always predict the browser result

The project also contained a deterministic, rank-stratified Lighthouse panel. All 477 selected pages had already passed the lightweight screen as same-domain 2xx HTML with no stored signature flag.

On the first headless Chrome and Lighthouse attempt:

• 18 pages surfaced an explicit HTTP 403 in Lighthouse;
• one redirected to an explicit CAPTCHA destination;
• 19 of 477 reversed to an explicit wall, or 3.98%.

The unweighted Wilson interval is 2.56% to 6.14%. Weighting the four sample strata to the 4,543-page eligible frame produces 4.34%, with a design-based 95% interval of 2.35% to 6.32%. Weighting changes the point estimate by 0.35 percentage points.

The intervals overlap substantially. The top-100 stratum has only 27 selected pages and one wall, so its interval is especially wide.

The first attempt is the defined endpoint. Eighteen of the 19 first-attempt wall cases had three recorded attempts and 18 never produced a numeric same-domain result in those records. One did produce such a result on another attempt. Follow-up was not uniform, so those later observations describe persistence but do not replace the first-attempt rate.

Client, timing, redirects, cookies, subresources, network reputation, configuration and ordinary site change all differ or can differ between the lightweight request and the browser run. The panel demonstrates conditional reversal, not browser-fingerprinting causality.

The primary rate is not carried by one rank band

Observed homepage outcomes varied by rank, but the smallest permitted stratum also carried the greatest uncertainty.

A leave-one-band-out check tests whether one stratum determines the headline:

The range is 7.15% to 7.39%, within 0.12 percentage points of the 7.27% primary estimate in either direction.

Measurement definition matters more. The strict two-signal intersection is 3.79%, HTTP 403 alone is 6.70%, and the union is 7.27%. These answer different questions. The union is retained because it captures explicit refusal plus a small set of clean-status responses with a stored content flag.

What access monitoring should record

A useful agent-access monitor needs a sequence of states, not one Boolean field:

1. Policy state: permitted, disallowed, unavailable, invalid or unresolved.

2. Transport state: status code, error, elapsed time and redirect chain.

3. Content state: content type, destination match and stored signature flag.

4. Browser state: first-attempt status, explicit challenge redirect and render outcome for a defined sample.

5. Context: timestamp, declared client, network vantage and the exact measurement definition.

The denominator should remain attached to every rate. A domain skipped by policy is not equivalent to a permitted request that returned content. A 200 response is not equivalent to verified destination content. A retry that later succeeds does not erase the first-attempt wall.

For production systems, an explicit wall should become a recorded terminal outcome or a policy-aware review item. It should not trigger an automatic bypass loop.

Method in brief

• Frame: Tranco list K9LZW, ranks 1-10,000; list created August 13, 2026.
• Census collection: August 14, 2026, 20:15:56 to 20:43:08 UTC.
• Client: declared ProxyWingResearch/1.0 research user agent.
• Gate: conservative robots.txt decision before the homepage request.
• Primary denominator: 5,601 gate-permitted homepage requests.
• Primary numerator: 407 responses with HTTP 403, a stored signature flag, or both.
• Browser panel: 477 deterministically selected, initially clean homepages; first Lighthouse attempt analyzed separately.
• Network: one shared execution environment.
• Safety: public GET requests only, no login, form, cart, checkout, challenge solving or bypass.

The primary 7.27% Wilson interval is 6.62% to 7.98%. It is a descriptive interval for the observed panel, not a probability-sample claim about the global web.

Sources

Official documentation and source records used for the figures above.

• Tranco. Tranco list K9LZW
• Tranco. Tranco research-oriented top-sites ranking
• IETF. RFC 9110 HTTP Semantics: 403 Forbidden
• IETF. RFC 9309 Robots Exclusion Protocol
• Cloudflare. Cloudflare interstitial Challenge Pages
• Cloudflare. Cloudflare Radar HTTP summary API
• Lighthouse project. Introduction to Lighthouse
• Lighthouse project. Lighthouse package version 13.4.1