Methodology v1.2 - published 2026-05-26
Methodology
How the ConsentMark scanner measures analytics governance, what the grade means, and what it deliberately does not claim. Versioned and citable.
Changelog
Older scans link to the methodology version they were scored under. The grade attached to a specific /scan/<id> URL is the grade as it was recorded on that date, under the methodology in force at the time.
- v1.22026-05-26
Precedent corpus expansion to 20 cited DPA enforcement decisions across DPC, ICO, CNIL, EDPB, AEPD and BfDI. New /precedent/ public route with per-case permalinks. 'Cite this scan' / 'Cite this case' citation blocks added.
- Registry expanded from 7 to 20 cases; AEPD (Spain) and BfDI (Germany) added as primary regulators.
- Public /precedent/ corpus route with filterable index, per-case pages, and versioned JSON export at /precedent/registry.v1.json.
- Citation block on /scan/[id] and /precedent/[slug] - BibTeX, APA, Markdown, plain text.
- Methodology page itself is now versioned. /methodology redirects to the latest version. Scan pages link to the version they were scored under.
- v1.12026-05-15
Three-zone narrative (Observed -> Regulatory context -> Precedent) goes live on /scan/[id]. Dual-citation Wayback snapshots on every registry entry.
- Public scan results render Observed / Regulatory / Precedent zones explicitly, never blended.
- Every primary-source URL in the enforcement registry carries a web.archive.org dual-citation.
- Static-vs-runtime methodology note added for Consent Mode v2 detection.
- v1.02026-04-12
Initial published methodology. Reject-state network-leak threshold (absolute + relative gate). Severity ladder caps grade at C, D, or F based on delta size.
- Reject-state leak detected when post-reject requests exceed pre-consent by both 10 absolute and 10% relative.
- Severity ladder: <=25 delta = Moderate (cap C), <=50 = High (cap D), >50 = Critical (cap F).
- Non-operable reject button caps grade at D regardless of leak size.
How we scan
The scanner loads a public website in a real browser (Playwright + Chromium), records every network request and cookie before consent, after accepting consent, and after rejecting consent. The resulting grade (A through F) is a summary of those observations - principally whether the site keeps making third-party tracking requests after the visitor has rejected.
Consent lifecycle: the scanner identifies the consent management platform (CMP), exercises its accept and reject controls, and observes the delta in network and cookie behaviour across the three phases. CMP detection happens via DOM signatures + window-object sniffing; no proprietary integration is required from the scanned site.
The reject-state leak threshold
A reject-state leak is recorded only when the post-reject phase fires materially more network requests than the pre-consent phase. Both gates must hold at the same time:
- Absolute floor. At least 10 additional requests after reject compared with the pre-consent phase.
- Relative floor. The additional requests amount to at least 10% of the pre-consent request count.
Worked example
A site fires 86 network requests on first load (pre-consent). After the visitor clicks Reject All, the site goes on to fire 185 requests in total. The delta is 99 - well above the absolute floor of 10 and the relative floor of 10% (99 / 86 = 115%). Both gates clear, so a leak Finding is recorded.
Sites that clear only one gate do not trigger a Finding. The two-gate design is deliberate - it stops single-pixel reload noise on small sites from registering, and it stops heavy-content sites from registering simply because 10-20 background requests is normal post-banner behaviour.
Severity ladder
When a Finding is recorded, the size of the leak determines the maximum grade the site can receive. The ladder is one-way - a Finding can only cap a grade, never raise it.
| Observation | Severity | Grade cap | Reading |
|---|---|---|---|
| delta <= 25 extra requests | Moderate | C | Reject is not clean, but the leak is contained. |
| delta <= 50 extra requests | High | D | Reject produces only token suppression of third-party traffic. |
| delta > 50 extra requests | Critical | F | Reject is effectively inert. Most third-party traffic continues unchanged. |
| Reject button cannot be operated | High | D | A consent platform is detected but no functional reject control could be exercised. |
When more than one Finding fires on the same scan, the strictest cap wins. A critical leak combined with a non-functional reject button caps at F.
Gate definitions
Each grade is the output of a small set of named gates. A scan can clear all the gates (Grade A), trip a content-only gate (Grade B), trip a consent-mechanics gate (Grade C), or trip a substantive reject-leak gate (Grade D or F).
- D1. Reject-state network leak above the severity-D threshold (delta greater than 25 requests).
- D3. Reject control is detected but cannot be exercised (e.g. missing event handler, asymmetric layer hierarchy, hidden behind a JavaScript-only "Settings" link).
- F-gates. Critical-severity reject leak (delta greater than 50), no detectable consent mechanism at all, or known-broken CMP integration.
- B / C bands. Surface findings about pre-consent inventory size, configuration weak points (e.g. Consent Mode v2 referenced in source but not wired at runtime), or excess third-party requests in the consent-accepted state.
- I. Inconclusive. The scanner could not observe a stable consent lifecycle - typically because the site failed to load or the CMP was unreachable within the scan budget.
What this is, and what it isn't
What the grade is
A reproducible, evidence-backed editorial signal. A summary of observable browser behaviour - which trackers fire, when they fire, and whether reject is honoured. Each grade is derived from a versioned set of Findings that the scan emits.
What we deliberately do NOT do
- No AI-generated verdicts. The narrative paraphrases regulator language, never invents legal conclusions.
- No analogising. The Precedent zone cites cases by case-ref, never says "this is like X".
- Dual-citation only. Every regulator URL has a web.archive.org Wayback snapshot beside it so a regulator site refactor cannot dead-link the citation.
Why these signals matter
The reject-state leak threshold is grounded in published regulator guidance and enforcement decisions, not in any private interpretation. The full corpus is published at /precedent/; representative anchors:
- EDPB Cookie Banner Taskforce - Report of work undertaken (Report 2/2023)
European Data Protection Board guidance on the practical application of Article 5(3) of the ePrivacy Directive and the Article 7 GDPR consent obligations to cookie banners and trackers.
- EDPB Guidelines 03/2022 on deceptive design patterns
EDPB's named taxonomy of dark patterns (overloading, skipping, stirring, obstructing, fickle, left in the dark) - the canonical regulator-language reference for any consent-banner design finding.
- CNIL SAN-2020-012 (Google) and SAN-2020-013 (Amazon)
Foundational French DPA cookie precedent: advertising cookies set on arrival without consent are unlawful under Article 82 of the French Data Protection Act.
Disputing a grade
If you believe a published scan misrepresents your site, write to contact@consentmark.com with the scan URL and the specific Finding you contest. We commit to:
- Acknowledging the dispute within two working days.
- A 14-day notification window before any change to the public grade of an affected site, where a regrade is the result of a methodology change rather than a fresh scan.
- Preserving the original scan record alongside any correction, so the audit trail is never silently rewritten.