How the SiteRemora store health score works

What the score is

The SiteRemora store health score is the share of our accessibility checks that your storefront passes. We scan your pages, run every check that applies to each one, weight each check by how much it affects people, and average the pages into one number from 0 to 100.

A 100 means every check we run passed on every page we scanned. Anything still open keeps the score at 99 or below, and an open critical problem keeps it at 89 or below.

How a page is scored

Each page runs a set of checks. Most are axe-core rules. The rest are our own detectors for things axe skips, like images inside hidden pop-ups. A check only counts on pages where it applies, so the form-label check counts on pages that have a form.

Each check passes or fails. A page scores the share of its check weight that passes, and the store scores the average of its pages.

Why the weights come from Lighthouse

Google's Lighthouse scores accessibility the same way. Each audit passes or fails, with no partial credit, and each is weighted by how badly axe says it affects people.^* A problem that stops someone outright, like a button with no name or an image with no alt attribute, weighs 10. Serious ones, like low contrast or a link with no name, weigh 7. Moderate ones, like a skipped heading level, weigh 3.^*

We use those weights for every rule Lighthouse scores. Rules Lighthouse leaves unweighted, like empty headings, count 1 here so they still show up. Missing alt text always counts 10, whichever of our detectors finds it.

Borrowing Lighthouse's table also means a developer who runs Lighthouse on one of your pages sees the same problems weighed the same way. We would rather use a scale people already know than invent our own.

Why the score is a rate

We divide by what we checked, so the score means the same thing for a small store and a large one.

Researchers looking for a fair accessibility score have landed in the same place. In 2011 a W3C research group surveyed the options, and one approach it describes is the failure rate: violations divided by the number of places a violation could happen.^* A pass rate is the same idea counted from the other end. WAQM, a metric Markel Vigo and colleagues published in 2007, works this way too.^* It divides each checkpoint's violations by the places that checkpoint could fail, then weights the results by priority.^*

A plain count of issues grows with the size of a site. A rate tells you what share passed, so you can compare your store with another one, or with itself after the theme grows.

To see it in numbers, take fifteen problems spread over six pages, the home page and five product pages, with every page running the same 47 checks. Put those six pages in an eight-page store, then in a 150-page store. We ran both through the code that scores every scan.

In the eight-page store, six of the eight pages have problems, and the score is 80. The 150-page store passes 99% of its check weight, since 144 of its pages are clean, but critical problems are still open, so its score holds at 89.

Why an open critical problem holds the score at 89

Go back to the eight-page store and fix its problems one group at a time.

Fixing contrast moves the score from 80 to 81, since that was one serious problem out of fifteen. Naming the buttons, links and newsletter field gets it to 88. Fixing the lists, nested controls and small tap targets takes the pass rate to 96, but the score stays at 89 because the product images still have no alt text. Fixing the headings takes the pass rate to 98, and the score is still 89. Once the last image has a description, it jumps to 99. Removing the empty headings gets it to 100.

The hold at 89 is on purpose. A store with a critical problem open should look unfinished, because for anyone who runs into that problem, it is. The hold at 99 does the same for smaller issues, so a perfect score appears only once every check passes.

Two more choices behind the number

Every scanned page counts the same. A problem on the cart page moves the score as much as the same problem on an old blog post, even though far more shoppers see the cart. Keeping pages equal keeps the score steady from one scan to the next.

A check passes or fails as a whole on each page. One unnamed button costs a page as much as twenty would, which is how Lighthouse works too.^* The issue list carries the detail: every failing element, the file and line it comes from, and every page it is on.

What the score covers

The score tracks what automated checks find. It shows progress, and it is not a compliance rating. Whether a site conforms to WCAG is a judgment about the whole standard, and a person has to make it.

Automated checks catch part of the picture. In 2017 GOV.UK's accessibility team built a page with 143 deliberate barriers and ran ten automated tools against it. The best single tool found 41% of them, and 42 got past every tool.^* Deque, the company behind axe-core, looked at more than 2,000 of its own first-time audits in 2021 and found that automated tests covered 57% of the issues.^* The two studies count differently, and both leave a large share for a person to find.

Any score built on automated checks shares that limit. When WAQM's authors compared its scores with an expert panel's ratings of 14 pages, the correlation was 0.55: related, and well short of a match.^*

So a 100 means every automated check passed on every page we scanned. Someone still needs to go through the store with a keyboard and a screen reader, and the checklist is a good place to start.

In SiteRemora, the score sits at the top of every scan with the count behind it, like 371 of 376 checks passing, and each failing check is listed below with its fix or the person who owns it.

The number tells you how much of your store passes, and the list underneath tells you what is left.