Methodology · Last data refresh 2026-08-11

How the scores are calculated

Every number on this site comes out of one table, one formula and one set of public sources. It is designed so you can disagree with it precisely — recompute it yourself, weight the pillars differently, or throw out the score and read the matrix.

The four statuses

Each tool is mapped against each of the 9 standards with one of four values, plus a note quoting what the documentation says.

✓

Documented

The vendor documents the capability, with enough detail to tell what it actually does. = 1 point

~

Partial

Documented but limited — gated behind a higher tier, capped, or only part of what the standard asks for. = 0.5 points

✗

Not offered

The vendor documents that it does not do this, or the docs make clear it is out of scope. = 0 points

?

Undocumented

Nothing public either way. Scores zero — a capability a buyer cannot verify is one they cannot count on. = 0 points

The formula

score = Σ weight(status of each of the 9 standards)
weight: documented = 1 · partial = 0.5 · not offered = 0 · undocumented = 0

All nine standards carry equal weight. That is a deliberate simplification: your team's weighting is almost certainly different, which is why every profile shows the full matrix rather than only the total. Worked example — CodeRabbit scores 6/9: 4 documented, 4 partial, 1 not offered, 0 undocumented.

Undocumented scoring zero is the most contestable choice here, so it is worth being explicit: it means a tool can be penalised for poor documentation rather than poor capability. We think that is the right bias for a buyer — you cannot put an undocumented promise in a procurement review — but each profile shows how many pillars are undocumented, so you can see when a low score is really a documentation problem.

What the score is not

• — It is not a benchmark. We do not claim to have measured how many real bugs each tool finds on your codebase; nobody can do that honestly from the outside.
• — It is not a quality rating. A tool can document all nine standards and still produce noisy reviews.
• — It is not editorial preference. The order of the directory is the arithmetic above, nothing else.
• — It is not permanent. Vendors ship. Every row carries a verification date and gets re-checked; stale rows are a bug, so report them.

Where the facts come from

Pricing, licensing, self-hosting and platform support are read from the vendor's own pricing page, documentation or public repository, and stamped with the date they were checked. Where a vendor does not publish a number, the field says so instead of carrying an estimate. Vendor-published benchmark figures are attributed and labelled as vendor-published — they are never folded into a score.

The full dataset is in the repository as one JSON file per tool. If a claim looks wrong, the fastest fix is an issue or a pull request against that file.