How much should you trust an AI index?

An index earns trust through its construct, evidence and reproducibility, not its visual precision.

Begin with the claim the number would make

Normalisation makes observations calculable on a chosen scale. It does not establish that components represent the intended construct or deserve their weights. The OECD/JRC handbook exposes such modelling choices; it does not validate Mayden’s dimensions. Handbook on Constructing Composite Indicators: Methodology and User Guide ↗.

Historical public-evidence editions used the strongest qualifying disclosure in each of nine themes, a fixed denominator and equal nominal weights. Arithmetic could be replayed, but one narrow implementation could fill a theme, unknown public evidence contributed no documented points, and access varied. These properties limit enterprise-capability interpretation. The current edition removes individual company scores from active public research.

The future instrument should distinguish a formative composite, reflective subscales and direct descriptive measures. A formative design defines a concept through selected components; high correlation is not automatically desirable. Reflective components need their own model and tests. Neither is validated by a convenient distribution. Conventional Wisdom on Measurement: A Structural Equation Perspective ↗; Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer ↗.

Keep source, claim and operating case separate

A document can support several claims; URLs can repeat one assertion; claims can describe one case; a group can appear through subsidiaries. URL volume is not independent confirmation. Record original assertion, document version, extracted passage, claim, review and release. A provenance graph exposes derivation, not truth. PROV-DM: The PROV Data Model ↗.

Legacy review records are not all bound to exact claim hashes, and some sources lack recorded dates. New controls should surface those gaps without rewriting history as if reviews had occurred. A queue cannot confer eligibility, confirm current operation or manufacture a second reviewer.

Missingness needs a reason: not searched, no qualifying evidence within a declared search, inaccessible source, outside scope or an observed negative finding. A participant instrument also needs unknown and not-applicable responses. Treating every state as zero creates precision about a different question.

Challenge modelling choices without inventing probabilities

The 11 October challenge tested 512 joint weight-corner specifications and 36 ordinal-spacing specifications against frozen v4, alongside a separately labelled baseline reference. Independent replay checked 43,840 specification-record results across 548 nonbaseline choices and 80 records. It verifies tested arithmetic, not source truth or construct validity.

Within the 512 weight choices, 53 of 59 historically ranked records changed position in at least one tested specification. Analysts selected the set; it is not a probability distribution, an exhaustive plausible model space or an error rate. This cannot become a claim that a corresponding percentage of companies was misclassified. Scope, disclosure inequality and intended use remain primary reasons to reconsider public comparison.

Sensitivity also needs source families, case dependence, coding disagreement, missing evidence and operating breadth. Nominal weight can differ from statistical influence when components vary and correlate. Estimating influence in a small selected corpus would not supply correct future weights. Weights and importance in composite indicators: Closing the gap ↗.

Make institutional trust part of the method

The publication model is deliberate. Inspected BCG Applied AI Index and McKinsey State of AI articles use aggregate respondent comparisons. BCG’s Most Innovative Companies 2023 publicly names and ranks firms using a different reputation/financial-performance composite. These observations establish neither client relationships nor universal consultancy policy. BCG AI research ↗; McKinsey AI research ↗; BCG innovation methodology ↗.

Mayden’s accepted direction is public sector research with attributed facts and a future confidential opt-in diagnostic. Previously published data cannot become anonymous. Preserve history and explain the change. Public facts do not imply participation or endorsement.

Appoint a research lead, a consequential-decision reviewer, a correction route and authority to halt unsupported release. The World Bank discontinued Doing Business after data irregularities: institutional standing does not substitute for controls. That case is not causal evidence about Mayden. World Bank statement ↗.

Predeclare intended decisions, units, sampling frame, validation criteria and analysis. Test content, comprehension, coding and role agreement, sector/language comparability and outcomes against simple alternatives. Report attrition and distinguish within-company change from composition or method change. Human appointments, fieldwork and empirical validation remain future gates.

Leadership agenda

Method lead: specify what each item measures and its required evidence.

Release editor: place denominator, period, access limits and assumptions beside each finding.

Programme owner: appoint accountable humans and protect their right to stop publication.