Methodology

A measurement pipeline,not a chatbot.

One model asked for an opinion gives you one opinion. Preditance asks hundreds of differently grounded people, measures their disagreement, and checks itself against what really happened.

The pipeline

Six steps

How a clearance is made

Six steps run between the document you paste and the verdict you read. Each one is described here in the order it happens.

01 — Grounding

Every agent holds a real position.

Personas are drawn from a vector-indexed corpus of demographic profiles, not invented for each run. The same decision meets the same kinds of people every time.

  • What a persona carries

    Region, income bracket, age band, sector, employment type, economic sensitivity and risk tolerance. Together they decide how a person weighs the decision in front of them.

  • Edge cases on purpose

    Regulators, journalists and activists sit inside the population. They are the people who turn a quiet change into a news cycle, so they are never sampled out.

  • Segments large enough to be evidence

    Grouping on a full four-part label would split 200 agents into about 42 buckets of four or five. The label is coarsened until the median segment holds at least eight agents, which yields about nine segments.

What agent count buys you
AgentsWhat it resolves
50A headline rate with a wide band, no segments
100A usable rate, segments still thin
200A rate plus about nine resolvable segments
500Fine-grained segments and tight bands

02 — Blinding and quality gate

Judge the decision, not the logo.

Two checks run before a single agent is spawned: one removes bad input, the other removes brand bias.

  • Quality gate

    Empty, truncated or nonsense documents are rejected before any compute is spent. You are never billed for a document that could not be run.

  • Decision blinding

    Company and product names are stripped from the document. Agents react to what the decision does, without brand loyalty or brand hostility colouring the answer.

  • Live context, with sources

    A research layer can pull current competitor pricing, precedents and measured outcomes from the web. Each claim is attributed to a source before it enters an agent's context.

03 — Adversarial simulation

Hundreds of reactions, each one structured.

Agents are asked to find the reasons the decision fails. Their answers come back in a form that can be counted, compared and traced to a segment.

  • In character, in parallel

    Each clearance runs between 50 and 500 agents, 200 by default. Every agent reacts from its own grounded position and returns a structured objection rather than free-form prose.

  • A pre-flight committee

    Before the mass run, economist, psychologist and regulatory specialists, a red team and a synthesis judge review the decision. The committee can stop a clearly broken decision early.

  • Model-independent

    Language models sit behind an abstraction and can run as an ensemble. A change at any single provider is a configuration change, and the weekly harness shows whether verdicts moved.

04 — Aggregation and confidence

A rate with its interval, or no call at all.

Reactions become a rejection rate, a confidence band and one of five verdicts. When the evidence is thin, the verdict says so.

  • Quasi-Monte Carlo sampling

    Random sampling clusters and leaves gaps, so the same decision can return a different number on each run. Sobol low-discrepancy sequences cover the space evenly, so estimates converge faster and stay reproducible.

  • Adaptive early stopping

    Sampling halts once the 90% confidence interval is tight enough. Agents that were not needed are refunded to your quota.

  • Bands, not point estimates

    Every rejection rate ships with its interval. A wide band is reported as a wide band, on the report, the certificate and the API response.

The five verdicts
ItemValue
ClearedResistance is low enough to ship as written.
Cleared with modificationsShip it, but address the named objections first.
High riskMaterial resistance across multiple segments.
Do not launchRejection is high enough to expect public damage.
Insufficient evidenceThe signal is too thin to call. Deliberately not a guess.

05 — Calibration and drift checks

Checked against what really happened.

Recorded outcomes feed back into the verdict. Accuracy is measured per decision type and shown to you, including when it is unflattering.

  • Isotonic calibration

    Raw scores are mapped onto observed outcome frequencies, so a stated 70% means roughly 70% in practice. Pricing, launch, policy and packaging decisions calibrate separately.

  • Sample size is disclosed

    Below 20 verified outcomes, a decision type is marked low-confidence on the report itself, in plain language.

  • Weekly drift harness

    Twelve canonical cases re-run every week against fixed verdict bands. If a model or data change shifts a verdict, we see it before a customer does.

  • Causal analysis of misses

    A miss is attributed to a named cause, and each cause maps to a correction for the next run of that decision type.

Attributed causes
ItemValue
MarketA market event shifted conditions between simulation and outcome.
CoverageA key demographic subgroup was missing from the population.
StalenessToo much time passed, and conditions changed beyond 90 days.
IntensityAdversarial intensity was too low, so the run was too optimistic.
Calibration dashboard comparing predicted and recorded outcomes by decision type.
The calibration dashboard in the Preditance console.

06 — Privacy

The verdict persists. The document does not.

Strategy documents are sensitive by definition. They are handled as briefly as the run allows, and nothing about the result can be quietly changed later.

  • Held in memory only

    Documents are held in memory for the duration of the run, then explicitly zeroed. They are never written to storage.

  • Blinded before any agent sees it

    Company and product names are removed at the start of the pipeline, so no agent context carries them.

  • Tamper-evident results

    A certificate hashes the verdict, the rejection rate with its confidence band, the decision's identifiers, the issue time and the report's own hash with SHA-256. Recompute it and any alteration shows.

Test the method on a decision you know.

Paste a decision whose outcome you already know. It is the fastest way to judge whether the verdict deserves your trust.