Multiverse and specification-curve analysis: decisions to results

Why decision sensitivity matters

Cleaning, variable definitions and models can admit several defensible choices. A multiverse makes their consequences visible. A specification curve organizes estimates and their decisions; sorting coefficients alone does not create valid inference.

Record scientifically justified alternatives before viewing results. The aim is not to select the most attractive finding, and a collection of invalid analyses remains invalid.

Map choices and dependencies

Document each choice, rationale, applicability condition and version. Multiplying option counts gives the path count only when combinations are compatible. Retain excluded combinations and reasons.

The synthetic example combines three prespecified data-quality rules and two descriptive models into six paths. Quality flags are already defined in the input; exclusions are not selected for favorable outcomes.

Preserve every result

For each path retain identifier, rules, sample size, estimate, units and execution status. The official R multiverse package supports such workflows. Our standard-Python example teaches a small version of the idea, not full package equivalence.

Different samples, adjustments or transformations may target different quantities. Do not average coefficients with incompatible units. The example reports score units per study hour and preserves sample and model labels.

Interpret without counting significance

Compare direction and magnitude alongside changed samples and decisions. The fraction of paths below a p-value threshold is not an independent test; paths commonly reuse observations. Whole-curve inference requires its own justified method.

The six-path teaching script returns descriptive coefficients, not unsupported p-values or confidence intervals. Add design-appropriate uncertainty for a real project.

Report and rerun

Publish the prespecified options, failures and reasons, labeling later additions. Keep the curve, decision table, code and permitted inputs in one release.

Rerun a predefined rule change and explain which estimate moved and why. Dependence of a claim on one particular path is a substantive limitation to report.

Completed teaching worksheet

This is a hypothetical teaching case, not observed data, an actual review or a publication acceptance. Numbers illustrate decisions.

Deliverables and completion checks

• Justified decision and option table.
• All-path results with sample counts and status.
• Code, synthetic inputs and six-path rerun outputs.

Version and scope references: multiverse — Official R package documentation · Center for Open Science — TOP 2025