Social science
Social science research, evaluated after the replication crisis
The last fifteen years taught social science what flexible analysis can do to a literature. The draft rubric below is written in that shadow, and the founding cohort will hold evidence to it.
What counts as evidence of social science skill
Social science reformed its evidence in public: preregistration, registered reports, open data and materials, large-scale replication projects. A preregistration honored in the final paper, or a replication conducted fairly, is now among the clearest evidence of research skill the field produces — clearer than most publication records.
Design is where judgment shows. A natural experiment that genuinely isolates its cause, an instrument that measures the construct it names, a survey whose sampling frame matches its claims — these are hard-won, and an experienced reader can score them from the artifact alone.
Outside academia, the same skills run policy evaluations, UX research programs, and people-analytics teams. A trial run for a ministry, or an honest program evaluation that reports a null result, is research work of the first order — and the people who produce it rarely have citation counts to show.
Reviewing skill leaves its own record: registered-report reviews, red-team readings of preregistrations, published comments that identified a confound the authors missed. So does methodological service — validating scales, building survey panels, writing the power analysis a team actually followed. These artifacts sample precisely the judgment evaluation requires, and they are readable without anyone's permission.
The social science evaluation rubric, first draft
Written after the replication crisis, this rubric treats a preregistered null as stronger evidence of skill than a flexible positive.
- Identification strategy
- The design isolates the causal claim it makes, and threats to identification are named and addressed rather than acknowledged in passing.
- Measurement validity
- Instruments measure the constructs they claim, with evidence, and the construct does not drift between measurement and conclusion.
- Analytic discipline
- Analysis choices are pre-specified or transparently explored, robustness is shown across reasonable forks, and the garden of forking paths is closed, not hidden.
- Generalization honesty
- Claims state the population they cover, sample limitations are priced into conclusions, and external validity is argued rather than assumed.
Founding social-science evaluators will refine this draft against preregistered and pre-crisis work alike, so the standard prices both eras fairly.
What founding social science evaluators will do
Challenge the rubric where the field is hardest to score: qualitative work, mixed methods, and designs where preregistration is genuinely impractical.
Run calibration rounds on public studies — preregistrations against outcomes, replication targets, applied evaluations — and study the disagreements between trained readers.
Accumulate the discipline's first calibration records, which will anchor evaluation weight when the community opens to the field at large.
Who this is for
The cohort needs social scientists whose skepticism survived the last fifteen years with their curiosity intact:
- Quantitative researchers who read the methods and the preregistration before the abstract.
- Applied researchers in policy, UX, and people analytics whose best studies never reach journals.
- Replicators and meta-scientists who have done the field's least rewarded, most informative work.
- Early-career researchers trained on open-science norms who want that training to count.
Who this is not for
Stated as plainly as the rubric demands of everyone else:
- Anyone wanting a platform for their own findings — evaluators here read other people's work.
- Researchers who experience methodological scrutiny as hostility; the rubric applies it evenly.
- Anyone unwilling to carry a visible record of their scoring accuracy.
- Anyone expecting live scoring today rather than a standard being drafted honestly first.
Apply to evaluate social science
Social science is pre-selected on the application. Link an ORCID or Scholar profile, an OSF page, or applied work we can read.