Preregistration as a Way to Prevent Questionable Research Practice

How not to get into trouble with problematic research practices such as HARKing (Hypothesizing After the Results are Known) and p-hacking (misuse of data to obtain statistical significance)? Preregistering your research may help.

Why Preregister?

A preregistration document allows researchers to outline and specify details of their research before a study is conducted. A preregistration is time-stamped and un-editable (in principle), and it may be deposited to an archive or repository and can be viewed by reviewers, editors, and fellow researchers. Preregistration promotes research transparency and reproducibility and future-proof research from questionable research practices, such as:

• HARKing, also known as “Hypothesizing After the Results are Known”, is about the presentation of a post-hoc hypothesis after analyzing data and reporting the research as if the study was designed to test the hypothesis. HARKing is in violation with the scientific method associated with confirmatory research for hypothesis testing.
• P-hacking, also known as “data dredging”, happens when a researcher exploits various degrees of freedom in data analyses, normally unspecified in the original research plan, to increase the likelihood of statistically significant results. Some examples for p-hacking include adding more data, excluding certain observations or “outliers”, and the inclusion or exclusion of certain control variables.
• Outcome switching: Selectively reporting pre-specified outcomes or introducing new outcomes that have not been set previously.
• Publication bias is unfortunately an underlying problem in scholarly publishing for long time. Researchers or editors are more biased towards publishing studies only if they have positive or novel results. However, replication study or null results often remain unpublished.

HARKing and p-hacking increase the rate of false positive in confirmatory research. These flaws could have been the consequence of researchers’ confirmation bias or lack of awareness of appropriate statistical methods. However, they are hard to be detected by peer reviewers and these faulty published results may mislead others into doing dead-end research or making ineffective decisions.

Preregistration is a low-cost solution to avoid such issues by increasing transparency of your research design and the credibility of your results. As a preregistration is made prior to data collection, it clearly states your intended research outcomes, and will sharpen the distinction between hypothesis-generating (exploratory research) and hypothesis-testing (confirmatory research). Also, like a research data management plan, research preregistration by nature helps researchers to plan more thoroughly at the beginning of a project, which in return could save researchers a lot of time in data analysis and results reporting.

Where to Preregister?

There are many venues to pre-register a study. Some publishers or funders have specific requirements. Over here we list out a few options.

Major platforms:

• OSF (Open Science Framework) Pre-registration 
• AsPredicted 

Subject specific platforms:

• Evidence in Governance and Politics (EGAP) Registry
• The American Economic Association’s registry for randomized controlled trials
• Registry for International Development Impact Evaluations (RIDIE) 
• ClinicalTrials (US)
• HKU Clinical Trials Registry

Registered Reports

Registered Reports is a publishing format in which research hypothesis and the methodology are pre-registered and peer-reviewed before data collection. After the research is conducted, the study’s results will be published regardless of the outcome, pending a second stage of peer review. This publication process addresses the issues of publication bias towards significance seeking and publication bias. The Centre of Open Science keeps a list of all journals that publish registered reports. Currently, over 300 journals use the Registered Reports publishing format either as a regular submission option or as part of a single special issue.

Final Notes

Preregistration is a good practice for improving transparency and credibility of research. Yet, having a research project pre-registered alone does not equal good science – It still relies heavily on researchers themselves to adhere to their proposed plan when carrying out the research, and avoid significance chasing and selective reporting of the results.

The practice of research preregistration is becoming common in all different research communities (i.e., not only limited to clinical trials or psychology studies), and it may also be a good idea to preregister your research with careful planning even if it’s exploratory. For inspirations and examples, you may browse or search in the OSF Registries.

References

• OSF Guides – Registrations
• COS Registered Reports
• Preregistration – PhD on Track 

– By Jennifer Gu, Library

Hits: 473