Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I was very surprised when I first learned that results could be published without first having been reproduced. I thought at the least they should have been published with a separate label.

This is a worthy initiative.



Traditionally, the public literature is how scientists communicated, which is what made republication possible in the first place. Lab A, perhaps in Germany, works on a research project, and gets some results. They write up their methodology and results, and communicate them to other scientists by publishing the write-up in a journal. The journal will have the write-up peer-reviewed before publishing it, but that doesn't mean verifying it's correct, only that it's a contribution to the scientific literature worth communicating to its readership. For example, it would make sure the paper has done appropriate data analysis, has cited relevant existing work, is not outright kookery, etc.

Now other scientists, anywhere in the world, will read this and decide what they think of it, which may include attempting to replicate it. If they find different results, or have a qualm with an interpretation or methodology, they can write a response, which will also (after peer-review) be communicated by the journal to its readership, either as another paper, or if shorter, as a letter. Thus the back-and-forth of claim/replication/critique happens, all in the open air of the journal's pages, with any lab anywhere in the world able to follow the conversation and jump in if they wish.

To me that's more transparent than doing all this in some kind of closed pre-publication process. Perhaps in the modern era, the open communication doesn't necessarily have to happen in journals anymore: perhaps it should happen on arXiv, and journals should become more conservative in what they publish. But I would like it to still be public.


I was very surprised when I first learned that results could be published without first having been reproduced.

A lot of psychology journals have been burned by that over the years,

http://chronicle.com/blogs/percolator/is-psychology-about-to...

http://simplystatistics.org/post/21326470429/replication-psy...

http://scholarlykitchen.sspnet.org/2012/04/26/reproducibilit...

and as a consequence, just within the last year or so, the most prestigious psychology journals are beginning to ask authors of papers on new experimental research to show replication across at least two distinct data sets. This, in turn, is driving more collaboration among researchers at different research centers. It's a long overdue development, one in accord with best practice in science,

http://norvig.com/experiment-design.html

and I hope this practice spreads to most disciplines that publish new research reports in the major journals of the discipline.

(Although the requirement for prepublication replication is not explicitly mentioned in the publication guidelines of Psychological Bulletin,

http://www.apa.org/pubs/journals/bul/index.aspx

which is the most prestigious psychology journal published in the United States, I have been told at the University of Minnesota behavior genetics "journal club" that replication across more than one data set is becoming a tacit requirement for publication in most of the better journals. Perhaps that guideline will eventually be made explicit for all of the better journals.)




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: