Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I think you're on to something regarding "significance." Over in my dept. we like to say that significance is a measure of sample size. The question, then, hinges on whether or not something has practical significance. Because we've built the whole research reputation incentive structure on the .05 significance level, studies can be designed to get that.

A great article on the problem with statistical significance (by a Frequentist) is here, called Why Most Published Research Findings Are False: http://www.plosmedicine.org/article/info:doi/10.1371/journal...

The punchline seems to be, well, that there's also a large human element that contributes to the problem. I think it's one thing to rail on the Frequentist way-of-thinking; it's entirely another to state that the institution of scientific research built on it creates unwanted incentives.



If you plan experiments and the like, the definition of a significance level (always) comes with a definition of sample size.


The relationship between significance level and sample size is reliant on a complex set of assumptions to say the least, and, when everything is stripped away, is perhaps best seen as a way of discovering just how difficult it will be to deblur the world. What power prescription we need.

Often (always?) these constraints are all so very much more complex than Gaussian power analysis states. You do it as a way of sketching the depth of a problem I think, not much more.

The linked paper is a pretty clear introduction of the high level problems. I think it's perhaps a little more grim than necessary, but then again that might just be my own bias.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: