The truth wears off?

[LanceGary]

see

http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=all

[Charles Pegge]

Fascinating Lance!

Scientific research relies too heavily on the formulaic application of statistics. Scientists are egoistically and financially motivated to obtain proof of theory. Editors of Scientific journals are disinclined to publish null results. Replicating previous experiments is not as interesting to funders as new research.

Yet our digital technology requires precision and reliability to the trillionth degree and we manage to achieve it.

Charles.

[danbaron]

I wonder how many claims of incredible experimental results would be destroyed if skeptics attempted to replicate them, and did enough trials so that the probability of chance was reduced to, say, one in a trillion?

James "The Amazing" Randi does something similar with claims pertaining to, "the paranormal, pseudo-scientific, and supernatural".

http://www.randi.org/site/

http://en.wikipedia.org/wiki/James_Randi

I bet the last thing many industry and academic researchers would want, is someone like him attempting (especially publicly) to replicate their results. Apparently, many "scientific" findings, would more accurately be referred to as, pseudo-scientific. And, you don't need a Ph.D. in statistics to realize that the more trials you do, the more accurate your results become, and the less likely it becomes that your results be will be what you want them to be. I don't see a mystery.

(My speculation.)
********|
********|
********V
verbal-overshadowing = f
cosmic-habituation***= f
Jonathan-Schooler****= f

:mad::p

[LanceGary]

I wonder how many claims of incredible experimental results would be destroyed if skeptics attempted to replicate them, and did enough trials so that the probability of chance was reduced to, say, one in a trillion?

James "The Amazing" Randi does something similar with claims pertaining to, "the paranormal, pseudo-scientific, and supernatural".

http://www.randi.org/site/

http://en.wikipedia.org/wiki/James_Randi

I bet the last thing many industry and academic researchers would want, is someone like him attempting (especially publicly) to replicate their results. Apparently, many "scientific" findings, would more accurately be referred to as, pseudo-scientific. And, you don't need a Ph.D. in statistics to realize that the more trials you do, the more accurate your results become, and the less likely it becomes that your results be will be what you want them to be. I don't see a mystery.

(My speculation.)
********|
********|
********V
verbal-overshadowing = f
cosmic-habituation***= f
Jonathan-Schooler****= f

:mad::p




I think your guesses are right but it misses the point. These researchers mostly followed the prescriptions given by good statisticians (for example in the gold standard randomised trials) and nevertheless ended up with false results. It is doubtful that any of them faked results. Schooler actually tried to replicate his results himself - so why would he be scared of someone else doing it? It seems that the scientific method as presently taught needs to be tightened up...

Lance

[danbaron]

I think that Schooler became famous, and probably achieved tenure because of results which are now suspect. Then he coined the phrase, "cosmic habituation", to me, implying that some mysterious phenomenon was involved. I think he was wrong twice.

I believe the researchers associated with the failures in the article, followed the accepted procedures for randomized trials. But, I also think there is a human tendency (but not an irresistible compulsion) to stop when you get the results you want. I think the inability to later reproduce previous results, could have been found initially, if more trials had been done the first time.

I don't think Schooler would be afraid of anyone else trying to verify his initial results, because, he volunteered that he could not reproduce them. But, I think he tried to, "make lemonade from a lemon", by claiming a second discovery from his first failure.

I think that theoretically the scientific method is fine. However, I don't think it specifies anything about what certainty in a presumed result, is good enough. I think the flaws in its actual implementation as performed by many researchers are obvious. I think they have crept into the process over many years. I don't think the reasons cited in the article for results which later cannot be replicated, would be surprising to anyone who has the ability to do scientific research. I think that if someone is going to do scientific research, then, he should be smart enough to realize when the practices he encounters in the academic and work environment, are suspect. If someone is going to do scientific research, then, he had better be capable of independent thought.

My suspicion is that there are many scientific researchers who never experience the problem of being unable to reproduce previous results. I think it depends on how important it is to the person to not publish something that may later be determined to be wrong. I think that those people are much more likely to perform many more trials than what a textbook says is adequate. And, I think they are much more likely to analyze their results upside down, backwards, and inside out, before ever letting them see the light of day. I guess it also depends on the intelligence of the researcher. A smarter guy is more likely to trust his own intellect and independent study, rather than what he was taught in school.

Additionally, if someone is going to do research that involves experimental trials, then it seems to me he would continually study statistics, so that he mastered and fully understood every facet of it, that any of his claims might ever be dependent on.

Dan :mad::p

[LanceGary]

I think that Schooler became famous, and probably achieved tenure because of results which are now suspect. Then he coined the phrase, "cosmic habituation", to me, implying that some mysterious phenomenon was involved. I think he was wrong twice.

I believe the researchers associated with the failures in the article, followed the accepted procedures for randomized trials. But, I also think there is a human tendency (but not an irresistible compulsion) to stop when you get the results you want. I think the inability to later reproduce previous results, could have been found initially, if more trials had been done the first time.

I don't think Schooler would be afraid of anyone else trying to verify his initial results, because, he volunteered that he could not reproduce them. But, I think he tried to, "make lemonade from a lemon", by claiming a second discovery from his first failure.

I think that theoretically the scientific method is fine. However, I don't think it specifies anything about what certainty in a presumed result, is good enough. I think the flaws in its actual implementation as performed by many researchers are obvious. I think they have crept into the process over many years. I don't think the reasons cited in the article for results which later cannot be replicated, would be surprising to anyone who has the ability to do scientific research. I think that if someone is going to do scientific research, then, he should be smart enough to realize when the practices he encounters in the academic and work environment, are suspect. If someone is going to do scientific research, then, he had better be capable of independent thought.

My suspicion is that there are many scientific researchers who never experience the problem of being unable to reproduce previous results. I think it depends on how important it is to the person to not publish something that may later be determined to be wrong. I think that those people are much more likely to perform many more trials than what a textbook says is adequate. And, I think they are much more likely to analyze their results upside down, backwards, and inside out, before ever letting them see the light of day. I guess it also depends on the intelligence of the researcher. A smarter guy is more likely to trust his own intellect and independent study, rather than what he was taught in school.

Additionally, if someone is going to do research that involves experimental trials, then it seems to me he would continually study statistics, so that he mastered and fully understood every facet of it, that any of his claims might ever be dependent on.

Dan :mad::p


There is probably nothing closer to slave labour than a graduate student bearing most of the teaching load in a university but unable to achieve tenure. More strength to schooler.

I don't see people as essentially motivated by selfishness or evil. I think (with a few exceptions) that Socrates was right - most people try to do good, though may well be mistaken about what good results will arise from their actions.

I think Schooler is right to give the phenomenon a name - the declining of interesting research findings, that is. The article makes clear that it happens frequently, and can happen even in the physical sciences. The article doesn't agree that it is anything cosmic - it arises from many social and psychological and methodological and isntitutional quirks that jointly lead to falsely significant results. The more we study the phenomenon the better we will be able to counter it.

You say "And, I think they are much more likely to analyze their results upside down, backwards, and inside out, before ever letting them see the light of day.". Hmm. I think this will make the problem worse. It seems to me that the easy availablity of stats packages means that instead of testing the most important claims and leaving it at that, most researchers now do what you suggest. This means that the more statistical procedures are carried out the more likely it is that some accidentally significant results will occur... (The term "family wise" error was coined by John Tukey to cover this kind of error, but built in controls will not work if the data is repeatedly analysed in different ways).

Cheers

Lance

[danbaron]

Maybe you're right, Lance.

I agree, most people try to do good.

This topic is not exactly clear to me.

Dan :mad::p