Scientifical accuracy not actually that high?

[erimir]

Unreliable research: Trouble at the lab | The Economist

I have often thought that publication bias was a problem. It's part of the reason that over time I've become less interested in going into academics. You have to publish papers, but it seems to me that being rigorous and finding positive results is hard, but if you try something and it doesn't work out, nobody is interested, even though it would still contribute to our knowledge.

The motivation to subtly fudge things is definitely there.

It also affects what types of things you even investigate, as the article notes. Replications aren't likely to get you anywhere (career-wise) unless you can show that some important finding is bunk.

[The Lone Ranger]

That's the problem with the "publish or perish" mentality, and why I hate it so. This sort of thing gets brought up occasionally, but so far at least, no one seems to have found a good solution.

E. O. Wilson has suggested that we really need to give more credit and recognition to those who write review articles, and I think he's dead-on. The way things are currently organized, practically the only way to get recognition is to publish original research, but this creates a number of serious problems.

For one thing, as noted, nobody publishes negative results. But there's a lot to be learned from studies that produce negative results; if nothing else, it helps us understand how things don't work.

Because negative results are almost never published, researchers are highly motivated to pick nice, safe projects that they're convinced won't produce negative results. This is especially true of those who're trying to earn a master's or doctorate. Unfortunately, this means an awful lot of potentially valuable research isn't being done, because nobody wants to "waste" several years doing research, only to come out of it with nothing that can be published.

The same thing goes for when you're trying to earn tenure. You get tenure by publishing original research. So you pick nice, safe research topics, where you can be reasonably certain you'll get publishable results. And, of course, if you don't get publishable results, there's a very real incentive to fudge the data.


Similarly, as noted, there's very little incentive to replicate previously-done research, since the name of the game is to publish original research. Who knows how many "positive" results are actually due to chance or poor experimental design, but because no one has bothered to replicate the experiments, the mistakes haven't been corrected?



That's why I think Wilson has an important point. In my opinion, we should be giving much more credit -- including credit when it comes to earning degrees and tenure -- to people who don't do truly original research, but instead replicate previous studies, to check them for accuracy.

[Dragar]

Another problem with negative results that are never published: statistically, given enough studies that should produce a negative result, one will spuriously yield a positive result. And that will get published.

It's particularly pronounced in the pharmacology industry (where I currently work). Industry has tried to reproduce a lot of results academics have published. The success rate is well under 30%.

[ceptimus]

Another reason why it's good to publish negative results is that it can sometimes help other scientists not to waste their time on research that is probably doomed to lead nowhere: say you think you have a good idea but when you check the literature to see if it's been done before, you find that twenty individuals/groups have already tried it and found nothing - unless you have some new wrinkle on how to conduct your research, then you'd probably be more useful trying something different.

With the present system, you just don't know - twenty people MAY have tried your research before, but if none of them found anything positive, then most likely nothing will have been published.

[Crumb]

Someone needs to start publishing the Journal of Negative Results.

[SR71]

A really big negative results file might be really helpful in some ways. As a wild example, let's say ambient temperature superconductor. So far as I know, there is no physical law that says it can't exist. Maybe, like Edison fiddling around looking for a durable incandescent lighting element, one might be found if enough trial and error took place. It might help if there were a big compilation of "Tried Already". The summary could contain a recap of important details about how the materials were tried, and any notes about anything interesting that was notable or interesting - some unexpected, unusual or unexplained response of the material.

[The Lone Ranger]

It's especially shameful that such a thing as a "negative results file" doesn't exist nowadays, given how easy it is to store and search information.

It was one thing when you had no choice but to go into books and journals and physically look for the information yourself. But nowadays, there's no excuse for not making easily-searchable databases that could compile massive amounts of such data. They could prove to be an invaluable resource.

[thedoc]

The problem is that negative results are not news-worthy. The recent FTL neutrinos were worth reporting because they contradicted relativity, but when someone discovered the error in the original experiments, the story went cold and nobody cared.

Some time ago there was a hunt for the first planet to be discovered orbiting another star, and at an astronomical conference there was an anticipated announcement of such a planet. But the first up was reveled to be a mistake. And the next one up was accurate and overshadowed the erroneous report. If I recall correctly, the false discovery was not written up, but the correct one was. Writing up the mistake could have been a cautionary tale for other astronomers.

[Dragar]

Originally Posted by Crumb

Someone needs to start publishing the Journal of Negative Results.

Such journals already are published.

[LadyShea]

How does the decline effect fit in?

[Vivisectus]

It IS the decline effect, I think

[Dragar]

It's likely related. Say some big important scientists measure a result: the charge on the electron. People try and copy the experiment, but if they disagree horribly they think they did the experiment wrong or it won't get published. Either way, they don't publish.

But someone does the experiment and it they get a consistent result, but measure the charge a bit higher. So they can publish that. And so on and so forth, all the published results slowly raise the measured value of the charge on the electron until it settles down on the 'true' value.

This is actually what happened (and is discussed by Feynman in 'Cargo Cult Science' if I recall).

We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of--this history--because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong--and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that. We've learned those tricks nowadays, and now we don't have that kind of a disease.

Of course, Feynman was talking about the physics community (and they are very good at not fooling themselves: data from the LHC, for example, is not only analysed by multiple separate teams, but they are given data that may not even be real to make sure all the analysis agrees!). Other fields (and I'm hesitant to say, but biology and its spinoffs in particular seems pretty bad for this) don't seem to have learned this lesson terribly well.

[Kael]

Aha! Science is unreliable, therefore aliens!
:shakehistorychannel:

[LadyShea]

Psychology has a huge decline effect problem

[lisarea]

This is a rebuttal article here:

My response to “Trouble at the lab” | watershedding

I am not a professional sciencer, but it seems to me that a lot of the problem lies with popular conceptions of what "science" even is. It's sort of a trendy concept, really. You know, "Science, bitches!" and t-shirts and memes proclaiming a generalized endorsement of "science" and the sort of jokey science analyses of fictional things, like stuff about Superman and Star Trek and whatever. And that popular conception tends to cast anything science related as totally, magically, simplistically objective. As though science consists of sticking predictable scenarios into a comprehensive model of the natural world and then running a simulation to see what happens, rather than the creeping, cumulative, culturally biased thing it actually is.

I mean, one of the clearest examples is "evolutionary psychology." It's a special kind of honorary "science" that is about as rigorous as Aesop's Fables. And yet, simply because it has the word "evolutionary" in the name, people often act like it's some big objective thing and not just a collection of popular confirmation biases made up by people sitting around mindlessly speculating. It's an extreme case, of course, but ultimately, there's always the influence of mindless speculation and biased observations.

We get fed a lot of bullshit by popular media using "science" as a magic word, through shitty news coverage of barely measurable results, by deceptive infographics and statistical data nobody actually looks at, that get turned into objective truths just by sticking the word science in front of them. And seriously, fuck any online news source that reports on study results and doesn't link to the study. And if the study itself is only available to paid subscribers, maybe at least don't breathlessly and unquestioningly reproduce the stupid press release any more than you'd reproduce witch doctors' reports of a penis theft epidemic.

In conclusion, science is stupid. Radical postmodernism 5eva. Bitches.

[lpetrich]

A way to make negative results seem worth publishing is to put a positive spin on them in some way. A common way is upper limits and lower limits. Particle Data Group has numerous upper limits for various processes and lower limits for the masses of various hypothetical particles.

PLOS ONE: “Positive” Results Increase Down the Hierarchy of the Sciences by Daniele Fanelli

... in some fields of research (which we will henceforth indicate as “harder”) data and theories speak more for themselves, whereas in other fields (the “softer”) sociological and psychological factors – for example, scientists' prestige within the community, their political beliefs, their aesthetic preferences, and all other non-cognitive factors – play a greater role in all decisions made in research, from which hypothesis should be tested to how data should be collected, analyzed, interpreted and compared to previous studies.

The hierarchy, from hard to soft:

• Physical sciences
• Biological sciences
• Social sciences

In one study, 222 scholars were asked to rate several academic disciplines by similarity. An analysis revealed three axes of variation:
Hard - soft
Pure - applied
Life - non-life

There's support for a hard - soft axis of variation from studies of lots of features, like number of colleagues acknowledged per paper, immediacy of references, and even the fraction of paper area dedicated to graphs.

This research examined the frontiers of the physical, biological, and social sciences, and indeed found differences associated with hardness -- researchers in the harder sciences tended to report more negative results, after correcting for various other factors. However, the difference was a matter of degree, not kind.

In his discussion, DF considers an odd conundrum: results in the physical sciences are typically much stronger statistically than results in the biological and social ones. So why do they get more negative results?

As I'd pointed out earlier, could the difference be in how easy it is to give a negative result a positive spin?

[Qingdai]

I disagree in the actual particulars, hard science is also subject to the "great men" problem of prestige of the researcher overwhelming the actual theory and data. The most famous example I can think of is the US HIV test researcher that cribbed his results from the French researchers. People who worked under him said they were afraid to criticize his work because he was responsible for grants to do their work, and he would threaten to make it so they couldn't get anymore funding.

Here's the article from SPY magazine (which is on some website I doubt is legit, but whatever). http://www.virusmyth.com/aids/hiv/srlabrat.htm

[lisarea]

I think it's even weird to call whole fields of knowledge "sciences." Science is just a methodology or category of methodologies. And the narrower and more physical a particular area of study is, the more likely it's going to be replicable and explicable using scientific method. It might seem really nitpicky, but I think it's an important distinction. Biology and physics aren't science any more than poetry or philosophy are. They're just easier to study by means of science.

The more complex and intertwined a particular type of phenomenon is, the 'softer' the science. Not because it's any less real, but because it's less isolatable. You can't accurately contain and study sociology or economics or psychology in a lab or using predictive models the way you often can with the 'harder' sciences like physics or chemistry (to a point, anyhows). Think how weird that model is, though. Like there are sciences, which are factual, objective, and explicable. Then, what's everything else? Is everything else in the world imaginary? Maybe some sort of wizardry or something?

I don't really think it's a super-common issue among actual scientists, at least not ones I know, but there does seem to be a popular conception that hard vs. soft 'sciences' exist on a spectrum of Objectively True to Totally Fictional Made Up Bullshit, based solely on how easily replicable an area of inquiry is.

In conclusion, scientists are, effectively, just extremely weak, limited purpose leprechauns.

[fragment]

Originally Posted by lisarea

I think it's even weird to call whole fields of knowledge "sciences." Science is just a methodology or category of methodologies. And the narrower and more physical a particular area of study is, the more likely it's going to be replicable and explicable using scientific method. It might seem really nitpicky, but I think it's an important distinction. Biology and physics aren't science any more than poetry or philosophy are. They're just easier to study by means of science.

Jumping off this tangent, I'm going to disagree with this. First, I don't think science can be really constrained to just a category of methodolgies - in fact I think it would be difficult to generalise the sciences in any way much more specific than something like "using repeated observation and logic to inform conceptual systems that describe stuff". The canonical "scientific method" descriptions that you can read on the internet about developing and testing hypotheses don't actually resemble a decent amount of what has gone on in the sciences. Taxonomy, for example, is pretty foundational to a chunk of biology, but it started off fairly free of theory and continued to run somewhat independently of it for some time. It's only more recently that other biological knowledge has been able to be fed back into more hypothesis-driven approaches. This doesn't make most of the history of taxonomy not-science. Scientists develop and adapt methodologies suited to the tasks they set themselves.

Secondly, I don't agree that disciplines can be defined only by their subject matter, without considering the approach taken to studying the subject matter, which is what you seem to be saying. At the very least, throwing everything everyone has ever thought and communicated about living stuff under the term "biology" leaves a gap in the language. It is useful to be able to distinguish the more systematic observation-based stuff from the other stuff, so we'd need to start talking about bio-scienticianism or something. Not to say the distinction is in any way hard and fast, but it can be useful to have labels for different parts of continua.

Actually, looking back at what I just wrote, the two points are kinda related. Adpating methodologies to suit the tasks is the flip side of the more scientifical approaches to particular areas of study. I guess I think it's problematic to try to separate the method from the subject when they've been developing together for centuries.

Hope this ramble makes some kind of sense. Should probably consider it some more, but I'm in a splurge mood on this right now.

[ceptimus]

Originally Posted by fragment

I think it would be difficult to generalise the sciences in any way much more specific than something like "using repeated observation and logic to inform conceptual systems that describe stuff".

I like, "Science is a logical search for better explanations."

[Ari]

[lisarea]

Originally Posted by fragment

Jumping off this tangent, I'm going to disagree with this. First, I don't think science can be really constrained to just a category of methodolgies - in fact I think it would be difficult to generalise the sciences in any way much more specific than something like "using repeated observation and logic to inform conceptual systems that describe stuff". The canonical "scientific method" descriptions that you can read on the internet about developing and testing hypotheses don't actually resemble a decent amount of what has gone on in the sciences. Taxonomy, for example, is pretty foundational to a chunk of biology, but it started off fairly free of theory and continued to run somewhat independently of it for some time. It's only more recently that other biological knowledge has been able to be fed back into more hypothesis-driven approaches. This doesn't make most of the history of taxonomy not-science. Scientists develop and adapt methodologies suited to the tasks they set themselves.

The funny thing is that I was actually thinking of taxonomies when I said that. Pretty much any academic field has its own taxonomies, and any (descriptive) taxonomy is subject to definitional fine-tuning and testing.

So identifying something as a pastoral elegy, a verbal phrase, an electron, a polymer, an animal, a hormone, an interpretive dance, a synthetic flavoring, a bird or a reptile is always dependent on some taxonomy or another, whether the taxonomy is prescriptive or descriptive.

Secondly, I don't agree that disciplines can be defined only by their subject matter, without considering the approach taken to studying the subject matter, which is what you seem to be saying.

NOOOOO! I'm trying to distinguish between disciplines vs. phenomena, or something like that. So phenomena would be just things the way they are, and a discipline would be a particular type of study of a defined subset of phenomena. Physics itself isn't 'science.' The study of physics (usually) is.

At the very least, throwing everything everyone has ever thought and communicated about living stuff under the term "biology" leaves a gap in the language. It is useful to be able to distinguish the more systematic observation-based stuff from the other stuff, so we'd need to start talking about bio-scienticianism or something. Not to say the distinction is in any way hard and fast, but it can be useful to have labels for different parts of continua.

That's actually pretty close to what I was saying. Not that people need to use some weird precise terminology, really, but just maybe that calling whole areas of knowledge 'science' contributes to a lot of popular misconceptions about what science is and what it is and isn't applicable to, and that this weird popular science fetishism is actually counterproductive in a lot of ways.

THE TAXONOMY OF SCIENCE HA HA. Seriously, I just realized that we are now arguing about the taxonomy of science.

Actually, looking back at what I just wrote, the two points are kinda related. Adpating methodologies to suit the tasks is the flip side of the more scientifical approaches to particular areas of study. I guess I think it's problematic to try to separate the method from the subject when they've been developing together for centuries.

I think it's problematic not to, not because I think it's necessarily inaccurate as a characterization, but because, like I said, just the word 'science' seems to create a sense of false authority, and because laying some kind of science vs. not science framework on top of entire fields of knowledge contributes to some too broadly dismissive attitudes toward critics.

In conclusion, metanarratives are witchcraft.

[Dingfod]

[Dingfod]

For science, bitches.

[Sauron]

Originally Posted by Dragar

It's particularly pronounced in the pharmacology industry (where I currently work).

Oh, riiight.



"Dragar, we have to cook!"