[LinearPhilosopher]

Somewhat related to a thread i'd made ages ago

http://climatecommun...vember-2016.pdf

According to this study over 2/3 of americains are down for taking more action to reduce global warming.

How much stock can I put in this survey? My gut is telling me a sample of 1200 compared to a total population 250m isn't exactly saying something.

And then there's this interesting video by veritasium that also says even with a 95% CI level (which this study has) it doesn't neccesarily mean we've found a true relationship.


So the open question is this? When can we accept a study as having identified a true relationship. What factors does one look for in order to determine its validity.

This post has been edited by LinearPhilosopher: 20 December 2016 - 08:17 PM

[Aptorian]

Quote

What factors does one look for in order to determine its validity.

I feel like this is kind of an impossible question.

A lot of what this guy is talking about is going over my head, but in terms of validity, proof can be very different between fields of science. If you're recreating something that is super "positivistic" in nature, like how heavy is a litre of water. Then the study is easily recreatable and you can try to falsify the test all you want until you trust the science behind it.

However if we're talking about the sciences in humanities, particularly science based on the answers of test subjects, then things become really fucking questionable. Not in terms of the "truthiness" of the science, but rather in how do you scientifically analyze a person's answers and the reasons behind why they answered the way they did? There the individual's answers might be less meaningful than what a thousand people said.

Furthermore I feel like, when you start to get really pedantic and esoteric in science, you reach a peak at some point where only a handful of people in the world are really equipped to handle the topic in any way that is constructive and understandable from an outside perspective. I mean you can try to digest the data and feed it to pople in a paper, but how many people are actually going to understand it, let alone have the time to go over it?

This post has been edited by Apt: 20 December 2016 - 09:44 PM

[Silencer]

I feel like the correct answer is: never. No single study can ever be considered to be true and verified on its own, regardless of the size of the sample or its CI or the way it was conducted. It would need to be repeated on different samples by a different team to be able to say with some confidence that it is drawing an appropriate conclusion. That’s just the minimum requirement, imo.

[Cause]

P scores are notoriously unreliable and most scientists know this. They are also the culprit of something called p-hacking. Its when a scientist changes his parameters, or data group until he gets the required p score. The problem is the way science is run these days. The pressure to publish is enormous and the controls to screen science are eroding. The p-score become a metric by which science was judged to be publishable or not. It missed the fact that it does not determine truth.

In the example given the p score was 0.01 so only 1 in a hundred experiments would give that result by chance. If we make no distinction between the neutral, negative and erotic images the average would go back to insignificance. So the first question is why was a distinction made? Did the scientist have any reason to believe that erotic images would enhance our ability to see the future? No its p-hacking. He introduced a meaningless variable to get a result. Further if we repeat the experiment 100 times we can expect the result to occur again by chance at least once. Its a significantly low probability. However if a person could see the future you would probably expect the person to be right closer to 100% of the time. That is the difference between statistically significant and experimentally meaningful. There is a growing awareness and I know of at least one journal that will no longer consider p-scores by which to judge papers as publishable or not.

A p score becomes more meaningful for a poll. If a 1000 people are questioned on global warnming and the likelihood of that polls outcome is say 1% by chance then yes its pretty trustworthy. With the caveat, if the poll is properly designed! Asking a 1000 baseball fans from new York if they support the new york Yankees baseball team will probably give you a very good idea of what percentage of baseball fans in new york support the home team. Asking 1200 Americans their thoughts on climate change will not be a reliable indicator of a country of 400 million people. Different levels in education, ethnic, cultural backgrounds even which state they are from among a slew of other factors such as their job and which newspaper they read could alter their answers. That poll is far to general.

http://fivethirtyeig...t-broken/#part1

[Sir Thursday]

As others have said, the validity of studies varies considerably from field to field. One thing to watch for is the problem summarized by this comic:



If you can see that the designers of the study have looked for multiple correlations in their dataset, then they need to have corrected for the above phenomenon, otherwise their conclusions aren't worth it. There are statistical techniques designed to do this - one of the most common being the Bonferroni correction. I will quite often look to see if the study indicates in their methodology that they've done something like this, and if they haven't then I'll probably disregard the results (at least until I see more studies that replicate their conclusions).

For polls like the one you've found, There's an issue of sample size in the subdivisions. The authors of the report have decided to breakdown their results by political orientation (presumably because they believe there is a correlation between political orientation and opinion on climate change). However, in each of the sub-categories there's a much smaller sample size, so the information about those sub-categories is more suspect. That in turn weakens any conclusion drawn from the overall dataset - it is acknowledged by the creators of the report that their sample is biased, so they need more data to be able to arrive at a legitimate sampling than they would have otherwise. An aggregation of similar polls would probably provide a better overall picture of the state of public opinion.

This is an extension of a general principle - when you're trying to do statistical analysis, you will almost always end up mapping your data onto a model of how you believe your measurable is distributed (you can't calculate things like p-values otherwise). Most of the time people pick a normal distribution as their basis, and there is some mathematical backing for defaulting to that (see the Central Limit Theorem). But when you map onto a model, you're making a lot of assumptions. Many times those assumptions aren't quite right, introducing errors. You see this in most fields, but it has larger impacts in certain places than others (I'm looking at you, Economics!). Checking the fine print of the statistical techniques you're using is difficult, but it's also very important if you want to actually use them properly. Sadly most people don't bother...and that's why we have a reproducibility crisis in science and a lack of trust in scientific findings generally.

ST

[Aptorian]

Wow. A wild Sir Thursday post observed in the wild. How often does a man get to see that in his lifetime?

[Gorefest]

You probably need the right bait to entice him out of his lair.

It's an interesting topic, though. Working in biomedical sciences myself, it is staggering to see the amount of staff, students and scientific papers that display a worrying lack of understanding of fundamental statistics. As mentioned before, often a statistical test is chosen not on the basis of the experimental design, but because it is the only one that generates a 'significant' (read p > 0.05) result. A fair few of our students don't seem able to distinguish a one-way ANOVA from a student T-test, never bother to check their distribution, or just go non-parametric instead of increasing their n-numbers to obtain the magic p-value that allows them to put a little asterisk above their bars. A shocking amount of journal articles get away with dodgy stats either because the reviewers don't bother to check or because the available info is confusing or too constrictive. The pressure to publish is one of the main drivers for this increasingly shaky output. Often the mindset of modern research seems to be: everthing is open source, so at some point any flaws in statistical validity will be picked up on through failure to reproduce or by conflicting data, so it'll iron out over time. If only. A large part of the Cochrane review project is centered around highlighting flaws in statistical analysis in research papers.

[LinearPhilosopher]

Sir Thursday, on 11 January 2017 - 01:09 PM, said:

As others have said, the validity of studies varies considerably from field to field. One thing to watch for is the problem summarized by this comic:



If you can see that the designers of the study have looked for multiple correlations in their dataset, then they need to have corrected for the above phenomenon, otherwise their conclusions aren't worth it. There are statistical techniques designed to do this - one of the most common being the Bonferroni correction. I will quite often look to see if the study indicates in their methodology that they've done something like this, and if they haven't then I'll probably disregard the results (at least until I see more studies that replicate their conclusions).

For polls like the one you've found, There's an issue of sample size in the subdivisions. The authors of the report have decided to breakdown their results by political orientation (presumably because they believe there is a correlation between political orientation and opinion on climate change). However, in each of the sub-categories there's a much smaller sample size, so the information about those sub-categories is more suspect. That in turn weakens any conclusion drawn from the overall dataset - it is acknowledged by the creators of the report that their sample is biased, so they need more data to be able to arrive at a legitimate sampling than they would have otherwise. An aggregation of similar polls would probably provide a better overall picture of the state of public opinion.

This is an extension of a general principle - when you're trying to do statistical analysis, you will almost always end up mapping your data onto a model of how you believe your measurable is distributed (you can't calculate things like p-values otherwise). Most of the time people pick a normal distribution as their basis, and there is some mathematical backing for defaulting to that (see the Central Limit Theorem). But when you map onto a model, you're making a lot of assumptions. Many times those assumptions aren't quite right, introducing errors. You see this in most fields, but it has larger impacts in certain places than others (I'm looking at you, Economics!). Checking the fine print of the statistical techniques you're using is difficult, but it's also very important if you want to actually use them properly. Sadly most people don't bother...and that's why we have a reproducibility crisis in science and a lack of trust in scientific findings generally.

ST

This is absolutely fantastic material! Thanks man