Description of the Scientific Process: Interpretting Your Results
Interpreting Results
You have done your statistical analysis, and you can show that two things are likely related (or not.) You just did the mechanical part, the boring part, the thing that does not yet have the inspiration of human spirit. Here you are now, results in hand. What in the world do you do now? You have to interpret what you have found. You have to take the results and tell yourself and others what it means, what the implications are, and how this changes our understanding of the world.
Statistical Significance
If you performed a test that told you that two or more means were statistically different, then you need to explain why you think that they are. Were you expecting them to be different? Were you expecting them to be the same? How different are they[1]? If your tests did not show statistically significant differences, then what do you make of them?
These questions can dog professional scientists, and honestly, there is no hard and fast rule that you can use. If you have a p-value of 0.049 (just under the 0.05 threshold), then you have shown statistically significant differences according to conventions, but you have barely done so. When you interpret that difference, you need to acknowledge that it is a weak one. Perhaps, more replications of your experiment will clear this problem up, but you may not have the time or resources to do more experimental work.
The bottom line is that you can make a claim that your mean or proportion is different with statistical significance. You will have to interpret what that means or implies. Be sure to relate what you found with what others have found in similar systems or with other measures in your system. If you found that the selenite crystals in a local park have greater specific gravity than is normal, you will need to speculate and perhaps do follow-up chemical tests to determine why it is so different.
Correlations
If you did correlation tests, then you will need to interpret what those correlations mean and suggest what other tests need to be considered. In general, correlations show that two measurable things vary with each other or against each other (or not at all.) You have all heard the phrase, “Correlation isn’t the same as causation,” or something to that effect. This statement is true, but correlation can imply causation.
When two things (A and B) correlate with each other, we can come away with four different conclusions: A causes B; B causes A; something other than A or B simultaneously causes both of them; or A and B correlate due to coincidence. Scientists are not interested in the last conclusion, except that they need to rule it out. Let’s start with the first two conclusions, A causes B and B causes A. You will have to use reason and common sense, along with whatever evidence you can find, to tease out which is true[2].
|
Figure 12: Recent Oklahoma earthquake frequency. |
Consider Oklahoma’s recent spate of earthquakes. Starting in 2009, the frequency of earthquakes with a magnitude of 3.0 or higher has gone up dramatically (Figure 12). As many have observed, this date is the approximate date that waste injection well use in fossil fuel extraction also increased[3]. The two things correlate. Let’s compare A causes B to B causes A. Would it be reasonable to say that injection wells could cause earthquakes? Waste injection wells push liquid deep underground at high pressure, which geologists surmise could alter the state of underground faults, thereby initiating earthquakes. Could we suppose that earthquakes cause injection well use? That would be absurd, of course. No one feels the ground shake and then decide to go and drill because of that.
Could something else cause both the recent increase in earthquakes and increased waste well injection? That seems almost as absurd as earthquakes causing drilling, so we should rule that out. The final possibility is that these two things are mere coincidence. That is not as absurd, and it should be seriously considered. The joint study performed by the United States Geological Survey and the Oklahoma Geological Survey concluded that it was unlikely to be coincidence after more rigorous statistical tests and the use of information from other locations, but you should always consider the possibility that correlations are coincidence.
Frequently, two observed trends will correlate, because some other factor is driving them. Sour foods can lead to tooth decay, if you’re not careful. How does sour hurt your teeth? Well, you taste acidic foods as being sour, and acids can wear away at tooth enamel. Both the sour flavor and the tooth decay are caused by acidity. Let’s try another. Sometimes, silly hats and fireworks co-occur[4]. While silly hats could lead to fireworks, it might just be that it’s New Year’s Eve. People like to wear silly hats on the holiday, and people like to display fireworks on certain ones, so both silly hat wearing and fireworks are caused by people’s reaction to New Years Day. Okay, one more. Alligators and cypress trees correlate. Does one cause the other? No, they both prefer swamps. When you look at your data, make sure that you find instances of these things.
|
Figure 13: Correlation sometimes has nothing to do with causation. Say what you will about Microsoft Internet Explorer, but it would be unreasonable to assume that it caused murders or that few murders caused people to stop using the software. Source: Gizmodo.com |
Now and then, correlation is total happenstance. Two things have positive or negative correlations, and it has absolutely nothing to do with them relating to each other. Sometimes, it’s really easy to tell when that is true (Figure 13)[5]. As with finding common causes for correlating variables, uncovering when they are simply happenstance is also an exercise in common sense and reason. If you cannot explain why two things are related, then you should assume that they are not. In fact, designing experiments to test whether two things are actually related and possibly why is a great enterprise in-and-of-itself.
Context
Science is not conducted in a vacuum. Your results relate to someone else’s work, and you worked on something that interacts with something else. You need to put your results in the context of those things. Here, your research of the systems and methods you used in your experiment will pay further dividends. What did other authors say about their systems? What did they say about the results they got? Borrow[6] those ideas and make them your own.
There is not any one-size-fits-all advice for finding the context of your work. It is generally a matter of knowing why you researched, hypothesized, experimented, and analyzed. Here are some tidbits that we can provide you, though:
- If another experimenter used the same methods that you did on another similar system, and you share results, what did he or she conclude? Do you think that you can make similar claims?
- What have other researchers discovered as unique about the system that you are working on? Could they be relevant? Why or why not[7]?
- How will this work impact what we understand about other systems related to your system?
- Is there an impact on people[8]?
- Does anything make you wonder what is really happening? You could have found something unusual that you can encourage others to look into.
When science is done, you test a hypothesis to challenge a theory. A theory’s resistance to those challenges is what makes it strong. Sometimes, when you have done your work, and you found no differences between treatments like you thought you should, you are tempted to despair. The lack of differences is called “negative results,” and even the wording implies failure. You have to put that into context, too, however. Do those negative results support or challenge the theory? Maybe they imply that the theory is not as strong as others think it to be. Depending on how well supported that theory is, you may have shown that it does not apply to your system, to a class of systems, or to anything at all. After all, if only one experiment in all of history has supported the theory, you may have just cast enough doubt on it to disprove it. On the other hand, if it has been supported by millions of prior experiments and has been shown to explain phenomena time and time again, it’s probably your experiment.
Negative results are difficult to make conclusions from, but they are an underappreciated part of science. It’s hard to get them published in the more prestigious journals, and many scientists don’t bother submitting papers about them to even the least of journals[9]. That’s a shame, really. If nothing else, it prevents other scientists from trying the same experiment when they could be doing something else. As a high school student, you should be proud of your results, even if they are considered negative. You might have discovered something new anyway.
[1] Since you are working on a high school student’s budget, if your results are significantly different, then they are probably meaningfully different. In studies with many replications and huge sample sizes, statistical differences can be found that are so minor in real world terms that they don’t really imply any worthwhile conclusion. This is most common in gigantic medical studies.
[2] You know, both could be simultaneously true. This would be an instance of feedback. Consider one plant shading another plant. They both need sunlight to grow, and the bigger plant is getting more of it. Because of that, it can grow faster, continuing to shade the smaller shaded plant. Thus, the shaded plant’s small size is preventing it from getting sunlight (small size is causing the plant to be shaded by larger plants), and the shade is stunting growth (shade is causing the small size.) Plants have means of escaping these situations, of course, but this might be true for some plants some of the time.
[3] Some modern fossil fuel extraction has a lot of waste water that needs to be disposed of. Drillers simply inject it deep underground to prevent it from harming the environment.
[4] Scientists often use the words “occur” and “co-occur” in ways that are kind of funny to non-scientists. You might say that wolves live in the forest, but a scientist would write in a paper that they occur there. If two things happen together or live together, we will say that they co-occur. This often implies correlation.
[5] Here are lots of spurious correlations that you can waste your time justifying.
[6] We say borrow and make your own, and that’s important for two reasons. If you want to be good at what you are doing, you have to internalize the ideas you are using and coming up with. This is a matter of practice and talking to other people. When you have aspects of your work internalized, you can really shine. Also, it will help you to avoid plagiarism when you write. Take note of where you got ideas, but also having them as your own ideas will help you to write original text, and that is the essence of avoiding plagiarism.
[7] Generally, if something isn’t relevant, you don’t need to report that it is. There are cases in which people have assumed something to be relevant, and research showed that it was not. Maybe you found something like that. If so, you’ll want to report it.
[8] Not all science is immediately relevant to human welfare, and that’s okay. If you can make a claim that yours is, though, it tends to get folks’ attention. As a dispassionate scientist, you shouldn’t care about that, but as a dispassionate scientist who needs to get a grant proposal accepted in order to get a paycheck, you probably should care. Just saying.
[9] Peer-reviewed scientific journals represent the most important way that scientists communicate their results. They are basically magazines (more online entities these days) that contain articles describing what scientists did and what they found. Each article has been examined by other experts in the field and vetted for its rigor and importance. Very prestigious journals such as Nature, Science, and Proceedings of the National Academy of Sciences have a very high bar for what they consider important. Every field has its own set of journals ranging from very prestigious to those that accept any paper, as long as the science was rigorous enough.