Description of the Scientific Process: Designing Your Experiment
Designing Your Experiment
To test your hypothesis, you need to come up with an experiment. This is when you get to decide what work you will actually do. You need to start by taking your hypothesis and the independent and dependent variables that you think that you can test and figuring out how you are going to test them.
|
Materials and methods sections of papers are really boring, and even scientists tend to skim over them. We do use them to figure out what we’re going to do, though. I may want to use someone else’s methodology on my system. Here is some advice for how to digest them:
|
In a traditional controlled experiment, you will need what is called the control sample and the treatment sample, or experimental sample, (maybe samples.) Let’s start with the word sample. It refers to a set of specimens, measurements, etc. that you will group together for repetition. The number of those things is called the sample size, and we will address that soon. Your control sample is the sample that is under somewhat normal conditions. The term “normal” is loaded, but the important thing is that you know what “normal” means for your experiment. Scientists argue and fuss about how valid controls are all the time, but when you feel satisfied that you have a set of normal conditions, then you should proceed. The treatment samples are those with conditions that somehow vary from your control in defined ways. For example, you water your control dandelions with 100 mL of water every day, you water your drought treatment dandelions with 50 mL of water every third day, and you keep your waterlogged dandelions semi-submerged. Here, you are varying water exposure relative to your control (you are also varying water volume and watering frequency, so you may have to adjust for that.)
It’s important that you consider what might be realistic for your treatments. What is a reasonable range of the variable? I’d bet that those waterlogged dandelions aren’t going to get very far, and it might not be a good level of exposure, since they will likely die within a few days. You have to consider this issue for your own experiment. Here is where you can dive into the literature. Scientific papers have to include their materials and methods, that is what things they used and how they used them. Check those out.
How do you find out what treatment extremes you could use? First, you should look at what others have
|
John’s anecdote: I once performed a study testing the rate of hybridization of shortleaf pine and loblolly pine in a shortleaf pine forest adjacent to a loblolly pine plantation. As expected, there were many hybrids in the sample site across a road from the plantation, but all of the other sites that were staggered away from the site had few hybrids. As it turned out, my results were not great, and I could only make weak claims in my paper. The lesson is that you should have a good idea of what scale your treatments act over. If my sample sites were all closer to the plantation, I may have been able to detect the distance that loblolly pine pollen was fertilizing the shortleaf pine cones. Andrew’s note: I LOVE pilot experiments! You can be wacky with them, really testing out extremes. If you don’t you may find yourself in the position of my first grad student who did lots of experiments being too small to get really interesting results. Of course, what other scientists have already published can be a pilot experiment for you as well, to help you understand the responsiveness of the study system. |
done in similar experiments. Yes, that means you have to look at the literature again, but if you used any scientific literature to get to the point you are now, then that same literature will probably have what you need. You can also ask an expert. Any scientist worth his or her salt would be delighted to correspond with you about setting up an experiment, and really, the biggest danger here is that they make your scope of study way too big to handle. Remember that what they tell you is advice and not a dictate. Finally, you can try a pilot experiment.
Pilot experiments are miniature trials that you can use to test your methods. (Professionals sometimes use them to justify grant requests.) If you have the time and resources, then you can try running your experiment with a lot of treatments to work the kinks out. In addition to informing you about extremes of treatment, pilot experiments can help you work out bugs in an apparatus you built or show you that you need to take extra care in some step.
Consider our dandelion example. Let’s say that you have already decided to put your plants into 2” × 2” × 3” pots. Now, you need to know how much water you should add to get reasonable plant growth. You will just want a range of watering conditions repeated two or three times to get a feel for their effects. Those conditions might be 2 mL/day, 5 mL/day, 25 mL/day, 50 mL/day, 100 mL/day, and 250 mL/day. Notice that these conditions do not scale in a linear way. That is, they are not in increments of 10 mL or something. This way, you can capture a better representation of varying growth conditions with only a few samples. You might also only grow your plants for 2 weeks in the pilot experiment instead of the full 4 weeks you would use in a planned experiment.
You will also need to visualize how your experiments will work and where they might introduce accidental experimental bias. Consider a flat of daisies in an experiment. It’s placed on a windowsill on the east facing side of a house. In the morning, the flat gets sun, so all seems well. You then notice that the easternmost daisies are growing faster, and you realize that they are shading out the other daisies. More than likely, this experiment has been ruined by an accidental bias introduced in the design. Even if the flat of daisies was placed under a sunlamp so that all of the plants got even light, you would notice that the plants around the edges are growing better, since they are less crowded. We have to ignore our edge plants and only collect data for the others. If you had designed the experiment so that all of your treatments were in neat rows, and your control plants had all been on the edge, you just lost your control samples. Ouch!
Besides edge effects, there are a great many things that can add bias to your results. If you need to test multiple samples with a device like a probe, each exposure could change the probe’s sensitivity. Again, if you measured all of your control samples first, and then measured all of your treatment samples, you might introduce a bias into your experiment through the order of sampling. Social scientists frequently use surveys and have to be very mindful of the wording of the questions they use. A given political affiliation or racial group might respond very differently from another just because some word with a loaded history was included in the question.
What do you do about this? First, randomize. Randomize the order of the plants in the flat or the order in time of the samples you measure with a probe. You can do this in a spreadsheet[1] or with a deck of playing cards. Second, include dummy samples. On your flat of daisies, just grow daisies around the edge that are not considered to be treatment or control plants. You can then ignore them in the analysis. If you are doing some sort of chemical assay on them, then they can act as practice specimens later. Be creative!
Sample size is the bane of scientific studies. We don’t know how many times we’ve seen a scientist say that their sample size was too small. They couldn’t end up making statistical claims that they wanted to make about their studies, because they couldn’t find enough smallmouth bass, or the small proportion of sample locations with rose rocks is too small to make meaningful conclusions. Sample size issues can be devilishly frustrating to deal with, but you can overcome them.
First, how large should your sample size be? It depends on how much variation you expect to see in your dependent variable and on how many samples you think you will have to discard as the experiment progresses. If you don’t know those things, you can again ask an expert, read the literature, or attempt a pilot experiment. When asked how large a sample size should be, most scientists will tell you that it should be as large as possible, and that’s true. There are trade-offs, though. Given limited resources, you can have more treatments with smaller sample sizes or fewer treatments with larger ones. If you can manage 100 specimens, you could make a control and a treatment with samples sizes of 50, or you could make a control and 3 treatments with samples sizes of 25. Which should you do?
If you have no prior information and just want to proceed with the experiment, going with 40 individuals is not a terrible rule of thumb[2]. You can often get good statistical significance with 30 individuals, but you must assume that you will lose some during the experimental process. If you can’t get that many, do not despair. If the differences among treatments and controls are large enough, you may still get great results with only 4 samples, though that is very risky.
Finally, you’ve designed your experiment. Now, ask yourself once more, how will this experiment test my hypothesis? If you aren’t sure, either redesign your experiment, or reword your hypothesis. Both approaches can be valid. Sometimes, you lose your way somewhere in the experimental design process, but sometimes, your original hypothesis was just too hard to test with the experimental resources you have. Do what you feel is best. It’s your project.
[1] See our Excel Tutorial.
[2] The website http://edis.ifas.ufl.edu/pd006 has a decent guide to finding out how big your sample should be.