Arguments in Context Unit VII Scientific Reasoning Chapter 21 Significant Correlations and Controlled Studies

CHAPTER 21 Significant Correlations and Controlled Studies SECTION I INTRODUCTION Science is in the business of offering specifically , it is in the business of trying to cover the causal relations in nature . Identifying a correlation can be one step in this process , and in the last chapter we looked at correlations and how they relate to causal thinking more broadly . As we saw , although a correlation can be a good place to start , by itself a correlation drawing a causal conclusion . After all , there are many possible explanations for the existence of a correlation . Moreover , we briefly talked about some ways of ruling out alternative explanations , and thereby identifying correlations that are suggestive of a causal relation . In this chapter we will look at some more sophisticated techniques for identifying significant often used in science and social science . We will look at two of the most common the observational study and the randomized trolled study . The following is not intended to prepare you to conduct your own a study requires expertise and resources ( and time ! most of us do not have . Rather , the aim is to show the reasoning behind these studies , and to give us the tools and vocabulary to be more informed and critical readers of scientific studies . SECTION OBSERVATIONAL AND EXPERIMENTAL STUDIES Suppose that you have spotted a pattern . Suppose that among your friends you have noticed that those who regularly eat breakfast have higher grade point averages ( than those who do not . This tion leads you to think that maybe you ought to eat breakfast , and maybe other students should too . Put in our terminology among your friends , regularly eating breakfast is correlated with GPA , and this tion may be indicative ofa causal relation between the two . So , is that this correlation establish that eating breakfast is causally relevant to GPA ?

Well no . To conclude that the eating breakfast is causally relevant to GPA is to risk a hasty explanation . Not only is this correlation based on a relatively small sample ( your friends ) but there are many possible explanations for any correlation , and we need to investigate the plausibility and adequacy of these alternatives before identifying a particular explanation as the best . Nevertheless , observing this pattern gives you a place to start . In order to determine whether this pattern is suggestive of a causal relation , you would need to scale up your inquiry in a way that would allow you to ( i ) find a genuine correlation if there is one , and ( ii ) rule out , or at least cast doubt upon , some of the ble explanations for the correlation . We will discuss two techniques a researcher might use . The first is called an observational study . To conduct an observational study in this case , a researcher would 237

238 THADDEUS systematically identify and examine a group of students to see whether there is a significant correlation between the two factors in question . Alternatively , a researcher might choose to conduct an experimental study . Unlike an observational study , in an experimental study researchers go beyond mere observation to intervene by systematically exposing subjects to the suspected cause . In this case , the researcher would ( among other things ) identify a specific group of students , give them breakfast ( or not ) and see what pens . How do these techniques take up or cast doubt upon alternative explanations for the correlations ?

Let us take a closer look . SECTION OBSERVATIONAL STUDIES There are different kinds of observational study . A retrospective observational study begins with the effect you are interested in and looks backwards in time to try to isolate a cause . To conduct a retrospective study on the hypothesis , you start with students and then look to see whether they eat breakfast or not . A prospective observational study , on the other hand , begins with a suspected cause and follows it forward in time to see if the effect follows . So , in the case at hand you start by looking at fast eating patterns and look for patterns in . There are other differences between prospective and retrospective studies ( retrospective studies tend to be much easier to conduct ) but in discussing the reasoning behind observational studies we will focus on a prospective study , though our conclusions will largely apply to retrospective studies as well . Let us return to the case at hand . In order to dig deeper into your tentative view that breakfast contributes to GPA , you decided to conduct a prospective observational study . How will this work ?

You have to start by identifying a group of subjects . These people will constitute your sample . The larger your sample is , the less likely your results will be mere coincidence . So , you want to work with the largest sample that is feasible given the limitations of time , resources , and so on . How should you choose the people in your sample ?

Here is a strategy that might seem intuitive find as many students as you can who regularly eat breakfast . Once you got this group , you could then simply look at their . This is too quick , ever . This strategy for identifying a sample wo work since , as we seen , correlations are comparative claims . If we are looking to see whether eating breakfast is significantly correlated with GPA , we need to compare the percentage of people who eat breakfast and have high to the percentage of people who do not eat breakfast who have high . That is , we need to look at two groups with respect to the experimental group that exhibits the suspected cause and the control group that does not . Let call the experimental and control groups in this case the Breakfast Group and the Group . Say you have identified a reasonably large sample of students , half of which tend to eat breakfast , half of which do not . This makes it more likely you find a genuine correlation ifthere is one , and mitigates against the possibility of coincidence . Are we ready to look at ?

Not yet . Remember , we want to know whether breakfast is a causal contributor to GPA . Finding a correlation between the two will not , all by itself , suggest this , since there are many possible explanations for any correlation . Recall the four types of explanation from Chapter 20 . It could be that breakfast contributes to GPA as you suspect ( Type ) but it could also be the other way around ( Type ) Moreover , there could be some underlying factor that explains the tion ( Type ) or it could be coincidence ( Type ) The of the study mitigates the possibility of coincidence , and background knowledge casts doubt on Type , since it seems unlikely that a GPA could influence breakfast habits . This leaves Type an underlying cause . Could there be some underlying factor that accounts for a person breakfast habits and their GPA ?

Sure . Of particular importance is the possibility ofone or more confounding factors ( sometimes called CORRELATIONS AND CONTROLLED STUDIES 239 factors ) A confounding factor for a particular study is a factor which is ( i ) correlated with the cause under investigation and ( ii ) a partial cause of the effect . ron by BY Here is an example . Suppose you were interested in the possible health benefits of the spice , saffron . To investigate this question , you might examine people who regularly eat saffron to see if they are healthier than those who never ( or rarely ) do . There are confounding factors here , however . Saffron is an extremely expensive spice as a consequence , people who regularly consume saffron also tend to be relatively wealthy . Thus , there is a correlation between wealth and the suspected cause ( saffron consumption ) In addition , there is also a correlation between wealth and the suspected effect , since wealthy people have access to regular preventative health care at a higher rate than people do . Thus , if there is a correlation between saffron consumption and health , this correlation may be explained by because saffron has any health benefits . That is , wealth is a confounding factor in the effort to determine whether saffron has health benefits . Let return to the question of whether eating breakfast is causally relevant to GPA . How could we conduct our study to avoid confounding factors ?

In brief , we will need to ( i ) think about the kind of factors that may be causally relevant to GPA , and ( ii ) make sure we take this into account in deciding how to construct the Breakfast and groups . Given this , let us consider factors that might influence a person GPA . One prominent determinant of a student GPA , for example , is the difficulty of their course schedule . In some classes it is more difficult to get an A than in others . To simplify , let us pretend that there is only one class like Organic Chemistry . In addition , imagine that 10 of the students in the Breakfast Group , but none of the students in the Group , are enrolled in Organic Chemistry . In this case , we would have a confounding factor , since being enrolled in Organic Chemistry is causally relevant to a dent GPA and is correlated with the suspected cause ( eating breakfast ) Why does this matter ?

Well , if we do somehow account for students course schedules , our results might not tell us anything definitive about whether eating breakfast is relevant to Is ! Here is why say that we find no difference in GPA between the groups . The natural inference here would be that eating least within this group of no effect on GPA . But if the groups differ in this way , it may be that that eating breakfast really did have an effect , but that this effect has been hidden or masked by the unusually low grades coming from the students enrolled in Organic Chemistry . Put in different terms , if eating breakfast is relevant , then you would know it , since its effects on GPA are mixed together or

240 THADDEUS ROBINSON founded with the effects of course difficulty . In order to determine if eating breakfast contributes to GPA we need to isolate it from other causes , so that we can spot its effects ( if there are any ) In order to prevent this kind of confounding , we need to take account of , or control for , the difficulty of dents courses . In general terms , to control for a particular factor or variable is to ensure that there is no difference with respect to that variable between the two groups you are comparing . This allows you to late the potential effects of the factor you are interested in . To control for the difficulty of students courses in this case would mean making sure that the Breakfast and groups are as similar as possible with respect to the number of students enrolled in Organic Chemistry . Similarly , to control for wealth in the saffron study discussed above would mean looking at two groups of equally wealthy of whom regularly eat saffron , some do not . SECTION LIMITATIONS OF OBSERVATIONAL STUDIES Observational studies , even when they control for known confounding factors and study large samples , do not always give accurate results . Perhaps the chief problem is that observational studies like this can not account for the effects of unknown confounding factors . In other words , an observational study can not rule out all the competing explanations for the phenomena in question . Take the case discussed above suppose we have conducted our study in a way that controls for a variety of relevant factors ( including course difficulty ) and that our original suspicion has been this much larger and more diverse group there is a correlation between eating breakfast and GPA . While this gives us stronger reasons to think that this correlation is not mere coincidence , and further that eating breakfast is causally can not be sure this is the case . The problem is that there may be unknown out there . For example , it may have not occurred to you that getting up early might be causally relevant to a person GPA . Let us assume for a moment that this is right , and that , further , ple who get up early tend to eat breakfast . If we have controlled for getting up early , then the results of our observational study will suggest that eating breakfast is causally relevant when , in reality , it is not . A case illustrating the limitations of observational studies involves hormone replacement . In the 19905 a number of professional observational studies correlated hormone replacement therapy with a decreased risk of heart disease in women . Scientists had a good sense of how this replacement therapy might work , and as a result many doctors recommended this to their patients . However , in the early it became clear that there must be a confounding factor at work in these studies . A large randomized controlled experiment showed that that though hormone replacement therapy could have positive health benefits for some women , it did not prevent heart disease and was actually associated with a number of negative health outcomes . As it turned out , overall health was the confounding factor . On balance , healthy women were more likely to pursue and be prescribed hormone replacement therapy than their unhealthy counterparts . Since a person overall health is causally relevant to the chance heart disease , this was a confounding factor . This shows that observational studies have their limits , and that there are alternative methods for ruling out competing explanations . Let us take a closer look at the kind of study which ultimately revealed the flaw in this observational randomized controlled experiment .

SIGNIFICANT CORRELATIONS AND CONTROLLED STUDIES 241 SECTION RANDOMIZED CONTROLLED EXPERIMENTS In observational studies researchers make careful choices about what populations to observe . While this is also true In an experimental study , a researcher conducting an experimental study goes one step further by manipulating the suspected causal variable . Consider the case discussed above . In an observational version of this study , you divide students into two carefully chosen Breakfast and No Breakfast then check to see whether students who eat breakfast end up having better than those who do not . In contrast , In an experimental study the researcher chooses which students will eat breakfast ( the experimental group ) and which will not ( control ) There are different kinds of experimental study , but here we will focus on a particularly important one the controlled experiment or trial . As one author recently put it , The randomized controlled Is one of the simplest , most powerful , and revolutionary tools of research . In this section we will explain what a randomized trial is , and why it is so powerful . By their very design experimental studies can cast doubt on many of the possible explanations for a . First , like an observational study , an experimental study takes a systematic look at a wide body of information , and In doing so limits the possibility of sheer coincidence ( Type explanations ) Second , given a correlation between and Ys , an experimental study can rule out the possibility that Ys are causing the instead of vice versa ( Type ) since causes precede effects . In an experimental study the researcher duces the suspected causal factor ( to subjects In which the suspected effect ( is absent . So if the effect is subsequently observed in the population you can be sure that this is not a case of Ys causing . As we have seen , observational studies are always subject to the possibility of unknown . ever , the possibility of , known and unknown , is greatly limited by a process of the value of the randomized controlled experiment . But wait . What , exactly , is randomized , and how does this mitigate worries about ?

What makes an experiment like this randomized is that the individuals or members of the experimental group and the control group are chosen randomly from a targeted population . To do so , a researcher uses a procedure that gives each member of the population an equal chance of being chosen . So , in this case you might assign students a number and use a random number generator that is available online to assign individuals to each group . Ok , but how does who is chosen for each group mitigate against ?

Let us return to the unknown confounder considered In the breakfast case for the sake of argument let us assume that it is not eating breakfast , but getting up early that is causally relevant to GPA ( perhaps because they are more alert during morning classes ) Because there are more people who get up early in the fast Group than In the Group your observational study will mistakenly suggest that breakfast is causally relevant to GPA . However , in a randomized controlled experiment this is much less likely . How so ?

By randomly assigning students to either the experimental group or the control group , you would likely tribute early risers evenly into both groups ( at least roughly ) in this way breaks the relation between breakfast eaters and early risers , and allows us to see more clearly whether eating breakfast , in itself , has any effect . The same will go for other unknown after all , arbitrarily splitting subjects into the control and experimental groups will likely distribute other possible evenly ( at least roughly ) between the groups . In other words , by randomly assigning subjects to the mental and control groups the researcher will thereby automatically control for unknown .

242 THADDEUS ROBINSON Start The Day With A Smile by This is the chief benefit of a randomized controlled reduces the chances that an unknown confounder will bias your results . Unfortunately , however , a study of this kind does not completely nate this possibility . It is always with a random assignment of some causally relevant factor will end up accidentally associated with the experimental group . Even so , a randomized trolled experiment is much more likely to give you an accurate picture of the relationship between two ( or more variables ) and these studies are considered the single best way to determine causal relationships . SECTION LIMITATIONS ON EXPERIMENTAL STUDIES The comparison above between different kinds of experimental techniques raises a question if randomized controlled experiments are the best way to identify significant correlations and causal relations , why do researchers ever do any other kind of study ?

First off , randomized controlled experiments are time and can be quite expensive to conduct . Second , in some cases it is not practical or ethical to do a randomized controlled experiment . To illustrate , consider the disease known as Ebola . Ebola is a viral hemorrhagic fever that carries a high risk of death for which there is no known cure . However , there are a number of experimental treatments . The problem is that in order to do a randomized controlled study for one of these treatments , researchers would need to set up a control group that received only a placebo . In this case , the control group would not get the possible benefits of the treatment , which in this case might realistically include survival . This point , in that when a person life is at stake , an experimental study can be simply cal . This point was made in a ( somewhat ) humorous way in an article from the British As with many interventions intended to prevent ill health , the effectiveness of parachutes has not been to rigorous evaluation by using randomized controlled trials . Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data . We think that everyone might benefit if the most radical protagonists of evidence based medicine organized and participated in a randomized placebo controlled , crossover trial of the parachute . Conducting a randomized study of the effectiveness of parachutes using real people would be obviously unethical , and so too when it comes to other potentially or life interventions .

CORRELATIONS AND CONTROLLED STUDIES 243 A third consideration is that simply because randomized controlled studies are the single best way to uncover significant correlations , does not mean that observational studies have no value . Observational studies tend to be more economical in terms of time , effort , and resources than experimental studies , and so are useful ( among other things ) for studying a hypothesis in a preliminary a way of deciding whether it is worth using the resources to conduct a randomized controlled experiment . Moreover , when multiple observational build upon one another and largely concur in their conclusions , then they can give us good reason to endorse causal conclusions . EXERCISES Exercise Set Directions Consider the following possible studies . What confounding factors might you need to control for ?

A study to assess whether being an early reader ( a person who learns to read before age ) causally tributes to academic success . A study to assess whether fluency in a second language improves scores on standardized tests . A study to assess whether taking prevents chronic illness . A study to assess whether listening to classical music while studying makes it easier to remember tion . Exercise Set Comment on the following experiment A vitamin company has developed a new pill intended to prevent strep throat ( a kind of bacterial infection ) The company claims that they pill was given to 2000 people daily for a month , and only of subject came down with strep throat during this time . On the basis of this experiment , the company sells the vitamin as a tive . What kind of study is the following , and what do you make of the study itself ?

A researcher is interested in investigating whether pet ownership contributes to lower blood pressure , The researcher identifies 30 people who have pets and 30 people who do not . She takes everybody blood pressure several times , and then averages the results , It turns out that the pet owners in the researcher sample have lower blood pressure than the owners . The researcher takes this to be good preliminary evidence of a causal connection .

244 THADDEUS In his book Exercised Why Something We Never Evolved to Do is Healthy and Rewarding , Daniel on a study of physical activity and health . He summarizes and evaluates the study as follows . Researchers put on a diverse sample of eight thousand Americans above the age of and then tallied up who died over the next four percent of the sample . Predictably , those who were more sedentary died at faster rates , but these rates were lower in people who rarely sat for long One flaw with this study is that people who are already sick are inherently less able to get up and be What kind of study is commenting on , and what correlation did it find ?

Also , explain criticism of the study . What kind of study is the following , and what do you make of the study itself ?

A researcher wants to study whether taking notes by hand is more effective than typing notes when it comes to remembering them . He identifies 30 students who are willing to participate , and allows them choose whether they be in the experimental group or the control group , and the groups end up even ( 15 people in each ) The researcher then requires students to attend the same lecture . One group takes notes by hand , the other takes notes by typing on their laptops . Students are allowed to study their notes and are given a test over the lecture days later . When the researcher crunches the numbers , it turns out that students in this population who took notes by hand got higher grades than those who took notes on a laptop . Come up with one experiment you love to know the results you had the time , money , and expertise to do so . What would it be , how you set it up , and why ?

Notes . Alejandro and , Murray ( 2007 ) Randomized Controlled Trials Questions , Answers , and Musings ed . MA , Smith , 2003 ) Parachute use to prevent death and major trauma related to gravitational challenge systematic review of controlled trials . 327 ( 7429 ) Daniel ( 2020 ) Exercised Why Something We Never Evolved to Do is Healthy and Rewarding . New York theon Books ,

Unit Summary In this unit we looked at some forms of reasoning commonly used in the sciences . We began by looking at some important components of statistics and their visual representations . In Chapter 20 we turned to an examination of causal language , and a particularly important kind of comparative statistic the correlation . Although correlations can be especially useful starting point for identifying likely relationships , not every correlation captures a causal relation . Before drawing an inference like this , we need to check to see whether other conditions have been met . Chief among them is whether there are other possible explanations for the correlation besides the hypothesized causal relation . As we saw , there are all kinds of techniques for identifying significant correlations , though some techniques are better than others . A Key Question for a Statistic What , exactly , was counted ?

Questions to Ask of Inferences from correlation to cause How likely is the proposed explanation ?

Are there other plausible explanations for the correlation ?

Would the truth of the proposed explanation be less surprising than the truth of any competitor ?

KEY TERMS Comparative Statistics Outliers Scale Relevantly Similar Statistics Causal Inference Complete Cause Partial Cause Reliable Cause Probabilistic Cause Positive Correlation Statistical Claim Negative Correlation Significant Correlation Observational Study Experimental Study Experimental Group Group Confounding Factor Controlling for a Variable Unknown Experimental Group Control Group FURTHER READING There are a number of short readable books about the pitfalls of dealing with statistics . A famous , but dated , source is How to Lie with Statistics by Darrell Irving . For more contemporary examples , you could check Best Damned Lies and Statistics or that a Fact ?

A Field Guide to Statistical and 245 246 THADDEUS ROBiNSON Information by Mark . For a deeper discussion of experimental structure and set up , see Randomized Controlled Trials Questions , Answers , and Musings by and Murray .

Concluding Thoughts In conclusion , let us briefly look back over the text as a whole . Taking this wider angle will allow us to see how the text has woven several themes together , and will put us in a position to think about reasoning and argument more broadly . As noted at the outset , the goal of the text is to help its readers reflectively enhance their general reasoning skills . We have pursued this goal by walking through the basics of ment identification , analysis , and evaluation within the social and psychological in which they occur . Along the way , we have gradually built up a vocabulary for describing the elements , standards , and circumstances for reasoning . The most important running theme in the text has been asking the right question . As we have seen , ing the intended structure of an argument means asking about the author main point and reasons , ing to resolve ambiguity , and so on . Turning to argument evaluation , we isolated three main questions to ask of any ( inductive ) argument . Evaluation The Three Main Questions Are the premises likely to be true ?

A check for Factual Correctness ) Would the truth of the premises make the truth of the conclusion probable ?

A check for internal Logical Strength ) Is there any other relevant information available ?

A check for external Logical Strength ) We discussed the process of asking each one of these questions . While checking for factual correctness can be difficult in some cases ( we spent most of our time thinking about how to determine logical strength . Indeed , when we know what kind we are talking about , there are more specific questions we can ask to determine an argument logical strength . We focused on four of the most mon types of ( inductive ) argument , and identified a series of more specific questions to help us determine logical strength for each type . Two Questions to Ask of Analogy Is the noted similarity relevant to the inferred similarity ?

Are there differences that are relevant ?

Three Questions to Ask of inferences to the Best Explanation How likely is the proposed explanation ?

Are there other plausible explanations ?

Would the truth of the proposed explanation be less surprising than the truth of any competitor ?

Two Questions to Ask of Inductive Generalizations 247 248 THADDEUS Is the sample large enough ?

Is the sample diverse enough ?

Two Questions to Ask of Inductive Applications Is the individual in question a member of the subject class or not a member of the predicate class ?

Is the individual in question a member of other relevant classes ?

Another main theme at work throughout the text has to do with the influence of social factors on our ing . Reasoning is not the solitary task it is sometimes imagined to be , and almost always takes place within a broader social context . We began by focusing on cooperative dialogue , and cooperative disagreement specifically . In addition , we have seen how social factors can activate biases and help mitigate against them . We have discussed the extent to which we depend on other people for information , and use them to check our own thinking . Moreover , we saw how important trust is for learning from other people , and took a close look at when trust is warranted , and when it is not . Lastly , we have seen how the internet and social media give us access to an incredible variety of new voices and information , while simultaneously amplifying information and making it difficult to know who to trust . The final theme is psychological . At the outset , we distinguished between largely automatic and implicit reasoning processes , on the one hand , and consciously directed ones , on the other . Although we have focused on the latter , we have considered how our conscious reasoning can be influenced by these implicit processes . bias , for example , can arise from largely intuitive responses and preferences . In addition , we saw that in some we are subject to a variety of biases or cognitive illusions . We often read as if they were symmetric even though they are not , and we tend to overestimate how common something is when it is especially interesting or provocative . Further , we underestimate the influence of factors when we explain other people behavior , and we tend to think other people are more like us than they really are . So where does this leave us ?

Hopefully , taking the time to reflectively work through this text has made you a more careful and active thinker , and has given you greater confidence to navigate the important , challenges , and decisions you will face .