Although the talk is aimed more at psychologists than economists, it's a nice accessible overview to these subjects, which apply across the social sciences--and toward the end, Nisbet has some comments that should be thought-provoking for economists about the potential strength of social influence. But at the start, Nisbet says:
"A huge range of science projects are done with multiple regression analysis. The results are often somewhere between meaningless and quite damaging. I find that my fellow social psychologists, the very smartest ones, will do these silly multiple regression studies, showing, for example, that the more basketball team members touch each other the better the record of wins.
I hope that in the future, if I’m successful in communicating with people about this, there’ll be a kind of upfront warning in New York Times articles: These data are based on multiple regression analysis. This would be a sign that you probably shouldn’t read the article because you’re quite likely to get non-information or misinformation. ...
What I most want to do is blow the whistle on this and stop scientists from doing this kind of thing. ... I want to do an article that will describe, similar to the way I have done now, what the problem is. I’m going to work with a statistician who can do all the formal stuff, and hopefully we’ll be published in some outlet that will reach scientists in all fields and also act as a kind of "buyer beware" for the general reader, so they understand when a technique is deeply flawed and can be alert to the possibility that the study they're reading has the self-selection or confounded-variable problems that are characteristic of multiple regression."The basic issue arises here when a study looks at two variables that are correlated with each other. However, the study doesn't take into account all the relevant factors which may be causing the correlation, and thus it misinterprets the correlation. Here's a quick-and-dirty example from Nisbet:
"A while back, I read a government report in The New York Times on the safety of automobiles. The measure that they used was the deaths per million drivers of each of these autos. It turns out that, for example, there are enormously more deaths per million drivers who drive Ford F150 pickups than for people who drive Volvo station wagons. Most people’s reaction, and certainly my initial reaction to it was, "Well, it sort of figures—everybody knows that Volvos are safe."
Let’s describe two people and you tell me who you think is more likely to be driving the Volvo and who is more likely to be driving the pickup: a suburban matron in the New York area and a twenty-five-year-old cowboy in Oklahoma. It’s obvious that people are not assigned their cars. We don’t say, "Billy, you’ll be driving a powder blue Volvo station wagon." Because of this self-selection problem, you simply can’t interpret data like that. You know virtually nothing about the relative safety of cars based on that study."Again, Nisbet's point is that just because pickup trucks are correlated with more accidents or deaths does not in any way prove that pickup trucks are less safe. Maybe they are. Or maybe the reason behind the correlation is that pickup trucks are more likely to attract certain kinds of drivers, or to be driven in certain ways, and thus have
Like many quick-and-dirty examples, this one is both useful and oversimplified. Multiple regression analysis is a way of doing correlations that lets you take a lot of other factors into account and hold them constant. In this example, one could include "control variables" about the drivers of cars, including age, gender, occupational category, parental status, rural/urban, state of residence, and the like. If you've got the detailed data, the kinds of factors that Nisbet mentions here can be taken into account in a multiple regression analysis--and if a study didn't take at least some of those factors into account, it's clearly statistical malpractice.
But what's even worse is the situation where the available data doesn't describe the key difference. A standard example in economics classrooms is the study of how much getting more education influences future wages. Sure, it's easy to draw up a correlation and show that, on average, college graduates make more money. But it's plausible that college graduates may differ from non-college graduates in other ways, too. Perhaps they have more persistence. Perhaps they are more likely to be personally related to people who can help them find high-paying jobs. Perhaps they are smarter. Perhaps employers treat college completion as a signal that that someone is more likely to be a good employee, and offer job possibilities accordingly.
Economists have been hyper-aware of these issues for at least a couple of decades, and arguably longer. For an overview of the approaches and methods they use to try to circumvent these problems, a useful starting point is the essay by Joshua D. Angrist and Jörn-Steffen Pischke, "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," which appeared in the Spring 2010 issue of the Journal of Economic Perspectives, (24:2, pp. 3-30).
Nisbet also points out that a lot of psychology experiments compare people people in one situation with people in a slightly different situation, but such studies can be hard to replicate because so many things are different about two situations, and people aren't very aware of what influences their decisions. Nisbet describes one of his own experiments this way:
"The single thing I’ve done that has gotten the most notice was my work with Tim Wilson showing that we have very little access to the cognitive processes that go on that produce our behavior. ... I’ll give an example of the experiment we did. We have two experiments, and in the first experiment—a learning experiment—people memorize word pairs. There might be a pair like "ocean-moon" for some subjects. Then we’re through with that experiment: "Thank you very much." The other fellow has this other experiment—on free association, where you are to give an example of something in a category we give you. So, one of the categories we give is, "Name a laundry detergent." People are much more likely to mention Tide, if they’ve seen that ocean-moon pair. You ask them, "Why did you come up with that? Why did you say Tide?" "Well, that’s what my mother uses," or "I like the Tide box." And you say, "Do you remember learning that word pair—ocean-moon?" "Oh, yeah." "Do you suppose that could’ve had an influence?" "No. I don’t think so.""Toward the end of the talk, Nisbet offers what I interpreted as a useful challenge to economists. When economists think about how to alter some behavior, they tend to think in terms of incentives for the individuals. Nisbet offers a useful reminder that social incentives often affect our behavior quite powerfully, and we are only dimly aware that these forces are operating on us. Nisbet offers some vivid example of social incentives in action:
I hear the word incentivize, I say, "If imagination fails, incentivize." There are so many more ways of getting people to do what’s in their own interests and society’s interests. Absolutely the most powerful way that we have is to create social influence situations where you see what it is that other people are doing and that’s what you do. I took up tennis decades ago and it turned out that most of my friends had taken up tennis. I dropped it a few years later and it turned out that the tennis courts were empty. I took up cross-country skiing, and how about that, these other people do it. Then we lost interest in it, and find out our friends don’t do that anymore.
How about minivans and fondue parties? You do things because other people do them. And one very practical important consequence of this was worked out by Debbie Prentice and her colleague at Princeton, [Dale Miller]. Princeton has a reputation of being a heavy drinking school. ... Prentice and Miller had the idea to find out how much drinking goes on. They had the strong intuition that less drinking goes on than people think goes on, because on Monday a kid comes in and says, "I was stoned all week," when in actuality he was studying all Sunday for the exam. In a setting where people are drinking a lot, you get prestige for drinking a lot. If you get good grades despite the fact that you’re drinking a lot, then that makes you look smarter. They found out how much people are actually drinking, and then they fed this information back to students and said, "This is how much drinking goes on." Drinking plummeted down to something closer to the level of what was actually going on.
Here's something that saved hundreds of millions of dollars and millions of tons of carbon dioxide being poured into the atmosphere in California by a social psychologist team led by Bob Cialdini. He hangs tags on people’s doors if they’re using more electricity than their neighbors saying, "You’re using more electricity than your neighbors," and that sends electricity usage down. However, you shouldn’t hang a tag on their door saying, "You’re using less electricity than your neighbors" because then people start using more electricity—unless you put a smiley face on the bottom. You’re using less electricity than your neighbor’s and a smiley face ... oh, that’s a good thing, I’ll keep it up.