Why Methodology Matters (really)

and

a Real-world Example of Good Intentions Undermined by Bad Methodology

(thanks to UNC’s Zeynep Tufekcifor much of this discussion)

Professors often like to say things like this:

“Facts” mean nothing without a lens (or theory) through which to view them. All the data in the world is meaningless or even harmful without an explanation as to how one collected those “facts” and a theory by which to interpret them.And since collecting data always means selecting what data to collect (as even the largest data set must exclude something), and since selection is always a research method, there is no knowledge outside of research methodology.

 

But it’s not always easy to see why any of that professor speak might be true.

But a recent internet sensation of sorts—a youtube video viewed more than 30 million times, discussed endlessly on social media, repeatedly parodied, and covered by CNN—demonstrates the importance of methodology.

See the original video here:

(“10 Hours of Walking in NYC as a Woman,” Rob Bliss Creative, 2014)

 

Critics at the time (Fall of 2014) immediately drew attention to the fact that nearly all the men allegedly catcalling to the white actress were people of color; indeed, most are black men. The video, however, contains text that reads: “100+” instances of “verbal street harassment” had occurred during the ten hours of shooting, from “people of all backgrounds.”

So, how to explain this seeming discrepancy between the producers’ claims and the video’s contents?

The video, in essence, presents itself as a research project not unlike those of your own senior capstone projects.  It, for example, has an implicit research question (“Do conventionally attractive white women in New York get verbally harassed?”) and generates its own answer: the video itself.

The video underscores the importance of methodology both for getting reliable answers and for the political consequences of research.  The Hollaback video, for instance, had been intended to be an indictment of street harassment.  But because the video’s producers failed to reflect upon (or publicly present) their own methodology, their work instead became interpreted as an indictment of black men.  See, for example, these twitter posts by prominent novelist Joyce Carol Oates where she suggests catcalling in New York occurs only in poor, non-white neighborhoods:

 

 

Without a careful presentation of the methods of their study, the racial demographics of Hollaback video lends itself to three wildly different hypotheses.

We can summarize these as:

Hypothesis 1: Men of color are more likely to catcall, especially to white females and, so, implicitly men of color are a threat to white women. Hypothesis 2: Although all men regardless of race are equally likely to catcall, the video’s makers were biased, consciously or unconsciously, against black men and so intentionally edited out men of other races Hypothesis 2 might be divided into three versions: 2.a Consciously: the video’s makers are racists and intentionally played to a long-standing (and prejudiced) notion in America that  “white women are endangered by black men” 2.b Unconsciously: The video’s makers employed a flawed methodology that made it appear as if men of color are more likely to catcall 2.c. Both 2.a. and 2.b are true.   Hypothesis 3: It’s a spurious correlation because the study was badly designed.  A spurious correlation occurs when a third, unrelated variable, causes a change in other variables in a study, which can then seem as if they are causally connected even though they are not.   (This third variable that creates the illusion of correlation is often called a confounding variable). An example of a spurious correlation can be see with ice cream and murder: as ice cream consumption goes up, so too do murders. The two variables are correlated. Perhaps popsicles make good murder weapons?  In reality, it’s a spurious correlation caused by a confounding variable: summer. There is more crime of all kinds in the summer, and people eat more ice cream in summer. So, although it looks as if more ice cream eating produces more murders, what’s really occurring is that warmer temperatures produce more of both murdered bodies and ice cream purchases. In the case of the video, a spurious correlation would be if some other factor caused black men to catcall more frequently than white men when the researchers went out into the field, but there was no real relationship between being black and the likelihood of catcalling.

 

The central methodological point is that because the video does not reflect upon or present its own methodology, the video can support all three wildly incompatible propositions.

In short, if you viewed just the video you could believe any of the three hypotheses.  Indeed, you would likely choose whichever best fits your own preoccupations and prejudices.

But which of the hypotheses best explains what the video presents?

The third hypothesis is the easiest to address.

The most obvious confounding variable here would have been if the Hollaback producers had their actress walk streets in minority neighborhoods but skipped over streets in white ones. The producers, however, insisted the actress walked all of New York rather than just minority neighborhoods.

So, where do the catcalls occur?

Thankfully, someone has done the homework and mapped out where each shot in the video occurred—something any researcher producing a two minute video from ten hours
should have done.

Here’s what that mapping found:

Shot Number	Landmark	Intersection	Neighborhood
Shot 1	Pathmark	129th + Lexington Ave.	Harlem
Shot 2	Coffee spot	Broome + Broadway	SoHo
Shot 3	Aldo	Broadway + Great Jones	Village
Shot 4	Apollo	125th + Adam Clayton Powell	Harlem
Shot 5	Manna’s	125th + Madison Ave.	Harlem
Shot 6	Canal St.	Canal St.	Canal
Shot 7	Unknown	Unknown	Unknown
Shot 8	Heartland Brewery	41st. St. + 8th Ave.	Times Square
Shot 9	Carmine’s	44th St. + 7th Ave.	Times Square
Shot 10	Duane Reade	40th St. + 8th Ave.	Times Square
Shot 11	Dr. Sandy F. Ray Building	W. 125th. St.	Harlem
Shot 12	H+M	W. 125th St.	Harlem
Shot 13	Marshall’s	W. 125th St.	Harlem
Shot 14	Street	W. 125th St.	Harlem
Shot 15	Street	125th St.	Harlem
Shot 16	Starbuck’s	125th + Malcolm X Blvd.	Harlem
Shot 17	Gap	125th + Fredrick Douglas Blvd.	Harlem
Shot 18	American Apparel	125th + Fredrick Douglas Blvd.	Harlem
Shot 19	Starbuck’s	125th + Malcolm X Blvd.	Harlem
Shot 20	Construction Underpass	125th St.	Harlem
Shot 21	Universal Church	125th St.	Harlem
Shot 22	Street	125th St.	Harlem
Shot 23	Gap	42nd St.	Times Square
Shot 24	Chevy’s	42nd St.	Times Square
Shot 25	Duane Reade	Unknown	Unknown

 

That information in the table above can be represented in this pie graph by neighborhood:

 

 

So, did the filmmakers: (a) intentionally cluster their filming on a few neighborhoods, or (b) film all of New York but encountered catcalling only in minority neighborhoods, which they then presented in their video?

Possibility (b) is what the author Joyce Carol Oates assumed in her tweets above. (A), however, is also possible.

But without careful reflection and presentation of methodology, one cannot know if the filmmakers are simply bad at research design or if black men actually catcall women more!

It’s also a revealing example of why personal experience or anecdotes from friends—however important they might be to our sense of identity—cannot and should not be trusted. Only systematic research geared to remove our biases can produce an understanding of the world.

But the above critiques are only the start of the methodological problems of the video.  The following questions will address others.

BEFORE YOU START (DONE IN CLASS)

Assumptions Need Acknowledgment:

All research has assumptions they are impossible to remove, but not impossible to define. An assumption is not as developed as a theory, but a simple an underlying belief or understanding. Implicit assumptions have implications and influence your findings.  The first step is to identify the underlying assumptions in the research. Identifying assumptions will help you define what you are researching and ultimately, what you are counting. What are the assumptions of the Hollaback video?

List at least three assumptions of the Hollaback video.

Definitions:

What is it you are researching? How does the Hollaback video define street harassment/”catcalls”?  While this seems like a simple question it requires thought.  Is any comment directed at a stranger street harassment?

Once you define street harassment are there multiple kinds? From the video is there a typology of street harassment.  Are types of verbal street harassment missing from the video? Please define.

The answers to the definitional questions are important, because they determine what data you are collecting and what you are counting.  What you are counting influences how you select cases to included, or excluded. More importantly they help determine how you SMAPLE and what you sample on.

(you should submit your responses to question 1-3 below in your HW)

Question 1: Random Sampling

First, read this discussion of random sampling.  Then, in four paragraphs of careful prose, explain: (a) how and why the producers of the video failed to address the importance of random sampling in their implicit research project and (b) what the producers would have had to do address concerns raised by the issue of random sampling.

 

Question 2: Construct Validity

First, read this discussion of construct validity.  Then, in four paragraphs of careful prose, (a) explain how and why the producers of the video failed to address the importance of construct validity in their implicit research project and (b) what the producers would have had to do address concerns raised by the issue of construct validity.

(keep in mind that concerns about construct validity might not necessarily be about the relative frequency of black & white catcalling; there are also other problems with the research design of the Hollaback video)

 

Question Three: Rater Reliability

First, read this discussion of inter-rater reliability.  Then, in four paragraphs of careful prose, explain (a) how and why the producers of the video failed to address the importance of inter-rater reliability in their implicit research project and (b) what the producers would have had to do address concerns raised by the issue of inter-rater reliability.

(keep in mind that concerns about inter-rater reliability might not necessarily be about the relative frequency of black & white catcalling; there are also other problems with the research design of the Hollaback video)