The Most Cited Study In The Transgender Athlete Debate Is Bad Science – The Daily Caller
The Most Cited Study In The Transgender Athlete Debate Is Bad Science The Daily Caller
Those who argue transgender athletes should be allowed to compete with women often cite the work of Joanna Harper. But Harper’s study is bad science.
Those who argue transgender athletes should be allowed to compete with women often cite the work of Joanna Harper as evidence that biological men who suppress their testosterone levels do not have an unfair competitive advantage against women. But the study really doesn’t prove anything from a scientific standpoint.
Harper freely admits this. “I’m certainly not suggesting that one study of eight athletes in one sport is in any sense a definitive study,” Harper told The Daily Caller. “But it was a start, and it’s got people interested in doing more research. There’s probably ten, twenty more years, probably, of work that needs to be done on this, but we’ve started.”
Harper, a scientist and competitive long distance runner, decided to research the topic in 2004, after personally experiencing a big drop in run times post transition. “I started on hormone therapy in August of 2004, and within 9 months I was running 12 percent slower. It was an astounding thing for me,” Harper said. The study was published in 2015.
Harper analyzed the run times of eight male long distance runners before and after they began suppressing their testosterone levels and competing against women. The study found the testosterone suppression slowed them down, to the extent that they were competing at about the same level against women as they had been against men. It’s been used repeatedly to push back against claims that biological men have an unfair advantage over women in sports, even after they suppress their testosterone levels, and informed the 2016 guidelines for transgender competition set by the International Olympic Committee. (RELATED: Liberals Are At War With Science In The Transgender Athlete Debate)
Joanna Harper talks transgender athletes on Fox News. (YouTube/Public/Fox News)
A closer look at the study, though, shows that while Harper seems to have conducted it in good faith, it’s is riddled with scientific problems. The biggest ones are with the size of the study and the way the data was collected, which Harper acknowledges, as well as a lack of controls and the context in which some of the data was collected.
“The best that might be said is that the study suggests that more exploration through a better-designed study would be useful,” American University math and statistics professor Mary Gray told the Caller, ticking off a list of the biggest flaws.
It took Harper seven years to collect the run times of just eight transgender athletes, mostly by reaching out to them online and asking them to participate. Those who agreed to participate self-reported before and after run times for races of varying distances, sometimes over a long span of time. Harper was able to verify just half of the times for six of the runners, and most of them did not submit times for all of the distances looked at in the study — 5k, 10k, half marathon and full marathon.
There was no control group, since this was a retrospective study, and no control for factors such as time spent training before and after transition, the kind of courses and distances run, injuries, and other potentially significant factors. Harper noted three of the runners reported factors other than testosterone suppression that clearly impacted their scores. (RELATED: The Real Story Behind USA Powerlifting’s Decision To Ban Transgender Athletes From Women’s Events)
One runner got injured, which could explain the much slower times after transition. Another gained weight and experienced a loss of motivation following transition, and eventually gave up on running altogether. Again, these factors, rather than lower testosterone levels, could explain the slower run times.
A third runner, on the other hand, sharply increased training post transition and lost weight, and performed much better against women than against men. Harper chose to include the data of the first two runners, but excluded the data of the third as an outlier in the statistical analysis, based on a subjective review of the weight of the various factors.
Harper used a mathematical model to adjust the runner’s scores according to their age before and after transition, known as age grading. A given runner’s time was compared to the fastest time ever run over that distance by someone of the same sex and gender, and then graded accordingly. Harper compared the age grade of each runner before and after transition, and found that collectively most of them scored about the same, indicating they were competing at about the same level against both sexes.
Here’s what the resulting table looked like for the runners who reported 10k race times:
The parentheticals indicate that runner two’s times were compared over a long time period, so they are less meaningful for the purposes of this study, while the times for runner six are among the most useful because they were gathered from runs on the same course over a relatively short three-year time period.
Harper then averaged the age grades for all of the runners and ran a statistical test, which found the changes in grade from before and after transition were not statistically significant. The same test was run on the best individual times for each runner, and again the statistical test did not find a meaningful difference in their grades. (The runner who got a lot faster post transition was not included in these tests, because of a reported dramatic increase in training.)
In addition to some of the flaws already noted — participants were not randomly selected, no control group — Ed Boone, a statistics professor at Virginia Commonwealth University, pointed to problems with the statistical analysis portion of the study. Boone said that a post hoc power analysis should be completed to determine whether the statistical analysis Harper conducted was powerful enough to detect a meaningful difference in the results.
“From a statistical standpoint the lack of finding a statistically significant difference does not imply that no difference exists,” Boone told the Caller. “It simply means that no difference was detected.”
Harper didn’t specify exactly what kind of test was run on the data from the last two tables, but Boone found a post hoc power of .0534 based on a test he ran. That means if there is a statistically meaningful difference between the group’s performance level before and after transition, the test would be able to detect it 5.34 percent of the time. “Typically for studies like this, one would need a power of over 0.8 or 0.9 to be sure that difference does not exist in the population,” Boone said.
Aside from these flaws, the design of the study is also limited in scope, since many of these runners were not elite level athletes and it looks only at long distance running. Harper concludes the findings of the study make it “possible” to say that transgender runners compete at about the same level before and after transition, but is careful to note: “This conclusion only applies to distance running and the author makes no claims as to the equality of the performances, pre- and post- gender transition, in any other sport.” (RELATED: Media Reports On Transgender Athletes All Seem To Be Missing One Thing)
Jay Kadane, a statistics professor at Carnegie Mellon University, said it’s not great science, but lauded Harper for a respectable and honest effort. “It is some evidence,” he told the Caller. “Is it conclusive? Certainly not. … There’s just a lot we don’t know. But that doesn’t mean this isn’t a good and honest first try.”
The differences that do exist between biological men and women in sport are perhaps the most pronounced in powerlifting. The USA Powerlifting Federation is in a dispute with a transgender athlete who filed a discrimination complaint in the state of Minnesota, because the federation does not allow biological men to compete with women. The policy is reinforced by research a panel of experts conducted for the federation, which found a statistically significant advantage remains among men who transition, even if they use suppress their testosterone levels.
JayCee Cooper (YouTube/Public/Vice News HBO)
The researchers looked at both available literature and data on transgender athletes and differences between men and women, and then conducted a statistical analysis of internal data and data from the International Powerlifting Federation and its affiliates. In contrast to Harper’s study, their analysis represented 17,351 lifters and 1,300 competitions at all ranges of age and competitive levels, over a 7 year period. They grouped the data into categories based on sex and body weight, and looked at lift totals with the aim of quantifying the advantage a typical biological male has over a typical biological woman in powerlifting.
They found men have between a 46 and 64 percent strength advantage over women, which is key, because the available research on men who transition to women indicates they experience a 10 percent drop in strength after reducing their testosterone levels. The researchers concluded it would not be fair to allow transgender athletes to compete with women.
Harper is familiar with the study, and told the Caller there does seem to be a clear advantage for transgender athletes competing against women in powerlifting especially, but also in other sports that rely on height and strength, such as volleyball. “Powerlifting, of course there’s some technique to it, but it’s almost all strength,” Harper said. “And there might be a valid point in not having transgender women and cisgender women compete for the same prizes.”
Harper has suggested transgender athletes be allowed to compete alongside women, but then be scored differently, so they can feel included in the competition without directly competing against the women. For volleyball, Harper suggested to the International Olympic Committee that teams be allowed a maximum of one transgender player each. (RELATED: Columbia Journalism Review Chooses Sides In Trans Athlete Debate)
“You see people screaming at each other on Twitter, and that’s no way to solve problems,” Harper told the Caller. “And there are people that I won’t name, who have been extraordinarily toxic on social media, and I think it’s extraordinarily counterproductive. These issues are very difficult, and it’s certainly reasonable that people will have disagreements, but hopefully you can get rational people and sit them down and talk these things out, and reach some kind of consensus.”
Harper will again advise the International Olympic Committee, when it meets this year to determine the rules for the next Olympic games. At the 2015 meeting, Harper said the group of about 20 discussed the available research (including Harper’s study) and each other’s point of view, and ultimately formed a general consensus on the rule changes. Transgender athletes were allowed to compete based on testosterone levels, which was a change from rules set in 2004 that required a person to legally change their sex, undergo surgery and adjust testosterone levels in order to compete.
This fall, Harper will move to London to work on new studies of transgender athletes, in this case following three individuals before, during and after their transition, in real time. Harper is also working on getting a follow up retrospective study published, along with several other scientists, by the end of this year. The study will similarly look at data from one rower, one sprinter, three long distance runners, and a cyclist.
Harper is excited about the continued work, but is not thrilled about the way some are using the initial study in the current debate. “There’s not a lot of nuance, and people are making hyperbolic claims on one or the other,” Harper told the Caller. “And yeah, there’s definitely people who have said, ‘Oh Joanna Harper’s study proves something.’ Well, of course it doesn’t prove anything, really.”
“It shows that hormones make a big difference,” Harper continued. “But beyond that it’s just a starting point for these studies, and so I’ve been a little chagrined that people have made claims about the study that are far greater than claims that I make about the study.”