Cynthia Than at Quartz brings us the delightful headline “There is no gender gap in tech salaries” based on a study from AAUW. The study is based on non-public data and is, unfortunately, not very rigorously documented.
The key conclusion from the article is that there is no gender gap in salaries for engineers and those working in math, computer and physical science occupations. As far as I can tell, that result comes from this figure in the study:
From this figure, you would think that male and female engineers have exactly the same average earnings, $55,046. There are a number of problems with the way this is presented and with the underlying study and its interpretation.
If you go to the underlying tables, you’ll find that male and female engineers do not have the same earnings. This data is based on earnings for full time employees in 2009, a year after the sample graduated college in 2008, and is restricted to those who were 35 years or younger at graduation. I wasn’t able to recreate the figure precisely using the Department of Education’s online tool, but I got pretty close to overall average earnings for this group of engineers of $55,076. Here’s what happens when you chart it out by sex:
That’s right: in levels, female engineers make about 11% less than men one year out of college.
Both Cynthia Than and the researchers behind the study claim that the difference is not statistically significant. That may be true, but it’s still misleading to simply show the same bars for the two groups. If you want to make a point about significance, use error boxes or something like that.
Is there, in fact, an insignificant difference? I’m not sure. NCES has a helpful tool for calculating the t-statistic:
The tool tells us that the value is 2.25, which means that the gender gap is statistically significant. I am not sure because it’s possible that more thought was put into Figure 8 than is apparent from the notes. Let’s get back to that.
You might think that it’s weird to only look at gender gap for earnings one year after graduation. It could be because the data set only contains that variable. But why did they pick that particular data set when other datasets, such as the American Community Survey, have more data? Perhaps to control for additional variables such as the rank of the college and GPA that are not present in ACS. There’s also an argument that restricting to earnings one year out of college helps control for the motherhood penalty and other factors.
That could be true, but it still only tells you something about earnings for a very limited set of workers. You wouldn’t be able to conclude that there is no gender gap in tech salaries; for example, they could show up later in the career.
Figure 13 of the study has what appears to be a carefully done regression that shows a statistically significant gender gap of 6.6% across all occupations, controlling for things such as hours worked, economic sector, undergraduate GPA and whether the undergraduate degree was from a very selective institution. That’s the only specification shown, so we are not shown any evidence that these extra controls actually affect the size of the gender gap (the regression coefficient on female).
The conclusion of the Quartz article was that there is no gender gap in tech. It’s not clear that Figure 8 is based on a regression that controls for anything at all. The notes say:
This chart shows average earnings 2007–08 bachelor’s degree recipients employed full time in 2009 and excludes graduates older than age 35 at bachelor’s degree complation. In occupations with red and green columns shown, men earned significantly more than women. In occupations with one blue column shown, there were no significant gender differences in earnings one year after graduation.
There’s no mention of control variables, or a regression, or how the regression was made (for example, restricting to tech workers, or introducing gender×occupation interaction dummies). It’s still a mystery how they arrived at the conclusion that there’s no difference, or why they created a misleading chart, or whether they performed a full analysis of the gender gap by occupation, and if so, why the results are not reported. And if we don’t control for these things, what’s the point of using a small study that only has data for earnings one year after graduation?
I do think it’s safe to say that this study, as reported, does not have enough evidence to conclude anything about occupational gender gaps, whether it’s for earnings one year after graduation or at any other time in workers’ careers.
Another question is whether all of those controls are appropriate (if they were included). One of my teachers once told me that social scientists overcontrol. What he meant was that a social scientist tends to be interested in the partial effect of one variable controlling for every other conceivable variable, but the public or policy makers may not be interested in the partial effect: if women make less than men, they may think that’s a problem, whatever the reason. You may still want to include plenty of controls for efficiency, but your reader may be more interested in a result that has those controls integrated away. In this case neither undercontrolled nor overcontrolled results are accurately reported.
Update: Cynthia Than tells me that the significance results from Figure 8 control for education, work status, occupation and career timing. As far as I can tell she got that in private email communication with one of the study’s authors. (The authors’ email addresses are not in the report or readily accessible on the AAUW website, by the way, so there’s no way for anyone else to ask questions about it.)
I don’t want to belabor this point too much because even if everything is done correctly, Quartz is still trying to draw general conclusions about the gender wage gap based only on salaries one year out of graduation. This is a point about external validity: all the arguments in favor of using salaries one year after graduation only establish that those salaries can be measured with less error and fewer confounding factors than, say, salaries 20 years after graduation. They do not establish that salaries one year after graduation are of particular interest.
However, I still think we should be careful with a report that lacks full documentation on its methodology and how the results are constructed. We are not given regression specifications, results from alternative specifications, sample sizes, t-statistics, or even a full list of control variables for Figure 8 (except what was privately communicated to Than). I don’t necessarily fault the study authors for omitting this in a report for general consumption when Figure 8 is only a minor aspect of the whole report. But without more information about that aspect of the study, I would be very cautious in relying on it, and in making the general conclusion that there is no gender gap in tech salaries.
I’ve been trying hard to find good data on this and again it’s been hard. The NCES tool initially looked useful. Presumably because of sample size related privacy issues, it refused to give me average wages by gender for tech professions other than Engineers. That raises concerns about statistical power. It’s possible that the online tool is based on a smaller sample than what the AAUW researchers had access to. If only either the online tool or the AAUW study would report sample sizes!