Friday, March 28, 2014

"Data Journalism" on College ROI at FiveThirtyEight: Where's the Critical Thinking?

NOTE: See the end of this entry for important updates, including one from 11/9/15.

A website called PayScale recently published a "College ROI Report" that purports to calculate the return on investment (ROI) of earning a Bachelor's degree from each of about 900 American colleges and universities. I found out about this report from an article on Nate Silver's new FiveThirtyEight website. The article appears under a banner called "DataLab," implying that it is an example of the new "data journalism" that Silver and his site are all about. Unfortunately, the article contains approximately zero critical thinking about the meaning of the PayScale report, its data sources, and its conclusions.

PayScale did a lot of number-crunching (read all about it here), but the computation resulted in two key numbers for each institution: (1) the cost of getting an undergraduate degree, taking into account factors like financial aid and time to graduation; and (2) the expected total earnings of a graduate over the next twenty years. The first one can be figured out from public data sources. The second one came from a survey by PayScale (more on this later). The ROI for a college was calculated by subtracting #1 from #2, and then further subtracting the expected total earnings of a person who skipped college and worked for 24–26 years instead (which happens to be about $1.1 million). The table produced by PayScale thus purports to show how much you would get back—in monetary income—on the "investment" of obtaining a degree from any particular college or university.

Indeed, PayScale says that "This measure is useful for high school seniors evaluating their likely financial return from attending and graduating college." But this is simply not true. As I read the FiveThirtyEight article on the PayScale report, I was waiting for them to point out the reasons why, but they never did. The only critical comments were about incorporating the effects of student debt.

What are the problems with the PayScale analysis? First of all, it only makes sense to speak of the comparative return on an investment when the investors have a choice of what to invest in. If every person could choose to attend any college (and to graduate from it and get a full-time job), or to skip college entirely, then it would be meaningful to ask which choice maximizes return. This is what we do when calculating a financial ROI: we try to figure out whether investing in stocks versus bonds, or one mutual fund versus another, or one business opportunity versus another, will be more profitable. But colleges have admissions requirements, so not everyone can go to whatever college he or she wants. Colleges select their students as much as students select their colleges. And in fact, the people who attend different colleges can be very different, and they can be even more different from the people who don't attend college at all.

This means that the Return in this "ROI" depends on much more than the Investment. It also depends on who is doing the investing. In fact, it is far from trivial to figure out the true ROI of going to Harvard versus Vanderbilt versus Wayland Baptist versus Nicholls State versus not attending college at all. To figure this out, you would have to control in the analysis for all the characteristics that make students at different colleges different from one another, and different from students who don't go to college. Factors like cognitive ability, ambition, work habits, parental income and education, where the students grew up and went to high school, what grades they got, and many others are likely to be important. In fact, those other factors could be so important that they might wind up explaining more of the variation in income between people than is explained by going to college—let alone which particular college people go to.

Even controlling for data we might be able to obtain, like the average SAT score and parental income of students who attend each college, would not completely solve the problem, because there could be factors that we can't measure that have an important effect. Only by randomly assigning students to different colleges (or to directly entering the workforce after high school) would we get an estimate of the true ROI (measured in money—which of course leaves aside all the other benefits one might get from college that don't show up in your next twenty years of paychecks).

Of course this ideal experiment won't ever happen, but clever researchers have tried to approximate it by doing things like looking at students who were accepted to both a higher-ranked and a lower-ranked school, and then comparing those who enroll in the higher-ranked one to those who enroll in the lower-ranked one. Since all the students in this analysis got into both schools, the problem of different schools having different students is mitigated. (Not erased entirely, though: for example, people who deliberately attend lower-ranked schools might be doing so because of financial circumstances, or their college experience may differ because they are likely to start out above average in ability and preparation for the school they attend, as compared to those who choose higher-ranked schools.)

FiveThirtyEight said nothing about this fundamental logical problem with the entire PayScale exercise. Nor did it address the other flaws in the analysis and presentation of the data.

It could have also asked about the confidence intervals around the ROI estimates provided by PayScale. When you give only point estimates (exact values that represent just the mean or median of a distribution), and proceed to rank them, you create the appearance of a world where every distinction matters—that the school ranked #1 really has a higher ROI than #2, which is higher than #3, and so on. PayScale's methdology page says, "the 90% confidence interval on the 20 year median pay is ±5%" (but 10% for "elite schools" and "small liberal arts schools or schools where a majority of undergraduates complete a graduate degree"). The narrowness of these intervals is a bit hard to believe, as well as their uniformity (how does every school in a category get the same confidence interval?). Why not just put the school-specific confidence intervals into the report, so that it is obvious that, for example, school #48 (Yale) is probably not significantly higher in ROI than, say, school #69 (Lehigh), but is probably lower in ROI than school #6 (Georgia Tech)?

It's hard to have much confidence in these confidence intervals anyhow, since we don't know how many people PayScale surveys at each college to make the income calculations (which will be the critical drivers of the variability in ROI). Many of the colleges are small; how reliable can the estimates of what their graduates will earn be? And are the surveys of college graduates unbiased with respect to what field the graduates work in? Or, for example, do engineers and teachers tend to respond to these surveys more than, say, baristas and consultants? The unemployed and under-employed are not included; this will have the effect of inflating the apparent ROI of schools whose graduates tend, for whatever reasons, not to have full-time jobs. Payscale says that non-cash compensation and investment income are not included, which might bias down the reported ROI of graduates of elite schools who go into financial careers.

Finally, perhaps FiveThirtyEight could have looked at whether the schools that stand out at either end of the distribution happen to be smaller than the ones in the middle. Ohio State, Florida State, et al. have so many students, drawn from such a broad distribution of ability and other personal traits, that they should be expected to have "ROI" values nearer to the middle of the overall distribution of universities than should small colleges, which through pure chance (having, by luck, more high- or low-income graduates) are more likely to land in the top or bottom thirds of the list. Some degree of mean reversion may be expected, so the rankings of PayScale will lose some predictive value for future ROIs, especially in the case of small schools.

The comments I have made all concern the underlying PayScale report, but I think it is FiveThirtyEight that has not upheld the best standards of "data journalism." If that term is to have any meaning, it can't simply refer to "journalism" that consists of the passing along of other people's flawed "data" (especially when those people are producing and promoting the data for commercial purposes). Nate Silver earned his reputation, and that of his FiveThirtyEight brand, largely by calling out—and improving on—just this kind of simplistic and misleading analysis. It's sad to see his "data journalism organization" no longer criticizing superficiality, but instead promoting it.

Postscripts: 3/29/14: After I first posted this piece, I realized three things. First, I hadn't mentioned mean reversion originally, so I added it in. But it's a minor issue compared to the others. Second, I didn't make it clear that notwithstanding what I wrote above, I am 100% in favor of more good data journalism. I agree with Nate Silver and others that journalists (and everyone!) should be more aware of the data that exists to answer questions, how to gather data that has not already been compiled, how to think about data, and so on. A great example of silly data-ignorant journalism is the series of articles the New York Post has been running on the "epidemic" of suicides and suspicious deaths in the financial industry. The proper question to start with is whether there is an epidemic, or even a significant excess over normal variation, as opposed to  a set of coincidences that would be expected to happen every so often. Perhaps there is an epidemic, but I am skeptical. The Post (and other outlets that have reported on these deaths) skip right over this crucial threshold issue. Maybe FiveThirtyEight could address it and teach its readers about the danger of jumping to conclusions after seeing nonexistent patterns in noise. Third, and finally, I should have mentioned that FiveThirtyEight has on board some people who really do know how to think seriously about data (and do it much better than I do), such as the economist Emily Oster. I hope Emily's influence will spread throughout the organization. 3/30/14: I removed text in the original version that asked whether outliers like hedge fund managers had their incomes included in PayScale's calculations. They won't have too much influence, regardless, because PayScale is reporting medians, not means. My apologies for the inadvertent error. 4/5/14: I changed the number of colleges included from 1310 to "about 900." There are 1310 entries in Payscale's table, but many colleges are listed more than once if they have different tuition options (e.g. state resident versus non-resident). 4/7/14: I added links to the Krueger & Dale (and Dale & Krueger) economics papers that tried to estimate the returns from attending more selective/elite colleges. I knew about these papers when I wrote the initial post, but had forgotten who the authors were.

Addendum, 11/9/15: In an article at washingtonpost.com, Nate Silver is quoted as saying the following when comparing his FiveThirtyEight site to Vox, one of his main competitors:
I think the best five or ten things they do are terrific, right? They have some great people working for them. I think they also have a lot of less than terrific things … I know how hard my writers and my editors work to try and get get the facts right, to not always go for the hot take that you can’t really provide evidence for, right? To avoid errors and mistakes. And so, you know, I obviously have some skin in the game where I feel like if people are taking a lot of shortcuts and things that have the sheen of being data driven and maybe aren’t very empirical and aren’t very self aware, then, yeah, I guess I get really annoyed.
I think "taking a lot of shortcuts and things that have the sheen of being data driven and aren't very empirical and aren't very self aware" is an excellent description of the FiveThirtyEight piece on PayScale's completely misleading ROI analysis. And the piece remains on the site, as far as I can tell just as it was when I wrote this entry, with no corrections or updates or qualifications of its superficial and non-self-aware reporting. But at least it wasn't published on Vox!

58 comments:

  1. Really great post, Thank you for sharing This knowledge.Excellently written article, if only all bloggers offered the same level of content as you, the internet would be a much better place. Please keep it up!..

    Working at best online mba programs

    ReplyDelete
    Replies
    1. This bit of spam I will not delete, since it is so flattering. I love you too!

      Delete
    2. Just saw this -- hahaha! Take it when you can get it!

      Delete
    3. "Data Journalism" article covers approximately zero serious thinking about the meaning of the PayScale report, its data sources, and its conclusions. Since I am a dissertation writer or tutor and give 100% guarantee on expulsion of linguistic defects and logical flaws while providing online dissertation editing services with any subject from economics, finance dissertation, psychology, medical dissertation, health and care, philosophy and business dissertation at Dissertation Help UK, and I love to read review and opinion written by you in which every point are very easy to understand.

      Delete
  2. ‘Tipping Point States’ are those states that tip the outcome of the election from one candidate to the other. In each simulation run, the states are lined up from best to worst for each candidate. The states are marked off sequentially until the candidate reaches 270 electroal votes. The state responsible for putting the candidate over the top to 270 electoral votes is the tipping point state for that simulation run.
    coursework service uk

    ReplyDelete
  3. This is Awesome. Simply Awesome clicks. I'm not going to let a little time between posts stop me from seeing if there's anything new... :) Take care, and thanks for posting!

    Resume writer @ professional resume writing service

    ReplyDelete
  4. Do you know why i like your post. I like your post because of your great information. You share lot of information in your blog. Now i am very satisfy for your great post. Thank you so much. essay writing

    ReplyDelete
  5. As you said best experimentation won't always happen, but intelligent researchers have always try to approximate it by doing efforts. And as writer of Dissertation Service UK I like your blog and content.

    ReplyDelete
  6. College is the best life and i can not forget these days... While you choose any field your 1st proroty is do hard work. thanks and more information about study see infinitymotion.com here.

    ReplyDelete
  7. The postings on your site are always awriter
    excellent. Thanks for the great share and keep up this great work! All the best to you.

    ReplyDelete
  8. The introduction in essay helps you to highlight the main point of your work and show its value to others. Don’t miss a chance to engage your readers and use appropriate phrases for it. Click on essay and papers mistakes to read the useful article.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Really Great Platforms for students One Of The Best Article Thanks for sharing this update news with us Excellent writer. Limitless Eddie Morra Jacket

    ReplyDelete
  11. I believe that data journalism can be a best help for dissertation writing for many students. Being writer at a dissertation writing service firm, I guess this post is really worth of good knowledge.

    ReplyDelete
  12. A great piece of hard work and research. It is something to be proud.I have something for students It is the most reliable and authentic assignment help review platform where students come to share feedback on assignment writing service provider websites.@ http://topassignmentreviews.com/

    ReplyDelete
  13. Once you realize the purpose and structure of a top quality personalized statement, you'll locate that writing a personal statement isn't only simple, but a wonderful opportunity to express yourself and let your personality shine by way of. read more

    ReplyDelete
  14. i am no longer all that familiar with the specific qualifications on the newspapers of those universities, however i'm familiar with college lifestyles. faculties of that size can also have multiple way that allows you to make a contribution to the journalism of the network of the dissertation writers UK you may function the newspaper's correspondent from the pre-med application. or the pre-med branch would possibly have its very own inner book for college students and alumni.

    ReplyDelete
  15. According to my experience of working with dissertation writing services, I have heard that the course of Journalism in many universities is still outdated that needs to reviewed.

    ReplyDelete
  16. The comments I have made all concern the underlying PayScale report, but I think it is FiveThirtyEight that has not upheld the best standards of "data journalism." If that term is to have any meaning, it can't simply refer to "journalism" that consists of the passing along of other people's flawed "data" (especially when those people are producing and promoting the data for commercial purposes assignment Writing Services

    ReplyDelete
  17. Present situations http://downloadterrariumtvapk.com/ performs the good HD movie streams provider

    ReplyDelete
  18. Download mirror 1 for click and watch the latest videos and Movies for free our PC.

    ReplyDelete
  19. What are the problems with the PayScale analysis? First of all, it only makes sense to speak of the comparative return on an investment when the investors have a choice of what to invest in. If every person could choose to attend any college (and to graduate from it and get a full-time job), or to skip college entirely, then it would be meaningful to ask which choice maximizes return. This is what we do when calculating a financial ROI: we try to figure out whether investing in stocks versus bonds, or one mutual fund versus another, or one business opportunity versus another, will be more profitable. But colleges have admissions requirements, so not everyone can go to whatever college he or she wants.

    ReplyDelete
  20. Data is a given in journalism. ... That's why data analytics technology is so important to journalism. By analyzing large amounts of information – both structured and unstructured – quickly, can help journalism gain knowledge, make decision options almost immediately. custom assignment writing service us

    ReplyDelete
  21. Nutralyfe Green Coffee review
    This Green coffee is an excellent option not only to lose weight but also controls your urge of eating. The users of this coffee are definitely giving positive feedback as it improves their metabolism. Reviews are coming due to the long lasting effective result.
    Nutralyfe green coffee price

    Green Coffee reviews & price

    ReplyDelete
  22. Nutralyfe Garcinia for weight loss

    Nutralyfe Garcinia Cambogia Reviews
    Garcinia Cambogia Herbs is well known as HCA that is clinically proven for its effectiveness in weight management. This anti-obesity drug is also used for blood pressure control and maintaining the cholesterol level. It is pure, 100% natural and has no side effects at all.

    ReplyDelete
  23. thanks so mush for this amazing Great post Keep posting such kind of info on your page. Am really impressed by your blog.
    make money online



    ReplyDelete
  24. Furniture Moving Company
    Furniture is one of the things that we can do a lot of effort and time in order to carry out transportation from one place to another, to any place in the Kingdom or anywhere outside the Kingdom. The transport works from services that seem to us easy and simple but ultimately Of the services that lead to exposure to problems is very difficult to be solved from the breakage, scratching, damage and loss, the company tops the excellence of the most important and best companies that achieve the highest level of transport services for furniture and the agreement with the Gulf Cooperation Center in order to maintain the furniture against Any problem, if you are puzzled by the matter of purity We are sure that you now have the best companies specialized in transportation by relying on providing a number of distinguished services.
    شراء الاثاث المستعمل بجدة

    ReplyDelete
  25. Using feature-based help services can be a decisive moment for students because the support they get from this company is top chocie for them.
    Agricultural Engineering Assignment Help
    Python Engineering Assignment Help

    ReplyDelete
  26. We are here to help you in all your celebrations as we provide one of the most beautiful and craziest gifts for all occasions. Whether it is birthday of your special ones or Wedding Anniversary of a dear couple you can have excellent gifts from here. Get gifts for all festivals and special occasions, reach us now
    Buy Gifts
    Valentine Day Gifts

    ReplyDelete
  27. Thank You for Sharing such a wonderful knowledge. This one is a great post.
    Dissertation Writing Help

    ReplyDelete
  28. MyAssignmenthelp.com has introduced write my term paper service for the students in Australia, UK and USA. Term Paper is a mandatory academic task for any student, especially in US, UK and Australia.

    ReplyDelete
  29. I found your post so interesting.Awesome keep sharing.
    Animetv app

    ReplyDelete
  30. This comment has been removed by the author.

    ReplyDelete
  31. This type article is rare on data journalism. I want to do journalism thanks for sharing such an informative article.

    Want to buy a microwave check here whirlpool microwave oven price list

    ReplyDelete
  32. I found many interesting things on your blog, especially the discussion. Really good article. Keep it up!

    บาคาร่า
    บาคาร่าออนไลน์
    goldenslot
    goldenslot

    ReplyDelete
  33. A debt of gratitude is in order for offering decent data to us.
    earbuds and over ear headphones

    ReplyDelete

Note: Only a member of this blog may post a comment.