The ridiculousness of rankings

The ridiculousness of rankings– AKA -We are Number #1 and (sort of) Proud of it!!!!

Imagine participating in a marathon and finishing first – in record time, no less. The awards ceremony arrives, and the race organizers call you to the top of the podium, pronouncing as they place the medal around your neck,“Congratulations! You drank more water per mile than anyone else. You are clearly the best!!!!”

Why would drinking water matter, and who was keeping track? These are two questions you might ask, among others. Strangely, this situation is analogous to the one we find ourselves in right now with regards to a recent national ranking putting Keystone #1 among private schools in the United States.

In the past week, The Washington Post put out its annual “Ranking of America’s Most Challenging High Schools.” This year, Keystone was ranked #1 out of all private schools in America (and #23 overall). This lofty status is not new. We have been ranked in the top 3 among private schools for the past three years. This ranking must reflect our incredible program and the quality of the amazing kids who go here, shouldn’t it?

Unfortunately, we are sheepish about the Post’s placement of Keystone at the top. For, the ranking is based on data that is an extremely poor indicator of school quality or challenge, and, frankly, the methodology behind the rankings is extremely troubling. Building off of the analogy of the Marathon racer, the Post, in essence, is making its pronouncement based on how much water schools are drinking, and for its data is using the number of cups each runner touches as its basis – even if the contact only happens while navigating the trash of crushed cardboard on the street.

Here is what the Washington Post does to get its rankings.

It allows schools to enter their own data (without any attempt to verify it) about the following:

  1. The number of AP (advanced placement) tests taken by the entire student body in a given year
  2. The number of seniors graduating that year

The Post simply divides the number of tests taken by the number of seniors graduating to get its “Challenge Index.”

There are two fundamental problems with this ranking process. The first is that it measures the wrong thing, and second is that it does a very bad job of measuring it!

The creator of the index – a writer named Jay Mathews – says that he wanted to create a measure of how challenging schools are, with the intention of both recognizing and encouraging schools to be rigorous. This sounds great. He decided that the number of AP tests (or international baccalaureate) tests that each student takes on average would be a good indicator of this challenge level. This sounds awful.

Here’s the problem. Anyone can take these tests – even students who have done no preparation for them whatsoever. The only requirement is to pay the fee (about $80 per test). Saying that a school where students take lots of these tests is challenging is equivalent to saying that someone who falls down a lot must be good at gymnastics.

Further compounding the questionable nature of the index are two methodological concerns. The first is that schools self-report the data – there is no apparent independent fact checking. The second is that the index uses the entire student body for the numerator, and only the size of the senior class for the denominator. So, if in a given year there is one senior who graduates, and 30 juniors who each take an AP exam, a school would have an index of 30 (which is more than the entire number of AP subjects available), and in a year with an extra-large senior class, a school’s index would be misleadingly small.

The biggest concern with the system is that schools that rank at the top of the Posts’ index can have students taking lots of AP tests, but we have no idea if the students actually have the skills and knowledge to do well on them. A true indicator of a school’s level of challenge and rigor would look at how well students perform on the tests they take. Mathews doesn’t look at this at all.

In response to this misleading index, we have developed our own – the Keystone Index, which looks both at the quantity and quality of AP exams students take.

When students take an AP exam, they receive a score of 1-5. A score of 1 reflects a very poor performance (and is the score that a student can get for simply writing his or her name). Many educators consider a score of 3 as a “passing” grade, indicating that a student has demonstrated a college level understanding of the material. However, most colleges that accept AP scores for credit (and not all do – more on that later) require scores of 4 or 5. On most tests, approximately 20% of students nationally get each of these scores. At Keystone, about 50% of our students’ exams result in scores of 5, and about 45% result in scores of 4, and the mean score for the past three years has been 4.5. On average, our students graduate having taken 7 AP tests. These numbers are very, very impressive individually, and unsurpassed collectively. The challenge and rigor of Keystone are reflected in the combination of the amount of tests taken and the quality of work on them.

The Keystone index is determined by multiplying together the mean score and the mean number of tests taken by the graduating class. Typically, the Keystone index for our students hovers at 32. This index is not hard to calculate, but Mathews and the Washington Post have stated that they think it would be too hard for schools to do. Unfortunately, we don’t know the index value for other schools, because they do not publish how well their students score on AP exams. We do know that some other schools have students take more tests than ours do, and some schools have high mean scores but on far fewer exams. We know for a fact that our students have the highest Keystone index of any private school in the US, and we suspect that we would be at the top of the index list for all schools as well.

Thus, we agree with the Post’s ranking, but not at all with the reason for it!

There are some important things to point out about both the Posts’ comments and our own process. The first is that there are many ways for schools to challenge students, and there are a lot of programs that are quite rigorous but that do not have AP or International Baccalaureate courses. Like all indexes, the Post’s and Keystone’s are not fair ones to use for a lot of good schools. The second is that different school policies regarding exams can have a big impact on these indexes. Some schools require students to take AP exams if they are enrolled in AP classes, while others do not. The former would score higher on the Posts’ index (though not on the Keystone Index, as we don’t give credit for scores of 1 or 2). At Keystone we do not require students to take AP exams, which means our index score could actually be higher. In fact, in a typical year, about one-third of our seniors do not take any exams (even though they are typically enrolled in at least three AP classes), because they know by the time of the exams (May) that the college they will be attending does not accept AP credit. Without the incentive for credit, there is no reason to pay the $80 for each exam.

  1. of course, your school requires you to take it and/or pays for you to do so, which, apparently, is the strategy of some schools. And, some schools (including some “high scoring” charter schools) are built on a system of whittling down class size as students bow to the pressure of their programs. These schools have an artificially high index, as the small number of seniors will appear to be taking many more AP exams than they actually are. Among the reasons a school might pursue these educationally counter-productive practices? to move up on the Washington Posts’ rankings!

This is the damaging part of all rankings – that they matter for public opinion, and that they can impact a school’s decisions about program, often in ways that undermine a school’s mission.

Renowned author, Malcolm Gladwell (The Tipping Point, Outliers, and David and Goliath) wrote a wonderful piece regarding college rankings, the misguided methodology behind them, and the lengths to which schools will go to chase a higher score. It can be found online at: http://www.newyorker.com/magazine/2011/02/14/the-order-of-things

In this article, Gladwell points out that the major challenges of rankings are:

1. They do not measure what really matters
2. They rely on perception of those who
  • Have no knowledge of what they are ranking
  • Often have a vested interest in the ranking
3. They have a huge impact on public opinion, which
  • Leads institutions to change practices
    • Admissions trolling
    • Passing failing students
    • Focusing on quantifiable outcomes only (teaching to the test)
    • Drifting from mission
  • Leads institutions to game the system
    • Admissions trolling
    • Test scoring
      • APs
      • SAT super-scores
  • Leads institutions to cheat

Our caution is simple: watch out for rankings. They can be very misleading, and, unfortunately, destructive to the ultimate purpose of organizations being ranked.