I’ve commented before on the murky consequences of the four PairWise comparison criteria. For this purpose, the easiest way to think of the PWR is that it’s the Ratings Percentage Index with several modifiers tacked on, each of which is meant to capture some notion of what a “good” team’s performance should look like.
As I’ve said, the PWR and the RPI track closely together, so this way of thinking makes sense — the other pieces of the PairWise are really meant to tweak the RPI in specific ways.
Common opponents? This seems reasonable; if you want to decide which of two teams is better, one piece of the decision-making process might involve looking at how they fared against the same teams.
Head-to-head competition is even more obvious. If Team A beats Team B two or three times, it seems clear that we ought to believe Team A is better unless we have a lot of other evidence to the contrary.
And finally, looking at the record against “Teams Under Consideration” — the top 25 of the RPI — satisfies the concept that beating the best is what a good team should do, and that special attention should be paid to those games.
As always, though, the law of unintended consequences applies. Here’s the problem in a nutshell: each of those three things is already built into the RPI. If Team A beats Team B, the win and the loss go into both team’s RPIs, but also in the head-to-head criterion. Same goes for common opponents — those wins and losses are baked into the RPI, and then essentially get counted twice.
The worst offender, though, is record against TUCs. This has always been a cherished annoyance of mine, because it simply doesn’t work the way it’s intended.
How can I say that, you ask? Isn’t it obvious that wins and losses against “good” opponents should be singled out for extra attention?
Yes, it is obvious — so obvious that once again, the RPI already does this. Beating, say, Boston University or Notre Dame provides a big boost to a team’s RPI, the most important part of the PWR. Conversely, beating American International (no offense to AIC fans, but AIC was the bottom team in the RPI as of Sunday night) doesn’t do much for your NCAA credentials.
No, what the record-against-TUCs criterion does is reward inconsistency. For a case study, I give you the Northeastern Huskies.
Disclaimer No. 1: some of you may recall my use of Northeastern as an example in another article recently, a Bracketology blog post. I do not live in Boston, I did not go to NU, and I am not campaigning for or against the Huskies. Northeastern just happens to be instructive this season in terms of the PairWise.
Disclaimer No. 2: I’m about to criticize a specific problem with the PairWise Rankings. This does not imply that I think that the PWR as a whole should be scrapped. I like the PairWise. I like its objectivity and its transparency. But I’d like it even more if it made more sense than it sometimes does.
Back to our narrative. Northeastern has an overall record of 20-8-4 and an RPI of .5675, fourth-best in the nation. But NU is only seventh in the PWR as a whole, mostly because of its record against TUCs: 7-8-4, for a .4737 winning percentage against TUCs. This is causing the Huskies to lose five PWR comparisons against teams with better TUC records but worse RPIs.
Check those two records a little more closely: 20-8-4 and 7-8-4. Northeastern is a perfect 13-0-0 against teams not “under consideration” — those currently outside the top 25 in the RPI. These are the teams that NU is definitely “supposed” to beat, and it does.
Big deal, right? So NU won a lot against bad teams? It becomes more impressive when you consider that no other team in the country has done that. Every other team has at least one little blip on its resume against non-TUC teams. Boston University lost to Providence. Notre Dame posted ties against Lake Superior State (twice) and Western Michigan.
I’m not saying this to criticize those teams, because you can’t win ’em all. And that’s exactly the point. According to some, “good” teams prove themselves by doing two things: you “beat the teams you’re supposed to” and then “win the tough ones.”
When you put those two together, you’re basically asking for perfection. Beat the teams you’re supposed to — the bad opponents. Win the tough ones — the good opponents. Any loss is a problem because you’re breaking one of those two rules.
The rule Northeastern breaks is about “the tough ones.” As we already mentioned, Northeastern is below .500 in games against TUCs despite being perfect against everyone else, and that’s damaging the Huskies’ PWR stats.
So let’s fiddle with history a bit. On Oct. 31, Northeastern lost to New Hampshire, and a couple of weeks later, on Nov. 16, NU lost to BU. Let’s imagine the Huskies had won those games. Now, let’s balance things out by taking NU’s two wins over Merrimack on Jan. 9 and 10 and making them losses instead.
Look sharp, now. Our revised version of Northeastern has played the same schedule, and has the same 20-8-4 record overall. For that matter, it has the same Hockey East record and is still in first place in the league, although the PWR doesn’t pay attention to that. All we’ve done is give NU two more wins against top competition and two more losses against Hockey East’s last-place team (for completeness, Merrimack would be ninth in Hockey East rather than last had the Warriors won those two games, but you get the idea).
Is this new version of Northeastern more impressive to you than the real Northeastern? Beat BU and UNH, lose twice to Merrimack. Kind of a wash, isn’t it?
Well, the PairWise doesn’t think so. This fantasy version of Northeastern has almost the same RPI as before (.5675 vs. .5676), which isn’t surprising — same schedule, same record, and those are the pieces of the RPI. But now two more of NU’s wins are against TUCs. The result? This version of Northeastern is third in the PairWise, as opposed to seventh.
Now let’s go crazy with our little experiment. On top of what we’ve already done, let’s trade out some more wins and losses — how about Northeastern loses to Alaska-Anchorage, Bentley, Rensselaer and once more against Merrimack (all non-TUCs), and let’s give the Huskies wins against Princeton, Minnesota, Vermont and another against BU (all TUCs).
Four more losses balanced out by four more wins. The same 20-8-4 overall record, but now produced by wild inconsistency. This version of Northeastern can beat anyone and lose to anyone with equal aplomb. There’s no telling what you’ll get on any given night.
And the PairWise loves it. This Jekyll-and-Hyde Northeastern is No. 1 in the PWR thanks to a spectacular record against TUCs of 13-2-4 — even though each of the TUC wins that we added to NU’s record was counterbalanced by a “bad” loss. (To be fair, the extra wins against BU are important in the direct comparison against the Terriers, but that’s only one comparison. The TUC record applies to every comparison.)
This is an extreme example, but the problem is real. Something like this happens every season, though less dramatically. It’s happening right now — not just to Northeastern, but to Ohio State, for instance — and that’s a concern. In short, the PairWise would rather have you beat a good team and lose to a bad one than vice versa.
That makes very little sense, and that is the point.
