New Macleans Ranking of Canadian Law Schools

There were some changes to the methodology used this year from last year, with the two biggest changes being (i) the addition of a new set of criteria relating to law professor positions secured by a school's LL.B. or JD graduates; and (ii) an expansion of the number of journals in which citations to faculty publications were counted.  Both of these changes ought to be regarded as significant improvements, in that they capture in new ways important, reliable, and publicly-verifiable information about a school's graduates and faculty scholarship, respectively. 

A problem common to each, however, is that improvements to a school's graduates or faculty will have an effect on each measure only with a dramatic lag.  Hiring strong young faculty, for example, will temporarily hurt a school's ranking in the per capita citation measure since young faculty will only begin being cited frequently a number of years after joining a faculty, but will immediately be added to the denominator.  A law dean who is acutely sensitive to this measure might be disinclined to hire promising young scholars and might prefer to try to poach scholars from other schools.  Hiring established scholars laterally from other schools would permit a much more immediate improvement on this measure.  Graduating stronger LL.B. and JD classes will also take some time to show up in the law professor counts.  It was a nice touch to include a per capita measure of both the total number of LL.B. and JD graduates of a school who are law professors, as well as the number of LL.B. and JD graduates of a school working as law professors at schools other than their alma mater (since it is well-known that schools tend to have a preference to hire their own graduates).  It may be unnecessary, however, since the correlation between the two measures is 0.96, which suggests that breaking the data out in this way doesn't add much information.  On the other hand, this blunts some of the incentive for schools to game the system by loading up on their own graduates, though I suspect that this would be unlikely in any event.  Perhaps in future years it would be better to count only LL.B. and JD grads hired by other schools as law professors, since this measure cannot be gamed directly (though a school would retain an indirect preference to hire one's own to preclude a rival from claiming an additional academic hire).

It seems there is still a problem with the elite firm hiring and national reach indicia.  It's unclear why the quotient is important, rather than the per capita number of graduates at more than the top three drawing elite firms.  Consider the following:

It seems to me, at least, that the per capita measure provides a more intuitive sense of the depth of influence of a school's graduates than the quotient measure, which measures the concentration of lawyers at the non-top three drawing elite firms versus the total number of associates at elite firms.  Better to get a sense of how broadly a school's graduates are placed at elite firms by using the per capita measure, i would have thought.  As it stands, the national reach data correlates somewhat poorly with the other measures of law school strength: with elite firm hiring (0.315), with SCC clerkships (0.091), with faculty citations (0.535), and with law academics (all) (0.188), and law academics (non-alma mater) (0.369).  If instead the per capita national reach measure was used, some of this weakness would evaporate, with correlations moving to: with elite firm hiring (0.821), with SCC clerkships (0.376), with faculty citations (0.632), and with law academics (all) (0.527), and law academics (non-alma mater) (0.661).  And this gain in consistency with the rest of the indicia does not come at a dramatic cost, actually; the correlation of the (in my view, flawed, quotient measure of national reach) with the per capita national reach measure is 0.754, which suggests a fairly strong relationship.  So the higher correlations with the other indicia is not being purchased at the cost of losing a ton of information from the quotient measure of national reach.  The difference is that the per capita national reach measure is not affected by the wonkiness of the quotient measure (which rewards not placing many students at elite firms, but placing students at many elite firms).  The per capita measure does not penalize as perversely placing an additional graduate at an elite firm that is a top three drawer of that school's graduates.  The reason is that it doesn't add a unit to the denominator (as in the quotient method), but instead is associated with the opportunity cost of not adding another unit to the numerator of the per capita measure.  I would have thought that this incentive was a better one to engender for career development offices, and that the per capita national reach measure more intuitively captures a law school's national reach.