Search This Blog


Friday, October 2, 2015

Limitations of Ranking

My previous post on the ranking framework has led to disappointment among several of my friends whose views I take seriously. And the common argument is that this is far superior to anything that we have, and yet I have chosen to criticize it, which is not quite right on my part. Fair enough. I do accept that the ranking framework is far superior to what various business houses have been doing over the last many years.

However, we started off this path by saying that QS/THE do not understand our universities and we need to somehow showcase the quality of our top institutions which we believe is significantly better than the ranks they are being given in those rankings. I have never understood how an India-specific ranking will showcase the quality to the rest of the world. How we can claim that we deserve to be in top 100 by having an India ranking. Yes, I have seen the argument. I good Indian ranking will spur the competition to get better. We will no longer be able to get a higher ranking by giving false data or by bribing the reporter or by buying ads in the private ranking. And this improvement will help us get into top 100. But notice that this argument actually admits that we need to be in top 100 based on what QS and THE decide as ranking parameters, and if that is the goal, then there are better ways to spur that competition than a government ranking.

Ranking are useful information for the stake holders, and I have written many blog articles in the past recommending that our institutions should take ranking seriously. However, I am scared of a situation where ranking is the only information that a stake holder uses for taking important decisions. There are serious limitations of ranking (or most measurements of quality) and it is difficult to assume that a common man would understand those limitations. So far, the common man was taking multiple inputs not because they understood the limitations of ranking, but they had an inherent mistrust in a ranking by private business house. But if we now have a government ranking, and granted that it will give inherently better information than private sector ranking, the common man is likely to use this information as the only or primary information, since it does not understand the limitation of the ranking. And while the current decision making has serious flaws, the new decision making can only have bigger flaws.

The Framework talks about ranking to be available before April 2016 so that these can be used by students and parents to take better admission decision. So the most important stake holder for these rankings are the prospective under-graduate students. What is the ideal decision for such a prospective student, and let us see how close the advice is to that ideal decision of the student.

Let us assume that the student is interested in getting education which will provide for a good career and a resultant happy life. And the admission decision is that among all the options for higher studies available to me, which one should I choose that is more likely to give me a good career and a happy life.

First of all, asking that question at the end of 12th class is quite ridiculous. If the same question was asked (or rather was allowed to be asked) a few years earlier, the decision may have been completely different. Note that the ranking can only tell you whose graduates are having a good career (even that, as we will see below, is not being said by the ranking, but let us not get ahead of ourselves). It can not say what would be good for an individual student. Success in career does not depend solely on the alma mater. It also depends on, to just give one example, whether you have an interest and passion for the kind of job you are working in. By suggesting that X is number one engineering college, you are really suggesting that for all disciplines this is a better college. So for all students, irrespective of their passion, should prefer this college. This is clearly nonsensical. No one could be best in everything. Of course, the framework says that they will use this framework for ranking not just colleges but also individual disciplines.

But even discipline based ranking does not help beyond a point. A student who is deeply passionate about research versus a student who is deeply passionate about entrepreneurship in largely the same discipline should probably go to different institutes. A student who is studying CS as a tool to be applied eventually to another discipline would probably require a different kind of program than someone who is studying CS to get a technical job in CS area, who would probably require a different kind of program than someone who is studying CS only because it hones your skills of abstraction, analysis, etc., which he wants to apply in management, finance, and other "non core" areas. Some locations may be more conducive for some and less conducive for others based on factors such as language of discourse in the hostels. Discipline based ranking will not help here. One has to look into the programs more deeply. What courses are on offer. What flexibility the program offers. What kind of culture and environment is there.

We already have a problem on hand. A very large number of students and parents only look at "placement statistics" to decide and then suffer. But there are many who still look for options, ask questions. That number will dwindle further if there is a government ranking out there.

Many of the readers will argue that most of what I have written above is not relevant because a 12th class student is not going to do research on colleges, does not know his passions, does not know whether would want to become a manager or a scientist, and is only looking at information on where would a typical student more likely to succeed. So individual differences are not important. And those few who know their passions so well are also aware of the limitations of the rankings and will do their own research.

Fair enough. But does ranking even give that statistical information. First of all, we will need to define "success in career and happiness in life" to say which institute is really causing more of its graduates to get there. The problem is not just in the definition of success, but how do we get data, and whether data is relevant for this batch. Assuming ideal information of all kinds, one could possibly look at alumni who graduated 15-30 years ago and have some way to figure out what percentage of them are successful. (Can we really have a binary decision here.) Assume you can have this information. But is that information relevant today. A college may be doing some magic 20 years ago which has caused great success to the alumni in their careers, but may have gone downhill since then. There may be new institutes who may turn out to be better 20 years from now in this regard.

And, therefore, we look at not how successful the alumni have been but assume certain parameters either correlate very highly with that success or cause that success, both without any scientific study. So the prospective students and parents assume that last year's placement statistics is the best indicator of future career success for a student who is joining this year and will graduate after 4 years. And a few lone voices like me would claim that quality of education causes that success, and therefore, a prospective student should look at not the placement data but do research on various colleges about things that affect quality of education.

And, of course, I would claim in support of my view that even if placement of 2014 was a good predictor of success in 2050, most colleges would give out wrong information, and most students and parents would look at wrong data (like top placement and average placement rather than percentage of students placed and median placement) and these two wrongs combined would ensure that you are really taking a lottery ticket.

Since the government ranking is created by professors, you would of course see a bias and they are closer to my views than the views of the students/parents. Quality of education is important. But of course, we will also give some decent weight to post-program outcomes (including placement, but as professors we would also like to see how many of graduates do well in exams like GATE, and go for higher education). So the placement per se will be a small factor.

The problem is then how do you judge the quality of education. Again, we don't know how to define quality of education, and we will assume that certain proxies for them will somehow be good predictors. So, a faculty-student ratio would be a great proxy for quality. Number of faculty members with PhDs would be a great proxy for quality. Why is faculty-student ratio a great proxy. Well, it is likely to result in smaller class sizes and it is assumed that smaller class sizes result in better delivery of education. Then why not just look at class sizes. Would a system (like in IIT Kanpur) where one faculty teaches 400+ students while a large number of faculty members teach less than 10 students a better model than a system where everyone teaches a 50-60 students. Would a system where a faculty student ratio is 1:15 but students do 6 courses a semester better compared to a system where the faculty-student ratio is 1:16 but students do only 5 courses a semester or a system where the faculty-student ratio is 1:17 but students do only 4 courses a semester.

Is it really true that in the Indian context, PhD faculty teaches better than non-PhD faculty on an average. Remember, this ranking is supposed to reflect Indian realities. The Indian reality is that the quality of PhD sucks big time. Only a few top institutes are able to find PhD faculty from good places. Others are hiring PhD faculty who know much less than BTechs from good places.

Is it really true that citation index is a good predictor of quality of education. Is it even true that a good researcher will be statistically a better teacher. May be it happens in IITs, but it does not seem to be happening across the country. And I hope this ranking, though created by IIT professors, is not meant for IITs alone.

How does inclusiveness improve quality of education. Inclusivity is a great social and national goal. I must applaud all those who care for inclusivity, but not all national goals imply improvement in quality of education. My fear is that this is beginning of politicization of ranking, even before they start. The same argument can now be extended to cover other national and social goals as well. Are you actively participating in Swatch Bharat Abhiyan. Nobody can deny that improving cleanliness should not be applauded.

In general, the education experts tell us that having diversity inside a class improves the quality of education. And it is good that the framework looks at diversity. But there are three kinds of diversity - in-state versus out of state/international, gender diversity, and having people from economically and socially disadvantaged classes. But let us look into the details. The maximum marks you get for geographical diversity is when you take 100% of students from outside the state, none from instate and none from a foreign country. Should diversity mean not having any student from the society which is hosting you and nurturing you as an institution. If geographical diversity is a great thing (and I believe that it is a great thing), would the Government free NITs of the in-state quota. Let them decide how they want to compete in this race, and tie their hands behind their back. If the government does not do this, then it is forcing NITs to get a poor rank. Is this fair to NITs.

Also, should diversity be counted only in terms of in-state versus out-of-state. Should we have some credit for number of different states represented on campus. After all, an institute in Delhi having students from Gurgaon and Noida would meet the diversity requirements but is that really helping the quality of education. And given the political nature of these factors, I am sure one day someone will say that presence of North East students must be part of the ranking (which, by the way, would actually improve diversity in most campuses) explicitly. 

To meet diversity goals in case of genders, is 50-50 the ideal for improving the quality of education. It is probably a great social goal, but I would guess from the quality of education perspective, having a substantial presence of both genders would be desirable, but not necessarily 50-50. So may be some mismatch should be acceptable, say 40-60 or 30-70, either more men, or more women. Again, the question will be that if this is the goal of the society and the government, would they allow IITs to do something (anything) to improve this ratio.

And having 50% students from economically and socially backward backgrounds, again, is it furthering the goal of diversity and quality of education, or a social goal. Note that there is no definition of economically and social backwardness. This is going to be a political hot potato. Can I only count SC/ST/OBC (Non creamy layer), or can I also count Muslims and anyone else whose income is less than 6 lakhs. If you allow all those who are non-creamy layer, irrespective of their caste and religion, then every single institute in this country, including some of the expensive private colleges would have the desired 50% or more people from this category. So why have this at all.

On the other hand, religion diversity is important for improving quality of education, which is not mentioned, clearly because that has political overtones. Another diversity which is extremely important for quality of education is having students study different subjects. So a university with many more departments should get some credit compared to narrowly focused universities. But do you really think that IITs were going to include that parameter in ranking.

In one of the curriculum workshop, I heard one very famous Computer Scientist say, that the number of courses in the curriculum is one of the strongest predictor of quality of education. The lower the number, the better is the quality. If the student is being asked to learn 6-7 courses in a semester, the outcomes will be poor. And he mentioned a large number of quality CS departments where he showed that the top departments typically have 4.5 courses per semester, good departments have 5 courses per semester, and then there is downhill. We could use that (and it not only gives students time to learn each subject, but the costs are reduced, the class sizes are reduced, more assignments can be graded, etc.). We don't seem to have such a simple predictor in our ranking.

The point of all this is not that the ranking framework is poor. Of course, it is much better than to ask people in diverse fields to name the top X colleges, the perception of non-experts seem to dictate the current rankings in India and even abroad. The point of all this is to understand that rankings are based on proxy variables and not a direct measurement of quality (since there is no direct measure). Those proxy variables are disputed, and have their own limitations. And hence rankings have limitations. This is the point that all stake holders need to understand. Rankings are just one more input and can be used to short list your potential places to study but then you must think of your own interests, preferences and personality, and do your own research of those colleges.

My concern with this ranking, as I said in the beginning of this article, is that people will have so much trust in this linear ordering that they will not do even the limited research that happens today. I am assuming that a poor quality private ranking supplemented by whatever little research goes on is better than better quality government ranking with no research.

The biggest advantage of this ranking process will be that reliable data will be available at a common portal for most good institutions (hoping that most of them would participate in this ranking). Not only that data will be available, there would be a system to challenge any information and hence colleges would hopefully provide honest data. And hopefully, the systems would be strong enough to ensure compliance. That is, if it is known that wrong data has been given then the college could be barred from ranking. And hopefully, there will be an interface where I could search, order colleges based on my queries.

What is even more disappointing is that the behavior that this framework and the government is hoping to encourage through competition for better ranks could have been achieved even otherwise. First of all, just the publication of this report will encourage the private players to modify their rankings in the right direction. Second, a simple way to do this would be to have NAAC dictate to colleges that they keep updated data on NAAC portal, otherwise they lose accreditation, and allow challenges to that data, similar to what the ranking framework is proposing. Allow people to search, and order accredited colleges, and so on. NAAC could even allow those colleges to upload their data where a formal accreditation has not been done. So the key is that we have good quality data available, which can be easily searched and colleges can be ordered on multitudes of queries. You really don't need a single government approved formal linear ordering to help the potential students and parents. After all the data for ranking and the data for NAAC have huge overlap. So avoid duplication of efforts. Avoid linear ordering. Avoid government approval of that linear order. And yet, give all that quality information to those who need it, and let them use it in interesting ways. In fact, I can see many people will do research on that data, come up with multitudes of lists, share them on the Internet, and that would be great since students and parents would then understand that the ranking depends on your perspective and encourage more research by them.

Frankly, the only reason to not use data with NAAC and NBA can be that IITs don't want to deal with those agencies. This could have been a great opportunity to overhaul accreditation, but IITs strong resistance to be compared with other institutes has done a great disservice to accreditation services in this country and by extension the whole higher technical education sector in this country. And this ranking framework is another outcome of that attitude of IITs.

The government could have done other things as well. Whatever it believes as parameters of good quality, it can incentivize colleges and universities to improve on those parameters. Government has all the power in the world to align incentives with desired outcomes, without forcing the institutes on those outcomes. You will get more grants if you do this or that. You will not get large projects unless all your data is with the central portal, and so on. But this will amount to giving autonomy to colleges and let them decide what goals are important to them vis-a-vis support they can get for those goals. Government does not work in those ways. It will dictate the goals and then have a complicated process to judge whether those goals are being achieved.

And finally, is there anything positive in this. Of course, there is. And perhaps I should support this framework just for that reason. It will allow lazy HR folks to take better decisions. Currently, if you see how many HR folks decide which colleges to go for campus placement (assuming no corruption), it is like let us go to IITs, NITS, IIITs, and BITS, if we have to go to 50 places. If we have to go 10 places, then old IITs, and a couple of places with whom we have friendly relations. It does not matter that a new NIT would have provided far poorer quality of education than some of the private colleges. It is just to avoid doing any research on whether the education in those institutes align with the requirements of the company. With this ranking in place, hopefully, some lazy HR manager will be able to say, let us go to top 50 places. (He will still not do research of his own.) And these top 50 would be a better list than set of IITs, NITS, IIITs, etc. It will, therefore, provide a chance for private colleges to prove that they too are providing quality education. They already appear in private rankings, but those rankings are not trusted. But what will happen if a deemed university not liked by Dr. Tandon Committee appears in the top 100.

To summarize, the ranking framework will certainly be a better predictor of quality than the current private rankings have been. But they do not do anything for our universities to appear in top 100 of international ranking. However, the government backing a linear order of colleges will have so much trust among the stake holders that they will not understand the limitations of the rankings and that will not be good for the decision making. We need rankings but in private sector. And we also need to do things to improve our ranks in international rankings. And most importantly, we need to do all this while fully recognizing the limitations of the rankings.

Wednesday, September 30, 2015

The ranking framework

So, finally we have the ranking framework, not for all types of institutions, but that too will happen soon. I find this obsession with numbers very interesting. But the framework has disappointed me. I was looking forward to a framework that would make it possible for all of us to claim that the top five engineering universities in the world are Ayee Ayee Tees at Kalyanpur, Guindy, Hijli, Hauz Khas and Powai, ahead of MIT, Princeton and Harvard. Alas, the framework is not sufficiently Indian, and may not yet result in top five slots coming to India. But if we follow the implementation strategy given in this article, we may still have a chance.

The framework has got a few things right. We should look for geographical diversity in terms of people from different states, and not judge them by the color of their passports alone. You see the US universities invariably have a lot of students from within the state they are located in, while in our Ayee Ayee Tees, everyone is from Kota or Hyderabad. Hauz Khas is an exception, as many students come from Delhi. They will have a reason to crib about their 5th rank. May be they should bribe all coaching centers to set up their teaching shops in Gurgaon and Noida. Working harder on teaching and research would not be as effective in improving the rank. But the old five should not be complacent about their ranks. Once we have the 40th institute in Daman fully functional, they will have 99% of their students from out of state, and their ranking will go higher.

In terms of graduation outcomes, we must insist on GATE performance as the only criteria of quality of graduates. Let Stanford graduates get 0 unless they can take the new Air India flight to Delhi and perform well in GATE. This will also ensure that Air India starts making profits. Even then, they will not be able to compete with Ayee Ayee Tees, unless we transfer the technology of impersonating in such exams to them. I can see a lot of business opportunities here. (Of course, we will have to figure out how to incentivize our own students to apply for GATE.)

The placement should be considered six months to a year in advance of graduation. No other place in the world would have such a crazy system of placement before the graduation, and we will win hands down. For further cementing our position, we should convert the offered salary into USD based on a flawed but useful PPP model. So a Rs. 10 lakh offer becomes US$ 1 lakh salary.

The inclusiveness must be checked only through reservations. If you don't have reservations for your local minorities, you get a zero on this factor too. But we got to think seriously about these marks for women share in the student population. These yankees you know are not family persons, send their women to college, not take care of family. Bad culture. May be we should give more marks if there are less women on campus. Promote Indian culture through these rankings.

The minor issue of student faculty ratio can be resolved by collaboration. The faculty of two institutes can be shown as recruited by both (and they can actually travel to the institute if there is any inspection). And remember, there are marks for collaborations too. In terms of lab infrastructure, we must insist on having a minimum number of PCs for each 100 students. Rest of the world has moved to bring your own device. We should also insist on all the AICTE guidelines being followed, including having an English language lab.

So, as you can see, there is enough scope for working out the detailed implementation in a way that the top five slots are occupied by Indian universities.

And, if by chance, some foreign entity shows up in the top five ranks, we can always give them a zero in perception. Hey, these are our rankings and we will decide who gets what.

On a more serious note, I think the problems with the higher education are far too obvious and the solutions are also far too obvious. Having an Indian framework for ranking does not help us claim that we are better in the world than what QS, THE, etc. are claiming. This whole business of Indian ranking system started off by saying that these foreigners do not understand our issues and our strengths are not given adequate weight. But would having an Indian ranking enable our universities to jump into top 100 of QS.

It would be of some help to students and parents during the admission time, but shouldn't we let the private sector come up with those rankings, instead of government controlling this. Yes, I am not happy with the private sector rankings like those of India Today. But the solution is not for the Government to compete with India Todays of the country, but to encourage them to improve their processes.

Saturday, August 29, 2015

OROP: Monetizing Respect

This is the first time I am writing anything on a topic other than education and Railways. This is because I have been very intrigued by this whole issue of One Rank One Pay. I have read a lot of articles (mostly from retirees of armed forces, as to why they should get OROP) in recent times and have asked my facebook friends to help me with better understanding of the issue.

Almost all articles on OROP will argue that it is such an obvious thing that there need to be no reason given. Of course, if it is so obvious that people retiring at the same rank across decades should today get the same pension, then shouldn't it be for civilian employees as well. A lot of authors guess that this question will be coming and try to answer it as follows.

The soldiers in army (and equivalent in other forces) retire after 20 years. The officers also retire at different points in time, but mostly after 20-30 years. They have given the best years of their lives for the nation. They do not always get jobs after they leave armed forces, and therefore, all this must be factored in while deciding the compensation package, and they must have a better compensation package than what it is right now, and having higher pension would be the best way to make it a better compensation package.

But, didn't you give all this in your representation to 6th pay commission, and for that matter to 5th pay commission, and now to the 7th pay commission. Why do you believe that the successive pay commissions have not already taken all these arguments into account while recommending your pay package.

The argument then becomes that we do not trust pay commissions to have done a fair job. The IAS lobby controls the pay commission. They always ensure a better deal for themselves, and give a raw deal to us. And look, they are not even implementing what pay commission recommended that a percentage of recruits in armed police forces may come from ex-servicemen.

The last line is somewhat of an argument. It is fair to assume that the pay commission would have assumed that many ex-servicemen (not officers) would have post retirement jobs, and decided their compensation accordingly. If the pay commission knew that the probability of getting a decent job will be lower, most probably it would have recommended a higher compensation package for them in some form. OK. So the jawans can get a slightly higher pension based on this logic. But why officers who mostly are able to get post-retirement jobs in private sectors. If OROP demand was only for jawans, it would make some sense based on this argument. And here too, we need to get higher compensation, and it can not be claimed that OROP is the only way to achieve that.

So what about officers. Well, it really come back to pay commissions being biased. They have really not factored the hardships, early retirement and slow promotions while deciding the compensation package.

The problem with this argument is this. Who decides whether the compensation is adequate or not, if not the pay commission. Should compensation for millions of people be decided by public protests? There has to be a better process than that. I don't know if it would help to have a person belonging to armed forces as a member of the pay commission. It is already chaired by a Judge of Supreme Court, whom we could easily consider as neutral between civilians and armed forces.

From the arguments I am reading, there is a certain level of discomfort. For example, arguments like my pension fixed long time ago is not enough today. This gives an impression that the pensioner is still getting the same pension as he was getting a few decades ago. The reality is that all pensions are protected against inflation and they are also given a jump with each pay commission. So each pensioner is getting a better pension (even after taking into account inflation) today than 10 years ago. And if the argument is that a colonel retired 20 years ago should have the same life style as a colonel retired yesterday, why shouldn't a professor retired 20 years ago have the same life style as a professor retired yesterday.

Some people have shown numbers that someone who works for 40 years and lives for another 20 years (in which s/he is free to work, if health permits and can find a job) gets a higher total compensation compared to someone who works for 20 years and lives for another 40 years (in which s/he is free to work, if health permits and can find a job). And this is supposed to be somehow unfair. Frankly, I don't understand. If someone works for 20 years less, why should the total compensation be about the same for him compared to someone who has worked for 20 years extra.

And if the argument is that armed forces need to be compensated better for shorter tenure, slow promotions, etc., and even if we agree that pay commissions have all been biased, why not increase the pay and perks while in service. What is the argument for OROP. After all, the employer should be able to structure the compensation package in a way that will attract the best people to do the job at hand. And if the employer believes that it is better to increase pay than pension, it should be possible.

The problem is that most people are not really looking at compensation for perceived biases. If everyone in Armed forces is given a couple of extra increment to bring that so-called parity with the IAS types, that only means a few more peanuts for them. (Yes, it would increase their status in the government hierarchy, and that is important. But monetarily, it really does not make much of a difference.) Also, that is only for current employees.

And if it is a matter of compensation, and we want to redraw today the compensation package of someone who joined army 50 years ago, why not just increase pension. Can it be 60% of the last pay drawn (as modified by successive pay commissions), instead of 50%. Why insistence on equal pension. All the arguments are for higher compensation package. I have not understood why this particular way of increasing the compensation package.

Demand for increased compensation should ideally be based on arguments like the following:
  1. We are finding it difficult to recruit talent despite our best efforts. Can we offer higher package.
  2. Someone else who is doing similar tasks, with similar efficiency, in similar operating environment is getting higher pay.
  3. There should be a certain minimum level of compensation for any employee (the idea of minimum pay).
In case of OROP, I am not sure what the argument is. (I am sure there can be more arguments than the three that I have stated above.)

To summarize, there is some argument (based on the 6th pay commission recommendation that was not implemented) in favor of increasing compensation, including pension, for non-officers. There is some argument  (based on the assumption that 6th pay commission was biased and 7th pay commission will be biased) for increasing compensation for officers, but ask new officers to join New Pension Scheme. But I am yet to see an argument in favor of equal pension for same rank. I welcome my readers to inform me of articles where such arguments have indeed been given. Of course, if early retirement is the primary issue, we should spend even more on skilling those in uniforms for their post-retirement careers, and other steps to improve their chances of decent employment.

But why are we not seeing articles in media opposing OROP or even seeking clarifications like the one I am seeking in this article. If the OROP is such an obvious thing to do, then what is government waiting for. We can't be thinking of a few thousand crores per year, if those are the legitimate dues of people who defend our borders. As someone said on my facebook discussion, war is expensive and to maintain war machine is expensive. We must be willing to pay that price for independence.

This is what I believe is happening.

Armed forces are arguably the most respected institution in the country. And in the era of cross-border terrorism, not many are willing to argue or discuss military pay. Keep them happy. Give them anything they ask.

There is also a fear that questioning the military pay would label one as unpatriotic. (And if one is careful in reading this article, I am not questioning military pay or perks or pension, even suggesting that they be increased, only seeking answer to the basis on which such a package should be decided. I certainly don't want to be labeled as unpatriotic.)

This is more so when both Congress and BJP have already promised OROP, and it is obvious to everyone that sooner or later, there would be a substantial increase in the compensation package, irrespective of any arguments. Why be considered unpatriotic when the deal is almost done and one would not have any influence on the deal. (But academicians always want to know the answers even when they have no influence.) By the way, I believe that since it is almost a done deal, we must implement it at the earliest, and close this chapter. Every day of this protest is affecting the country negatively.

The veterans on the other hand have figured this out. The public has huge respect for armed forces. Also, the public at large has strong negative feelings about the bureaucracy and the politicians. By making this a public issue and essentially blaming the IAS and politicians for the mess (and not waiting for the 7th pay commission report), they have a much better chance of improving their compensation package.

But this, sadly, is monetizing respect.