Macquarie and the rankings
Written by Steven Schwartz on November 12th, 2007
Macquarie’s place in the latest Times Higher Education Supplement-QS World University Rankings has been the subject of some comment, including a front page article in the local newspaper. The rankings showed Macquarie has dropped from 82 to 168 in the international rankings and from seventh to ninth among local universities.
Should we be concerned? While it’s always nice to go upwards in the rankings there are valid criticisms of the THES-QS including that it uses a relatively subjective methodology, a view reinforced by Professor Ian Chubb, Vice-Chancellor of the ANU, the THES top-ranked university in Australia. He commented that “I think it shows that if you choose your indicators, you can get a number of Australian universities in the world’s top something-or-other”.
I think the easiest way to understand what happened is to imagine a group of 100 people queued up according to increasing height. The shortest person is number 1. Then you order the same group according to age and you get a different order. The first person becomes number 80, say. Does this mean the person dropped 80 places? No, it simply means different criteria of measurement have been used. There has been a change in methodology with the rankings that has impacted adversely on Macquarie, but over time this should be ironed out.
And judging by a wide selection of independent ranking systems, including those of Shanghai Jiao Tong and the Melbourne Institute, and national awards for excellence such as the Carrick Awards and Eureka Prizes, Macquarie University has enjoyed an extremely successful year in 2007. One blog respondent claims that “the message from ‘the market’ is very clear … MU is going down, down, down”. But on most credible measures, Macquarie is going up, up, up.
It’s also worth pointing out that some data used by THES-QS refer to 2005 and to data in 2006 that were determined by decisions made in 2005, so they are hardly a contemporary market reaction. For example, the data for international faculty numbers provided this year were for 2006, while the previous year was for 2005. Also, we do not know what we would have ranked last year using the current formula
Regardless of what the rankings show, I want to assure you of our determination to pursue the University’s strategic plan outlined in the Macquarie@50 document – see here and here.
The main goal is to ensure that, by age 50, Macquarie is among the top eight research universities in Australia and one of the top 200 in the world. To achieve this goal, the plan calls for the University to increase its research activity while also delivering the highest possible standard of teaching. The plan calls for a new management structure, new resource streams (such as fundraising) and many changes in the way we conduct our day-to-day operations.
Over the past two years we have begun to institute the changes necessary to reach those goals. We are on the way, but much work remains to be done. We need to stick to our plan, be determined to deliver excellence in research and teaching, and we will achieve our objective.
Steven Schwartz adds:
Thank-you all for your feedback on this topic. Some good points have been made, and some strong opinions expressed.
I’d just like to make a few points in reply: the THES-QS ratings were dramatically changed from last year with many universities around the world moving 30, 40 or more places - which is just not credible to anyone who works in universities. Also, it is not unreasonable to point out that changes in the THES-QS methodology impacted negatively on Macquarie among others.
Further, the data behind the THES rankings are not public and cannot be examined. Even by their own admission, their rankings are not based solely on academic criteria but on the subjective opinions of unknown persons being asked unstated questions. The Shanghai ratings are public, can be examined on the web and are objective rather than subjective. Thus, it is sensible to want to rely on the Shanghai ratings but not the THES.
The previous message suggests that the change results from a change in methodology that has caused the reranking. If it is known which component or weighting change lead to the significant downgrade then It would be interesting to know the specifics of that change.
The Macquarie University score could be adversely influenced by a number of the changes in procedures including the use of standardization to reduce the impact of outliers on particular factor scales. Macquarie’s previous ranking may have relied on an abnormally good performance on a few factors.
Also of interest is the consistency of the underlying data. Large, sudden falls in THES-QS have previously arisen from inconsistent data ( http://rankingwatch.blogspot.com/ ).
A potential issue is whether the latest data contains any error, or whether the older data is in fact incorrect leading to MU being over-rated for previous years.
The components for Macquarie University for the THES-QS rankings will be very interesting when they are published.
This is a weird post. On the one hand you are claiming that these ranking measures are not especially valid (I personally wouldn’t disagree) nor have good reliability, but on the other hand you want to be in the top 200. These things seem really contradictory (unless you somehow hate the THES but love the SJT). If you really think these measures are so poor, then you could refuse to participate as some universities in the US do to the USA today survey (this might have some meaning if you convince others to do it as a block). There are lots of other good things MQ could market itself on that probably mean a whole lot more than some of these rankings (especially given the new hospital/company links).
Isn’t it a typical response by Macquarie that as long as the feedback is that things are going well, the source is ‘credible’, however if the cries of ‘foul’ from the students and industry is saying otherwise it is ‘more work to do’. Funny how similar that sentiment sounds to the flailing NSW government and their superfluous spending on useless infrastructure.
shhhh…. just don’t listen Macquarie and hear what you want to.
As a student at this university, I personally think this rank does not consider that Macquarie is advancing for the future. And the future is very different from the past, old style of education apathy. If other Australian universities are still practising old education style
(from those I exclude Melbourne University and Australian National University) it makes sense that this ranking will communicate something for them. And for the ranking indicators, anyone could manage a new ranking for anything. Now one question - when this data has been compiled? Has it been compiled during this year or prior years when another policy was still in place? I personally embrace Macquarie new standards for its new education plan
(that is shared by the rest of the world class universities for years). In USA, there are lots of rankings and what one does is to make a careful analysis of what criteria was used for ranking measurement before taking it seriously.
I am tempted to think that is just the case with this ranking results for Macquarie University.
I’m a younger brother, and is most likely the case, when I was a little younger I was usually the butt-end of one of my older brothers jokes. I used to go to my mother and complain what was happening and she always used to look me square in the face and say: “quit your complaining and get on with the job.” Perhaps there is a lesson after all…
Radcon- there’s loads of money and prestige involved, hence the “losers” attack the methodology, and the “winners” trumpet their success. Rank ordering doesn’t really tell you much anyway, does anyone seriously choose a university because it’s ranked 7th over one that’s ranked 9th?
“there are valid criticisms of the THES-QS including that it uses a relatively subjective methodology” Are there valid praises? Can you provide proof of this ‘relatively subjective methodology’ other than your allegory?
Good point radcon. In addition, the University was boasting about their Times ranking when it was good. Isn’t it equally possible that MQ’s previous high ratings were incorrect? Or it may be it is more likely that the rating system is not accurate. It is obvious that the university believed in the Times until they got the drop in the ranking. If they really thought it meant nothing they would not have commented. Relax MQ if you are doing everything right you have nothing to worry about. Just watch your lane. Don’t look so far into your future (ranking goals) and forget about us in the here and now.
Much has been written about the unpredictability of university rankings and the poor methodologies they employ. Taking this as valid criticism I am continually confused as to why the main goal as stated in the MQ@50 document seems to be based on a rankings measure - to be among the top eight research universities in Australia and one of the top 200 in the world.
The most obvious reason is that it allows for measurability of the university’s progress. I’m aware that the document lists other numerical measures as indicators of goal achievement (i.e. increasing Macquarie’s number of first preference applications to 10%).
These are my questions - With all the current concern over rankings, is it still advisable to use rankings as the basis for goal measurement? If so, which rankings are used? Depending on the system used, is Macquarie not already in a position to achieve its goals? If so, is Macquarie merely seeking to maintain its position in an ever improving field of competition?
I believe universities can benefit from a move away from the ‘objectivity’ of rankings. Not disregarding the complexity of long-term strategic approaches, there must surely still be value for organisations in the more traditional measures - knowing that you are doing a good job; identifying areas of improvement; and ensuring that users are satisfied. The challenge is to get the balance right and not ascribe more significance to rankings than they warrant.
Dear Prof Schwartz. The new ranking is obviously based on academic measures, albeit slightly different criteria to the last ranking. Therefore, I do not accept your “straw man” argument that the new ranking is as different as age is to height. I am sure that if MQ had improved in ranking we would all be congratulating ourselves.
To use a sporting metaphor, when you decide to join in a game, it is poor form to blame the umpire when you don’t get the outcome you want.
I agree with radcon. I think there is general agreement that all such ranking are flawed. If Macquaire takes such an agressive stance against THES, then it will not go down well when we trumpet our achievement in some other ranking.
If THES is not to be trusted, how will we measure our place in the top 200? Has the “reliable” survey already been identified, or will we wait to see which gives the results we want?
Though such rankings are “flawed”, this does not mean that MQU name has not been questioned.
This new ranking is very disturbing, this will affect everyone in the school.
I think saying, “oh this ranking is flawed and it doesn’t matter cause we are doing well” is not enough! Something must be done to make sure that we progress and not fall back.
Currently,I feel like I invested my education in to the wrong university at the moment.
I hope that this ranking as a slap in the face, instead of brushing it aside.
I just want to be a little ignorant for the moment.
I left my previous university, UWS, to come and learn at Macquarie.
Why is it that UWS is ranked higher? All i know is I had to break out some D’s to get into this place and now i’m being told my old place out at Parramatta is better? I don’t think so:) That is all I know on the subject. I wouldn’t worry about it.
There is a post on the topic at University Ranking Watch [http://rankingwatch.blogspot.com]. Macquarie’s decline seems to be caused by the “peer review” and by the international faculty section.
If the drop in ranking is due the the change of methodology, then how come the other Australian universities or even yet, the other universities across the world still seemed to be ranked in fairly the same place compared to last year? Stop trying to justify this to save yourself and the reputation of Macquarie and do something about it.
It doesnt matter. Students need to prove themselves, not the uni. The higher your performance, the better you are . . .
I came to Macquarie because it is what it says it is - innovative, and this is the best goal Macquarie can set. In any case, how do we know and who is to say whether what is being measured statistically is involutionary or evolutionary? When I find the ‘grey men’ infiltrating Macquarie, I will choose more colourful people under whom to study.
The university that continues to embrace technology and updates its premises to become more flexible will be the organization that succeeds
For example, video podcasting, ilectures, online facilities such as an hour each week where live interaction can occur through macquarie university webct site for subjects etc.
This makes students learn and will encourage freely and creatively the application of our studies.
Back to video podcasting, a criticism may be that it would stop students coming to lectures. This logic is flawed. It is the choice of the student to attend university and as such, this aid would increase the marks and reputation of the university as a flexible provider of education. It would help the good students to go further, and bring education in the modern context that it should be placed
I think its time universities started to think outside the square, and the days of just a piece of chalk and a blackboard are slowing coming to a close.
Thanks very much
Totally agree with ‘VN with love’.
It’s not a university but students ourselves who need study and work harder to prove ourselves in our work fields.
THES hasn’t been credible and will not in the future. Also in my home country, newspapers argued that there is significant imperfectness in rating and ranking universities such that unis in UK and its friends (former British colonies) have abnormally high points in some columns which can be assessed ambiguously and subjectively.
The problem is that MQ is ranked in an even lower place in most of other institutions’ ranking tables.
In bottom lines, we don’t need to worry about this huge drop in THES table but students and uni staffs need some willpower to take some action to make it better.
Who said UWS was even in the top 400? Macquarie is still 9th in Australia, behind University of Western Australia.
As long as the uni is striving to improve things for students, I think it is on the right track. Granted, there is a long way to go. But the changes over the last semester (more places for students to hang out together and study or relax) are evidence of improvement.
Kudos to the uni for finally making these changes with us in mind!
The VC is right in that the THES is widely regarded as somewhat dodgy in its methodology, placing a relatively high weighting (40%, I believe) on a form of peer review that is not particularly transparent. It is more a measure of perceptions of the university than its actual quality.
On the other hand, it may be worrying that the perception of our quality has fallen so substantially. I’m not going to speculate on possible reasons for that here - it needs further investigation.
All rankings, by their nature, will be biased in some way, as none can measure ALL of the many variables involved in judging a university’s quality. The JT, for example, while it uses a consistent methodology, focuses more on reesarch outcomes and is biased towards universities with substantial output in the sciences. In short, rankings only produce rankings on what they measure - so Ian Chubb is right, you could get any number of Australian unis into the world’s top 100 if you measure the right things.
Remember the “Macquarie beats Oxford” ads of a few years ago? - our debating team won an international competition. Does this make us a “better” uni than Oxford? YES! (In debating in that year…)
Maybe we should create a global ranking of universities on the basis of their contribution to the microeconomy of North Ryde!
All jokes aside (as Peter Garrett might have said to Steve Price), the real issue for students to think about shouldn’t be where MQ is ranked as a whole in some global index. In my view it’s more important to think about:
* whether MQ has enthusiastic and high quality researchers and teachers in the field that you are interested in studying; * whether the experience of studying here is a positive, inspiring and engaging one; and
* whether it engenders a love of learning and an appreciation and respect for your chosen discipline.
Show me the ranking of all those things, for EACH subject area, not the university as a whole, and I MIGHT start to think that a ranking has some real value in informing students and staff (and potential students and staff) about their university choices.
Back to my marking…
Though THES-QS rankings are quite subjective and not based solely on academic criteria, it is also true that THES-QS ranking is the most influencing raking that informing potential students how good this university is.
I remember that MQ made 63th (67th?) in 2005 and it was quite impressed (and that’s one of the main reasons why I choosed MQ than other universities). I firmly believe that MQ have to more care about the ranking rather than just ignore and leave it.