[BOAI] Re: Ranking of repositories

[Jean-Claude Guédon]

I fully agree with Isidro that  "Perhaps the problem is not with the
Rankings themselves, but with authorities not applying quality criteria
in the evaluation of such classifications".

But feeding tools for ranking is also part of the problem. Witness this
quotation from one of Eugene Garfield's papers: "I myself deplore the
quotation of impact factors to three decimal places. ISI uses three
decimal places to reduce the number of journals with the identical
impact rank. It matters very little whether the impact of JAMA is quoted
as 21.5 rather than 21.455 (Garfield, Eugene, 2005, “The Agony and the
Ecstasy - The History and Meaning of the Journal Impact Factor”, paper
presented at the International Congress on Peer Review And Biomedical
Publication, Chicago, September 16, 2005. Available on-line at
http://www.garfield.library.upenn.edu/papers/jifchicago2005.pdf). [my
emphasis, JCG]

In other words, the ranking takes precedence over the meaning of the
figures produced to support it. This is very strange to say the least,
unless one considers that science is like a spectator sport and that we
need to separate champions in every possible way (perhaps to intensify
the excitement of the spectators).

So, yes, the research administrators and university management teams
should be a little wiser in their evaluation techniques and place
quality back at the center of evaluations, but those providing the
numbers should also carefully repeat and underscore the caveats attached
to these numbers, even at the cost of losing themselves some impact as a
consequence. In any case, impact factors with three decimals simply mean
nothing.

Finally, the rage to rank also dominates sports. It is the reason that
sprinting events were clocked to the nearest one hundredth of a second
so as to produce more world records faster (but a false start,
curiously, is still measured at one tenth of a second after the signal
is given). This looks suspiciously like three decimals in the impact
factor. 

It is also the reason why steroids and other enhancement drugs are
messing up sports everywhere. Interestingly, cheating is also on the
rise in science. Could it have something to do with the obsession with
ranking?

Jean-Claude Guédon




Le jeudi 04 août 2011 à 08:43 +0200, Isidro F. Aguillo a écrit :

> Dear colleagues,
> 
> In my country sometimes we said: "the best is enemy of the good" and
> certainly there are far better tools for analyzing empirically the OAI
> but at this moment the key objective is IMPACT and according to my
> personal experience several universities are promoting their
> institutional repositories for improving their position in the
> Rankings. Perhaps the problem is not with the Rankings themselves, but
> with authorities not applying quality criteria in the evaluation of
> such classifications. Only in this way can be explained that a lot of
> people believe that the unethical Times Higher Education Ranking is
> very "prestigious".
> 
> Best regards,
> 
> 
> 
> El 03/08/2011 17:52, Jean-Claude Guédon escribió: 
> 
> > Personally, I regret these constant efforts to create "rankings"
> > leading to the identification of "excellence". They completely
> > distort the quality issues which, IMHO, are far more important.
> > Would it not be much better to create evaluation thresholds
> > corresponding to quality levels. This would encourage lower-level
> > repositories to try moving up a category, and then perhaps two?
> > 
> > Some may object that category classifications are nothing more than
> > rough, crude ranking. This is not false, but there is a distinction
> > to be observed, however: quality thresholds do not put competition
> > at the center of everything, and it does not rely on competition to
> > identify quality. 
> > 
> > Some may think that competition is a good way to create quality, but
> > this is not the case. Just to give an example: the US health system
> > is largely dominated by competitive rankings of all kinds. This
> > leads to two opposite results: the US has many of the best health
> > centers in the world and a great many Nobel prizes in medicine; yet,
> > the US ranks about 35th in the world for life expectancy, which is
> > shockingly low. If one were to choose between having the medical
> > champions of the world, versus having a population with a better
> > general health, one would tend to prefer the latter. At least that
> > would be my choice.
> > 
> > In other words, fighting for excellence as the over-arching
> > principle of quality creation leads to the concentration of quality
> > at the very top, and it often leads to the neglect of overall
> > quality.
> > 
> > I believe science needs quality everywhere, and not just at the top.
> > A bit of competition is also needed, but only at the very top, to
> > stimulate the very best to go one step further. Competition
> > everywhere does not work because those that cannot hope to come even
> > close to the very best, the gold medals, simply give up.
> > 
> > Incidentally, OA corresponds to a massive vote in favor of quality,
> > as the many discussions about quality control and peer review that
> > are appearing in its wake demonstrate. Excellence is all right if it
> > is limited to the very top of science, where the paradigm shifts
> > occur. But most of science is not about paradigm shifting, far from
> > it. Let us value excellence, but let us keep it also in its proper
> > place. Meanwhile, let us grow quality all over and Open Access is a
> > powerful tool to that end.
> > 
> > My two cents' worth.
> > 
> > Jean-Claude Guédon
> > 
> > Le mercredi 03 août 2011 à 10:04 -0400, Peter Suber a écrit :
> > 
> > > [Forwarding from Isidro F. Aguillo, via the AmSci OA Forum.
> > >  --Peter Suber.]
> > > 
> > > 
> > > The second edition of the 2011 Ranking Web of Repositories has
> > > been published at the end of July.  It is available from the
> > > Webometrics portal:
> > > 
> > > http://repositories.webometrics.info/
> > > 
> > > 
> > > The number of repositories is growing fast, especially in academic
> > > institutions from developing countries. As in previous editions
> > > the subject repositories still appear in the top positions, with
> > > large institutional ones following them.
> > > 
> > > 
> > > There are no relevant changes in this edition, but the editors are
> > > making a plea to the Open Access community regarding a few aspects
> > > related to intellectual property issues.
> > > 
> > > 
> > > The papers and other documents deposited in institutional
> > > repositories are probably the main asset of those institutions. As
> > > important as giving free access to others is the proper
> > > recognition of the authorship of the scientific documents.
> > > Unfortunately a few institutions are hosting their repositories in
> > > websites outside the main webdomain of its organization and many
> > > repositories are recommending to use systems like handle and
> > > others purl-like URLs for citing (linking) the deposited items.
> > > This means that moral rights regarding institutional authorship
> > > are ignored, relevant information about authors is missed and the
> > > semantic possibilities of the web address are not explored.
> > > 
> > > 
> > > Nowadays it is already common to add the URL address of the full
> > > text document in the bibliographic references of the published
> > > papers. Logically the link to the full text in the institutional
> > > repository can be used for that purpose, but researchers are
> > > facing options that ignore their institutional affiliation, with
> > > strange meaningless codes, prone to typos or other mistakes and
> > > pointing to metadata pages not to the full text documents.
> > > Obviously for authors it could be more profitable to host the
> > > papers in their personal pages instead doing it in institutional
> > > repositories whose naming policies have relevant copyright issues.
> > > 
> > > 
> > > Our position is that end-users should be taken into account, that
> > > web addresses are going to place in important role in citing
> > > behavior, that citations are the key tool for evaluation of
> > > authors, that institutions are investing large amounts of money in
> > > their repositories in exchange of prestige and impact and that
> > > providing permanent address is the duty of the institution, nor
> > > responsibility of external third-parties.
> > > 
> > > 
> > > Comments are welcomed
> > > 
> > >  
> > > 
> > > 
> > > ===============================
> > > 
> > > 
> > > Isidro F. Aguillo, HonPhD
> > > 
> > > The Cybermetrics Lab
> > > IPP-CCHS-CSIC
> > > Albasanz, 26-28 (3C1)
> > > 28037 Madrid. Spain
> > > 
> > > isidro.aguillo @ cchs.csic.es
> > > 
> > > ===============================
> > > 
> > > 
> > 
> > 
> > -- 
> > Jean-Claude Guédon
> > Professeur titulaire
> > Littérature comparée
> > Université de Montréal
> 
> 
> 
> 
> -- 
> ===============================
> 
> Isidro F. Aguillo, HonPhD
> 
> The Cybermetrics Lab
> IPP-CCHS-CSIC
> Albasanz, 26-28 (3C1)
> 28037 Madrid. Spain
> 
> isidro.aguillo @ cchs.csic.es
> 
> ===============================


-- 
Jean-Claude Guédon
Professeur titulaire
Littérature comparée
Université de Montréal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/boai-forum/attachments/20110804/252d731b/attachment-0001.html