<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you for your contribution, Lizzie!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I love this quote from your The blind and the elephant... "</div>
<span style="color: rgb(157, 51, 51); font-family: Vollkorn, serif; font-size: 18px; font-style: italic; text-align: center; background-color: rgb(255, 255, 255); display: inline !important">“When using any indicator for purposes that have rewards attached
– especially when the entity is small – you should use metrics with extreme care.”</span>
<div>
<div style="text-align: center;"><font color="#9d3333" face="Vollkorn, serif"><span style="font-size: 18px;"><i><br>
</i></span></font></div>
<div style="text-align: left;"><span style="color: rgb(0, 0, 0);">I agree. This is a key point that I am trying to make. Metrics based on substantive collective knowledge make sense. Let's aim to achieve the CO2 emissions reductions that are needed to avoid
catastrophic climate change (and wouldn't it be nice if newspapers decided to report our collective progress on this prominently on a daily basis)? Similarly, we can assess our collective progress in preventing and treating cancer through epidemiological data.</span></div>
<div style="text-align: left;"><span style="color: rgb(0, 0, 0);"><br>
</span></div>
<div style="text-align: left;"><span style="color: rgb(0, 0, 0);">However, evaluating an individual scholar or scholarly article on the basis of citations is problematic because there are rewards attached for the small entity, the individual scholar - from
job loss to promotion, prestige, grant funding. This creates an incentive to overstate positive findings, understate limitations, see patterns in data that aren't really there, and even to commit fraud. Less (or no) reliance on metrics in this case would be
in the best interests of advancing our collective knowledge.</span></div>
<div style="text-align: left;"><span style="color: rgb(0, 0, 0);"><br>
</span></div>
<div style="text-align: left;">Metrics (like most things) are neither good nor bad in and of themselves. Whether metrics are beneficial or otherwise depends on who is using them, how, and for what purpose. Some beneficial examples of metrics (from my perspective)
are to understand and ameliorate bias in hiring, salaries, grant funding, etc. </div>
<div style="text-align: left;"><br>
</div>
<div style="text-align: left;">best,</div>
<div style="text-align: left;"><span style="color: rgb(0, 0, 0);"><br>
</span></div>
<div>
<div id="Signature">
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px">Dr. <span style="font-size:12pt">Heather Morrison</span></p>
<p style="margin-top:0px; margin-bottom:0px">Associate Professor, School of Information Studies, University of Ottawa</p>
<p style="margin-top:0px; margin-bottom:0px">Professeur Agrégé, École des Sciences de l'Information, Université d'Ottawa<br>
</p>
<p style="margin-top:0px; margin-bottom:0px"></p>
<span id="ms-rterangepaste-start"></span>
<p style="margin-top:0px; margin-bottom:0px; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px">
Principal Investigator, Sustaining the Knowledge Commons, a SSHRC Insight Project</p>
<p style="margin-top:0px; margin-bottom:0px; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px">
sustainingknowledgecommons.org</p>
<span id="ms-rterangepaste-end"></span>
<p style="margin-top:0px; margin-bottom:0px">Heather.Morrison@uottawa.ca<br>
</p>
<p style="margin-top:0px; margin-bottom:0px"><a href="https://uniweb.uottawa.ca/?lang=en#/members/706" class="OWAAutoLink">https://uniweb.uottawa.ca/?lang=en#/members/706</a></p>
<p style="margin-top:0px; margin-bottom:0px">[On research sabbatical July 1, 2019 - June 30, 2020]</p>
</div>
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Elizabeth Gadd <E.A.Gadd@lboro.ac.uk><br>
<b>Sent:</b> Wednesday, October 23, 2019 3:54 AM<br>
<b>To:</b> Heather Morrison <Heather.Morrison@uottawa.ca><br>
<b>Cc:</b> scholcomm@lists.ala.org <scholcomm@lists.ala.org>; Global Open Access List (Successor of AmSci) <goal@eprints.org>; Julie Bayley <JBayley@lincoln.ac.uk>; g.derrick@lancaster.ac.uk <g.derrick@lancaster.ac.uk><br>
<b>Subject:</b> Re: [SCHOLCOMM] Evaluation and metrics: why not (a critical perspective)</font>
<div> </div>
</div>
<div dir="auto">
<div style="font-size:9pt; font-family:'calibri'; color:#FF4233"><b>Attention : courriel externe | external email</b></div>
<div>Hi Heather
<div><br>
</div>
<div>Thanks for your email! A few thoughts:</div>
<div><br>
</div>
<div>1) The UK has given the notion of measuring impact quite a lot of thought, having had this measured as part of their national research assessments since 2009. I would refer you to the brilliant work by Julie Bayley (<a href="https://juliebayley.blog/">https://juliebayley.blog/ </a>) and
Gemma Derrick (<a href="https://www.palgrave.com/gp/book/9783319636269">https://www.palgrave.com/gp/book/9783319636269</a>) (copied in) in this space for a full exploration of all the issues, including the negative impacts you describe, something Gemma’s group
have termed ‘grimpacts’. </div>
<div>2) With regards to your statement that the assessing of scholarly work does not require metrics, I would refer you to a piece I’ve written called ‘The Blind and the elephant: bringing clarity to our conversations about responsible metrics’. (<a href="https://thebibliomagician.wordpress.com/2019/05/15/the-blind-and-the-elephant-bringing-clarity-to-our-conversations-about-responsible-metrics/">https://thebibliomagician.wordpress.com/2019/05/15/the-blind-and-the-elephant-bringing-clarity-to-our-conversations-about-responsible-metrics/</a>) In
it I argue that we need to be a bit careful about sweeping statements about metrics, because there are many reasons we evaluate research (I name six) and at many different levels of granularity (individual, group, country, etc). In some settings, the use of
metrics can be helpful, in others not. I would always generally argue that metrics + peer review give us the best chance of responsible assessment, as metrics can mitigate against unconscious bias in peer review.</div>
<div>3) To find other examples of best practice in research evaluation, DORA are compiling these on their website (<a href="https://sfdora.org/good-practices/research-institutes/">https://sfdora.org/good-practices/research-institutes/</a>). </div>
<div>4) For more discussion on these issues there are a number of dedicated discussion lists now. In the US, there is the RESMETIG list; in Canada the BRICS group; in the UK the LIS-Bibliometrics group, and finally an international working group looking at
research evaluation called INORMS Research Evaluation Working Group.</div>
<div><br>
</div>
<div>I hope this is helpful?</div>
<div><br>
</div>
<div>All best</div>
<div>Lizzie</div>
<div><br>
<div dir="ltr">Dr Elizabeth Gadd
<div>Research Policy Manager (Publications)</div>
<div>Research Office</div>
<div>Loughborough University</div>
<div>Loughborough, Leicestershire, UK</div>
<div><br>
</div>
<div>T: +44 (0)1509 228594</div>
<div>S: lizziegadd</div>
<div>E: e.a.gadd@lboro.ac.uk</div>
<div><br>
</div>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On 22 Oct 2019, at 22:04, Heather Morrison <Heather.Morrison@uottawa.ca> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
</div>
<p style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:Calibri,sans-serif">
<span style="color:black; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Rigorous scholarly work requires periodic assessment of our underlying assumptions. If these are found to be incorrect, then any logical arguments or empirical work based on
these assumptions should be questioned.</span></p>
<p style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:Calibri,sans-serif">
<span style="color:black"><br>
</span></p>
<p style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:Calibri,sans-serif">
<span style="color:black; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Assumptions underlying metrics-based evaluation include:</span></p>
<div style="margin:0px 0cm 0.000133333px; font-size:12pt; font-family:Calibri,sans-serif">
<ol>
<li><span style="color:black; font-family:Calibri,Helvetica,sans-serif; font-size:12pt">impact is a quality of good scholarship at the level of individual works</span></li><li><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">aiming for impact is desirable in scholarly work</span></li></ol>
<div><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Let's consider the logic and an example.</span></div>
<div><br>
</div>
<div>
<ol>
<li><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Is impact a good thing? Consider what "impact" means in other contexts. Hurricanes and other natural disasters have impact; when we seek to work in harmony with the environment, we try
to avoid impact. "Impact" is not essentially tied to the quality of "good".</span></li><li><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Is aiming for impact at the level of individual scholarly works desirable? According to Retraction Watch, one of the top 10 most highly cited papers includes "the infamous</span><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"> </span><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><em style="font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit; margin:0px; outline:0px; color:rgb(51,51,51)">Lancet</em></span><span style="color:rgb(51,51,51); font-family:Calibri,Helvetica,sans-serif; font-size:12pt"> </span><span style="color:rgb(51,51,51); font-family:Calibri,Helvetica,sans-serif; font-size:12pt">paper
by Andrew Wakefield that</span><span style="color:rgb(51,51,51); font-family:Calibri,Helvetica,sans-serif; font-size:12pt"> </span><a href="http://retractionwatch.com/2011/01/06/some-quick-thoughts-and-links-on-andrew-wakefield-the-bmj-autism-vaccines-and-fraud/" style="border-bottom:1px solid rgb(51,51,51); font-family:"Noto Serif",serif; font-size:19px; outline:0px; color:rgb(51,51,51)"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">originally
suggested a link between autism and childhood vaccines</span></a><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">" (from: </span><a href="https://retractionwatch.com/the-retraction-watch-leaderboard/top-10-most-highly-cited-retracted-papers/" id="LPlnk257332" style="font-size:12pt"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">https://retractionwatch.com/the-retraction-watch-leaderboard/top-10-most-highly-cited-retracted-papers/</span></a><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">).
This article has been highly cited in academic papers both before and after retraction, widely quoted in traditional and social media, and I argue can demonstrate real-world impact (in the form of the return of childhood diseases that were on track to worldwide
eradication) that is truly exceptional. Any way you measure impact, this article had it. Could this be a fluke? I argue that there are logical reasons why this is would not be a fluke. When researchers are rewarded for impact, this is an incentive to overstate
the conclusions, see positive and interesting results beyond what the data shows, and even outright fraud. </span></li></ol>
<div><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">It is important to distinguish the consequences of impact at the level of an individual research work and scholarly consensus based on a substantial body of evidence (such as climate
change). </span></div>
</div>
<div><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">It is also important to consider some of the implications of metrics-based evaluation on individual scholars. Social biases such as those based on gender, ethnic origin, and Western
centrism are common in our society, including in academia. There is some recognition of this is traditional academic work and some work to counter bias (such as blind reviews), however this cannot be controlled in the downstream academic environment and it
seems obvious that metrics that go beyond academic citations will tend to amplify such biases. </span></div>
<div><br>
</div>
<div><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Evaluation of the quality of scholarly work does not require metrics. Anyone who is a researcher needs to do a great deal of reading and assessment of scholarly works. Professors read
and grade papers and theses. When I evaluate dossiers for scholarships or grants or tenure and promotion committees, I read and evaluate the works. </span></div>
<div><br>
</div>
<div><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">The University of Ottawa has what I consider a good, non-metrics-based approach to evaluating research. Although it was written some time ago, it is still leading-edge. To obtain promotion
and tenure, for example, a professor needs to demonstrate that they are contributing a sufficient amount of original research beyond their dissertation. It is recognized that there are many different kinds of knowledge generation. A scientist may publish journal
articles; a professor in theatre may accomplish innovations in production of plays. There is no need to add preprints; this is already covered. If you know of other good non-metric models for evaluation, please share with the list. </span></div>
<div><br>
</div>
<div><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">This e-mail is a brief piece on a topic that I've written about in quite a bit more detail. Anyone who has the time is invited to read this book chapter in the process of publication
that I wrote: "</span><span style="font-size:12pt; font-family:Calibri,Helvetica,sans-serif">What counts in research? Dysfunction in knowledge creation & moving beyond". In addition to a critical view of metrics-based evaluation (traditional and altmetrics),
readers may be interested in learning about how metrics feed into university rankings and the growing role of Elsevier in this space. When the book is published, I'll refer to the work of fellow authors for an explanation of the problems associated with university
rankings per se.</span></div>
</div>
<p style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:Calibri,sans-serif">
<span style="color:black"><span style=""><a href="http://hdl.handle.net/10393/39088"><span style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">http://hdl.handle.net/10393/39088</span></a></span></span></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">best,</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">Dr. </span><span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">Heather Morrison</span><br>
</div>
<div id="x_Signature">
<div></div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px">Associate Professor, School of Information Studies, University of Ottawa</p>
<p style="margin-top:0px; margin-bottom:0px">Professeur Agrégé, École des Sciences de l'Information, Université d'Ottawa<br>
</p>
<p style="margin-top:0px; margin-bottom:0px"></p>
<span id="x_ms-rterangepaste-start"></span>
<p style="margin-top:0px; margin-bottom:0px; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px">
Principal Investigator, Sustaining the Knowledge Commons, a SSHRC Insight Project</p>
<p style="margin-top:0px; margin-bottom:0px; font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px">
sustainingknowledgecommons.org</p>
<span id="x_ms-rterangepaste-end"></span>
<p style="margin-top:0px; margin-bottom:0px">Heather.Morrison@uottawa.ca<br>
</p>
<p style="margin-top:0px; margin-bottom:0px"><a href="https://uniweb.uottawa.ca/?lang=en#/members/706" class="x_OWAAutoLink">https://uniweb.uottawa.ca/?lang=en#/members/706</a></p>
<p style="margin-top:0px; margin-bottom:0px">[On research sabbatical July 1, 2019 - June 30, 2020]</p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>