<div dir="ltr">Re-posted to GOAL as of potential interest:<div><br><div class="gmail_quote">On Dec 18, 2014, at 10:57 AM, David Wojick <<a href="mailto:dwojick@CRAIGELLACHIE.US" target="_blank">dwojick@CRAIGELLACHIE.US</a>> wrote:<div style="word-wrap:break-word"><div><blockquote type="cite"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
This does sound interesting, Stevan, especially if you got an unexpected
result.</blockquote></blockquote>The objective is actually not to get an unexpected result, David, but to generate a battery of metrics that predicts the actual REF2014 peer ranking as closely as possible, so that in REF2020 it can be the metrics rather than the peers that do the ranking.</div><div><blockquote type="cite"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">But I doubt it would validate or invalidate any scientometric
predictors.</blockquote></blockquote>A high correlation would certainly validate the REF battery, for the REF.</div><div><blockquote type="cite"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> It is basically a decision model for a single organization
going through a more or less single, albeit complex, decision exercise. To begin with, it is just one organization.</blockquote></blockquote>All researchers, at all UK institutions, in each discipline, is a “single organization”? </div><div><br></div><div>(To paraphrase an erstwhile UK researcher: "<i>some organization!" "some singularity!"</i>)</div><div><br></div><div>The UK does 6-11% of the world’s research. Not a bad sample, I’d say, for a first pass at validating those metrics.</div><div><blockquote type="cite"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Then too, simple multiple
regression seems like a very crude way to derive such a model.</blockquote></blockquote>Simple multiple regression is a natural first step. (I agree that after that more sophisticated analyses will be possible too.)</div><div><blockquote type="cite"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> The large
number of factors is also a concern, as others have noted, especially if
we are trying to establish causality.</blockquote></blockquote>For the REF, all you need is predictivity. But I agree that causality too is important, and with continuous assessment instead of just stratified post-hoc sampling, it will be possible to make much more powerful use of the time domain.</div><div><br></div><div>(I don’t think a starting battery of 30 metrics would be too many -- far from it. But some of them will prove to have low or no Beta weights. That’s why metric validation is an <i>empirical</i> exercise.)</div><div><blockquote type="cite"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I would think that the more factors
used the less credible the result.</blockquote></blockquote>The credibility of each metric will be the proportion of the total variance that it accounts for. It is an empirical question whether a few metrics will account for the lion’s share of the variance, and the rest will have negligibly small or no weights.</div><div><blockquote type="cite"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> But then we also need to think that we
have all the significant factors, don't we? Perhaps not. Are there useful
precedents for this?</blockquote></blockquote><div>I am certain that my back-of-the-matchbox list of candidate metrics was neither exhaustive nor optimal. It was just indicative. All other credible candidates are welcome!</div><div><br></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">REF2020Rank</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)"> = </span></div></div><div><div><br style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px"></div></div><div><div><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w1</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(pubcount) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w2</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(JIF) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w3</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(cites) +</span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w4</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(art-age) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w5</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(art-growth) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w6</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(hits) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w7</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(cite-peak-latency) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w8</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(hit-peak-latency) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w9</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(citedecay) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w10</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(hitdecay) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w11</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(hub-score) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w12</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(authority+score) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w13</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(h-index) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w14</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(prior-funding) +</span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w15</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(bookcites) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w16</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(student-counts) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w17</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(co-cites + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w18</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(co-hits) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w19</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(co-authors) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w20</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(endogamy) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w21</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(exogamy) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w22</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(co-text) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w23</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(tweets) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w24</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(tags), + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w25</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(comments) + </span><strong style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px">w26</strong><span style="color:rgb(51,51,51);font-family:verdana,arial,helvetica,sans-serif;font-size:13px;background-color:rgb(255,255,255)">(acad-likes) etc. etc.</span></div></div></blockquote><div><div><br></div><blockquote type="cite"><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Finally, is all the needed data available and how
much might this cost?</blockquote></div></blockquote>The <a href="http://www.ref.ac.uk" target="_blank">REF2014 data</a> were released today and are available at once, for testing against metrics, discipline by discipline.</div><div><br></div><div>What’s still very sparse and gappy is the availability of the 26 OA metrics sketched above — and that’s because a lot of the source material is not yet OA. The proprietary databases (like WoS and SCOPUS) are not OA either. But if the papers were all OA, then the metrics could all easily be harvested and calculated from them.</div><div><blockquote type="cite"><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I guess that if I were peer reviewing this as a preliminary proposal I
would be positive but not enthusiastic. More information is needed about
the proposed project and its goals.</blockquote></div></blockquote>I wasn’t actually counting on your recommendation for peer review of the proposal to validate metrics against REF2014, David: I was rather hoping it might help inspire you to <a href="http://openaccess.eprints.org/index.php?serendipity%5Baction%5D=search&serendipity%5BsearchTerm%5D=wojick&serendipity%5BsearchButton%5D=%3E" target="_blank">recommend the right OA policy model to OSTI</a> for which you consult. That way we would have a better hope of making the all-important OA data available when President Obama’s OSTP directive is implemented...</div><div><br><blockquote type="cite"><div><div>At 07:23 AM 12/18/2014, you wrote:<br>
<blockquote type="cite"><blockquote type="cite">On Dec 18, 2014, at 3:39
AM, [<i>name deleted because posted off-list]</i>
wrote:<br><br>
that's very high dimensionality in that equation.</blockquote><br>
I don't think 30 metric predictors for about 6% of the planet's
annual research output (UK) is such an under-fit.<br><br>
(But we could start with the most likely metrics first, and then see how
much variance is accounted for by adding more.)<br>
<br>
<blockquote type="cite">you don't have enough data
points to have any decent confidence about those weights - i
</blockquote><br>
That cannot be stated in advance. First we need to calculate the multiple
regression on the REF2014 rankings and determine how much each metric
contributes.<br><br>
<blockquote type="cite">suggest you look at the REF
data… and see how many different journal/venues and all over the ACM
Classification hierarchy, the 7000 odd outputs appeared in - you'll find
in any given venue, topic you rarely have more than a handful of items -
your variance will be terrible</blockquote><br>
The proposal is not to assess the predictive power of any one of the 4
publications submitted. <br><br>
The REF2014 peer rankings themselves are based on peers (putatively)
re-reading those 4 pubs per researcher, <i>but the regression equation I
sketched is based on (OA) data that go far beyond that.</i> <br><br>
(In point of fact, it's absurd and arbitrary to base the REF assessment
on just 4 papers in a 6-year stretch. That restriction is dictated by the
demands of the peers having to read all those papers, but open-access
metrics can be harvested and have no such human bottleneck constraint on
them. What you <i>could</i> complain, legitimately, is that<i> not all
those potential data are OA yet</i>... Well, yes — and that’s part of
the point.)<br><br>
<b>REF2020Rank</b> = <br><br>
<b>w1</b>(pubcount) + <b>w2</b>(JIF) + <b>w3</b>(cites)
+<b>w4</b>(art-age) + <b>w5</b>(art-growth) + <b>w6</b>(hits) +
<b>w7</b>(cite-peak-latency) + <b>w8</b>(hit-peak-latency) +
<b>w9</b>(citedecay) + <b>w10</b>(hitdecay) + <b>w11</b>(hub-score) +
<b>w12</b>(authority+score) + <b>w13</b>(h-index) +
<b>w14</b>(prior-funding) +<b>w15</b>(bookcites) +
<b>w16</b>(student-counts) + <b>w17</b>(co-cites + <b>w18</b>(co-hits) +
<b>w19</b>(co-authors) + <b>w20</b>(endogamy) + <b>w21</b>(exogamy) +
<b>w22</b>(co-text) + <b>w23</b>(tweets) + <b>w24</b>(tags), +
<b>w25</b>(comments) + <b>w26</b>(acad-likes) etc. etc.<br><br>
<blockquote type="cite">and the result of munging all
those _different_ distributions into one single model will be to prssure
people to move their work areas to the best fit topic/venue, which is not
a true measure of anything desired by us of HEFCE or
<a href="http://rc.uk/" target="_blank">RC.UK</a> to my knowledge.</blockquote><br>
I cannot fathom what one, two, three or N things a researcher can do in
order to maximize their score on the above equation (other than to try to
do good, important, useful work…).<br><br>
<blockquote type="cite">please do the detailed
work…</blockquote><br>
Will try. But there a few details you need to get straight too…
(<:3<br><br>
<blockquote type="cite"><br>
On Wed, Dec 17, 2014 at 3:38 PM, Stevan Harnad
<<a href="mailto:harnad@ecs.soton.ac.uk" target="_blank">harnad@ecs.soton.ac.uk</a>
> wrote:<br>
<dl><br>
<blockquote type="cite">
<dd>On Dec 17, 2014, at 9:54 AM, Alan Burns
<<a href="mailto:alan.burns@YORK.AC.UK" target="_blank">alan.burns@YORK.AC.UK</a>>
wrote:<br><br>
</dd><dd>Those that advocate metrics have never, to at least my satisfaction,
answered the<br>
</dd><dd>argument that accuracy in the past does not mean effectiveness in the
future,<br>
</dd><dd>once the game has changed.</dd></blockquote><br>
<dd>I recommend Bradley on metaphysics and Hume on
<a href="http://plato.stanford.edu/entries/induction-problem/" target="_blank">
induction</a>:<br><br>
</dd><dd>
"<a href="https://www.goodreads.com/quotes/1369088-the-man-who-is-ready-to-prove-that-metaphysical-knowledge" target="_blank">
The man who is ready to prove that metaphysical knowledge is wholly
impossible… is a brother metaphysician with a rival theory</a>â€
Bradley, F. H. (1893) Appearance and Reality<br>
<br>
</dd><dd>One could have asked the same question about apples continuing to
fall down in future, rather than up.<br><br>
</dd><dd>Yes, single metrics can be abused, but not only van abuses be named
and shamed when detected, but it become harder to abuse metrics when they
are part of a multiple, inter-correlated vector, with disciplinary
profiles on their normal interactions: someone dispatching a robot to
download his papers would quickly be caught out when the usual
correlation between downloads and later citations fails to appear. Add
more variables and it gets even harder,<br><br>
<blockquote type="cite">
<dd>Even if one was able to define a set of metrics that perfectly
matches REF2014.<br>
</dd><dd>The announcement that these metric would be used in REF2020
would<br>
</dd><dd>immediately invalidate there use.</dd></blockquote><br>
</dd><dd>In a weighted vector of multiple metrics like the sample I had
listed, it’s no use to a researcher if told that for REF2020 the mertic
equation will be the following, with the following weights for their
particular discipline:<br><br>
<dl>
<dd>w1(pubcount) + w2(JIF) + w3(cites) +w4(art-age) +
w5(art-growth) w6(hits) +w7(cite-peak-latency) +
w8(hit-peak-latency) +w9(citedecay) +w10(hitdecay) + w11(hub-score) +
w12(authority+score) + w13(h-index) + w14(prior-funding) +w15(bookcites)
+ w16(student-counts) + w17(co-cites + w18(co-hits) + w19(co-authors) +
w20(endogamy) + w21(exogamy) + w22(co-text) + w23(tweets) + w24(tags),
+w25(comments) + w26(acad-likes) etc. etc.<br><br>
</dd></dl><br>
</dd><dd>The potential list could be much longer, and the weights can be
positive or negative, and varying by discipline.<br><br>
<dl>
<dd>
"<a href="https://www.goodreads.com/quotes/1369088-the-man-who-is-ready-to-prove-that-metaphysical-knowledge" target="_blank">
The man who is ready to prove that metric knowledge is wholly impossible…
is a brother metrician with rival
m</a>etrics<font face="verdana" color="#333333">…†<br>
</font><br>
</dd></dl><br>
<blockquote type="cite">
<dd>if you wanted to do this properly, you should have to take a lot of
outputs that were NOT submitted and run any metric scheme on them as well
as those submitted. <br>
<blockquote type="cite">
<dd>too late:)</dd></blockquote></dd></blockquote></dd></dl></blockquote>
<br>
You would indeed — and that’s why it all has to be made OA…<br><br>
</blockquote></div></div></blockquote></div><br></div></div></div></div>