<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>Sally<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>The situation is much more complex than this. Yep, oversimplifying, but it’s natural. Publishers’ sites are crawled by Googlebot because (a) robots are allowed in to the public areas of publisher sites, and (b) there are relatively few publishers, well indexed. Google Scholar is a selective service based on Googlebot’s results: it chooses what to include (little) and what to leave out (the vast majority). Google Scholar has algorithms that select what from the publisher’s site is an article or the metadata thereof, and what is plain publisher guff (like subscription info, guidelines for authors, etc).<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>BTW, Google Scholar does not crawl separately, it uses selectivity on the Googlebot results.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>Repositories are very unlikely to bar robot entry (through robots.txt), though I would not say categorically that it has never happened. You actually have to extra work to bar robots from a website, and I can’t understand why a manager would do so. (Of course password protected data or behind a search barrier is inaccessible to a robot anyway.)<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>However, it is not so clear what is a repository, how many there area or where they are. The number keeps changing. This is problem No 1. <o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>The second problem is that Google Scholar seems to apply different rules to repositories than publisher sites. Repositories contain all sorts of things that are not ‘articles’, such as archival material, unpublished works, conference presentations, etc. One theory is that Google Scholar is happiest if it finds an open-access pdf hanging off a metadata entry in a repository. In other words, if the file is in XML, XHTML, Word, iBook, etc formats, it is not regarded as an article. And what to do if the metadata has several pdfs attached to it (or other formats), which is common with theses? When in doubt, leave it out... Google Scholar is about being ultra-selective on the Internet.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>The third problem is that Googlebot does not always crawl the entire site. It optimizes its time to best use. One trick it uses is to limit the depth of the link tree to search. Another is not to go too far at any one level. In the case of a publisher site the depth is relatively shallow and each list is short. One finds the list of issues, then each leads to a list of articles, and bingo! Or possibly years -> issues -> articles. Repositories are not so well organized, necessarily. Unless they are optimized for Googlebot (EPrints is) the robot might well find a year index, leading to 1000s of ‘articles’ per year. Googlebot gives up well before the end. Next time it may well do the same. Optimal is to have a link to ‘most recent deposits’ high on the home page (so the robot finds it early), and to provide Googlebot with an easy way to eventually search all of the site. The Google database may build up over time.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>And finally Problem No 4. How does Google Scholar regard the metadata? It prefers publisher formats. This is referred to in the paper cited.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>BTW Note that Wouter Gerritsma’s comments indicate that Problem No 2 or Problem No 4 dominate over No 3 for his repository. Google knows about the article (Googlebot indexed it) but Google Scholar doesn’t.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>I agree with Stevan Harnad that we need fuller repositories, but disagree that there is any inconsistency in also pressing for improved performance by Google Scholar. The people who deposit papers in repositories, and the programmers in Google are almost completely disjoint groups (Google workers don’t publish much – they keep the processes a commercial secret). We can do both at once, and they will have a synergistic effect on each other. Lack of synergy holds back open access. But enough of that. I just wanted to explain what was happening with Google Scholar.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>Arthur Sale<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>Tasmania, Australia<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> goal-bounces@eprints.org [mailto:goal-bounces@eprints.org] <b>On Behalf Of </b>Sally Morris<br><b>Sent:</b> Saturday, 5 January 2013 11:14 PM<br><b>To:</b> 'Global Open Access List (Successor of AmSci)'<br><b>Subject:</b> [GOAL] Re: Searching for OA vs. Providing OA<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>It's my understanding that Google (and Google Scholar) find published articles because the publishers enable crawling - whether the content is freely available or not (if I'm oversimplifying, someone will no doubt set me right). Are repository managers unintentionally blocking this?</span><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>Sally</span><o:p></o:p></p><div><p class=MsoNormal> <o:p></o:p></p></div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Sally Morris</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>South House, The Street, Clapham, Worthing, West Sussex, UK BN13 3UU</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Tel: +44 (0)1903 871286</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Email: <a href="mailto:sally@morris-assocs.demon.co.uk">sally@morris-assocs.demon.co.uk</a></span><o:p></o:p></p><div><p class=MsoNormal> <o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div class=MsoNormal align=center style='text-align:center'><span lang=EN-US><hr size=3 width="100%" align=center></span></div><p class=MsoNormal>On Fri, Jan 4, 2013 at 5:03 PM, Gerritsma, Wouter <<a href="mailto:Wouter.Gerritsma@wur.nl" target="_blank">Wouter.Gerritsma@wur.nl</a>> wrote:<o:p></o:p></p><div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> </span><span lang=EN-GB style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Google Scholar is a very good fulltext scholarly search engine, no doubt about it. But it doesn’t find all the ftxt available on the web, albeit it does a good job.</span><span lang=EN-GB><o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Take e.g. one of my articles <a href="http://scholar.google.com/scholar?cluster=17014920805021872143&hl=en&as_sdt=0,5" target="_blank">http://scholar.google.com/scholar?cluster=17014920805021872143&hl=en&as_sdt=0,5</a> GS found two PDF version’s but not the one on our universities repository. That is still not fully indexed. Although it gets close <a href="http://library.wur.nl/WebQuery/wurpubs/lang/380005" target="_blank">http://library.wur.nl/WebQuery/wurpubs/lang/380005</a> it found our metadata reocrd, but not the ftxt.</span><span lang=EN-GB><o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>I guess this is still the case with many repositories. Earlier this year it was even reported in the literature:</span><span lang=EN-GB><o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=NL style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Arlitsch, K. & P.S. O'Brien (2012). </span><span lang=EN-GB style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Invisible institutional repositories: addressing the low indexing ratios of IRs in Google. Library Hi Tech, 30(1): 60-81 <a href="http://dx.doi.org/10.1108/07378831211213210" target="_blank">http://dx.doi.org/10.1108/07378831211213210</a></span><span lang=EN-GB><o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>So Google Scholar is still not the cure all for all OA available in the world. Interestingly our repository is better indexed in the standard Google search engine rather than the Scholar version.</span><span lang=EN-GB><o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-GB style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>So my point is, doing a search on GS, and finding a lot of hits still doesn’t guarantee to find all the ftxt of those papers. </span><span lang=EN-GB><o:p></o:p></span></p></div></div></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div></div></div></body></html>