<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi All, <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Over the last few days, we've been sorting out a few kinks with the with fulltext searching / index creation on our local EPrints repository and thought I'd pass along the notes in the hopes that it might help out others. The issues
were noted upon performing the query noted by Paolo Tealdi a few days back seeking malformed content in the eprint index table:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:black">select *,length(word) from eprint__rindex where length(word) > 35</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In our local results we noted an number of 'word' values corresponding to eprints with pdf documents in which series of valid words were string together with assorted Unicode interspersed.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The offending / troublesome Unicode values interspersed were inserted in the export from pdf to text, as called by eprints to generate the source fulltext to be indexed (called as '$(pdftotext) -enc UTF-8 -layout $(SOURCE) $(TARGET)').
Owing to the '-layout' argument, many spaces, line endings and paragraph endings were converted to UTF-8 formatting characters not handled by the default tokenizer (e.g. space to 'NON BREAKING SPACE' "chr(0x0a)", line ending to 'LINE SEPARATOR' - "\x{2028}"
and paragraph ending to 'PARAGRAPH SEPARATOR' - "\x{2029}"). <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">These are easily identifiable for insertion into the list of delimiters, however, it seems that the list of delimiters ('FREETEXT_SEPERATOR_CHARS') is defined in both ~eprints/archives/{archiveid}/cfg/cfg.d/indexing.pl and ~eprints/perl_lib/EPrints/Index/Tokenizer.pm,
only the latter of which appears to have any effect. (The former may be orphaned code specific to our repository)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As may also be of note - in our case, resetting the indexed values seemed to require reloading the config (restarting apache and the indexer - to update Tokenizer.pm), as well as dropping the contents of the eprint__rindex table all before
finally running epadmin erase_fulltext_index. To any who might be having their search misbehave, hopefully this may be of some help - any warnings, criticisms or comments welcome!
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">NB: as our config could differ significantly from those out there, it might be best to test the above on a non-critical / test repository if it is of interest to you.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal">Casey<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-CA">Casey Hilliard<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-CA">PC Consultant,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-CA">Health Sciences Library / QE2 Systems,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-CA">Memorial University<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-CA">Phone: 709-777-2387 (HSL)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-CA">Phone: 709-864-6267 (QE2)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;mso-fareast-language:EN-CA">This communication is intended as a private communication for the sole use of the primary addressee. The information contained herein is private and confidential. If you are not the
intended receipient, you are hereby notified that copying, forwarding or other dissemination or distribution of this communication by any means is prohibited. If you are not specifically authorized to receive this communication and you believe that you have
received it in error, please notify the original sender immediately.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>This electronic communication is governed by the terms and conditions at http://www.mun.ca/cc/policies/electronic_communications_disclaimer_2012.php
</body>
</html>