<html><body>
<p><font size="2" face="sans-serif">Hi,</font><br>
<br>
<font size="2" face="sans-serif">we can reproduce the behavior:</font><br>
<br>
<font size="2" face="sans-serif">Advanced search (which goes to the SQL index): Ågren, ågren, "Ågren" and "ågren" all fail</font><br>
<br>
<font size="2" face="sans-serif">Quick search (which goes to the Xapian index:) both creators_name:ågren and creators_name:Ågren find results (creators_name is the field name we use for authors)</font><br>
<br>
<font size="2" face="sans-serif">perl_lib/EPrints/Index/Tokenizer.pm contains a translation list that maps Unicode characters to ASCII - Å is missing there. Maybe this is the clue?</font><br>
<br>
<font size="2" face="sans-serif">Best regards,</font><br>
<br>
<font size="2" face="sans-serif">Martin</font><br>
<br>
<font size="2" face="sans-serif">--</font><br>
<font size="2" face="sans-serif">Dr. Martin Brändle</font><br>
<font size="2" face="sans-serif">Zentrale Informatik</font><br>
<font size="2" face="sans-serif">Universität Zürich</font><br>
<font size="2" face="sans-serif">Stampfenbachstr. 73</font><br>
<font size="2" face="sans-serif">CH-8006 Zürich</font><br>
<br>
<br>
<img width="16" height="16" src="cid:1__=4EBBF5CEDFA398258f9e8a93df9@lotus.uzh.ch" border="0" alt="Inactive hide details for Christer Enkvist ---17/02/2016 17:20:34---Hello all! I have encountered a weird UTF-8 related problem"><font size="2" color="#424282" face="sans-serif">Christer Enkvist ---17/02/2016 17:20:34---Hello all! I have encountered a weird UTF-8 related problem when querying names in the advanced sear</font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">Von: </font><font size="1" face="sans-serif">Christer Enkvist <christer.enkvist@slu.se></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">An: </font><font size="1" face="sans-serif">"eprints-tech@ecs.soton.ac.uk" <eprints-tech@ecs.soton.ac.uk></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Datum: </font><font size="1" face="sans-serif">17/02/2016 17:20</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Betreff: </font><font size="1" face="sans-serif">[EP-tech] Searching fails when database field contains Å (utf8 %c3%85)</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Gesendet von: </font><font size="1" face="sans-serif">eprints-tech-bounces@ecs.soton.ac.uk</font><br>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<font size="2" face="Arial">Hello all!</font><br>
<font size="2" face="Arial"> </font><br>
<font size="2" face="Arial">I have encountered a weird UTF-8 related problem when querying names in the advanced search. If the name of an author contains Å, like Ångström, (UTF-8 %c3%85, A with a ring above) then querying will fail. I have not seen the problem for any other character, e.g. no problem with ”å” (a with ring above), %c3%a5, or any other non A-Z letter such as ä,Ä,ö, or Ö. The problem is when the database entry itself contains an Å, which is typically when the character is the first in the name like Ångström or in a hyphened name like Per-Åke.</font><br>
<font size="2" face="Arial"> </font><br>
<font size="2" face="Arial">Furthermore, if the queryterm contains an “Å” then it will fail. A few examples:</font><br>
<font size="2" face="Arial"> </font><br>
<font size="2" face="Arial">Mårten – works</font><br>
<font size="2" face="Arial">mårten – works</font><br>
<font size="2" face="Arial">MåRTEN -- works</font><br>
<font size="2" face="Arial">MÅRTEN -- fails</font><br>
<font size="2" face="Arial">mÅrten -- fails</font><br>
<font size="2" face="Arial"> </font><br>
<font size="2" face="Arial">The query field is (normally) case insensitive so it shouldn’t matter if I write “ångström” or “Ångström”. However, hit or miss in this case depends on if the database have an Å and/or the query term contains an Å as it seems like Eprints cannot handle “Å”. Always, displays correct and is correctly written into the database. Only problem is the advanced search.</font><br>
<font size="2" face="Arial"> </font><br>
<font size="2" face="Arial">Should add that querying the database using SQL works without any problems (incl all upper/lower combinations). Any ideas what may be wrong with Eprints and where to start looking? </font><br>
<font size="2" face="Arial"> </font><br>
<font size="2" face="Arial">Regards,</font><br>
<font size="2" face="Arial">Christer</font><br>
<font size="2" face="Arial"> </font><br>
<font size="2" face="Arial"> </font><br>
<font size="2" face="Calibri"><b>Christer Enkvist, Ph D</b></font><br>
<font size="2" face="Calibri">System Administrator/System Librarian</font><br>
<font size="2" face="Calibri">Division of Scholarly Communication </font><br>
<font size="2" face="Calibri">Swedish University of Agricultural Sciences</font><br>
<font size="2" face="Calibri">Uppsala, Sweden</font><br>
<font size="2" face="Calibri"> </font><br>
<font size="2" face="Calibri">Telephone: 018-671042</font><br>
<font size="2" face="Calibri"> </font><tt><font size="2">*** Options: </font></tt><tt><font size="2"><a href="http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech">http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech</a></font></tt><tt><font size="2"><br>
*** Archive: </font></tt><tt><font size="2"><a href="http://www.eprints.org/tech.php/">http://www.eprints.org/tech.php/</a></font></tt><tt><font size="2"><br>
*** EPrints community wiki: </font></tt><tt><font size="2"><a href="http://wiki.eprints.org/">http://wiki.eprints.org/</a></font></tt><tt><font size="2"><br>
*** EPrints developers Forum: </font></tt><tt><font size="2"><a href="http://forum.eprints.org/">http://forum.eprints.org/</a></font></tt><tt><font size="2"><br>
</font></tt><br>
</body></html>