<html><body>
<p><font size="2" face="sans-serif">Hi,</font><br>
<br>
<font size="2" face="sans-serif">Unfortunately, we cannot produce the pages in advance because the repo is so large (in the order of 300K authors). </font><br>
<font size="2" face="sans-serif">If we were to do that, we would use up all the availabe disk space.</font><br>
<br>
<font size="2" face="sans-serif">Best regards,</font><br>
<br>
<font size="2" face="sans-serif">Martin</font><br>
<br>
<br>
<font size="2" face="sans-serif">--</font><br>
<font size="2" face="sans-serif">Dr. Martin Brändle</font><br>
<font size="2" face="sans-serif">Zentrale Informatik</font><br>
<font size="2" face="sans-serif">Universität Zürich</font><br>
<font size="2" face="sans-serif">Stampfenbachstr. 73</font><br>
<font size="2" face="sans-serif">CH-8006 Zürich</font><br>
<br>
<br>
<img width="16" height="16" src="cid:1__=4EBB0A4FDFC7370B8f9e8a93df9@lotus.uzh.ch" border="0" alt="Inactive hide details for John Salter ---07/03/2017 15:03:32---Hi Jens, We had a similar issue a while ago - which was being ca"><font size="2" color="#424282" face="sans-serif">John Salter ---07/03/2017 15:03:32---Hi Jens, We had a similar issue a while ago - which was being caused by on-the-fly (or request-drive</font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">Von: </font><font size="1" face="sans-serif">John Salter <J.Salter@leeds.ac.uk></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">An: </font><font size="1" face="sans-serif">"eprints-tech@ecs.soton.ac.uk" <eprints-tech@ecs.soton.ac.uk></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Datum: </font><font size="1" face="sans-serif">07/03/2017 15:03</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Betreff: </font><font size="1" face="sans-serif">Re: [EP-tech] Bot net Attacks</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Gesendet von: </font><font size="1" face="sans-serif">eprints-tech-bounces@ecs.soton.ac.uk</font><br>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<font size="2" color="#1F497D" face="Calibri">Hi Jens,</font><br>
<font size="2" color="#1F497D" face="Calibri">We had a similar issue a while ago - which was being caused by on-the-fly (or request-driven) view generation.</font><br>
<font size="2" color="#1F497D" face="Calibri">For author views, where some authors do not have a unique ID, this is computationally intensive, and it is better addressed by making sure you are in control of the page generation.</font><br>
<font size="2" color="#1F497D" face="Calibri">If the pages have already been generated, normal bot activity should not be an issue.</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">We addressed the issue by:</font>
<ul style="padding-left: 36pt"><font size="2" color="#1F497D" face="Calibri">1) Changing the max_age of the person view (and in the end, all views) so regeneration is never triggered by a request</font><br>
<font size="2" color="#1F497D" face="Calibri">2) Changing the generate_views to be able to accept multiple viewids</font><br>
<font size="2" color="#1F497D" face="Calibri">3) Changing the cronjob to generate all views *</font><font size="2" color="#1F497D" face="Calibri"><b>except</b></font><font size="2" color="#1F497D" face="Calibri">* the person one daily, and the person one once a week</font></ul>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">The details are here:</font><br>
<a href="https://github.com/eprints/eprints/pull/417"><font size="2" color="#0000FF" face="Calibri"><u>https://github.com/eprints/eprints/pull/417</u></font></a><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">Our 'subject' and 'year' views have </font><br>
<font size="2" color="#1F497D" face="Calibri">max_menu_age => 10*24*60*60, #10 days</font><br>
<font size="2" color="#1F497D" face="Calibri">max_list_age => 10*24*60*60, #10 days</font><br>
<font size="2" color="#1F497D" face="Calibri">But our 'people' view has:</font><br>
<font size="2" color="#1F497D" face="Calibri">max_menu_age => 20*24*60*60, #20 days</font><br>
<font size="2" color="#1F497D" face="Calibri">max_list_age => 20*24*60*60, #20 days</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">Our crontab looks a bit like this (with the above additions in place):</font><br>
<font size="2" color="#1F497D" face="Calibri"># # # Generate subject and year views Mon-Fri</font><br>
<font size="2" color="#1F497D" face="Calibri">10 3 * * 1-5 <eprints_root>/bin/generate_views <archiveid> --view year --view subject</font><br>
<font size="2" color="#1F497D" face="Calibri"># # # Generate people view on Sunday</font><br>
<font size="2" color="#1F497D" face="Calibri">10 5 * * 0 <eprints_root>/bin/generate_views <archiveid> --view people</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">It might be worth checking to see if your person views are being generated on-the-fly - look at the modification time of the browse pages - compare them to the bot activity. If they are being re-generated by a request the above will help. If not… more thinking to do!</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" color="#1F497D" face="Calibri">Cheers,</font><br>
<font size="2" color="#1F497D" face="Calibri">John</font><br>
<font size="2" color="#1F497D" face="Calibri"> </font><br>
<font size="2" face="Calibri"><b>From:</b></font><font size="2" face="Calibri"> eprints-tech-bounces@ecs.soton.ac.uk [</font><font size="2" face="Calibri"><a href="mailto:eprints-tech-bounces@ecs.soton.ac.uk">mailto:eprints-tech-bounces@ecs.soton.ac.uk</a></font><font size="2" face="Calibri">] </font><font size="2" face="Calibri"><b>On Behalf Of </b></font><font size="2" face="Calibri">jens.vieler@id.uzh.ch</font><font size="2" face="Calibri"><b><br>
Sent:</b></font><font size="2" face="Calibri"> 07 March 2017 13:30</font><font size="2" face="Calibri"><b><br>
To:</b></font><font size="2" face="Calibri"> eprints-tech@ecs.soton.ac.uk</font><font size="2" face="Calibri"><b><br>
Subject:</b></font><font size="2" face="Calibri"> [EP-tech] Bot net Attacks</font><br>
<font size="3" face="Times New Roman"> </font>
<p><font size="2" face="Arial">Dear List</font><font size="3" face="Times New Roman"><br>
</font><font size="2" face="Arial"><br>
during the last 2 weeks we observe some really tricky bot net attacks. Thousands of requests seem to ask for author information and as a result, we ran into disk space problems during the massive pagecreation for /view/authorsnew into the cache.</font><font size="3" face="Times New Roman"><br>
</font><font size="2" face="Arial"><br>
We did some interesting experiments with apaches mod_evasive. But unfortunately bot nets change their behaviour to maximum allowed acces per timeframe we configured. Looks like they know what to do to drive eprints into trouble...</font><font size="3" face="Times New Roman"><br>
</font><font size="2" face="Arial"><br>
What do you recommend?</font><font size="3" face="Times New Roman"><br>
</font><font size="2" face="Arial"><br>
Jens</font><font size="3" face="Times New Roman"><br>
</font><font size="2" face="Arial"><br>
-- <br>
Jens Vieler<br>
Zentrale Informatik<br>
Universität Zürich<br>
Stampfenbachstrasse 73<br>
CH-8006 Zürich<br>
<br>
mail: </font><a href="mailto:jens.vieler@id.uzh.ch"><font size="2" color="#0000FF" face="Arial"><u>jens.vieler@id.uzh.ch</u></font></a><font size="2" face="Arial"><br>
phone: +41 44 63 56777</font><font size="2" color="#0000FF" face="Arial"><u><br>
</u></font><a href="http://www.id.uzh.ch/"><font size="2" color="#0000FF" face="Arial"><u>http://www.id.uzh.ch</u></font></a><font size="3" face="Times New Roman"><br>
<br>
</font><img src="cid:1__=4EBB0A4FDFC7370B8f9e8a93df9@lotus.uzh.ch" width="16" height="16" alt="Inactive hide details for Andrew Collington ---07.03.2017 12:44:27---Hi John, That helps a huge amount, thanks you so much! Th"><font size="2" color="#424282" face="Arial">Andrew Collington ---07.03.2017 12:44:27---Hi John, That helps a huge amount, thanks you so much! This gives me a great base on which to maybe</font><font size="3" face="Times New Roman"><br>
</font><font size="1" color="#5F5F5F" face="Arial"><br>
Von: </font><font size="1" face="Arial">Andrew Collington <</font><a href="mailto:a.p.collington@sussex.ac.uk"><font size="1" color="#0000FF" face="Arial"><u>a.p.collington@sussex.ac.uk</u></font></a><font size="1" face="Arial">></font><font size="1" color="#5F5F5F" face="Arial"><br>
An: </font><font size="1" face="Arial">"</font><a href="mailto:eprints-tech@ecs.soton.ac.uk"><font size="1" color="#0000FF" face="Arial"><u>eprints-tech@ecs.soton.ac.uk</u></font></a><font size="1" face="Arial">" <</font><a href="mailto:eprints-tech@ecs.soton.ac.uk"><font size="1" color="#0000FF" face="Arial"><u>eprints-tech@ecs.soton.ac.uk</u></font></a><font size="1" face="Arial">></font><font size="1" color="#5F5F5F" face="Arial"><br>
Datum: </font><font size="1" face="Arial">07.03.2017 12:44</font><font size="1" color="#5F5F5F" face="Arial"><br>
Betreff: </font><font size="1" face="Arial">Re: [EP-tech] Easier way to do this in a citation?</font><font size="1" color="#5F5F5F" face="Arial"><br>
Gesendet von: </font><a href="mailto:eprints-tech-bounces@ecs.soton.ac.uk"><font size="1" color="#0000FF" face="Arial"><u>eprints-tech-bounces@ecs.soton.ac.uk</u></font></a><br>
<hr width="100%" size="2" align="left" noshade><br>
<font size="3" face="Times New Roman"><br>
<br>
</font><font size="2" color="#1F497D" face="Calibri"><br>
Hi John,<br>
<br>
That helps a huge amount, thanks you so much! This gives me a great base on which to maybe add some functionality of our own and the citation reworking using the choose/when does make it a bit clearer until I do (hopefully) get something written. So many thanks for a very helpful response!<br>
<br>
Andy<br>
<br>
</font><font size="2" face="Calibri"><b><br>
From:</b></font><font size="2" face="Calibri"> </font><a href="mailto:eprints-tech-bounces@ecs.soton.ac.uk"><font size="2" color="#0000FF" face="Calibri"><u>eprints-tech-bounces@ecs.soton.ac.uk</u></font></a><font size="2" face="Calibri"> [</font><a href="mailto:eprints-tech-bounces@ecs.soton.ac.uk"><font size="2" color="#0000FF" face="Calibri"><u>mailto:eprints-tech-bounces@ecs.soton.ac.uk</u></font></a><font size="2" face="Calibri">] </font><font size="2" face="Calibri"><b>On Behalf Of </b></font><font size="2" face="Calibri">John Salter</font><font size="2" face="Calibri"><b><br>
Sent:</b></font><font size="2" face="Calibri"> 03 March 2017 15:58</font><font size="2" face="Calibri"><b><br>
To:</b></font><font size="2" face="Calibri"> </font><a href="mailto:eprints-tech@ecs.soton.ac.uk"><font size="2" color="#0000FF" face="Calibri"><u>eprints-tech@ecs.soton.ac.uk</u></font></a><font size="2" face="Calibri"><b><br>
Subject:</b></font><font size="2" face="Calibri"> Re: [EP-tech] Easier way to do this in a citation?<br>
</font><font size="2" color="#1F497D" face="Calibri"><br>
Hi Andrew,<br>
I don't think there is a easy way to do what you require in the existing epscript functions (but there is a way to add it - more on that below!).<br>
<br>
You could alter your existing code to make the tests clearer:<br>
<choose><br>
<when test="event_title and event_location and event_dates"><br>
<print expr="event_title" />, <print expr="event_location"/>, <print expr="event_dates"/>.<br>
</when><br>
<when test="event_title and event_location"><br>
<print expr="event_title" />, <print expr="event_location"/>.<br>
</when><br>
<when test="event_title "><br>
<print expr="event_title" />.<br>
</when><br>
</choose><br>
<br>
But this does feel a bit noisy too.<br>
<br>
To add (inject) a custom method to EPrints::Script::Compiled, see the example here: </font><a href="https://wiki.eprints.org/w/ORCID#Rendering_the_ORCID_in_a_citation"><font size="2" color="#0563C1" face="Calibri"><u>https://wiki.eprints.org/w/ORCID#Rendering_the_ORCID_in_a_citation</u></font></a><font size="2" color="#1F497D" face="Calibri"><br>
This keeps the added code in the repository config - and should work over upgrades (unless there's a major rewrite of EPrints::Script).<br>
In my opinion, any files in <eprints_root>/archives/<archiveid>/cfg/cfg.d/ should be checked as part of an upgrade.<br>
<br>
For a similar example, in White Rose Research Online, we wanted to render event dates in a more 'friendly' human way.<br>
They are stored in the database as 'yyyy-mm-dd - yyyy-mm-dd' (or just yyyy-mm-dd if it was a one-day event), and we wanted e.g. '1-3 Mar 2017' '28 Feb - 1 Mar 2017' or '31 Dec 2016 - 2 Jan 2017'.<br>
This </font><a href="https://gist.github.com/jesusbagpuss/491086533294f864de63115c66719def"><font size="2" color="#0563C1" face="Calibri"><u>https://gist.github.com/jesusbagpuss/491086533294f864de63115c66719def</u></font></a><font size="2" color="#1F497D" face="Calibri"> adds a method to EPrints::Script::Compiled that does this conversion. The citation uses:</font><font size="2" face="Calibri"><br>
<if test="event_dates"><print expr="wrro_human_event_dates(event_dates)"/></if></font><font size="2" color="#1F497D" face="Calibri"><br>
<br>
Hope that helps!<br>
Cheers,<br>
John<br>
<br>
<br>
</font><font size="2" face="Calibri"><b><br>
From:</b></font><font size="2" face="Calibri"> </font><a href="mailto:eprints-tech-bounces@ecs.soton.ac.uk"><font size="2" color="#0563C1" face="Calibri"><u>eprints-tech-bounces@ecs.soton.ac.uk</u></font></a><font size="2" face="Calibri"> [</font><a href="mailto:eprints-tech-bounces@ecs.soton.ac.uk"><font size="2" color="#0563C1" face="Calibri"><u>mailto:eprints-tech-bounces@ecs.soton.ac.uk</u></font></a><font size="2" face="Calibri">] </font><font size="2" face="Calibri"><b>On Behalf Of </b></font><font size="2" face="Calibri">Andrew Collington</font><font size="2" face="Calibri"><b><br>
Sent:</b></font><font size="2" face="Calibri"> 03 March 2017 13:49</font><font size="2" face="Calibri"><b><br>
To:</b></font><font size="2" face="Calibri"> </font><a href="mailto:eprints-tech@ecs.soton.ac.uk"><font size="2" color="#0563C1" face="Calibri"><u>eprints-tech@ecs.soton.ac.uk</u></font></a><font size="2" face="Calibri"><b><br>
Subject:</b></font><font size="2" face="Calibri"> [EP-tech] Easier way to do this in a citation?<br>
<br>
Hi all,<br>
<br>
I’ve recently had to add a few new rules in a citation for a conference proceeding. At the moment I have a number of checks that look something like this:<br>
<br>
<if test="is_set(event_title)"><br>
<print expr="event_title"/><br>
</if><br>
<if test="event_location"><br>
<if test="is_set(event_title)">, </if><br>
<print expr="event_location"/><br>
</if><br>
<if test="is_set(event_dates)">, <print expr="event_dates"/></if><br>
<if test="is_set(event_title) or is_set(event_location) or is_set(event_dates)">.</if><br>
<br>
So it’ll only add a comma before the location if the title is supplied, etc. and the full-stop at the end if any event details are shown. But, well, as you can see it’s a pretty messy way to do things and wondered if there were something a little more streamlined available that would allow you to supply a list of fields and it’ll then automatically put commas between values if the values are there and a full-stop at the end if needs be? I’m only trying to do this with cite tags in citations/eprints/default.xml.<br>
<br>
I did see a ‘pretty_list’ function in Compiled.pm that looks like it may do something like what I want, but despite trying I didn’t see how I could pull this into a citation nor could I find any documentation on the subject. Is that possible?<br>
<br>
If this kind of functionality doesn’t already exist, then what’s the best course of action to adding new types of actions to cite tags? Is it possible to create my own class do add extra actions, or should I update existing modules? (which seems like a bad idea if ever wanting to upgrade). Is there any documentation about doing this kind of thing?<br>
<br>
Many thanks for any advice,<br>
<br>
Andy<br>
<br>
<br>
-- <br>
Andrew Collington<br>
Web Programmer, ITS Client Services<br>
ITS-CS Shawcross, University of Sussex, Falmer, Brighton, BN1 9QT<br>
<br>
T: (01273) 872591 (ext. 2591)<br>
E: </font><a href="mailto:a.p.collington@sussex.ac.uk"><font size="2" color="#0000FF" face="Calibri"><u>a.p.collington@sussex.ac.uk</u></font></a><font size="2" face="Calibri"><br>
</font><font size="2" face="Courier New">*** Options: </font><a href="http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech"><font size="2" color="#0000FF" face="Courier New"><u>http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech</u></font></a><font size="2" face="Courier New"><br>
*** Archive: </font><a href="http://www.eprints.org/tech.php/"><font size="2" color="#0000FF" face="Courier New"><u>http://www.eprints.org/tech.php/</u></font></a><font size="2" face="Courier New"><br>
*** EPrints community wiki: </font><a href="http://wiki.eprints.org/"><font size="2" color="#0000FF" face="Courier New"><u>http://wiki.eprints.org/</u></font></a><font size="2" face="Courier New"><br>
*** EPrints developers Forum: </font><a href="http://forum.eprints.org/"><font size="2" color="#0000FF" face="Courier New"><u>http://forum.eprints.org/</u></font></a><tt><font size="2">*** Options: </font></tt><tt><font size="2"><a href="http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech">http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech</a></font></tt><tt><font size="2"><br>
*** Archive: </font></tt><tt><font size="2"><a href="http://www.eprints.org/tech.php/">http://www.eprints.org/tech.php/</a></font></tt><tt><font size="2"><br>
*** EPrints community wiki: </font></tt><tt><font size="2"><a href="http://wiki.eprints.org/">http://wiki.eprints.org/</a></font></tt><tt><font size="2"><br>
*** EPrints developers Forum: </font></tt><tt><font size="2"><a href="http://forum.eprints.org/">http://forum.eprints.org/</a></font></tt><tt><font size="2"><br>
</font></tt><br>
<br>
</body></html>