<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Hi James,</p>
<p>Technically, there is also mtop (m for mysql) but I don't think it is in the standard package repositories and is only useful for keeping an eye on something at the time it is happening, unless you can hack a way to get it to log like atop.</p>
<p>Regards</p>
<p>David Newman<br>
</p>
<div class="moz-cite-prefix">On 05/03/2020 14:37, James Kerwin wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAKkNZ9CEDPC1ygkBQFUsKA_tU39b&#43;im_31Tv3ZAOkxKKawTa2w@mail.gmail.com">
<div dir="ltr">Thanks David!
<div><br>
</div>
<div>I've used top and htop, but never atop. I've just installed it so I will get to investigating it now. Sounds like it could be really useful. Improves on my previous idea of staying up until the suspected failure time and looking at which processes were
 running.</div>
<div><br>
</div>
<div>I'll get working on point two now.</div>
<div><br>
</div>
<div>I may delay point three until I'm really stuck. Although I have just noticed that the Elements &quot;get_records&quot; script appears to run for longer than an hour, so it's for example still running the 1pm script when the 2pm script starts.</div>
<div><br>
</div>
<div>I'm trying to decide if there's any great harm in doing frequent curl calls to the homepage from another server to see at which point it fails so I can pin down a more precise time for the problem.</div>
<div><br>
</div>
<div>So much to investigate!</div>
<div><br>
</div>
<div>Thanks again for your advice. It's greatly appreciated.</div>
<div><br>
</div>
<div>James</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Mar 5, 2020 at 1:37 PM Newman D.R. &lt;<a href="mailto:drn@ecs.soton.ac.uk" moz-do-not-send="true">drn@ecs.soton.ac.uk</a>&gt; wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi James,</p>
<p>Several suggestions:</p>
<p>1. Try install atop [1], this creates log files similar to what you get from running the top command.&nbsp; This will allow you to look back later to see what was going on at the times when the server was not responding.&nbsp; By default it takes a snapshot every
 10 minutes.&nbsp; It might be worth swaping this to every minute or couple of minutes.</p>
<p>2. Edit MySQL's configuration to introduce a log file for slow running queries [2].</p>
<p>3. I use something called pt-kill to kill very long running queries that may be blocking other queries [3].<br>
</p>
<p>Regards</p>
<p>David Newman<br>
</p>
<p>[1] <a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinux.die.net%2Fman%2F1%2Fatop&amp;data=01%7C01%7C%7C7ad1d54da0074526f43508d7c113d248%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=OIoZx72wcd08HLfEZOUCg0FVQd7mC45wR20QpnJp0ik%3D&amp;reserved=0" originalsrc="https://linux.die.net/man/1/atop" shash="wjBvX4k85q2ZnqK4cKzDxr&#43;jPOxHNaPhCg0DIMXwBL&#43;Q/M/l7nJf8suuMXSKD&#43;kAKKzuKdPIgIc3YK4Zjn8E9rvle1BFotc9zK4DObTVZiKdkcz85uEylmH9Ybi&#43;rWa1yr5E/d0vZV0ruOipNMr9fK7za5xHQQD/P4qdpa/lRbs=" originalsrc="https://linux.die.net/man/1/atop" shash="BafrTeY5CMUTF/QKb1PSu6M2s0OKlzvRRMCLJon86VBEN1&#43;Iq0jhYmrMT/UmbwInWq8Skcf9cb0ENVvkCdYgJwBk/LpCJZDFAUg1E47zwgiGDjJmNZMuUTHSLBrkhMgSs5AX5nJheDmiTj6kvswzV2Zo8XOzy9UvwSJLgrF2CsU=" target="_blank" moz-do-not-send="true">
https://linux.die.net/man/1/atop</a><br>
</p>
<p>[2] <a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.mysql.com%2Fdoc%2Frefman%2F5.7%2Fen%2Fslow-query-log.html&amp;data=01%7C01%7C%7C7ad1d54da0074526f43508d7c113d248%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=GyF%2BaaqHZqQPWOMRzdYuvcYsOxUPw94UOYlOES8HbL0%3D&amp;reserved=0" originalsrc="https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html" shash="xZiAj8A1TLygGutH22E2lG0C&#43;FCZpd3B1DA1k7zMop62Dga6Hi/a/84H8BVO4xZNvA0wVi8wiillFHB4mjnyNgRsl8AdZiBjuWH32K7Vrn9J0OxUkBzioR/JZ10Gnmppcc4ZB0EwnvhKxOOA07UdulWiXC67D8WvsHCahRx3H9w=" originalsrc="https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html" shash="WJFCxEwFqtb&#43;OKK2yoaTLvvyOxH14ovz0MThcAMycyGH9dSFUxq8j2uaqMtQysbEVoTz3WeTAMfdUaBHLwKI7yRAf53jxQvd&#43;c8BU7QcR8Aehs0qYkmY9jOtUnpHr8fTughEkSnWzbCe2QPm8VwL&#43;gGP7TlygAGcisGlO8/s/QQ=" target="_blank" moz-do-not-send="true">
https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html</a></p>
<p>[3] <a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.percona.com%2Fdoc%2Fpercona-toolkit%2FLATEST%2Fpt-kill.html&amp;data=01%7C01%7C%7C7ad1d54da0074526f43508d7c113d248%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=SYE5WWmN2jsLi73ejPR29tMU8VSvYLD2d3MiZb3XaHQ%3D&amp;reserved=0" originalsrc="https://www.percona.com/doc/percona-toolkit/LATEST/pt-kill.html" shash="Tz13gwe6/sbwlrP6A8Azq5Kxgh2zkToTGnuGtQDyWxTxN0kMSjK0zrDTU9F7iYOiwUBO81a9ZWxeYOjd8qJhrCVURaLmRhgxff8&#43;qnbW9uVhyCK8/8oTZzEndoMIBCgQOWODXh6yO/OHEVoF//4&#43;9lJPoJMyyi7gkDLRo2UJ09s=" originalsrc="https://www.percona.com/doc/percona-toolkit/LATEST/pt-kill.html" shash="TxyHw5sn80Cn0w6xVSqCZMME&#43;0PXh&#43;sFOEaCRDhnOGm8AL9LhyI3LRX/FOseANgSwq6nFgNJuziv6oU6vbwSLPBc7Tzl&#43;Wz/YDgz0JDHQDkfIl9jbOxXShzFbzGolSOdEGq/6o9JbmSVWZvsVUbBxFvLwIR6q0HwCe2vREWqWbQ=" target="_blank" moz-do-not-send="true">
https://www.percona.com/doc/percona-toolkit/LATEST/pt-kill.html</a></p>
<p><br>
</p>
<div>On 05/03/2020 12:46, James Kerwin via Eprints-tech wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><font face="tahoma, sans-serif">Hi All,<br>
</font>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif">This isn't necessarily directly EPrints related, but its about a server running EPrints.</font></div>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif">I've noticed a problem this week with the repository. In the early hours of the morning the number of users drops to zero for several hours between 2am and 6am (according to Google Analytics). Due to having&nbsp; a cold I've
 been up between these times and can confirm that the repository website times out when I try to connect from home.</font></div>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif">I don't get any memory or CPU warnings from our monitoring software. My gut instinct is that it's an issue with MySQL connections not closing in a timely manner. We do have cron jobs that run at 1:30, 2:30 and 3:30 which
 I'm aware fall right within the problem zone, but these have been running at the same time for years and have never caused an issue.</font></div>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif">Has anybody experienced anything similar to this or have suggestions as to how I could chase it down?</font></div>
<div><font face="tahoma, sans-serif"><br>
</font></div>
<div><font face="tahoma, sans-serif">It's a Ubuntu server with MySQL running EPrints 3.3.14<font color="#000000">. I don't think it's an EPrints issue, but there is nothing in the log files to suggest what's happening.
</font><font color="#000000">The apache error log is blank for the hours that the server won't connect.</font></font></div>
<div><font face="tahoma, sans-serif"><font color="#000000"><br>
</font></font></div>
<div><font face="tahoma, sans-serif"><font color="#000000">Thanks,</font></font></div>
<div><font face="tahoma, sans-serif"><font color="#000000">James</font></font></div>
</div>
<br>
<fieldset></fieldset>
<pre>*** Options: <a href="http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech" target="_blank" moz-do-not-send="true">http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech</a>
*** Archive: <a href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&amp;data=01%7C01%7C%7C7ad1d54da0074526f43508d7c113d248%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=FKqf8b7%2B7qH4yuWrOA7FA6Vk9QULZL6O4h9WLHUwjvQ%3D&amp;reserved=0" originalsrc="http://www.eprints.org/tech.php/" shash="YpUnZ3sf3KV&#43;zKVrm3H/KBRs04lXZOoCzUGGtBk4XDPg5LyctyDxrreglxyXNsRbEbnBVzuv2GnYfK1upWxXsE6b/ufbbxShmPV/sCTGBsmCahXYyfWXlF7Irwif109s0PEnPVlTkayBNLO/zVlawBkEIcf6aZx6k5GlnSb6RkY=" originalsrc="http://www.eprints.org/tech.php/" shash="DCl2vsBg8eZET&#43;WeORic3CNHp5DhuqF4KKLxJOMs8A7oD7G2rCKzVv3MHx&#43;VMO03Fg8QPPtVEtqCh4gLc2Jahz3XQ4Baafh9wC8ivkqswJP52FtsBDcTVTJ5qc0axYcMGUb&#43;7TY6eAgiqv6Us1LPZ/p&#43;EuHnKm1cLI&#43;rXafMPMs=" target="_blank" moz-do-not-send="true">http://www.eprints.org/tech.php/</a>
*** EPrints community wiki: <a href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&amp;data=01%7C01%7C%7C7ad1d54da0074526f43508d7c113d248%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=XrQ1V01jWKSMYvBJJ5%2B5LTM7CcztIXFDus0puKQLd1M%3D&amp;reserved=0" originalsrc="http://wiki.eprints.org/" shash="okD/pSxP8XDlRHg3V8e2OgPBBM6Nb1FVI5EDSsOhgxanrXyrvBn7qNizssY2Z/ptGJITP8MO&#43;RzVnpqqlEgLj8SgN8D/AW8Jc5o1g8WT7ZBUn8NoRqZWwHGb754JQP85cVetXGHEi4fyfRV&#43;EC3OCkTnHxII3yog0849UmlPFYE=" originalsrc="http://wiki.eprints.org/" shash="elzUrUiJowR9NPFTThwyj8UOIzKVUlmRjD4FZ5T/mcYV/jKqqB&#43;fUERlHpYcFQDfyT/dYWi1Xj3E4sEdv1mz83nHazafffyo7eoDFpCL9q7018TUv2Zrb3q6B4bFjSIhp3ucG68UtEYVXqC9OH81ymW&#43;MlkC8nTxLS9Q9k5XSQI=" target="_blank" moz-do-not-send="true">http://wiki.eprints.org/</a></pre>
</blockquote>
<div id="gmail-m_-9132979355531325136DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br>
<table style="border-top:1px solid rgb(211,212,222)">
<tbody>
<tr>
<td style="width:55px;padding-top:13px"><a href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.avg.com%2Femail-signature%3Futm_medium%3Demail%26utm_source%3Dlink%26utm_campaign%3Dsig-email%26utm_content%3Demailclient&amp;data=01%7C01%7C%7C7ad1d54da0074526f43508d7c113d248%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=qXKNpNaVQNJGrQmY7EMaMSdq5NDxTVk8%2BjiKHOxHF1w%3D&amp;reserved=0" originalsrc="http://www.avg.com/email-signature?utm_medium=email&amp;utm_source=link&amp;utm_campaign=sig-email&amp;utm_content=emailclient" shash="HY3ZfXkTuJftPrHmRvniAEvrBtHkC/8UVM70zd0Zd8YRHqD2UdO&#43;kk9EddlIUH0ujQPbpUWr/8B8n1F8AEQkDltGNmFBQAwBEvffurMMBuOLhDQlhBkCr7kgCMEC0fG51XpjskTyO7MV7I1uoaZ5wmu8elghARm7AYtigBTd9L0=" originalsrc="http://www.avg.com/email-signature?utm_medium=email&amp;utm_source=link&amp;utm_campaign=sig-email&amp;utm_content=emailclient" shash="BcG1yL1wLiEmkfhXnwFhBxSlSYPWQDm2WM0StJJYuA6lL/qV7Flc0p7/iBDNej0vRbMgOvBazUP7U70Ml6cLBxVmcZ3inBLMXlIDCLlhqaUJR&#43;RNTQLGre2wcz9GAdph0PtCrpSOEE3Kpxt&#43;TVsd6d4mNJGiyvs9w2T0qd8oNsE=" target="_blank" moz-do-not-send="true"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png" alt="" style="width: 46px; height: 29px;" moz-do-not-send="true" width="46" height="29"></a></td>
<td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">
Virus-free. <a href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.avg.com%2Femail-signature%3Futm_medium%3Demail%26utm_source%3Dlink%26utm_campaign%3Dsig-email%26utm_content%3Demailclient&amp;data=01%7C01%7C%7C7ad1d54da0074526f43508d7c113d248%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&amp;sdata=qXKNpNaVQNJGrQmY7EMaMSdq5NDxTVk8%2BjiKHOxHF1w%3D&amp;reserved=0" originalsrc="http://www.avg.com/email-signature?utm_medium=email&amp;utm_source=link&amp;utm_campaign=sig-email&amp;utm_content=emailclient" shash="HY3ZfXkTuJftPrHmRvniAEvrBtHkC/8UVM70zd0Zd8YRHqD2UdO&#43;kk9EddlIUH0ujQPbpUWr/8B8n1F8AEQkDltGNmFBQAwBEvffurMMBuOLhDQlhBkCr7kgCMEC0fG51XpjskTyO7MV7I1uoaZ5wmu8elghARm7AYtigBTd9L0=" originalsrc="http://www.avg.com/email-signature?utm_medium=email&amp;utm_source=link&amp;utm_campaign=sig-email&amp;utm_content=emailclient" shash="BcG1yL1wLiEmkfhXnwFhBxSlSYPWQDm2WM0StJJYuA6lL/qV7Flc0p7/iBDNej0vRbMgOvBazUP7U70Ml6cLBxVmcZ3inBLMXlIDCLlhqaUJR&#43;RNTQLGre2wcz9GAdph0PtCrpSOEE3Kpxt&#43;TVsd6d4mNJGiyvs9w2T0qd8oNsE=" style="color:rgb(68,83,234)" target="_blank" moz-do-not-send="true">
www.avg.com</a> </td>
</tr>
</tbody>
</table>
</div>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>