[EP-tech] Server timeouts - MySQL Problems

Newman D.R. drn at ecs.soton.ac.uk
Thu Mar 5 13:37:27 GMT 2020


Hi James,

Several suggestions:

1. Try install atop [1], this creates log files similar to what you get from running the top command.  This will allow you to look back later to see what was going on at the times when the server was not responding.  By default it takes a snapshot every 10 minutes.  It might be worth swaping this to every minute or couple of minutes.

2. Edit MySQL's configuration to introduce a log file for slow running queries [2].

3. I use something called pt-kill to kill very long running queries that may be blocking other queries [3].

Regards

David Newman

[1] https://linux.die.net/man/1/atop<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinux.die.net%2Fman%2F1%2Fatop&data=01%7C01%7C%7C000643adaba24a95e26e08d7c10a5835%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=WQ%2BPdrQT03IPHqhkvuUyVmIKXGxvihkNu06sXDkSng0%3D&reserved=0>

[2] https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdev.mysql.com%2Fdoc%2Frefman%2F5.7%2Fen%2Fslow-query-log.html&data=01%7C01%7C%7C000643adaba24a95e26e08d7c10a5835%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=ELlj0BKcWrtYho%2Bl3vTbL26gEBzUnIMLP9qyM9b3NiM%3D&reserved=0>

[3] https://www.percona.com/doc/percona-toolkit/LATEST/pt-kill.html<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.percona.com%2Fdoc%2Fpercona-toolkit%2FLATEST%2Fpt-kill.html&data=01%7C01%7C%7C000643adaba24a95e26e08d7c10a5835%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=jq6BD8ZQTcRztgD7jv3tlMLWMSrYnb03HdhCshvyC3w%3D&reserved=0>


On 05/03/2020 12:46, James Kerwin via Eprints-tech wrote:
Hi All,

This isn't necessarily directly EPrints related, but its about a server running EPrints.

I've noticed a problem this week with the repository. In the early hours of the morning the number of users drops to zero for several hours between 2am and 6am (according to Google Analytics). Due to having  a cold I've been up between these times and can confirm that the repository website times out when I try to connect from home.

I don't get any memory or CPU warnings from our monitoring software. My gut instinct is that it's an issue with MySQL connections not closing in a timely manner. We do have cron jobs that run at 1:30, 2:30 and 3:30 which I'm aware fall right within the problem zone, but these have been running at the same time for years and have never caused an issue.

Has anybody experienced anything similar to this or have suggestions as to how I could chase it down?

It's a Ubuntu server with MySQL running EPrints 3.3.14. I don't think it's an EPrints issue, but there is nothing in the log files to suggest what's happening. The apache error log is blank for the hours that the server won't connect.

Thanks,
James



*** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
*** Archive: http://www.eprints.org/tech.php/<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.eprints.org%2Ftech.php%2F&data=01%7C01%7C%7C000643adaba24a95e26e08d7c10a5835%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=y7HqOhW%2FKC85j5RQFnS00tuyZlLN0SsWMQOcgeBjLOY%3D&reserved=0>
*** EPrints community wiki: http://wiki.eprints.org/<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwiki.eprints.org%2F&data=01%7C01%7C%7C000643adaba24a95e26e08d7c10a5835%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=oUjAm%2FTM7AlM%2FyV2lvtzKm9rm3BlQ%2B74kxLHCjQYq2U%3D&reserved=0>

[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.avg.com%2Femail-signature%3Futm_medium%3Demail%26utm_source%3Dlink%26utm_campaign%3Dsig-email%26utm_content%3Demailclient&data=01%7C01%7C%7C000643adaba24a95e26e08d7c10a5835%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=FMpcV%2BspSI%2BeF2SJm8EH4DqWUKyhWPO05DL4%2BqEZFPs%3D&reserved=0>     Virus-free. www.avg.com<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.avg.com%2Femail-signature%3Futm_medium%3Demail%26utm_source%3Dlink%26utm_campaign%3Dsig-email%26utm_content%3Demailclient&data=01%7C01%7C%7C000643adaba24a95e26e08d7c10a5835%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=FMpcV%2BspSI%2BeF2SJm8EH4DqWUKyhWPO05DL4%2BqEZFPs%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20200305/05783b12/attachment-0001.html 


More information about the Eprints-tech mailing list