[EP-tech] PIRUS plugin issue?
Cox, Alan G.
agc at nerc.ac.uk
Fri Mar 13 17:28:24 GMT 2015
I’d appreciate hearing whether anyone using the PIRUS plugin has seen anything similar to the following, or from anyone else with wisdom to offer …
I have noticed that netstat on our eprints service shows many connections to 54.72.175.35:80 in CLOSE_WAIT state, originating from instances of the httpd process (see below) and that these connections take a long time to get to LAST_ACK and then stay in that state for a long time. If I understand what I read correctly, CLOSE_WAIT occurs when the remote end has requested that the TCP connection be closed (sent a FIN packet), to which the local protocol stack has responded with an ACK, and is then waiting for the local process with the connection open to close it.
54.72.175.35 is an Amazon AWS VM that appears to host a number of services but it appears most likely that the service of relevance here is http://www.jusp.mimas.ac.uk/ and what we’ve got is the PIRUS plugin reporting each full-text download to the JUSP COUNTER application.
So it looks as to me as though the plugin is failing to close() the connection promptly on receipt of a close request from the JUSP COUNTER application.
I’m a dilettante when it comes to Perl and eprints code but, from glancing at the plugin code, I cannot see anything obviously amiss, so I’m guessing that the answer lies inside LWP::UserAgent or LWP::ConnCache as used by the plugin.
Does anyone else recognise this behaviour or have any suggestions on how to fix it, or can tell me I’m barking up the wrong tree?
This came to light because on a few occasions recently the NERC eprints service has become completely unresponsive with connections hanging and timing out and the logs recording many HTTP 500 errors and “Software caused connection abort at /opt/eprints3/perl_lib/EPrints/Page.pm line 78.\n”.
From the netstat output I guessed that we were hitting the limit on the number of httpd processes and was able to recover by stopping Apache, waiting until the connections cleared, then restarting it. With this done, eprints springs back into life.
Thanks,
Alan.
[50%20email%20logo]
Alan Cox | Infrastructure Team
NERC Research Technology Services
Polaris House, North Star Avenue, Swindon, SN2 1EU, UK
Tel: +44 (0)1793 411963 | Email: agc at nerc.ac.uk<mailto:agc at nerc.ac.uk>
NERC<http://www.nerc.ac.uk/> | Planet Earth Online<http://www.planetearth.nerc.ac.uk/> | Follow @NERCscience<https://twitter.com/NERCscience> & @NewOnNORA<https://twitter.com/NewOnNORA>
netstat output extract:
tcp 1 1 139.166.209.11:45805 54.72.175.35:80 LAST_ACK -
tcp 1 0 139.166.209.11:45810 54.72.175.35:80 CLOSE_WAIT 12784/httpd
tcp 1 0 139.166.209.11:45822 54.72.175.35:80 CLOSE_WAIT 12704/httpd
tcp 1 0 139.166.209.11:45820 54.72.175.35:80 CLOSE_WAIT 12633/httpd
tcp 1 0 139.166.209.11:45768 54.72.175.35:80 CLOSE_WAIT 11758/httpd
tcp 1 0 139.166.209.11:45785 54.72.175.35:80 CLOSE_WAIT 11773/httpd
tcp 1 0 139.166.209.11:45856 54.72.175.35:80 CLOSE_WAIT 12497/httpd
tcp 0 0 139.166.209.11:45870 54.72.175.35:80 ESTABLISHED 12217/httpd
tcp 1 0 139.166.209.11:45866 54.72.175.35:80 CLOSE_WAIT 10528/httpd
tcp 1 0 139.166.209.11:45864 54.72.175.35:80 CLOSE_WAIT 12620/httpd
tcp 1 0 139.166.209.11:45865 54.72.175.35:80 CLOSE_WAIT 11561/httpd
tcp 1 0 139.166.209.11:45835 54.72.175.35:80 CLOSE_WAIT 11774/httpd
tcp 1 0 139.166.209.11:45832 54.72.175.35:80 CLOSE_WAIT 12785/httpd
tcp 1 0 139.166.209.11:45833 54.72.175.35:80 CLOSE_WAIT 12780/httpd
tcp 1 1 139.166.209.11:45846 54.72.175.35:80 LAST_ACK -
tcp 1 0 139.166.209.11:45844 54.72.175.35:80 CLOSE_WAIT 12605/httpd
tcp 1 0 139.166.209.11:45842 54.72.175.35:80 CLOSE_WAIT 8836/httpd
tcp 1 1 139.166.209.11:45843 54.72.175.35:80 LAST_ACK -
tcp 1 1 139.166.209.11:45840 54.72.175.35:80 LAST_ACK -
tcp 1 0 139.166.209.11:45841 54.72.175.35:80 CLOSE_WAIT 12701/httpd
tcp 1 0 139.166.209.11:45854 54.72.175.35:80 CLOSE_WAIT 12783/httpd
tcp 1 0 139.166.209.11:45852 54.72.175.35:80 CLOSE_WAIT 12632/httpd
tcp 1 0 139.166.209.11:45853 54.72.175.35:80 CLOSE_WAIT 12781/httpd
tcp 1 0 139.166.209.11:45850 54.72.175.35:80 CLOSE_WAIT 12777/httpd
tcp 1 0 139.166.209.11:45848 54.72.175.35:80 CLOSE_WAIT 12789/httpd
tcp 1 0 139.166.209.11:45849 54.72.175.35:80 CLOSE_WAIT 12782/httpd
________________________________
This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system.
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20150313/1014bca5/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 17818 bytes
Desc: image001.png
Url : http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20150313/1014bca5/attachment-0001.png
More information about the Eprints-tech
mailing list