[EP-tech] PIRUS plugin issue?

Cox, Alan G. agc at nerc.ac.uk
Fri Mar 13 17:28:24 GMT 2015


I’d appreciate hearing whether anyone using the PIRUS plugin has seen anything similar to the following, or from anyone else with wisdom to offer …
I have noticed that netstat on our eprints service shows many connections to 54.72.175.35:80 in CLOSE_WAIT state, originating from instances of the httpd process (see below) and that these connections take a long time to get to LAST_ACK and then stay in that state for a long time. If I understand what I read correctly, CLOSE_WAIT occurs when the remote end has requested that the TCP connection be closed (sent a FIN packet), to which the local protocol stack has responded with an ACK, and is then waiting for the local process with the connection open to close it.
54.72.175.35 is an Amazon AWS VM that appears to host a number of services but it appears most likely that the service of relevance here is http://www.jusp.mimas.ac.uk/ and what we’ve got is the PIRUS plugin reporting each full-text download to the JUSP COUNTER application.
So it looks as to me as though the plugin is failing to close() the connection promptly on receipt of a close request from the JUSP COUNTER application.
I’m a dilettante when it comes to Perl and eprints code but, from glancing at the plugin code, I cannot see anything obviously amiss, so I’m guessing that the answer lies inside LWP::UserAgent or LWP::ConnCache as used by the plugin.
Does anyone else recognise this behaviour or have any suggestions on how to fix it, or can tell me I’m barking up the wrong tree?
This came to light because on a few occasions recently the NERC eprints service has become completely unresponsive with connections hanging and timing out and the logs recording many HTTP 500 errors and “Software caused connection abort at /opt/eprints3/perl_lib/EPrints/Page.pm line 78.\n”.
From the netstat output I guessed that we were hitting the limit on the number of httpd processes and was able to recover by stopping Apache, waiting until the connections cleared, then restarting it. With this done, eprints springs back into life.
Thanks,
            Alan.

[50%20email%20logo]

Alan Cox | Infrastructure Team
NERC Research Technology Services
Polaris House, North Star Avenue, Swindon, SN2 1EU, UK
Tel: +44 (0)1793 411963 | Email: agc at nerc.ac.uk<mailto:agc at nerc.ac.uk>
NERC<http://www.nerc.ac.uk/> | Planet Earth Online<http://www.planetearth.nerc.ac.uk/> | Follow @NERCscience<https://twitter.com/NERCscience> & @NewOnNORA<https://twitter.com/NewOnNORA>



netstat output extract:

tcp        1      1 139.166.209.11:45805        54.72.175.35:80             LAST_ACK    -
tcp        1      0 139.166.209.11:45810        54.72.175.35:80             CLOSE_WAIT  12784/httpd
tcp        1      0 139.166.209.11:45822        54.72.175.35:80             CLOSE_WAIT  12704/httpd
tcp        1      0 139.166.209.11:45820        54.72.175.35:80             CLOSE_WAIT  12633/httpd
tcp        1      0 139.166.209.11:45768        54.72.175.35:80             CLOSE_WAIT  11758/httpd
tcp        1      0 139.166.209.11:45785        54.72.175.35:80             CLOSE_WAIT  11773/httpd
tcp        1      0 139.166.209.11:45856        54.72.175.35:80             CLOSE_WAIT  12497/httpd
tcp        0      0 139.166.209.11:45870        54.72.175.35:80             ESTABLISHED 12217/httpd
tcp        1      0 139.166.209.11:45866        54.72.175.35:80             CLOSE_WAIT  10528/httpd
tcp        1      0 139.166.209.11:45864        54.72.175.35:80             CLOSE_WAIT  12620/httpd
tcp        1      0 139.166.209.11:45865        54.72.175.35:80             CLOSE_WAIT  11561/httpd
tcp        1      0 139.166.209.11:45835        54.72.175.35:80             CLOSE_WAIT  11774/httpd
tcp        1      0 139.166.209.11:45832        54.72.175.35:80             CLOSE_WAIT  12785/httpd
tcp        1      0 139.166.209.11:45833        54.72.175.35:80             CLOSE_WAIT  12780/httpd
tcp        1      1 139.166.209.11:45846        54.72.175.35:80             LAST_ACK    -
tcp        1      0 139.166.209.11:45844        54.72.175.35:80             CLOSE_WAIT  12605/httpd
tcp        1      0 139.166.209.11:45842        54.72.175.35:80             CLOSE_WAIT  8836/httpd
tcp        1      1 139.166.209.11:45843        54.72.175.35:80             LAST_ACK    -
tcp        1      1 139.166.209.11:45840        54.72.175.35:80             LAST_ACK    -
tcp        1      0 139.166.209.11:45841        54.72.175.35:80             CLOSE_WAIT  12701/httpd
tcp        1      0 139.166.209.11:45854        54.72.175.35:80             CLOSE_WAIT  12783/httpd
tcp        1      0 139.166.209.11:45852        54.72.175.35:80             CLOSE_WAIT  12632/httpd
tcp        1      0 139.166.209.11:45853        54.72.175.35:80             CLOSE_WAIT  12781/httpd
tcp        1      0 139.166.209.11:45850        54.72.175.35:80             CLOSE_WAIT  12777/httpd
tcp        1      0 139.166.209.11:45848        54.72.175.35:80             CLOSE_WAIT  12789/httpd
tcp        1      0 139.166.209.11:45849        54.72.175.35:80             CLOSE_WAIT  12782/httpd
________________________________
This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system.
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20150313/1014bca5/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 17818 bytes
Desc: image001.png
Url : http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20150313/1014bca5/attachment-0001.png 


More information about the Eprints-tech mailing list