[EP-tech] PIRUS plugin issue?

Cox, Alan G. agc at nerc.ac.uk
Fri Mar 13 17:28:24 GMT 2015

I’d appreciate hearing whether anyone using the PIRUS plugin has seen anything similar to the following, or from anyone else with wisdom to offer …
I have noticed that netstat on our eprints service shows many connections to in CLOSE_WAIT state, originating from instances of the httpd process (see below) and that these connections take a long time to get to LAST_ACK and then stay in that state for a long time. If I understand what I read correctly, CLOSE_WAIT occurs when the remote end has requested that the TCP connection be closed (sent a FIN packet), to which the local protocol stack has responded with an ACK, and is then waiting for the local process with the connection open to close it. is an Amazon AWS VM that appears to host a number of services but it appears most likely that the service of relevance here is http://www.jusp.mimas.ac.uk/ and what we’ve got is the PIRUS plugin reporting each full-text download to the JUSP COUNTER application.
So it looks as to me as though the plugin is failing to close() the connection promptly on receipt of a close request from the JUSP COUNTER application.
I’m a dilettante when it comes to Perl and eprints code but, from glancing at the plugin code, I cannot see anything obviously amiss, so I’m guessing that the answer lies inside LWP::UserAgent or LWP::ConnCache as used by the plugin.
Does anyone else recognise this behaviour or have any suggestions on how to fix it, or can tell me I’m barking up the wrong tree?
This came to light because on a few occasions recently the NERC eprints service has become completely unresponsive with connections hanging and timing out and the logs recording many HTTP 500 errors and “Software caused connection abort at /opt/eprints3/perl_lib/EPrints/Page.pm line 78.\n”.
From the netstat output I guessed that we were hitting the limit on the number of httpd processes and was able to recover by stopping Apache, waiting until the connections cleared, then restarting it. With this done, eprints springs back into life.


Alan Cox | Infrastructure Team
NERC Research Technology Services
Polaris House, North Star Avenue, Swindon, SN2 1EU, UK
Tel: +44 (0)1793 411963 | Email: agc at nerc.ac.uk<mailto:agc at nerc.ac.uk>
NERC<http://www.nerc.ac.uk/> | Planet Earth Online<http://www.planetearth.nerc.ac.uk/> | Follow @NERCscience<https://twitter.com/NERCscience> & @NewOnNORA<https://twitter.com/NewOnNORA>

netstat output extract:

tcp        1      1             LAST_ACK    -
tcp        1      0             CLOSE_WAIT  12784/httpd
tcp        1      0             CLOSE_WAIT  12704/httpd
tcp        1      0             CLOSE_WAIT  12633/httpd
tcp        1      0             CLOSE_WAIT  11758/httpd
tcp        1      0             CLOSE_WAIT  11773/httpd
tcp        1      0             CLOSE_WAIT  12497/httpd
tcp        0      0             ESTABLISHED 12217/httpd
tcp        1      0             CLOSE_WAIT  10528/httpd
tcp        1      0             CLOSE_WAIT  12620/httpd
tcp        1      0             CLOSE_WAIT  11561/httpd
tcp        1      0             CLOSE_WAIT  11774/httpd
tcp        1      0             CLOSE_WAIT  12785/httpd
tcp        1      0             CLOSE_WAIT  12780/httpd
tcp        1      1             LAST_ACK    -
tcp        1      0             CLOSE_WAIT  12605/httpd
tcp        1      0             CLOSE_WAIT  8836/httpd
tcp        1      1             LAST_ACK    -
tcp        1      1             LAST_ACK    -
tcp        1      0             CLOSE_WAIT  12701/httpd
tcp        1      0             CLOSE_WAIT  12783/httpd
tcp        1      0             CLOSE_WAIT  12632/httpd
tcp        1      0             CLOSE_WAIT  12781/httpd
tcp        1      0             CLOSE_WAIT  12777/httpd
tcp        1      0             CLOSE_WAIT  12789/httpd
tcp        1      0             CLOSE_WAIT  12782/httpd
This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material supplied to NERC may be stored in an electronic records management system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20150313/1014bca5/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 17818 bytes
Desc: image001.png
Url : http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20150313/1014bca5/attachment-0001.png 

More information about the Eprints-tech mailing list