<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Arial","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">I’d appreciate hearing whether anyone using the PIRUS plugin has seen anything similar to the following,
or from anyone else with wisdom to offer …<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">I have noticed that netstat on our eprints service shows many connections to
</span><span style="font-size:10.0pt;font-family:"Lucida Console"">54.72.175.35:80</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D"> in CLOSE_WAIT state, originating from instances of the httpd process (see below) and that
these connections take a long time to get to LAST_ACK and then stay in that state for a long time. If I understand what I read correctly, CLOSE_WAIT occurs when the remote end has requested that the TCP connection be closed (sent a FIN packet), to which the
local protocol stack has responded with an ACK, and is then waiting for the local process with the connection open to close it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">54.72.175.35</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D"> is an Amazon AWS VM that appears to host a number of services but it appears most
likely that the service of relevance here is <a href="http://www.jusp.mimas.ac.uk/">http://www.jusp.mimas.ac.uk/</a> and what we’ve got is the PIRUS plugin reporting each full-text download to the JUSP COUNTER application.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">So it looks as to me as though the plugin is failing to close() the connection promptly on receipt
of a close request from the JUSP COUNTER application.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">I’m a dilettante when it comes to Perl and eprints code but, from glancing at the plugin code, I cannot
see anything obviously amiss, so I’m guessing that the answer lies inside LWP::UserAgent or LWP::ConnCache as used by the plugin.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">Does anyone else recognise this behaviour or have any suggestions on how to fix it, or can tell me
I’m barking up the wrong tree?<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">This came to light because on a few occasions recently the NERC eprints service has become completely
unresponsive with connections hanging and timing out and the logs recording many HTTP 500 errors and “Software caused connection abort at /opt/eprints3/perl_lib/EPrints/Page.pm line 78.\n”.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">From the netstat output I guessed that we were hitting the limit on the number of httpd processes and
was able to recover by stopping Apache, waiting until the connections cleared, then restarting it. With this done, eprints springs back into life.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D"> Alan.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" style="border-collapse:collapse">
<tbody>
<tr style="height:56.1pt">
<td width="129" valign="top" style="width:97.05pt;padding:0cm 5.4pt 0cm 5.4pt;height:56.1pt">
<p class="MsoNormal" align="right" style="text-align:right;mso-line-height-alt:1.0pt">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><img border="0" width="121" height="81" id="Picture_x0020_1" src="cid:image001.png@01D05DB3.1B78B8D0" alt="50%20email%20logo"></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
</td>
<td width="392" valign="top" style="width:294.25pt;padding:0cm 5.4pt 0cm 5.4pt;height:56.1pt">
<p class="MsoNormal" align="right" style="text-align:right;mso-line-height-alt:1.0pt;background:white">
<span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">Alan Cox | Infrastructure Team<br>
NERC Research Technology Services<br>
<i>Polaris House, North Star Avenue, Swindon, SN2 1EU, UK<br>
</i><b>Tel: </b>+44 (0)1793 411963 | <b>Email:</b> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="mailto:agc@nerc.ac.uk"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">agc@nerc.ac.uk</span></a><br>
<a href="http://www.nerc.ac.uk/" target="_blank"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">NERC</span></a>
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">|
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="http://www.planetearth.nerc.ac.uk/" target="_blank"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">Planet Earth Online</span></a>
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">| Follow
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://twitter.com/NERCscience" target="_blank"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">@NERCscience</span></a></span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">
& </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://twitter.com/NewOnNORA"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">@NewOnNORA</span></a></span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D">netstat output extract:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 1 139.166.209.11:45805 54.72.175.35:80 LAST_ACK - </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45810 54.72.175.35:80 CLOSE_WAIT 12784/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45822 54.72.175.35:80 CLOSE_WAIT 12704/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45820 54.72.175.35:80 CLOSE_WAIT 12633/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45768 54.72.175.35:80 CLOSE_WAIT 11758/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45785 54.72.175.35:80 CLOSE_WAIT 11773/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45856 54.72.175.35:80 CLOSE_WAIT 12497/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 0 0 139.166.209.11:45870 54.72.175.35:80 ESTABLISHED 12217/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45866 54.72.175.35:80 CLOSE_WAIT 10528/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45864 54.72.175.35:80 CLOSE_WAIT 12620/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45865 54.72.175.35:80 CLOSE_WAIT 11561/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45835 54.72.175.35:80 CLOSE_WAIT 11774/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45832 54.72.175.35:80 CLOSE_WAIT 12785/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45833 54.72.175.35:80 CLOSE_WAIT 12780/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 1 139.166.209.11:45846 54.72.175.35:80 LAST_ACK - </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45844 54.72.175.35:80 CLOSE_WAIT 12605/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45842 54.72.175.35:80 CLOSE_WAIT 8836/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 1 139.166.209.11:45843 54.72.175.35:80 LAST_ACK - </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 1 139.166.209.11:45840 54.72.175.35:80 LAST_ACK - </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45841 54.72.175.35:80 CLOSE_WAIT 12701/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45854 54.72.175.35:80 CLOSE_WAIT 12783/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45852 54.72.175.35:80 CLOSE_WAIT 12632/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45853 54.72.175.35:80 CLOSE_WAIT 12781/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45850 54.72.175.35:80 CLOSE_WAIT 12777/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45848 54.72.175.35:80 CLOSE_WAIT 12789/httpd </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lucida Console"">tcp 1 0 139.166.209.11:45849 54.72.175.35:80 CLOSE_WAIT 12782/httpd </span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D"><o:p></o:p></span></p>
</div>
<hr>
<small>This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material
supplied to NERC may be stored in an electronic records management system</small>.
<hr>
</body>
</html>