[EP-tech] Re: Eprints Server Freezes Often

Francis Jayakanth franc at ncsi.iisc.ernet.in
Tue May 22 13:30:36 BST 2012


On Tue, 22 May 2012, Yuri wrote:

> Il 22/05/2012 08:43, Francis Jayakanth ha scritto:
>> On Mon, 21 May 2012, Yuri wrote:
>>
>>> Lower the number of child and forked processes in apache config. The
>>> default config is usual for static pages. For example I've:
>>>
>>> <IfModule mpm_prefork_module>
>>>      StartServers          4
>>>      MinSpareServers       2
>>>      MaxSpareServers       8
>>>      MaxClients           30
>>>      MaxRequestsPerChild  1000
>>> </IfModule>
>>>
>>> It seems to work quite well. Also, I had to install mod_bw in apache to
>>> limit the max number of connection from the same ip, to calm down search
>>> engines and bots that download every pdf they find in the world... :-)
>>>
>> Thanks, Yuri, for the suggestions. I have incorporated the above changes
>> in the apache config file. Keeping a watch on the web server behaviour.
>> Will let the list know if the changes made to the apache config file is
>> helping us or not.
>>
>> Also, I have noticed that the following error message has been reported
>> quite often in the web server error log file:
>>
>> -----------------EPrints System Error ----------------------------
>> Error connecting to MySQL server: Too many connections. To fix this
>> increase max_connections in my.cnf:
>>
>> [mysqld]
>> max_connections=300
>>
>> ------------------------------------------------------------------
>>
>> Should I incorporate the above 'mysqld' directive or will the changes made
>> in the apache config take care of the MySQL error?
>
> if they happens again, yes. But I think they're related to many
> connections you've.
>
>  This are my my.cnf modifications for tuning:
>
> key_buffer     = 16M
> max_allowed_packet   = 16M
> thread_stack      = 128K
> thread_cache_size = 40
> table_cache = 1024
>
> # * Query Cache Configuration <- I don't know if this 2 are already default
> #
> query_cache_limit       = 1M
> query_cache_size        = 16M
>
>
> you can see how many queries your mysql is processing doing this:
>
> show processlist;
>
> opening mysql client. Then you can see how much loaded your mysql is.
>
>
>
Thanks, Yuri.

Its been almost 24 hours since I made the changes in the apache config 
file. What I have noticed after the changes is that the number of httpd 
processes is averaging less than 30. Before making the changes in config 
file, I have noticed that at times the number of httpd processes would be 
130+. That's when the server would freeze.

So, the changes in the config file is keeping the number of httpd process 
under control. However, just a while ago, I noticed that the eprints 
server was taking a long time to respond. I checked the number of httpd 
processes, there were 22 of them and the 'top' command produced the 
following output:

top - 17:03:33 up 5 days,  5:17,  1 user,  load average: 2.94, 3.55, 2.80
Tasks: 167 total,   2 running, 165 sleeping,   0 stopped,   0 zombie
Cpu(s): 25.6%us,  0.6%sy,  0.0%ni, 49.2%id, 24.6%wa,  0.0%hi,  0.0%si, 
0.0%st
Mem:   4051536k total,  4025724k used,    25812k free,     5304k buffers
Swap:  8193108k total,  4098224k used,  4094884k free,    66116k cached

There is clear indication that the  free memory is quite low. I restarted 
the httpd server and the output of the 'top' command right after the 
restart is:

top - 17:05:57 up 5 days,  5:19,  1 user,  load average: 12.27, 6.11, 3.78
Tasks: 160 total,   1 running, 159 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.2%us,  0.2%sy,  0.0%ni, 92.8%id,  4.8%wa,  0.0%hi,  0.0%si, 
0.0%st
Mem:   4051536k total,   335168k used,  3716368k free,     7644k buffers
Swap:  8193108k total,   263748k used,  7929360k free,    69500k cached

More than 3.5GB of main memory has been freed after the httpd restart. Is 
there anything else that needs to be done to maintain the optimal memory 
level to ensure that the eprints server response level is consistent?

Thanks for your attention, Francis



>>
>> Thanks once again.
>>
>>    - Francis
>>
>> JRDTML, IISc
>> Bengaluru, India
>>
>> eprints.iisc.ernet.in
>>
>>
>>> Il 19/05/2012 13:22, Francis Jayakanth ha scritto:
>>>> Dear Members, Our eprints server freezes (stops responding) almost every
>>>> week or so. The only solution to bring up the server again is to do a hard
>>>> boot.
>>>>
>>>> I have looked into the Web Server error log file and find that around the
>>>> time the system freezes, the following error is recorded in the error log
>>>> file:
>>>>
>>>> [Thu May 18 06:10:05 2012] [error] (12)Cannot allocate memory: fork:
>>>> Unable to fork new process
>>>>
>>>> So, the obvious reason for the system freeze is, the system is running
>>>> out of main memory. Once the server is re-booted, everything is back to
>>>> normal, but with the passage of time, the free memory keeps gradually
>>>> dropping.
>>>>
>>>> The 'top' command issued right ofter the re-boot, gives the
>>>> following output:
>>>>
>>>> top - 12:41:49 up 2 days, 55 min,  1 user,  load average: 0.90, 0.52, 0.35
>>>> Tasks: 167 total,   1 running, 166 sleeping,   0 stopped,   0 zombie
>>>> Cpu(s):  1.1%us,  0.2%sy,  0.0%ni, 97.3%id,  1.3%wa,  0.0%hi,  0.0%si,
>>>> 0.0%st
>>>> Mem:   4051536k total,  1162736k used,  2888800k free,    23300k buffers
>>>> Swap:  8193108k total,   378836k used,  7814272k free,   438404k cached
>>>>
>>>> Over the period of few hours, the the usage of the main memory gradually
>>>> goes up.
>>>>
>>>> The output of the 'top' command after 3 hours is:
>>>>
>>>> top - 15:42:07 up 2 days,  4:02,  1 user,  load average: 0.23, 0.22, 0.18
>>>> Tasks: 173 total,   1 running, 172 sleeping,   0 stopped,   0 zombie
>>>> Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
>>>> 0.0%st
>>>> Mem:   4051536k total,  3048740k used,  1002796k free,   129020k buffers
>>>> Swap:  8193108k total,   378828k used,  7814280k free,   796732k cached
>>>>
>>>> For a time gap of 3 hours, the free memory size has come down from
>>>> 2888800k to 1002796k. The drop will continue over the period of time and
>>>> eventually, the system freezes.
>>>>
>>>> I restarted the apache server after about 3 hours and the output of the
>>>> 'top' command is:
>>>>
>>>> top - 15:52:59 up 2 days,  4:07,  1 user,  load average: 0.11, 0.14, 0.16
>>>> Tasks: 168 total,   1 running, 167 sleeping,   0 stopped,   0 zombie
>>>> Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
>>>> 0.0%st
>>>> Mem:   4051536k total,  1686412k used,  2365124k free,   130712k buffers
>>>> Swap:  8193108k total,   378820k used,  7814288k free,   801620k cached
>>>>
>>>> More than 1.3 GB of memory has been freed after restarting the apache
>>>> server.
>>>>
>>>> How do I ensure that memory level doesn't gradually drop and eventually
>>>> cause the server to freeze? Should something be done at the apache
>>>> configuration level?
>>>>
>>>> Any help/pointers to overcome the said problem will of great help to us.
>>>>
>>>> We have recently upgraded our eprints software from eprints-3.2.8 to
>>>> eprints-3.3.9. The above behaviour was very much there with the 3.2.8
>>>> version as well.
>>>>
>>>> Apache Server version: Apache/2.2.3 on 64-bit RHES
>>>>
>>>> Many thanks in advance for your attention.
>>>>
>>>>
>>>>     - Francis
>>>>
>>>> JRD Tata Memorial Library
>>>> IISc, Bengaluru, India
>>>>
>>> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>>> *** Archive: http://www.eprints.org/tech.php/
>>> *** EPrints community wiki: http://wiki.eprints.org/
>>>
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>>
>
> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
> *** Archive: http://www.eprints.org/tech.php/
> *** EPrints community wiki: http://wiki.eprints.org/
>
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.



More information about the Eprints-tech mailing list