[EP-tech] Re: Eprints Server Freezes Often

Yuri yurj at alfa.it
Tue May 22 13:16:10 BST 2012


I think you can't do much other than:

a - restart via cronjob the apache and mysql process, every night - this 
help
b - find why you've so much connections and from where
c - limit who is abusing from the service

for b, you can use the logs.

for c, you can use mod_bw from Apache. In debian just do this:

apt-get install libapache2-mod-bw

and add this to 
/usr/share/eprints3/archives/<your-archive>/var/auto-apache.conf 
(remember to do this when you re-run regeneration of apacheconf with 
epadmin tools)

         # turn bw limitation on
         BandwidthModule On

         # force limitation on every request
         ForceBandWidthModule On
               BandWidth all 0
               MaxConnection all 20


inside the before </VirtualHost>

in this way abusers can connect only 20 times and not infinite. Usually 
they throttle or stop to abuse you :)

Try also the mysql tuning, they help expecially the thread_cache_size 
and table_cache.


Il 22/05/2012 14:30, Francis Jayakanth ha scritto:
> On Tue, 22 May 2012, Yuri wrote:
>
>> Il 22/05/2012 08:43, Francis Jayakanth ha scritto:
>>> On Mon, 21 May 2012, Yuri wrote:
>>>
>>>> Lower the number of child and forked processes in apache config. The
>>>> default config is usual for static pages. For example I've:
>>>>
>>>> <IfModule mpm_prefork_module>
>>>>       StartServers          4
>>>>       MinSpareServers       2
>>>>       MaxSpareServers       8
>>>>       MaxClients           30
>>>>       MaxRequestsPerChild  1000
>>>> </IfModule>
>>>>
>>>> It seems to work quite well. Also, I had to install mod_bw in apache to
>>>> limit the max number of connection from the same ip, to calm down search
>>>> engines and bots that download every pdf they find in the world... :-)
>>>>
>>> Thanks, Yuri, for the suggestions. I have incorporated the above changes
>>> in the apache config file. Keeping a watch on the web server behaviour.
>>> Will let the list know if the changes made to the apache config file is
>>> helping us or not.
>>>
>>> Also, I have noticed that the following error message has been reported
>>> quite often in the web server error log file:
>>>
>>> -----------------EPrints System Error ----------------------------
>>> Error connecting to MySQL server: Too many connections. To fix this
>>> increase max_connections in my.cnf:
>>>
>>> [mysqld]
>>> max_connections=300
>>>
>>> ------------------------------------------------------------------
>>>
>>> Should I incorporate the above 'mysqld' directive or will the changes made
>>> in the apache config take care of the MySQL error?
>> if they happens again, yes. But I think they're related to many
>> connections you've.
>>
>>   This are my my.cnf modifications for tuning:
>>
>> key_buffer     = 16M
>> max_allowed_packet   = 16M
>> thread_stack      = 128K
>> thread_cache_size = 40
>> table_cache = 1024
>>
>> # * Query Cache Configuration<- I don't know if this 2 are already default
>> #
>> query_cache_limit       = 1M
>> query_cache_size        = 16M
>>
>>
>> you can see how many queries your mysql is processing doing this:
>>
>> show processlist;
>>
>> opening mysql client. Then you can see how much loaded your mysql is.
>>
>>
>>
> Thanks, Yuri.
>
> Its been almost 24 hours since I made the changes in the apache config
> file. What I have noticed after the changes is that the number of httpd
> processes is averaging less than 30. Before making the changes in config
> file, I have noticed that at times the number of httpd processes would be
> 130+. That's when the server would freeze.
>
> So, the changes in the config file is keeping the number of httpd process
> under control. However, just a while ago, I noticed that the eprints
> server was taking a long time to respond. I checked the number of httpd
> processes, there were 22 of them and the 'top' command produced the
> following output:
>
> top - 17:03:33 up 5 days,  5:17,  1 user,  load average: 2.94, 3.55, 2.80
> Tasks: 167 total,   2 running, 165 sleeping,   0 stopped,   0 zombie
> Cpu(s): 25.6%us,  0.6%sy,  0.0%ni, 49.2%id, 24.6%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Mem:   4051536k total,  4025724k used,    25812k free,     5304k buffers
> Swap:  8193108k total,  4098224k used,  4094884k free,    66116k cached
>
> There is clear indication that the  free memory is quite low. I restarted
> the httpd server and the output of the 'top' command right after the
> restart is:
>
> top - 17:05:57 up 5 days,  5:19,  1 user,  load average: 12.27, 6.11, 3.78
> Tasks: 160 total,   1 running, 159 sleeping,   0 stopped,   0 zombie
> Cpu(s):  2.2%us,  0.2%sy,  0.0%ni, 92.8%id,  4.8%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Mem:   4051536k total,   335168k used,  3716368k free,     7644k buffers
> Swap:  8193108k total,   263748k used,  7929360k free,    69500k cached
>
> More than 3.5GB of main memory has been freed after the httpd restart. Is
> there anything else that needs to be done to maintain the optimal memory
> level to ensure that the eprints server response level is consistent?
>
> Thanks for your attention, Francis
>
>
>
>>> Thanks once again.
>>>
>>>     - Francis
>>>
>>> JRDTML, IISc
>>> Bengaluru, India
>>>
>>> eprints.iisc.ernet.in
>>>
>>>
>>>> Il 19/05/2012 13:22, Francis Jayakanth ha scritto:
>>>>> Dear Members, Our eprints server freezes (stops responding) almost every
>>>>> week or so. The only solution to bring up the server again is to do a hard
>>>>> boot.
>>>>>
>>>>> I have looked into the Web Server error log file and find that around the
>>>>> time the system freezes, the following error is recorded in the error log
>>>>> file:
>>>>>
>>>>> [Thu May 18 06:10:05 2012] [error] (12)Cannot allocate memory: fork:
>>>>> Unable to fork new process
>>>>>
>>>>> So, the obvious reason for the system freeze is, the system is running
>>>>> out of main memory. Once the server is re-booted, everything is back to
>>>>> normal, but with the passage of time, the free memory keeps gradually
>>>>> dropping.
>>>>>
>>>>> The 'top' command issued right ofter the re-boot, gives the
>>>>> following output:
>>>>>
>>>>> top - 12:41:49 up 2 days, 55 min,  1 user,  load average: 0.90, 0.52, 0.35
>>>>> Tasks: 167 total,   1 running, 166 sleeping,   0 stopped,   0 zombie
>>>>> Cpu(s):  1.1%us,  0.2%sy,  0.0%ni, 97.3%id,  1.3%wa,  0.0%hi,  0.0%si,
>>>>> 0.0%st
>>>>> Mem:   4051536k total,  1162736k used,  2888800k free,    23300k buffers
>>>>> Swap:  8193108k total,   378836k used,  7814272k free,   438404k cached
>>>>>
>>>>> Over the period of few hours, the the usage of the main memory gradually
>>>>> goes up.
>>>>>
>>>>> The output of the 'top' command after 3 hours is:
>>>>>
>>>>> top - 15:42:07 up 2 days,  4:02,  1 user,  load average: 0.23, 0.22, 0.18
>>>>> Tasks: 173 total,   1 running, 172 sleeping,   0 stopped,   0 zombie
>>>>> Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
>>>>> 0.0%st
>>>>> Mem:   4051536k total,  3048740k used,  1002796k free,   129020k buffers
>>>>> Swap:  8193108k total,   378828k used,  7814280k free,   796732k cached
>>>>>
>>>>> For a time gap of 3 hours, the free memory size has come down from
>>>>> 2888800k to 1002796k. The drop will continue over the period of time and
>>>>> eventually, the system freezes.
>>>>>
>>>>> I restarted the apache server after about 3 hours and the output of the
>>>>> 'top' command is:
>>>>>
>>>>> top - 15:52:59 up 2 days,  4:07,  1 user,  load average: 0.11, 0.14, 0.16
>>>>> Tasks: 168 total,   1 running, 167 sleeping,   0 stopped,   0 zombie
>>>>> Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
>>>>> 0.0%st
>>>>> Mem:   4051536k total,  1686412k used,  2365124k free,   130712k buffers
>>>>> Swap:  8193108k total,   378820k used,  7814288k free,   801620k cached
>>>>>
>>>>> More than 1.3 GB of memory has been freed after restarting the apache
>>>>> server.
>>>>>
>>>>> How do I ensure that memory level doesn't gradually drop and eventually
>>>>> cause the server to freeze? Should something be done at the apache
>>>>> configuration level?
>>>>>
>>>>> Any help/pointers to overcome the said problem will of great help to us.
>>>>>
>>>>> We have recently upgraded our eprints software from eprints-3.2.8 to
>>>>> eprints-3.3.9. The above behaviour was very much there with the 3.2.8
>>>>> version as well.
>>>>>
>>>>> Apache Server version: Apache/2.2.3 on 64-bit RHES
>>>>>
>>>>> Many thanks in advance for your attention.
>>>>>
>>>>>
>>>>>      - Francis
>>>>>
>>>>> JRD Tata Memorial Library
>>>>> IISc, Bengaluru, India
>>>>>
>>>> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>>>> *** Archive: http://www.eprints.org/tech.php/
>>>> *** EPrints community wiki: http://wiki.eprints.org/
>>>>
>>>> --
>>>> This message has been scanned for viruses and
>>>> dangerous content by MailScanner, and is
>>>> believed to be clean.
>>>>
>> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>> *** Archive: http://www.eprints.org/tech.php/
>> *** EPrints community wiki: http://wiki.eprints.org/
>>
>> -- 
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>



More information about the Eprints-tech mailing list