[EP-tech] Re: Eprints Server Freezes Often
Yuri
yurj at alfa.it
Tue May 22 13:16:10 BST 2012
I think you can't do much other than:
a - restart via cronjob the apache and mysql process, every night - this
help
b - find why you've so much connections and from where
c - limit who is abusing from the service
for b, you can use the logs.
for c, you can use mod_bw from Apache. In debian just do this:
apt-get install libapache2-mod-bw
and add this to
/usr/share/eprints3/archives/<your-archive>/var/auto-apache.conf
(remember to do this when you re-run regeneration of apacheconf with
epadmin tools)
# turn bw limitation on
BandwidthModule On
# force limitation on every request
ForceBandWidthModule On
BandWidth all 0
MaxConnection all 20
inside the before </VirtualHost>
in this way abusers can connect only 20 times and not infinite. Usually
they throttle or stop to abuse you :)
Try also the mysql tuning, they help expecially the thread_cache_size
and table_cache.
Il 22/05/2012 14:30, Francis Jayakanth ha scritto:
> On Tue, 22 May 2012, Yuri wrote:
>
>> Il 22/05/2012 08:43, Francis Jayakanth ha scritto:
>>> On Mon, 21 May 2012, Yuri wrote:
>>>
>>>> Lower the number of child and forked processes in apache config. The
>>>> default config is usual for static pages. For example I've:
>>>>
>>>> <IfModule mpm_prefork_module>
>>>> StartServers 4
>>>> MinSpareServers 2
>>>> MaxSpareServers 8
>>>> MaxClients 30
>>>> MaxRequestsPerChild 1000
>>>> </IfModule>
>>>>
>>>> It seems to work quite well. Also, I had to install mod_bw in apache to
>>>> limit the max number of connection from the same ip, to calm down search
>>>> engines and bots that download every pdf they find in the world... :-)
>>>>
>>> Thanks, Yuri, for the suggestions. I have incorporated the above changes
>>> in the apache config file. Keeping a watch on the web server behaviour.
>>> Will let the list know if the changes made to the apache config file is
>>> helping us or not.
>>>
>>> Also, I have noticed that the following error message has been reported
>>> quite often in the web server error log file:
>>>
>>> -----------------EPrints System Error ----------------------------
>>> Error connecting to MySQL server: Too many connections. To fix this
>>> increase max_connections in my.cnf:
>>>
>>> [mysqld]
>>> max_connections=300
>>>
>>> ------------------------------------------------------------------
>>>
>>> Should I incorporate the above 'mysqld' directive or will the changes made
>>> in the apache config take care of the MySQL error?
>> if they happens again, yes. But I think they're related to many
>> connections you've.
>>
>> This are my my.cnf modifications for tuning:
>>
>> key_buffer = 16M
>> max_allowed_packet = 16M
>> thread_stack = 128K
>> thread_cache_size = 40
>> table_cache = 1024
>>
>> # * Query Cache Configuration<- I don't know if this 2 are already default
>> #
>> query_cache_limit = 1M
>> query_cache_size = 16M
>>
>>
>> you can see how many queries your mysql is processing doing this:
>>
>> show processlist;
>>
>> opening mysql client. Then you can see how much loaded your mysql is.
>>
>>
>>
> Thanks, Yuri.
>
> Its been almost 24 hours since I made the changes in the apache config
> file. What I have noticed after the changes is that the number of httpd
> processes is averaging less than 30. Before making the changes in config
> file, I have noticed that at times the number of httpd processes would be
> 130+. That's when the server would freeze.
>
> So, the changes in the config file is keeping the number of httpd process
> under control. However, just a while ago, I noticed that the eprints
> server was taking a long time to respond. I checked the number of httpd
> processes, there were 22 of them and the 'top' command produced the
> following output:
>
> top - 17:03:33 up 5 days, 5:17, 1 user, load average: 2.94, 3.55, 2.80
> Tasks: 167 total, 2 running, 165 sleeping, 0 stopped, 0 zombie
> Cpu(s): 25.6%us, 0.6%sy, 0.0%ni, 49.2%id, 24.6%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 4051536k total, 4025724k used, 25812k free, 5304k buffers
> Swap: 8193108k total, 4098224k used, 4094884k free, 66116k cached
>
> There is clear indication that the free memory is quite low. I restarted
> the httpd server and the output of the 'top' command right after the
> restart is:
>
> top - 17:05:57 up 5 days, 5:19, 1 user, load average: 12.27, 6.11, 3.78
> Tasks: 160 total, 1 running, 159 sleeping, 0 stopped, 0 zombie
> Cpu(s): 2.2%us, 0.2%sy, 0.0%ni, 92.8%id, 4.8%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 4051536k total, 335168k used, 3716368k free, 7644k buffers
> Swap: 8193108k total, 263748k used, 7929360k free, 69500k cached
>
> More than 3.5GB of main memory has been freed after the httpd restart. Is
> there anything else that needs to be done to maintain the optimal memory
> level to ensure that the eprints server response level is consistent?
>
> Thanks for your attention, Francis
>
>
>
>>> Thanks once again.
>>>
>>> - Francis
>>>
>>> JRDTML, IISc
>>> Bengaluru, India
>>>
>>> eprints.iisc.ernet.in
>>>
>>>
>>>> Il 19/05/2012 13:22, Francis Jayakanth ha scritto:
>>>>> Dear Members, Our eprints server freezes (stops responding) almost every
>>>>> week or so. The only solution to bring up the server again is to do a hard
>>>>> boot.
>>>>>
>>>>> I have looked into the Web Server error log file and find that around the
>>>>> time the system freezes, the following error is recorded in the error log
>>>>> file:
>>>>>
>>>>> [Thu May 18 06:10:05 2012] [error] (12)Cannot allocate memory: fork:
>>>>> Unable to fork new process
>>>>>
>>>>> So, the obvious reason for the system freeze is, the system is running
>>>>> out of main memory. Once the server is re-booted, everything is back to
>>>>> normal, but with the passage of time, the free memory keeps gradually
>>>>> dropping.
>>>>>
>>>>> The 'top' command issued right ofter the re-boot, gives the
>>>>> following output:
>>>>>
>>>>> top - 12:41:49 up 2 days, 55 min, 1 user, load average: 0.90, 0.52, 0.35
>>>>> Tasks: 167 total, 1 running, 166 sleeping, 0 stopped, 0 zombie
>>>>> Cpu(s): 1.1%us, 0.2%sy, 0.0%ni, 97.3%id, 1.3%wa, 0.0%hi, 0.0%si,
>>>>> 0.0%st
>>>>> Mem: 4051536k total, 1162736k used, 2888800k free, 23300k buffers
>>>>> Swap: 8193108k total, 378836k used, 7814272k free, 438404k cached
>>>>>
>>>>> Over the period of few hours, the the usage of the main memory gradually
>>>>> goes up.
>>>>>
>>>>> The output of the 'top' command after 3 hours is:
>>>>>
>>>>> top - 15:42:07 up 2 days, 4:02, 1 user, load average: 0.23, 0.22, 0.18
>>>>> Tasks: 173 total, 1 running, 172 sleeping, 0 stopped, 0 zombie
>>>>> Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.6%id, 0.4%wa, 0.0%hi, 0.0%si,
>>>>> 0.0%st
>>>>> Mem: 4051536k total, 3048740k used, 1002796k free, 129020k buffers
>>>>> Swap: 8193108k total, 378828k used, 7814280k free, 796732k cached
>>>>>
>>>>> For a time gap of 3 hours, the free memory size has come down from
>>>>> 2888800k to 1002796k. The drop will continue over the period of time and
>>>>> eventually, the system freezes.
>>>>>
>>>>> I restarted the apache server after about 3 hours and the output of the
>>>>> 'top' command is:
>>>>>
>>>>> top - 15:52:59 up 2 days, 4:07, 1 user, load average: 0.11, 0.14, 0.16
>>>>> Tasks: 168 total, 1 running, 167 sleeping, 0 stopped, 0 zombie
>>>>> Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.6%id, 0.4%wa, 0.0%hi, 0.0%si,
>>>>> 0.0%st
>>>>> Mem: 4051536k total, 1686412k used, 2365124k free, 130712k buffers
>>>>> Swap: 8193108k total, 378820k used, 7814288k free, 801620k cached
>>>>>
>>>>> More than 1.3 GB of memory has been freed after restarting the apache
>>>>> server.
>>>>>
>>>>> How do I ensure that memory level doesn't gradually drop and eventually
>>>>> cause the server to freeze? Should something be done at the apache
>>>>> configuration level?
>>>>>
>>>>> Any help/pointers to overcome the said problem will of great help to us.
>>>>>
>>>>> We have recently upgraded our eprints software from eprints-3.2.8 to
>>>>> eprints-3.3.9. The above behaviour was very much there with the 3.2.8
>>>>> version as well.
>>>>>
>>>>> Apache Server version: Apache/2.2.3 on 64-bit RHES
>>>>>
>>>>> Many thanks in advance for your attention.
>>>>>
>>>>>
>>>>> - Francis
>>>>>
>>>>> JRD Tata Memorial Library
>>>>> IISc, Bengaluru, India
>>>>>
>>>> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>>>> *** Archive: http://www.eprints.org/tech.php/
>>>> *** EPrints community wiki: http://wiki.eprints.org/
>>>>
>>>> --
>>>> This message has been scanned for viruses and
>>>> dangerous content by MailScanner, and is
>>>> believed to be clean.
>>>>
>> *** Options: http://mailman.ecs.soton.ac.uk/mailman/listinfo/eprints-tech
>> *** Archive: http://www.eprints.org/tech.php/
>> *** EPrints community wiki: http://wiki.eprints.org/
>>
>> --
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>
More information about the Eprints-tech
mailing list