Memory leak / buffer overrun

bosskong · Post by **bosskong** » Sun Apr 09, 2006 5:41 pm

I have an Athlon 64 server running Gentoo, with 1 GB RAM and 1 GB swap memory and I'm running modect on 3 cameras. Under most conditions, the load is below 1%. But every once in a while, the server will stop responding or will respond slowly. When I check /var/log/messages, I see tons of buffer overrun messages. If I run "top" or "ps -aux", I see that most of my memory is used, but I can't figure out what is using it. It seems like a memory leak, but I'm not sure that zoneminder is causing the leak. Today, I tried to access ZM and found that the mysql database wasn't running and all my memory was used. When I have these problems, a reboot is usually all I need to do to fix it.

Does anyone have any tips for how to figure out what is causing the problem?

Thanks

Brad

keyboardgnome · Post by **keyboardgnome** » Mon Apr 10, 2006 4:26 am

lsof- see what has open sockets running, and other related hooks;

you should also know that linux uses 'shared' resources (virtual threads). read this: http://techpatterns.com/forums/about687.html and http://virtualthreads.blogspot.com/2006 ... linux.html

I take it none of your swap is being eaten; check your HD performance anyways with hdparm; if it is slower than it is suppose to be then modify the system settings to tweak it.

your other options are to turn off processes and slowly turn them back on until you see the problem (change control).

HTH

Ben · Post by **Ben** » Mon Apr 10, 2006 11:09 am

Hi,

I have the same problem. I use MAndriva 2006. I already post my problem... The problem is ZM... I use 2 camera's...

Post by **zoneminder** » Mon Apr 10, 2006 5:03 pm

In case you were aware you can sort the output of top by memory usage, though it can be a little non-intuitive what it displays, for instance static and shared memory is included in each process that is accessing it so it can look like you are using a lot more than you really are.

Ben · Post by **Ben** » Tue Apr 11, 2006 9:00 am

Thanks Phil,

When I have more time to solve this problem I will send a screenshot of my "Top".

johabba · Post by **johabba** » Fri Jun 09, 2006 5:19 pm

I'm having a similar problem on ZM v1.22.1. After running for a few (8-10) hours all ram (and most of swap) is gone. Apache and zmc and zma are using the most RAM according to top. Once zmc and zma where using over 200MB (according to top's SHR column)!

This is a new install so I imagine the customer is happily clicking everything they can get their grubby little mouse pointer on. ZM is watching a fairly high traffic fruit smoothie shop in Yuppieville, USA.

Intel(R) Celeron(R) CPU 2.66GHz
512 PC3200 DDR Ram
1-chip 4-output dvr card
ZM in modetect mode
All monitors at 320x240
Debian Sarge

Has this specific problem been addressed in the new version? I don't really want to upgrade ZM on this system because the customer is a pain in the ass...unless this specific problem as been resolved in a new version.

Is there anything else i can try to keep all the memory from being eaten?

tzr · Post by **tzr** » Fri Jun 09, 2006 7:31 pm

Ben wrote:Hi,

I have the same problem. I use MAndriva 2006. I already post my problem... The problem is ZM... I use 2 camera's...

Mandriva 2006, 512MB ram, 1GB swap

if I use KDE, top says that memory gets eaten up, both RAM and swap.
But if I don't use the GUI but only the console (init level 3),
no problem at all

give it a try

eddyhkim · Post by **eddyhkim** » Fri Jun 09, 2006 10:04 pm

I had similar symptoms on my 1.8Ghz celeron with 1 GB of ram. After running about 12 hours or so, the system was so slow and unresponsive a login would take about 30 seconds or so to complete, and the httpd server was extremely slow to respond. I was running a basic system, with 4 camera inputs.

1st: I upped the shared memory. I checked top and the graphical system analyser and memory wasn't being fully used, I still had about 1000K free, and virtually no swap was being used. I upped it to 256MB of shared memory in sysctl.conf even though regular ram didn't seem to be getting hit that hard.

2nd: I throttled the cameras in the configuration down to 5 or 10 fps depending on the camera.

The change was done recently so I don't know if it's a long term solution but it hasn't happened since.

RedFlyer · Post by **RedFlyer** » Sun Jul 09, 2006 1:33 pm

Hi everybody,

I'm getting the same problem.

My config is :
MB Advantech SBC PIII 800MHz
SDRAM 133 512MB
BT878 single chip capture board - 4 channels
Monitor 1 - CAM in MODECT 5fps at 352x288 RGB24
Monitor 2 - CAM in MONITOR 5fps at 352x288 RGB24
The distro is the CTU core 1.21.3 from Cordel upgraded to ZM 1.22.2
This distro is headless (no X server, only text console)

I get the problem after 1 or 2 days of working.

I'll post 'top' display before and after the problem later on.

If somebody have found the solution ...

Oliver
________
WIKI VAPORIZER

jameswilson · Post by **jameswilson** » Sun Jul 09, 2006 5:06 pm

yes i dpost your top as all linux machines i have build recently report all their ram is 'gone' but most is used for cache! and then comes from cache when needed

RedFlyer · Post by **RedFlyer** » Sun Jul 09, 2006 10:23 pm

Hi James,

You are right, the increasing memory usage is mostly used by cache.
I've analysed the information given by TOP and the memory used by ZMC and ZMA process is the same, but the cache is increasing.

Here a capture from this afternoon

Code: Select all

top - 15:22:12 up  2:35,  1 user,  load average: 0.31, 0.25, 0.19
Tasks:  62 total,   2 running,  60 sleeping,   0 stopped,   0 zombie
Cpu(s): 12.3% us,  1.0% sy,  0.0% ni, 86.7% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:    483264k total,   188292k used,   294972k free,    20764k buffers
Swap:   983032k total,        0k used,   983032k free,    71500k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4386 apache    15   0  160m  45m  41m S  6.6  9.6   8:08.27 zmc
 4435 apache    15   0  132m  16m  13m S  6.0  3.5   7:10.91 zma
 3415 apache    16   0 18480  11m 7104 S  0.0  2.5   0:05.00 httpd
 3414 apache    16   0 18472  11m 7072 S  0.0  2.5   0:04.46 httpd

and here a capture just now

Code: Select all

top - 00:19:14 up 11:32,  2 users,  load average: 0.21, 0.22, 0.28
Tasks:  66 total,   1 running,  65 sleeping,   0 stopped,   0 zombie
Cpu(s): 11.9% us,  1.0% sy,  0.0% ni, 81.5% id,  5.6% wa,  0.0% hi,  0.0% si
Mem:    483264k total,   244588k used,   238676k free,    32652k buffers
Swap:   983032k total,        0k used,   983032k free,   110212k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4386 apache    15   0  160m  45m  41m S  6.9  9.6  43:43.12 zmc
 4435 apache    15   0  132m  16m  13m S  5.9  3.6  39:15.74 zma
 5361 apache    15   0 18624  11m 7060 S  0.0  2.5   0:02.42 httpd
 3414 apache    15   0 18472  11m 7092 S  0.0  2.5   0:09.24 httpd

Between them they are four more processes, but this is just sshd and shell.

What is causing the cached memory increasing ? and buffers also ...
How fix this ?

Best regards,

Oliver
________
XW FALCON

RedFlyer · Post by **RedFlyer** » Sun Jul 09, 2006 10:38 pm

While browsing linux forum, i saw that if the system need more memory the cache usage will be freed by the system before using swap.
But here, this is not the case as the swap is used without cache 'cleaning'

Any idea ?

regards,

Oliver
________
New Jersey Marijuana Dispensary

jameswilson · Post by **jameswilson** » Mon Jul 10, 2006 7:12 am

I may be wrong but from what i understand;-

You will see all your 'free' memory disappear. Although top reports there is no actaully free memory, you will find 10's of megs are 'cached'. From what i understand this cache is basically a disk cache and even if you had 16gig of ram it would still be used for this. I beleive you can tell linux not to, but there really isnt any need as it does this for speed reasons. If your machine suddenly needs another 15 meg of ram for a process, memory will be moved from 'cache' to the process requireing it. All this is standard for linux. The buffers i cant explain though but im sure if you read about memory management on one of the linux pages it would explain this too. I read on hera bout people running init 3 and not using all there ram. This suprises me as all my linux machines (centos, mandrake, fc3 or pclos) always use all their ram. I have machines doing different jobs ranging from 256 meg to 4 gig and on the 4 gig machine i have a 2.8 gig cache!!

James

Flash_ · Post by **Flash_** » Mon Jul 10, 2006 8:06 am

I may also be wrong, but I was under a similar opinion - that Linux will use all available ram for cache if it's not otherwise being used. The OS is smart enough to clear that cache if another program asks for more memory.

the reasoning, as I understand it, is that memory is there to be used. If using it for cache might help things run smoother, it will.

Both my debian servers run at very high memory usage continually and haven't caused any problem so I simply stopped worrying about the red bars in phpsysinfo and htop; system only stops being responsive when there is heavy cpu or disk load, the latter on ide system seems to have more of an impact to quickness of response.

A quick google brought up: http://www.novell.com/coolsolutions/qna/2327.html

In general Linux uses as much of ram as it can sensibly get and use. So, at first glance it appears that Windows XP would use memory more efficiently, because more is reported to be free.

The truth is that by using all of your memory right from the start and shuffling this around as needed, Linux is much more efficent while Windows does not make use of all the free ram.

Linux just uses a fundamentally different, but more efficient memory management system.