High CPU when camera is disconnected

Forum for questions and support relating to the 1.30.x releases only.
Locked
DoubleHP
Posts: 18
Joined: Fri Sep 02, 2016 1:13 pm

High CPU when camera is disconnected

Post by DoubleHP »

Hello; got 6 cams on a weak server. reduced the frame rate to 3 inside each camera. My CPU usage is about 1/3 by night when activity is low around, and about half during day. I have found settings that work for me, most of the time.

3FPS is low rate, but, considering other factors ... does the job for me.

But if for any reason a camera is unreachable (cable disconnected, dead supply, maintainance), then the CPU load goes 70-80% for a single missing cam; and 100% for 3. In short, disconnect or break half of my cams, and the remaining ones also stop working due to system overload.

I have found that missing camera also produce a HUGE AMOUNT OF LOGS. Like hundreds MB per day.

How to reduce CPU usage, system load, and log lines when cam goes away ?

One possible scenario: I go on holidays for 1 week, first one cam goes off (dead psu), when I come back server is completely down due to disk full.
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: High CPU when camera is disconnected

Post by rockedge »

I am running into a problem I first noticed working with the master development version 1.30.1 where the /dev/shm creeps upwards until it hits 100% and cameras begin to shut down and the log indicates

Code: Select all

disk_free_space returned false for /dev/shm
I wonder if these are related.
the first camera to shutdown is prone to disconnect then reconnect which seems to be WiFi related.
DoubleHP
Posts: 18
Joined: Fri Sep 02, 2016 1:13 pm

Re: High CPU when camera is disconnected

Post by DoubleHP »

I don't have SHM issues; I monitor everything via Munin, and SHM use does not change significantly during these issues.

Which log file tells you that ?

# grep -nr "disk_free_space" /var/log/zm/
#

... nothing.
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: High CPU when camera is disconnected

Post by rockedge »

that message came directly from the log setup as installed. Log was red and I clicked on it and that was in the log displayed in the window. First time I ever saw this message or error since version 1.26.5. In the options logging is set to write to the database.
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: High CPU when camera is disconnected

Post by rockedge »

turns out another log file, not related to ZM, had kept growing and loaded into RAM and eventually took up the available memory. I deleted the log and have not had the problem since. I will manage this log better and the problem looks solved.
DoubleHP
Posts: 18
Joined: Fri Sep 02, 2016 1:13 pm

Re: High CPU when camera is disconnected

Post by DoubleHP »

Use Munin, and plugins loggrep and df, to get warnings about real causes weeks before this kind of trouble occurs.
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: High CPU when camera is disconnected

Post by knight-of-ni »

Workaround to high cpu usage when a camera drops offline is to put a frame rate slightly higher than the incoming stream into the Max FPS field.
For IP streams never place a value that is lower than the incoming frame rate in this field. This is described in the help text.

So in your case, put a 4 in the max fps field.

You can't reduce the amount of log entries going into the logs. The solution to this is to design a system where the cameras do no drop offline.
If however, you cameras go offline during a known time period, then you could make use of run states to disable the camera in zoneminder during those times.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: High CPU when camera is disconnected

Post by knight-of-ni »

rockedge wrote:I am running into a problem I first noticed working with the master development version 1.30.1 where the /dev/shm creeps upwards until it hits 100% and cameras begin to shut down and the log indicates

Code: Select all

disk_free_space returned false for /dev/shm
I wonder if these are related.
the first camera to shutdown is prone to disconnect then reconnect which seems to be WiFi related.
This is happening when a camera is restarted w/o restarting zoneminder. One or more of the perl daemons are keeping open file handles on the mmap files when they shouldn't. It'll be fixed by the next release.

You can see this happening with a "sudo lsof /dev/shm"
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: High CPU when camera is disconnected

Post by rockedge »

knnniggett wrote:This is happening when a camera is restarted w/o restarting zoneminder. One or more of the perl daemons are keeping open file handles on the mmap files when they shouldn't. It'll be fixed by the next release.

You can see this happening with a "sudo lsof /dev/shm"
Thank you! That does seem to be exactly whats happening. I'll live with it until the next release.
DoubleHP
Posts: 18
Joined: Fri Sep 02, 2016 1:13 pm

Re: High CPU when camera is disconnected

Post by DoubleHP »

YES, Maximum FPS had an impact on CPU usage. Setting this value reduced CPU usage even below the usage ratio I normally have when cams are working.

But logs generated by ZMA were almost the same. So, that variable affected only the CPU usage; not the log flood. About two lines per second, for the whole duration of issue, when disconnecting 3 cams. Or, one line per second per cam (depending how I read my stats). If I got out for one week, that can be a huge amount of disk space for a small server like a rPi.
StephenOz
Posts: 6
Joined: Sat Dec 30, 2006 12:27 am

Re: High CPU when camera is disconnected

Post by StephenOz »

I had an issue were ZoneMinder was jumping to 100% CPU usage if the network interface was down - setting the maximum FPS solved this. Thanks for the info!
PacoLM
Posts: 971
Joined: Wed Dec 02, 2009 9:55 pm
Location: Spain

Re: High CPU when camera is disconnected

Post by PacoLM »

knnniggett wrote:
rockedge wrote:I am running into a problem I first noticed working with the master development version 1.30.1 where the /dev/shm creeps upwards until it hits 100% and cameras begin to shut down and the log indicates

Code: Select all

disk_free_space returned false for /dev/shm
I wonder if these are related.
the first camera to shutdown is prone to disconnect then reconnect which seems to be WiFi related.
This is happening when a camera is restarted w/o restarting zoneminder. One or more of the perl daemons are keeping open file handles on the mmap files when they shouldn't. It'll be fixed by the next release.

You can see this happening with a "sudo lsof /dev/shm"
I also noticed it. /run/shm keeps growing. In my case, the workaround is to restart Zoneminder from time to time.

After more than 15 years, no longer using ZM as surveillance system.
Now in the dark side, using a commercial system...
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: High CPU when camera is disconnected

Post by knight-of-ni »

PacoLM wrote:
knnniggett wrote:
rockedge wrote:I am running into a problem I first noticed working with the master development version 1.30.1 where the /dev/shm creeps upwards until it hits 100% and cameras begin to shut down and the log indicates

Code: Select all

disk_free_space returned false for /dev/shm
I wonder if these are related.
the first camera to shutdown is prone to disconnect then reconnect which seems to be WiFi related.
This is happening when a camera is restarted w/o restarting zoneminder. One or more of the perl daemons are keeping open file handles on the mmap files when they shouldn't. It'll be fixed by the next release.

You can see this happening with a "sudo lsof /dev/shm"
I also noticed it. /run/shm keeps growing. In my case, the workaround is to restart Zoneminder from time to time.
This has been fixed in master:
https://github.com/ZoneMinder/ZoneMinder/pull/1717
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
rockedge
Posts: 1173
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA

Re: High CPU when camera is disconnected

Post by rockedge »

good work!
DoubleHP
Posts: 18
Joined: Fri Sep 02, 2016 1:13 pm

Re: High CPU when camera is disconnected

Post by DoubleHP »

Don't have direct relation between /dev/shm use and cam unreachable. Events may happen at the same time, or not.
Locked