debugging load average and ZM jpeg saving vs shinob i h264

Forum for questions and support relating to the 1.30.x releases only.
Post Reply
marcmerlin
Posts: 65
Joined: Thu Jan 17, 2013 6:13 pm

debugging load average and ZM jpeg saving vs shinob i h264

Post by marcmerlin » Tue Mar 13, 2018 5:18 am

I've had zoneminder for a long time, upgraded to 1.30 not that long ago (yes, I know, 1.31 just came out it seems), and I can tell my system is stressed by the load, even if it manages still it seems.
I have 12 cameras, most are 1080p. I have them as Mocord, which does mean they are recording at all times.
I have maximum FPS set to 7 and alarm maximum FPS also set to 7
All cameras are set for jpeg or mjpeg capture, so there is no H264 video decoding
Cameras are recording to a double drive raid0 array for a bit of extra speed, on btrfs on top of bcache backed by an ssd.
CPU-wise I have a quad core Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
32GB of RAM
OS is currently 32bit to save on resources (smaller pointers, smaller binaries), but kernel is 64bit

I was curious that yahoo claims
"JPEG Storage is awful
Saving each frame as a separate file in JPEG format can have a seriously detrimental effect on storage space and the hardware itself.
yahoo saves to WebM and MP4 files. While MP4 takes a fair amount of space.. its level of CPU usage during encoding for H.264 streams is just amazing."

Usually I like benchmarks and data over an empty sentence that includes "speed is amazing" with absolutely no info on the I/O vs CPU tradeoff.
But I'm curious, have others looked into this? Am I better off staying with jpeg frame storage over H264 from yahoo?
Part of my "problem" I guess is that I'm using mocord, and currently have bulk frames set to 100, but this is not working well for me because the whole point of mocord is that I want to record even if no motion is detected, and if ZM detects no motion it collapses a lot of frames together, even though they're not quite the same, and they may have data I wanted to look at (this has happened before sadly and the video frames I wanted were gone because ZM didn't see movement and junked them as part of bulk processing).
If I were recording to video instead, the video codec would record just the difference and let me replay the frames and let me see if there is a difference that's important to me, but not enough to trigger motion detection.

That being said, I'm also getting really weird recording from event recording.
Look below, I have recordings of 22 seconds, and others of 55mn with 9811 frames (!)

Id Name Monitor Cause Time(v) Duration Frames Alarm Frames Total Score Avg. Score Max. Score
4882250 Event-4882250 cam-frontyard Continuous 03/12 15:34:34 00:05:25 2151 0 0 0 0
4882227 Event-4882227 cam-frontyard Continuous 03/12 15:30:00 00:04:36 1677 7 46 6 8
4882198 Event-4882198 cam-frontyard Continuous 03/12 15:20:00 00:09:59 3811 0 0 0 0
4882174 Event-4882174 cam-frontyard Continuous 03/12 15:10:00 00:10:00 3780 0 0 0 0
4882145 Event-4882145 cam-frontyard Continuous 03/12 15:05:38 00:04:21 1556 0 0 0 0
4882069 Event-4882069 cam-frontyard Continuous 03/12 14:37:36 00:27:58 5209 4950 265363 53 60
4882055 Event-4882055 cam-frontyard Continuous 03/12 14:31:44 00:05:53 1842 397 21995 55 59
4882044 Event-4882044 cam-frontyard Continuous 03/12 14:30:00 00:01:45 413 227 13115 57 60
4882039 Event-4882039 cam-frontyard Continuous 03/12 14:29:30 00:00:30 203 0 0 0 0
4882033 Event-4882033 cam-frontyard Continuous 03/12 14:28:37 00:00:52 225 94 5316 56 59
4882030 Event-4882030 cam-frontyard Continuous 03/12 14:27:33 00:01:06 258 121 6999 57 60
4882010 Event-4882010 cam-frontyard Continuous 03/12 14:20:46 00:06:47 1528 1127 61162 54 60
4882003 Event-4882003 cam-frontyard Continuous 03/12 14:20:00 00:00:46 226 4 221 55 56
4882000 Event-4882000 cam-frontyard Continuous 03/12 14:19:54 00:00:05 35 0 0 0 0
4881996 Event-4881996 cam-frontyard Continuous 03/12 14:19:28 00:00:25 138 1 59 59 59
4881992 Event-4881992 cam-frontyard Continuous 03/12 14:18:29 00:00:59 329 30 1736 57 59
4881991 Event-4881991 cam-frontyard Continuous 03/12 14:18:03 00:00:25 123 10 576 57 59
4881989 Event-4881989 cam-frontyard Continuous 03/12 14:16:26 00:01:37 513 58 3311 57 59
4881985 Event-4881985 cam-frontyard Continuous 03/12 14:15:01 00:00:00 1 0 0 0 0
4881986 Event-4881986 cam-frontyard Continuous 03/12 14:15:01 00:01:24 462 26 1473 56 59
4881844 Event-4881844 cam-frontyard Continuous 03/12 13:20:00 00:55:00 9811 8642 409480 47 60
4881826 Event-4881826 cam-frontyard Continuous 03/12 13:17:26 00:02:33 976 0 0 0 0
4881809 Event-4881809 cam-frontyard Continuous 03/12 13:10:00 00:07:12 1672 748 14318 19 30
4881798 Event-4881798 cam-frontyard Continuous 03/12 13:08:13 00:01:46 671 0 0 0 0
4881793 Event-4881793 cam-frontyard Continuous 03/12 13:05:39 00:02:28 596 15 75 5 5

This may be a totally different problem though.
Anyway, back to recording and jpeg recording vs yahoo h264, what are your thoughts?

bbunge
Posts: 2205
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by bbunge » Tue Mar 13, 2018 12:41 pm

First off you have two settings that should be blank unless you have local cameras: Maximum FPS and Alarm Maximum FPS
With streaming cameras (AKA IP cameras) set the frame rate at each camera as well as the resolution. My recommendation is 5 FPS with 640x480.
1.31.x is still in beta testing and not officially released.
Performance of your system depends on a lot of factors: LAN quality, camera output, server processor and hard drive speed. I'm running 17 IP cameras at 640x480 5 FPS on an i5 Quad Core (3.2 GHZ), 16 GIG RAM and a WD Blue 1 TB desktop drive. Four cameras on Mocord during the day with at least 5 PC's running Montage of the cameras. The system is not stressed at all and I'm considering reducing the RAM to 8 GIG.
I've recently set up a used Poweredge server with RAID 1 on 250 GIG drives for the OS and a 1 TB drive for events (see: https://wiki.zoneminder.com/Common_Issu ... ive_or_NAS). This "old dog" is fast in testing compared to a single drive system and may get upgraded to Ubuntu 18.04 before it is put in service.
But I've dodged the issue of .jpg files vs. video. And that like all things depends. Video, like movie film, is still created from a series of still pictures. So it boils down to where you want to use processing power. A security camera system is not a HD movie machine but a balance of frame rate and image size to catch that thieving rodent!

marcmerlin
Posts: 65
Joined: Thu Jan 17, 2013 6:13 pm

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by marcmerlin » Wed Mar 14, 2018 12:27 am

Thanks bbunge. I do jpeg pulls, isn't that speed controlled by my server? (I don't like H264 pushed by the camera because if my PC hangs for a bit, or the network is congested, the H264 stream gets damaged and I get broken frames. With jpeg pulls, I know my PC controls the data pull, and doesn't have to waste CPU in H264 decoding).

As for doing 640x480, not really, I do at least 1920x1080 for my lowest res cams, some are a bit higher res. Yes, that's expensive, but if I can't see who is trying to steal from my mailbox or get their license plate, let's say, what's the point of the camera? :)
For your system doing ok, you' have 4 to 6x less load per camera than I do due to the lower res, so I'm not surprised yours is doing fine.
As saving output as H264 video instead of jpeg frames, it sure would save me disk space if I turned down bulk from 100 to 0, but I'm indeed worried about the CPU impact and don't quite trust shinob i's claim that "speed is amazing".
I'm kind of curious if people have tried both and can report the results.

User avatar
knight-of-ni
Posts: 2180
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by knight-of-ni » Wed Mar 14, 2018 12:43 pm

Hey, I like the way you think. I don't have to tell you that, the author of the software of which you speak, loves to make inaccurate statements, just like the one you quoted, to promote himself. Those of us with real experience in the industry know that jpegs are still quite useful, especially since storage is so cheap. In any case, you are asking the right questions. Ignore the hype. Rather, collect real data. Then make your own decision on what product you want to use.

Along the lines of what bbunge said, you've told us what values you've got in the max fps fields, but the important thing to know is the framerate of the stream itself coming from the camera. If the framerate in the camera is higher than 7, then you aren't saving cpu resources at all. You are increasing the cpu load.

If you want to stick to the latest release of zoneminder, the way to minimize cpu usage is:
- set the frame rate in the camera to something low. I like to use 5 fps, and yes that is plenty, even for moving objects. bbunge likes to go lower.
- don't use the max fps fields (This is a bug that awaits someone with time to fix it. You *can* put a value in here higher than the camera's framerate. )
- set your camera streams to mjpeg rather than h264. Cpu usage will drop like a rock, at the expense of significantly more bandwidth.
- Lowering the resolution will significantly lessen cpu load, but I totally get it if you want to keep that 1080p image. I would too.

If you want to live on the edge, try one of our development packages:
- set the framerate in the camera to something low
- keep max fps fields blank
- enable h264 passthrough in the monitor config, which will save straight to mp4
- disable jpeg recording

I saw your other thread about bulk frames. Despite having used ZoneMinder for a decade, there are still some features I've just never used, and that is one of them. It will be interesting to see if you get any responses. Generally speaking however, the things that use the most cpu resources are resolution, framerate, compression type. Motion detection not so much these days, and I would not expect bulk frames to matter much either.

Lastly, the author if which you speak loves to masquerade under different names. If a post shows up in this thread from a new user, trash talking zoneminder or those that participate in the project, I'll give you one guess who that is.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/

marcmerlin
Posts: 65
Joined: Thu Jan 17, 2013 6:13 pm

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by marcmerlin » Wed Mar 14, 2018 3:26 pm

Thanks for the reply. Yes, I smelled BS there, glad to hear confirmation :)
But you are both talking about framerate coming from the camera. I'm doing jpeg http pulls from the camera. It's not pushing anything to me, I'm pulling from it.
I agree that even 3fps is enough actually, I'm just not even getting that most of the time, because I have a bottleneck somewhere I haven't found yet.
But you're saying I should use mjpeg push, sadly many of my cameras do it badly or not at all. Is it really better than jpeg pull? I thought that outside of the http round trip for each jpeg pull, it'd be about the same?
Thanks also on the warning about the other software's author :)

User avatar
knight-of-ni
Posts: 2180
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by knight-of-ni » Wed Mar 14, 2018 5:00 pm

Ah, I see. We call that snapshot mode. I misunderstood that was what you were doing, so yes you certainly can use the max fps fields.

I don't know for certain how much better mjpeg would be than jpeg snapshots.
It would be worth it to test which method is more cpu efficient. Watch your zma & zmc processes of a particular camera in snapshot mode to get a rough idea of the cpu utilization. Then change over to an mjpeg stream and repeat the process. That is of course if the camera in question supports mjpeg.

All other other things being equal, I am reasonably certain the mjpeg stream will be more efficient, but I am unsure by how much. It comes down to just how expensive it is to constantly open & close a connection to the camera, which is what is going on in snapshot mode, as you have noted.

I see what you mean from your first post about unexpected event statistics. It's really hard to know what caused some of the more unusual looking statistics without knowing a lot more about what was going on with the hardware, the motion zone configuration, and what was in the camera's field of view at the time the event occurred. I am curious to know what drove you to choose this mode. Would you be willing to use normal modect with a super sensitive motion zone? It's typically easy to make the zone very sensitive so it records even the slightest movement. I don't know at this point if disk io is your bottleneck, but switching over to record only on motion would certainly help with that.
Speaking of bottlenecks, zoneminder will leave a warning in your logs when it can't keep up and your buffers are about to be overrun. Do you see anything like that?
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/

marcmerlin
Posts: 65
Joined: Thu Jan 17, 2013 6:13 pm

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by marcmerlin » Thu Mar 15, 2018 5:13 am

thanks for mjpeg vs jpeg pull. I'm pretty sure it lowers my frame rate a bit, but doesn't otherwise affect me much, so I'm not too worried there.
Why mocord? If I use modetect and have a super sensitive zone, then I'll have even more data written down to disk since bulk won't work, and I'll have to tweak my job that deletes 0 score frames after x days intead of y days (y>x)
Mar 14 22:05:55 gargamel zma_m2[23240]: INF [In overload mode, 2 frames of 3 remaining]
I only have a single camera overloaded, and unsurprisingly it's the only one I can't do jpeg pulls from and that is pushing h264 to me.

For the rest, I have:
Mar 14 22:07:57 gargamel zmc_m2[11336]: INF [cam-fmr: 879000 - Capturing at 5.08 fps]
Mar 14 22:08:16 gargamel zma_m5[22542]: INF [cam-fmr-kitch: 787000 - Analysing at 4.95 fps]
Mar 14 22:08:19 gargamel zma_m10[31832]: INF [cam-kitchen: 1458000 - Analysing at 9.62 fps]
Mar 14 22:08:33 gargamel zmc_m4[11337]: INF [cam-garage: 865000 - Capturing at 5.00 fps]
Mar 14 22:08:34 gargamel zmc_m10[11334]: INF [cam-kitchen: 1653000 - Capturing at 9.90 fps]
Mar 14 22:08:36 gargamel zmc_m24[26146]: INF [cam-mailbox-front: 10000 - Capturing at 1.95 fps]
Mar 14 22:08:42 gargamel zma_m11[24888]: INF [cam-lvr: 1327000 - Analysing at 9.71 fps]
Mar 14 22:08:47 gargamel zma_m4[4446]: INF [cam-garage: 723000 - Analysing at 4.98 fps]
Mar 14 22:09:03 gargamel zmc_m3[11352]: INF [cam-frontdoor: 812000 - Capturing at 4.85 fps]
Mar 14 22:09:04 gargamel zmc_m11[11342]: INF [cam-lvr: 1676000 - Capturing at 10.00 fps]
Mar 14 22:09:05 gargamel zma_m2[23240]: INF [cam-fmr: 752000 - Analysing at 5.03 fps]
Mar 14 22:09:44 gargamel zmc_m6[11323]: INF [cam1-garage: 115000 - Capturing at 0.70 fps]
Mar 14 22:10:00 gargamel zma_m10[31832]: INF [cam-kitchen: 1459000 - Analysing at 9.90 fps]
Mar 14 22:10:12 gargamel zma_m16[22954]: INF [cam-backyard: 613000 - Analysing at 4.55 fps]
Mar 14 22:10:14 gargamel zma_m6[12123]: INF [cam1-garage: 115000 - Analysing at 0.70 fps]
Mar 14 22:10:14 gargamel zmc_m10[11334]: INF [cam-kitchen: 1654000 - Capturing at 10.00 fps]
Mar 14 22:10:23 gargamel zma_m11[24888]: INF [cam-lvr: 1328000 - Analysing at 9.90 fps]
Mar 14 22:10:29 gargamel zmc_m17[25716]: INF [cam-mailbox-rear2: 303000 - Capturing at 6.02 fps]
Mar 14 22:10:45 gargamel zmc_m16[11330]: INF [cam-backyard: 684000 - Capturing at 4.57 fps]
Mar 14 22:10:45 gargamel zmc_m11[11342]: INF [cam-lvr: 1677000 - Capturing at 9.90 fps]
Mar 14 22:10:46 gargamel zmc_m5[11354]: INF [cam-fmr-kitch: 863000 - Capturing at 4.95 fps]
Mar 14 22:11:14 gargamel zmc_m2[11336]: INF [cam-fmr: 880000 - Capturing at 5.08 fps]
Mar 14 22:11:32 gargamel zma_m23[27367]: INF [cam-mailbox-rear: 303000 - Analysing at 1.92 fps]
Mar 14 22:11:37 gargamel zma_m5[22542]: INF [cam-fmr-kitch: 788000 - Analysing at 4.98 fps]
Mar 14 22:11:41 gargamel zma_m10[31832]: INF [cam-kitchen: 1460000 - Analysing at 9.90 fps]
Mar 14 22:11:54 gargamel zmc_m4[11337]: INF [cam-garage: 866000 - Capturing at 4.98 fps]
Mar 14 22:11:55 gargamel zmc_m10[11334]: INF [cam-kitchen: 1655000 - Capturing at 9.90 fps]

Interesting that I don't get the same interval of logging for all cameras, but oh well...
Anyway, the logs look ok enough, but in real life I often get multi second gaps between frames. I need to go back to see if it's only in non alarm state or in all states...

rockedge
Posts: 923
Joined: Fri Apr 04, 2014 1:46 pm
Location: Connecticut,USA
Contact:

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by rockedge » Thu Mar 15, 2018 3:00 pm

I have tested both sets of software extensively... I found ZM to be more reliable in the long run especially with motion detection. ZM can be modified and "suped up" and being able to interact with ZM using PERL, PYTHON, PHP or shell scripts put it into a different league.
Overall system loads in my experience and general wear and tear on the machine hardware seem less with ZM than the software who's name we don't mention. The potential to use the jpegs for object recognition analysis is much greater than trying to do the same using the streams produced by Shinob i. Just my experience.

marcmerlin
Posts: 65
Joined: Thu Jan 17, 2013 6:13 pm

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by marcmerlin » Thu Mar 15, 2018 3:20 pm

Thanks rockedge, this is very useful feedback. I very much appreciate that you're saving me the time to try the other software to see whether it's faster on my system, or not.

User avatar
knight-of-ni
Posts: 2180
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: debugging load average and ZM jpeg saving vs shinob i h264

Post by knight-of-ni » Thu Mar 15, 2018 5:30 pm

Excellent objective feedback from Rockedge. Thank you.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests