Alarms are dropping frames.

[skidd]

For some reason.. when alarms (or web based forced alarms) fire, I'm dropping frames. I'm running out of things to check. The kicker is, I have a 2nd test machine that I threw ZM on, and it works perfectly. So.. it's something about my main machine, but I can't yet seem to figure out what. Except for the disk (which is my suspicion) the 2 machines are very similar.

My main server is an older Phenom II quad core 3.2Ghz with 8G of ram.
6 SATA drives, with a software RAID1 root partition. and a software RAID5 data partion.
Teh test machine was a 2 core 3Ghz Phenom II with 8G or ram, and a single 500G SATA drive.
Both are X64 Arch Linux Distro machines.

When running a single H264 based camera, the CPUs sit at about 30%, and the Disk IO sits about below 20% as well.

I have my suspicions that it's a disk IO problem, but I'm not convinced. I also suspect MySQL has some blame, but also not totally convinced.

Currently I have my ZM (1.28.1) setup with a link to a single outdoor camera. It's a H264 3MP camera setup to stream only 10fps.
ZM has no trouble reading the camera using ffmpeg. the live view mode looks perfect, an no frames are missing.
ZM is setup with just a simple "AlarmPixles' zone to watch for motion. It seems to pick up motion just fine.
If i trigger a web-based force alarm, or if it detects motion, the frames recorded quickly drops from 10fps down to about 5fps.
It starts out decent recording about 2 seconds at 10fps, but by the time it's into the alarm frames, its' down to 5fps and never recovers.

But... if I switch the camera to "Record", it gets every single frame without fail. I set it churn on Record for about 10 minutes, and it captured every single frame no problem. Not one missed. Which tells me, its' capable of capturing the frames, but something is going wrong when in Modect mode.


The camera as 2 aux streams, I tried setting up a MJPG aux stream at 704x572 and 10fps. I setup a 2nd monitor and set it up as the alarm trigger for the other main stream.
It did better.. much better.. but still dropped frames. Both the lower res MJPG stream and the main X264 stream dropped about the same frames. On average both managed between 6-9fps. Some short 10fps runs also. But.. still not the perfect 10fps that "Record" manages.

I know software raid isn't awesome, but I've never had an issue with it's IO.
Running a simple IO test on my drives.

Code: Select all

dd if=/dev/zero of=tmp.img bs=1G count=1 oflag=dsync

produces between 80MB/s and 100MB/s on my RAID5 drive.
And a less than ideal 40-50MB/s on my root drive.

I've tried moving the mysql files to a stand alone SATA partition, and it didn't seem to make any improvement.
Iv'e tried tweaking various innodb values in my.conf to increase performance, but it didn't seem to help.
I have a gigabit homenetwork, so I even tried hooking ZM to a a MySQL database on my Laptop (i7, 32G ram, SDD disk) but it made no difference.

One thing I notice though, is the mysql database on my main machine really doesn't seem snappy. It takes almost 4x as long to run the zm_create.sql on it as it does on my test machine.

So.... my guess is ZMA has something to do with this. Is it somehow failing to tell ZMC to capture various frames? The reason I suspect ZMA is really just because when it's removed from the equation (Record mode) I get every frame.

I do get the usual share memory warnings in my log file, but I don't know if that is the cause.

Code: Select all

ZMU - Shared data not initialised by capture daemon, some query functions may not be available or produce invalid results for monitor FrontYard

Sorry for the long post... but I wanted to try and provide as much info as I could.
Any insight on what to look for ?

Cheers
Shane

[skidd]

P.S. I forgot to mention, I tried OPT_FRAME_SERVER but it didn't help at all either.

[skidd]

A bit more info for this strange issue.
I once again offloaded the mysql database to a dedicated temporary machine. They are hooked to decent quality gigabit switch, so there is very little, if any, DB lag.
In fact, the "zm_create.sql" when run on the server machine, to the remote database, runs 4x faster than the same script on the server to it's local db. That in itself is telling to me.
Ok.. so.. DB should be fast now.
I left enable the OPT_FRAME_SERVER flag, as it did seem to help a little.
When I fire up a manual alarm, I get most, but not all frames. Here is the interesting part.
My ZM log shows some new.. interesting errors. A sequence of these in around the time the alarm is running.

Code: Select all

2015-10-11 11:24:33.534271	zmf	4891	INF	Frame server socket open, awaiting images	/tmp/pacaurtmp-spowell/zoneminder/src/ZoneMinder-1.28.1/src/zmf.cpp	86
2015-10-11 11:24:33.413326	zmf	4891	WAR	Socket closed at remote end	/tmp/pacaurtmp-spowell/zoneminder/src/ZoneMinder-1.28.1/src/zmf.cpp	274
2015-10-11 11:24:33.404964	zmf	4891	ERR	Only read 219224 bytes of 555487	/tmp/pacaurtmp-spowell/zoneminder/src/ZoneMinder-1.28.1/src/zmf.cpp	267
2015-10-11 11:24:33.401445	zma_m1	4917	ERR	Incomplete frame write: 219264 of 555527 bytes written	/tmp/pacaurtmp-spowell/zoneminder/src/ZoneMinder-1.28.1/src/zm_event.cpp

Which.. once again leads to ZMA being somehow the guilty party in my problem.
Anybody have a guess what could be causing this? Seems like a unix socket issue of some sort. Is there a limiting factor to the IO of sockets?

Thanks for any insight anybody can provide.

[skidd]

Still more experiments.
I setup a tmpfs (in memory Filesystem) for both the zoneminder/tmp and zoneminder/socks and zoneminder/swap directories.
No issue at all with disk IO now.
Same thing.. a lot , but not all frames are recorded.
Teh above zmf errors are still being recorded. It cant' be a socket issue.
ZMA .. is for some reason failing to send some frames, and as a result they are being dropped.

[bbunge]

Try turning off CREATE_ANALYSIS_IMAGES.

Would guess you are using ffmpeg? Have you tried changing the key frame interval on your camera to something shorter?

You did put PATH_MAP and PATH_SWAP in your RAM disk?

My experiences with software RAID in the past have not been very successful. I've also discovered that I have problems with Seagate drives. Just something about them that they rack up errors as reported by S.M.A.R.T. even though SeaTools testing passes the drive. You may want to find a way to check each drive in your RAID. One bad lemon can spoil the bunch...

[skidd]

Turning off the analysis images dropped the number of those "only read 219224 bytes" errors down from quite a few, to only a couple. In the end I didn't plan to leave those turned on anyway.. but it's good to know I can at least (almost) get rid of that error.

I tried dropping the i-frames down from 50 to10 on my camera (10fps).. didn't do any good.
PATH_MAP (/dev/shm) and PATH_SWAP (/tmp/zm) are both indeed in RAM disk.

All my drives are Western Digital. I too have not had great luck with SG drives in the past.

So.. the fact that my slightly less powerful other test machine, is only really different in that it's running a single drive, and no Software raid. and as much as I hate to admit it, I'm thinking it could (if not must) be related to that. For a test, I removed one of the 2 drives in the RAID1 that my root partition is on. my dd IO test jumped from 50MB/s to 80MB/s. Now... I still lose frames, but only about a total of.. oh... 5-10 over a 20 second alarm. that's very much a step in the right direction. The odd thing about this though... most of the data files are held on my RAID5 partition, which manages over 70MB/s most of the time.

At one time, I tried moving all of the mysql DB files onto a non-raid extra partition, and also all related ZM files onto there as well. Thinking at the time it "might" be Disk IO related. Now, I think more so, it somehow is indeed disk IO related. I seem to recall nmon showing me a great deal of IO still going onto my main root partition though.. which seemed odd at the time. I'm going to re-try that test again. .. tomorrow...

Thanks for your help.

[skidd]

So far.. it's all working. .. I say so far because I've jumped the gun a couple of times already here.. but...

Ok.. so my main root partition running RAID1 is only good for <50MB/s. My main data RAID5 is good for >90MB/s.
I moved my mysql DB files to the RAID5 disk. SQL test scripts I had ran 5-6x faster.
That almost totally fixed the problem. I was getting almost every single frame.. except for perhaps one or two.

One thing that was still happening was the ZMF server was missing frames due to a buffer size error. So... I dug deeper into that one. Turns out my system default in /proc/sys/net/core/wmem_max was set for 219228. Almost the exact same size as the buffer count error ZMF was giving me. So. .I upped the wmem_max and restart ZM. Still got the error. So.. I used the setting in ZM that allows me to specify the socket size. Now.. that error is gone too.

The only error I still get is the "Shared data not initialized by capture daemon for monitor" error.


Moral of the story... dont' dismiss the disk performance requriesments of the MySQL database. I thought 50MB/s was enough.. it wasn't.

[skidd]

Yep.. jumped the gun.. it is indeed quite a bit better... so I'm on the right track.. but I'm still losing a few frames during alarms.

[knight-of-ni]

In my experience, zmf has always done the opposite of what it claims. Rather than speed things up, it tends to get in the way and slow things down. That isn't to say that someone has managed to make it helpful, I've just never seen it.

Out of the box, mysql installs completely untuned and is incredibly slow.
The query cache isn't even turned on.

Most everyone should spend time using mysqltuner and reading about the various parameters the mysqltuner tool suggests to modify. Run mysqltuner, make the changes it suggests, then keep running mysqltuner once a day for a week to make sure it does not discover anything new.

Most notably, one should turn on the query cache and make sure the innodb buffer pool is large enough to get all your tables into memory. The difference in dB performance will be significant. I think bbunge may have written a guide on this if I remember. Should be in the Wiki somewhere.

[bbunge]

No specific guide...yet. Guess I've got some work to do...

bb

[skidd]

Thank for that guide bbunge.
An update to this saga of mine, trying to record every frame.... it's working.. again. Yeah. yeah.. I've said that before.. hopefully I won't be posting another "spoke too soon" post.
I used your DB tuning wiki
http://www.zoneminder.com/wiki/index.ph ... cks.2FTips
And proceeded to switch to files per table.
In the process... some of the steps just kept getting a little off track.. like random errors on the upgrade script and all-db.sql import script.
so I said enough.... I did a fresh-brand new "mysql_install_db" and a re-import of all my databases... after setting up the "innodb_file_per_table" setting.
The all-db.sql import when error free. The forced upgrade went error free. This time, it all looked good.
Fired up apache, mysql and zoneminder... now.. so far.. all my frames are being recorded.
(fingers crossed) it will keep working!!

P.S. I'd been working with mysqltuner to try and get it running better. I'll keep working with it to be sure I'm getting all the performance I should.

[dmitripr]

Hi Skidd, did you resolve your issue?

I've got a brand new setup and I think I have exactly the same problem as you. I've followed your "journey" here, tried the same stuff, but still have the issue. I've opened a separate topic, since I've got 1.29 version. viewtopic.php?f=34&t=24663

If anyone else has any other advice, I'd be happy to hear it. Thanks!