Need help to understand what's bringing down my ZM

[tcg]

I have a ZM 1.30.4 server with 25 cameras running on Ubuntu Server 16.04. It's a mixture of analog cameras and IP cameras. All cams use ffmpeg, 13 pulled from an older analog DVR and the rest from IP cams. A couple of weeks ago I upgraded the server and increased RAM from 16GB to 32GB and added additional hard drive space. I reduced the resolution of all cameras to 640x360 and the fps to 15 or less to reduce the load on the server for analysis. I also set up additional nodect monitors for all of the IP cameras and configured them to trigger from their lower-resolution counterparts when motion was detected so that I could get an additional HD capture during motion. Shortly after the upgrade, I started experiencing problems. A camera or two would drop out and then the problem would kind of cascade through all the rest of the cams. In montage view you could see most of the camera go offline and then return within 20-30 seconds and the Source column on the main ZM page would turn red. It's typically much worse when there is a lot of motion happening. It almost always starts with the older analog streams but then affects all the others.

When this cascading outage occurs, I find a lot of mysql errors like this:

Code: Select all

2018-02-23T13:37:09.331389Z 28076 [Note] Aborted connection 28076 to db: 'zm' user: 'zmuser' host: 'localhost' (Got an error reading communication packets)

I've been researching this for days. I can't decide whether the problem is on the MySQL side and it ripples into ZM or if it's the other way around. I've run mysqltuner and adjusted various things in MySQL but it hasn't helped so I've backed out most of the suggestions.

My CPU load is around 70-80% and my network throughput is 80-90Mbps. ifconfig shows no dropped packets. My SHM looks like this:

Code: Select all

tmpfs            16G  4.9G   11G  31% /dev/shm

I'm attaching a couple of log files I could sure use some help figuring this out.

[iconnor]

probably out of mysql connections. Typical mysql can only do 214 without additional configuration.

[rockedge]

looking over your mysql_error_log.txt and I saw this exact error occurring on a test platform. I was using an Ubuntu 16.04 derivative and was in the process of setting up ZM 1.30.4. In my case it was the Apache configuration and the ZM database. I will look at my notes at the moment I am pressed for time so I will see what I can find again in a few hours or sooner.

[tcg]

probably out of mysql connections. Typical mysql can only do 214 without additional configuration.

As a test, I raised the limits to 65535 for files and 1024 for max_connections. The MySQL log no longer shows the 214 limit on connections as it did in my previous attachment. Still having the same problem though. MySQL "show processlist" shows around 160 rows. I don't think I've seen it peak much higher than that.

[bbunge]

Running mysqltuner was the right thing to do!
However with the number of cameras, including duplicates, max_connections = 1024 may not be enough. Might also need to increase innodb_buffer_pool_size =. As you have a good bit of RAM start with 2G and set innodb_buffer_pool_instances = 2
With all your cameras check the tmpfs (/dev/shm). 32 GIG of RAM should result in a tmpfs of 16 GIG which should be more than enough.
You are running a 64 BIT version of Ubuntu? I ask this because even with 32 GIG of installed RAM a 32 BIT system will still use only about 4 GIG unless PAE is working.
I would drop the nodect cameras, they eat up space in /dev/shm, and just use the main cameras for motion detection. reduce the frame rate to 5 FPS and if possible increase the key frame interval. This is a security camera system and not a HD movie maker after all.

[tcg]

Thanks for the tips. I've made the changes you suggest plus a few others but the "aborted connection" errors are still coming:

Code: Select all

innodb_buffer_pool_size = 2G
innodb_buffer_pool_instances = 2
innodb_log_buffer_size = 32M
innodb_log_file_size = 512M
innodb_open_files = 5000
table_open_cache = 1000
net_read_timeout = 60
net_write_timeout = 120
connect_timeout = 20
max_connections = 5000

Yes, my system is 64bit.

You mentioned dropping the nodect cams. So my idea was to provide a lower resolution for mobile devices connecting via zmNinja, reduce CPU load for analysis and still get HD captures. I named all of the nodect cameras with a leading _ to distinguish them but I'm still seeing analysis logs for them. I thought nodect avoided doing motion detection. Maybe I misunderstood what it's for.

Is there a better way to accomplish what I'm trying to do here? I have to admit I didn't have these problems until after I made all these changes to try to make things better. :/