ZM on CentOS 7 ESXi VM getting disk full

Forum for questions and support relating to the 1.30.x releases only.
Locked
melloa
Posts: 25
Joined: Tue Apr 05, 2016 12:54 am

ZM on CentOS 7 ESXi VM getting disk full

Post by melloa »

I've been running into this issue from time to time. The VM will stop without space and I have to restore a snapshot to get it going.

My ZM VM has a 1TiB VHD and the PurgeWhenFull filter is running and set to 89% of the disk space. I even set a cron job to alert me if the disk space utilization went above 90% and runs every hour, but didn't get any alert. It seems that it might have happened within the hour, between cron runs, as I've tested the script for a full day with a low threshold and got alerts.

Today, after couple hours running, my partitions shows:

Code: Select all

[root@zoneminder ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/cl-root 1015574016 101567680 914006336  11% /
devtmpfs               1930500         0   1930500   0% /dev
tmpfs                  1941156    587656   1353500  31% /dev/shm
tmpfs                  1941156     16916   1924240   1% /run
tmpfs                  1941156         0   1941156   0% /sys/fs/cgroup
/dev/sda1              1038336    233756    804580  23% /boot
/dev/mapper/cl-home   52403200     32964  52370236   1% /home
tmpfs                   388232         0    388232   0% /run/user/0
So I'm using 11% of my root partition - I customized the partitions to allocate the space to root and not need to move the storage.

Is there anything in the logs that would help me fix this problem? I'd assume if the PurgeWhenFull filter is running correctly, I'd never run into this situation.

The problem is when it explodes, the VM won't even start, so I can't access it to get any information from logs, free-up space, etc. Just restore the snapshot.

Any thoughts appreciated.
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: ZM on CentOS 7 ESXi VM getting disk full

Post by knight-of-ni »

Your events folder should always be a dedicated drive or partition.

See the wiki:
https://wiki.zoneminder.com/Using_a_ded ... Hard_Drive
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
melloa
Posts: 25
Joined: Tue Apr 05, 2016 12:54 am

Re: ZM on CentOS 7 ESXi VM getting disk full

Post by melloa »

Before I moved ZM to a VM, I was running it on a CentOS 7 server with 2x2TiB HDDs with the same partition scheme, where root had 90% on the total space (~3.6TiB) for several months without encountering this issue.

I'll follow your suggestion to test and create a small VHD for the OS/ZM and a second one for the events.

On a side note, I did see the above mentioned article, but as it says "you may wish" didn't take as I should have a separated partition.

Thanks again for your reply.
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: ZM on CentOS 7 ESXi VM getting disk full

Post by knight-of-ni »

Well, yes it does say "you may wish", but if you don't do that then you put yourself at risk to exactly what just happened to your system. The decision is yours. You have to understand the risks.

If you think there is something in the wiki which needs clarification, then feel free to edit it. The wiki is maintained by the community.

To describe what happened in generic terms, regardless of the application software or even the operating system, if you install a piece of software which, by design, fills up the hard disk it was writing to, would you really let it do that to your boot volume? And if you do choose to go down that path, would you really be surprised bad things happen when the boot volume fills up completely?? I'm being rhetorical.

The fact that it didn't happen on your previous system simply means you were lucky. You didn't run into a situation where the purge filter did not or could not delete events faster than were being written to the disk.

For a production system, setting up your events folder onto a dedicated drive or partition is the first thing you should do. The next thing you must do is, tune your purge filter. Once you get your events folder off the boot volume, monitor the system when it reaches the threshold of the purge filter. If I recall, the default is 95% full. Once the purge filter kicks in, the disk percentage at any given time should stay between 93-96%, give or take. If it goes above that, then you need to tell the filter to delete more events at a time. If the other extreme occurs, then you should tell the filter to delete fewer events.

Note that, the best time to monitor the system is during a period of high activity. I don't know what your cameras are watching, but a thunderstorm makes a good test for a system with outdoor cameras. Also make sure ZM_OPT_FAST_DELETE is off. It is off by default.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
melloa
Posts: 25
Joined: Tue Apr 05, 2016 12:54 am

Re: ZM on CentOS 7 ESXi VM getting disk full

Post by melloa »

Not saying you don't have a point. I have taken all measures to prevent Zoneminder to cause problems and it seems to be working correctly, as designed. I've noticed that VMWare is filling the datastore with disks and was using 98% of the dataset. That could be the reason that the VM stopped without send me a notification from cron that the disks was getting full.
I'm going to change my VM as you've suggested later, as I want to get to the bottom of this to make sure VMWare is the bad guy here. I'm sure ZM filter is working, as I've tested several times, so no issues for ZM, but want to provide this status to the ZM users that are running it in a VM.
Thank you again for your attention.
Locked