Video Storage deletion issues

Post here to ask any questions about hardware suitability, configuration in ZoneMinder, or experiences. If you just want to know if something works with ZoneMinder or not, please check the Hardware Compatibility sections in the forum, and the Wiki first. Also search this topic as well.
Post Reply
dgossage
Posts: 4
Joined: Tue Jul 18, 2017 8:01 pm

Video Storage deletion issues

Post by dgossage » Thu Nov 30, 2017 3:05 pm

I have a system with 15 cameras.

It's running Centos7 and the storage is using zfs raidz6 4TB WD Red drives. @15TB useable space currently

I'm starting to run into the issue where I am writing faster than the system is cleaning up the images and directories.
I've even taken to writing a cleanup script to use perl to unlink files faster than the filter will with rm.

Curious if anyone else has come across the issue. ZFS does have a little more overhead when removing files so I have considered if I should just move back to xfs. I have also thought maybe I should split zoneminder between multiple servers spreading the storage out.

I'm looking at adding more cameras in future so trying to plan out a sustainable model going forward.

marcopete87
Posts: 40
Joined: Wed Dec 04, 2013 4:53 pm

Re: Video Storage deletion issues

Post by marcopete87 » Fri Dec 08, 2017 8:21 am

dgossage wrote:
Thu Nov 30, 2017 3:05 pm
I have a system with 15 cameras.

It's running Centos7 and the storage is using zfs raidz6 4TB WD Red drives. @15TB useable space currently

I'm starting to run into the issue where I am writing faster than the system is cleaning up the images and directories.
I've even taken to writing a cleanup script to use perl to unlink files faster than the filter will with rm.

Curious if anyone else has come across the issue. ZFS does have a little more overhead when removing files so I have considered if I should just move back to xfs. I have also thought maybe I should split zoneminder between multiple servers spreading the storage out.

I'm looking at adding more cameras in future so trying to plan out a sustainable model going forward.
I still wonder which kind of issue do you have.

dgossage
Posts: 4
Joined: Tue Jul 18, 2017 8:01 pm

Re: Video Storage deletion issues

Post by dgossage » Fri Dec 08, 2017 2:23 pm

Mainly that the system can't delete the older files faster than the new ones are written.

Curious if anyone else has systems with that many cameras running to one storage system with any similar problems.

User avatar
iconnor
Posts: 532
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Video Storage deletion issues

Post by iconnor » Fri Dec 08, 2017 3:59 pm

It gets better with h264 storage, as you are only deleting 1 or 2 files instead of thousands.

dgossage
Posts: 4
Joined: Tue Jul 18, 2017 8:01 pm

Re: Video Storage deletion issues

Post by dgossage » Fri Dec 08, 2017 4:14 pm

That's coming in 1.31 if i recall?

marcopete87
Posts: 40
Joined: Wed Dec 04, 2013 4:53 pm

Re: Video Storage deletion issues

Post by marcopete87 » Fri Dec 08, 2017 5:57 pm

iconnor wrote:
Fri Dec 08, 2017 3:59 pm
It gets better with h264 storage, as you are only deleting 1 or 2 files instead of thousands.
Iconnor, as you told me some time ago, raid5 doesn't boost performance on write; does raid6 do the same?

dgossage, how many seconds last an event? I had same issue (with buffer overflow due disk slowness), mitigated by setting 60 seconds events.

dgossage
Posts: 4
Joined: Tue Jul 18, 2017 8:01 pm

Re: Video Storage deletion issues

Post by dgossage » Fri Dec 08, 2017 6:51 pm

I have it set to default I think so they are in 10 minute chunks. So yes I could likely change it to 1 minute a piece which would give me more directories but with fewer images in each. So hopefully each rm command would return faster.

I doubt raid6 is great for performance but with 4TB drives rebuilding one when a drive dies takes a bit of time so I like the extra parity.

User avatar
iconnor
Posts: 532
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Video Storage deletion issues

Post by iconnor » Fri Dec 08, 2017 10:59 pm

raid6 is definitely a better thing than raid5, but offers no write performance improvements.

What would be nice would be a raid with a lazy redundancy. Because I wouldn't really care if I lost a file or two, As long as most were there.

I think zfs and btrfs have some more interesting raid options, but I havn't had time to look into them.

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests