S3 storage

Discussions related to the 1.36.x series of ZoneMinder
Post Reply
User avatar
Andyrh
Posts: 125
Joined: Sat Oct 28, 2017 3:55 am

S3 storage

Post by Andyrh »

I was curious is anyone is using S3 storage?
I was thinking about dropping an S3 server at a friends house to store the last week of events.

Just looking for a bit of advice, do's, dont's, that sort of thing.

The bandwidth will not be an issue. I generate ~8GB of events per day.
Andy
o||||o

Ubuntu 20.04
ZM 1.36.14
i7-2700K @ 3.5GHz
16 GB RAM
6 cameras -> 54 FPS modect
Magic919
Posts: 866
Joined: Wed Sep 18, 2013 6:56 am

Re: S3 storage

Post by Magic919 »

I don't use it for this purpose, but I'd suggest looking at Rclone and run a sync from time to time.
-----------------------------------------------------------------------
I'm nothing to do with ZM, ZMNinja or ZMES, I just use them.
Contribute funding to the devs here - https://opencollective.com/zoneminder
User avatar
iconnor
Posts: 1917
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: S3 storage

Post by iconnor »

It works. I recommend recording initially to local storage and use a filter to move events to s3 in the background. Latency is the problem when recording.
User avatar
Andyrh
Posts: 125
Joined: Sat Oct 28, 2017 3:55 am

Re: S3 storage

Post by Andyrh »

I was thinking of filters to manage the copies, all events will first be written to local disks. Will a filter do well deleting old events from the S3 copy?

As weekend time is available I will post progress. 1st attempt will be to a Ubuntu VM as an S3 server. If that goes well 2nd attempt will be to a Raspberry Pi running Ubuntu. Final step will be the Pi remote.

Things I have to learn:
1. What happens if the S3 server goes away.
2. What happens if I fall behind.
3. Best way to remove old events from S3.
4. Things I don't even know to ask yet.
Andy
o||||o

Ubuntu 20.04
ZM 1.36.14
i7-2700K @ 3.5GHz
16 GB RAM
6 cameras -> 54 FPS modect
dougmccrary
Posts: 548
Joined: Sat Aug 31, 2019 7:35 am

Re: S3 storage

Post by dougmccrary »

Magic suggested Rclone, or rsync would take care of deletions with the right options. It'll basically keep them in sync if you run it periodically.
User avatar
Andyrh
Posts: 125
Joined: Sat Oct 28, 2017 3:55 am

Re: S3 storage

Post by Andyrh »

How would those tools perform having to scan the events folders (a ridiculous 540k events) to find the events since the last sync and delete the remote copies that are more than a week old? I am not looking to sync all 6TB, just the last 7 days. The remote system will be a Pi with limited storage.
I generate ~1,100 events per day.
I assume that if I use filters to copy the events ZM will be able to play the event from the remote site. With rsync that would not be possible because ZM would be unaware of the copy.

I will add those tools to the list of things to try.
Andy
o||||o

Ubuntu 20.04
ZM 1.36.14
i7-2700K @ 3.5GHz
16 GB RAM
6 cameras -> 54 FPS modect
Magic919
Posts: 866
Joined: Wed Sep 18, 2013 6:56 am

Re: S3 storage

Post by Magic919 »

I suppose you'll find out. I can't see playing the remote S3 stuff in ZM going well, but I've never tried it.
-----------------------------------------------------------------------
I'm nothing to do with ZM, ZMNinja or ZMES, I just use them.
Contribute funding to the devs here - https://opencollective.com/zoneminder
User avatar
iconnor
Posts: 1917
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: S3 storage

Post by iconnor »

I think backup case (in case thieves steal your nvr) is valid. Whether it works or not has a lot to do with your connection speed etc.

At worst it will be a little laggy. At best it will be fine. Was worse back when savign a billion jpegs. With mp4 storage it is much better.

In practice we found deleting from ZM the S3 storage couldn't keep up. Most want to delete by age every night. YMMV.

If anyone is looking to play with this Bifrost Cloud (bifrostcloud.ca) is offering a commission to ZM users and are cheaper than Amazon.
Post Reply