[HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster

Add any particular hints or tricks you have found to help with your ZoneMinder experience.
Post Reply
vox
Posts: 43
Joined: Tue Aug 30, 2016 12:51 pm

[HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster

Post by vox » Wed Nov 15, 2017 1:58 pm

Music for brain storming : Best Music Mix 2017 - Best of EDM Remixes Of Popular Songs 2017
This tutorial coming from [Tuto/HowTo] [GNU/Linux] [EN] Multi Server Zoneminder - fully secure and scalable ZM cluster
This tuto cover ZM 1.30.4 version on Ubuntu-server 16.04.

Warning : the procedure is working, but the tuto need to be test and corrected

Introduction
  • This system require minimum 3 machines with Quadcore x64 processor (min i3). Not working with ARM.
    This tuto is write for Ubuntu but, normally, it's working too on Debian.
    In this thread i explain to you how to creat :
    1. a Database cluster (mariadb + galera)
    2. a FileSystem cluster (unified disk space with btrfs JBOD on each machine and unified server space over network with glusterfs)
    3. a ZoneMinder cluster (multi machine)
    To facilitate the setting up we config each server to cumulate all the jobs. But it is not obligatory that every machine hosts all the services.
Last edited by vox on Wed Nov 15, 2017 3:59 pm, edited 2 times in total.

vox
Posts: 43
Joined: Tue Aug 30, 2016 12:51 pm

Re: [HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster

Post by vox » Wed Nov 15, 2017 2:05 pm

Install and configure the DataBase Cluster and ZoneMinder
  1. For secure communication between the node of the DataBase cluster, on one machine install a VPN (example OpenVPN or WireGuard) and connect all your Database Servers/Clients on that VPN. With this method you increase security without so much boring manipulations.
  2. Connect to admin user (root)
  3. Add key and repository for the last software version
    • On Ubuntu 16.04
      • Code: Select all

        apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
        add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mariadb.cu.be/repo/10.2/ubuntu xenial main'
        add-apt-repository -y ppa:ondrej/apache2
        add-apt-repository ppa:iconnor/zoneminder
        

      On Debian 9 (stretch)
      • Code: Select all

        sudo apt-get install software-properties-common dirmngr
        apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xF1656F24C74CD1D8
        add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mariadb.mirror.nucleus.be/repo/10.2/debian stretch main'
        echo "deb http://www.deb-multimedia.org stretch main non-free" >> /etc/apt/sources.list
        
      For other distro, check this link :)
  4. Install all softwares
    • Code: Select all

      apt-key update
      apt-get update
      apt-get install -y btrfs-tools glusterfs-server attr glusterfs-client mariadb-server galera-3 mariadb-client zoneminder php-gd php-mysql apache2
      btrfs-tools glusterfs-server attr => filesystem cluster softwares
      glusterfs-client => filesystem client software
      mariadb-server galera-3 => database cluster softwares
      mariadb-client => database client software
      zoneminder php-gd php-mysql apache2 => ZoneMinder and friends
  5. Lunch following commands
    • Code: Select all

      adduser www-data video
      chmod 740 /etc/zm/zm.conf
      chown -R www-data:www-data /usr/share/zoneminder/
      sudo ln -s /etc/apparmor.d/usr /etc/apparmor.d/disable/.sbin.mysqld
      mkdir /usr/share/zoneminder/www/sounds
      chown www-data:www-data -R /usr/share/zoneminder/www/sounds
      chmod 774 -R /usr/share/zoneminder/www/sounds
      a2enmod cgi
      a2enmod rewrite
      a2enmod http2
      sed -ie "/AllowOverride None/c\AllowOverride All" /etc/apache2/apache2.conf
      a2enconf zoneminder
      a2dismod status # enforce security (disable apache2 server status)
      echo "ServerTokens Prod" >> /etc/apache2/apache2.conf # enforce security (disable apache2's serverTokens)
      echo "ServerSignature Off" >> /etc/apache2/apache2.conf # enforce security (disable apache2's ServerSignature)
      sudo echo "User-agent: *" > /usr/share/zoneminder/www/robot.txt; sudo echo "Disallow: /" >> /usr/share/zoneminder/www/robot.txt; sudo chown www-data:root /usr/share/zoneminder/www/robot.txt # disable google indexing
      
      systemctl enable zoneminder
      
  6. Add you timezone in /etc/php/7.0/apache2/php.ini (CTRL+w for search), remove the ; before the line. (for europe past "Europe/Brussels", for USA "America/New_York"
  7. Edit mariadb the configuration's file /etc/mysql/conf.d/galera.cnf
    • Code: Select all

      nano /etc/mysql/conf.d/galera.cnf
      Add the following line after have adapt it (ip, hostname) to your config
      • Code: Select all

        [mysqld]
        #mysql settings
        sql_mode = NO_ENGINE_SUBSTITUTION
        binlog_format=ROW
        default-storage-engine=innodb
        innodb_autoinc_lock_mode=2
        query_cache_size=0
        query_cache_type=0
        innodb_flush_log_at_trx_commit=0
        innodb_buffer_pool_size=256M
        bind-address=10.8.1.42 # EDIT ME
        
        #Galera settings
        wsrep_provider=/usr/lib/libgalera_smm.so  # /usr/lib/libgalera_smm.so OR /usr/lib/galera/libgalera_smm.so
        wsrep_cluster_name="DataBaseCluster" # this line need be same on all machine !
        wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3" # EDIT ME
        #wsrep_cluster_address="gcomm://" # uncomment this line of one machine for first start or when your cluster is break
        wsrep_sst_method=rsync
        wsrep_on=ON
        wsrep_node_address="hostnameServer1"  # EDIT ME
        wsrep_node_name="hostnameServer1" # EDIT ME
        
        bind-address=10.8.1.42 => add the STATIC IP of this server here
        wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3" => add the hostname of all your machine. These hostname need be config in your DNS or in your /etc/hosts files
        wsrep_node_address="hostnameServer1" => add the hostname (or the static IP address), don't forget to edit your /etc/hosts for add the same address than the bind-address parameter for the hostname of this server
        wsrep_node_name="hostnameServer1" => add the hostname of this machine
  8. On the "first" node of your cluster, stop mariadb
  9. On this node edit the file/etc/mysql/conf.d/galera.cnf
    • Code: Select all

      nano /etc/mysql/conf.d/galera.cnf
      Comment the line with the list of your nodes "wsrep_cluster_address" and add the following line.
  10. Creat the cluster
  11. Restart mariadb on all of your nodes
  12. Check if it's work
    • Code: Select all

      mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
      • screenshot-2017_08_22-Commade_Galera_Cluster_mariadb_Status_Cluster.png
        screenshot-2017_08_22-Commade_Galera_Cluster_mariadb_Status_Cluster.png (5.88 KiB) Viewed 2556 times
  13. If it's work, edit on your "first" node the file /etc/mysql/conf.d/galera.cnf for remove "wsrep_cluster_address="gcomm://" and uncomment your node list.
    • Code: Select all

      wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3"
  14. Edit zoneminder main configuration's file /etc/zm/zm.conf and adapt the following paramaters
    • ZM_DB_HOST=10.8.0.42 # need be same than wsrep_node_address, you can use hostname (with /etc/hosts correspondence) or Static IP address
      ZM_DB_PASS="your_Super_Strong_DataBase_Password"
      ZM_SERVER_HOST= choose an unique name for this server, need be same than you add in ZM Webui in Options -> Servers
  15. Reload main services
    • Code: Select all

      service apparmor restart
      mysqladmin -uroot -p reload
      service mysql restart
      service apache2 restart
  16. Creat your Database user
    • Code: Select all

      mysql -uroot -p -e "grant all on zm.* to 'zmuser'@%.%.%.% identified by 'zmpass';"
      Replace zmpass by a very strong password.
  17. Initialize your database
    • Code: Select all

      mysql -uroot -p < /usr/share/zoneminder/db/zm_create.sql
  18. Start ZoneMinder and check if it's working.
Last edited by vox on Wed Nov 15, 2017 2:28 pm, edited 3 times in total.

vox
Posts: 43
Joined: Tue Aug 30, 2016 12:51 pm

Re: [HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster

Post by vox » Wed Nov 15, 2017 2:08 pm

Unify memory
Explanation :
  • First we unified disk on each server via btrfs JBOD without striping. (in this tuto /media/superPartition)
    After we use glusterfs for creat a virtual JBOD without stripping over all your servers. (in this tuto /media/ZoneMinder)

On all your Data server
  1. List your disk
    • Code: Select all

      fdisk -l
      Recover the path for each disk we want use for camera's data (example /dev/sdb)
  2. Creat the btrfs partition on one or more disk (you can add more after if need)
    • Code: Select all

      mkfs.btrfs -d single /dev/sdb /dev/sdc
      screenshot-2017_08_11-Tuto-BTRFS-Creation_JBOD_01.png
      screenshot-2017_08_11-Tuto-BTRFS-Creation_JBOD_01.png (65.15 KiB) Viewed 2556 times
      If after you wan add one disk, use the following command
      • Code: Select all

        btrfs device add /dev/sdc /media/superPartition/
        Where /dev/sdc is the path of your new disk. The partition need already be mount.
  3. Creat the mount point
    • Code: Select all

      mkdir /media/superPartition
      mkdir /media/ZoneMinder
  4. Mount the btrfs partition
    • Code: Select all

      mount /dev/sdb /media/superPartition
  5. Recover the UID of one of your disk in your btrfs partition
    • Code: Select all

      blkid | grep "btrfs" | grep -E " UUID=\"([0-9a-z]{1,}[\-]{0,}){0,}\""
  6. Edit /etc/fstab
  7. Adapt the UUID and add the following lines
    • Code: Select all

      UUID=blablabla-uid-de-votre-disque-blablabla /media/superPartition           btrfs   defaults,rw,noexec      0  0
      127.0.0.1:/ZoneMinder /media/ZoneMinder glusterfs defaults,_netdev,backupvolfile-server=hostnameServer2,noexec 0 0

On one lonely Zoneminder server
  1. Add all your servers (peers) in your glusterfs cluster (repeat the command for each hostname/server)
    • Code: Select all

      gluster peer probe hostnameServer2
  2. Creat your JBOD over all your servers
    • Code: Select all

      gluster volume create ZoneMinder transport tcp hostnameServer1:/media/superPartition hostnameServer2:/media/superPartition hostnameServer3:/media/superPartition force
  3. Start your new network JBOD
  4. Mount your volume
    • Code: Select all

      mount -t glusterfs 127.0.0.1:/ZoneMinder /media/ZoneMinder
  5. Move the old zoneminder's data folder to the new destination and give it the good permissions.
    • Code: Select all

      mv  /var/cache/zoneminder /var/cache/zoneminderOLD
      cp  -R -p -v /var/cache/zoneminderOLD/* /media/ZoneMinder
      chown www-data:root -R /media/ZoneMinder/*
      chmod 770 -R /media/ZoneMinder/*
      
On each server now you can mount the network JBOD via the next command line, or you can directly reboot. On all your ZM servers, add a symbolic link (or modify the Path of "events", "images" and "sound" in the WEBUI of Zneminder in "Options" => "Path", it's same result)
  • Code: Select all

    ln -s /media/ZoneMinder /var/cache/zoneminder
By default glusterfs don't encrypt communications between server-server and between client-server, for encrypt communications follow this tuto : [Tuto/HowTo] [GNU/Linux] Gluster and TLS (SSL). This is not required if your communications are going through VPN tunnels.

User avatar
iconnor
Posts: 633
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: [HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster

Post by iconnor » Wed Nov 15, 2017 3:29 pm

Nice tutorial.

Just a couple of points:
1. I found glusterfs to be impractical for ZM use, due to our use of millions of small jpegs. Storing h264 video should work better here though going forward.
2. My dev branch allows multiple storage areas, so removes the need for a unified storage.
3. This all works fine on arm too.

vox
Posts: 43
Joined: Tue Aug 30, 2016 12:51 pm

Re: [HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster

Post by vox » Wed Nov 15, 2017 3:42 pm

Thanks :) I tried to reduce the size to the maximum but hard hard ^^
iconnor wrote:
Wed Nov 15, 2017 3:29 pm
1. I found glusterfs to be impractical for ZM use, due to our use of millions of small jpegs. Storing h264 video should work better here though going forward.
I have try on Localhost (on SSD) it's working fine. But i m not a glusterfs fan (i hate his bugs (example)), if another one filesystem flashes me, the tutorial will be edited.
Of course, in cluster world glusterfs has a bad reputation with little files but, actually, glusterfs is the simplest I've tested. (but well for french speaker, test all cluster filesystem take amount of time with the doc that is never translated :P )
iconnor wrote:
Wed Nov 15, 2017 3:29 pm
2. My dev branch allows multiple storage areas, so removes the need for a unified storage.
We can't actually on stable branch (1.30.4) no? (i don't have read any parameter for that)
iconnor wrote:
Wed Nov 15, 2017 3:29 pm
3. This all works fine on arm too.
Zm work on ARM (but lag, ARM CPU aren't fastly (screenshot on Ordroid-XU4 (8 core))).
MariaDB cluster not work on ARM (i have try, the galera dependencies are not satisfied). It's not working too with MySQL Cluster on ARM (i have try too and report the problem on mysql dev forum).

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest