HOSAFE 1MB1W - Keeps rebooting on 1.32 and Ubuntu 18.04

[maturola]

Hi,

Been using zoneminder for a while and besides a few random issues here and there the system had been pretty stable. I just recently got me a new poweredge R620 for a home lab (ESXI) and I re-installed Zoneminder (1.32.3) on a Ubuntu server 18.04.1 and now my 2 HOSAFE 1MB1W keep resetting. I had tried many settings and no luck, they both reboot every 5 mins or so.

I still have the old server (1.30.4) and this doesn't happen there. (Before I go back to 1.30 or anything else drastic like keeping zoneminder in the old server I would like to see if anyone have any ideas I could try.

Here is some information about my systems

Original server
FREENAS 11.1-U6 running on a Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz.
Zoneminder is installed on a Freenas VM (Bhyve) running ubuntu 16
VM specs: 4 Virtual CPUs, 24GB RAM
Zoneminder 1.30.4
I have total of 6 monitors, only 3 of then are Modect (2x HOSAFE 1MB1W, 3x Wansview W3 , 1x Wansview K2 -B)

New Server
Dell Poweredge R620, Running ESXI 6.5.0 Update 2, 12 CPUs x Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz and 64GB RAM
Zoneminder is installed on a VM running ubuntu 18.04.1
VM specs: 4 Virtual CPUs, 16GB RAM
Zoneminder 1.32.3
I setup the same monitors as the old server using the same settings.

ESXI Monitor show that the most memory it had consumed is 9GB, average is 6GB

Camera settings (same exact setting On both severs)
Name Front Test
Server None
Source Type FFmpeg
Function Modect
Enabled Check
Linked Monitors none
Analysis FPS 15.00
Maximum FPS (?) 15.00
Alarm Maximum FPS (?) 15.00
(I set max fps to 10 on the camera, I also try with these FPS field empty and it make no difference, they still reset)

Source Path rtsp://admin:admin@192.168.3.5
Method (?) TCP
Options (?) allowed_media_types=video
Target colorspace 32Bit
Capture Width (pixels) 1280
Capture Height (pixels) 720

Any ides are welcome, would love to have the new version and the new server up and running and not have to default to the NAS server.

Thanks

[bbunge]

As always check the free space in tmpfs. You had 24 gig in the old vm why only 16 in the new? You might also switch the 1.32.3 to save jpg and not video to see if that helps.

[maturola]

As always check the free space in tmpfs. You had 24 gig in the old vm why only 16 in the new? You might also switch the 1.32.3 to save jpg and not video to see if that helps.

Thank you sir, will do. Can you elaborate on checking the tmpfs? (i'll search the forum and see if I find a reference)

I have 16, just to start, I can easily change it to more, but I am keeping an eye on the ESXI monitor and that VM have no consumed more than 9GB at any giving time.

I'll try 1.32.3 and see if I see any difference.

[maturola]

No luck with 1.32.3 on jpeg, same issue.

Here are some of the errors that happens right before the camera reboots

Code: Select all

web_php		8288	ERR	Socket /var/run/zm/zms-236730s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information.	/usr/share/zoneminder/www/includes/functions.php	2048
2018-12-28 12:04:19	web_js		16390	ERR	getStreamCmdResponse stream error: Socket /var/run/zm/zms-236730s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information. - checkStreamForErrors()	?view=watch	
2018-12-28 12:04:18	zms_m3		23354	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	538
2018-12-28 12:04:18	zms_m3		23354	ERR	Unable to connect to monitor id 3 for streaming	zm_stream.cpp	48
2018-12-28 12:04:18	zms_m3		23354	ERR	Unable to connect to zmc process for monitor 3	zms.cpp	254
2018-12-28 12:04:18	web_php		8288	ERR	Socket /var/run/zm/zms-616489s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information.	/usr/share/zoneminder/www/includes/functions.php	2048
2018-12-28 12:04:18	web_js		8288	ERR	getStreamCmdResponse stream error: Socket /var/run/zm/zms-616489s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information. - checkStreamForErrors()	?view=watch	
2018-12-28 12:04:17	zmwatch		3321	ERR	Memory map file '/dev/shm/zm.mmap.3' should have been 1160 but was instead 0	zmwatch.pl	
2018-12-28 12:04:17	web_js		1682	ERR	getStreamCmdResponse stream error: Socket /var/run/zm/zms-285459s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information. - checkStreamForErrors()	?view=watch	
2018-12-28 12:04:17	web_php		1682	ERR	Socket /var/run/zm/zms-285459s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/latest/faq.html#why-can-t-i-see-streamed-images-when-i-can-see-stills-in-the-zone-window-etc for more information.

Code: Select all

2018-12-28 11:53:48	zmwatch		3321	ERR	Memory map file '/dev/shm/zm.mmap.5' should have been 1160 but was instead 0	zmwatch.pl	
2018-12-28 11:53:45	zmdc		2123	ERR	'zma -m 5' exited abnormally, exit status 255	zmdc.pl	
2018-12-28 11:53:45	zma_m5		27453	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	538
2018-12-28 11:53:40	zmdc		2123	ERR	'zma -m 5' exited abnormally, exit status 255	zmdc.pl	
2018-12-28 11:53:39	zma_m5		26859	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	538
2018-12-28 11:53:38	zma_m5		26734	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	538
2018-12-28 11:53:38	zmdc		2123	ERR	'zma -m 5' exited abnormally, exit status 255	zmdc.pl	
2018-12-28 11:53:37	zmwatch		3321	ERR	Memory map file '/dev/shm/zm.mmap.5' does not exist. zmc might not be running.	zmwatch.pl	
2018-12-28 11:53:27	zmc_m5		14406	ERR	Failed to prime capture of initial monitor	zmc.cpp	253
2018-12-28 11:53:27	zmc_m5		14406	ERR	Unable to open input rtsp://admin@192.168.3.7/live1.264 due to: Interrupted system call	zm_ffmpeg_camera.cpp	357
2018-12-28 11:53:13	zmc_m5		14406	ERR	Unable to open input rtsp://admin@192.168.3.7/live1.264 due to: Operation now in progress	zm_ffmpeg_camera.cpp	357
2018-12-28 11:53:13	zmc_m5		14406	ERR	Failed to prime capture of initial monitor	zmc.cpp	253
2018-12-28 11:52:14	zmwatch		3321	ERR	Memory map file '/dev/shm/zm.mmap.3' should have been 1160 but was instead 0	zmwatch.pl	
2018-12-28 11:52:10	zma_m3		18054	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	538
2018-12-28 11:52:10	zmdc		2123	ERR	'zma -m 3' exited abnormally, exit status 255	zmdc.pl	
2018-12-28 11:52:05	zma_m3		17543	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	538
2018-12-28 11:52:05	zmdc		2123	ERR	'zma -m 3' exited abnormally, exit status 255	zmdc.pl	
2018-12-28 11:52:04	zma_m3		17392	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	538
2018-12-28 11:52:04	zmdc		2123	ERR	'zma -m 3' exited abnormally, exit status 255	zmdc.pl	
2018-12-28 11:52:03	zmwatch		3321	ERR	Memory map file '/dev/shm/zm.mmap.3' does not exist. zmc might not be running.	zmwatch.pl	
2018-12-28 11:51:53	zmc_m3		16077	ERR	Unable to open input rtsp://admin@192.168.3.5/live1.264 due to: Operation now in progress	zm_ffmpeg_camera.cpp	357
2018-12-28 11:51:53	zmc_m3		16077	ERR	Failed to prime capture of initial monitor	zmc.cpp	253
2018-12-28 11:51:52	zms_m3		16071	ERR	Got empty memory map file size 0, is the zmc process for this monitor running?	zm_monitor.cpp	538
2018-12-28 11:51:52	zms_m3		16071	ERR	Unable to connect to monitor id 3 for streaming	zm_stream.cpp	48
2018-12-28 11:51:52	zms_m3		16071	ERR	Unable to connect to zmc process for monitor 3	zms.cpp	254
2018-12-28 11:51:51	zmwatch		3321	ERR	Memory map file '/dev/shm/zm.mmap.3' does not exist. zmc might not be running.	zmwatch.pl	
2018-12-28 11:51:41	zmc_m3		9657	ERR	Unable to open input rtsp://admin@192.168.3.5/live1.264 due to: Interrupted system call	zm_ffmpeg_camera.cpp	357
2018-12-28 11:51:41	zmc_m3		9657	ERR	Failed to prime capture of initial monitor	zmc.cpp	253
2018-12-28 11:51:27	zmc_m3		9657	ERR	Unable to open input rtsp://admin@192.168.3.5/live1.264 due to: Operation now in progress

I don't think memory is the issue, tmpfs is at 4% (I increased it too 70% allocation)

Code: Select all

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            9.8G     0  9.8G   0% /dev
tmpfs           2.0G  1.2M  2.0G   1% /run
/dev/sda2        59G   12G   45G  21% /
tmpfs            14G  422M   14G   4% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           9.9G     0  9.9G   0% /sys/fs/cgroup
/dev/loop0      184M  184M     0 100% /snap/nextcloud/10314
/dev/loop1       90M   90M     0 100% /snap/core/6130
/dev/loop2       87M   87M     0 100% /snap/core/4917
tmpfs           2.0G     0  2.0G   0% /run/user/1000

[maturola]

I'm trying to check if PATH_ZMS is correct, but I not longer see PATHS on the Options menu (on 1.32.4), am i missing something?

[rockedge]

the paths are now in a config file located in /etc/zm/conf.d (on Ubuntu variants)
you can modify it by placing a zmcustom.conf file in /etc/zm/conf.d if you need to change the /cgi-bin path ...check to make sure the nph-zms and zms are in the correct /cgi-bin and you have that path.

here is an example of one of mine:

/etc/zm/conf.d/zmcustom.conf

Code: Select all

# These values were autogenerated by zmupdate.pl
# You may edit these values. ZoneMinder will not overwrite them.
#
ZM_WEB_USER=webuser
ZM_WEB_GROUP=webgroup
ZM_PATH_ZMS=/cgi-bin/nph-zms

[maturola]

the paths are now in a config file located in /etc/zm/conf.d (on Ubuntu variants)
you can modify it by placing a zmcustom.conf file in /etc/zm/conf.d if you need to change the /cgi-bin path ...check to make sure the nph-zms and zms are in the correct /cgi-bin and you have that path.

here is an example of one of mine:

/etc/zm/conf.d/zmcustom.conf

Code: Select all

# These values were autogenerated by zmupdate.pl
# You may edit these values. ZoneMinder will not overwrite them.
#
ZM_WEB_USER=webuser
ZM_WEB_GROUP=webgroup
ZM_PATH_ZMS=/cgi-bin/nph-zms

Thank you, Checked and path seems correct

Code: Select all

# ZoneMinder url path to the zms streaming server
ZM_PATH_ZMS=/zm/cgi-bin/nph-zms

I've tried everything I can come up with, I think I am going back to 1.30 and stick to that one.

[rockedge]

did you check how Apache has the /cgi-bin aliased? if that path matches the actual location of nph-zms
what happens if you try this URL ->

Code: Select all

http://yourZMserver/zm/cgi-bin/nph-zms

I have slightly changed from the default location and set the alias to this (config for Hiawatha 10.8.2)

Code: Select all

Alias = /cgi-bin:/usr/local/libexec/zoneminder/cgi-bin

from the iconner ppa for ubuntu the alias looks like->

Code: Select all

Alias =/zm/cgi-bin:/usr/lib/zoneminder/cgi-bin

[maturola]

did you check how Apache has the /cgi-bin aliased? if that path matches the actual location of nph-zms
what happens if you try this URL ->

Code: Select all

http://yourZMserver/zm/cgi-bin/nph-zms

I have slightly changed from the default location and set the alias to this (config for Hiawatha 10.8.2)

Code: Select all

Alias = /cgi-bin:/usr/local/libexec/zoneminder/cgi-bin

Where do i set the Apache Aliased?

When i go to that URL I get nothing, empty page.

wouldn't that affect all cameras if it is set wrong?

[rockedge]

wouldn't that affect all cameras if it is set wrong?

yes it would

[rockedge]

do you have enough memory in /dev/shm? perhaps it is a memory problem what happens if you turn off other cameras?

the Apache configs are /etc/apache2 look at /etc/apache2/conf-avialable and /etc/apache2/conf-enabled
but it doesn't sound like that is the problem

[maturola]

do you have enough memory in /dev/shm? perhaps it is a memory problem what happens if you turn off other cameras?

the Apache configs are /etc/apache2 look at /etc/apache2/conf-avialable and /etc/apache2/conf-enabled
but it doesn't sound like that is the problem

well, all other 4 cameras are working just fine, I think it may be related to the Cam configuration. However they work fine under 1.30.4. OR maybe something related to Ubuntu 18.04, the older server is running Ubuntu 16.04 ( I may spin a VM with ubuntu 16.04 and install 1.32.3 and see if the issue persist).

Memory seem ok, /dev/shm show 4%

Code: Select all

Filesystem      Size  Used Avail Use% Mounted on
udev            9.8G     0  9.8G   0% /dev
tmpfs           2.0G  1.2M  2.0G   1% /run
/dev/sda2        59G   12G   44G  22% /
tmpfs            14G  422M   14G   4% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           9.9G     0  9.9G   0% /sys/fs/cgroup
/dev/loop0      184M  184M     0 100% /snap/nextcloud/10314
/dev/loop1       90M   90M     0 100% /snap/core/6130
/dev/loop2       87M   87M     0 100% /snap/core/4917
tmpfs           2.0G     0  2.0G   0% /run/user/1000

This is what i have under .../conf-enable/zoneminder.conf

Code: Select all

                                             
# Remember to enable cgi mod (i.e. "a2enmod cgi").
ScriptAlias /zm/cgi-bin "/usr/lib/zoneminder/cgi-bin"
<Directory "/usr/lib/zoneminder/cgi-bin">
    Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
    AllowOverride All
    Require all granted
</Directory>

# Order matters. This alias must come first.
Alias /zm/cache /var/cache/zoneminder/cache
<Directory /var/cache/zoneminder/cache>
    Options -Indexes +FollowSymLinks

How about this error:

Code: Select all

Memory map file '/dev/shm/zm.mmap.5' should have been 1160 but was instead 0

Any clues here?

I turned off all others cameras, no change, these 2 keep rebooting every few minutes. =(

[maturola]

I spun a new VM with Ubuntu 16.04 Server and installed fresh Zonmeminde 1.32.3.

Both 1MB1W are currently streaming with no issues (No reboots reported yet). I have not added the other 4 cameras yet. I will do that and report back.

These errors are showing once in a while but streaming is solid.

Code: Select all

2018-12-28 19:06:42	web_php		20117	ERR	Timed out waiting for msg /var/run/zm/zms-235193s.sock	/usr/share/zoneminder/www/ajax/stream.php	90
2018-12-28 19:06:42	web_php		20117	ERR	No data to read from socket	/usr/share/zoneminder/www/includes/functions.php	2048
2018-12-28 19:06:42	web_php		20116	ERR	Timed out waiting for msg /var/run/zm/zms-517965s.sock	/usr/share/zoneminder/www/ajax/stream.php	90
2018-12-28 19:06:42	web_php		20116	ERR	No data to read from socket	/usr/share/zoneminder/www/includes/functions.php	2048

UPDATE 2

Well, once I added the other monitors, they started to reset. They don't reset as often, but about every 10 to 15mins (instead of every 3 to 5 mins)

I also increased the memory to 32GB and 6 CPUs just to eliminate the possibility of too little memory or processing power.