SOLVED: Ubuntu 18.04 + nginx cgi issue - no streaming

[AfricanSwallow]

Hi All,

I know I'm beating a dead horse here, but even with all of the resources online about this exact issue, none of them have helped. I'm hoping to get a bit of assistance here because i am soooo close. So i have the typical no streaming issue, though stills exist as expected. I get the typical socket doesn't exsit error:
2018/06/23 13:50:18 [error] 6124#6124: *8 connect() to unix:/var/run/fcgiwrap.socket failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: host.duckdns.org, request: "GET /zm/cgi-bin/zms%20?mode=jpeg&scale=100&maxfps=5&buffer=1000&monitor=1&auth=b8b406fd18035c7181695ba1cc796c8e&connkey=679563&rand=1529779817 HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "host.duckdns.org", referrer: "http://host.duckdns.org/zm/index.php?view=watch&mid=1"
2018/06/23 13:50:21 [error] 6124#6124: *8 FastCGI sent in stderr: "PHP message: ERR [Socket /var/run/zm/zms-679563s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information.]" while reading response header from upstream, client: 127.0.0.1, server: host.duckdns.org, request: "POST /zm/index.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "host.duckdns.org", referrer: "http://host.duckdns.org/zm/index.php?view=watch&mid=1"

I have ubuntu 18.04 and ZoneMinder 1.30.4 per isaac connor's ppa. Im using the nginx config provided in the zoneminder wiki with no variances. I updated the cgi_path to include /zm/, and even followed a few other forums to modify group of /var/run/fcgiwrap.socket.

After spending about a week trying to solve this one particular issue, Im finally seeking psychiatric and forum help.

Thank you.

cat /etc/nginx/host.duckdns.conf
location /zm/cgi-bin {
gzip off;
alias /usr/lib/zoneminder/cgi-bin;

include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_FILENAME $request_filename;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}

location /zm {
# if ($scheme ~ ^http:){
# rewrite ^(.*)$ https://$host$1 permanent;
# }

gzip off;
alias /usr/share/zoneminder/www;
index index.php;

location ~ \.php$ {
if (!-f $request_filename) { return 404; }
expires epoch;
include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_FILENAME $request_filename;
fastcgi_index index.php;
fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
}


location ~ \.(jpg|jpeg|gif|png|ico)$ {
access_log off;
expires 33d;
}

location /zm/api/ {
alias /usr/share/zoneminder/www/api;
rewrite ^/zm/api(.+)$ /zm/api/index.php?p=$1 last;
}
}

[bbunge]

For Bionic you need to use the patched version of the ZM 1.30.4 as referenced in the WIKI instructions:https://wiki.zoneminder.com/Ubuntu_Serv ... .2C_PHP.29

I will run the install on my test machine to see if anything has changed.

bb

[bbunge]

Ran the install for Bionic with NGINX and everything works!

https://wiki.zoneminder.com/Ubuntu_Serv ... .2C_PHP.29

in the "server" section after listen [::] :80 default_Server; add

include /etc/nginx/zoneminder.conf;

the line should look like: listen [::] :80 default_Server;include /etc/nginx/zoneminder.conf;

[AfricanSwallow]

I've modified nginx to match that, though I dont know how that would help since it appears the config is loading OK. Anyway, went ahead and matched that exactly to rule it out.

I've also uninstalled and reinstalled zoneminder by doing the following:
sudo apt-get remove zoneminder
wget --no-check-certificate 'https://docs.google.com/uc?export=downl ... zwtOcCiisX' -O /tmp/zoneminder-1.30.4-bionic-amd64.deb
sudo apt-get install /tmp/zoneminder-1.30.4-bionic-amd64.deb
chown -R www-data:www-data /usr/share/zoneminder/

I then had a few more issues to take care of which included:
chown -R www-data:www-data /etc/zm/
chown -R www-data:www-data /var/cache/zoneminder/

I am now back to a mostly functioning stack though streaming still causes:
"2018/06/25 00:25:53 [error] 30261#30261: *449 FastCGI sent in stderr: "PHP message: ERR [Socket /var/run/zm/zms-624089s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information.]" while reading response header from upstream,"

Also, saw these such errors upon disabling the camera and setting its function to none:
Jun 25 00:31:46 AweSwa-Formula zmdc[30357]: WAR [Can't find process with command of 'zma -m 2']
Jun 25 00:31:46 AweSwa-Formula zmdc[30357]: INF ['zmc -m 2' sending stop to pid 31510 at 18/06/25 00:31:46]
Jun 25 00:31:46 AweSwa-Formula zmc_m2[31510]: INF [Got signal 15 (Terminated), exiting]
Jun 25 00:31:46 AweSwa-Formula zmdc[30357]: INF ['zmc -m 2' exited normally]
Jun 25 00:31:46 AweSwa-Formula zmdc[30357]: WAR [Can't find process with command of 'zmtrack.pl -m 2']
Jun 25 00:31:47 AweSwa-Formula zmdc[30357]: WAR [Can't find process with command of 'zma -m 2']

Overall, im back at the same issue, with no idea how to debug. For full disclosure, im also running pihole and nextcloud on this same machine, though I'm not sure how either of those would cause this issue.

thank you

[bbunge]

Have discovered that the NGINX procedure works until the server is rebooted. Then the cameras will not stream. Am in process of reloading Bionic with ZM and NGINX to see if I can figure out the issues. May take me a while as there are other "important" things in process...

[AfricanSwallow]

Well, it sure is a relief to know its not just me, i've tried all kinds of things. Let me know if there is anything i can do to help, any kind of debug logging to enable or anything like that.

thanks

[AfricanSwallow]

further, if you have any sneaking suspicions and can point me in a direction, i can attempt to further troubleshoot - just as it is i've hit a wall.

[AfricanSwallow]

Hate to simply bump the thread - but I'm really hoping to get this figured out. I am considering switching to apache, but I already have several services running under nginx and a proxy. Long story short, I'd rather spend that time debugging ZoneMinder than switching to apache until the issue is resolved.

Are there any guides or tips you can provide for me to enable some debugging with this? Do you believe it is an nginx/cgi issue or specific to ZoneMinder.

[knight-of-ni]

In a perfect world, I'd rather we all switched to nginx. Unfortunately, I don't use it as much as I want to, though.

Looking at the screenshot shown in an earlier post, PATH_ZMS is not set correctly.

Value of PATH_ZMS from the sreenshot: /zm/cgi-bin/zms

What is should be set to: /zm/cgi-bin/nph-zms

[AfricanSwallow]

Sorry for the delay, I had gone out of town for a few days. Anyway, back to respond to this. thank you for taking some time here Knight.

1. Value of PATH_ZMS was currently /zm/cgi-bin/zms due to me having tried multiple things. I see that when looking at the /usr/lib/zoneminder/cgi-bin/nph-zms that it is simply a symlink to /usr/lib/zoneminder/cgi-bin/zms so i was trying many things to get this to work including changing the path.
-rwxr-xr-x 1 www-data www-data 1271072 May 9 2017 zms
lrwxrwxrwx 1 www-data www-data 3 May 9 2017 nph-zms -> zms

2. I have reset the PATH to /zm/cgi-bin/nph-zms. The problem continues.
3. I saw that the path actually had a space on the end in the config, not sure if that could cause an issue, i removed that and have confirmed it does stay removed after saving. The problem continues.
4. I have checked many permissions throughout the system and they all look good. Anything questionable i went ahead end gave read permissions to www-data, just in case. The problem continues.
5. To double check cgi i created a test.php and confirmed cgi does indeed return info. The problem continues.
6. I can't think of much else to check. After restarting all of nginx and zoneminder, the problem continues.

Again, Im glad to keep going with this troubleshooting, but hoping to get some ideas of what/where to check. Here is a line from error.log and access.log from nginx:
Error
2018/07/25 12:42:03 [error] 4449#4449: *73 FastCGI sent in stderr: "PHP message: ERR [Socket /var/run/zm/zms-952196s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information.]" while reading response header from upstream, client: 127.0.0.1, server: zonemindermine.duckdns.org, request: "POST /zm/index.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "zonemindermine.duckdns.org", referrer: "http://zonemindermine.duckdns.org/zm/in ... atch&mid=2"
2018/07/25 12:42:03 [error] 4449#4449: *71 connect() to unix:/var/run/fcgiwrap.socket failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: zonemindermine.duckdns.org, request: "GET /zm/cgi-bin/nph-zms?mode=jpeg&scale=100&maxfps=5&buffer=1000&monitor=2&auth= d0d15027763a24537b38fa1f55d&connkey= 9159126&rand= 23287 HTTP/1.1", upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: "zonemindermine.duckdns.org", referrer: "http://zonemindermine.duckdns.org/zm/in ... atch&mid=2"

Access
127.0.0.1 - - [25/Jul/2018:12:47:28 -0500] "GET /zm/cgi-bin/nph-zms?mode=jpeg&scale=100&maxfps=5&buffer=1000&monitor=2&auth=d0d15027763a24537b38fa1f55d&connkey=9159126&rand=23287 HTTP/1.1" 502 182 "http://zonemindermine.duckdns.org/zm/in ... atch&mid=2" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0"
127.0.0.1 - - [25/Jul/2018:12:47:28 -0500] "POST /zm/index.php HTTP/1.1" 200 25 "http://zonemindermine.duckdns.org/zm/in ... atch&mid=2" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0"

As a last thing, here is a lsof run grepping the process zoneminder is running:
zmdc.pl 4493 www-data cwd DIR 8,53 4096 29532 /usr/share/zoneminder/www
zmdc.pl 4493 www-data rtd DIR 8,53 4096 2 /
zmdc.pl 4493 www-data txt REG 8,53 2093624 263105 /usr/bin/perl
zmdc.pl 4493 www-data mem REG 8,53 14176 7439 /usr/lib/x86_64-linux-gnu/perl5/5.26/auto/Sys/Mmap/Mmap.so
zmdc.pl 4493 www-data mem REG 8,53 88904 395207 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/Storable/Storable.so
zmdc.pl 4493 www-data mem REG 8,53 27088 395186 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/File/Glob/Glob.so
zmdc.pl 4493 www-data mem REG 8,53 18616 395209 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/Sys/Syslog/Syslog.so
zmdc.pl 4493 www-data mem REG 8,53 43392 395206 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/Socket/Socket.so
zmdc.pl 4493 www-data mem REG 8,53 544104 395218 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/re/re.so
zmdc.pl 4493 www-data mem REG 8,53 31680 6975 /lib/x86_64-linux-gnu/librt-2.27.so
zmdc.pl 4493 www-data mem REG 8,53 26880 395211 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/Time/HiRes/HiRes.so
zmdc.pl 4493 www-data mem REG 8,53 39424 395171 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/Data/Dumper/Dumper.so
zmdc.pl 4493 www-data mem REG 8,53 18704 395192 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/IO/IO.so
zmdc.pl 4493 www-data mem REG 8,53 105992 395200 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/POSIX/POSIX.so
zmdc.pl 4493 www-data mem REG 8,53 18456 395184 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/Fcntl/Fcntl.so
zmdc.pl 4493 www-data mem REG 8,53 47568 6924 /lib/x86_64-linux-gnu/libnss_files-2.27.so
zmdc.pl 4493 www-data mem REG 8,53 96616 6871 /lib/x86_64-linux-gnu/libgcc_s.so.1
zmdc.pl 4493 www-data mem REG 8,53 1615312 274236 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
zmdc.pl 4493 www-data mem REG 8,53 116960 7006 /lib/x86_64-linux-gnu/libz.so.1.2.11
zmdc.pl 4493 www-data mem REG 8,53 4236448 268416 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.9
zmdc.pl 4493 www-data mem REG 8,53 120744 427282 /usr/lib/x86_64-linux-gnu/perl5/5.26/auto/DBD/mysql/mysql.so
zmdc.pl 4493 www-data mem REG 8,53 133048 410053 /usr/lib/x86_64-linux-gnu/perl5/5.26/auto/DBI/DBI.so
zmdc.pl 4493 www-data mem REG 8,53 47656 395194 /usr/lib/x86_64-linux-gnu/perl/5.26.1/auto/List/Util/Util.so
zmdc.pl 4493 www-data mem REG 8,53 10281936 267385 /usr/lib/locale/locale-archive
zmdc.pl 4493 www-data mem REG 8,53 39208 6844 /lib/x86_64-linux-gnu/libcrypt-2.27.so
zmdc.pl 4493 www-data mem REG 8,53 2030544 6834 /lib/x86_64-linux-gnu/libc-2.27.so
zmdc.pl 4493 www-data mem REG 8,53 144976 6967 /lib/x86_64-linux-gnu/libpthread-2.27.so
zmdc.pl 4493 www-data mem REG 8,53 1700792 6897 /lib/x86_64-linux-gnu/libm-2.27.so
zmdc.pl 4493 www-data mem REG 8,53 14560 6857 /lib/x86_64-linux-gnu/libdl-2.27.so
zmdc.pl 4493 www-data mem REG 8,53 170960 6806 /lib/x86_64-linux-gnu/ld-2.27.so
zmdc.pl 4493 www-data 0u unix 0xffff9bacc9d17c00 0t0 2108498 type=DGRAM
zmdc.pl 4493 www-data 1u unix 0xffff9bae11d11000 0t0 2106256 type=STREAM
zmdc.pl 4493 www-data 2w REG 0,22 4 1448 /run/zm/zm.pid
zmdc.pl 4493 www-data 3u unix 0xffff9bae856f9000 0t0 2108014 /var/run/zm/zmdc.sock type=STREAM

[AfricanSwallow]

i guess ill just have to bite the bullet and go to apache on a separate port to run zoneminder. Not sure what else i can do here.

[knight-of-ni]

For the distro you are using, /zm/cgi-bin/nph-zms is the only correct value for PATH_ZMS with no trailing spaces when doing normal streaming. Don't be confused by the symlink. zms knows how it was called, and will do something different if it was executed as just "zms" rather than "nph-zms".

The problem you are experiencing is definitely related to your webserver configuration. The error you are receiving is not caused by file permissions or anything else.

This is how I would approach this:

• verify php-fpm and fcgiwrap are installed
• verify the fcgiwrap service is running
• there are two socket files your nginx config is lookging for. Verify those are getting created in the correct folders.
• verify fcgiwrap is spawning sufficient processes. On my pi 3, I've got DAEMON_PROCS=5
• verify your php-fpm config file is generating sufficient child processes

I'm running nginx on a pi 3 running fedora, and this is what mine looks like:

Code: Select all

# Uncomment these on machines with little memory
pm = ondemand
pm.max_children = 10
pm.process_idle_timeout = 10s

[bbunge]

I worked on Bionic a week or so ago and got it to stream more than one camera. See:
https://wiki.zoneminder.com/Ubuntu_Serv ... .2C_PHP.29

You may be able to fix your system from the changes you made...

If it does not exist create

nano /etc/default/fcgiwrap

and insert (with the number of children -c equal to the number of cameras)

DAEMON_OPTS=-c 10

then restart the fcgiwrap service.

systemctl restart fcgiwrap

[AfricanSwallow]

thank you @bbunge and @knight-of-ni, I have finally figured out what the issue is. fcgiwrap was indeed not starting properly and I failed to see that in the syslog. The error was:
fcgiwrap[23674]: /usr/sbin/fcgiwrap: invalid option -- ' '
fcgiwrap[23674]: Unknown option `- '.
fcgiwrap.service: Main process exited, code=exited, status=1/FAILURE

I had created the file as bbunge mentioned at /etc/default/fcgiwrap and had inserted into it, according to the wiki -https://wiki.zoneminder.com/Ubuntu_Serv ... e_easy_way

DAEMON_OPTS="-f -c 1"

the -f is what was causing the issue. once this changed to the following, fcgiwrap starts correctly and the camera stream is function as expected:
DAEMON_OPTS="-c 1"

I have since modified it to allow more cameras and will test once possible
DAEMON_OPTS="-f -c 10"

I think the wiki should either be noted that for 18.04 the wiki article will cause the issue, though I'd be willing to bet the same issue arises in 17.10 as well, though i cannot confirm that, and should be modified to remove the -f. While i have your attention, do either of you know what that flag is for? I do not see it noted on the fcgiwrap man page http://www.linuxcertif.com/man/8/fcgiwrap/.

Again, thank you both for all of your help here! Hopefully this also helps someone else who may find themselves with the same issue.

[bbunge]

The 18.04 WIKI was updated with the information I gave you. I should update the 16.04 WIKI but the 16.10 and 17.XX WIKI articles will be obsoleted when I get around to it.