Can't get zmNinja's Android notification via FCM/websockets working

[SpikeyGG]

I'm using ZM 1.34.16 in a docker (dlandon's). I've got yolo and face detection working but I can't seem to get notifications to work using FCM nor websockets.

I can use nearly all of the features of zmNinja without issue but I can't get notifications to work. When I look at the logs I see "Failed to connect to mythbox.mynet/192.168.0.10:9001" which looks like a very strange address showing both my domain name and my private IP. I have to host the port on 9001 since 9000 is already in use on that machine. Looking above at the initalize websocket it shows the correct string "wss://mythbox.mynet:9001". Do I need to use SSL and certs with the notifications? I ask because I don't have that configured.

Code: Select all

Jul 2, 2020 05:06:24 PM DEBUG Setting server version to:1.34.16
Jul 2, 2020 05:06:24 PM DEBUG getAPI version succeeded with 1.34.16
Jul 2, 2020 05:06:24 PM INFO Got API version: 1.34.16
Jul 2, 2020 05:06:24 PM INFO First invocation of TimeZone, asking server
Jul 2, 2020 05:06:24 PM INFO EventSever: Initializing Websocket with URL wss://mythbox.mynet:9001
Jul 2, 2020 05:06:24 PM DEBUG EventSever: Using native websockets...
Jul 2, 2020 05:06:24 PM DEBUG Calling pushInit()
Jul 2, 2020 05:06:24 PM INFO EventSever: Setting up push registration
Jul 2, 2020 05:06:24 PM DEBUG Transitioning state to: app.events with param {}
Jul 2, 2020 05:06:24 PM INFO getMonitors:Loading all monitors
Jul 2, 2020 05:06:24 PM INFO Checking value of ZM_MIN_STREAMING_PORT for the first time
Jul 2, 2020 05:06:24 PM DEBUG Push permission returned: {"isEnabled":true}
Jul 2, 2020 05:06:24 PM DEBUG CACHE: NOT found for:cached_timezone reverting to HTTP
Jul 2, 2020 05:06:24 PM DEBUG CACHE: NOT found for:cached_multi_port reverting to HTTP
Jul 2, 2020 05:06:24 PM DEBUG EventSever: Push Notification registration ID received: {"registrationId":"<long_token_string_removed>","registrationType":"FCM"}
Jul 2, 2020 05:06:24 PM INFO getMonitors:Loading all monitors
Jul 2, 2020 05:06:24 PM INFO Checking value of ZM_MIN_STREAMING_PORT for the first time
Jul 2, 2020 05:06:24 PM DEBUG CACHE: NOT found for:cached_multi_port reverting to HTTP
Jul 2, 2020 05:06:25 PM DEBUG EventSever: Failed to connect to WebSocket: code: 1006, reason: undefined, exception: Failed to connect to mythbox.mynet/192.168.0.10:9001
Jul 2, 2020 05:06:25 PM INFO ZM has recaptcha disabled - good
Jul 2, 2020 05:06:25 PM DEBUG Real value of PRIVACY is:0
Jul 2, 2020 05:06:25 PM DEBUG CACHE: storing key data in cache now, with expiry of 86400
Jul 2, 2020 05:06:25 PM DEBUG CACHE: storing key data in cache now, with expiry of 86400

From what I understand, I should be able to use FCM to alert my cellphone no matter where I am but how do I get that to work? I see the tokens.txt file in my docker instance but it never gets any data from zmNinja (it is still a zero byte file), even though I can see the registrationId tokens in the zmNinja logs.

[asker]

The way this works is the phone must be able to reach 192.168.0.10:9001 for the first time to register its push token with the ES.
If that doesn't happen, you won't get push because the ES doesn't know your token.

That error message is an internal exception message the websocket library generates - it might just be a concatenation of DNS name and IP name.

You should focus on the reachability part from phone to ES.

1. Is 192.168.0.10:9001 reachable from your phone? (Often people attempt this over a phone's cellular network and it won't work as this looks like an internal ip, which may mean you have a hostname to internal IP mapping

2. If it is reachable, and you are using self signed certs there may be issues with your certs. One way to test is to disable "WSS" on the server side and on zmNinja (just use ws) and try it (see here). If it is reachable, you have an SSL cert issue. If not, you have a reachability issue.

[SpikeyGG]

Thanks asker,

1. Is 192.168.0.10:9001 reachable from your phone? (Often people attempt this over a phone's cellular network and it won't work as this looks like an internal ip, which may mean you have a hostname to internal IP mapping

I was worried about that. I'm only really trying to get this to work over WiFi at this point so my phone is on the same network as the zoneminder server, it does have to go through the router so I checked the logs and I see allowed traffic on the router from the cellphone (WiFi) to the zoneminder server.

2. If it is reachable, and you are using self signed certs there may be issues with your certs. One way to test is to disable "WSS" on the server side and on zmNinja (just use ws) and try it (see here). If it is reachable, you have an SSL cert issue. If not, you have a reachability issue.

In an effort to simplify things, I am not using the certs. I just tried using ws instead of wss and the error messaging looks the same and the tokens.txt file is still empty. Is there some way I can test two-way traffic on port 9001 to guarantee that it's open and workable?

EDIT
After posting this question I found this article on serverfault. It shows how you can monitor the ethernet for traffic on a specific port. I used it to monitor 9001 on the zoneminder server and I see it receive and respond when I launch the app and when I click the save button on the EventServer config page:

Code: Select all

$ sudo tcpdump -i eth0 -n tcp 'port 9001'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:08:33.303415 IP 192.168.0.1.40762 > 192.168.0.10.9001: Flags [S], seq 3105867478, win 65535, options [mss 1460,sackOK,TS val 1924918582 ecr 0,nop,wscale 8], length 0
08:08:33.303668 IP 192.168.0.10.9001 > 192.168.0.1.40762: Flags [R.], seq 0, ack 3105867479, win 0, length 0
08:08:50.925724 IP 192.168.0.1.40798 > 192.168.0.10.9001: Flags [S], seq 3374061652, win 65535, options [mss 1460,sackOK,TS val 1924936206 ecr 0,nop,wscale 8], length 0
08:08:50.925881 IP 192.168.0.10.9001 > 192.168.0.1.40798: Flags [R.], seq 0, ack 3374061653, win 0, length 0
^C
4 packets captured
10 packets received by filter
0 packets dropped by kernel
$

Still nothing in tokens.txt though.

[asker]

If you are saying you see a message from the device to the ES, let's do this:

1. Make sure WSS is turned off on both sides (ES+zmNinja) and ES is restarted, so we are sure it is not expecting WSS
2. Make sure ES debug logs are on
3. Load app and register with ES

Please post any logs at the ES side.

I'm thinking of the following:

a) If you see the ES getting a message, that means its a payload related issue
b) If you don't see the ES getting the message, that means your server is receiving it, but not going up to the ES, which may mean:
b.1) There is some intermediary process that you may not have described running in the ES machine (a proxy/firewall etc.) that is dropping the connection
b.2) There is a TLS version difference between the device and the server (which should not happen if you turn off WSS)

[SpikeyGG]

ok, I think when you say that WSS is turned off on the ES side you're referring to:

Code: Select all

# ... cut a bunch of stuff out
[ssl]
# Enable SSL (default: yes)
enable = no

cert = !ES_CERT_FILE
key = !ES_KEY_FILE

#cert = /etc/apache2/ssl/zoneminder.crt
#key = /etc/apache2/ssl/zoneminder.key

# Location to SSL cert (no default).
# cert = /etc/apache2/ssl/yourportal/zoneminder.crt

# Location to SSL key (no default).
# key = /etc/apache2/ssl/yourportal/zoneminder.key

[customize]
# Display messages to console (default: no).
console_logs = yes
# ... cut a bunch of stuff out

I just updated the console_logs to 'yes' and restarted ZM. (I'm running zmeventnotification as OPT_USE_EVENTNOTIFICATION turned on, should I run it manually?) When I watch the ZM log system using the web browser I see this for zmeventnotification component:

Code: Select all

Date/Time
Component	Server	PID	Level	Message	File	Line
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: Web Socket Event Server listening on port 9001	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: Secure WS is disabled...	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: Re-loading monitors	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: Initializing MQTT connection...	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: Event Notification daemon v 5.15-Docker starting	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	WAR	PARENT: WARNING: SSL is disabled, which means all traffic will be unencrypted	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: |------- Starting ES version: 5.15-Docker ---------|	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: MQTT Enabled	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: Push enabled via FCM	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: using secrets file: /etc/zm/secrets.ini	zmeventnotification.pl	
2020-07-03 09:19:42	zmeventnotification		32034	INF	PARENT: using config file: /etc/zm/zmeventnotification.ini	zmeventnotification.pl

While that was opened, I went into my zmNinja app and hit save on the Event Server screen which generated a 9001 packet I saw via tcpdump and no other messages were printed to the log file. I am running ZM from a docker but I wouldn't think the docker translation should have any effect on the packet. My external 9001 port should be connected to the docker's internal 9000 port which is used for the websocket stuff, but it doesn't look like the zmeventnotification service is receiving the message.

[asker]

1. To be sure, enable debug logs
https://zmeventnotification.readthedocs ... ng#logging

2. To disable WSS both sides, https://zmeventnotification.readthedocs ... e-wss-mode


Remember to restart ES

(General note: also make sure you have read the docs/FAQs thoroughly - it helps when trying to debug)

[SpikeyGG]

Remember to restart ES

If I restart ZM doesn't that restart ES as well if I'm using OPT_USE_EVENTNOTIFICATION? I thought that ES was equivalent to zmeventnotification?

(General note: also make sure you have read the docs/FAQs thoroughly - it helps when trying to debug)

I didn't have the debug logs enabled, I turned that stuff on and I'll try running it again. I've scoured through the docs and there's very little on setting up FCM and websockets -- it appears to just work for the majority of folks or people haven't run into this problem before. The zmNinja app works great on WiFi, I can pull up all my cameras and events so clearly I can get to the ZM server but this notification thing is boggling my mind.

I know that the zmeventnotification recognizes that I have WSS disabled because of these messages:

Code: Select all

	PARENT: Secure WS is disabled...
	PARENT: WARNING: SSL is disabled, which means all traffic will be unencrypted

Is there some way to test FCM independently? I'm not sure how to configure that and get it working and the docs don't seem to say much about FCM, specifically. Maybe I need an SSL setup with certs to make FCM possible?

[asker]

If I restart ZM doesn't that restart ES as well if I'm using OPT_USE_EVENTNOTIFICATION? I thought that ES was equivalent to zmeventnotification?

Yes it does.

I didn't have the debug logs enabled, I turned that stuff on and I'll try running it again. I've scoured through the docs and there's very little on setting up FCM and websockets -- it appears to just work for the majority of folks or people haven't run into this problem before.

When you are trying to figure out a problem, enabling debug logs helps. They are there for a reason - they print out a lot of intermediary messages that might give you further hints before an INF log is generated. Lots of people run into lots of different problems - and that is where debug logs help - more information than trying to guess. In your case, I don't know if debug logs help or not (especially if the ES is not getting your message) - but we won't know till we try. Plus it saves me a lot of time while trying to help others.

The zmNinja app works great on WiFi, I can pull up all my cameras and events so clearly I can get to the ZM server but this notification thing is boggling my mind.

I've seen situations before where users have some sort of proxy/whatever that is a perimeter process that rejects connections for some reason, the most common one being an nginx proxy in between that is messing with things. Take a look at this thread https://stackoverflow.com/questions/193 ... -code-1006 and see if one of the reasons apply to you. Specifically, the answer about nginx timeouts, if you are using nginx.