ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Forum for questions and support relating to the 1.29.x releases only.
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

Long story short: I have a Raspberry Pi 2 with ZM 1.29.0, Raspbian Jessie, followed your guide plus some other for using an external SSD to record events and images, I've put a higher shmmax value, enabled the v4l for the embedded PiCam, etc.
I have 4 cam, 3 foscam 720P IP camera and the embedded PiCam, working at 720p @ around 3 fps, modect. All is working great except that...

At random times (can happens once a day or so) one or more camera stop to be received! :? But the cameras theyself are healty (I can see them all through their own IP address). If i disable and reenable the cam that stopped working in ZM... images are resuming to be received correctly!
The same if I restart zoneminder.
So there should be something that stop working in zoneminder itself.

My question is... is there some good people that can help me troubleshoot this issue? What can I check?

Thank you in advance,
Henrik
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

Trying to understand this morning I've found one stuck camera (the first, most of the time this is the camera that zoneminder stops to receive). As always I've restart zoneminder and the camera came back to life (the camera has no problem at all).
Trying to figure out I've checked events. My last event was at 5:31 this morning (when the camera switched from IR to color) then nothing more. So the problem arised after that time. I've checked the log filtering "WAR" level events and found this:

Code: Select all

2016-06-29 07:22:55.429793	zmc_m3		24122	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:22:55.406757	zmc_m3		24122	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-06-29 07:22:55.382123	zmc_m3		24122	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-06-29 07:22:47.165318	zmc_m3		24122	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:22:47.147609	zmc_m3		24122	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-06-29 07:21:41.276651	zmc_m3		24113	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:21:41.261383	zmc_m3		24113	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-06-29 07:21:41.245903	zmc_m3		24113	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-06-29 07:21:25.016459	zmc_m3		24113	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:21:24.998704	zmc_m3		24113	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-06-29 07:20:17.420954	zmc_m3		24104	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:20:17.404553	zmc_m3		24104	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-06-29 07:20:17.385630	zmc_m3		24104	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-06-29 07:20:03.192335	zmc_m3		24104	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:20:03.177822	zmc_m3		24104	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-06-29 07:18:55.533363	zmc_m3		24096	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:18:55.515506	zmc_m3		24096	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-06-29 07:18:55.499909	zmc_m3		24096	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-06-29 07:18:41.434615	zmc_m3		24096	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:18:41.418099	zmc_m3		24096	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-06-29 07:17:33.537102	zmc_m3		24073	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:17:33.516627	zmc_m3		24073	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-06-29 07:17:33.496088	zmc_m3		24073	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-06-29 07:17:19.365889	zma_m3		21224	WAR	Signal: Lost	zm_monitor.cpp	1420
2016-06-29 07:17:19.317145	zmc_m3		24073	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:17:19.297080	zmc_m3		24073	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-06-29 07:16:11.465686	zmc_m3		24023	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:16:11.450716	zmc_m3		24023	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-06-29 07:16:11.434560	zmc_m3		24023	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-06-29 07:07:16.444739	zma_m3		21224	WAR	Signal: Reacquired	zm_monitor.cpp	1420
2016-06-29 07:07:15.426828	zma_m3		21224	WAR	Signal: Lost	zm_monitor.cpp	1420
2016-06-29 07:07:15.391718	zmc_m3		24023	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:07:15.379376	zmc_m3		24023	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-06-29 07:06:08.453708	zmc_m3		22048	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-06-29 07:06:08.438754	zmc_m3		22048	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-06-29 07:06:08.423581	zmc_m3		22048	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
I think that zmc_m3 refers to that camera as the monitor link hovering mouse to that camera is http://[MYIP]/zm/index.php?view=watch&mid=3 , am I right?

The strange is that I can see similar events about other cameras too but it seems that when this sequence is longer the camera stops to get received totally.

Please, isn't there someone that can help me to troubleshoot the issue? :?
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

I'm further investigating. I've enabled LOG_LEVEL_FILE to level Debug, LOG_DEBUG = On, LOG_DEBUG_LEVEL = 5 and waited that a problem arrive.
This time was another camera that stopped to be received, so here is the last lines of zm_debug.log.29959 (the PID of the last process that was handling that camera):

Code: Select all

06/30/16 01:38:53.498563 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:38:56.001367 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:38:58.504143 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:01.006199 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:03.508991 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:06.011808 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:08.512938 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:11.015915 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:13.518727 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:16.021505 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:18.524339 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:21.027118 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:23.529941 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:26.032788 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:28.535642 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:31.038472 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:33.541324 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:36.043924 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:38.546170 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:41.049005 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:43.551837 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:46.054663 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:48.556187 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:51.056674 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:53.559457 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:56.062240 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:39:58.563148 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:01.065973 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:03.568883 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:06.071731 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:08.574536 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:11.077313 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:13.579648 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:16.082500 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:18.585354 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:21.088141 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:23.590917 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:26.096174 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:28.598938 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/30/16 01:40:28.629307 zmc_m4[29959].INF-zm_signal.cpp/40 [Got signal 15 (Terminated), exiting]
06/30/16 01:40:28.643998 zmc_m4[29959].ERR-zm_remote_camera_http.cpp/175 [Select error: Interrupted system call]
06/30/16 01:40:28.664074 zmc_m4[29959].ERR-zm_remote_camera_http.cpp/628 [Unable to read header]
06/30/16 01:40:28.681099 zmc_m4[29959].ERR-zm_remote_camera_http.cpp/1141 [Unable to get response, disconnecting]
06/30/16 01:40:28.693769 zmc_m4[29959].DB3-zm_remote_camera_http.cpp/132 [Disconnected from host]
06/30/16 01:40:28.706668 zmc_m4[29959].DB3-zm_remote_camera_http.cpp/132 [Disconnected from host]
06/30/16 01:40:28.814637 zmc_m4[29959].DB1-zm_logger.cpp/239 [Terminating Logger]
I don't know how to interpret that... What could be wrong? What other action can I take to try to understand what is hanging??? Please help me... :(
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by bbunge »

With 4 cameras on a pi you are likely running out of memory. Try to lower the camera resolution and frame rate.
And you should not mess with shared memory settings.
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

But... I had my zoneminder 1.28 on Debian Stretch perfectly working for one year without a single issue receiving the same 4 cameras with same parameters! I never had to restart it in one year! Now every day I have at least one camera that doesn't work... There should be a small configuration, a small change in something to fix that!

That said I did the math... Every camera is 1280x720 @ 24bpp so we have 2,64MB per frame. I've set the buffer to 30 frames (3xsec=10seconds), so every camera consume 79,11MB. All the four camera should be 316,41MB. I dimensioned shmmax to 512MB so we should be ok. Or maybe that means that less memory will be available to the rest of processes? I don't remember how much was shmmax with my last working configuration :(

But is'nt there a more scientifical way for understandin what is the problem that hangs the receive of a camera? What should I check in the log?
I'm very near to throw away my raspberry to other tasks and buy a NVR... But that would be a defeat... :(
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by bbunge »

energywave wrote:But... I had my zoneminder 1.28 on Debian Stretch perfectly working for one year without a single issue receiving the same 4 cameras with same parameters! I never had to restart it in one year! Now every day I have at least one camera that doesn't work... There should be a small configuration, a small change in something to fix that!

That said I did the math... Every camera is 1280x720 @ 24bpp so we have 2,64MB per frame. I've set the buffer to 30 frames (3xsec=10seconds), so every camera consume 79,11MB. All the four camera should be 316,41MB. I dimensioned shmmax to 512MB so we should be ok. Or maybe that means that less memory will be available to the rest of processes? I don't remember how much was shmmax with my last working configuration :(

But is'nt there a more scientifical way for understandin what is the problem that hangs the receive of a camera? What should I check in the log?
I'm very near to throw away my raspberry to other tasks and buy a NVR... But that would be a defeat... :(
Well, that was 1.28... Bet your tmpfs, ramdisk, or whatever you want to call it is running out of space. Lower the resolution and frame rate to get it working then push those up, if you need to, till something breaks...
marco123
Posts: 25
Joined: Thu Mar 12, 2015 7:38 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by marco123 »

on my pi3 with 1.29 my shmmax is 134217728 image buffer 50
No reboot, no stop, no pb with 2 ip cam in 640x480 10fps
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

Hm... I'll try to lower my shmmax and buffer size, trying to lower memory usage. However I'm sure that 1.29 can work as well as 1.28. There something for sure that I have to change. I'll try that and post result. Thank you to who replied to me in the meanwhile :)
bbunge
Posts: 2934
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by bbunge »

energywave wrote:Hm... I'll try to lower my shmmax and buffer size, trying to lower memory usage. However I'm sure that 1.29 can work as well as 1.28. There something for sure that I have to change. I'll try that and post result. Thank you to who replied to me in the meanwhile :)
Get away from fooling with shmmax. That has not been needed since 1.26.x or so. Zoneminder has been compiled to use mmap since at least 1.26.x! Most Linux distros will use half of the RAM for tmpfs. You cam "adjust" tmpfs depending upon your need. I use Ubuntu and have set /dev/shm to 4 GIG out of 16 GIG RAM. You can also set PATH_MAP and PATH_SWAP to use tmpfs to speed things up a bit.
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

I've experimented a lot and I have more data. I'm rather sure to have found the root of the problem but not a solution (could be a bug).
shmmax, however, seems to be an important part as zoneminder seems to still use shared memory to share stream buffer between processes (even the wiki for setting up raspberry pi with zoneminder tells you to change it's value: https://wiki.zoneminder.com/Raspbian
Furthermore here my df -h:

Code: Select all

File system     Dim. Usati Dispon. Uso% Montato su
/dev/root        15G  2,7G     12G  20% /
devtmpfs        427M     0    427M   0% /dev
tmpfs           432M  211M    221M  49% /dev/shm
tmpfs           432M   44M    388M  11% /run
tmpfs           5,0M  4,0K    5,0M   1% /run/lock
tmpfs           432M     0    432M   0% /sys/fs/cgroup
/dev/mmcblk0p1   63M   21M     43M  33% /boot
/dev/sda1       230G   37G    181G  17% /media/ssd
tmpfs            87M     0     87M   0% /run/user/1000
As you can see /dev/shm is at 211M of usage. My settings are now 1280x720x24bppx20 buffer frames = 52.73MB per camera, 4 camera are 210,94MB + some overhead. It means that shared memory is in use by zoneminder and shmmax and shmall are vital so set in a small environment like Raspberry Pi.
I've set these params in /etc/sysctl.conf:

Code: Select all

kernel.shmmax = 300000000
kernel.shmall = 73242
From what I understood shmmax is the maximum amount in bytes that a single buffer can allocate in shared memory (I set it to 300M) and shmall is the maximum size of shared memory allowed to allocate between all buffers. shmall is expressed in bytes while shmall is expressed in pages (usually 4.096 bytes).
Doing the math it seems that I've set it correctly.

I've found some threads that seems to refer to the same problem I have, in particular I've found one that helped me better understand what is happeneing. This uses is running zoneminder on a horsepower PC so that's not bound to low resources of the raspberry. Here the link: viewtopic.php?t=23120

Today I had the "luck" to be connected to my zoneminder while a camera stopped working. I've checked it while it was happening and the camera worked correctly. Here the log content:

Code: Select all

2016-07-04 16:32:50.256155	zms		16348	ERR	Terminating, last frame sent time 1467642769.755924 secs more than maximum of 10.000000	zm_monitor.cpp	4392
2016-07-04 16:13:43.621386	zmc_m3		16261	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:13:43.603312	zmc_m3		16261	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-07-04 16:13:43.577055	zmc_m3		16261	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-07-04 16:13:31.395876	zmc_m3		16261	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:13:31.377467	zmc_m3		16261	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-07-04 16:12:19.975779	zmc_m3		16253	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:12:19.959037	zmc_m3		16253	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-07-04 16:12:19.940299	zmc_m3		16253	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-07-04 16:12:08.948780	zmc_m3		16253	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:12:08.936542	zmc_m3		16253	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-07-04 16:10:56.122267	zmc_m3		16245	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:10:56.108872	zmc_m3		16245	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-07-04 16:10:56.096659	zmc_m3		16245	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-07-04 16:10:46.795600	zmc_m3		16245	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:10:46.785760	zmc_m3		16245	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-07-04 16:09:34.272013	zmc_m3		16194	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:09:34.261324	zmc_m3		16194	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-07-04 16:09:34.248381	zmc_m3		16194	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-07-04 16:09:25.049615	zmc_m3		16194	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:09:25.035273	zmc_m3		16194	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-07-04 16:08:12.493046	zmc_m3		16190	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:08:12.482883	zmc_m3		16190	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-07-04 16:08:12.473241	zmc_m3		16190	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-07-04 16:08:03.176769	zmc_m3		16190	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:08:03.165773	zmc_m3		16190	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
2016-07-04 16:06:50.604359	zma_m3		16107	WAR	Signal: Lost	zm_monitor.cpp	1420
2016-07-04 16:06:50.566478	zmc_m3		15836	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 16:06:50.556780	zmc_m3		15836	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-07-04 16:06:50.544222	zmc_m3		15836	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
It seems that the camera replied for some time to zoneminder with a "text/plain" response, that suggest an error message, and at certain time zoneminder says: Terminating, last frame sent time 1467642769.755924 secs more than maximum of 10.000000 (it's the last event, at first position above). Now, as suggested in the thread I've linked above, 1,467,642,769.755924 seconds are about 46 years... So it seems that the time of last frame received get corrupted (ora nulled) and the math Last frame time - Current time generates that value that triggers the 10 seconds threshold and the process is killed and never restarted.
Could it be a bug in post 1.28 zoneminder?

The problem could be not bound to a broken reply as I've seen today the same thing happens with another camera (in monitor mode, so it's not modect the problem!). The process was restarted and the same problem as above arised:

Code: Select all

2016-07-04 19:05:55.012375	web_php		14835	ERR	socket_sendto( /var/run/zm/zms-431515s.sock ) failed: No such file or directory	/usr/share/zoneminder/www/includes/functions.php	2371
2016-07-04 19:05:48.247443	zms		17157	ERR	Terminating, last frame sent time 1467651947.747218 secs more than maximum of 10.000000	zm_monitor.cpp	4392
2016-07-04 19:03:13.478690	zmdc		17143	INF	'zmc -m 4' started at 16/07/04 19:03:13	zmdc.pl	
2016-07-04 19:03:13.478650	zmdc		8284	INF	'zmc -m 4' starting at 16/07/04 19:03:13, pid = 17143	zmdc.pl	
2016-07-04 19:03:11.897190	zmwatch		8421	INF	Restarting capture daemon for Soggiorno, shared data not valid	zmwatch.pl	
2016-07-04 19:03:01.849450	zmdc		8284	INF	'zmc -m 4' exited normally	zmdc.pl	
2016-07-04 19:03:01.729028	zmc_m4		17136	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 19:03:01.713583	zmc_m4		17136	ERR	Unable to read header	zm_remote_camera_http.cpp	628
2016-07-04 19:03:01.695217	zmc_m4		17136	ERR	Select error: Interrupted system call	zm_remote_camera_http.cpp	175
2016-07-04 19:03:01.648074	zmc_m4		17136	INF	Got signal 15 (Terminated), exiting	zm_signal.cpp	40
2016-07-04 19:03:01.150530	zmdc		8284	INF	'zmc -m 4' sending stop to pid 17136 at 16/07/04 19:03:01	zmdc.pl	
2016-07-04 19:02:59.430670	zmwatch		8421	INF	Restarting capture daemon for Soggiorno, time since last capture 6 seconds (1467651779-1467651773)	zmwatch.pl	
2016-07-04 19:02:53.429754	zmc_m4		17136	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-07-04 19:02:53.416247	zmc_m4		17136	ERR	Unrecognised content type 'text/plain'	zm_remote_camera_http.cpp	856
So I'm not alone. There should be a bug somewhere. What could I do, other than buying a NVR? I'm sad to not being able to let zoneminder works in a stable way... :(
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

Latest finding from log files: it seems not related to events I've posted above but, when it stops to work, I find this in the log:

Code: Select all

06/29/16 23:40:19.112759 zmc_m4[29959].DB3-zm_remote_camera_http.cpp/147 [Request sent]
06/29/16 23:40:21.615482 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:24.118339 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:26.621128 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:29.123936 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:31.626701 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:34.128694 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:36.631466 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:39.134238 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:41.636151 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:44.138983 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:46.641745 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:49.144519 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:51.646145 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:54.149000 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:56.651775 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:40:59.154594 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:01.657397 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:04.160240 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:06.663009 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:09.164173 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:11.664983 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:14.165844 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:16.666633 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:19.167303 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:21.668078 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:24.168865 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:26.671530 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:29.174352 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:31.677143 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:34.179972 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:36.680807 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:39.183566 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:41.686331 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:44.189161 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:46.691987 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:49.194693 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:51.697420 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:54.200189 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:56.703012 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:41:59.205875 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:42:01.708764 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:42:04.211619 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:42:06.714476 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:42:09.216187 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:42:11.719014 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:42:14.221829 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
06/29/16 23:42:16.724638 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
... and so on until I restart zoneminder and the process log ends with:

Code: Select all

06/30/16 01:40:28.629307 zmc_m4[29959].INF-zm_signal.cpp/40 [Got signal 15 (Terminated), exiting]
06/30/16 01:40:28.643998 zmc_m4[29959].ERR-zm_remote_camera_http.cpp/175 [Select error: Interrupted system call]
06/30/16 01:40:28.664074 zmc_m4[29959].ERR-zm_remote_camera_http.cpp/628 [Unable to read header]
06/30/16 01:40:28.681099 zmc_m4[29959].ERR-zm_remote_camera_http.cpp/1141 [Unable to get response, disconnecting]
06/30/16 01:40:28.693769 zmc_m4[29959].DB3-zm_remote_camera_http.cpp/132 [Disconnected from host]
06/30/16 01:40:28.706668 zmc_m4[29959].DB3-zm_remote_camera_http.cpp/132 [Disconnected from host]
06/30/16 01:40:28.814637 zmc_m4[29959].DB1-zm_logger.cpp/239 [Terminating Logger]
So I have now to understand what the hell is "Select timed out [...]" and what is caused by... :?
If someone can help me what direction to take it would be much appreciated :wink:
User avatar
knight-of-ni
Posts: 2404
Joined: Thu Oct 18, 2007 1:55 pm
Location: Shiloh, IL

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by knight-of-ni »

Didn't see you show up in our IRC like I asked in my github reply, so I'll respond here.

This is a classic problem when ZoneMinder runs out of hardware or network resources. Seen it many times.
More specifically, no one who works on this project has ever stated you could reliably run four HD cameras with ZoneMinder on any Raspberry Pi. 1-2 HD cameras is the recommendation. Yeah, I know you said you had it working for zm 1.28, but we have never stated, much less guaranteed, that would work. Go back to using 1.28 if you want. A second Pi might help, but not so much if the problem is your network.

For starters, do the following:
- Undo everything you have done relating to shmall and shmax. Put it back to default. Like bbunge said, those parameters have not been used for a long time. shmall and shmax are for shared memory. ZoneMinder used mapped memory, which is not the same. I'm sorry you may have come across community documentation stating otherwise. Feel free to contact the author of the guide you followed and ask him or her to update it.

- After you reduced the buffer settings to 20, your /dev/shm of 211MB looks reasonable if that value is correct. Keep an eye on it. Resolution, buffersize, and number of cameras directly affect /dev/shm and the cameras you are using can easily fill that up in the blink of an eye.

- You said the framerate was 3 fps. Verify it still is and that value has been set in the camera. You are guaranteed to have big problems if you try to limit the frame rate in zoneminder rather than in the camera.

- I don't recall if arm chips have SSE instruction set. I think not, but try setting your color depth to 32bit. If it works, you will get a performance boost.

Messages like this imply a network interruption:

Code: Select all

06/30/16 01:40:28.693769 zmc_m4[29959].DB3-zm_remote_camera_http.cpp/132 [Disconnected from host]

Code: Select all

06/29/16 23:42:11.719014 zmc_m4[29959].DB4-zm_remote_camera_http.cpp/168 [Select timed out timeout was 0 secs 0 usecs]
In addition to memory, disk i/o and cpu resources are limited on the pi. We need more information to get a better idea of your system:

How much cpu resources are being consumed?

What kind of disk are you recording events to? How is the disk connected to the Pi?
The Pi does not have a good high i/o solution. Writing to an sd card can be problematic, and going through the USB bus to something else will present a new bottleneck which will affect all attached usb devices. The best you can hope for is mapped or iscsi connection to a NAS box, over the wired ethernet port.

We know how the picam is attached, but what kind of network connection do you have for your other three cameras and the raspberry pi?
Please don't say Wifi.

We need to see the complete camera configuration for all four of your cameras. Show us the General and Source tabs. Screenshots are preferred. Knowing the make and model of your Foscam cameras would help too.

So there you go. These are some things to look into and some data to provide. Again, we make no guarantees you will be able to get this to work, nor have we ever stated it will, using a Raspberry Pi. My free time is valuable to me, so if you continue with the assumption it must be a bug or continue to make implications this is somehow our problem, which we must fix for you for free, then I will simply spend my free time helping someone else.
Visit my blog for ZoneMinder related projects using the Raspberry Pi, Orange Pi, Odroid, and the ESP8266
All of these can be found at https://zoneminder.blogspot.com/
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

Hello Andrew, first of all thank you for your reply! Before to going further: please wait and read again what I wrote with another mood. I'm sure it's my fault, for sure, maybe because english is not my first language. I was not pretending anything, absolutely! I know your spare time (and other too) is valuable, like mine. I'm a programmer too (dot net in Windows, but that's another story) and I too do things for free so let's archive this misunderstanding :)
My intention was to improve zoneminder, if possible, by helping you find a problem that maybe could be there and could be handled more gracefully (to write something more easy to understand in the main log, so that average user understands what is happening), in addition to resolving my problem, of course.
So I was trying to be helpful to you too, if I can. Please don't assume I was requesting assistance for free because I'm on your side and I hate too pretentious people that wants you to work for them for free and fast.
Sorry about IRC, I missed that piece (but even if I didn't I couldn't, I'm at work right now).
This evening I'll post a comment here with all you requested. If we could resolve my problem we could improve the wiki page about setting up zoneminder on raspberry pi, I can do it, if you want :)
energywave
Posts: 10
Joined: Fri Jun 24, 2016 10:20 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by energywave »

Hello Andrew, thank you again for interesting! Here the data you requested:
let's start with my configuration.

Camera id=1: picam
GENERAL
Image

SOURCE
Image

BUFFERS
Image


Camera id=3: Foscam FI9804W (720p h264 cam)
GENERAL
Image

SOURCE
Image

BUFFERS
same as cam id 1

Camera id=2: Foscam FI9853EP (720p h264 cam)
GENERAL
same as cam id 3

SOURCE
same as cam id 3 (except ip address)

BUFFERS
same as cam id 1

Camera id=4: Foscam FI9818W V2 (720p h264 cam pan tilt)
GENERAL
Image

SOURCE
same as cam id 3 (except ip address)

BUFFERS
same as cam id 1

As you see I've set FPS to 2 to reduce CPU usage (the max I can achieve is around 3).
I've undone my shmall, shmax values (many information searching with google tells you to set them, I know it's not your fault!) I'll try to find the authors to inform them to remove that outdated info. I didn't knew about shared memory / mapped memory. I'm a Windows developer, that's another beast.

Here my df -h:

Code: Select all

File system     Dim. Usati Dispon. Uso% Montato su
/dev/root        15G  2,8G     12G  20% /
devtmpfs        427M     0    427M   0% /dev
tmpfs           432M  211M    221M  49% /dev/shm
tmpfs           432M  5,9M    426M   2% /run
tmpfs           5,0M  4,0K    5,0M   1% /run/lock
tmpfs           432M     0    432M   0% /sys/fs/cgroup
/dev/mmcblk0p1   63M   21M     43M  33% /boot
/dev/sda1       230G   43G    175G  20% /media/ssd
tmpfs            87M     0     87M   0% /run/user/1000
And here my /proc/meminfo:

Code: Select all

MemTotal:         882796 kB
MemFree:          198408 kB
MemAvailable:     356620 kB
Buffers:           71164 kB
Cached:           327612 kB
SwapCached:            0 kB
Active:           332372 kB
Inactive:         288672 kB
Active(anon):     222504 kB
Inactive(anon):   227660 kB
Active(file):     109868 kB
Inactive(file):    61012 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        102396 kB
SwapFree:         102396 kB
Dirty:                 4 kB
Writeback:             0 kB
AnonPages:        221076 kB
Mapped:           283740 kB
Shmem:            227904 kB
Slab:              24044 kB
SReclaimable:      15620 kB
SUnreclaim:         8424 kB
KernelStack:        1472 kB
PageTables:         3544 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      543792 kB
Committed_AS:    1631120 kB
VmallocTotal:    1179648 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:           8192 kB
CmaFree:            3728 kB
I know the math behind this, that's why I've reduced buffer to 20 and bpp to 24 (knowing that this prevents the 32bits alignments but my guess was that RAM was more precious than some more cpu cycles, maybe I was wrong)

The fps, as said, is 2, not set in camera as I'm not receiving the h264 stream (I tried but that's not so reliable... sometimes the frame is corrupted and that generates an alarm everytime. That's well known to Foscam users :( ) so I used the picture snapshots via http. That's slower but works better. (I was running the same camera config for one year using 1.28)

It seems that raspberry doesn't support SSE, this is my /proc/cpuinfo

Code: Select all

processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 57.60
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 1
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 57.60
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 2
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 57.60
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 3
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 57.60
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

Hardware        : BCM2709
Revision        : a01041
Serial          : 00000000dbe46f95
Should I test images at 32bpp anyway?

The cpu load, hm, could htop tell you that info? If it does here it is:
Image

I know about usb bottleneck of the raspberry but I'm using from ever a self powered USB SSD (I've set max_usb_current=1 in /boot/config.txt and it's working well).
I know that's not optimal but saving events @ 2fps should not be a problem on a SSD, even with raspberry USB bottleneck, right? That worked well for the past year.

Oh my cam are NOT connected with wifi :) they are all wired ethernet, connected to various switches. Network connection isn't a issue.

I enabled a level 5 files debug and I have the files on my PC, if you wants me to look at something. Or I can produce another one, just tell me what you want to know.
In the meanwhile... THANK YOU SO MUCH for your valuable time!!!

Best regards,
Henrik
bonsainuser
Posts: 2
Joined: Thu Jul 07, 2016 8:52 am

Re: ZM 1.29.0 on Raspberry Pi 2 stop receiving IP cameras!

Post by bonsainuser »

Hi all,
we had the same problem but it was solved with a good power supply.I mean one is working stable.
do you check this?

cheers
Locked