/dev/shm increasing TO 100%

[tommisgr]

Zm version: 1.34.20
14 cameras mixed resolution
720p 8
640x480 4
960 2

Have reduced fps on cameras that support it
on the rest I use "Motion Frame Skip" to reduce cpu usage

Lately the shm keeps rising up to 100%.

I added a /etc/crontab to check it and restart zmpkg.pl when over 85% as a last resort


example on normal operation

df -h /dev/shm

Code: Select all

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           5,9G  2,1G  3,8G  36% /dev/shm 

ls -Shla /dev/shm

Code: Select all

total 2,1G
-rw-------  1 www-data www-data 370M Αυγ  31 21:30 zm.mmap.3
-rw-------  1 www-data www-data 229M Αυγ  31 21:30 zm.mmap.6
-rw-------  1 www-data www-data 211M Αυγ  31 21:30 zm.mmap.1
-rw-------  1 www-data www-data 211M Αυγ  31 21:30 zm.mmap.8
-rw-------  1 www-data www-data 194M Αυγ  31 21:30 zm.mmap.2
-rw-------  1 www-data www-data 194M Αυγ  31 21:30 zm.mmap.5
-rw-------  1 www-data www-data 194M Αυγ  31 21:30 zm.mmap.9
-rw-------  1 www-data www-data 148M Αυγ  31 21:31 zm.mmap.4
-rw-------  1 www-data www-data 106M Αυγ  31 21:30 zm.mmap.7
-rw-------  1 www-data www-data  99M Αυγ  31 21:30 zm.mmap.12
-rw-------  1 www-data www-data  47M Αυγ  31 21:30 zm.mmap.10
-rw-------  1 www-data www-data  47M Αυγ  31 21:30 zm.mmap.11
-rw-------  1 www-data www-data  47M Αυγ  31 21:30 zm.mmap.13
-rw-------  1 www-data www-data  47M Αυγ  31 21:30 zm.mmap.15

lsof | grep /dev/shm

Code: Select all

zmc        9286              www-data  mem       REG               0,26   221190312          3 /dev/shm/zm.mmap.1
zmc        9286              www-data    5u      REG               0,26   221190312          3 /dev/shm/zm.mmap.1
zma        9295              www-data  mem       REG               0,26   221190312          3 /dev/shm/zm.mmap.1
zma        9295              www-data  mem       REG               0,26   110598632          9 /dev/shm/zm.mmap.7
zma        9295              www-data    4u      REG               0,26   221190312          3 /dev/shm/zm.mmap.1
zma        9295              www-data    6u      REG               0,26   110598632          9 /dev/shm/zm.mmap.7
zmc        9300              www-data  mem       REG               0,26   202758232          4 /dev/shm/zm.mmap.2
zmc        9300              www-data    5u      REG               0,26   202758232          4 /dev/shm/zm.mmap.2
zma        9309              www-data  mem       REG               0,26   202758232          4 /dev/shm/zm.mmap.2
zma        9309              www-data  mem       REG               0,26   110598632          9 /dev/shm/zm.mmap.7
zma        9309              www-data  mem       REG               0,26   221190312          3 /dev/shm/zm.mmap.1
zma        9309              www-data    4u      REG               0,26   202758232          4 /dev/shm/zm.mmap.2
zma        9309              www-data    6u      REG               0,26   221190312          3 /dev/shm/zm.mmap.1
zma        9309              www-data    7u      REG               0,26   110598632          9 /dev/shm/zm.mmap.7
zmc        9315              www-data  mem       REG               0,26   387079032          5 /dev/shm/zm.mmap.3
zmc        9315              www-data    5u      REG               0,26   387079032          5 /dev/shm/zm.mmap.3
zmc        9330              www-data  mem       REG               0,26   202758232          7 /dev/shm/zm.mmap.5
zmc        9330              www-data    5u      REG               0,26   202758232          7 /dev/shm/zm.mmap.5
zma        9335              www-data  mem       REG               0,26   202758232          7 /dev/shm/zm.mmap.5
zma        9335              www-data    4u      REG               0,26   202758232          7 /dev/shm/zm.mmap.5
zmc        9346              www-data  mem       REG               0,26   239622392          8 /dev/shm/zm.mmap.6
zmc        9346              www-data    5u      REG               0,26   239622392          8 /dev/shm/zm.mmap.6
zmc        9356              www-data  mem       REG               0,26   110598632          9 /dev/shm/zm.mmap.7
zmc        9356              www-data    5u      REG               0,26   110598632          9 /dev/shm/zm.mmap.7
zma        9357              www-data  mem       REG               0,26   239622392          8 /dev/shm/zm.mmap.6
zma        9357              www-data    4u      REG               0,26   239622392          8 /dev/shm/zm.mmap.6
zma        9362              www-data  mem       REG               0,26   110598632          9 /dev/shm/zm.mmap.7
zma        9362              www-data    4u      REG               0,26   110598632          9 /dev/shm/zm.mmap.7
zmc        9367              www-data  mem       REG               0,26   221190312         10 /dev/shm/zm.mmap.8
zmc        9367              www-data    5u      REG               0,26   221190312         10 /dev/shm/zm.mmap.8
zmc        9377              www-data  mem       REG               0,26   202758232         11 /dev/shm/zm.mmap.9
zmc        9377              www-data    5u      REG               0,26   202758232         11 /dev/shm/zm.mmap.9
zma        9378              www-data  mem       REG               0,26   221190312         10 /dev/shm/zm.mmap.8
zma        9378              www-data    4u      REG               0,26   221190312         10 /dev/shm/zm.mmap.8
zma        9383              www-data  mem       REG               0,26   202758232         11 /dev/shm/zm.mmap.9
zma        9383              www-data    4u      REG               0,26   202758232         11 /dev/shm/zm.mmap.9
zmc        9394              www-data  mem       REG               0,26    49157992         12 /dev/shm/zm.mmap.10
zmc        9394              www-data    5u      REG               0,26    49157992         12 /dev/shm/zm.mmap.10
zma        9399              www-data  mem       REG               0,26    49157992         12 /dev/shm/zm.mmap.10
zma        9399              www-data    4u      REG               0,26    49157992         12 /dev/shm/zm.mmap.10
zmc        9410              www-data  mem       REG               0,26    49157992         13 /dev/shm/zm.mmap.11
zmc        9410              www-data    5u      REG               0,26    49157992         13 /dev/shm/zm.mmap.11
zma        9415              www-data  mem       REG               0,26    49157992         13 /dev/shm/zm.mmap.11
zma        9415              www-data    4u      REG               0,26    49157992         13 /dev/shm/zm.mmap.11
zmc        9426              www-data  mem       REG               0,26   103686152         14 /dev/shm/zm.mmap.12
zmc        9426              www-data    5u      REG               0,26   103686152         14 /dev/shm/zm.mmap.12
zma        9431              www-data  mem       REG               0,26   103686152         14 /dev/shm/zm.mmap.12
zma        9431              www-data    4u      REG               0,26   103686152         14 /dev/shm/zm.mmap.12
zmc        9436              www-data  mem       REG               0,26    49157992         15 /dev/shm/zm.mmap.13
zmc        9436              www-data    5u      REG               0,26    49157992         15 /dev/shm/zm.mmap.13
zma        9441              www-data  mem       REG               0,26    49157992         15 /dev/shm/zm.mmap.13
zma        9441              www-data    4u      REG               0,26    49157992         15 /dev/shm/zm.mmap.13
zmc        9446              www-data  mem       REG               0,26    49157992         16 /dev/shm/zm.mmap.15
zmc        9446              www-data    5u      REG               0,26    49157992         16 /dev/shm/zm.mmap.15
zma        9451              www-data  mem       REG               0,26    49157992         16 /dev/shm/zm.mmap.15
zma        9451              www-data    4u      REG               0,26    49157992         16 /dev/shm/zm.mmap.15
zmc        9534              www-data  mem       REG               0,26   154834824          6 /dev/shm/zm.mmap.4
zmc        9534              www-data    5u      REG               0,26   154834824          6 /dev/shm/zm.mmap.4
zma        9535              www-data  mem       REG               0,26   154834824          6 /dev/shm/zm.mmap.4
zma        9535              www-data    4u      REG               0,26   154834824          6 /dev/shm/zm.mmap.4

even now df usage doesnt match ls usage
when `shm` usage rises `ls` stays the same and `df` rises

Any Ideas on the cause.

[burger]

What does the output of lscpu say? I.e. what hardware are you using? May just need more processing power.

You didn't say what FPS you are running. There's a big difference in CPU use between 30FPS and 6 FPS.

I reboot 1.34 once a week, as there were some memory issues. But I have ~30 cameras.

[tommisgr]

CPU : FX 4300 4core (4treads) 3.9MHz
5-7fps on most of the cameras
4 of them run at 15-20fps (cannot change from camera)
With logging on CPU usage is under 4
when I disable logging usually around 1-2 so no CPU issues.

Are u suggesting I add a cron job for rebooting the machine?

[fontiano]

I've got similar problem on my environment (ryzen 5 2600 with 32Gb RAM) but it happens only if ZM can't get the stream from at least one camera (camera power off, freeze, lose radio bridge etc.).

My /dev/shm is 14GB:
tmpfs 14G 7,7G 6,4G 55% /dev/shm

If a camera goes down zoneminder gets crazy and start filling /dev/shm

Try to control logs and check if before the fill-up event there is a lost of stream from your cameras.

Usually I can see this on my logs when a camera has problems:

zma_mX[XXXX]: ERR [zma_mX] [Shared data not initialised by capture daemon for monitor XXXXXX]

or

zmc_mX[XXXX]: ERR [zmc_mX] [Unable to open input rtsp://xxxxx:xxxxx@10.x.x.x:554/1 due to: No route to host]

I think it's a bug.

As workaround we can made a cron script that disable cameras when they're offline and re-enable them when they're online again but it would be great if the developers could fix it in the zoneminder code

[tommisgr]

3 of the cameras are wifi only (2 of which old phones running IPcam), so they loose connection at times.
Logs do go crazy when that happens but couldn't pin point it as the cause of /dev/shm filling up.

How do u turn them of and back on? (states or scripting)
If scripting can u please give me a starting point.

Isn't there a way to delete files not used in /dev/shm ?

also the "df -h /dev/shm" does not equal the "ls -la /dev/shm". The former (df) is bigger.

Any ideas on that?

some errors form syslog

Code: Select all

Sep  1 19:02:17 ZM-Mint19 zmc_m7[26973]: ERR [zmc_m7] [Unable to read packet from stream 0: error -1094995529 "Invalid data found when processing input".]
Sep  1 19:11:08 ZM-Mint19 zma_m7[28055]: ERR [zma_m7] [Got empty memory map file size 0, is the zmc process for this monitor running?]
Sep  1 19:11:08 ZM-Mint19 zmdc[11287]: ERR ['zma -m 7' exited abnormally, exit status 255]
Sep  1 19:35:17 ZM-Mint19 zmc_m7[28056]: ERR [zmc_m7] [Unable to read packet from stream 0: error -1094995529 "Invalid data found when processing input".]
Sep  1 19:39:36 ZM-Mint19 zmc_m7[28056]: ERR [zmc_m7] [Unable to read packet from stream 0: error -1094995529 "Invalid data found when processing input".]
Sep  1 19:49:20 ZM-Mint19 zmc_m7[28056]: ERR [zmc_m7] [Unable to read packet from stream 0: error -1094995529 "Invalid data found when processing input".]
Sep  1 20:07:26 ZM-Mint19 zmc_m3[11344]: ERR [zmc_m3] [Unable to send packet Invalid data found when processing input, continuing]
Sep  1 20:30:09 ZM-Mint19 zma_m13[2280]: ERR [zma_m13] [Got empty memory map file size 0, is the zmc process for this monitor running?]
Sep  1 20:30:09 ZM-Mint19 zmdc[11287]: ERR ['zma -m 13' exited abnormally, exit status 255]
Sep  1 20:30:26 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Unable to open input rtsp://xxxxx:xxxxx@10.10.xxxxx:554/video.mp4 due to: No route to host]
Sep  1 20:30:39 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Unable to open input rtsp://xxxxx:xxxxx@10.10.xxxxx:554/video.mp4 due to: No route to host]
Sep  1 20:30:39 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Failed to prime capture of initial monitor]
Sep  1 20:30:52 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Unable to open input rtsp://xxxxx:xxxxx@10.10.xxxxx:554/video.mp4 due to: No route to host]
Sep  1 20:30:52 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Failed to prime capture of initial monitor]
Sep  1 20:33:36 ZM-Mint19 zmc_m6[11382]: ERR [zmc_m6] [Unable to send packet Invalid data found when processing input, continuing]
Sep  1 20:33:36 ZM-Mint19 zmc_m6[11382]: ERR [zmc_m6] [Unable to send packet Invalid data found when processing input, continuing]

[burger]

I have seen CPU use go up from cameras disconnecting, but I have not noticed /dev/shm maxing out. I have cameras disconnect occasionally, and I've not needed to run any such scripts.

This is why choosing a good brand of cameras, and using proper setup (i.e. the forums here generally recommend to avoid wireless) is important. If you have cheap cameras, you get flaky compatibility, and things don't work 100%. Wireless should technically work, though. But RF is unpredictable. Lots of things can interfere.

[iconnor]

The space differences are because there are deleted files still held open by some process.

sudo lsof | grep DEL | grep /dev/shm/zm

Should tell us which process is the culprit.

[tommisgr]

sudo lsof | grep DEL | grep /dev/shm/zm

Code: Select all

  
zma        4263              www-data  DEL       REG               0,26                      9 /dev/shm/zm.mmap.7
zma        4272              www-data  DEL       REG               0,26                     23 /dev/shm/zm.mmap.7
zma        4272              www-data  DEL       REG               0,26                     22 /dev/shm/zm.mmap.7
zma        4272              www-data  DEL       REG               0,26                     21 /dev/shm/zm.mmap.7
zma        4272              www-data  DEL       REG               0,26                     18 /dev/shm/zm.mmap.7
zma        4272              www-data  DEL       REG               0,26                     17 /dev/shm/zm.mmap.7

Now how can I delete them automatically?
Are they left there by a bug or some miss-configuration on my part?

As mentioned on top I implemented a cron job restarting zm whenever shm goes over 85% (runs every 30 mins)
I also change state morning and night, which also resets the shm to normal values (36-41%), but it keeps rising slowly
The script does a restart almost once a day in addition to the 2 state changes daily.


Would rather find the problem before it gets out of hand => continuous restarts of zm.

[burger]

Are u suggesting I add a cron job for rebooting the machine?

I have a cron job that reboots once a week (I could probably do so once every other week). But once a day is unnecessary, and restarting ZM every 30 minutes, is insane.

Try to narrow down which camera is causing /dev/shm to fill up, and either troubleshoot, or just replace it with a good known camera. If it's just one camera, you can turn off that camera. Set monitor to none, and remove check from checkbox so it's disabled.

I.e., which monitor # is 4272. Fix that one.

[fontiano]

3 of the cameras are wifi only (2 of which old phones running IPcam), so they loose connection at times.
Logs do go crazy when that happens but couldn't pin point it as the cause of /dev/shm filling up.

How do u turn them of and back on? (states or scripting)
If scripting can u please give me a starting point.

Isn't there a way to delete files not used in /dev/shm ?

also the "df -h /dev/shm" does not equal the "ls -la /dev/shm". The former (df) is bigger.

Any ideas on that?

some errors form syslog

Code: Select all

Sep  1 19:02:17 ZM-Mint19 zmc_m7[26973]: ERR [zmc_m7] [Unable to read packet from stream 0: error -1094995529 "Invalid data found when processing input".]
Sep  1 19:11:08 ZM-Mint19 zma_m7[28055]: ERR [zma_m7] [Got empty memory map file size 0, is the zmc process for this monitor running?]
Sep  1 19:11:08 ZM-Mint19 zmdc[11287]: ERR ['zma -m 7' exited abnormally, exit status 255]
Sep  1 19:35:17 ZM-Mint19 zmc_m7[28056]: ERR [zmc_m7] [Unable to read packet from stream 0: error -1094995529 "Invalid data found when processing input".]
Sep  1 19:39:36 ZM-Mint19 zmc_m7[28056]: ERR [zmc_m7] [Unable to read packet from stream 0: error -1094995529 "Invalid data found when processing input".]
Sep  1 19:49:20 ZM-Mint19 zmc_m7[28056]: ERR [zmc_m7] [Unable to read packet from stream 0: error -1094995529 "Invalid data found when processing input".]
Sep  1 20:07:26 ZM-Mint19 zmc_m3[11344]: ERR [zmc_m3] [Unable to send packet Invalid data found when processing input, continuing]
Sep  1 20:30:09 ZM-Mint19 zma_m13[2280]: ERR [zma_m13] [Got empty memory map file size 0, is the zmc process for this monitor running?]
Sep  1 20:30:09 ZM-Mint19 zmdc[11287]: ERR ['zma -m 13' exited abnormally, exit status 255]
Sep  1 20:30:26 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Unable to open input rtsp://xxxxx:xxxxx@10.10.xxxxx:554/video.mp4 due to: No route to host]
Sep  1 20:30:39 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Unable to open input rtsp://xxxxx:xxxxx@10.10.xxxxx:554/video.mp4 due to: No route to host]
Sep  1 20:30:39 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Failed to prime capture of initial monitor]
Sep  1 20:30:52 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Unable to open input rtsp://xxxxx:xxxxx@10.10.xxxxx:554/video.mp4 due to: No route to host]
Sep  1 20:30:52 ZM-Mint19 zmc_m13[2281]: ERR [zmc_m13] [Failed to prime capture of initial monitor]
Sep  1 20:33:36 ZM-Mint19 zmc_m6[11382]: ERR [zmc_m6] [Unable to send packet Invalid data found when processing input, continuing]
Sep  1 20:33:36 ZM-Mint19 zmc_m6[11382]: ERR [zmc_m6] [Unable to send packet Invalid data found when processing input, continuing]

sorry, I wanted to write "we should write a script that disables cameras that lose signal" as I haven't done so yet.

It would be cool if the script did a test on the stream before reactivating a monitor

Here is the documentation for making changes to monitors via API https://zoneminder.readthedocs.io/en/stable/api.html

[tommisgr]

As it stands now /dev/shm rises about 10% every hour which means every 6 hours zoneminder restarts via script which checks /dev/shm usage %.

But I also do a state change every 12 hours or so same affect as a restart.
will look into the monitor the remains in memory.

[iconnor]

I would have bet that zma was solid in this respect, but the evidence is clear, it is not.

Is zmc exiting or crashing?

Your cron job could probably just do sudo zmdc.pl restart zma -m 7

[tommisgr]

Changed the offending cameras source path from a http to a rtsp stream and its more stable now.
Does a hiccup every now and then but shm does not fill up.

My 2 daily state changes day/Night are enough to reset shm size.
Keeping my shm size check script as a safety.

Thanks for all the help and suggestions.

Issue remains but under control.