Many errors in log. ERR Socket /run/zm/zms-does not exist

Discussions related to the 1.36.x series of ZoneMinder
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

A huge number of errors appear in the logs - 80+ thousand in a few days for the last time for example.
Linux Debian OS. Originally there was version 1.34, then updated to 1.36.14, now 1.36.15.
022-05-26 15:08:17 web_php 20113 ERR Socket /run/zm/zms-283538s.sock does not exist. This file is created by sms, and since it does not exist, either ms did not run, or zms exited early. Please check your sms logs and ensure that SHE is enabled in apache and check that the PATH_KS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (mc) is not running the sims will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information. /usr/share/zoneminder/www/includes/functions.php 1907
5 ip 2MP hikvision rtsp 1080p 25 frames + 1 rtsp from rvi dvr
ffmpeg, modest, rtsp tcp, 1080p,
h264 camera pass through, JPEG disabled.
Buffer 3; max 0; I threw processors and memory with a margin - it doesn't help (virtual machine).

dmesg |tail
[ 6004.837063] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/zoneminder.service,task=zmc,pid=19181,uid=33
[ 6004.837123] Out of memory: Killed process 19181 (zmc) total-vm:6679792kB, anon-rss:5955848kB, file-rss:0kB, shmem-rss:24308kB, UID:33 pgtables:12176kB oom_score_adj:0.

After the update, new mass errors:
2022-05-26 15:35:55 zms_m 17 19355 ERROR sdm monitor is not connected zm_stream.cpp 78
2022-05-26 15:35:55 Error zms_m 18 20147 Could not connect to monitor id 18 for streaming zm_stream.cpp 60
2022-05-26 15:35:55 zms_m18 20147 ERROR Cannot open the memory card file /dev/shm/zm.map.18: There is no such file or directory zm_monitor.cpp 927

What can be done, what can be added for diagnostics?

upd: 60 thousand error messages per day
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by iconnor »

1.36.16 should fix this. Packages building.
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

updated to 1.36.16, errors are pouring in - 3500 in an hour and a half

Image

upd: already 9500 errors in 3 hours
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by iconnor »

One of your monitors is not running when it should be. Why? Check it's log.
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

All monitors are working, only constantly turns off and starts up again and I can't figure out why. Buffer settings without restrictions, could this be the reason? I tried adding memory, now for 6 cameras 20 GB, set up to 24 GB, then returned 20. upd: made 24GB again

In the first message I wrote about out of memory

dmesg |tail -n 30
[357883.117368] [ 146102] 33 146102 60976 131 163840 1767 0 apache2
[357883.117372] [ 146103] 33 146103 60988 761 163840 1457 0 apache2
[357883.117376] [ 146110] 33 146110 60974 808 163840 1435 0 apache2
[357883.117380] [ 146820] 33 146820 89109 8893 344064 4199 0 nph-zms
[357883.117383] [ 147076] 33 147076 60978 267 163840 1740 0 apache2
[357883.117387] [ 147088] 33 147088 60974 820 163840 1433 0 apache2
[357883.117390] [ 147111] 33 147111 89109 7503 331776 4185 0 nph-zms
[357883.117395] [ 148765] 33 148765 60992 817 163840 1440 0 apache2
[357883.117407] [ 148801] 33 148801 60974 769 163840 1435 0 apache2
[357883.117412] [ 148805] 33 148805 60974 781 163840 1433 0 apache2
[357883.117416] [ 148806] 33 148806 89109 9902 344064 2634 0 nph-zms
[357883.117420] [ 148808] 33 148808 89109 7967 339968 4688 0 nph-zms
[357883.117424] [ 150208] 33 150208 89488 12151 331776 1340 0 nph-zms
[357883.117428] [ 150221] 33 150221 60972 933 163840 1318 0 apache2
[357883.117431] [ 150224] 33 150224 89109 11413 348160 1170 0 nph-zms
[357883.117435] [ 150225] 33 150225 60974 930 163840 1325 0 apache2
[357883.117438] [ 150382] 33 150382 6971 2661 94208 410 0 zmdc.pl
[357883.117442] [ 150410] 33 150410 1635515 1394833 12320768 83624 0 zmc
[357883.117446] [ 150415] 33 150415 1995698 1740139 14970880 63272 0 zmc
[357883.117449] [ 150420] 33 150420 374772 296170 2695168 1676 0 zmc
[357883.117452] [ 150425] 33 150425 603707 504886 4407296 2529 0 zmc
[357883.117456] [ 150430] 33 150430 604560 501794 4370432 1375 0 zmc
[357883.117460] [ 150439] 33 150439 608223 516291 4485120 1266 0 zmc
[357883.117463] [ 150444] 33 150444 10554 2686 118784 3924 0 zmfilter.pl
[357883.117466] [ 150449] 33 150449 10542 4882 118784 1716 0 zmfilter.pl
[357883.117470] [ 150457] 33 150457 6810 2934 94208 0 0 zmwatch.pl
[357883.117473] [ 150469] 33 150469 7903 3973 98304 0 0 zmupdate.pl
[357883.117476] [ 150473] 33 150473 6769 2871 90112 0 0 zmstats.pl
[357883.117489] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/zoneminder.service,task=zmc,pid=150415,uid=33
[357883.117570] Out of memory: Killed process 150415 (zmc) total-vm:7982792kB, anon-rss:6936248kB, file-rss:0kB, shmem-rss:24308kB, UID:33 pgtables:14620kB oom_score_adj:0
Last edited by leksand on Mon May 30, 2022 3:11 pm, edited 3 times in total.
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

So many mistakes for today in 6 hours

Image

upd: in half hour + 8000
upd2: 233000 errors per day

What else can be applied for a more accurate diagnosis?
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

The problem is still relevant. Should I write to another place or will they do it? I am ready to help with diagnostics and information collection
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

Updated to 1.36.17. Cleared logs. Monitor dumps again. Again Out of memory: Killed process

dmesg |tail -n 30
[ 281.960838] [ 672] 33 672 89109 12508 344064 76 0 nph-zms
[ 281.960840] [ 678] 33 678 60958 2552 155648 249 0 apache2
[ 281.960843] [ 693] 33 693 89109 10730 331776 1859 0 nph-zms
[ 281.960845] [ 695] 33 695 60958 2549 155648 267 0 apache2
[ 281.960854] [ 696] 33 696 60958 2576 155648 264 0 apache2
[ 281.960857] [ 726] 33 726 89109 11453 335872 1132 0 nph-zms
[ 281.960859] [ 752] 33 752 89109 12553 344064 539 0 nph-zms
[ 281.960867] [ 764] 33 764 60958 1634 155648 251 0 apache2
[ 281.960870] [ 769] 0 769 3676 304 61440 7 0 sshd
[ 281.960873] [ 772] 1000 772 3845 296 69632 0 0 systemd
[ 281.960875] [ 773] 1000 773 41734 483 90112 180 0 (sd-pam)
[ 281.960877] [ 790] 1000 790 3729 275 65536 38 0 sshd
[ 281.960879] [ 791] 1000 791 2020 348 49152 0 0 bash
[ 281.960882] [ 801] 33 801 60958 2599 155648 249 0 apache2
[ 281.960884] [ 874] 33 874 60958 2517 155648 249 0 apache2
[ 281.960886] [ 909] 33 909 60958 2485 155648 249 0 apache2
[ 281.960888] [ 1081] 33 1081 6969 315 98304 2755 0 zmdc.pl
[ 281.960891] [ 1109] 33 1109 1328219 1170836 10088448 48617 0 zmc
[ 281.960893] [ 1114] 33 1114 1514047 1323168 11649024 88208 0 zmc
[ 281.960895] [ 1119] 33 1119 407689 299098 2801664 11744 0 zmc
[ 281.960898] [ 1124] 33 1124 586326 452853 4374528 49316 0 zmc
[ 281.960900] [ 1131] 33 1131 2423349 2233753 18587648 20053 0 zmc
[ 281.960903] [ 1138] 33 1138 619179 522167 4550656 1532 0 zmc
[ 281.960905] [ 1143] 33 1143 10543 916 118784 5695 0 zmfilter.pl
[ 281.960907] [ 1149] 33 1149 10538 902 122880 5700 0 zmfilter.pl
[ 281.960910] [ 1159] 33 1159 6833 678 94208 2260 0 zmwatch.pl
[ 281.960912] [ 1166] 33 1166 7901 3927 106496 45 0 zmupdate.pl
[ 281.960914] [ 1171] 33 1171 6764 2868 98304 1 0 zmstats.pl
[ 281.960926] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/zoneminder.service,task=zmc,pid=1131,uid=33
[ 281.961010] Out of memory: Killed process 1131 (zmc) total-vm:9693396kB, anon-rss:8910704kB, file-rss:0kB, shmem-rss:24308kB, UID:33 pgtables:18152kB oom_score_adj:0

Image
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

10 cores, 24 GB RAM for 6 cameras (virtual machine, debian)
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by iconnor »

If you don';t want to go OOM, set a MaxIMageBuffer value.

There will be other errors logged but at least they'll be different and might offer clues.
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

The video ends on a timer (EVENT_CLOSE_MODE=time). Does this overlap with the buffer settings?

What values should I set so that there is no error due to a small buffer?
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by iconnor »

Given what is happening I don't think there is any value that will do that.

You are hitting the end of the buffer because something can't clear it fast enough.

It COULD be that your cameras have a MASSIVE keyframe interval. Ideally the buffer would be a bit larger than 2*keyframe interval. I have seen cameras that goes thousands of frames between keyframes.(And the UI is so broken we can't configure them).
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

Changed the buffer settings

Image

5 hikvision DS-2CD2122FWD-IS cameras and 1 stream with nvr RVi-HDR04LA-M
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

The current load is ~80-90% of 10 vCPUs and 50-60% of 24 GB RAM.
What can I change in the settings to reduce the load? (6 modect monitors)

The current load is ~80% of 10 vCPUs and 50% of 24 GB RAM.
What can I change in the settings to reduce the load? (6 modect monitors)
Saw on the graph - when there were monitor crashes before changing the buffer settings. I raised the values due to the fact that the recording stopped even though there was movement. In another topic, I was given advice to change the Alarm Reference Image Blend %gel settings and reduced all values in the zone settings from the best high sensitivity preset by 2 times - the recording goes fine

Image
leksand
Posts: 48
Joined: Thu May 26, 2022 12:03 pm

Re: Many errors in log. ERR Socket /run/zm/zms-does not exist

Post by leksand »

Now the video of the events does not open or opens but stands still (according to the time stamp from the camera) and other errors (because of this, I just increased the buffer size)

Image
Post Reply