Hi! I've been using Zoneminder at several locations for several years. I have had upgrade issues before, but never like this.
After upgrading a Centos8 box to 1.36.12, all of the cameras stopped capturing. The past events are still there, no other issues are evident with zoneminder. Here's a snippet of errors from /var/log/messages:
Jan 1 00:35:46 DVRBox zmdc[3364]: ERR ['zmc -m 3' exited abnormally, exit status 6]
Jan 1 00:35:47 DVRBox zmdc[3364]: ERR ['zmc -m 5' exited abnormally, exit status 6]
Jan 1 00:35:47 DVRBox zmdc[3364]: ERR ['zmc -m 2' exited abnormally, exit status 6]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Got signal 6 (Aborted), crashing]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Signal address is 0x3000000e13, from 0x7ff18983037f]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 0: /usr/bin/zmc(+0xd1431) [0x556751cf0431]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 1: /lib64/libpthread.so.0(+0x12c20) [0x7ff18cf8ac20]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 2: /lib64/libc.so.6(gsignal+0x10f) [0x7ff18983037f]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 3: /lib64/libc.so.6(abort+0x127) [0x7ff18981adb5]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 4: /usr/bin/zmc(+0x55118) [0x556751c74118]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 5: /usr/bin/zmc(+0xbaf17) [0x556751cd9f17]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 6: /usr/bin/zmc(+0xbc3a8) [0x556751cdb3a8]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 7: /lib64/libstdc++.so.6(+0xc2ba3) [0x7ff18a21aba3]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 8: /lib64/libpthread.so.0(+0x817a) [0x7ff18cf8017a]]
Jan 1 00:35:48 DVRBox zmc_m4[3603]: ERR [zmc_m4] [Backtrace 9: /lib64/libc.so.6(clone+0x43) [0x7ff1898f5dc3]]
I've never had a problem like this. Most of my previous ZM upgrade problems required running ZMupdate.pl or fixing the apache conf files.
SELinux is still disabled.
Is anyone else seeing this?
Thanks!
ZMC processes dying after upgrade to 1.36.12
Re: ZMC processes dying after upgrade to 1.36.12
No.
There must be some errors before that.
Make sure all ZM processes have stopped. Restart Apache. Then restart your DB. Then try starting ZM. See what comes up.
There must be some errors before that.
Make sure all ZM processes have stopped. Restart Apache. Then restart your DB. Then try starting ZM. See what comes up.
-
Re: ZMC processes dying after upgrade to 1.36.12
Magic919, thanks for your response.
I restarted mariadb and apache, and then started zoneminder. Here are the starting entries in /var/log/messages:
Jan 1 10:07:02 DVRBox systemd[1]: Started The Apache HTTP Server.
Jan 1 10:07:02 DVRBox httpd[9024]: Server configured, listening on: port 80
Jan 1 10:07:20 DVRBox systemd[1]: Starting ZoneMinder CCTV recording and security system...
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Got signal 6 (Aborted), crashing]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Signal address is 0x3000002449, from 0x7f805658937f]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 0: /usr/bin/zmc(+0xd1431) [0x5576a834e431]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 1: /lib64/libpthread.so.0(+0x12c20) [0x7f8059ce3c20]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 2: /lib64/libc.so.6(gsignal+0x10f) [0x7f805658937f]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 3: /lib64/libc.so.6(abort+0x127) [0x7f8056573db5]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 4: /usr/bin/zmc(+0x55118) [0x5576a82d2118]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 5: /usr/bin/zmc(+0xbaf17) [0x5576a8337f17]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 6: /usr/bin/zmc(+0xbc3a8) [0x5576a83393a8]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 7: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f8056f73ba3]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 8: /lib64/libpthread.so.0(+0x817a) [0x7f8059cd917a]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 9: /lib64/libc.so.6(clone+0x43) [0x7f805664edc3]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Got signal 6 (Aborted), crashing]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Signal address is 0x300000244f, from 0x7f790b6d537f]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 0: /usr/bin/zmc(+0xd1431) [0x561f6cd84431]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 1: /lib64/libpthread.so.0(+0x12c20) [0x7f790ee2fc20]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 2: /lib64/libc.so.6(gsignal+0x10f) [0x7f790b6d537f]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 3: /lib64/libc.so.6(abort+0x127) [0x7f790b6bfdb5]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 4: /usr/bin/zmc(+0x55118) [0x561f6cd08118]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 5: /usr/bin/zmc(+0xbaf17) [0x561f6cd6df17]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 6: /usr/bin/zmc(+0xbc3a8) [0x561f6cd6f3a8]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 7: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f790c0bfba3]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 8: /lib64/libpthread.so.0(+0x817a) [0x7f790ee2517a]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 9: /lib64/libc.so.6(clone+0x43) [0x7f790b79adc3]]
These, however, are only from the very general /var/log/messages log. There are many others in the /var/log/zoneminder folder.
Here is the zmdc log, which is very different:
/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::
size_type) [with _Tp = char; _Alloc = std::allocator<char>; std::vector<_Tp, _Alloc>::reference = char&; std::vector<_Tp, _Alloc>::size_type
= long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
01/01/2022 10:07:23.151850 zmdc[9261].ERR [ZMServer:718] ['zmc -m 2' exited abnormally, exit status 6]
/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::
size_type) [with _Tp = char; _Alloc = std::allocator<char>; std::vector<_Tp, _Alloc>::reference = char&; std::vector<_Tp, _Alloc>::size_type
= long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
01/01/2022 10:07:23.240760 zmdc[9261].ERR [ZMServer:718] ['zmc -m 3' exited abnormally, exit status 6]
/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::
size_type) [with _Tp = char; _Alloc = std::allocator<char>; std::vector<_Tp, _Alloc>::reference = char&; std::vector<_Tp, _Alloc>::size_type
= long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
01/01/2022 10:07:23.405250 zmdc[9261].ERR [ZMServer:718] ['zmc -m 12' exited abnormally, exit status 6]
These entries look to me, as if the C++ code is using the wrong variable type.
The web_js log contains entries like:
12/31/21 23:00:45.789553 web_js[1018].ERR [10.39.128.117] [getStreamCmdResponse stream error: Socket /var/lib/zoneminder/sock/zms-593045s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information. - checkStreamForErrors()] at ?view=watch line
12/31/21 23:00:48.787348 web_js[1022].ERR [10.39.128.117] [getStreamCmdResponse stream error: Socket /var/lib/zoneminder/sock/zms-563370s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information. - checkStreamForErrors()] at ?view=watch line
31/12/21 23:17:49.121230 web_js[6780].ERR [10.39.128.205] [getStreamCmdResponse%20stream%20error%3A%20No%20data%20to%20read%20from%20socket%20-%20checkStreamForErrors()] at ?view=watch&mid=3 line
31/12/21 23:17:50.539274 web_js[1048].ERR [10.39.128.205] [getStreamCmdResponse%20stream%20error%3A%20socket_bind(%20%2Fvar%2Flib%2Fzoneminder%2Fsock%2Fzms-077149w.sock%20)%20failed%3A%20Address%20already%20in%20use%20-%20checkStreamForErrors()] at ?view=watch&mid=3 line
Of course, I checked /etc/zm/conf.d in the file 01-system-paths.conf and found the lines :
# ZoneMinder url path to the zms streaming server
ZM_PATH_ZMS=/cgi-bin-zm/nph-zms
Which seem to be correct.
Any ideas?
I restarted mariadb and apache, and then started zoneminder. Here are the starting entries in /var/log/messages:
Jan 1 10:07:02 DVRBox systemd[1]: Started The Apache HTTP Server.
Jan 1 10:07:02 DVRBox httpd[9024]: Server configured, listening on: port 80
Jan 1 10:07:20 DVRBox systemd[1]: Starting ZoneMinder CCTV recording and security system...
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Got signal 6 (Aborted), crashing]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Signal address is 0x3000002449, from 0x7f805658937f]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 0: /usr/bin/zmc(+0xd1431) [0x5576a834e431]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 1: /lib64/libpthread.so.0(+0x12c20) [0x7f8059ce3c20]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 2: /lib64/libc.so.6(gsignal+0x10f) [0x7f805658937f]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 3: /lib64/libc.so.6(abort+0x127) [0x7f8056573db5]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 4: /usr/bin/zmc(+0x55118) [0x5576a82d2118]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 5: /usr/bin/zmc(+0xbaf17) [0x5576a8337f17]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 6: /usr/bin/zmc(+0xbc3a8) [0x5576a83393a8]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 7: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f8056f73ba3]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 8: /lib64/libpthread.so.0(+0x817a) [0x7f8059cd917a]]
Jan 1 10:07:23 DVRBox zmc_m2[9289]: ERR [zmc_m2] [Backtrace 9: /lib64/libc.so.6(clone+0x43) [0x7f805664edc3]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Got signal 6 (Aborted), crashing]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Signal address is 0x300000244f, from 0x7f790b6d537f]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 0: /usr/bin/zmc(+0xd1431) [0x561f6cd84431]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 1: /lib64/libpthread.so.0(+0x12c20) [0x7f790ee2fc20]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 2: /lib64/libc.so.6(gsignal+0x10f) [0x7f790b6d537f]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 3: /lib64/libc.so.6(abort+0x127) [0x7f790b6bfdb5]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 4: /usr/bin/zmc(+0x55118) [0x561f6cd08118]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 5: /usr/bin/zmc(+0xbaf17) [0x561f6cd6df17]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 6: /usr/bin/zmc(+0xbc3a8) [0x561f6cd6f3a8]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 7: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f790c0bfba3]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 8: /lib64/libpthread.so.0(+0x817a) [0x7f790ee2517a]]
Jan 1 10:07:23 DVRBox zmc_m3[9295]: ERR [zmc_m3] [Backtrace 9: /lib64/libc.so.6(clone+0x43) [0x7f790b79adc3]]
These, however, are only from the very general /var/log/messages log. There are many others in the /var/log/zoneminder folder.
Here is the zmdc log, which is very different:
/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::
size_type) [with _Tp = char; _Alloc = std::allocator<char>; std::vector<_Tp, _Alloc>::reference = char&; std::vector<_Tp, _Alloc>::size_type
= long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
01/01/2022 10:07:23.151850 zmdc[9261].ERR [ZMServer:718] ['zmc -m 2' exited abnormally, exit status 6]
/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::
size_type) [with _Tp = char; _Alloc = std::allocator<char>; std::vector<_Tp, _Alloc>::reference = char&; std::vector<_Tp, _Alloc>::size_type
= long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
01/01/2022 10:07:23.240760 zmdc[9261].ERR [ZMServer:718] ['zmc -m 3' exited abnormally, exit status 6]
/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::
size_type) [with _Tp = char; _Alloc = std::allocator<char>; std::vector<_Tp, _Alloc>::reference = char&; std::vector<_Tp, _Alloc>::size_type
= long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
01/01/2022 10:07:23.405250 zmdc[9261].ERR [ZMServer:718] ['zmc -m 12' exited abnormally, exit status 6]
These entries look to me, as if the C++ code is using the wrong variable type.
The web_js log contains entries like:
12/31/21 23:00:45.789553 web_js[1018].ERR [10.39.128.117] [getStreamCmdResponse stream error: Socket /var/lib/zoneminder/sock/zms-593045s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information. - checkStreamForErrors()] at ?view=watch line
12/31/21 23:00:48.787348 web_js[1022].ERR [10.39.128.117] [getStreamCmdResponse stream error: Socket /var/lib/zoneminder/sock/zms-563370s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information. - checkStreamForErrors()] at ?view=watch line
31/12/21 23:17:49.121230 web_js[6780].ERR [10.39.128.205] [getStreamCmdResponse%20stream%20error%3A%20No%20data%20to%20read%20from%20socket%20-%20checkStreamForErrors()] at ?view=watch&mid=3 line
31/12/21 23:17:50.539274 web_js[1048].ERR [10.39.128.205] [getStreamCmdResponse%20stream%20error%3A%20socket_bind(%20%2Fvar%2Flib%2Fzoneminder%2Fsock%2Fzms-077149w.sock%20)%20failed%3A%20Address%20already%20in%20use%20-%20checkStreamForErrors()] at ?view=watch&mid=3 line
Of course, I checked /etc/zm/conf.d in the file 01-system-paths.conf and found the lines :
# ZoneMinder url path to the zms streaming server
ZM_PATH_ZMS=/cgi-bin-zm/nph-zms
Which seem to be correct.
Any ideas?
Re: ZMC processes dying after upgrade to 1.36.12
Yeah that's pretty not good.
Unfortunately those logs don't tell us where the problem is, so we need to narrow it down. We need you to turn on debug logging, which hopefully will give us a rough idea where in the code it is dying.
Options->Logging->ZM_DEBUG
Best to go to level 3, but even level 1 will give us lots more info.
Unfortunately those logs don't tell us where the problem is, so we need to narrow it down. We need you to turn on debug logging, which hopefully will give us a rough idea where in the code it is dying.
Options->Logging->ZM_DEBUG
Best to go to level 3, but even level 1 will give us lots more info.
Re: ZMC processes dying after upgrade to 1.36.12
Thank you for the instructions.
I checked debug_log, set to level 3 and specified a log file location.
There are a LOT of entries here. I think this is a chunk that may allow you to see what is happening:
01/01/22 15:10:32.093754 zmc_m9[16103].DB1-zm_logger.cpp/232 [LogOpts: level=DB3 effective=DB3, screen=OFF, database=ERR, logfile=DB3->/var/log/zoneminder/debug, syslog=ERR]
01/01/22 15:10:32.093789 zmc_m9[16103].DB1-zm_utils.cpp/198 [Detected a x86\x86-64 processor with AVX2]
01/01/22 15:10:32.094182 zmc_m9[16103].DB1-zm_storage.cpp/91 [No id passed to Storage constructor. Using default path /var/lib/zoneminder/events instead]
01/01/22 15:10:32.094191 zmc_m9[16103].DB1-zm_monitor.cpp/493 [Have camera type Local]
01/01/22 15:10:32.094194 zmc_m9[16103].DB1-zm_monitor.cpp/2336 [Reloading linked monitors for monitor WarehouseRight, '(null)']
01/01/22 15:10:32.094198 zmc_m9[16103].DB3-zm_monitor.cpp/570 [Decoding enabled: 1 function 5 Mocord savejpegs 1 videowriter 0]
01/01/22 15:10:32.094211 zmc_m9[16103].DB1-zm_packetqueue.cpp/660 [Setting pre_event_video_packet_count to 5]
01/01/22 15:10:32.094215 zmc_m9[16103].DB1-zm_packetqueue.cpp/654 [Setting max_video_packet_count to 20]
01/01/22 15:10:32.094218 zmc_m9[16103].DB1-zm_monitor.cpp/668 [mem.size(8) SharedData=760 TriggerData=560 VideoStoreData=4128 timestamps=48 images=3x1440000 = 4320000 total=4325560]
01/01/22 15:10:32.094229 zmc_m9[16103].DB2-zm_camera.cpp/69 [New camera id: 9 width: 800 line size: 2400 height: 600 colours: 3 subpixelorder: 6 capture: 1]
01/01/22 15:10:32.094241 zmc_m9[16103].DB1-zm_remote_camera.cpp/111 [1 addresses returned]
01/01/22 15:10:32.094248 zmc_m9[16103].DB1-zm_ffmpeg.cpp/86 [Not enabling ffmpeg logs, as LOG_FFMPEG and/or LOG_DEBUG is disabled in options, or this monitor is not part of your debug targets]
01/01/22 15:10:32.094265 zmc_m9[16103].DB2-zm_rtsp.cpp/169 [RTSP Local SSRC is 9480198, url is rtsp://192.168.0.113:7070/]
01/01/22 15:10:32.094270 zmc_m9[16103].DB2-zm_rtsp.cpp/176 [# of auth parts 2]
01/01/22 15:10:32.094308 zmc_m9[16103].DB3-zm_monitor.cpp/2328 [Reloading zones for monitor WarehouseRight have 0]
01/01/22 15:10:32.094381 zmc_m9[16106].DB1-zm_comms.cpp/471 [connect(): Trying '192.168.0.113', family '2', proto '6']
01/01/22 15:10:32.094560 zmc_m9[16103].DB1-zm_zone.cpp/839 [Got 0 zones for monitor WarehouseRight]
01/01/22 15:10:32.094576 zmc_m9[16103].DB1-zm_monitor.cpp/2330 [Reloading zones for monitor WarehouseRight have 0]
01/01/22 15:10:32.094584 zmc_m9[16103].DB1-zm_monitor.cpp/694 [Loaded monitor 9(WarehouseRight), 0 zones]
01/01/22 15:10:32.094588 zmc_m9[16103].DB2-zmc.cpp/217 [1 monitors loaded]
01/01/22 15:10:32.094591 zmc_m9[16103].INF-zmc.cpp/220 [Starting Capture version 1.36.12]
01/01/22 15:10:32.094599 zmc_m9[16103].DB3-zm_monitor.cpp/917 [Connecting to monitor. Purpose is 1]
01/01/22 15:10:32.094610 zmc_m9[16103].DB3-zm_monitor.cpp/930 [Success opening mmap file at (/dev/shm/zm.mmap.9)]
01/01/22 15:10:32.094617 zmc_m9[16103].DB3-zm_monitor.cpp/963 [MMap file size is 0]
01/01/22 15:10:32.094622 zmc_m9[16103].DB1-zm_monitor.cpp/968 [Unable to map file /dev/shm/zm.mmap.9 (4325560 bytes) to locked memory, trying unlocked]
01/01/22 15:10:32.094627 zmc_m9[16103].DB1-zm_monitor.cpp/971 [Mapped file /dev/shm/zm.mmap.9 (4325560 bytes) to unlocked memory]
01/01/22 15:10:32.094629 zmc_m9[16103].DB3-zm_monitor.cpp/1004 [Aligning shared memory images to the next 64 byte boundary]
01/01/22 15:10:32.094633 zmc_m9[16103].DB3-zm_monitor.cpp/1014 [Allocated 3 3 image buffers]
01/01/22 15:10:32.096418 zmc_m9[16103].DB3-zm_monitor.cpp/1067 [Success connecting]
01/01/22 15:10:32.096615 zmc_m9[16103].DB1-zm_db.cpp/189 [Success running sql query INSERT INTO Monitor_Status (MonitorId,Status,CaptureFPS,AnalysisFPS) VALUES (9, 'Running',0,0) ON DUPLICATE KEY UPDATE Status='Running',CaptureFPS=0,AnalysisFPS=0]
01/01/22 15:10:32.096623 zmc_m9[16103].DB2-zm_remote_camera_rtsp.cpp/130 [Waiting for sources]
01/01/22 15:10:32.102717 zmc_m9[16106].DB2-zm_rtsp.cpp/43 [Sending RTSP message: OPTIONS rtsp://192.168.0.113:7070/ RTSP/1.0
User-Agent: ZoneMinder/1.36.12
CSeq: 1
]
01/01/22 15:10:32.108736 zmc_m2[16071].ERR-zm_signal.cpp/50 [Got signal 6 (Aborted), crashing]
01/01/22 15:10:32.108781 zmc_m2[16071].DB1-zm_signal.cpp/60 [Signal information: number 6 code -6 errno 0 pid 16068 uid 48 status 0]
01/01/22 15:10:32.108795 zmc_m2[16071].ERR-zm_signal.cpp/80 [Signal address is 0x3000003ec4, from 0x7fb46aa4e37f]
01/01/22 15:10:32.108992 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 0: /usr/bin/zmc(+0xd1431) [0x55e734d29431]]
01/01/22 15:10:32.109006 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 1: /lib64/libpthread.so.0(+0x12c20) [0x7fb46e1a8c20]]
01/01/22 15:10:32.109014 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 2: /lib64/libc.so.6(gsignal+0x10f) [0x7fb46aa4e37f]]
01/01/22 15:10:32.109022 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 3: /lib64/libc.so.6(abort+0x127) [0x7fb46aa38db5]]
01/01/22 15:10:32.109030 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 4: /usr/bin/zmc(+0x55118) [0x55e734cad118]]
01/01/22 15:10:32.109048 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 5: /usr/bin/zmc(+0xbaf17) [0x55e734d12f17]]
01/01/22 15:10:32.109055 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 6: /usr/bin/zmc(+0xbc3a8) [0x55e734d143a8]]
01/01/22 15:10:32.109062 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 7: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fb46b438ba3]]
01/01/22 15:10:32.109069 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 8: /lib64/libpthread.so.0(+0x817a) [0x7fb46e19e17a]]
01/01/22 15:10:32.109077 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 9: /lib64/libc.so.6(clone+0x43) [0x7fb46ab13dc3]]
01/01/22 15:10:32.109084 zmc_m2[16071].INF-zm_signal.cpp/108 [Backtrace complete, please execute the following command for more information: addr2line -e /usr/bin/zmc 0x55e734d29431 0x7fb46e1a8c20 0x7fb46aa4e37f 0x7fb46aa38db5 0x55e734cad118 0x55e734d12f17 0x55e734d143a8 0x7fb46b438ba3 0x7fb46e19e17a 0x7fb46ab13dc3]
01/01/22 15:10:32.117860 zmc_m2[16070].DB1-zm_db.cpp/189 [Success running sql query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1641085832.108736, 'zmc_m2', 0, 16071, -2, 'ERR', 'Got signal 6 (Aborted), crashing', 'zm_signal.cpp', 50 )]
01/01/22 15:10:32.127370 zmc_m2[16070].DB1-zm_db.cpp/189 [Success running sql query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1641085832.108795, 'zmc_m2', 0, 16071, -2, 'ERR', 'Signal address is 0x3000003ec4, from 0x7fb46aa4e37f', 'zm_signal.cpp', 80 )]
Please let me know what info you need. I have split the log file into chunks according to stop/start instances, but even a chunk will be too large to post here.
FYI, this box is an i5-4690 with 8GB RAM and SSD (SATA) storage.
Thanks,
-Jeff
I checked debug_log, set to level 3 and specified a log file location.
There are a LOT of entries here. I think this is a chunk that may allow you to see what is happening:
01/01/22 15:10:32.093754 zmc_m9[16103].DB1-zm_logger.cpp/232 [LogOpts: level=DB3 effective=DB3, screen=OFF, database=ERR, logfile=DB3->/var/log/zoneminder/debug, syslog=ERR]
01/01/22 15:10:32.093789 zmc_m9[16103].DB1-zm_utils.cpp/198 [Detected a x86\x86-64 processor with AVX2]
01/01/22 15:10:32.094182 zmc_m9[16103].DB1-zm_storage.cpp/91 [No id passed to Storage constructor. Using default path /var/lib/zoneminder/events instead]
01/01/22 15:10:32.094191 zmc_m9[16103].DB1-zm_monitor.cpp/493 [Have camera type Local]
01/01/22 15:10:32.094194 zmc_m9[16103].DB1-zm_monitor.cpp/2336 [Reloading linked monitors for monitor WarehouseRight, '(null)']
01/01/22 15:10:32.094198 zmc_m9[16103].DB3-zm_monitor.cpp/570 [Decoding enabled: 1 function 5 Mocord savejpegs 1 videowriter 0]
01/01/22 15:10:32.094211 zmc_m9[16103].DB1-zm_packetqueue.cpp/660 [Setting pre_event_video_packet_count to 5]
01/01/22 15:10:32.094215 zmc_m9[16103].DB1-zm_packetqueue.cpp/654 [Setting max_video_packet_count to 20]
01/01/22 15:10:32.094218 zmc_m9[16103].DB1-zm_monitor.cpp/668 [mem.size(8) SharedData=760 TriggerData=560 VideoStoreData=4128 timestamps=48 images=3x1440000 = 4320000 total=4325560]
01/01/22 15:10:32.094229 zmc_m9[16103].DB2-zm_camera.cpp/69 [New camera id: 9 width: 800 line size: 2400 height: 600 colours: 3 subpixelorder: 6 capture: 1]
01/01/22 15:10:32.094241 zmc_m9[16103].DB1-zm_remote_camera.cpp/111 [1 addresses returned]
01/01/22 15:10:32.094248 zmc_m9[16103].DB1-zm_ffmpeg.cpp/86 [Not enabling ffmpeg logs, as LOG_FFMPEG and/or LOG_DEBUG is disabled in options, or this monitor is not part of your debug targets]
01/01/22 15:10:32.094265 zmc_m9[16103].DB2-zm_rtsp.cpp/169 [RTSP Local SSRC is 9480198, url is rtsp://192.168.0.113:7070/]
01/01/22 15:10:32.094270 zmc_m9[16103].DB2-zm_rtsp.cpp/176 [# of auth parts 2]
01/01/22 15:10:32.094308 zmc_m9[16103].DB3-zm_monitor.cpp/2328 [Reloading zones for monitor WarehouseRight have 0]
01/01/22 15:10:32.094381 zmc_m9[16106].DB1-zm_comms.cpp/471 [connect(): Trying '192.168.0.113', family '2', proto '6']
01/01/22 15:10:32.094560 zmc_m9[16103].DB1-zm_zone.cpp/839 [Got 0 zones for monitor WarehouseRight]
01/01/22 15:10:32.094576 zmc_m9[16103].DB1-zm_monitor.cpp/2330 [Reloading zones for monitor WarehouseRight have 0]
01/01/22 15:10:32.094584 zmc_m9[16103].DB1-zm_monitor.cpp/694 [Loaded monitor 9(WarehouseRight), 0 zones]
01/01/22 15:10:32.094588 zmc_m9[16103].DB2-zmc.cpp/217 [1 monitors loaded]
01/01/22 15:10:32.094591 zmc_m9[16103].INF-zmc.cpp/220 [Starting Capture version 1.36.12]
01/01/22 15:10:32.094599 zmc_m9[16103].DB3-zm_monitor.cpp/917 [Connecting to monitor. Purpose is 1]
01/01/22 15:10:32.094610 zmc_m9[16103].DB3-zm_monitor.cpp/930 [Success opening mmap file at (/dev/shm/zm.mmap.9)]
01/01/22 15:10:32.094617 zmc_m9[16103].DB3-zm_monitor.cpp/963 [MMap file size is 0]
01/01/22 15:10:32.094622 zmc_m9[16103].DB1-zm_monitor.cpp/968 [Unable to map file /dev/shm/zm.mmap.9 (4325560 bytes) to locked memory, trying unlocked]
01/01/22 15:10:32.094627 zmc_m9[16103].DB1-zm_monitor.cpp/971 [Mapped file /dev/shm/zm.mmap.9 (4325560 bytes) to unlocked memory]
01/01/22 15:10:32.094629 zmc_m9[16103].DB3-zm_monitor.cpp/1004 [Aligning shared memory images to the next 64 byte boundary]
01/01/22 15:10:32.094633 zmc_m9[16103].DB3-zm_monitor.cpp/1014 [Allocated 3 3 image buffers]
01/01/22 15:10:32.096418 zmc_m9[16103].DB3-zm_monitor.cpp/1067 [Success connecting]
01/01/22 15:10:32.096615 zmc_m9[16103].DB1-zm_db.cpp/189 [Success running sql query INSERT INTO Monitor_Status (MonitorId,Status,CaptureFPS,AnalysisFPS) VALUES (9, 'Running',0,0) ON DUPLICATE KEY UPDATE Status='Running',CaptureFPS=0,AnalysisFPS=0]
01/01/22 15:10:32.096623 zmc_m9[16103].DB2-zm_remote_camera_rtsp.cpp/130 [Waiting for sources]
01/01/22 15:10:32.102717 zmc_m9[16106].DB2-zm_rtsp.cpp/43 [Sending RTSP message: OPTIONS rtsp://192.168.0.113:7070/ RTSP/1.0
User-Agent: ZoneMinder/1.36.12
CSeq: 1
]
01/01/22 15:10:32.108736 zmc_m2[16071].ERR-zm_signal.cpp/50 [Got signal 6 (Aborted), crashing]
01/01/22 15:10:32.108781 zmc_m2[16071].DB1-zm_signal.cpp/60 [Signal information: number 6 code -6 errno 0 pid 16068 uid 48 status 0]
01/01/22 15:10:32.108795 zmc_m2[16071].ERR-zm_signal.cpp/80 [Signal address is 0x3000003ec4, from 0x7fb46aa4e37f]
01/01/22 15:10:32.108992 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 0: /usr/bin/zmc(+0xd1431) [0x55e734d29431]]
01/01/22 15:10:32.109006 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 1: /lib64/libpthread.so.0(+0x12c20) [0x7fb46e1a8c20]]
01/01/22 15:10:32.109014 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 2: /lib64/libc.so.6(gsignal+0x10f) [0x7fb46aa4e37f]]
01/01/22 15:10:32.109022 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 3: /lib64/libc.so.6(abort+0x127) [0x7fb46aa38db5]]
01/01/22 15:10:32.109030 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 4: /usr/bin/zmc(+0x55118) [0x55e734cad118]]
01/01/22 15:10:32.109048 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 5: /usr/bin/zmc(+0xbaf17) [0x55e734d12f17]]
01/01/22 15:10:32.109055 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 6: /usr/bin/zmc(+0xbc3a8) [0x55e734d143a8]]
01/01/22 15:10:32.109062 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 7: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fb46b438ba3]]
01/01/22 15:10:32.109069 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 8: /lib64/libpthread.so.0(+0x817a) [0x7fb46e19e17a]]
01/01/22 15:10:32.109077 zmc_m2[16071].ERR-zm_signal.cpp/101 [Backtrace 9: /lib64/libc.so.6(clone+0x43) [0x7fb46ab13dc3]]
01/01/22 15:10:32.109084 zmc_m2[16071].INF-zm_signal.cpp/108 [Backtrace complete, please execute the following command for more information: addr2line -e /usr/bin/zmc 0x55e734d29431 0x7fb46e1a8c20 0x7fb46aa4e37f 0x7fb46aa38db5 0x55e734cad118 0x55e734d12f17 0x55e734d143a8 0x7fb46b438ba3 0x7fb46e19e17a 0x7fb46ab13dc3]
01/01/22 15:10:32.117860 zmc_m2[16070].DB1-zm_db.cpp/189 [Success running sql query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1641085832.108736, 'zmc_m2', 0, 16071, -2, 'ERR', 'Got signal 6 (Aborted), crashing', 'zm_signal.cpp', 50 )]
01/01/22 15:10:32.127370 zmc_m2[16070].DB1-zm_db.cpp/189 [Success running sql query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1641085832.108795, 'zmc_m2', 0, 16071, -2, 'ERR', 'Signal address is 0x3000003ec4, from 0x7fb46aa4e37f', 'zm_signal.cpp', 80 )]
Please let me know what info you need. I have split the log file into chunks according to stop/start instances, but even a chunk will be too large to post here.
FYI, this box is an i5-4690 with 8GB RAM and SSD (SATA) storage.
Thanks,
-Jeff
Re: ZMC processes dying after upgrade to 1.36.12
Okay it is crashing during opening the camera stream.
The Remote monitor still exists and should work, but is relatively unmaintained.
Try the ffmpeg type. It has many advantages.
The Remote monitor still exists and should work, but is relatively unmaintained.
Try the ffmpeg type. It has many advantages.
Re: ZMC processes dying after upgrade to 1.36.12
Thanks, iConnor, the problem was indeed the Remote type. Ffmpeg worked without crashing.
For those who find this applies to their situation, here are more details.
After switching General > Source Type to ffmpeg, I used Source > Source Path of form rtsp://<login>:<pass>@<IP>:<port>
Source > Method of TCP, although UDP seemed to work just as well.
Unfortunately, H.264 decoding didn't work well at all, although it had been perfect under the Remote type. Lots of artifacting and "jumping" with ffmpeg. On the 11 sending cameras, I selected MJPEG for the streams. All are now sending steady, artifact-free 800x600 streams to the ZM box.
There are new problems that didn't exist under the Remote type. In the events folders, about every other .jpeg file is missing.
I am also seeing errors like this run continuously through the logs:
2022-01-02 21:42:46 zmc_m6 52332 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
2022-01-02 21:42:46 zmc_m2 54009 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
2022-01-02 21:42:46 zmc_m10 50560 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
2022-01-02 21:42:46 zmc_m9 50894 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
These log errors have been running non-stop for the zmc process for each monitor.
The missing frames aren't caused by network issues - pings to all cameras are excellent, like:
50 packets transmitted, 50 received, 0% packet loss, time 50166ms
rtt min/avg/max/mdev = 0.539/0.620/0.796/0.061 ms
Storage space is adequate.
I installed ZM on this Centos8 box via the RPMFusion repo. I am wondering if I should throw everything out and try reinstalling?
Thanks,
-Jeff
For those who find this applies to their situation, here are more details.
After switching General > Source Type to ffmpeg, I used Source > Source Path of form rtsp://<login>:<pass>@<IP>:<port>
Source > Method of TCP, although UDP seemed to work just as well.
Unfortunately, H.264 decoding didn't work well at all, although it had been perfect under the Remote type. Lots of artifacting and "jumping" with ffmpeg. On the 11 sending cameras, I selected MJPEG for the streams. All are now sending steady, artifact-free 800x600 streams to the ZM box.
There are new problems that didn't exist under the Remote type. In the events folders, about every other .jpeg file is missing.
I am also seeing errors like this run continuously through the logs:
2022-01-02 21:42:46 zmc_m6 52332 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
2022-01-02 21:42:46 zmc_m2 54009 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
2022-01-02 21:42:46 zmc_m10 50560 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
2022-01-02 21:42:46 zmc_m9 50894 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
These log errors have been running non-stop for the zmc process for each monitor.
The missing frames aren't caused by network issues - pings to all cameras are excellent, like:
50 packets transmitted, 50 received, 0% packet loss, time 50166ms
rtt min/avg/max/mdev = 0.539/0.620/0.796/0.061 ms
Storage space is adequate.
I installed ZM on this Centos8 box via the RPMFusion repo. I am wondering if I should throw everything out and try reinstalling?
Thanks,
-Jeff
Re: ZMC processes dying after upgrade to 1.36.12
Well, as I said, the Remote type SHOULD work so now I know where to look and can hopefully find & fix it.
I have seen those errors when ffmpeg is starved for cpu, normally because you have put a value in the maxfps fields in the first tab of monitor edit.
I have seen those errors when ffmpeg is starved for cpu, normally because you have put a value in the maxfps fields in the first tab of monitor edit.
Re: ZMC processes dying after upgrade to 1.36.12
I've been using ZM for quite a few years, and know that speed should be throttled at the camera, and not with the MaxFPS option 
Also, analysis is not enabled, no motion detection zones are defined, and even timestamps are turned off, so post-processing/analysis is minimal.
CPU idle seems to be averaging around 69%, which is definitely lower than before switching from remote to ffmpeg.
Thanks iConnor
Also, analysis is not enabled, no motion detection zones are defined, and even timestamps are turned off, so post-processing/analysis is minimal.CPU idle seems to be averaging around 69%, which is definitely lower than before switching from remote to ffmpeg.
Thanks iConnor
Re: ZMC processes dying after upgrade to 1.36.12
Update:
I installed AlmaLinux 8.5 cleanly. Pulled all updates, set up repos and installed Zoneminder without any issues or errors.
I tried to create a monitor of type Remote and began seeing the ZMC crashes immediately. So, the Remote type seems to be hopelessly broken in Centos/AlmaLinux/RH 8.5 for now.
After a lot of experimentation with the Acti cameras, I found out why they were generating errors like:
2022-01-02 21:42:46 zmc_m9 50894 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
These Acti cams (that I had inherited from a client's proprietary NVR when the HD died) were continuously switching output between the different available streams!! Thus, stream2, which I had been ignoring, would send some frames after stream1 had sent several, causing hopeless confusion for ZM! After duplicating the desired settings in both streams, the problem was resolved. I have worked with a number of cams from Grandstream, Hikvision, Axis, and a few others, and had never seen such lunacy!
SO, the "Unable to send packet" errors were NOT caused by Zoneminder or the ffmpeg type, but by idiot camera firmware when pulling an RTSP stream.
My apologies for the false alarm, Isaiah.
I installed AlmaLinux 8.5 cleanly. Pulled all updates, set up repos and installed Zoneminder without any issues or errors.
I tried to create a monitor of type Remote and began seeing the ZMC crashes immediately. So, the Remote type seems to be hopelessly broken in Centos/AlmaLinux/RH 8.5 for now.
After a lot of experimentation with the Acti cameras, I found out why they were generating errors like:
2022-01-02 21:42:46 zmc_m9 50894 ERR Unable to send packet Invalid data found when processing input, continuing zm_ffmpeg.cpp 519
These Acti cams (that I had inherited from a client's proprietary NVR when the HD died) were continuously switching output between the different available streams!! Thus, stream2, which I had been ignoring, would send some frames after stream1 had sent several, causing hopeless confusion for ZM! After duplicating the desired settings in both streams, the problem was resolved. I have worked with a number of cams from Grandstream, Hikvision, Axis, and a few others, and had never seen such lunacy!
SO, the "Unable to send packet" errors were NOT caused by Zoneminder or the ffmpeg type, but by idiot camera firmware when pulling an RTSP stream.
My apologies for the false alarm, Isaiah.