More Fixes + XML API for ZM 1.24.2

If you've made a patch to quick fix a bug or to add a new feature not yet in the main tree then post it here so others can try it out.
mitch
Posts: 169
Joined: Thu Apr 30, 2009 4:18 am

More Fixes + XML API for ZM 1.24.2

Post by mitch »

The broad details:
This patch includes the following improvements:
*Allow for longer camera Paths to avoid truncation that could occur
*Cameras eventually reach very slow FPS playback on streams until stream reconnects (if network is not 100%), this is fixed
*Zone minder would act like it was recording when in reality there were no new frames coming from a camera, it would even continue to timestamp the last frame making it somewhat hard to detect.
*If a stream was refreshed but the connKey did not change several items were broken including pausing (or anything with the swap) and stream stats (fps, status, etc), to enable this feature under options->Images, REDUCTION_FPS_RESET_TIME - set to a value other than 0, not anything overly low I use 240.
*If web client would run into a connection refused error when trying to get stats from zms (say its still starting up) we now wait a few seconds and try again before failing
*Added the start of an external web API for some items not previously publicly exposed, for example http://127.0.0.1/zm/?view=api&action=list_monitors will return a list of the monitors and their config.
*Added generic XML output instead of just ajax output. This can be used now on ALL ZM ajax calls to get XML back instead, all you have to do is say usexml=1 in the query string for example: http://127.0.0.1/zm/?view=request&reque ... c&usexml=1
This gives a very large external API that can now be used over XML rather than just JSON
*A few other minor things and IE fixes

The patch is at: http://mitchcapper.com/zmpatch090704.patch
To apply this patch in your ZM checkout folder just do patch -p0 < zmpatch090704.patch if you want to check if it will work against the dryrun option like: patch -p0 --dry-run < zmpatch090704.patch . This is meant to be applied against trunk, but also most likely applies clean against 1.24.2

Also one should do the following insert into the db:

Code: Select all

insert into Config set Id = 183, Name = 'ZM_REDUCTION_FPS_RESET_TIME', Value = '0', Type = 'integer', DefaultValue = '0', Hint = 'integer', Pattern = '(?-xism:^(\d+)$)', Format = ' $1 ', Prompt = 'After how many seconds should the maxfps of a stream go back to normal after it has been throttled', Help = 'When a stream cannot send frames as fast as requested (generally due to too slow of a connection) it will throttle it back, unfortunately if the cause of this was only temporary the FPS will never pickup again.  By setting this to a value greater than 0 it will automatically go back to the original FPS after X seconds from when the last throttle took place.', Category = 'image', Readonly = '0', Requires = '';

Some more information:
*Zone minder slows down the FPS (halving it) on a stream if it cannot send fast enough. This is all well and good, but if the cause of this is only temporary then your FPS is permanently lowered. IE if you are streaming at 20fps, then the network becomes congested the FPS may drop down to say 2 FPS, but once the congestion is resolved ZM never resets the FPS back up. Now there is a configuration variable for after X seconds it will reset the FPS back to the original amount.
*Zone minder did not properly check if a directory was able to be created or not in several places, causing minor issues specifically related to swap path creation.
*If the stream was refreshed but the connkey didnt change it would, in most cases, result in the swap directory and socket being removed by the old streaming process but this would not happen until after the new one had already been created causing the new one to lose its socket and swap dir. The two options to avoid this were to either A) if a process was already streaming for a specific connection key, terminate it or B) Don't delete the swap directory and socket if a new process has started already for our connKey when we exit. I took the second approach as A) would require signaling the process it needed to die, and possibly waiting awhile for items to timeout (not to mention it may be desireable to not require a different connkey each time). So now any time zms is streaming it obtains a read lock on a lock file for the connectionKey, when it terminates it then releases its read lock, waits 10 seconds (incase another zms is starting up, don't want one to startup while we are deleting items), and try to obtain a write lock on the lock file. If write lock is obtained then no one is reading it and we can erase the swap dir safely. If we cannot obtain a write lock just leave everything and bail out, the other process (which has a read lock) will do our cleanup when it terminates.
*Some minor SQL injection holes in the ajax status tool existed specifically around sorting and ordering. We now parse for ints where ints are needed, and properly parse out columns for order by rather than just taking the str from the input.
*IE doesn't support getText, so use innerHTML which is more stnadard
*The new api calls are exposed by api.php in web/skins/classic/views/ it exposes a few functions (zone_info,list_monitors,get_connkey,event_stats) that were not previously exposed by JSON.
*The new XML support for all json function calls was done with a few minor changes the major one being an xmlEncode function in functions.php and a global override that xml_encode_not_json (set when usexml=1) that, when jsonEncode is called will call xmlEncode instead of true.


As for my new app, it is pretty much ready, ive been slowed a bit lately due to a conflict issue with network traffic. Within a week or so an initial version will be released that is a decent windows client with the debugging talked about previously.
User avatar
cordel
Posts: 5210
Joined: Fri Mar 05, 2004 4:47 pm
Location: /USA/Washington/Seattle

Post by cordel »

Mitch, Thanks again.
ahagadorn
Posts: 61
Joined: Mon Aug 11, 2008 1:55 pm
Location: Burnt Hills, NY

Post by ahagadorn »

I will probably use the XML capabilities in future releases of jZMConsole. It currently has it's own transaction engine on the server side that returns XML.

Andy
mrd
Posts: 269
Joined: Wed Apr 26, 2006 12:39 am
Location: Boston USA

Post by mrd »

I think this is exactly what I need! I'm working on an Android front end for ZM and was hoping to find an XML API. It looks like you have done it, but I'm not quite sure how I can use it since my install is not from source. I used the ZMARCH live CD installed method.

Just so I'm clear, the API you developed should be usable by any external interface to ZM and can produce XML for most if not all standard queries for the information that you would expect to need for a frontend viewer/client right.

Thanks for this hard work!
MRD
mitch
Posts: 169
Joined: Thu Apr 30, 2009 4:18 am

Post by mitch »

Thats correct, most of the current interface was AJAX I believe any AJAX interface with my patch will work in XML. I added a few other plugs for things that were only exported in HTML but I believe you should be able to do most everything with it. As for how to use it my XML patch I *believe* is a php modification only so even a CD install could have the web patch applied (although you may have to do it by hand).
mrd
Posts: 269
Joined: Wed Apr 26, 2006 12:39 am
Location: Boston USA

Post by mrd »

That sounds great! I'll have to give it a shot. Any chance this stuff will make it into the next release?
MRD
mitch
Posts: 169
Joined: Thu Apr 30, 2009 4:18 am

Post by mitch »

i would guess so, generally my previous patches have, but no promises.
User avatar
Normando
Posts: 219
Joined: Sun Aug 17, 2008 5:34 am
Location: Rosario - Argentina

Post by Normando »

Hi Mitch!. Your patch work like a charm!

This have a minimal issue, and is this:

In the DB and ConfigAdmin.pm you use "Category = image" and should be "Category = images" (look the 's').

Another issue. I don't know if this is important or not, but looking for log files when throttling in live mode, and I close the monitor window, I get the error:

ERR [Unable to send stream frame: Broken pipe]

Is this ok or not?

Thank you for your work.

Normando
mitch
Posts: 169
Joined: Thu Apr 30, 2009 4:18 am

Post by mitch »

Yes, the message is good that means it detected the stream was broken and stops, if it didn't it could continue to run forever.
User avatar
Normando
Posts: 219
Joined: Sun Aug 17, 2008 5:34 am
Location: Rosario - Argentina

Post by Normando »

Thank you Mitch. I was confirmed that this issue is no excluvelly for your path. Also give me the same ERR without the patch. One thing I think is why Phill not use [INF] instead of [ERR] in the code if this is a "normal" operation?
mitch
Posts: 169
Joined: Thu Apr 30, 2009 4:18 am

Post by mitch »

Its been honestly awhile since i worked in the ZM code, a good reason is probably the ERR call prints the error and bails out (abort) where as an INFO wouldn't. In the case of the stream being broken you want to abort so rather than have a separate abort but INFO handler just called the same.
User avatar
Normando
Posts: 219
Joined: Sun Aug 17, 2008 5:34 am
Location: Rosario - Argentina

Post by Normando »

Thanks Mitch for your reply.

I was patched manually the latest zm code from repository with your path, and all the functions you was implemented run like a char.

There is only one that concer me. The sock and lock files not removed from the memory (or disk in my case for testing purposes)

I was see the differences between zm_monitor.cpp v1.24.2 and the latest from the repo (svn 3117) and there are no significative IMHO. Here are the diff between 1.24.2 and svn 3117:

Code: Select all

--- zm_monitor.cpp-1.24.2    2009-06-24 12:22:23.000000000 +0200
+++ zm_monitor.cpp-svn3117    2010-08-02 18:35:58.000000000 +0200
@@ -1,5 +1,5 @@
 //
-// ZoneMinder Monitor Class Implementation, $Date: 2009-06-09 15:15:02 +0100 (Tue, 09 Jun 2009) $, $Revision: 2916 $
+// ZoneMinder Monitor Class Implementation, $Date: 2010-05-27 14:18:43 +0200 (Thu, 27 May 2010) $, $Revision: 3066 $
 // Copyright (C) 2001-2008 Philip Coombes
 // 
 // This program is free software; you can redistribute it and/or
@@ -53,7 +53,7 @@ Monitor::MonitorLink::MonitorLink( int p
 
 #if ZM_MEM_MAPPED
     map_fd = -1;
-    snprintf( mem_file, sizeof(mem_file), "%s/.zm.mmap.%d", config.path_map, id );
+    snprintf( mem_file, sizeof(mem_file), "%s/zm.mmap.%d", config.path_map, id );
 #else // ZM_MEM_MAPPED
     shm_id = 0;
 #endif // ZM_MEM_MAPPED
@@ -344,7 +344,7 @@ Monitor::Monitor(
 
     Debug( 1, "mem.size=%d", mem_size );
 #if ZM_MEM_MAPPED
-    snprintf( mem_file, sizeof(mem_file), "%s/.zm.mmap.%d", config.path_map, id );
+    snprintf( mem_file, sizeof(mem_file), "%s/zm.mmap.%d", config.path_map, id );
     map_fd = open( mem_file, O_RDWR|O_CREAT, (mode_t)0600 );
     if ( map_fd < 0 )
         Fatal( "Can't open memory map file %s, probably not enough space free: %s", mem_file, strerror(errno) );
@@ -542,8 +542,8 @@ Monitor::~Monitor()
     }
 
 #if ZM_MEM_MAPPED
-    if ( msync( mem_ptr, mem_size, MS_INVALIDATE ) < 0 )
-        Fatal( "Can't msync: %s", strerror(errno) );
+    if ( msync( mem_ptr, mem_size, MS_SYNC ) < 0 )
+        Error( "Can't msync: %s", strerror(errno) );
     if ( munmap( mem_ptr, mem_size ) <0>timestamp );
@@ -933,29 +933,36 @@ void Monitor::DumpZoneImage( const char 
     zone_image.Colourise();
     for( int i = 0; i <n_zones>Id() == exclude_id )
+        if ( exclude_id && (!extra_colour || extra_zone.getNumCoords()) && zones[i]->Id() == exclude_id )
             continue;
 
         Rgb colour;
-        if ( zones[i]->IsActive() )
+        if ( exclude_id && !extra_zone.getNumCoords() && zones[i]->Id() == exclude_id )
         {
-            colour = RGB_RED;
-        }
-        else if ( zones[i]->IsInclusive() )
-        {
-            colour = RGB_ORANGE;
-        }
-        else if ( zones[i]->IsExclusive() )
-        {
-            colour = RGB_PURPLE;
-        }
-        else if ( zones[i]->IsPreclusive() )
-        {
-            colour = RGB_BLUE;
+            colour = extra_colour;
         }
         else
         {
-            colour = RGB_WHITE;
+            if ( zones[i]->IsActive() )
+            {
+                colour = RGB_RED;
+            }
+            else if ( zones[i]->IsInclusive() )
+            {
+                colour = RGB_ORANGE;
+            }
+            else if ( zones[i]->IsExclusive() )
+            {
+                colour = RGB_PURPLE;
+            }
+            else if ( zones[i]->IsPreclusive() )
+            {
+                colour = RGB_BLUE;
+            }
+            else
+            {
+                colour = RGB_WHITE;
+            }
         }
         zone_image.Fill( colour, 2, zones[i]->GetPolygon() );
         zone_image.Outline( colour, zones[i]->GetPolygon() );
@@ -978,8 +985,8 @@ void Monitor::DumpImage( Image *dump_ima
     {
         static char filename[PATH_MAX];
         static char new_filename[PATH_MAX];
-        snprintf( filename, sizeof(filename), "%s.jpg", name );
-        snprintf( new_filename, sizeof(new_filename), "%s-new.jpg", name );
+        snprintf( filename, sizeof(filename), "Monitor%d.jpg", id );
+        snprintf( new_filename, sizeof(new_filename), "Monitor%d-new.jpg", id );
         dump_image->WriteJpeg( new_filename );
         rename( new_filename, filename );
     }
@@ -992,7 +999,7 @@ bool Monitor::CheckSignal( const Image *
     static unsigned char green_val;
     static unsigned char blue_val;
 
-    if ( camera->IsLocal() && config.signal_check_points > 0 )
+    if ( config.signal_check_points > 0 )
     {
         if ( static_undef )
         {
@@ -1004,11 +1011,22 @@ bool Monitor::CheckSignal( const Image *
 
         const unsigned char *buffer = image->Buffer();
         int pixels = image->Pixels();
+        int width = image->Width();
         int colours = image->Colours();
 
+        int index = 0;
         for ( int i = 0; i < config.signal_check_points; i++ )
         {
-            int index = (rand()*pixels)/RAND_MAX;
+            while( true )
+            {
+                index = (int)(((long long)rand()*(long long)(pixels-1))/RAND_MAX);
+                if ( !config.timestamp_on_capture || !label_format[0] )
+                    break;
+                // Avoid sampling the rows with timestamp in
+                int y = index / (width * colours);
+                if ( y <label_coord> label_coord.Y()+Image::LINE_HEIGHT )
+                    break;
+            }
             const unsigned char *ptr = buffer+(index*colours);
             if ( (RED(ptr) != red_val) || (GREEN(ptr) != green_val) || (BLUE(ptr) != blue_val) )
             {
@@ -2471,7 +2489,15 @@ Monitor *Monitor::Load( int id, bool loa
 
 int Monitor::Capture()
 {
-    if ( camera->Capture( image ) == 0 )
+    int captureResult = camera->Capture( image );
+    if ( captureResult == 1 )
+    {
+        // Unable to capture image for temporary reason
+        // Fake a signal loss image
+        image.Fill( signal_check_colour );
+        captureResult = 0;
+    }
+    if ( captureResult == 0 )
     {
         if ( orientation != ROTATE_0 )
         {
I hope you can help me with this issue.
User avatar
Normando
Posts: 219
Joined: Sun Aug 17, 2008 5:34 am
Location: Rosario - Argentina

Post by Normando »

If you want I can upload the patched file, because is too large to post here.
henke
Posts: 60
Joined: Thu Nov 10, 2005 8:16 am

Post by henke »

Hi,

Thanks for this patch.
Do you know if it is included int the 1.24.3 version?
User avatar
Normando
Posts: 219
Joined: Sun Aug 17, 2008 5:34 am
Location: Rosario - Argentina

Post by Normando »

henke wrote:Hi,

Thanks for this patch.
Do you know if it is included int the 1.24.3 version?
I don't know
Post Reply