Tip: handling huge logs created by mjpeg-streamer

Blazer · Post by **Blazer** » Mon Mar 16, 2009 7:21 pm

If you are a user of mjpeg-streamer you have no doubt noticed that it spews out tons of informational messages to /var/log/messages, such as:

Mar 16 11:50:33 hostname MJPG-streamer [2857]: serving client: 127.0.0.1

It does this for literally every frame grabbed. If you have 6+ cameras connected to a single server like I do, the /var/log/messages file can get very huge, very quickly.

By default mjpeg-streamer logs to syslog using the "INFO" log level (this can be changed by editing mjpeg-streamer.h).

So here are two tips for dealing with the massive output (sadly there is no option to simply turn it off).

1. Using syslog.conf to create a dedicated info.log
You can edit your /etc/syslog.conf (some distros use /etc/rsyslog.conf or /etc/ng-syslog.conf) like so:

Code: Select all

#*.info;mail.none;authpriv.none;cron.none                /var/log/messages
mail.none;authpriv.none;cron.none                /var/log/messages
*.info                                                   /var/log/info.log

So basically we commented out the normal entry which defines what events go into /var/log/messages, and created a new entry that has everything except *.info, and then added an additonal line that directs *.info events to go to /var/log/info.log

Once you restart your syslog/rsyslog/syslog-ng daemon, all of the mjpeg-streamer messages (and anything else with INFO level which includes some ZM messages) will go to the info.log.

Well that takes care of separating those messages to another logfile, but the logfile still becomes huge as mjpeg-streamer is constantly spewing messages to it...what then?

2. Using a named-pipe and syslog to send INFO messages into a "pipe"
A named-pipe is a cool, little-used unix flesystem trick. It is a special kind of "file", that any data written into it, goes into a small FIFO buffer that just rolls over when it gets full. If anything reads from this special "file", it gets whatever is in the buffer.

In other words, you can write endless amounts of data to this "file", and it will use no disk space. If you want to see the data that is going to the file, you can just read it (cat), and you will see the data...its like magic

First, we must create the named-pipe. As root, execute the command:
mkfifo /var/log/info.log

If you look at this file, you will see that it looks just like a normal file, except the file type is a "p", which indicates that it is a pipe:
prw-r--r-- 1 root root 0 2009-03-16 12:13 /var/log/info.log

So now we have a pipe file, is the rest as simple as option #1? ALMOST. The only difference is, in syslog.conf, you must put a pipe symbol (|) in front of the filename if it is a named-pipe. So our conf would look like so:

Code: Select all

#*.info;mail.none;authpriv.none;cron.none                /var/log/messages
mail.none;authpriv.none;cron.none                /var/log/messages
*.info                                                   |/var/log/info.log

Again you have to restart your syslog daemon for this to take effect. Your mjpeg-streamer output should now be going to the /var/log/info.log. If you run the command:

cat /var/log/info.log

You will see the data that is being sent there. Note that the "tail" command does not work on named pipe files, and you don't really need "tail" because since it is a FIFO buffer you are basically catting the last lines of the data anyway.

A similar technique can be used to redirect and control zoneminder logs if you need, although with ZM you should be able to just turn down/off any debug options and it wont create too much data.