[SOLVED] Wrong load and disk usage

[montagdude]

I am running ZoneMinder 1.30.4 on Slackware (using the build script here: http://slackbuilds.org/repository/14.2/ ... oneMinder/, modified only to bump the version to 1.30.4). I recently migrated the database from my laptop, where I was testing it, to a new dedicated server. This server is just a mini PC with 4x 1.6 GHz Celeron processors, so not extremely powerful, but I am only running one camera right now anyway (will soon increase to 4). Everything was working fine on my laptop, and for the most part it's working fine on the server, except that the Load and Disk % listed in the console are wrong. Even when ZoneMinder is not running, the load is listed as about 1 - 1.5, and it remains about the same when ZoneMinder is running. However, if I look at the load in top, it's only 0 - 3% when ZoneMinder is not running and 5 - 8% when it is running. The zmc process itself is hovering at around 10-15% of a single core. I noticed that the htop "Load average" seems to be the same as what is listed on the ZoneMinder console, but also not making sense based on the breakdown by core and by process.

I'm thinking that since htop shows the same thing, this is some sort of problem with my system as opposed to with my ZoneMinder configuration. On my laptop, the htop output is fine. For what it's worth, I have a python script that reports CPU load based on the information in /proc/stat, and it agrees with the 5-8% load reported by top. The first two screenshots below show output from top and htop when ZoneMinder is running on the server.

Similarly, my disk usage percentage seems to be wrong. It is listed as 5% on the console, but according to df -h, it should only be 1%. /dev/shm is right, at least.

Code: Select all

dan@zmserver:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            32M 1016K   32M   4% /run
devtmpfs        8.0M     0  8.0M   0% /dev
/dev/sda1        30G  6.7G   22G  24% /
tmpfs           1.9G  704M  1.2G  37% /dev/shm
cgroup_root     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/sda2        99G  110M   94G   1% /var/www
/dev/sda3       785G   55G  691G   8% /home
cgmfs           100K     0  100K   0% /run/cgmanager/fs

A picture of my Paths configuration is in the last screenshot. When I was running this on my laptop, I found that I needed to enter the full path to the events, images, and sounds in order to get the correct disk usage %, and I've done the same thing on the new server.

[montagdude]

So I've dug a little deeper into the disk usage issue. It turns out that the numbers come from the php functions disk_total_space and disk_free_space. I made this simple php script to test it:

Code: Select all

<?php
 $total = disk_total_space("/var/www/htdocs/zm/events");
 $free = disk_free_space("/var/www/htdocs/zm/events");
 $space = round((($total - $free) / $total) * 100);
 echo "total: $total\n";
 echo "free: $free\n";
 echo "usage: $space\n";
?>

The output:

Code: Select all

dan@zmserver:~$ php -f ./test.php 
total: 105555197952
free: 100029231104
usage: 5

Dividing by 1024^3, the total is 98.3 GB, and 93.2 GB is free. The output of df:

Code: Select all

dan@zmserver:~$ df -h /var/www/htdocs/zm/events
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        99G  134M   94G   1% /var/www

This actually agrees with the php script, if you round the total and free numbers up to the next GB. The issue seems to be that there are 5 GB in the size of the partition that are unaccounted for. Looking on my laptop, this seems to be a common thing with Linux disk partitions. So I think the 5% number is okay, based on how it's being calculated. df gets 1% by dividing used / (available + used) and rounding up to the nearest percent.

[montagdude]

Regarding the disk load, this comes from the php function sys_getloadavg. It is also the same number that uptime and htop report. Anyone know where that number comes from? It does not agree with what top and /proc/stat report, and it seems way too high, considering it remains above 1 even when neither ZoneMinder nor anything else significant is running on the system.

[montagdude]

I found a couple pages that explain the difference between CPU utilization and load average. Apparently, the load average is not really just the average utilization over a length of time. It is possible to have a low utilization but high load average, or vice versa.

http://blog.scoutapp.com/articles/2009/ ... d-averages

https://serverfault.com/questions/66707 ... ad-average

Also, I realized that top also reports the load average in addition to the instantaneous CPU utilization percent, and the load average agrees with what I've seen from ZoneMinder, htop, etc. I would still think that for a server running a number of services, over time the load average would be close to the CPU utilization sampled a number of times, but obviously that's not necessarily the case.

Anyway, I'm still well below 4, so I'll just keep an eye on it and not get too worried unless I see a big increase when I add more cameras.

[montagdude]

I have been monitoring the load average more closely. I found that even if I had next to nothing running on the system, the lowest the load average would go was 1. It turns out it is because there is a process called "rtsx_usb_ms_2" that is constantly waiting on disk I/O.

Code: Select all

dan@zmserver:~# ps -e v | awk '$3 ~/D/'  
  713 ?        D      0:00      0     0     0     0  0.0 [rtsx_usb_ms_2]

Because the load average includes processes waiting on disk I/O, this process was keeping the load average pegged at 1+. I upgraded from Linux kernel 4.4.88 to 4.9.66, and now it's more confusing. I still have rtsx_usb_ms_2 with "D" status, but now the load average goes to 0. So this is kind of solved, but not totally. I am going to mark the thread as solved just because this is certainly not related to ZoneMinder.

[montagdude]

I ended up fixing the disk usage display with the following changes in /var/www/htdocs/includes/functions.php:

Code: Select all

function getDiskPercent($path = ZM_DIR_EVENTS) {
  $total = disk_total_space($path);
  if ( $total === false ) {
    Error("disk_total_space returned false. Verify the web account user has access to " . $path );
    return 0;
  } elseif ( $total == 0 ) {
    Error("disk_total_space indicates the following path has a filesystem size of zero bytes" . $path );
    return 100;
  }
  $free = disk_free_space($path);
  if ( $free === false ) {
    Error("disk_free_space returned false. Verify the web account user has access to " . $path );
  }
-  $space = round((($total - $free) / $total) * 100);
+  if ( $path === ZM_DIR_EVENTS ) {
+    $reserved = 5408562240;
+  } else {
+    $reserved = 0;
+  }
+  $space = round((($total - $free - $reserved) / ($total - $reserved)) * 100);
  return( $space );
}

Now the disk usage displayed in the console is accurate. I figured out the reserved space by using the df command and subtraction total - used - available. Be aware that df prints out in kB by default, and php gets the info in bytes. IMO, ZoneMinder should be able to get this right. I'm aware that PHP only has functions to get the total and free space, so there's no native way to get the reserved space, but it seems like the kernel or filesystem must make that info available somewhere. I may submit a bug report /enhancement request for this.