Page 1 of 1

Extremely high iowait time & load

Posted: Wed Apr 19, 2017 3:33 pm
by ranciso
Hello,
We are seeing extremely high load times and iowait times, sometime over 50 load average. There doesn't seem to be a lot of disk activity related to the wait time (see attached). I'm having a really hard time pinning this issue down. Here is our zm setup:

ZM on a VM
16 vCPUs Physical CPUs on host are at 3GHz
32GB of RAM
26TB Disk Direct Attached SAS (RAID5)
CENTOS 7.2.1151 (core)
100 Monitors @ 2fps
I have a filter setup to remove events after 30 days.
I've tried several configuration under options related to zmaudit and how often to run the filter and nothing seems to bring down the load.
Network seems to be ok at max we are pulling about 12Mbps.

Any help would be appreciated. Thanks

Re: Extremely high iowait time & load

Posted: Wed Apr 19, 2017 4:57 pm
by ranciso
More info on this - we have about 49 monitors doing motion detection while the others are set to continuously record.

Re: Extremely high iowait time & load

Posted: Wed Apr 19, 2017 5:41 pm
by iconnor
Ok, so you get that raid5 means that the write speed is that of 1 disk. So all the zma's are trying to write to your raid5 and are backed up.

Re: Extremely high iowait time & load

Posted: Wed Apr 19, 2017 6:20 pm
by ranciso
I understand how raid 5 works and I do get that the zma's are getting backed up but it doesn't seem like they should be with the disk sub system being what is.

Re: Extremely high iowait time & load

Posted: Thu Apr 20, 2017 3:10 pm
by ranciso
Fixed my problem. Turns out I was using the LSI Logic SCSI controller for the VM. Switched to para virtualized SCSI controller. Enabled pvscsi driver in OS. Had to modify vmx file to use pvscsi. Removed and re-added disks to PV Controller. Also had to removed vm from inventory and re-registered the vm in vcenter. Some info here: http://allesschonvergeben.blogspot.com/ ... apter.html

Night and day difference. Load is now at around 2-4. Hopefully this helps others.