Extremely high iowait time & load

Forum for questions and support relating to the 1.30.x releases only.
Locked
ranciso
Posts: 20
Joined: Wed Apr 19, 2017 3:19 pm

Extremely high iowait time & load

Post by ranciso »

Hello,
We are seeing extremely high load times and iowait times, sometime over 50 load average. There doesn't seem to be a lot of disk activity related to the wait time (see attached). I'm having a really hard time pinning this issue down. Here is our zm setup:

ZM on a VM
16 vCPUs Physical CPUs on host are at 3GHz
32GB of RAM
26TB Disk Direct Attached SAS (RAID5)
CENTOS 7.2.1151 (core)
100 Monitors @ 2fps
I have a filter setup to remove events after 30 days.
I've tried several configuration under options related to zmaudit and how often to run the filter and nothing seems to bring down the load.
Network seems to be ok at max we are pulling about 12Mbps.

Any help would be appreciated. Thanks
Attachments
top screen
top screen
Screen Shot 2017-04-19 at 11.20.59 AM.png (273.51 KiB) Viewed 2290 times
iotop screen
iotop screen
Screen Shot 2017-04-19 at 11.21.11 AM.png (308.02 KiB) Viewed 2290 times
options screen
options screen
Screen Shot 2017-04-19 at 11.28.51 AM.png (300.12 KiB) Viewed 2290 times
ranciso
Posts: 20
Joined: Wed Apr 19, 2017 3:19 pm

Re: Extremely high iowait time & load

Post by ranciso »

More info on this - we have about 49 monitors doing motion detection while the others are set to continuously record.
User avatar
iconnor
Posts: 2880
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Extremely high iowait time & load

Post by iconnor »

Ok, so you get that raid5 means that the write speed is that of 1 disk. So all the zma's are trying to write to your raid5 and are backed up.
ranciso
Posts: 20
Joined: Wed Apr 19, 2017 3:19 pm

Re: Extremely high iowait time & load

Post by ranciso »

I understand how raid 5 works and I do get that the zma's are getting backed up but it doesn't seem like they should be with the disk sub system being what is.
ranciso
Posts: 20
Joined: Wed Apr 19, 2017 3:19 pm

Re: Extremely high iowait time & load

Post by ranciso »

Fixed my problem. Turns out I was using the LSI Logic SCSI controller for the VM. Switched to para virtualized SCSI controller. Enabled pvscsi driver in OS. Had to modify vmx file to use pvscsi. Removed and re-added disks to PV Controller. Also had to removed vm from inventory and re-registered the vm in vcenter. Some info here: http://allesschonvergeben.blogspot.com/ ... apter.html

Night and day difference. Load is now at around 2-4. Hopefully this helps others.
Locked