Extremely high iowait time & load

Forum for questions and support relating to the 1.30.x releases only.
Post Reply
ranciso
Posts: 20
Joined: Wed Apr 19, 2017 3:19 pm

Extremely high iowait time & load

Post by ranciso » Wed Apr 19, 2017 3:33 pm

Hello,
We are seeing extremely high load times and iowait times, sometime over 50 load average. There doesn't seem to be a lot of disk activity related to the wait time (see attached). I'm having a really hard time pinning this issue down. Here is our zm setup:

ZM on a VM
16 vCPUs Physical CPUs on host are at 3GHz
32GB of RAM
26TB Disk Direct Attached SAS (RAID5)
CENTOS 7.2.1151 (core)
100 Monitors @ 2fps
I have a filter setup to remove events after 30 days.
I've tried several configuration under options related to zmaudit and how often to run the filter and nothing seems to bring down the load.
Network seems to be ok at max we are pulling about 12Mbps.

Any help would be appreciated. Thanks
Attachments
Screen Shot 2017-04-19 at 11.20.59 AM.png
top screen
Screen Shot 2017-04-19 at 11.20.59 AM.png (273.51 KiB) Viewed 1395 times
Screen Shot 2017-04-19 at 11.21.11 AM.png
iotop screen
Screen Shot 2017-04-19 at 11.21.11 AM.png (308.02 KiB) Viewed 1395 times
Screen Shot 2017-04-19 at 11.28.51 AM.png
options screen
Screen Shot 2017-04-19 at 11.28.51 AM.png (300.12 KiB) Viewed 1395 times

ranciso
Posts: 20
Joined: Wed Apr 19, 2017 3:19 pm

Re: Extremely high iowait time & load

Post by ranciso » Wed Apr 19, 2017 4:57 pm

More info on this - we have about 49 monitors doing motion detection while the others are set to continuously record.

User avatar
iconnor
Posts: 1228
Joined: Fri Oct 29, 2010 1:43 am
Location: Toronto
Contact:

Re: Extremely high iowait time & load

Post by iconnor » Wed Apr 19, 2017 5:41 pm

Ok, so you get that raid5 means that the write speed is that of 1 disk. So all the zma's are trying to write to your raid5 and are backed up.

ranciso
Posts: 20
Joined: Wed Apr 19, 2017 3:19 pm

Re: Extremely high iowait time & load

Post by ranciso » Wed Apr 19, 2017 6:20 pm

I understand how raid 5 works and I do get that the zma's are getting backed up but it doesn't seem like they should be with the disk sub system being what is.

ranciso
Posts: 20
Joined: Wed Apr 19, 2017 3:19 pm

Re: Extremely high iowait time & load

Post by ranciso » Thu Apr 20, 2017 3:10 pm

Fixed my problem. Turns out I was using the LSI Logic SCSI controller for the VM. Switched to para virtualized SCSI controller. Enabled pvscsi driver in OS. Had to modify vmx file to use pvscsi. Removed and re-added disks to PV Controller. Also had to removed vm from inventory and re-registered the vm in vcenter. Some info here: http://allesschonvergeben.blogspot.com/ ... apter.html

Night and day difference. Load is now at around 2-4. Hopefully this helps others.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests