Server OS Upgrade Performance hit (Ubuntu 18.04)

Forum for questions and support relating to the 1.30.x releases only.
Locked
basketcase
Posts: 31
Joined: Thu Aug 22, 2013 10:04 pm

Server OS Upgrade Performance hit (Ubuntu 18.04)

Post by basketcase »

Hi Folks,

I spent the weekend upgrading Ubuntu 17.04 now quite long out of support to 18.04 LTS. To give myself time and ease of rollback, I grabbed a new SSD and started fresh knowing I can plug in the former build and get the server is back online.

So I followed the wiki https://wiki.zoneminder.com/Shell_scrip ... all-bionic which worked a treat. Thanks for that!

After standing everything up on the same hardware, DB restore, config and everything is mirrored, sits at double it's former utilisation. Both installs are ZM 1.30.4

Ubuntu 17.04 CPU Load Around 2.0 doing it's all cams Mocord normal run

Ubuntu 18.04 CPU Load Around 4.0-4.5

Both measurements are after a few hours so the server is "settled" doing it's thing. I've since rolled back to the 17.04 SSD so I can troubleshoot.

The only differences are the 17.04 was actually a Desktop build, I kept the 18.04 to the server only and only installed the dependencies as I wanted to keep it trim... Both were running an OpenHab instance also, but it runs pretty light.

If anyone can give me some starting points to look at the camera streams themselves, I can see it's all across the ZMC's and all are just a 2%ish higher
Nothing obvious in the logs... if I didn't know it normally ran at half the utilisation I'd think it was a healthy build (aside from over utilising the 4 available cores...)

Any suggestions appreciated
Cheers!
Attachments
snips.jpg
snips.jpg (166.37 KiB) Viewed 4480 times
jwarfin
Posts: 41
Joined: Mon Jul 23, 2018 4:36 am

Re: Server OS Upgrade Performance hit (Ubuntu 18.04)

Post by jwarfin »

I'm assuming you upgraded the same PC/server to 18.04 - so, no hardware changes (other than the SSD).

So, according to this: https://blog.ubuntu.com/2018/01/12/melt ... tus-update , 17.04's kernel never got any of the Spectre or Meltdown mitigations. However, 18.04 does have all the mitigations. The mitigations result in a big hit on performance and especially related to I/O (eg: disk, net, I/O related to buffers/caches in memory, etc.). The performance hit varies from case to case, but in general a hit of 20-30% is common. High I/O apps (like zoneminder) can take a bigger hit than that. Really depends on the app and CPU. At the moment, AMD CPUs (in general) only need mitigations for Spectre. However, Intel CPUs (in general), need mitigations for both Spectre and Meltdown.

The mitigations can be turned off through kernel boot options: https://wiki.ubuntu.com/SecurityTeam/Kn ... onControls

If a system/server is on a well protected subnet and is not used for accessing the internet (ie: people using web browsers and the like), then some sysadmins may choose to disable the mitigations. It's a classic choice quandry: performance vs risk.
basketcase
Posts: 31
Joined: Thu Aug 22, 2013 10:04 pm

Re: Server OS Upgrade Performance hit (Ubuntu 18.04)

Post by basketcase »

Good one! I should have thought of that, I remember when we patched Spectre at work and seeing up to 15% performance hits, but those are proper enterprise servers, not my flogged out consumer level already high I/O ZM server - credit though it's run for about 6 years 24/7 smashing camera feeds without issue.

Thanks for that, you've given me something to test tonight. Exactly what I was after!
bbunge
Posts: 2930
Joined: Mon Mar 26, 2012 11:40 am
Location: Pennsylvania

Re: Server OS Upgrade Performance hit (Ubuntu 18.04)

Post by bbunge »

Did you do a minimum install with a net install iso? I use that in lieu of a server install as I can get a really minimal system. I do not need the containers or other stuff in a server install to run ZM. I have not noticed a performance hit with Bionic but have to admit my test platform uses just a couple of cameras on an i3 with a laptop hard drive.
jwarfin
Posts: 41
Joined: Mon Jul 23, 2018 4:36 am

Re: Server OS Upgrade Performance hit (Ubuntu 18.04)

Post by jwarfin »

bbunge wrote: Mon Aug 06, 2018 2:43 am Did you do a minimum install with a net install iso? I use that in lieu of a server install as I can get a really minimal system. I do not need the containers or other stuff in a server install to run ZM. I have not noticed a performance hit with Bionic but have to admit my test platform uses just a couple of cameras on an i3 with a laptop hard drive.
I've been tuning an Xubuntu Bionic install for a couple weeks now. Wish I'd just done a minimum install as you describe. But I've de-installed and turned off so much non-essential stuff my system is probably equivalent to a minimal setup by now.

At this point my 1.30.4 ZM system has 7 IP cams: 720p@6fps 8bit grey scale. I still have to add a few old school analog cams.

Of all the things I've done to improve performance, the things that seem to have contributed the most improvement:

- Running 8bit grey scale at a lower frame rate
- Using mysqltuner regularly and tuning the database
- Setting spectre_v2=off in kernel boot params.
- Tuning kernel params for better network performance
- NOT logging to the database.

I've also tuned the typical ZM options in in favor of performance.

I'm presently running with Motion Frame Skip set to 2 for each cam. This is supposed to reduce analysis load somewhat and I'm seeing that. So far I'm not having any issues with motion events being missed - but I'm still evaluating.

Hardware wise, the biggest performance contributors are a fast multi-threaded CPU (Ryzen 2600) and plenty of fast ram (16gb of DDR4 @ 2666mhz).

Anyway my system now hovers at around 0.30 load with all cams running (no motion events). The actual CPU usage is typically low - like 8-15%. What contributes the most to the load factor numbers on my system is I/O related.
basketcase
Posts: 31
Joined: Thu Aug 22, 2013 10:04 pm

Re: Server OS Upgrade Performance hit (Ubuntu 18.04)

Post by basketcase »

I've had a busy week and didn't find the time to test this until now.

Testing only the spectre kernel changes firstly...

For anyone who's not done it before, In ubunutu 18.04 making only the changes to grub

Code: Select all

sudo nano /etc/default/grub
In grub… modifying only this line adding: spectre_v2=off nopti pti=off

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity spectre_v2=off nopti pti=off"
exit

Code: Select all

sudo update-grub
reboot.

After 3 hours bedding in, same services running it's very close to the former utilisation, in fact, marginally improved.

CPU load averages: 1.92 (1 mins) , 1.82 (5 mins) , 1.73 (15 mins)

However, I have now exposed a known CVE on an internet facing server which isn't great but given it is a home server and I have a number of compensating controls I'm comfortable with it, but each to their own.

Thanks for the suggestion, the impact on ZM of the Page table isolation was far greater than I'd expected.
jwarfin
Posts: 41
Joined: Mon Jul 23, 2018 4:36 am

Re: Server OS Upgrade Performance hit (Ubuntu 18.04)

Post by jwarfin »

I'm running an AMD processor. I tested "spectre_v2=off" alone and then included "nopti pti=off" & tested again. I didn't see much more improvement with the added "nopti pti=off", so I just boot with "spectre_v2=off" .
Locked