Stop the "Windows update madness" on WS2016!
Dear Microsoft Windows Server team. We would like _complete control_ over when we would like our our WS2016 to install windows updates! No more slow / timeconsuming server restarts because of "Getting Windows ready. Do not turn off your computer" messages when we just would like a quick restart of a production server.. No more TiWorker.exe using 100% of a CPU core during "*********" and so on.. And please fix the issues with '2017-08 Update for Windows Server 2016 for x64-based Systems (KB4035631)' asap. Thank you.
KjB
shared this idea
76 comments
-
Karl W-E.
commented
There was a discussion about a workaround on patchmanagement.org Google Group
here it is - read the outcome below."%SystemRoot%\System32\sfc.exe" /scannow
"%SystemRoot%\System32\Dism.exe" /online /Cleanup-Image /AnalyzeComponentStore
"%SystemRoot%\System32\Dism.exe" /online /Cleanup-Image /CheckHealth
"%SystemRoot%\System32\Dism.exe" /online /Cleanup-Image /ScanHealth
"%SystemRoot%\System32\Dism.exe" /online /Cleanup-Image /RestoreHealth /Source:WIM:<your install.wim location and index>
"%SystemRoot%\System32\Dism.exe" /online /Cleanup-Image /StartComponentCleanup"Optional: "%SystemRoot%\System32\Dism.exe" /online /Cleanup-Image /StartComponentCleanup /resetbase
Verdict: The workaround – as above - appear to work just on GUI based servers, that are default and never had run a cleanup, or install / uninstall of features.
Advice: check the notes [1]-[4] before making early conclusionsGood news: it will speed update update process
Bad news: it will still take ages compared to later Server 2016 releases, just as 1703 or 1803 or later (SAC) or 2019 1809 LTSC / SAC or laterAt a glance:
Sfc + dism healthchecks and clean-ups (as commands below)2016 1607 LTSC GUI: 2 hours 9 minutes 18 seconds[1]
2016 1607 LTSC Core: 40 minutes 49 seconds[1]Patching after clean-ups and restart:
Searching for updates:
2016 1607 LTSC GUI: no time – update list was already cached (usually 13-20 seconds)
2016 1607 LTSC Core: 20 seconds (VM Uptime 1:21) [3]Download:
2016 1607 LTSC GUI: 8 minutes 34 seconds (start VM Uptime 1:05, end 9:39) [2][3]
2016 1607 LTSC Core: 12 minutes 8 seconds (start VM Uptime 1:26, end 13:34) [2][3]Install:
2016 1607 LTSC GUI: 8 minutes 20 seconds (start VM Uptime 9:39, end 17:54)
2016 1607 LTSC Core: 13 minutes 22 seconds (start VM Uptime 13:34 end 26:56)Restart + Logon:
2016 1607 LTSC GUI: 6 minutes 44 seconds (start VM 18:08, end 23:18+1:34)
2016 1607 LTSC Core: 6 minutes 50 seconds (start VM Uptime 27:51, end 34:06+0:35)Total Time:
2016 1607 LTSC GUI: with cleanup: 23 minutes 58 seconds / without cleanup: 44 minutes 28 seconds, as tested in February with latest SSU[1][4]
2016 1607 LTSC Core: with cleanup: 32 minutes 40 seconds / without cleanup: 23 minutes 26 seconds, as tested in February with latest SSU[1][4]2016 1803 LTSC Core: without cleanup: 6 minutes 56 seconds, as tested in February with latest SSU[1][4]
2019 1809 LTSC Core: without cleanup: 3 minutes 56 seconds, as tested in February with latest SSU[1][4]
2019 1809 LTSC GUI: without cleanup: 4 minutes 40 seconds, as tested in February with latest SSU[1][4][1]Note this process will very like take a lot longer in usual virtual environments. I’ve 4 cores + HT at 4.6 GHz and full NVMe Flash! (usual Hyper-V scheduling, full spectre / NG enabled)
[2] note that DL is bugged in Server 2016 1607. Taskmanager will not show any network activity and the TIworker Process will run after download and already prepare update installation
While it is still saying “download”. The download is fairly finished in a few time (450 Mbits downstream).
[3] VM Uptime are mentioned for time measure references and later video editing
[4]https://www.askwoody.com/2019/server-2016-ltsc-patches-take-for-e-ver-there-are-numerous-reasons-why-and-not-much-you-can-do-about-it/Interesting facts: Note that Server 2016 1607 Core, unlike its GUI pendant does not benefit from the cleanup. Likely because its component store is less overloaded with pre-installed roles / features.
In junction the cleanup was much faster ready than on the GUI pendant, as there was apparently less to do.Best regards,
Karl (@tweet_alqamar) -
Karl W-E.
commented
Hi Koen,
I've suggested that long ago. You need Software assurance and also take care of your RDP and user / device CALS for Windows Server being updated. Most upgrade servers and forget about these CALs. -
Koen
commented
Just updated my server to 1809 so server 2019(inplace upgrade). This is the fastest way to get updates :-) took les then an hour !
installing updates afterword just works as it should. -
Koen
commented
Happy I found this site. I thought there was a problem with the hardware of my machine :-) then I found out that there is a high disk usage to get the files into the c: \windows\softwaredistribution\ folder so constantly working on updates.
regeards,
Microsoft should get us a free license to upgrade to 1809!
-
Anonymous
commented
On our server the March 12, 2019 CU (KB4489882) took nearly *four hours* to install. Microsoft, please, please fix this.
-
Anonymous
commented
We peaked at 3 hours last month with installing a single Cumulative update patch during our Server downtime. That is unacceptable. We have server environments that are three or four dependencies deep. We can't afford to be down for that kind of time especially given we only have a 6 hour downtime window to patch over 1,000 servers in our datacenter. Also, the file size is much too large. Last month it clocked in a 1.37 GB!
We need Microsoft to find a way to reset this with Windows 2016 server the way they do with Win 10 versions (patch size gets "reset" to a smaller size with each version upgrade) Why can't Microsoft find a way to port that same functionality into Windows 2016?
-
Jerem
commented
2 or 3 hours with this fu****** Getting Ready after reboot when we install update on 2016 LTSC
Thanks Microsoft !!! -
Anonymous
commented
WU is a nightmare here too (Windows Server 2016 1607)... for no reason some servers are updating fine in 5-10 minutes with the CU 2019-04, while others can fail up to 10 times with the timeout error code 0x800705b4 (even clean install...), just click retry 'till it would finally work and take hours to complete...
+ Windows Update take 50% of the CPU... migrating again to W2019 is not a solution, to me server 2016 is like Windows Vista :)
We update throught WSUS.
-
Karl W-E.
commented
Hi John, I cannot agree there is any improvement on my lab.
I've tried following:2016 1607 LTSC GUI patched to 2019-02, incl. SSU - not ok
2016 1607 LTSC Core patched to 2019-02, incl. SSU - not ok2016 1803 SAC Core patched to 2019-02, incl. SSU - ok
2019 1809 LTSC Core patched to 2019-02, incl. SSU - ok
2019 1809 LTSC GUI patched to 2019-02, incl. SSU - okall trying to update to 2019-03. not ok mean - not improvment in speed compared to my previous benchmarks / comparisons.
another thing that I noticed while comparing. During "download", when it is really downloading, there is no network activity in the VM taskmanager in 1607 LTSC GUI only.
2019 LTSC GUI shows correct network activity while downloading.Imho 2016 is "limited" in many aspects. Windows Update is just one of them.
https://twitter.com/tweet_alqamar/status/1107188019268849664
-
Karl W-E.
commented
Hi John,
I will recap the situation with the 2019-03 updates. All VMs I've measured had previous updates. I currently doubt there will be any improvement but I really want to double check your findings.Yesterday updated 2016 1607 2x GUI 2xCore at a customer 4 VMs from 2018-08 to 2019-02 via Windows Admin Center. The process took about about 45 Minutes including restarts
-
John
commented
Hi guys, I did some testing too with W2016 GUI boxes too and I experienced there is huge difference between the clean install update and when you work with updates aplied after Jan 2019.
As I have tested it, it took about 6 hours to make a first update with .iso image MLF_X21-3035 downloaded from Microsoft and used for a clean install. But since applied patches in the end of Jan it's just a matter of minutes to update the install and it's very fast (Feb 2019 updates took nearly 7mins to apply and WinDef daily updates are just a matter of 1-2 mins without restart - using sconfig or UI it doesnt matter - i am testing it behind FW + proxy and other HTTP protections on the way).
Now I am working on W2019 Core and UI tests so hopefully will get some numbers too. -
Karl W-E.
commented
" a fix to the current Service Stack might have fixed it in those who are still using 1607 if I'm reading what you said, though on reread it seems this only fixed a DL and not the install/reboot."
Currently there is no fix. It can happen though on fresh installs that the display of download / installing phase are displayed correctly without affecting the total time.
"Do you have more information on the procedure you were using to update the OS of the existing 1607 boxes to 1809? A link to documentation or perhaps what exactly to look for?"
There is no documentation for this as this does not work. GUI <> Core switch is no longer possible (was with 2012R2 but got removed with 2016).
If you need GUI and have a license or active SA for CALs and Server you can easily mount your 2019 ISO and do an inplace upgrade. (Backup / temporarily snapshot in backhand)
Ironically this upgrade wents nearly faster than any monthly CU installation ;)
Not telling there won't be a fix at all in future - while I doubt it - as it disagrees with the LTSC terminology - but all efforts to get MS involved directly failed so far.
Maybe being a GOLD partner is not enough.Thanks for everyone your feedback and appreciations of my works. It helps a lot get equal the investments of free time into this topic.
-
Karl W-E.
commented
"Not seeing a 180* version of 2016 with GUI at all. :( So guessing we either need core instances or start using 2019. :/"
Yes it is like this it is definitely fixed in Server 2016 1809 Core, which means there is no transition from GUI to Core without reinstalling + you may have limitations with apps - lesser of these with 2019 Core (due to Feature on Demand package).
However you need an active Software Assurance for the Server 2016 Core 1809 as it is an SAC branch. you may not use it without active SA. On top SAC Server Core branch has only 18 months support but inplace upgrades are the way to get up just as with GUI from 2016 > 2019 etc.
tl;dr So either you can transist to 2016 or 2019 Core with Software Assurance or use 2019 GUI to workaround this issue.
We have had IT press coverage already and I am trying to get more involved.please note by terms of use it says Server Uservoice is NOT for bug reporting - so reporting here might not be the right place. Reporting via the MS Support however they likely let you down to tell you it is an isolated issue - which it is not.
-
Mike Estes
commented
Ah, I think I misunderstood what you said and what is available. Not seeing a 180* version of 2016 with GUI at all. :( So guessing we either need core instances or start using 2019. :/
-
Mike Estes
commented
Hi Karl, thank you for taking the time to investigate this issue, was one of my tasks to finally get back to as this has gotten much worse over time. This last patch Tuesday was a nightmare. Probably due to having more boxes on that version of Server 2016.
What's the TLDR? I see you mention in an earlier suggestion to update the OS to a newer version 1809, but it sounds like a fix to the current Service Stack might have fixed it in those who are still using 1607 if I'm reading what you said, though on reread it seems this only fixed a DL and not the install/reboot.
Do you have more information on the procedure you were using to update the OS of the existing 1607 boxes to 1809? A link to documentation or perhaps what exactly to look for?
Thank you,
Mike -
Karl W-E.
commented
John / EFD no need to argue. Please check my stats. The issue is any Windows 10 / Server 2016 1607 alone. Any later does not have any issues with updating and Performance.
Belongs to Server 2019 GUI / Core (1809) as well as Server 2016 Core 1803 etc. -
John
commented
2 EFD - having no internet connection at the time of Azure & AWS and hybrid infrastructure? Sorry but its 21st century and advanced firewalling is the solution for today not to be closed in a bunker. And when I compare Linux machines, clean installs, updates with W2016/2019 it's like a heaven and ****. In case of Linux it's transparent, all infos & progress provided to admins every second and all update process takes a few minutes. In case of W2016 I had been working on clean installs this weekend and I saw rolling icon & don't turn of you computer for many hours. However W2016 is fast and stable system but WU it's something horrible. And SCCM has nothing to do with it because WU is internal and not the orchestration around it and SCCM. There can be just improvement like using WSUS and some internal issues W10/W2016 have with searching for online update MS services and downloading http .cab crazy stuff. The new WU is a replacement of a previoud Update services that W10/W2016 provide and this is the crazy stuff that Microsoft introduced with W10.
-
EFD
commented
Also, what are your updates scoped to? Are you installing critical CVE patches or everything? I can assure you that installing anything other than CVE and security patches is the wrong way to go so that may be why we dont see this issue. We dont install anything else, there is no point
-
EFD
commented
I'm curious to know what you are using as an update platform? We have no issues on server 2016/2019 with updates. We are using SCCM, but then again our configuration is fine tuned and our servers have zero internet connection, which should be the standard for any corporation looking to secure its infrastructure
-
Anonymous
commented
Server 2016 updates = TOTAL NIGHTMARE !!!