Patch Tuesday – two words that will bring some amount of fear in the hearts of many Microsoft Windows administrators. It’s the day of the month (usually the second Tuesday) when Microsoft releases its monthly batch of software updates across the entire family of Microsoft products. Most of the time updates will just be applied without any problems (other than the mandatory system restart) but every now and then things just don’t go as planned … in some very extreme cases the updates lead to blue screens, application crashes and pretty much all-nighters for the stressed Windows admins.
To some extent a little bit of this happened this week. On 11/11/14 Microsoft made available the updates for November 2014. Among the various updates a particular one of notice is MS14-066 (KB2992611) – https://technet.microsoft.com/library/security/MS14-066.
It’s a security patch marked as critical affecting pretty much every operating system (both client and server) that Microsoft currently supports. The ‘critical’ label comes from the fact that it fixes a remote code execution bug which pretty much means that an attacker can do something bad to a system without any action being needed on part of the user. This particular patch deals with a problem in SChannel which is Microsoft’s component that deals with https traffic and cryptography.
It was pretty hard to read the news this week and not come across some story or article that referred to MS14-066 as the Windows Shellshock bug (in reference to Shellshock and Heartbleed – two very serious security bugs that were discovered in the last few months affecting the open source community and *nix operating systems in particular). Bottom line – this is one patch for Windows that you want to install as soon as possible especially if you maintain IIS web servers that run https sites.
And so we did. We applied this to servers running Windows Server 2012 with IIS and all of a sudden we started seeing problems with https and Google Chrome. Users running the latest version of Google Chrome (as of now 38.0.2125.122) were not able anymore to hit https sites served by IIS servers that had this patch applied to them. Chrome users were getting some very strange errors such as ERR_CONNECTION_ABORTED and ERR_TIMED_OUT. At the same time, users with Internet Explorer 11 and Firefox 33 (latest versions for those browsers) were able to browse the same https sites without a problem.
Very odd … what exactly is going on here?
It appears to be a problem that only comes up under some very specific (and not fully understood) conditions. It’s unclear for example if both the server and the client OS must have MS14-066 installed in order for the Google Chrome problems to start happening.
It took a few days but slowly signs that others were seeing similar issues began to emerge. See for example https://social.technet.microsoft.com/Forums/windowsserver/en-US/218cf562-3dab-4d09-adcc-74f65d0f29f1/winshock-kb2992611-patch-breaks-iis?forum=winserversecurity.
The Microsoft KB support page for KB2992611 began to offer some hints that not everything was ok – https://support.microsoft.com/kb/2992611. It provides some details on how to remove 4 new security ciphers that were added as part of this update – ciphers that apparently can lead to some problems with clients trying to establish https connections using TLS 1.2 (Google Chrome being one of them).
So what can system administrators do if they have systems that are encountering this problem? The choices are few and none of them is really ideal:
- wait for Microsoft to fully and properly fix this issue (is it really Microsoft’s fault? did Microsoft mess up something with the new ciphers and TLS 1.2?)
- wait for Google to fix Chrome (why is Chrome the only browser having problems with this update? is it doing something non-standard when it comes to TLS 1.2?)
- attempt to remove just the 4 new ciphers according to Microsoft’s instructions (this involves registry edits and does not appear to actually work since some of the registry keys mentioned are not directly accessible by administrators due to how permissions on those keys are set up)
- fully uninstall the KB2992611 patch and stay unsecured (we confirmed that uninstalling that patch from the IIS server enables Google Chrome users to again connect via https to those sites)
- either disable https until this is fixed or terminate the https connection in front of the IIS web server using a load balancer
What are some lessons that we can learn from here?
First – Microsoft still has some work to do in its QA processes … somehow it let this Google Chrome issue go by uncaught (yes Microsoft – please note that there are users out there who use browsers other than Internet Explorer).
Second – this is really the important lesson from a software development lifecycle point of view: don’t mix security fixes with new features in the same patch. Never. Ever.
MS14-066 should have been just a security fix. You found some new vulnerability that you have to address? Great, stick it in a patch all on its own. Don’t mix in some new features (such as additions to available TLS cipher suites – see the “Does this update contain any additional security-related changes to functionality?” answer in the FAQ portion of the bulletin) in the same patch that fixes some critical security bug. As it so happens, the new features are now the ones that actually break functionality but Microsoft has no easy way to just tell people to remove the new features because that involves removing the critical fix which it still wants users to keep.
Oh well … just another fun day in IT.
Update: Amazon AWS is aware of this issue involving TLS 1.2 after the update is applied and has an advisory page for it: http://aws.amazon.com/security/security-bulletins/ms14-066-advisory/
Update 2: disabling TLS 1.2 on the server appears to fix the problem (temporarily) because that forces clients to use at most TLS 1.1 and get around the new ciphers added by Microsoft
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server] "Enabled"=dword:00000000
One can also use the nice IIS Crypto tool – https://www.nartac.com/Products/IISCrypto/Default.aspx to manage security protocols and ciphers on IIS.
Update 3: this blog post was picked up by a very comprehensive InfoWorld article collecting data about other problem reports with this patch (thank you Woody Leonhard).
Update 4: I also came across a blog post by Toby Meyer where he documents how the default order of cipher suites was actually modified by this patch on Windows Server 2008 R2 and Windows Server 2012 … yet another twist … a ‘feature/improvement’ that was not mentioned anywhere in Microsoft’s technical documentation for the patch.