Microsoft patch MS14-066 leads to https problems with IIS and Google Chrome

Patch Tuesday – two words that will bring some amount of fear in the hearts of many Microsoft Windows administrators. It’s the day of the month (usually the second Tuesday) when Microsoft releases its monthly batch of software updates across the entire family of Microsoft products. Most of the time updates will just be applied without any problems (other than the mandatory system restart) but every now and then things just don’t go as planned … in some very extreme cases the updates lead to blue screens, application crashes and pretty much all-nighters for the stressed Windows admins.

To some extent a little bit of this happened this week. On 11/11/14 Microsoft made available the updates for November 2014. Among the various updates a particular one of notice is MS14-066 (KB2992611) – https://technet.microsoft.com/library/security/MS14-066.

It’s a security patch marked as critical affecting pretty much every operating system (both client and server) that Microsoft currently supports. The ‘critical’ label comes from the fact that it fixes a remote code execution bug which pretty much means that an attacker can do something bad to a system without any action being needed on part of the user. This particular patch deals with a problem in SChannel which is Microsoft’s component that deals with https traffic and cryptography.

It was pretty hard to read the news this week and not come across some story or article that referred to MS14-066 as the Windows Shellshock bug (in reference to Shellshock and Heartbleed – two very serious security bugs that were discovered in the last few months affecting the open source community and *nix operating systems in particular). Bottom line – this is one patch for Windows that you want to install as soon as possible especially if you maintain IIS web servers that run https sites.

And so we did. We applied this to servers running Windows Server 2012 with IIS and all of a sudden we started seeing problems with https and Google Chrome. Users running the latest version of Google Chrome (as of now 38.0.2125.122) were not able anymore to hit https sites served by IIS servers that had this patch applied to them. Chrome users were getting some very strange errors such as ERR_CONNECTION_ABORTED and ERR_TIMED_OUT. At the same time, users with Internet Explorer 11 and Firefox 33 (latest versions for those browsers) were able to browse the same https sites without a problem.

Very odd … what exactly is going on here?

It appears to be a problem that only comes up under some very specific (and not fully understood) conditions. It’s unclear for example if both the server and the client OS must have MS14-066 installed in order for the Google Chrome problems to start happening.

It took a few days but slowly signs that others were seeing similar issues began to emerge. See for example https://social.technet.microsoft.com/Forums/windowsserver/en-US/218cf562-3dab-4d09-adcc-74f65d0f29f1/winshock-kb2992611-patch-breaks-iis?forum=winserversecurity.

The Microsoft KB support page for KB2992611 began to offer some hints that not everything was ok – https://support.microsoft.com/kb/2992611. It provides some details on how to remove 4 new security ciphers that were added as part of this update – ciphers that apparently can lead to some problems with clients trying to establish https connections using TLS 1.2 (Google Chrome being one of them).

So what can system administrators do if they have systems that are encountering this problem? The choices are few and none of them is really ideal:

  • wait for Microsoft to fully and properly fix this issue (is it really Microsoft’s fault? did Microsoft mess up something with the new ciphers and TLS 1.2?)
  • wait for Google to fix Chrome (why is Chrome the only browser having problems with this update? is it doing something non-standard when it comes to TLS 1.2?)
  • attempt to remove just the 4 new ciphers according to Microsoft’s instructions (this involves registry edits and does not appear to actually work since some of the registry keys mentioned are not directly accessible by administrators due to how permissions on those keys are set up)
  • fully uninstall the KB2992611 patch and stay unsecured (we confirmed that uninstalling that patch from the IIS server enables Google Chrome users to again connect via https to those sites)
  • either disable https until this is fixed or terminate the https connection in front of the IIS web server using a load balancer

What are some lessons that we can learn from here?

First – Microsoft still has some work to do in its QA processes … somehow it let this Google Chrome issue go by uncaught (yes Microsoft – please note that there are users out there who use browsers other than Internet Explorer).

Second – this is really the important lesson from a software development lifecycle point of view: don’t mix security fixes with new features in the same patch. Never. Ever.
MS14-066 should have been just a security fix. You found some new vulnerability that you have to address? Great, stick it in a patch all on its own. Don’t mix in some new features (such as additions to available TLS cipher suites – see the “Does this update contain any additional security-related changes to functionality?” answer in the FAQ portion of the bulletin) in the same patch that fixes some critical security bug. As it so happens, the new features are now the ones that actually break functionality but Microsoft has no easy way to just tell people to remove the new features because that involves removing the critical fix which it still wants users to keep.

Oh well … just another fun day in IT.

Update: Amazon AWS is aware of this issue involving TLS 1.2 after the update is applied and has an advisory page for it: http://aws.amazon.com/security/security-bulletins/ms14-066-advisory/

Update 2: disabling TLS 1.2 on the server appears to fix the problem (temporarily) because that forces clients to use at most TLS 1.1 and get around the new ciphers added by Microsoft

One can also use the nice IIS Crypto tool – https://www.nartac.com/Products/IISCrypto/Default.aspx to manage security protocols and ciphers on IIS.

Update 3: this blog post was picked up by a very comprehensive InfoWorld article collecting data about other problem reports with this patch (thank you Woody Leonhard).

Update 4: I also came across a blog post by Toby Meyer where he documents how the default order of cipher suites was actually modified by this patch on Windows Server 2008 R2 and Windows Server 2012 … yet another twist … a ‘feature/improvement’ that was not mentioned anywhere in Microsoft’s technical documentation for the patch.

16 thoughts on “Microsoft patch MS14-066 leads to https problems with IIS and Google Chrome

  1. Why not edit your Chrome shortcut to *not* use TLS 1.2 (temporarily)?

    From http://www.saotn.org/time-disable-sslv3-or-what/:
    ====
    Due to a bug(*) in the Chrome UI you have to specify a command line parameter for the chrome.exe command (shortcut):
    –ssl-version-max – Specifies the maximum SSL/TLS version (“ssl3″, “tls1″, “tls1.1″, or “tls1.2″)
    –ssl-version-min – Specifies the minimum SSL/TLS version (“ssl3″, “tls1″, “tls1.1″, or “tls1.2″)

    Your shortcut command would become:

    “C:\Program Files (x86)\Google\Chrome\Application\chrome.exe” –ssl-version-min=tls1 –ssl-version-max=tls1.1

    (*) https://productforums.google.com/forum/#!topic/chrome/mE-KUuYBkSU
    ====

  2. I have experienced this error on 2012R2, 2008R2, and even 2008 server to varying degrees on each kind of system. (2008 lost https, sftp, and RDP) And none of the “fixes” or workarounds have restored proper https function either, including uninstalling the patch. I have removed the new ciphers as Microsoft directed, turned off TLS1.2, and uninstalled patch, none of which has made any difference on any of the above OS’s. I had previously made registry changes to disable ss3.0 and enable secure ciphers on most of the affected servers as well for PCI compliance, but everything was fully functional before this patch, although those changes being there could have complicated how the patch was applied. Those change however don’t seem that uncommon since a lot of people had to make similar changes once ssl3 was exposed, so it’s not like I did something unusual.
    Hoping to see a fixed patch from MS soon so I can finish patching my unpatched servers and the server I had to restore after patching disabled most secure functions.

  3. FYI, this post is ignoring the fact that the antivirus might be causing this issue – before you disable anything, download the latest definitions for your antivirus software.
    I have trend, as soon as I downloaded the definitions everything works again.

    • I did consider that aspect when troubleshooting on some of my systems (especially if the antivirus software is configured to inspect web traffic realtime) but in the end I didn’t mention it because I certainly came across problems with systems running in virtual machines where no antivirus was even installed.

      Even if it is the case that antivirus in web filtering mode would start working again with the https changes after its own internal updates – the point of the article still stands I think: Microsoft in this particular patch introduced undocumented changes in IIS to the https protocols and ciphers – changes which appear to affect more and more devices as new reports of problems continue to surface.

  4. Microsoft re-released update MS14-066 / update 2992611

    https://technet.microsoft.com/library/security/ms14-066
    – Reason for Revision: V2.0 (November 18, 2014): Bulletin revised
    to announce the reoffering of the 2992611 update to systems
    running Windows Server 2008 R2 and Windows Server 2012. The
    reoffering addresses known issues that a small number of
    customers experienced with the new TLS cipher suites that were
    included in the original release. Customers running Windows
    Server 2008 R2 or Windows Server 2012 who installed the 2992611
    update prior to the November 18 reoffering should reapply the
    update. See Microsoft Knowledge Base Article 2992611 for more
    information.

    (and don’t forget to apply MS14-068!)

  5. We are still facing this issue. None of my users (Windows 7) have been able to access a particular site using https with Chrome. I modified the shortcut like your suggestion but it still doesn’t load. I even tried deleting the registry keys suggested in the chrome forum but no go as well. This is getting ridiculous.

Leave a Comment