Virtual Security Research, LLC. http://www.vsecurity.com/ Security Advisory -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -=-=-=- Advisory Name: Multiple Cisco CSS / ACE Client Certificate and HTTP Header Manipulation Vulnerabilities Release Date: 2010-07-02 Application: Cisco Content Services Switch (CSS) / ACE Products Versions: Cisco CSS 11500 - 08.20.1.01 Cisco ACE 4710 - Version A3(2.5) [build 3.0(0)A3(2.5) (Other versions may be affected) Severity: High (in specific configurations) Author: George D. Gal <ggal (a) vsecurity . com> Vendor Status: Cisco CSS vulnerability remains unpatched, workarounds available Cisco ACE workarounds available CVE Candidate: CVE-2010-1575 - Certificate Spoofing Flaw CVE-2010-1576 - HTTP Request Parsing Flaw Reference: http://www.vsecurity.com/resources/advisory/20100702-1/ -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -=-=-=- Product Description ------------------- From [1]: "The Cisco CSS 11500 Series Content Services Switch is a high-performance, high-availability modular architecture for Web infrastructures. As the premiere switch for the Cisco Web Network Services Software, the Cisco CSS 11500 Series helps businesses to build global Web networks optimized for content delivery and e-commerce. By activating HTTP headers, the CSS 11500 Series helps to ensure availability, optimize utilization, reduce latency, increase scalability, and enhance security for Websites, server farms, cache clusters, and firewall systems." From [2]: "Cisco(R) ACE Application Control Engine application switches represent the state of the art in next-generation application switches for increasing the availability, performance, and security of data center applications. The Cisco ACE family of application switches includes the Cisco ACE Service Module for the Cisco Catalyst(R) 6500 Series Switches and Cisco 7600 Series Routers, as well as the Cisco ACE 4710 Appliance in a standalone form factor for discrete data center deployments." Vulnerability Overview ---------------------- On June 4th 2009, VSR identified multiple weaknesses in the Cisco CSS 11500's handling of HTTP header interpretation and client-side SSL certificates. Individually, these issues may be considered minor, but combined they could allow for the compromise of an application that relies on a vulnerable CSS to assist in authenticating clients. If successfully exploited, an attacker could spoof another application user's identity without possession of the victim's client certificate. Additionally, due to the fact that the Cisco CSS product has been effectively superceded by the Cisco ACE, VSR has also identified similar issues to those described below in the ACE in particular configurations. These issues may affect any CSS installation, but would have the greatest impact on deployments that have the following feature enabled in the configuration: ssl-server <context> http-header client-cert Similarly, on the Cisco ACE, these issues may manifest themselves when using a policy map with a class-default class, as shown below: policy-map type loadbalance first-match SLB-VIP-REDIRECT class class-default serverfarm TEST-FARM action DO-SOMETHING-WITH-HEADERS insert-http X-SRC-IP header-value "%is" Issue 1: Weak Enforcement of Authority in HTTP Certificate Headers ------------------------------------------------------------------ Cisco Bug Id - CSCSZ04690 Affects - Cisco CSS The first weakness affecting the Cisco CSS is that, in a typical client certificate configuration, HTTP clients may confuse web applications by injecting their own certificate headers. When utilizing the CSS to terminate SSL communications, SSL client certificates are first authenticated by the CSS. From there, the CSS will normally pass the client's identity to the back-end web server in the form of several HTTP headers as shown below: ClientCert-Subject: XXX ClientCert-Subject-CN: XXX ClientCert-Fingerprint: XXX ClientCert-Subject-CN: XXX ClientCert-Issuer-CN: XXX ClientCert-Certificate-Version: XXX ClientCert-Serial-Number: XXX ClientCert-Data-Signature-Algorithm: XXX ClientCert-Subject: XXX ClientCert-Issuer: XXX ClientCert-Not-Before: XXX ClientCert-Not-After: XXX ClientCert-Public-Key-Algorithm: XXX ClientCert-RSA-Modulus-Size: XXX ClientCert-RSA-Modulus: XXX ClientCert-RSA-Exponent: XXX ClientCert-X509v3-Subject-Key-Identifier: XXX ClientCert-X509v3-Authority-Key-Identifier: XXX ClientCert-Signature-Algorithm: XXX ClientCert-Signature: XXX However, there is no attempt by the CSS to prevent clients from supplying their own ClientCert-* headers. Depending on how application developers handle multiple copies of these headers, an attacker may be able to impersonate other users. For example, assuming that a back-end web application simply trusts the user identity supplied by the CSS in the ClientCert-Subject-CN header and userX wants to impersonate userY, he may simply insert the following HTTP header(s) in the HTTP request issued to the server: ClientCert-Subject-CN: CN=userY or ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userY Upon injecting the attacker-supplied HTTP headers the application would receive an HTTP request similar to that shown below: POST /targetapp HTTP/1.1 Content-Type: text/xml; charset=utf-8 ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userY ClientCert-Subject-CN: CN=userY Host: test.vsecurity.com Content-Length: 1024 ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX ClientCert-Subject-CN: CN=userX ClientCert-Fingerprint: XXX ClientCert-Subject-CN: XXX ClientCert-Issuer-CN: XXX ClientCert-Certificate-Version: XXX ClientCert-Serial-Number: XXX ClientCert-Data-Signature-Algorithm: XXX ClientCert-Subject: XXX ClientCert-Issuer: XXX ClientCert-Not-Before: XXX ClientCert-Not-After: XXX ClientCert-Public-Key-Algorithm: XXX ClientCert-RSA-Modulus-Size: XXX ClientCert-RSA-Modulus: XXX ClientCert-RSA-Exponent: XXX ClientCert-X509v3-Subject-Key-Identifier: XXX ClientCert-X509v3-Authority-Key-Identifier: XXX ClientCert-Signature-Algorithm: XXX ClientCert-Signature: XXX Since existing ClientCert-* headers are left intact, application developers are expected to trust only the last instance of a given certificate header. This approach is clearly prone to error if application developers do not carefully test this attack scenario. An alternative approach to securing these headers can be achieved through an optional configuration where the CSS places an additional prefix string on the inserted certificate headers [4]. For instance, a server administrator could select a random header prefix through a command such as: ssl-server <context> http-header prefix "<random_prefix>" This would cause the new certificate headers to be included with the form: <random_prefix>-ClientCert-* So long as an attacker could not discover this random prefix, then there would be no way to spoof these headers. However, this solution is far from ideal, since there may be several ways for an attacker to obtain a copy of these headers. Examples include TRACE/TRACK requests being honored by the back-end web servers or debugging components in web applications which echo client headers. Issue 2: Lack of HTTP Request Validation ---------------------------------------- Cisco Bug Id - CSCTA04885 Affects - Cisco CSS & ACE A second weakness that manifests itself on the CSS and ACE through different interpretation of HTTP newline sequences between the content switch and commonly used web servers. RFC 2616 [3] defines the US ASCII carriage return/line feed (CRLF) sequence as the end-of-line marker for protocol elements (excluding the entity-body). Indeed, the CSS and ACE appear to adhere relatively closely to this requirement. Popular web servers, however, permit various permutations of the CRLF sequence as end-of-line markers, including: LF, CR, and LFCR. This difference in interpretation could lead to serious consequences if the device relies on any of these headers and end-of-line delimiters or makes changes to any of them. This has been well documented in the past in relation to HTTP request smuggling and related attacks [5]. Discussion ---------- It is difficult to consider either of these two issues, taken alone, as extremely critical flaws. In the case of certificate header handling, there are clearly work-arounds for header spoofing attacks, even if they are error-prone or imperfect. For HTTP newline interpretation, it is difficult to fault Cisco for adhering strictly to the RFC. However, in combining these issues in typical deployment scenarios the end result could be catastrophic for an application using a CSS and relying on client certificates for user identification and authorization. During testing, VSR found that use of invalid newline sequences caused the CSS to fail to insert it's own ClientCert-* headers, though the back-end Apache web server accepted these newline sequences. This clearly defeats the approach that some application developers might take in trying to rely only on the last set of certificate headers. The following simple HTTP request demonstrates how a client could cause the CSS to omit its insertion of the ClientCert-* headers: GET /protected_resource HTTP/1.0\x0a ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX\x0a ClientCert-Subject-CN: CN=userX\x0a\x0a However, various combinations of end-of-line delimiters should be possible. In fact, Cisco describes their logic for the fix as follows: "The HTTP request detection will attempt to determine the end-of-line marker following the HTTP/1.x line. If it consists of a single LF, then the logic will attempt to identify a LFLF." Unfortunately, this logic may be problematic when attempting to parse HTTP requests consisting of various combinations of end-of-line markers within the same HTTP request. For instance, many web servers accept requests with mixed newline sequences such as: GET /protected_resource HTTP/1.0\x0d\x0a ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX\x0a ClientCert-Subject-CN: CN=userX\x0a\x0a While testing a Cisco ACE appliance, VSR was able to cause rules that normally add or remove headers in HTTP requests to be omitted when the class-default class was used in a policy map as opposed to a layer 7 class for http communications. It is possible that if a class-default class were used for client certificate processing on the Cisco ACE Module, that client certificate headers could also possibly be omitted, allowing an attacker to insert their own headers used to set the authenticated user context. Cisco's recommendation to utilize a layer 7 policy class map as shown below appears to mitigate this issue, preventing malformed or unexpected end-of-line delimiters: class-map type http loadbalance match-any http-cm 2 match http url .* policy-map type loadbalance http first-match http-pm class http-cm serverfarm TEST-FARM action DO-SOMETHING-WITH-HEADERS insert-http X-SRC-IP header-value "%is" Recommendations for Cisco and Competing Vendors ----------------------------------------------- VSR recommends that Cisco (and any other vendors who provide similar products) implement more stringent request validation and/or corrections when receiving requests which do not utilize HTTP-compliant newlines. While failing to validate requests forwarded to back-end servers is a reasonable approach when requests are not interpreted or modified, VSR believes that some responsibility for correctness should be assigned when HTTP requests are modified in transit. Three primary approaches are possible for Cisco CSS/ACE devices and similar load balancers upon receiving invalid newlines (CR which lack a trailing LF and LFs which lack a preceeding CR): * Requests with invalid newlines could be rejected outright * Invalid newlines could be stripped prior to header interpretation or modification * Invalid newlines could be converted to valid newlines prior to header interpretation or modification Each of these approaches carries its own set of security, speed and compatibility risks which should be weighed carefully. However, given that load balancers are border devices which commonly terminate SSL, some level of validation should be implemented to help protect internal systems. In addition to addressing newline issues, Cisco CSS devices should provide better tools and documentation on securing certificate headers. VSR recommends one or both of the following approaches be adopted: * The CSS and ACE should strip any client certificate headers provided by clients prior to adding new ones. Note that this approach cannot be guaranteed to work if CR/LF ambiguities are not first corrected. Note: Cisco has added a new command to the CSS firmware, currently only available by TAC to address this issue via the following command: ssl pre-remove-http-hdr * The CSS and ACE should require device administrators to specify a random header prefix when configuring client certificates, but provide a mechanism for existing deployments to disable or opt out of a secure by default configuration to support exisitng or legacy applications. One way to accomplish this would be to prevent the client-cert header insertion configuration from taking effect until a device administrator has configured the header prefix using the following command syntax: ssl-server <context> http-header prefix "<random_prefix>" This <random_prefix> parameter should be documented as "password" or "key" and users should be urged to select an unpredictable one, known only by the CSS / ACE administrators and the back-end applications. Recommendations for Web Server Vendors -------------------------------------- Web server vendors should provide options for requiring strict HTTP compliance in HTTP headers. That is, instances of bare CR or bare LF characters should not be accepted as delimiters of HTTP headers. These should either be ignored (and considered a part of the HTTP header content) or requests containing bare CR and/or bare LF characters should be rejected. Eventually, such a configuration option should be enabled by default. It is impossible to predict the number of combinations of HTTP proxies, load balancers and web servers which, in combination, could allow for serious vulnerabilities along the lines of HTTP request smuggling, or in this case, certificate header spoofing. Web server vendors should stop accepting non-compliant requests in order to help head off future attacks. Recommendations for Customers ----------------------------- There are currently no known workarounds for the end-of-line marker parsing vulnerability on the Cisco CSS. On the Cisco ACE customers are encouraged to utilize a layer 7 class map rather than simply using a VIP redirect. In order to mitigate the risk of the ClientCert header insertion that may lead to user impersonation VSR recommends the use of the following CSS/ACE command: ssl-server <context> http-header prefix "<random_prefix>" Where <random_prefix> is a difficult to guess value that is comprised of a combination of 3 of the 4 following character classes: lowercase alphabetic, uppercase alphabetic, numeric and special characters. The value should consist of a minimum of 8 characters, decreasing the likelihood of successful brute-force attacks. VSR strongly recommends disabling the TRACE/TRACK HTTP methods on the target web/application server and periodically changing the random prefix. Other work arounds are possible if application administrators/developers are able to perform additional validation of user identities. One possibility would be to simply disable SSL termination at the Cisco CSS, instead performing certificate validation at the application server level. However, this may not be feasible in many environments. As an alternative, user certificate headers could undergo a second round of validation at the application server level. For instance, randomized user IDs or user-specific secret values could be embedded in user certificate headers which would be passed along by the CSS to applications. Once received, application servers would validate this additional parameter to help prevent header spoofing in a user-specific way. However, while this could prevent blind certificate spoofing, an attacker who could perform man-in-the-middle attacks would likely be able to obtain certificate headers of other users and subsequently bypass this protection on the server side. Versions Affected ----------------- The described end-of-line interpretation behavior exists in all versions of the Cisco CSS and Cisco ACE (Application Control Engine) and has only been partially updated in version 8.20.4.02 and ACE A2(3.0). Cisco has stated that the new behavior to address this issue in the CSS applicance is to look for the terminator of LFLF if the separator that follows the HTTP/1.X is a single LF. However if it is a CRLF pair the CSS will only search for CRLF as end-of-line markers, however it does not currently address situations where mixed end-of-line markers are used. The client supplied header insertion vulnerability continues to affect all versions of the Cisco CSS and ACE (Application Control Engine), however a mitigation exists and is described above. Vendor Response --------------- The following timeline details Cisco's response to the reported issue: 2009-06-05 VSR submitted a security bug report to Cisco PSIRT 2009-06-06 Cisco confirmed receipt of bug report 2009-07-02 Cisco acknowledged the presence of VSR submitted vulnerabilities 2009-08-04 Cisco confirmed release plans for end-of-line marker parsing vulnerability 2009-10-14 Cisco provided update on defect notes and remediation approach 2010-03-11 VSR reviewed the Cisco release notes on potential CSS bug fix and provided Cisco with notice indicating that these fixes are inadequate 2010-03-12 Cisco confirmed receipt of correspondence 2010-04-07 Conference call between VSR and Cisco to discuss security ramifications and understand implementation specifics of Cisco ACE 2010-05-21 VSR performed testing against Cisco ACE 4710 in a lab verifying end-of-line parsing issue in default class map configuration 2010-05-26 Cisco provided guidance on utilizing a layer 7 class map to address issue on Cisco ACE 2010-06-28 VSR performed verification of layer 7 class map in a lab verifying correct device behavior 2010-06-29 Cisco and VSR begin coordinating advisory release 2010-07-01 Cisco was provided a draft advisory 2010-07-02 VSR advisory released Common Vulnerabilities and Exposures (CVE) Information ------------------------------------------------------ The Common Vulnerabilities and Exposures (CVE) project has assigned the numbers CVE-2010-1575 and CVE-2010-1576 to these issues. These are candidates for inclusion in the CVE list (http://cve.mitre.org), which standardizes names for security problems. Acknowledgements ---------------- Thanks to Cisco for response and cooperation, Tim Morgan for assistance in attempting to verify this issue on other competing load-balancers, and Ken Pierce for providing a lab environment to conduct Cisco ACE testing. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -=-=-=- References: 1. Cisco CSS 11500 Series Content Services Switches http://www.cisco.com/en/US/products/hw/contnetw/ps792/ 2. Cisco ACE Application Control Engine Application Switches http://www.cisco.com/en/US/prod/collateral/modules/ps2706/ps6906/AAG_Cis co_ACE_Application_Control_Engine_Application_Switches.pdf 3. RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1 http://tools.ietf.org/html/rfc2616 4. Configuring SSL Termination: Adding a Prefix to the Fields Inserted in the HTTP Header http://www.cisco.com/en/US/docs/app_ntwk_services/data_center_app_servic es/css11500series/v8.10/configuration/ssl/guide/terminat.html#wp1026217 5. HTTP Request Smuggling http://www.cgisecurity.com/lib/HTTP-Request-Smuggling.pdf -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -=-=-=- Copyright 2010 Virtual Security Research, LLC. All rights reserved.