I've identified a couple of security flaws affecting the Varnish reverse proxy which may allow privilege escalation. These issues were reported by email to the vendor but he feels that it is a configurational issue rather than a design flaw. Whilst I can partially see his point in that the administrative interface can be disabled, I'm not convinced that making a C compiler available over a network interface without authentication is sound practice, especially when the resultant compiled code can be made to run as root rather trivially. Tim -- Tim Brown <mailto:timb (at) nth-dimension.org (dot) uk [email concealed]> <http://www.nth-dimension.org.uk/> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Nth Dimension Security Advisory (NDSA20090908) Date: 26th September 2009 Author: Tim Brown <mailto:timb (at) nth-dimension.org (dot) uk [email concealed]> URL: <http://www.nth-dimension.org.uk/> / <http://www.machine.org.uk/> Product: Varnish 2.0.4 <http://www.varnish-cache.org/> Vendor: Redpill Linpro <http://www.redpill-linpro.com/> Risk: High Summary The Varnish reverse proxy server is vulnerable to unauthenticated code execution as the root user. Depending on the exact circumstances this can lead to either privilege escalation or a full remote compromise. After discussions with the vendor, this vulnerability was reported to the Debian security team. The Debian security team assigned CVE-2009-2936 to these vulnerabilities. Further investigations indicated that the versions of Varnish to be found in both the Fedora and Ubuntu GNU/Linux distributions were also affected. Technical Details The Varnish reverse proxy server is vulnerable to unauthenticated remote code execution as the root user due to a design flaw. Varnish in its default state consists of two processes; one master process which runs as root (allows configuration) and one which runs as nobody (serves proxy requests). The master process has a listening socket which handles requests. It was identified that it was possible to ask the master process to read arbitrary VCL config files using the vcl.load directive and if it couldn't parse them then it returned the first line in the error. This can be used to get the obfuscated root password from /etc/shadow: $ nc localhost 6082 200 154 - ----------------------------- Varnish HTTP accelerator CLI. - ----------------------------- Type 'help' for command list. Type 'quit' to close CLI session. vcl.load shadow /etc/shadow 106 173 Message from VCC-compiler: Syntax error at (input Line 1 Pos 5) root:!:14623:0:99999:7::: - ----#-------------------- Running VCC-compiler failed, exit 1VCL compilation failed More seriously, the master process also allows the values of the initial arguments to be reconfigured, including the user under which the proxy process runs. It is possible to set this to root and restart the proxy process: $ nc localhost 6082 200 154 - ----------------------------- Varnish HTTP accelerator CLI. - ----------------------------- Type 'help' for command list. Type 'quit' to close CLI session. param.show user 200 441 user nobody (65534) ... param.set user root 200 47 Change will take effect when child is restarted stop 200 0 start 200 0 param.show user 200 441 user root (0) ... Varnish allows VCL config files to include arbitrary C calls by wrapping them within C{ and }C tags in the config. Varnish then compiles this C on the fly to be used to process proxy requests. Using this, is is possible to either ask the Varnish master process to read a malicious VCL config file or the new configuration can be injected directly over the TCP connection to the master process using the vcl.inline directive. This code is now executed by the now root proxy process. Whilst it is acknowledged that the inclusion of this functionality is entirely intentional and that the vendor does not believe it to be a problem, the default behaviour of the server could be exploited particularly where the user is unaware of the servers existance. In the case of both Debian and Ubuntu, this meant that the server was likely to be started by the init subsystem. Note that whilst on Debian and Fedora, the master process is only bound to localhost, in other cases, it may be possible for an attacker to connect directly to the master process over the network and recreate the attack outlined. Where the attacker can not route directly to the listening port on the master process there is also a cross-site request forgery based attack (similar to the vulnerability on the Tor control port[1] some years back) since the master process is very tolerant of malformed requests. If a attacker can cause a legitimate admin to make a POST request on their behalf then they may be able to inject commands in that way. Solutions In order to completely protect against the vulnerability (in the short term), Nth Dimension recommend turning off the server and replacing it with another reverse proxy such as Squid. Should this not be possible, Nth Dimension would strongly recommend that users confirm that the master process is not listening on an external network interface. In the latter case, users should confirm that only trusted users have SSH access to the system. History Following vendor notification on the 3rd September 2009, Nth Dimension were directed to a configuration setting[2] which it was claimed would require the user to authenticate before making any configuration changes. Nth Dimension attempted to clarify with the vendor as to why Varnish was shipped in an insecure state where a root owned process listened on a TCP port without authentication and could be made to load and execute arbitrary C. The vendors response was essentially that they believed satisfactory warnings were in place and that any vulnerability was configurational rather than a design flaw. Nth Dimension contacted the Debian project on the 4th and after notifying them of the initial vulnerability, continued researching. Whilst doing this, Nth Dimension were able to confirm that authentication functionality did exist, however the changeset[3] in which it was added had not been ported to the latest stable release 2.0.4 and was only available in trunk. Additionally, whilst the vendor was correct that the master process socket needed to be socket explicitly enabled, the vendors own packaging for Redhat (from which other distro packages are derived) has a sysconfig script[4] that switches it on. On 13th September 2009, the Debian package maintainer proposed patches to Nth Dimension that disable the support for arbitrary C (within .vcl configuration files only). They also noted that a new feature would likely be introduced in the next stable upstream release which would allow the feature to be enabled and disabled at run time by passing the appropriate flags to the Varnish process when starting it. Current As of 25th March 2010, the state of these vulnerabilities is believed to be as follows: | | 2.0.6 | 2.1.0 | | Unauthenticated access to master process | | Fixed | | Master process accepts malformed XSRF request | | | | Arbitrary file read via master process | | | | Master process can reconfigure proxy as root | | | | Inline C can be injected via file | | | | Inline C can be injected via vcl.inline | | | 2.1.0 has entered Debian unstable and a patch to the packaging has been prepared for 1.1.2 in Debian stable which disables the listening TCP socket on the master process. Disabling or password protecting the listening TCP socket of the master process satisfactorily mitigates the issues outlined above. Nth Dimension are not aware of fixes for Fedora or any distribution. Vendor Response The vendor asked for the following to be included verbatim within the published advisory: This "security-advisory" is mostly bunk and pointless speculation, and may not even have anything to do with Varnish. Let me explain why that is: Varnish is a tool for professionals, and as such it contains very powerful bits and sharp edges because it is designed to move data at wire-speed, no matter how much money you spent on your hardware. If it helps the reader get into the right frame of mind, think of Varnish as "Husqvarna Extreme Logging Chainsaw" as opposed to "Toys'R'Us Plastic Saw For Babies". It is the Varnish users responsibility to aim and apply these powerful bits and sharp edges wisely: Towards purposeful application and away from harm. If that is not possible, the user should be intelligent enough to not enable these features in the first place. In accordance with my personal version of the Principle Of Least Astonishment, these features, in particular the CLI, are disabled by default in Varnish: It takes explicit command line arguments to enable them. If the people who distribute Varnish as packages decide otherwise, and enable the CLI by default, they supposedly to know what they are doing, and presumably also why. Furthermore, Varnish is written to not need root priviledge to run. The typical reason to run Varnish as root anyway, is that the OS/kernel requires this priviledge to bind to TCP port 80. All qualified observers find this requirement a serious security flaw in the POSIX standard. (You have no idea how much I wish I could fix that.) The long litany of speculation, in this so-called "security advisory", about what you can do with the Varnish CLI in various circumstances, if it is enabled, is therefore fundamentally misguided and pointless, in the same way an enumerated list of which pieces of household furniture a Husqvarna chainsaw can possibly cut through, once you start it, would be. If you start Varnish as root, and tell it to format your disks through the CLI or in-line C-code, it will, by design, faithfully attempt to follow your instructions and format your disk. If you do not want Varnish to format your disk, then you should not ask it to. - From this it is immediately obvious to any casual observer, that if you do not protect access to the Varnish CLI, or send unwise commands to it, you will be screwed. If this was news or a surprise to you, you should step away from the keyboard and start growing cabbage instead. The only reason I do not declare this so-called "security advisory" totally bunk, is that it may contain one tiny fig-leaf of substance: If somebody, who distributes software in package format, unwisely selects a non-POLA-compliant configuration as default, the users of that package may get screwed. This is truth in the absolute sense, not just for Varnish, but for all possible software. To the extent this "security advisory" document such an inconsistent application of POLA, relative to some specific distribution channels packaging of Varnish, it would have a valid point. But I don't see that point being made, much less clearly documented, anywhere in the text. The closest we get is the sentence: "Note that whilst on Debian and Fedora, the master process is only bound to localhost," But no attempt is made to compare this fact to the respective and expected POLA guidelines for these distributions ? Instead, the above factual observation is immediately followed by totally vacuous speculation: "in other cases, it may be possible for an attacker to connect directly to the master process over the network [...]" Which is as informative and helpful as the statement: "Chain-saws could conceiveably hurt people". And thus, in conclusion, I can't help but wonder, how much better off we would all be, if "security researchers" tried to design security into software, rather than to waste time, theirs and ours, on pointless security-advisory trophy hunts. Poul-Henning Kamp, aka: phk (at) FreeBSD (dot) org [email concealed], aka: Author of Varnish, And not at all amused. Thanks Whilst their intervention was not successful, Nth Dimension would like to thank Joey Schulze and Moritz Muehlenhoff of the Debian security team and the package maintainer Stig Sandbeck Mathisen for their attempts to mediate a satisfactory solution. Nth Dimension would also like to thank various other unnamed individuals with whom the vulnerability was discussed during the research phase. [1] http://www.securityfocus.com/bid/25188/discuss [2] http://www.varnish-cache.org/wiki/CLI [3] http://www.varnish-cache.org/changeset/3865 [4] http://www.varnish-cache.org/browser/trunk/varnish-cache/redhat/varnish. sysconfig -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBCAAGBQJLsFckAAoJEPJhpTVyySo7+pUP/3010oniD5fna8vxUsOUEetf dksx2r20cVqbF8mLCXsMe4Uo+oqomUkDhPCp1ofvJ4qbQ3io9x22ckzz6OA6Ai8o QQV4MBXSBA7ikLUH1SCRER7NVQ48Y0a4jJxiMApwFSlULxwEVhH1mX5v22LhsHaf HCTZ7xn5rt5YaugSfTpjavW+AKEkIaoPCj7amvZNCEg4CVq3M+qrx2GUPtmmFCH8 8T9r9x78es9DMGzXad5N8Y1z6JfLaqJqx1zy3XoQ1GvN16a7q/0PkEzg1b/c0oZE Y6csWY2Db3zrCYcLL/Cv59nNcIog9Jtt9rvKeM5v4+NynNqVcZ1XTCsiZfAAGGy2 ivgbgjdL8kQSlQ0ZRdLoBBj4sLdwGN9l1s4q1AWZLpZfws9JaP19SyPn3YUOsunf DzaE85Pgx8gDbPP6pu9jrmSxR9DhexskJ6LHfDjeKD1CRJBxmUK7285dBVlgDSOA TgSr7penbUTzE64O3IlfjdE+2TzUd+8z1wqWCVVi9/rsN1P50SdtYGYCvhoqXCEW ai3kzawjBvfMWV7OoNSqKDNFVLkI2Q4pzp6uQfgdOuFzSAHrt7+1eO3/LNM2lF07 aVbQ2cd4V5WDe8yH+4RABY8+B/J9QMa4H44HM0J5kTOa13ZvWXln1KuD8CiB5UYf uKFUKQTvO69HTP+lufC1 =n6jW -----END PGP SIGNATURE----- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAABCAAGBQJLsF/VAAoJEPJhpTVyySo72IkP/2Wa2SlrrGqGYZoc7zFRbjjR IwT0HMEcuAJMWRToLlWNi5rADxLigDmo8fuTP/y3UQOQ6de9JLnPcK9xEOfO+rqq FJ3a5h/zGs5A5pIOtG+faU3DUNFj9FbOpmbdFA08PVC6PgUjFMHhABTKrxeQ4Tcu hblcfYxkTEvfsaocYtsdno201cAyJ6QuF2cFE14rCI24xZKzWhO5sz5PLRhm9193 tr3znxd9URmhFVwLL+1fkHU21vE5BYoSk8qwq6zexG5ftcmIzosDR6qfsuYI3uyX pEKhM0s5OiDWndsBZHUqWD2LaCdR1/e+uWJ88gQ6Xy0eChWX1HdWN6WWuLcyJt0/ WVye5UTYhmZJBwBbPIpjWBGOUZanShUGUTUA/ir3i1tgAJWJ+ssS0HsHwLvg3aXb OdVgVpjoyFAxsK96x+Ie/tPScG6379UYxHATLX9U+28zW5xZWOGaoWLO0h2EABK1 PrLnf1oKg9AP4enJ0nTh566QtSJ1q1cHPMIq5M46+5q4V594tUNd8lupJ6iVMz8d xpV3Ivi2KvvI+LioHlAFypLCvqOeVY6iaPboFzTqZ3tTtCgHAPJAtJYYCl3orbal eTqAOONWI2A7tpvWXt+/XS5tVZHIOtPdLmhUIs8U4NmI1uxrRk2/r8fCGMeR+VLq v8osapcZHjNZDhpVDq2D =BaSu -----END PGP SIGNATURE-----