Acquired!

As those of you who know me (and probably a fair few that don't) will already know, PythonAnywhere was acquired by Anaconda, Inc back in June of this year. We're still the same team, and I'm still leading it, but now we're part of a larger company.

It's been quite a ride. Due diligence and negotiation in the months up to the close was just as tough as I'd always been told it would be (and that's despite the fact that according to our lawyers it was a pretty smooth one as these things go). And now I have to get used to having a boss again, which is weird... but is helped by the fact that said boss is a great guy, and is aligned with us (you can tell from the lingo that I work for a larger company now, right?) on keeping the platform up and running as it was, while investing into it so that it can get better and grow faster.

So, all good news :-)

I've been vaguely considering putting together a few blog posts outlining what happens during an acquisition -- just a general discussion of the steps and what they involve. I wouldn't be putting anything in about this particular deal, of course -- there are strict non-disclosures about the terms and so on -- but just a description of what happens might be useful for other people in the position I was in earlier on this year. I had to learn a lot of stuff very quickly, and while our lawyers were awesome and explained things brilliantly, it would have been useful to have some kind of layman's background information.

What do you think -- worth posting?

A somewhat indirect way of reporting stolen cards to the bank

One of the interesting things about having a business that accepts cards on the Internet is seeing what odd things people do when trying to use your site. A case in point is someone we've noticed over the last few months, who appears to be using our site as a rather indirect way to report stolen cards.

The behaviour that we see is that they run some kind of script that signs up for a bunch of accounts, with randomly-generated usernames, and then try to upgrade them all using stolen card numbers.

Naturally, our fraud-prevention systems pick that up pretty much immediately, and we run our own script that identifies every account that they've created, finds the card details used for them, and reports every transaction and attempted transaction as fraudulent. This means that our payment processor, Stripe, can flag the card numbers as stolen, so that they can't be used elsewhere without triggering fraud alerts to the other merchants. And, if a charge actually goes through (most of the cards tend to be pre-paid with no money on them, so most charges fail), then we refund it as fraudulent, which not only notifies Stripe, but I believe notifies the bank that the card number is circulating amongst card fraudsters.

Now, the fact that we do this should be obvious to them. Every time they run their scripts, it causes a minor inconvenience to us (the scripts that we have to handle the problem are getting ever-simpler to use), and it means that every card that they tried on our site is now significantly less valuable as an asset to them. They're essentially paying money for lists of stolen card numbers, and then burning it up.

Given that we're doing this, and they must know that we're doing it, the only explanation I can think of is that they're actually running some kind of strange public service where they buy lists of stolen card details and then get them blocked. It does seem a very roundabout way to do it, though. Surely it would be easier to just tell the banks directly?

But perhaps there's something I'm missing.

Or perhaps they really are dim enough to be using us to check stolen cards for validity, and haven't yet noticed that doing so against a site that reports every fraudulent transaction to the card processor is not a terribly good idea...

Parsing website SSL certificates in Python

A kindly PythonAnywhere user dropped us a line today to point out that StartCom and WoSign's SSL certificates are no longer going to be supported in Chrome, Firefox and Safari. I wanted to email all of our customers who were using certificates provided by those organisations.

We have all of the domains we host stored in a database, and it was surprisingly hard to find out how I could take a PEM-formatted certificate (the normal base-64 encoded stuff surrounded by "BEGIN CERTIFICATE" and "END CERTIFICATE") in a string and find out who issued it.

After much googling, I finally found the right search terms to get to this Stack Overflow post by mhawke, so here's my adaptation of the code:

from OpenSSL import crypto

for domain in domains:
    cert = crypto.load_certificate(crypto.FILETYPE_PEM, domain.cert)
    issuer = cert.get_issuer().CN
    if issuer is None:
        # This happened with a Cloudflare-issued cert
        continue
    if "startcom" in issuer.lower() or "wosign" in issuer.lower():
        # send the user an email

pam-unshare: a PAM module that switches into a PID namespace

Today in my 10% time at PythonAnywhere (we're a bit less lax than Google) I wrote a PAM module that lets you configure a Linux system so that when someone sus, sudos, or sshes in, they are put into a private PID namespace. This means that they can't see anyone else's processes, either via ps or via /proc. It's definitely not production-ready, but any feedback on it would be very welcome.

In this blog post I explain why I wrote it, and how it all works, including some of the pitfalls of using PID namespaces like this and how I worked around them.

[ Read more ]

An HTTP request's journey through a platform-as-a-service

I'm definitely getting better as a public speaker :-) At EuroPython in Berlin last month, I gave a high-level introduction to PythonAnywhere's load-balancing system. There's a video up on PyVideo: An HTTP request's journey through a platform-as-a-service. And here are the slides [PDF].

A fun bug

While I'm plugging the memory leaks in my epoll-based C reverse proxy, I thought I might share an interesting bug we found today on PythonAnywhere. The following is the bug report I posted to our forums.

So, here's what was happening.

Each web app someone has on PythonAnywhere runs on a backend server. We have a cluster of these backends, and the cluster is behind a loadbalancer. Every backend server in the cluster is capable of running any web app; the loadbalancer's job is to spread things out between them so that each one at any given time is only running an appropriately-sized subset of them. It has a list of backends, which we can update in realtime as we add or remove backends to scale up or down, and it looks at incoming requests and uses the domain name to work out which backend to route a request to.

That's all pretty simple. The twist comes when we add the code that reload web apps to the mix.

Reloading a PythonAnywhere web app is simply a case of making an authenticated request to a specific URL. For example, right now (and this might change, it's not an official API, so don't do anything that relies on it) to reload www.foo.com owned by user fred, you'd hit the URL http://www.pythonanywhere.com/user/fred/webapps/www.foo.com/reload

Now, the PythonAnywhere website itself is just another web app running on one of the backends (a bit recursive, I know). So most requests to it are routed based on the normal loadbalancing algorithm. But calls specifically to that "reload" URL need to be routed differently -- they need to go to the specific backend that is running the site that needs to be reloaded. So, for that URL, and that URL only, the loadbalancer uses the domain name that's specified second-to-the-end in the path bit of the URL to choose which backend to route the request to, instead of using the hostname at the start of the URL.

So, what happened here? Well, the clue was in the usernames of the people who were affected by the problem -- IronHand and JoeButy. Both of you have mixed-case usernames. And your web apps are ironhand.pythonanywhere.com and joebuty.pythonanywhere.com.

But the code on the "Web" tab that specifies the URL for reloading the selected domain specifies it using your mixed-case usernames -- that is, it specifies that the reload calls should go to the URL for IronHand.pythonanywhere.com or JoeButy.pythonanywhere.com.

And you can probably guess what the problem was -- the backend selection code was case-sensitive. So requests to your web apps were going to one backend, but reload messages were going to another different backend. The fix I just pushed made the backend selection code case-insensitive, as it should have been.

The remaining question -- why did this suddenly crop up today? My best guess is that it's been there for a while, but it was significantly less likely to happen, and so it was written off as a glitch when it happened in the past.

The reason it's become more common is that we actually more than doubled the number of backends yesterday. Because of the way the backend selection code works, when there's a relatively small number of backends it's actually quite likely that the lower-case version of your domain will, by chance, route to the same backend as the mixed-case one. But the doubling of the number of servers changed that, and suddenly the probability that they'd route differently went up drastically.

Why did we double the number of servers? Previously, backends were m1.xlarge AWS instances. We decided that it would be better to have a larger number of smaller backends, so that problems on one server impacted a smaller number of people. So we changed our system to use m1.large instances instead, span up slightly more than twice as many backend servers, and switched the loadbalancer across.

So, there you have it. I hope it was as interesting to read about as it was to figure out :-)

SNI-based reverse proxying with Go(lang)

Short version for readers who know all about this kind of stuff: we build a simple reverse-proxy server in Go that load-balances HTTP requests using the Hosts header and HTTPS using the SNIs from the client handshake. Backends are selected per-host from sets stored in a redis database. It works pretty well but we won't be using it because it can't send the originating client IP to the backends when it's handling HTTPS. Code here.

We've been looking at options to load-balance our user's web applications at PythonAnywhere; this post is about something we considered but eventually abandoned; I'm posting it because the code might turn out to be useful to other people.

A bit of background first; if you already know what a reverse proxy is and how load-balancing and virtual hosting work, you can skip forward a bit.

Imagine an old-fashioned shared hosting environment. You're able to run a web application on a machine that's being used by lots of other people, and you're given that machine's IP address. You set up your DNS configuration so that your domain points to that IP address, and it all works. When a connection comes in from a browser to access your site, the web server on the machine needs to work out which person's web app it should route it to. It does this by looking at the HTTP request and finding a Host header in it. So, by using the Host header, the shared hosting provider can keep costs down by sharing an IP address and a machine between multiple clients. This is called virtual hosting.

Now consider the opposite case -- a high-traffic website, where one machine isn't enough to handle all of the traffic. Processing a request for a page on a website can take a certain amount of machine resources -- database lookups, generating dynamic pages from templates, and so on. So a single web server might not be enough to cope with lots of traffic. In this case, people use what's called a reverse proxy, or load-balancer. In the simplest case, this is just a machine running on a single IP. When a request comes in, it selects a backend -- that is, one of a number of web servers, each of which is running the full website's code. It then just sends the request down to one of them, and copies all data that comes back from that backend up to the browser that made the request. Because just copying data around from backend to browser and vice versa is much easier work than processing the actual request, a single load-balancer can handle many more requests than any of the backend web servers could, and if it's configured to select backends appropriately it can spread the load smoothly across them. Additionally, this kind of setup can handle outages gracefully -- if one backend stops responding, it can stop routing to it and use the others as backups.

Now let's combine those two ideas. Imagine a platform-as-a-service, where each outward-facing IP might be responsible for handling large numbers of websites. But for reliability and performance, it might make sense to have each website backed by multiple backends. So, for example, a PaaS might have a thousand websites backed by one hundred different webservers, where website one is handled by backends one, two and three, website two by backends two, three and four, and so on. This means that the PaaS can keep costs down (running ten web apps per backend server) and reliability and performance up (each website having three independent backends).

So, that's the basics. There are a number of great tools which can be used to operate as super-efficient proxies that can handle this kind of many-hostnames-to-many-backends mapping. nginx is the most popular, but there are also haproxy and hipache. We are planning to choose one of these for PythonAnywhere (more about that later), but we did identify one slight problem with all of them. The code I'm shortly going to show was our attempt at working around that problem.

The description above of how virtual hosting works is fine when we're talking about HTTP. But increasingly, people want to use HTTPS for secure connections.

When an HTTPS connection comes in, the server has a problem. Before it can decode what's in the request and get the Host header, it needs to establish a secure link. Its first step to establish that link is to send a certificate to the client to prove it is who it says it is. But each of the different virtual hosts on the machine will need a different certificate, because they're all on different domains. So there's a chicken-and-egg problem; it needs to know which host it is meant to be in order to send the right certificate, but it needs to have sent the certificate in order to establish a secure connection to find out which host it is meant to be. This was a serious problem until relatively recently; basically, it meant that every HTTPS-secured site had to have its own dedicated IP address, so that the server could tell which certificate to serve when a client connected by looking at the IP address the connection came in on.

This problem was solved by an extension to the TLS protocol (TLS being the latest protocol to underly HTTPS) called "Server Name Indication". Basically, it takes the idea of the HTTP Host header and moves it down the stack a bit. The initial handshake message that a client connecting to a server used to just say "here I am and here's the kind of SSL protocol I can handle -- now what's your certificate?" With SNI the handshake also says "here's the hostname I expect you to have"

So with SNI, a browser connects to a server, and the server looks at the handshake to find out which certificate to use. The browser and server establish a secure link and then the browser sends the normal HTTP request, which has a Host header, which it then uses to send the request to the appropriate web app.

Let's get back to the proxy server that's handling incoming requests for lots of different websites and routing them to lots of different backends. With all of the proxies mentioned above -- nginx, hipache and haproxy -- a browser makes a connection, the proxy does all of the SNI stuff to pick the right certificate, it decodes the data from the client, works out which backend to send it to using the Host header in the decoded data, and then forwards everything on.

There's an obvious inefficiency here. The proxy shouldn't have to decode the secure connection to get the Host header -- after all, it already knows that from the information in the SNI. And it gets worse. Decoding the secure connection uses up CPU cycles on the proxy. And either the connection between the proxy and the backends is non-secure, which could be an issue if a hacker got onto the network, or it's secure, in which case the proxy is decoding and then encoding everything that goes through it -- even more CPU load. Finally, all of the certificates for every site that the proxy's handling -- and their associated private keys -- have to be available to the proxy. Which is another security risk if it gets hacked.

So, probably like many people before us, we thought "why not just route HTTPS based on the SNI? It can't be that hard!" And actually, it isn't. Here's a GitHub project with a simple Go application that routes HTTP requests using the hosts header, and HTTPS using the SNI. It never needs to know anything about the certificates for the sites it's proxying for, and all data is passed through without any decryption.

So why didn't we decide to use it? Access logs and spam filters. The thing is, people who are running websites like to know who's been looking at their stuff -- for their website metrics, for filtering out spammy people using tools like Akismet, and so on. If you're using a proxy, then the backend sees every request as coming from the proxy's IP, which isn't all that useful. So normally a proxy will add an extra header to HTTP requests it passes through -- X-Forwarded-For is the usual one.

And the problem with an SNI proxy is the same as its biggest advantage. Because it's not decoding the secure stream from the browser, it can't change it, so it can't insert any extra headers. So all HTTPS requests going over any kind of SNI-based reverse proxy will appear to come from the proxy itself. Which breaks things.

So we're not going to use this. And TBH it's not really production-level code -- it was a spike and is also the first Go code I've ever written, so it's probably full of warts (comments very much welcomed!). Luckily we realised the problem with the backends not knowing about the client's IP before we started work on rewriting it test-first.

On the other hand, it might be interesting for anyone who wants to do stuff like this. The interesting stuff is mostly in handleHTTPSConnection, which decodes the TLS handshake sent by the client to extract the SNI.

I did a bit of very non-scientific testing just to make sure it all works. I started three backends servers with simple Flask apps that did a sleep on every request to simulate processing:

from flask import Flask
import time
from socket import gethostname



app = Flask(__name__)



@app.route("/")
def index():
    time.sleep(0.05)
    return "Hello from " + gethostname()



if __name__ == "__main__":
    app.run("0.0.0.0", 80, processes=4)

Then ran the Apache ab tool to see what the performance characteristics were for one of them:

root@abclient:~# ab -n1000 -c100 http://198.199.83.71/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/



Benchmarking 198.199.83.71 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests






Server Software:        Werkzeug/0.9.2
Server Hostname:        198.199.83.71
Server Port:            80



Document Path:          /
Document Length:        19 bytes



Concurrency Level:      100
Time taken for tests:   21.229 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      172000 bytes
HTML transferred:       19000 bytes
Requests per second:    47.10 [#/sec] (mean)
Time per request:       2122.938 [ms] (mean)
Time per request:       21.229 [ms] (mean, across all concurrent requests)
Transfer rate:          7.91 [Kbytes/sec] received



Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3   7.4      0      37
Processing:    73 2025 368.7   2129    2387
Waiting:       73 2023 368.4   2128    2386
Total:        103 2028 363.7   2133    2387



Percentage of the requests served within a certain time (ms)
  50%   2133
  66%   2202
  75%   2232
  80%   2244
  90%   2286
  95%   2317
  98%   2344
  99%   2361
 100%   2387 (longest request)
root@abclient:~# 

Then, after adding records to the proxy's redis instance to tell it to route requests with the hostname proxy to any of the backends, and hacking the hosts file on the ab client machine to make the hostname proxy point to it:

root@abclient:~# ab -n1000 -c100 http://proxy/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/



Benchmarking proxy (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests






Server Software:        Werkzeug/0.9.2
Server Hostname:        proxy
Server Port:            80



Document Path:          /
Document Length:        19 bytes



Concurrency Level:      100
Time taken for tests:   7.668 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      172000 bytes
HTML transferred:       19000 bytes
Requests per second:    130.41 [#/sec] (mean)
Time per request:       766.803 [ms] (mean)
Time per request:       7.668 [ms] (mean, across all concurrent requests)
Transfer rate:          21.91 [Kbytes/sec] received



Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.7      0       9
Processing:    93  695 275.4    617    1228
Waiting:       93  693 275.4    614    1227
Total:         99  696 274.9    618    1228



Percentage of the requests served within a certain time (ms)
  50%    618
  66%    799
  75%    948
  80%    995
  90%   1116
  95%   1162
  98%   1185
  99%   1204
 100%   1228 (longest request)
root@abclient:~#

So, it works. I've not done ab testing with the HTTPS side of things, but I have hacked my own hosts file and spent a day accessing Google and PythonAnywhere itself via the proxy. It works :-)

As to what we're actually going to use for load-balancing PythonAnywhere:

• nginx is great but stores its routing config in files, which doesn't easily scale to large numbers of hosts/backends. It's doable, but it's just a nightmare to manage, especially if things go wrong.
• haproxy is the same -- worse, it needs to be fully restarted (interrupting ongoing connections) if you change the config.
• hipache stores data in redis (which is what inspired me to do something similar for this proxy) so it can gracefully handle rapidly-changing rounting setups. But it's written in Node.js, so while it's pretty damn fast, it's not as fast as nginx.

But... as the dotcloud people who wrote hipache recently pointed out (bottom of the post), nginx's built-in lua scripting support is now at a level where you can store your routing config in redis -- so with a bit of work, you can get the speed of nginx with the ease of configuration of hipache. So that's where we're heading. We'll just have to make sure the proxy and its certificates are super-secure, and live with the extra CPU load.

How many Python programmers are there in the world?

We've been talking to some people recently who really wanted to know what the potential market size was for PythonAnywhere, our Python Platform-as-a-Service and cloud-based IDE.

There are a bunch of different ways to look at that, but the most obvious starting point is, "how many people are coding Python?" This blog post is an attempt to get some kind of order-of-magnitude number for that.

First things first: Wikipedia has an estimate of 10 million Java developers (though I couldn't find the numbers to back that up on the cited pages) but nothing for Python -- or, indeed, any of the other languages I checked. So nothing there.

A bit of Googling around gets one interesting hit; in this Stack Overflow answer, "Tall Jeff" says that the 2007 version of Learning Python estimated that there were 1 million Python programmers in the world. Using Amazon's "Look inside" feature on the current edition, they still have the same number but for the present day, but let's assume that they were right originally and the number has grown since then. Now, according to the Python wiki, there were 586 people at the 2007 PyCon. According to the front page at PyCon.org, there were 2,500 people at PyCon 2013. So if we take that as a proxy for the growth of the language, we get one guess of the number of Python developers: 4.3 million.

Let's try another metric. Python.org's web statistics are public. Looking at the first five months of this year, and adding up the total downloads, we get:

Jan:	2,584,754
Feb:	2,539,177
Mar:	3,182,946
Apr:	3,199,012
May:	2,855,033

Averaging that over a year gives us 34,466,213 downloads per year. It's worth noting that these are overwhelmingly Windows downloads -- most Linux users are going to be using the versions packaged as part of their distro, and (I think, but correct me if I'm wrong) the same is largely going to be the case on the Mac.

So, 34.5 million downloads. There were ten versions of Python released over the last year, so for let's assume that each developer downloaded each version once and once only; that gives us 3.5 million Python programmers on Windows.

What other data points are there? This job site aggregator's blog post suggests using searches for resumes/CVs as a way of getting numbers. Their suggested search for Python would be

(intitle:resume OR inurl:resume) Python -intitle:jobs -resumes -apply

Being in the UK, where we use "CV" more than we use "resume", I tried this:

(intitle:resume OR inurl:resume OR intitle:cv OR inurl:cv) Python -intitle:jobs -resumes -apply

The results were unfortunately completely useless. 338,000 hits but the only actual CV/resume on the first page was Guido van Rossum's -- everything else was about the OpenCV computer vision library, or about resuming things.

So let's scrap that. What else can we do? Well, taking inspiration (and some raw data) from this excellent blog post about estimating the number of Java programmers in the world, we can do this calculation:

• Programmers in the world: 43,000,000 (see the link above for the calculation)
• Python developers as per the latest TIOBE ranking: 4.183%, which gives 1,798,690
• Python developers as per the latest LangPop.com ranking: 7% (taken by an approximate ratio of the Python score to the sum of the scores of all languages), which gives 2,841,410

OK, so there I'm multiplying one very approximate number of programmers by a "percentage" rating that doesn't claim to be a percentage of programmers using a given language. But this ain't rocket science, I can mix and match units if I want.

The good news is, we're in the same order of magnitude; we've got numbers of 1.8 million, 2.8 million, 3.5 million, and 4.3 million. So, based on some super-unscientific guesswork, I think I can happily say that the number of Python programmers in the world is in the low millions.

What do you think? Are there other ways of working this out that I've missed? Does anyone have (gasp!) hard numbers?

A super-simple chat app with AngularJS, SockJS and node.js

We're planning to move to a more advanced JavaScript library at PythonAnywhere. jQuery has been good for us, but we're rapidly reaching a stage where it's just not enough.

There are a whole bunch of JavaScript MVC frameworks out there that look tempting -- see TodoMVC for an implementation of a simple app in a bunch of them. We're asking the people we know and trust which ones are best, but in the meantime I had a look at AngularJS and knocked up a quick chat app to see how easy it would be. The answer was "very".

Here's the client-side code:

<html ng-app>
<head>
<script src="http://cdn.sockjs.org/sockjs-0.3.min.js"></script>
<script src="http://ajax.googleapis.com/ajax/libs/angularjs/1.0.4/angular.min.js"></script>



<script>
    var sock = new SockJS('http://192.168.0.74:9999/chat');
    function ChatCtrl($scope) {
        $scope.messages = [];
        $scope.sendMessage = function() {
            sock.send($scope.messageText);
            $scope.messageText = "";
        };



        sock.onmessage = function(e) {
            $scope.messages.push(e.data);
            $scope.$apply();
        };
    }
</script>



</head>



<body>



<div ng-controller="ChatCtrl">
    <ul>
        <li ng-repeat="message in messages">{{message}}</li>
    </ul>



    <form ng-submit="sendMessage()">
        <input type="text" ng-model="messageText" placeholder="Type your message here" />
        <input type="submit" value="Send" />
    </form
</div>



</body>
</html>

Then on the server side I wrote this server (in node.js because I've moved to Shoreditch and have ironic facial hair it was easy to copy, paste and hack from the SockJS docs -- I'd use Tornado if this was on PythonAnywhere):

var http = require('http');
var sockjs = require('sockjs');



var connections = [];



var chat = sockjs.createServer();
chat.on('connection', function(conn) {
    connections.push(conn);
    var number = connections.length;
    conn.write("Welcome, User " + number);
    conn.on('data', function(message) {
        for (var ii=0; ii < connections.length; ii++) {
            connections[ii].write("User " + number + " says: " + message);
        }
    });
    conn.on('close', function() {
        for (var ii=0; ii < connections.length; ii++) {
            connections[ii].write("User " + number + " has disconnected");
        }
    });
});



var server = http.createServer();
chat.installHandlers(server, {prefix:'/chat'});
server.listen(9999, '0.0.0.0');

And that's it! It basically does everything you need from a simple chat app. Definitely quite impressed with AngularJS. I'll try it in some of the other frameworks we evaluate and post more here.

Reverse proxying HTTP and WebSockets with virtual hosts using nginx and tcp_proxy_module

I spent today trying to work out how we could get PythonAnywhere to support WebSockets in our users' web applications. This is a brief summary of what I found, I'll put it in a proper post on the PythonAnywhere blog sometime soon...

We use nginx, and it can happily route HTTP requests through to uwsgi applications (which is the way we use it) and can even more happily route them through to other socket-based servers running on specific ports (which we don't use but will in the future so that we can support Twisted, Tornado, and so on -- once we've got network namespacing sorted).

But by default, nginx does not support reverse proxying WebSockets requests. There are various solutions to this posted around the net, but they don't explain how to get it working with virtual hosts. I think that this is because they're all a bit old, because it's actually quite easy once you know how.

(It's worth mentioning that there are lots of cool non-nginx solutions using excellent stuff like haproxy and hipache. I'd really like to upgrade our infrastructure to use one of those two. But not now, we all too recently moved from Apache to nginx and I'm scared of big infrastructure changes in the short term. Lots of small ones, that's the way forward...)

Anyway, let's cut to the chase. This excellent blog post by Johnathan Leppert explains how to configure nginx to do TCP proxying. TCP proxying is enough to get WebSockets working if you don't care about virtual hosts -- but because arbitrary TCP connections don't necessarily have a Host: header, it can't work if you do care about them.

However, since the post was written, the nginx plugin module Johnathan uses has been improved so that it now supports WebSocket proxying with virtual hosts.

To get nginx to successfully reverse-proxy WebSockets with virtual host support, compile Nginx with tcp_proxy_module as per Johnathan's instructions (I've bumped the version to the latest stable as of today):

export NGINX_VERSION=1.2.4
curl -O http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz
git clone https://github.com/yaoweibin/nginx_tcp_proxy_module.git
tar -xvzf nginx-$NGINX_VERSION.tar.gz
cd nginx-$NGINX_VERSION
patch -p1 < ../nginx_tcp_proxy_module/tcp.patch
./configure --add-module=../nginx_tcp_proxy_module/
sudo make && make install

Then, to use the new WebSockets support in tcp_proxy_module, put something like this in your nginx config:

worker_processes  1;



events {
    worker_connections  1024;
}



tcp {
    upstream site1 {
        server 127.0.0.1:1001;



        check interval=3000 rise=2 fall=5 timeout=1000;
    }



    server {
        listen 0.0.0.0:80;
        server_name site1.com;



        tcp_nodelay on;
        websocket_pass site1;
    }
}






tcp {
    upstream site2 {
        server 127.0.0.1:1002;



        check interval=3000 rise=2 fall=5 timeout=1000;
    }



    server {
        listen 0.0.0.0:80;
        server_name site2.com;



        tcp_nodelay on;
        websocket_pass site2;
    }
}

Hopefully that's enough to help a few people googling around for help like I was this morning. Leave a comment if you have any questions!