How many Python programmers are there in the world?

Posted on 24 June 2013 in Programming, Python, PythonAnywhere

We've been talking to some people recently who really wanted to know what the potential market size was for PythonAnywhere, our Python Platform-as-a-Service and cloud-based IDE.

There are a bunch of different ways to look at that, but the most obvious starting point is, "how many people are coding Python?" This blog post is an attempt to get some kind of order-of-magnitude number for that.

First things first: Wikipedia has an estimate of 10 million Java developers (though I couldn't find the numbers to back that up on the cited pages) but nothing for Python -- or, indeed, any of the other languages I checked. So nothing there.

A bit of Googling around gets one interesting hit; in this Stack Overflow answer, "Tall Jeff" says that the 2007 version of Learning Python estimated that there were 1 million Python programmers in the world. Using Amazon's "Look inside" feature on the current edition, they still have the same number but for the present day, but let's assume that they were right originally and the number has grown since then. Now, according to the Python wiki, there were 586 people at the 2007 PyCon. According to the front page at PyCon.org, there were 2,500 people at PyCon 2013. So if we take that as a proxy for the growth of the language, we get one guess of the number of Python developers: 4.3 million.

Let's try another metric. Python.org's web statistics are public. Looking at the first five months of this year, and adding up the total downloads, we get:

Jan:2,584,754
Feb:2,539,177
Mar:3,182,946
Apr:3,199,012
May:2,855,033

Averaging that over a year gives us 34,466,213 downloads per year. It's worth noting that these are overwhelmingly Windows downloads -- most Linux users are going to be using the versions packaged as part of their distro, and (I think, but correct me if I'm wrong) the same is largely going to be the case on the Mac.

So, 34.5 million downloads. There were ten versions of Python released over the last year, so for let's assume that each developer downloaded each version once and once only; that gives us 3.5 million Python programmers on Windows.

What other data points are there? This job site aggregator's blog post suggests using searches for resumes/CVs as a way of getting numbers. Their suggested search for Python would be

(intitle:resume OR inurl:resume) Python -intitle:jobs -resumes -apply

Being in the UK, where we use "CV" more than we use "resume", I tried this:

(intitle:resume OR inurl:resume OR intitle:cv OR inurl:cv) Python -intitle:jobs -resumes -apply

The results were unfortunately completely useless. 338,000 hits but the only actual CV/resume on the first page was Guido van Rossum's -- everything else was about the OpenCV computer vision library, or about resuming things.

So let's scrap that. What else can we do? Well, taking inspiration (and some raw data) from this excellent blog post about estimating the number of Java programmers in the world, we can do this calculation:

  • Programmers in the world: 43,000,000 (see the link above for the calculation)
  • Python developers as per the latest TIOBE ranking: 4.183%, which gives 1,798,690
  • Python developers as per the latest LangPop.com ranking: 7% (taken by an approximate ratio of the Python score to the sum of the scores of all languages), which gives 2,841,410

OK, so there I'm multiplying one very approximate number of programmers by a "percentage" rating that doesn't claim to be a percentage of programmers using a given language. But this ain't rocket science, I can mix and match units if I want.

The good news is, we're in the same order of magnitude; we've got numbers of 1.8 million, 2.8 million, 3.5 million, and 4.3 million. So, based on some super-unscientific guesswork, I think I can happily say that the number of Python programmers in the world is in the low millions.

What do you think? Are there other ways of working this out that I've missed? Does anyone have (gasp!) hard numbers?

A super-simple chat app with AngularJS, SockJS and node.js

Posted on 12 February 2013 in JavaScript, Programming, PythonAnywhere

We're planning to move to a more advanced JavaScript library at PythonAnywhere. jQuery has been good for us, but we're rapidly reaching a stage where it's just not enough.

There are a whole bunch of JavaScript MVC frameworks out there that look tempting -- see TodoMVC for an implementation of a simple app in a bunch of them. We're asking the people we know and trust which ones are best, but in the meantime I had a look at AngularJS and knocked up a quick chat app to see how easy it would be. The answer was "very".

Here's the client-side code:

<html ng-app>
<head>
<script src="http://cdn.sockjs.org/sockjs-0.3.min.js"></script>
<script src="http://ajax.googleapis.com/ajax/libs/angularjs/1.0.4/angular.min.js"></script>

<script> var sock = new SockJS('http://192.168.0.74:9999/chat'); function ChatCtrl($scope) { $scope.messages = []; $scope.sendMessage = function() { sock.send($scope.messageText); $scope.messageText = ""; };

sock.onmessage = function(e) { $scope.messages.push(e.data); $scope.$apply(); }; } </script>

</head>

<body>

<div ng-controller="ChatCtrl"> <ul> <li ng-repeat="message in messages">{{message}}</li> </ul>

<form ng-submit="sendMessage()"> <input type="text" ng-model="messageText" placeholder="Type your message here" /> <input type="submit" value="Send" /> </form </div>

</body> </html>

Then on the server side I wrote this server (in node.js because I've moved to Shoreditch and have ironic facial hair it was easy to copy, paste and hack from the SockJS docs -- I'd use Tornado if this was on PythonAnywhere):

var http = require('http');
var sockjs = require('sockjs');

var connections = [];

var chat = sockjs.createServer(); chat.on('connection', function(conn) { connections.push(conn); var number = connections.length; conn.write("Welcome, User " + number); conn.on('data', function(message) { for (var ii=0; ii < connections.length; ii++) { connections[ii].write("User " + number + " says: " + message); } }); conn.on('close', function() { for (var ii=0; ii < connections.length; ii++) { connections[ii].write("User " + number + " has disconnected"); } }); });

var server = http.createServer(); chat.installHandlers(server, {prefix:'/chat'}); server.listen(9999, '0.0.0.0');

And that's it! It basically does everything you need from a simple chat app. Definitely quite impressed with AngularJS. I'll try it in some of the other frameworks we evaluate and post more here.

Reverse proxying HTTP and WebSockets with virtual hosts using nginx and tcp_proxy_module

Posted on 5 October 2012 in Programming, PythonAnywhere

I spent today trying to work out how we could get PythonAnywhere to support WebSockets in our users' web applications. This is a brief summary of what I found, I'll put it in a proper post on the PythonAnywhere blog sometime soon...

We use nginx, and it can happily route HTTP requests through to uwsgi applications (which is the way we use it) and can even more happily route them through to other socket-based servers running on specific ports (which we don't use but will in the future so that we can support Twisted, Tornado, and so on -- once we've got network namespacing sorted).

But by default, nginx does not support reverse proxying WebSockets requests. There are various solutions to this posted around the net, but they don't explain how to get it working with virtual hosts. I think that this is because they're all a bit old, because it's actually quite easy once you know how.

(It's worth mentioning that there are lots of cool non-nginx solutions using excellent stuff like haproxy and hipache. I'd really like to upgrade our infrastructure to use one of those two. But not now, we all too recently moved from Apache to nginx and I'm scared of big infrastructure changes in the short term. Lots of small ones, that's the way forward...)

Anyway, let's cut to the chase. This excellent blog post by Johnathan Leppert explains how to configure nginx to do TCP proxying. TCP proxying is enough to get WebSockets working if you don't care about virtual hosts -- but because arbitrary TCP connections don't necessarily have a Host: header, it can't work if you do care about them.

However, since the post was written, the nginx plugin module Johnathan uses has been improved so that it now supports WebSocket proxying with virtual hosts.

To get nginx to successfully reverse-proxy WebSockets with virtual host support, compile Nginx with tcp_proxy_module as per Johnathan's instructions (I've bumped the version to the latest stable as of today):

export NGINX_VERSION=1.2.4
curl -O http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz
git clone https://github.com/yaoweibin/nginx_tcp_proxy_module.git
tar -xvzf nginx-$NGINX_VERSION.tar.gz
cd nginx-$NGINX_VERSION
patch -p1 < ../nginx_tcp_proxy_module/tcp.patch
./configure --add-module=../nginx_tcp_proxy_module/
sudo make && make install

Then, to use the new WebSockets support in tcp_proxy_module, put something like this in your nginx config:

worker_processes  1;

events { worker_connections 1024; }

tcp { upstream site1 { server 127.0.0.1:1001;

check interval=3000 rise=2 fall=5 timeout=1000; }

server { listen 0.0.0.0:80; server_name site1.com;

tcp_nodelay on; websocket_pass site1; } }

tcp { upstream site2 { server 127.0.0.1:1002;

check interval=3000 rise=2 fall=5 timeout=1000; }

server { listen 0.0.0.0:80; server_name site2.com;

tcp_nodelay on; websocket_pass site2; } }

Hopefully that's enough to help a few people googling around for help like I was this morning. Leave a comment if you have any questions!

Raspberry Pi setup notes part 1: getting the display to work!

Posted on 20 June 2012 in Raspberry Pi

I received my Raspberry Pi yesterday, and today got it working well enough to display a text-based console on a DVI monitor using Arch Linux. There were a few hiccups along the way, so here are the complete notes so that anyone googling for the same errors as the ones I saw can benefit from my experience.

tl;dr: the file /boot/config.txt sets up various things before the OS is loaded, including HDMI settings. The one in the Arch Linux SD card image didn't work with my machine setup. The system default, which you can get just by removing that file completely, worked just fine.

Here are the details showing how I got to that...

  • I started with a Stock Arch Linux SD image
  • The monitor was a basic Acer DVI one, 1280x1024. It was attached to the RPi's HDMI port using a DVI -> HDMI adapter (labelled DVI-D -> HDMI) from Nikkai via Maplin. NB there are lots of different kinds of DVI plugs/sockets: here are some diagrams on Wikipedia. The socket on the adapter had holes for DVI-I dual link, and the plug from the monitor's cable was DVI-D single link. Nonetheless, as I eventually got it working, I guess that they are intercompatible to some degree.
  • As well as the monitor, the RPi was plugged into a USB keyboard, and an ethernet switch.
  • When I plugged my MicroUSB mains adapter into the RPi, the display remained blank -- it said there was no signal.
  • I checked out the logs of my DHCP server and saw that something called alarmpi had just got a lease, so I sshed to the IP and was able to log in using the default username/password for the Arch Linux distro (root/root) It was obviously working :-) :-) :-) (BTW the best guess around the office is that alarmpi is Arch Linux ARM Pi.
  • A bit of poking around with Google found various forum posts, many of which suggested that the key was the settings in /boot/config.txt
  • So I decided to look at that file in my ssh session. On the image I was using, it contained this:
    hdmi_mode=19
    #arm_freq=800
    disable_overscan=1
    
  • A forum post referenced this page, which appears to be a script to build a Debian wheezy image for the RPi; specifically, it contains the complete /boot/config.txt for that image, which has detailed comments explaining what a bunch of the switches do. So I edited my own config, merging that file in with the original settings (which I flagged with ### Original OOB setting):
    # uncomment if you get no picture on HDMI for a default "safe" mode
    #hdmi_safe=1
    

    # uncomment this if your display has a black border of unused pixels visible # and your display can output without overscan #disable_overscan=1 ### Original OOB setting: disable_overscan=1

    # uncomment the following to adjust overscan. Use positive numbers if console # goes off screen, and negative if there is too much border #overscan_left=16 #overscan_right=16 #overscan_top=16 #overscan_bottom=16

    # uncomment to force a console size. By default it will be display's size minus # overscan. #framebuffer_width=1280 #framebuffer_height=720

    # uncomment if hdmi display is not detected and composite is being output #hdmi_force_hotplug=1

    # uncomment to force a specific HDMI mode (this will force VGA) #hdmi_group=1 #hdmi_mode=1 ### Original OOB setting: hdmi_mode=19

    # uncomment to force a HDMI mode rather than DVI. This can make audio work in # DMT (computer monitor) modes #hdmi_drive=2

    # uncomment to increase signal to HDMI, if you have interference, blanking, or # no display #config_hdmi_boost=4

    # uncomment for composite PAL #sdtv_mode=2

    #uncomment to overclock the arm. 700 MHz is the default. #arm_freq=800

    # for more options see http://elinux.org/RPi_config.txt

  • I noticed that saving the file was slow. My best current guess for why this is is that the boot stuff is mounted from a separate tiny partition at the start of the SD card, FAT-formatted, which the RPi's chipset knows how to read to start the whole bootstrap process before there's even an operating system installed. And writing to that partition is just harder work than writing to a normal one. But perhaps I'm completely off-base about the reason for the slowdown, although I'm pretty sure about the bootstrap/partition thing.
  • Anyway, the hdmi_safe setting looked interesting, so I tried uncommenting it and rebooting. That didn't help.
  • Another likely one was suggested in this forum post: setting hdmi_force_hotplug=1. But that didn't help either.
  • I'd also noticed that people were suggesting running /opt/vc/bin/tvservice as a diagnostic tool; I ran it, but it failed:
    [root@alarmpi ~]# /opt/vc/bin/tvservice
    /opt/vc/bin/tvservice: error while loading shared libraries: libvcos.so: cannot open shared object file: No such file or directory
    
  • Clearly something important was either missing or just not on the search path. Check:
    [root@alarmpi ~]# ldd /opt/vc/bin/tvservice
            libvcos.so => not found
            libpthread.so.0 => /lib/libpthread.so.0 (0x400f9000)
            libdl.so.2 => /lib/libdl.so.2 (0x40074000)
            librt.so.1 => /lib/librt.so.1 (0x4007f000)
            libc.so.6 => /lib/libc.so.6 (0x4013f000)
            /lib/ld-linux.so.3 (0x4004c000)
    
  • Right. This forum post made it clear where it was, so:
    [root@alarmpi ~]# ls /opt/vc/lib/
    libbcm_host.so    libEGL.so        libGLESv2_static.a  libkhrn_static.a  libmmal.so       libvcfiled_check.a  libvcos.so
    libcontainers.so  libEGL_static.a  libilclient.a       libluammal.so     libopenmaxil.so  libvchiq_arm.a      libvmcs_rpc_client.a
    libdebug_sym.so   libGLESv2.so     libkhrn_client.a    liblua.so         libOpenVG.so     libvchostif.a       libWFC.so
    [root@alarmpi ~]# export LD_LIBRARY_PATH=/opt/vc/lib/:LD_LIBRARY_PATH
    [root@alarmpi ~]# ldd /opt/vc/bin/tvservice
            libvcos.so => /opt/vc/lib/libvcos.so (0x400c2000)
            libpthread.so.0 => /lib/libpthread.so.0 (0x4012d000)
            libdl.so.2 => /lib/libdl.so.2 (0x40038000)
            librt.so.1 => /lib/librt.so.1 (0x400ff000)
            libc.so.6 => /lib/libc.so.6 (0x4014d000)
            /lib/ld-linux.so.3 (0x40010000)
    
  • Good. Now let's get some diagnostics:
    [root@alarmpi ~]# /opt/vc/bin/tvservice -s
    state 0x40002, 720x480 @ 60Hz, interlaced
    [root@alarmpi ~]# /opt/vc/bin/tvservice -m CEA
    Group CEA has 0 modes:
    [root@alarmpi ~]# 
    
  • Hmmm. Not terribly helpful. But maybe there are more options? This forum post suggested one:
    [root@alarmpi ~]# /opt/vc/bin/tvservice -m DMT
    Group DMT has 13 modes:
               mode 4: 640x480 @ 60Hz, progressive
               mode 5: 640x480 @ 72Hz, progressive
               mode 6: 640x480 @ 75Hz, progressive
               mode 8: 800x600 @ 56Hz, progressive
               mode 9: 800x600 @ 60Hz, progressive
               mode 10: 800x600 @ 72Hz, progressive
               mode 11: 800x600 @ 75Hz, progressive
               mode 16: 1024x768 @ 60Hz, progressive
               mode 17: 1024x768 @ 70Hz, progressive
               mode 18: 1024x768 @ 75Hz, progressive
               mode 21: 1152x864 @ 75Hz, progressive
               mode 35: 1280x1024 @ 60Hz, progressive
               mode 36: 1280x1024 @ 75Hz, progressive
    
  • ooookay... mode 35 looks the most relevant for my monitor...
  • But reading the forum post further, it also sounds like removing the config.txt and rebooting is a good thing for debugging. Let's do that first.
    [root@alarmpi ~]# mv /boot/config.txt .
    [root@alarmpi ~]# ls /boot
    arm128_start.elf  bootcode.bin      kernel_emergency.img  start.elf
    arm192_start.elf  cmdline.txt       kernel.img
    arm224_start.elf  kernel_debug.img  loader.bin
    [root@alarmpi ~]# ls .
    config.txt
    
  • Rebooted, and... well, bugger me, it worked!

Running Django unit tests on PythonAnywhere

Posted on 21 May 2012 in Programming, PythonAnywhere

I was working on a side project today, a Django app hosted at PythonAnywhere. While writing some initial unit tests, I discovered a confusing bug. When you try to run the tests for your app, you get an error message creating the database (for the avoidance of doubt, USERNAME was my PA username):

18:57 ~/somewhere (master)$ ./manage.py test
Creating test database for alias 'default'...
Got an error creating the test database: (1044, "Access denied for user 'USERNAME'@'%' to database 'test_USERNAME$default'")
Type 'yes' if you would like to try deleting the test database 'test_USERNAME$default', or 'no' to cancel: no
Tests cancelled.

The problem is that PythonAnywhere users don't have the privileges to create the database test_USERNAME$default (whose name Django's unit testing framework has auto-generated from the USERNAME$default that is the DB name in settings.py). PA only allows you to create new databases from its web interface, and also only allows you to create databases that are prefixed with your-username$

After a bit of thought, I realised that you can work around the problem by setting TEST_NAME in settings.py to point to a specific new database (say, USERNAME$unittest) and then creating a DB of that name from the MySQL tab. Once you've done that, you run the tests again; you get an error like this:

19:02 ~/somewhere (master)$ ./manage.py test
Creating test database for alias 'default'...
Got an error creating the test database: (1007, "Can't create database 'USERNAME$unittest'; database exists")
Type 'yes' if you would like to try deleting the test database 'USERNAME$unittest', or 'no' to cancel:

You just enter "yes", and it will drop then recreate the database. This works, because when you created it from the MySQL page, the settings were set up correctly for you to be able to create and drop it again in the future. Once this has been done once, tests run just fine in the future, with no DB errors.

Obviously we'll be fixing this behaviour in the future (though I can't offhand see how...). But there's the workaround, anyway.

New business idea

Posted on 10 April 2012 in Rants

So, here's the plan. We write an iPhone app. iPhone-only, no Android. It's a simple social network, adding friends and chatting and sharing photos and all that crap. The cool thing is, it monitors your location. If you ever spend more than 50% of one week outside Shoreditch in London, the East Village in New York, or SoMa in San Francisco, it kicks you out -- you can never log in again. Once a week, it asks you a question about post-1900 conceptual art or artisan food vendors in your area. If you get it wrong, it kicks you out. Every day you have to take a photo of yourself, and other users get to vote on your outfit/fixed-gear bike/ironic facial hair. If you get less than a 50% approval rating, it kicks you out. Finally, the app comes with a guarantee that if the company's ever bought by Facebook, 10% of the purchase price goes to its few remaining members.

Who's with me? What should we call it?

PythonAnywhereAnywhere

Posted on 27 February 2012 in Programming, Python, PythonAnywhere, Resolver Systems

We recently added something cool to PythonAnywhere, our Python online IDE and web hosting environment -- if you're writing a tutorial, or anything else where you'd find a Python console useful in a web page, you can use one of ours! Check it out:

What's particularly cool about these consoles (apart from the fact that they advertise the world's best Python IDE-in-a-browser) is that they keep the session data on a per-client basis -- so, if you put one on multiple pages of your tutorial, the user's previous state is kept as they navigate from page to page! The downside (or is it an upside?) is that this state is also kept from site to site, so if they go from your page to someone else's, they'll have the state they had when they were trying out yours.

Bug or feature? Let me know what you think in the comments...

Teaching programming

Posted on 14 October 2011 in Programming

One thing we wanted to do with PythonAnywhere, our Python online IDE and web hosting environment, was put together a short introduction to Python for non-programmers. I wrote the first cut the other day.

I've always been a fan of the Pimsleur language lessons. Unlike very traditional ways of teaching foreign languages, they don't make you learn vocabulary lists and grammatical rules. Unlike more modern systems, they don't try to teach you phrases. There's no written textbook, just a bunch of CDs (or these days MP3s). And they throw you right in at the deep end.

At the start of each Pimsleur course, a soothing voice says something like "Welcome to Pimsleur Klingon part one, lesson one. Listen to the following conversation." And then you hear something that sounds like this. The soothing voice comes back and says "in 30 minutes you'll hear that again, and you'll understand it." And then you're introduced to the different sentences, and told to repeat stuff. "Repeat this: tlhab 'oS 'Iw HoHwI' So' batlh" they'll say, and you'll stumble your way through it. Then they'll get you to repeat "tlhab" on its own a few times. For complicated words they'll break it down, and get you to repeat the last bit first, gradually building up. "I". "wI". "oHwI". "HoHwI". After 30 minutes of this, your brain is aching, and then they play the conversation you heard at the start -- and it makes perfect sense!

Quite incredible.

So what's this all got to do with teaching programming? Well, we were looking for a decent "Python for non-programmers" course that we could use, and while we found a couple, they all seemed to have the same problem -- they'd work by gradually introducing simple concepts, and after twenty pages you might have learned enough to write a trivial program.

That's not how any of us learned programming. Instead, we tended to pick it up bit by bit by inspection of complete working bits of code. For me, it was typing in listings from the home computing magazines that I bought. (Note for younger readers: listings were printed programs in magazines in the days when disks were too expensive to give away with magazines. (Note for even younger readers: disks were primitive USB-stick-like things that people used to use to store data, and were often attached to magazines as a way of passing data from magazine to reader in the days when you couldn't just put a URL in the article text. (Note for yet younger readers: magazines were collections of very thin slices of wood on which text and images were "printed" using chemical dyes, which could be purchased in places called shops. They were popular in the Dark Ages before the Facebook.)))

That way of learning -- which must be even easier now that any aspiring programmer can look at the source of tons of open source software, or just view the source of any web page -- was quick, efficient, and got you straight to a position where you felt you'd achieved something. You'd written your first program! You could change the bits you understood, and leave the rest for a later day when you knew more. A great way to learn, as well as being excellent training for later days when you find yourself maintaining some horrific codebase written by someone who thinks that operator overloading is the best thing since sliced bread.

So, my theory for the first part of this tutorial -- and, if it works out, for any following ones -- is that each should start with a program listing that the reader won't understand yet, but should be able to understand with a little explication. Core concepts should be drilled in by relentless repetition. No baby-talk -- start using the technical terms straight away, and then repeat them enough that if the reader forgets what "variable" means then they can work it out from context. And aim to get them from knowing nothing to simple OO and functional programming in five half-hour lessons.

What do you think?

If programming languages were literary genres...

Posted on 24 June 2011 in Funny, Programming

From a conversation at the pub this evening with Jonathan and Glenn...

  • JavaScript would be science fiction. Easy to get into, and while there's an incredible amount of crap out there, there are also many gems that, carefully considered, will completely change your outlook.
  • C and C++ would be "literary" fiction. Somewhat old-fashioned, occasionally brilliant, but frequently clunky and never quite as good as its devotees and creators like to think it is...
  • LISP and Scheme would be obscure branches of poetry; densely packed with information, a few words express something that would take that would take tens of thousands to express in other languages. But only a few dozen people in the world would understand it.

...and somewhere around there, we ran out of steam. Java as corporate mission statements? Not quite... VBA as newspaper cartoons? Hmm. COBOL as... God only knows. And we couldn't even place Python.

Anyone got any ideas?

Resolver is hiring

Posted on 16 May 2011 in Resolver Systems

We’re looking for a Software Developer to work on PythonAnywhere, Resolver One, and our other products, based in our offices in Clerkenwell, London, UK. More information on the Resolver jobs page.