Recovered!

In early 2007, I had a nasty DB issue on my web server and (I thought) lost the first year or so's worth of posts to gilesthomas.com -- including the history of my attempts to automate my backups using s3sync, which are inexplicably popular :-). Even worse, the subsequent restarting of post IDs meant that Google Reader stopped noticing updates to the website, leading to a number of readers thinking I'd gone quiet.

Today, I found an old backup, and I've managed to merge it in with the recent posts -- so hopefully all of these problems are now fixed. There's a risk that some RSS readers might mark a lot of old posts "unread", so apologies if it looked to you like I'd just made a few dozen new posts.

All should be well now.

[Update] I've checked Google Reader and it looks like posts are feeding through again. w00t!

I came for the article, I stayed for the comments

An OK but somewhat oversimplified post on short selling leads to a fascinating discussion on the topic from market practitioners. Well worth reading to the end, you'll wind up holding at least two irrefutable but mutually contradictory opinions about the issue...

Why use IronPython?

I just posted this on the Joel on Software discussion board, in answer to someone's question about using IronPython for their new company. Hopefully it will be of interest here.

We've been using IronPython for three years now with a lot of success. The great thing about it is that it allows you to benefit from Python's syntax while getting most of the advantages of .NET:

• All of the .NET libraries are available.
• UIs look nice. I've never seen a pure traditional Python application that looked good, no matter how advanced its functionality.
• We use a bunch of third-party components -- for example, Syncfusion's Essential Grid -- without any problems.
• Reasonably decent multithreading using the .NET libraries -- CPython, the normal Python implementation, has the problem of the Global Interpreter Lock, an implementation choice that makes multithreading dodgy at best.
• We can build our GUI in Visual Studio, and then generate C# classes for each dialog, and then subclass them from IronPython to add behaviour. (We never need to look at the generated code.)
• When things go wrong, the CLR debugger works well enough -- it's not perfect, but we've never lost a significant amount of time for want of anything better.

Of course, it's not perfect. Problems versus using C#:

• It's slower, especially in terms of startup time. They are fixing this, but it's a problem in the current release. This hasn't bitten us yet -- all of the non-startup-related performance issues we've had have been due to suboptimal algorithms rather than language speed. However, it you're writing something that's very performance-intensive, you may want to look elsewhere.
• No LINQ yet.
• If you're considering IP then you presumably already know this, but dynamic languages have no compile-time to perform sanity checks on your codebase, so problems can come up at runtime. We write all of our code test-first and so we aren't impacted by that. However, if you're not writing a solid amount of test code (and if you're not, you should :-) then you might want to use a statically-typed language.

Problems versus using CPython:

• No cross-platform. Linux or Mac support is one of our more frequently-requested enhancements, and it will be a lot of work to add. The reason for this is that many third-party .NET components -- for example, the Synfusion grid -- are not "pure" .NET; they drop into win32 for certain operations, I assume for performance reasons. This means that if you use them, your application won't run on non-Windows .NET platforms.
• No use of CPython's C extensions, like numpy (a numerical functions library). This has hit us pretty hard, so we're working on an open-source library to interface between C extensions and IronPython -- however, it's still a work in progress.

Hope this was of some help.

Evolution in action

At Resolver Systems, we've recently split into two teams; about two thirds of us work on the core Resolver One platform that is our main product (this group is inventively called the Platform team), and the other third build new spreadsheet/Python programs, using Resolver One, for specific clients' custom needs (the Apps team). This is great, because we are now not only building business solutions for people, as well as a generic platform (which means more money for us), but we are also dogfooding -- so we can be sure we're adding features and fixing bugs which really do help our users.

The problem with doing this is that everyone in the company has different preferences about how much time they want to spend in each team. Some people really like writing programs to fix business problems, and others are keener on abstract algorithms. We could have just said "stuff it" and swapped people around so that everyone was doing a 1:2 rotation, but it was much more fun to solve the problem in software :-) My aim was to somehow generate, for each of the next twelve iterations (the two-week development cycles we work in), a list of people who would form that iteration's Apps team and the people who'd form the iteration's Platform team.

So I put together a spreadsheet: an evolutionary algorithm for team scheduling. If you're using Windows, you can download it and take a look (you can get a free version of Resolver One to run it on if you haven't already). You enter your team's preferences -- in terms of the percentage of time they'd like to spend on the Apps team -- in the "Preferences" sheet (which also shows some results from the last run), and then some numbers to guide the evolution (number of generations, population size, etc) in the "Parameters" sheet, and then get the best schedule it can generate in "Rota" sheet.

To be honest, it's using the spreadsheet more as a display mechanism than anything else. But it's a fun bit of code, although I'm sure that anyone who actually works on evolutionary algorithms would find it trivially simple (and probably broken :-). The function GenerateSchedule in the pre-formulae user code (for Resolver newbies: in the box below the grid - the section with a green background) is the interesting bit -- everything below there is just presentation logic. Here's how it works:

• We generate a random set of schedules, each of which is created by picking three random people from our team and putting them into the Apps team, leaving the remainder in the Platform team.
• We then run through as many generations as the user specified. In each generation:
  • Every schedule in our population is assigned a weight. This is generated by a function called WeightSchedule, which is what people who study evolutionary algorithms would call a fitness function. Basically, the higher the number it returns, the less good the schedule is.
  • We sort the schedules by their weights, and then we kill off the worst of them.
  • We then create a new generation comprising the survivors from the cull, and a set of new schedules that are "parented" by those survivors, using the function MutateSchedule. We apply a slight bias so that the fitter schedules have a better chance of reproducing than the others.
  • And on we go for another generation.

WeightSchedule was the most difficult function in the code to get right. (This is in keeping with what I've heard about evolutionary algorithms in general.) Its job is to return a number that is high for bad schedules, and low for good ones. I found I got the best results by returning an arbitrary "high" value for any schedule that failed to meet certain must-have criteria, and then working out, for each person, the difference between the amount of time they wanted to spend in a given team and the actual amount of time they spent there in the current schedule. I then raised those per-person errors to the power of four (to make it clear that three people 5% out is better than one person 15% out) and then summed the results. This seemed to work just fine.

For MutateSchedule I had a bit of fun. It's purpose is to generate a new child from a single parent schedule (I chose to use an asexual reproduction model because, in my experience, sexual reproduction and spreadsheets rarely mix well). My initial implementation just switched one pair of people around for every iteration -- that is, one person who was originally on the Apps team was now on the Platform team, and vice versa. I then made the number of such swaps a user-settable parameter, so that people could increase the extent of mutations. This sounded like a good idea, but didn't help much -- indeed, increasing the number of swaps invariably made the system less likely to produce a good schedule. My "background radiation" level was clearly too high. So I then changed things so that you could specify a fractional number of swaps. A swap level of 0.1 meant that each iteration has a one in ten chance of a having someone swapped around. This seemed to work well -- indeed, 0.1 seemed pretty close to the sweet spot for the number of swaps. I suppose this makes sense -- you can imagine that a schedule with twelve iterations in it that is almost perfect is more likely to be improved if you switch around two people in just one of its iterations than if you make a swap for every iteration.

So that's it -- a simple evolutionary algorithm in a spreadsheet. I've deliberately not over-tidied the code in the version you can download above -- I've just sanitised the data so that no-one on the team's privacy is harmed, and then added a few comments for the more impenetrable bits of code. But it should all be pretty easy to understand, and I'd love to hear from anyone with comments (especially if they know more about this kind of thing than me...)