Writing a reverse proxy/loadbalancer from the ground up in C, part 3: Lua-based configuration

This is the third step along my road to building a simple C-based reverse proxy/loadbalancer so that I can understand how nginx/OpenResty works -- more background here. Here's a link to the first part, where I showed the basic networking code required to write a proxy that could handle one incoming connection at a time and connect it with a single backend, and to the second part, where I added the code to handle multiple connections by using epoll.

This post is much shorter than the last one. I wanted to make the minimum changes to introduce some Lua-based scripting -- specifically, I wanted to keep the same proxy with the same behaviour, and just move the stuff that was being configured via command-line parameters into a Lua script, so that just the name of that script would be specified on the command line. It was really easy :-) -- but obviously I may have got it wrong, so as ever, any comments and corrections would be much appreciated.

Just like before, the code that I'll be describing is hosted on GitHub as a project called rsp, for "Really Simple Proxy". It's MIT licensed, and the version of it I'll be walking through in this blog post is as of commit 615d20d9a0. I'll copy and paste the code that I'm describing into this post anyway, so if you're following along there's no need to do any kind of complicated checkout.

The first thing I should probably explain, though, is why I picked Lua for this. I'm founder of a company called PythonAnywhere, so why not Python? Well, partly it's a kind of cargo-cult thing. nginx (and particularly OpenResty) use Lua for all of their scripting, so I probably should too (especially if that's what I'm trying to emulate.

Another reason is that Lua is really, really easy to integrate into C programs -- it was one of the design goals. Python is reasonably easy to embed, but as soon as you want to get objects out, you have to do a lot of memory management and it can get hairy (just scroll down that page a bit to see what I mean). From what I've read, Lua makes this kind of thing easier. I may learn better later.

Finally, there's the fact that Lua is just very very fast. As a language, I think it's not as nice as Python. But perhaps the things that look like language flaws to me were important tradeoffs in making it so fast. LuaJIT, in particular, is apparently blindingly fast. I've seen the words "alien technology" floating around to refer to it...

So there we go. Let's look at how it can be integrated into the proxy from the last post. Remember, all we're doing at this stage is using it as a glorified config file parser; more interesting stuff will come later.

The first step is to get hold of a Lua library to use. Inspired by the whole alien technology thing, I went for LuaJIT, and installed it thusly:

git clone http://luajit.org/git/luajit-2.0.git
cd luajit-2.0
make && sudo make install

Now, we need to build it into the program. A few changes to the Makefile; LuaJIT installs itself to /usr/local/, so:

INCLUDE_DIRS := -I/usr/local/include/luajit-2.0/
LDFLAGS := -Wl,-rpath,/usr/local/lib
LIBS := -lluajit-5.1



...



%.o: %.c $(HEADER_FILES)
       $(CC) -c -Wall -std=gnu99 $(INCLUDE_DIRS) $<



...



rsp: $(OBJ_FILES)
       $(CC) -o $@ $(OBJ_FILES) $(LDFLAGS) $(LIBS)

Note the use of -Wl,-rpath,/usr/local/lib in the LDFLAGS instead of the more traditional -L/usr/local/lib. This bakes the location of the library into the rsp executable, so that it knows to look in /usr/local/lib at runtime rather than relying on us always setting LD_LIBRARY_PATH to tell it where to find the libluajit-5.1.so file.

Now, some code to use it. For now, it's all in rsp.c. At the top, we need to include the headers:

#include <luajit.h>
#include <lauxlib.h>

And now we can use it. Jumping down to main, there's this line:

lua_State *L = lua_open();

That's all you need to do to create a new Lua interpreter. The capital L seems to be the tradition amongst Lua programmers when embedding the interpreter, so we'll stick with that for now. Next, we want to load our config file (with an appropriate error if it doesn't exist or if it's not valid Lua, both of which conditions are reported to us with appropriate error messages by the Lua interpreter):

    if (luaL_dofile(L, argv[1]) != 0) {
        fprintf(stderr, "Error parsing config file: %s\n", lua_tostring(L, -1));
        exit(1);
    }

That call to lua_tostring with its -1 parameter is worth a bit more discussion. All data that's passed from Lua to C goes across a stack, which is maintained by the interpreter. lua_tostring(L, -1) means "get the top thing from the interpreter's C communication stack, assume it's a string, and put it in a char* for me". The function (actually, macro) luaL_dofile, if it fails, returns a non-zero code and pushes an error message on to the stack -- so we can use that to extract the error message.

So, once that code's been run, we have a Lua interpreter in which our config file has been run. Now we need to extract the configuration values we want from it. The code that does this in main uses a simple utility function, get_config_opt:

    char* server_port_str = get_config_opt(L, "listenPort");
    char* backend_addr = get_config_opt(L, "backendAddress");
    char* backend_port_str = get_config_opt(L, "backendPort");

...and get_config_opt looks like this:

char* get_config_opt(lua_State* L, char* name) {
    lua_getglobal(L, name);
    if (!lua_isstring(L, -1)) {
        fprintf(stderr, "%s must be a string", name);
        exit(1);
    }
    return (char*) lua_tostring(L, -1);
}

Again, that stack: lua_getglobal gets the value of the given global variable and puts it on the stack. In Lua, if a global is not defined, it has the value nil, so this won't break here; instead, we next ask the interpreter if the thing on the top of the stack is a string using lua_isstring -- this covers the nil case, and also any other cases where something weird has been put in the variable. Once we've determined that the thing on the top of the stack is a string, we extract it using the lua_tostring function we used before.

So, what's the next step? There is no next step! That was all that we needed to do to use Lua to configure the proxy.

Now it's time to do some serious stuff -- parsing the HTTP headers so that we can delegate incoming client connections to backends based on their host header (and perhaps other things). Stay tuned!

[UPDATE: actually, the next post is going to be about fixing a bug in the previous version; more here]