Getting MathML to render properly in Chrome, Chromium and Brave

The other day I posted about adding mathematical typesetting to this blog using markdown2, LaTeX and MathML. One problem that remained at the end of that was that it looked a bit rubbish; in particular, the brackets surrounding matrices were just one line high, albeit centred, like this:

...rather than stretched to the height of the matrix, like this example from KaTex:

After posting that, I discovered that the problem only existed in Chromium-based browsers. I saw it in Chromium, Chrome and Brave on Android and Linux, but in Firefox on Linux, and on Safari on an iPhone, it rendered perfectly well.

Guided by the answers to this inexplicably-quiet Stack Overflow question, I discovered that the prolem is the math fonts available on Chromium-based browsers. Mathematical notation, understandably, needs specialised fonts. Firefox and Safari either have these pre-installed, or do something clever to adapt the fonts you are using (I suspect the former, but Firefox developer tools told me that it was using my default body text font for <math> elements). Chromium-based browsers do not, so you need to provide one in your CSS.

Using Frédéric Wang's MathML font test page, I decided I wanted to use the STIX font. It was a bit tricky to find a downloadable OTF file (you specifically need the "math" variant of the font -- in the same way as you might find -italic and -bold files to download, you can find -math ones) but I eventually found a link on this MDN page.

I put the .otf file in my font assets directory, then added the appropriate stuff to my CSS -- a font face definition:

@font-face {
    font-family: 'STIX-Two-Math';
    src: url('/fonts/STIXTwoMath-Regular.otf') format('opentype');
}

...and a clause saying it should be used for <math> tags:

math {
    font-family: STIX-Two-Math;
    font-size: larger;
}

The larger font size is because by default it was rendering about one third of the height of my body text -- not completely happy about that, as it feels like an ad-hoc hack, but it will do for now.

Anyway, mathemetical stuff now renders pretty well! Here's the matrix from above, using my new styling:

I hope that's useful for anyone else hitting the same problem.

[Update: because RSS readers don't load the CSS, the bad rendering still shows up in NewsBlur's Android app, which I imagine must be using Chrome under the hood for its rendering. Other RSS readers are probably the same :-(]

Adding mathematical typesetting to the blog

I've spent a little time over the weekend adding the ability to post stuff in mathematical notation on this blog. For example:

It should render OK in any browser released after early 2023; I suspect that many RSS readers won't be able to handle it right now, but that will hopefully change over time. [Update: my own favourite, NewsBlur, handles it perfectly!]

Here's why I wanted to do that, and how I did it.

[ Read more ]

Installing the unifi controller on Arch

This is more of a note-to-self than a proper blog post. I recently got a new Ubiquiti access point, and needed to reinstall the unifi controller on my Arch machine in order to run it.

There's no formal package for unifi, so you have to install the AUR. I use yaourt for that, and if you do a simple

yaourt -S unifi

...then it will try to install MongoDB from source. According to the Arch Wiki, this requires "180GB+ free disk space, and may take several hours to build (i.e. 6.5 hours on Intel i7, 1 hour on 32 Xeon cores with high-end NVMe.)". So not ideal.

The trick is to install MongoDB from binary first:

yaourt -S mongodb-bin

And only after that:

yaourt -S unifi

Finally, activate the service:

sudo systemctl enable unifi
sudo systemctl start unifi

...and then go to https://localhost:8443/, accept the self-signed cert, and you're all set.

Creating a time series from existing data in pandas

pandas is a high-performance library for data analysis in Python. It's generally excellent, but if you're a beginner or you use it rarely, it can be tricky to find out how to do quite simple things -- the code to do what you want is likely to be very clear once you work it out, but working it out can be relatively hard.

A case in point, which I'm posting here largely so that I can find it again next time I need to do the same thing... I had a list start_times of dictionaries, each of which had (amongst other properties) a timestamp and a value. I wanted to create a pandas time series object to represent those values.

The code to do that is this:

import pandas as pd
series = pd.Series(
    [cs["value"] for cs in start_times],
    index=pd.DatetimeIndex([cs["timestamp"] for cs in start_times])
)

Perfectly clear once you see it, but it did take upwards of 40 Google searches and help from two colleagues with a reasonable amount of pandas experience to work out what it should be.

Parsing website SSL certificates in Python

A kindly PythonAnywhere user dropped us a line today to point out that StartCom and WoSign's SSL certificates are no longer going to be supported in Chrome, Firefox and Safari. I wanted to email all of our customers who were using certificates provided by those organisations.

We have all of the domains we host stored in a database, and it was surprisingly hard to find out how I could take a PEM-formatted certificate (the normal base-64 encoded stuff surrounded by "BEGIN CERTIFICATE" and "END CERTIFICATE") in a string and find out who issued it.

After much googling, I finally found the right search terms to get to this Stack Overflow post by mhawke, so here's my adaptation of the code:

from OpenSSL import crypto

for domain in domains:
    cert = crypto.load_certificate(crypto.FILETYPE_PEM, domain.cert)
    issuer = cert.get_issuer().CN
    if issuer is None:
        # This happened with a Cloudflare-issued cert
        continue
    if "startcom" in issuer.lower() or "wosign" in issuer.lower():
        # send the user an email

Reverse proxying HTTP and WebSockets with virtual hosts using nginx and tcp_proxy_module

I spent today trying to work out how we could get PythonAnywhere to support WebSockets in our users' web applications. This is a brief summary of what I found, I'll put it in a proper post on the PythonAnywhere blog sometime soon...

We use nginx, and it can happily route HTTP requests through to uwsgi applications (which is the way we use it) and can even more happily route them through to other socket-based servers running on specific ports (which we don't use but will in the future so that we can support Twisted, Tornado, and so on -- once we've got network namespacing sorted).

But by default, nginx does not support reverse proxying WebSockets requests. There are various solutions to this posted around the net, but they don't explain how to get it working with virtual hosts. I think that this is because they're all a bit old, because it's actually quite easy once you know how.

(It's worth mentioning that there are lots of cool non-nginx solutions using excellent stuff like haproxy and hipache. I'd really like to upgrade our infrastructure to use one of those two. But not now, we all too recently moved from Apache to nginx and I'm scared of big infrastructure changes in the short term. Lots of small ones, that's the way forward...)

Anyway, let's cut to the chase. This excellent blog post by Johnathan Leppert explains how to configure nginx to do TCP proxying. TCP proxying is enough to get WebSockets working if you don't care about virtual hosts -- but because arbitrary TCP connections don't necessarily have a Host: header, it can't work if you do care about them.

However, since the post was written, the nginx plugin module Johnathan uses has been improved so that it now supports WebSocket proxying with virtual hosts.

To get nginx to successfully reverse-proxy WebSockets with virtual host support, compile Nginx with tcp_proxy_module as per Johnathan's instructions (I've bumped the version to the latest stable as of today):

export NGINX_VERSION=1.2.4
curl -O http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz
git clone https://github.com/yaoweibin/nginx_tcp_proxy_module.git
tar -xvzf nginx-$NGINX_VERSION.tar.gz
cd nginx-$NGINX_VERSION
patch -p1 < ../nginx_tcp_proxy_module/tcp.patch
./configure --add-module=../nginx_tcp_proxy_module/
sudo make && make install

Then, to use the new WebSockets support in tcpproxymodule, put something like this in your nginx config:

worker_processes  1;

events {
    worker_connections  1024;
}

tcp {
    upstream site1 {
        server 127.0.0.1:1001;

        check interval=3000 rise=2 fall=5 timeout=1000;
    }

    server {
        listen 0.0.0.0:80;
        server_name site1.com;

        tcp_nodelay on;
        websocket_pass site1;
    }
}


tcp {
    upstream site2 {
        server 127.0.0.1:1002;

        check interval=3000 rise=2 fall=5 timeout=1000;
    }

    server {
        listen 0.0.0.0:80;
        server_name site2.com;

        tcp_nodelay on;
        websocket_pass site2;
    }
}

Hopefully that's enough to help a few people googling around for help like I was this morning. Leave a comment if you have any questions!

Raspberry Pi setup notes: getting the display to work!

I received my Raspberry Pi yesterday, and today got it working well enough to display a text-based console on a DVI monitor using Arch Linux. There were a few hiccups along the way, so here are the complete notes so that anyone googling for the same errors as the ones I saw can benefit from my experience.

tl;dr: the file /boot/config.txt sets up various things before the OS is loaded, including HDMI settings. The one in the Arch Linux SD card image didn't work with my machine setup. The system default, which you can get just by removing that file completely, worked just fine.

Here are the details showing how I got to that...

[ Read more ]

Running Django unit tests on PythonAnywhere

I was working on a side project today, a Django app hosted at PythonAnywhere. While writing some initial unit tests, I discovered a confusing bug. When you try to run the tests for your app, you get an error message creating the database (for the avoidance of doubt, USERNAME was my PA username):

18:57 ~/somewhere (master)$ ./manage.py test
Creating test database for alias 'default'...
Got an error creating the test database: (1044, "Access denied for user 'USERNAME'@'%' to database 'test_USERNAME$default'")
Type 'yes' if you would like to try deleting the test database 'test_USERNAME$default', or 'no' to cancel: no
Tests cancelled.

The problem is that PythonAnywhere users don't have the privileges to create the database test_USERNAME$default (whose name Django's unit testing framework has auto-generated from the USERNAME$default that is the DB name in settings.py). PA only allows you to create new databases from its web interface, and also only allows you to create databases that are prefixed with your-username$

After a bit of thought, I realised that you can work around the problem by setting TEST_NAME in settings.py to point to a specific new database (say, USERNAME$unittest) and then creating a DB of that name from the MySQL tab. Once you've done that, you run the tests again; you get an error like this:

19:02 ~/somewhere (master)$ ./manage.py test
Creating test database for alias 'default'...
Got an error creating the test database: (1007, "Can't create database 'USERNAME$unittest'; database exists")
Type 'yes' if you would like to try deleting the test database 'USERNAME$unittest', or 'no' to cancel:

You just enter "yes", and it will drop then recreate the database. This works, because when you created it from the MySQL page, the settings were set up correctly for you to be able to create and drop it again in the future. Once this has been done once, tests run just fine in the future, with no DB errors.

Obviously we'll be fixing this behaviour in the future (though I can't offhand see how...). But there's the workaround, anyway.

Bare Git repositories

We started a new project at Resolver today -- I'm pretty excited about it, and will be blogging about it soon. However, in the meantime, here's something that's half a note-to-self and half something to help people googling for help with Git problems.

We've previously been using Subversion as our main source code control system, but for more recent projects we've moved to Mercurial. When we started the new one today, we decided to try out Git for a change; I use GitHub for my personal stuff, but hadn't used it for anything involving multiple developers -- and various people had been telling us that it wasn't subject to some of the problems we'd had with Mercurial.

So we created a new Git repo on a shared directory, by creating a directory and then running git init in it. We then cloned it into a working directory on my machine, and started work. After a while, we had our first checkin ready, so we a dded the files, committed them, and then decided to push to the central repo to make sure everything worked OK. We got this error message:

remote: error: refusing to update checked out branch: refs/heads/master
remote: error: By default, updating the current branch in a non-bare repository
remote: error: is denied, because it will make the index and work tree inconsistent
remote: error: with what you pushed, and will require 'git reset --hard' to match
remote: error: the work tree to HEAD.
remote: error:
remote: error: You can set 'receive.denyCurrentBranch' configuration variable to
remote: error: 'ignore' or 'warn' in the remote repository to allow pushing into
remote: error: its current branch; however, this is not recommended unless you
remote: error: arranged to update its work tree to match what you pushed in some
remote: error: other way.
remote: error:
remote: error: To squelch this message and still keep the default behaviour, set
remote: error: 'receive.denyCurrentBranch' configuration variable to 'refuse'.

It took us a while to work out precisely what this meant, because we'd never heard of "bare" repositories before. It turns out that there are two kinds of repository in Git: bare and non-bare. A non-bare repository is the same as the ones we were used to in Mercurial; it has a bunch of working files, and a directory containing the version control information. A bare repository, by contrast, just contains the version control information -- no working files.

Now, you can (in theory) push and pull between repositories regardless of whether they are bare or not. But if you were to push to a non-bare repository, it would cause problems. Part of the SCC data that Git keeps is an index, which basically tells it what the head of the current branch looks like. Now, if you push to a non-bare repository, Git will look at the working files, compare them to the index, and see that they differ -- so it will think that the working files have changed! For example, if your push added a new file, it would notice that the working directory didn't include that file, and would conclude that it had been deleted. There's a step-by-step example here.

You can see how that could be confusing. So bare repositories exist as a way of having central repositories that a number of people can push to. If you want to transfer changes from a non-bare repository to another, the correct way is to pull from the destination rather than push from the target -- which makes some kind of sense when you think about it. In general, any repository that someone is working on is not something that should be receiving changes without their approval... on the other hand, we've not encountered problems with pushing to regular repositories with Mercurial.

Anyway, this was our first checkin, so we had no history to lose, we fixed the problem by creating a new central repository using git --bare init in a new directory on the shared drive, cloning it to a new working repo, copying our files over from the old working repo to the new one, committing, and pushing back to the bare repository. It worked just fine. If we'd done multiple checkins before we tried our first push, we could have saved things by hand-editing the central repository; it had no working files (because we'd only just created it) so we could have moved the contents of the .git directory up to the repository's root, and deleted .git -- this would have "bared" it so that we could have pushed from our working repo. That would have been a bit scary, though.

An odd crontab problem

This took a little while to work out, so it's worth sharing here just in case anyone else has the same problems and is googling for solutions. We had a problem on one of our web servers at Resolver which manifested itself in some (but not all) cron jobs being run twice, which was causing all kinds of problems. Here's how we tracked it down and solved it.

[ Read more ]