Windows Subsystem For Astronomers

It's been dead around here of late because I just can't work up the energy to read a paper. Soon, perhaps.

Today, I want to do something a bit different : extol the virtues of Windows Subsystem for Linux. You can hate me all you want, but I will defend to the death (a) the Star Wars prequels and (b) Microsoft Windows. I cannot for the life of me understand people who prefer Linux, let alone people who - urrgh ! - prefer command line "interfaces" to GUIs. These people are why we cannot have nice things. The main purpose of Linux, in my view, is to break your computer and make the problems absolutely incomprehensible so you have no idea what the hell just happened.

But I digress. Anyway, what is WSL and why should you, an astronomer, care ?

WSL is a way of running Linux under Windows almost like any other Windows program. You get a full Linux installation which you can open from a terminal on your Windows desktop. And although the support isn't native, it's entirely possible to get Linux GUIs to run as well. And run properly, in a useful, no messing about way. In short, WSL is ideal for people like me : those who prefer using Windows but need Linux for work.

Of course, there are alternatives. The classic is to dual boot your machine. I managed this once, but my Linux installation was forever unstable and would randomly freeze, making it unusable for any real work. And it's not a terribly convenient solution having to restart your PC every time you want to switch operating systems.

Another option is to create a virtual machine to run Linux from Windows. I haven't tried this, but as far as I can determine the main drawback of this is a performance penalty, and integrating the two systems is difficult.

My experience of WSL has been nothing but positive. It's easy to install (though not as easy as installing a regular Windows program), integrates seamlessly with Windows, and suffers no loss of performance (when I tested it, I actually found my Python code was running slightly faster under WSL than Windows !*). You can have apps running directly on the Windows desktop or inside a window showing a full Linux desktop. Basically it does all the things. Well, very nearly all, as we'll see.

* Okay, fine. You can take that as evidence of Windows as a crappy operating system if you really want to. Happy now ?

By way of a demonstration, then, let me walk you through exactly what I've been able to get WSL to do - and how.

Installation : 1 - The Basics

Although not quite as straightforward as downloading a regular .exe file and running it, I found this process to be simple enough. There's nothing more involved than copy and pasting some commands into a standard Windows command prompt. Specifically, mirroring the official instructions found here, let me go through this step by step.

First, you may as well check if you have a compatible system, so run :

• systeminfo | find "System Type

This will tell you if you have an x64 or ARM64 system. I haven't a clue what this is, but apparently you need to know this for the next step :

• ver

This is simply to check you're running a compatible version of Windows : for x64 systems : version 1903 or higher, with build 18362 or higher; for ARM64 systems : version 2004 or higher, with build 19041 or higher. If it looks good you can get on with things. So now enable WSL as a feature by running : 

• dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

Next, enable the built-in virtual machine feature of Windows :

• dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Now you need to download the core Linux package. If you have x64, use this link. If you have ARM64, use this one instead. Run these like any other Windows program, enabling permissions as necessary.

Next, enable WSL version 2 as your default :

• wsl --set-default-version 2

Finally, download the particular version of Linux you want to use from the Windows Store. This is as easy as installing any other Windows program : click the version and then "get". It'll then take a minute or so to set everything up, and you have to create a new user account for Linux. 

I'm using Debian, because this is the one I'm familiar with. You'll get a little Debian icon you can click on to open a Debian terminal, or you can use the new "Open Linux shell here" option added to the Windows shift+right-click menu (taking you inside Linux directly to the folder from where you clicked). Or, if that's not enough, you can run "wsl" or "debian" from a Windows command prompt or PowerShell to start it that way. While WSL gives itself some dedicated Linux space (all the usual /usr/local etc. directory structures are there), you can access all your regular Windows folder through /mnt/c (or whatever drive letter you use). Just cd into this like a normal directory to access everything. 

Although this process is a bit more elaborate than the usual download->run program approach, it's very forgiving. I made an unrelated error later on than broke my Debian installation, so I just re-ran the whole process and everything worked once again. What's more, this didn't uninstall the Linux programs I'd already set up.

Installation : 2 - GUIs

Congratulations, you now have a fully-functioning Linux installation running under Windows. But to my mind this is absolutely useless unless you're manically obsessed with text adventure games. Come on, any serious work requires a graphical interface, for goodness sake. This isn't 1992.

Getting GUIs to run took a little more digging since this isn't supported natively for some reason, at least not yet. It's a bit more work than getting WSL itself to work, but in the end it's not too difficult. Look, I hate this kind of computing-wizardry crap, and even I managed it. If I can do it, and you're the sort of person reading this blog, then you can do it too.

Essentially what you have to do is use a X-server, just as if you were remotely logging in to a computer on another network. Except this time you're logging in to your own computer, which is a bit weird but it works (and is very much faster than remote logging, which is normally better through VNC).

Normally I use XMing for this stuff. However, on encountering some complications in the process, I decided to follow the various guides step-by-step to ensure I was doing everything 100% correctly. So here I'm using VcXrv. I'm sure it's possible to use XMing instead, with the right settings, but this worked for me.

Running Linux programs on the Windows desktop

Run an X-server

Install VcXrv on Windows like any other program. Let's start with assuming that you'll want to run Linux programs on the Windows desktop - we'll get to running a full Linux desktop later on as that's a bit harder. So run VcXrv and on the first screen set it to use multiple windows. Leave the options on the second screen unchanged. In the third start-up screen, enable the option "disable access control" (I believe that adding a switch "-ac" to the shortcut does this as well). You can then save yourself some considerable hassle by saving the configuration files from the fourth and final start-up screen. This is much more useful than saving you from having to set all the options each time, and we'll see why a bit later on.

If you're very lucky, running the X-server will already be enough : with the X-server running, try launching the built-in "xeyes" program in WSL to test it. If it works, it'll launch instantly. If not, you may get some error message, or it may just do nothing except make the Linux prompt unavailable until you hit ctrl+c to cancel. If that happens, the next two steps should solve this.

Configure Linux to accept the X-server

It took a lot of Google searching but I finally figured out that there were two independent problems impairing my access to Linuxy goodness. First, I had to configure Linux so that it would interface with the X-server correctly. Apparently this is done by running the command (in a WSL prompt) :

• export DISPLAY=XXX.XX.XXX.XX:0; where the correct address can be found in the file /etc/resolv.conf. This is a tiny file so you can read it with the built-in "more" command easily enough.

Try running xeyes again, it might already work. But even if it does, there's a complication here : that address changes. It was horribly annoying to have to adjust this every time I started Linux, but fortunately there's a workaround for that. Before that though, I had to solve the second problem. Despite trying numerous solutions (mostly infinite variations on the export DISPLAY command), I still couldn't get GUIs working. Eventually I found that the problem was - of all things - the firewall. Of course, if you've got the app working already, you can skip the next step.

Tell the Windows firewall that everything's okay

It seems bizarre that you need to alter firewall settings to get GUIs to open from programs running locally, but extensive searching and testing proved to me that yep, this is exactly what you need to do. As soon as I disabled my firewall (it only takes a few seconds to test the xeyes program, so the risk is probably minimal) I could launch Linux GUIs no problem. Of course, disabling the firewall to run Linux is Seriously Bloody Stupid, maybe even Darwin Award Stupid, for ordinary use, but all you really need to do is enable a port. I'm using McAfee, and I had to do the following :

• In McAfee go to "PC Security", "Firewall", "Ports and System Services" and then add a new one. The name and description etc. are just convenient labels in case you need to refer back to them in the future. The important thing is to enter "6000" in the "Local TCP/IP Ports" section and make sure "Forward port activity" is ENABLED.

That's it. There should be no need to restart the PC, though you'll probably need to close the X-server and Linux (if they're still running) and reload them. So do that, try xeyes again, and with any luck it should now work perfectly.

Adjust your Linux .bashrc file so we never need to do any of this ungodly horror again

This is nice, but still leaves us with the need to open an X-server and find the address we need for the export DISPLAY command every time we start Linux. This is... unpleasant. Fortunately, there's a way to automatically generate the full command :

• export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0

Isn't just Linux just lovely and so marvellously intuitive ? Hell, no of course not. But we need never bother with this ugly thing in practise - or with manually starting the X-server either. In fact one of the really neat tricks about WSL is that we can start Windows programs from within Linux. So all we need to do is set up an alias that can configure Linux and launch the X-server in Windows all at once. That's where saving the X-server configuration file comes in handy. In Linux, I edited my .bashrc file to include the following :

• export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0
• echo "For GUI functionality either run 'xapps' (apps only) or 'xdesktop' (full LXDE desktop)"
• alias xapps="'/mnt/c/Program Files/VcXsrv/xlaunch.exe' -run 'C:My\Directory\AppsOnly.xlaunch'"

EDIT : On Ubuntu, it may be necessary to slightly alter the last command :

• alias xapps="'/mnt/c/Program Files/VcXsrv/xlaunch.exe' -run 'C:My\Directory\AppsOnly.xlaunch' &> /dev/null &"

Without this "&> /dev/null &" on the end, the .bashrc file may (for some silly reason) not process the alias correctly, so it will run with no errors but not actually do anything. This modified command causes no problems on Debian, so if in doubt use this version.

The first step is exactly as before, to configure the Linux display for the X-server. The third generates a command "xapps". Typing this in my Linux console automatically runs the X-server using the configuration file I saved earlier (Note : if you move this file, the command will break and you'll have to edit the new path in the .bashrc file. This is worth remembering ! I did this once and completely forgot about all this setup, so for quite a while I was dumbfounded as to why it had suddenly stopped working). The second line just prints a helpful reminder to myself, every time I start a new Linux session, about how to run this very important command.

Running a full Linux desktop under Windows

You may have noticed that my reminder statement in the last step includes a command to launch a full Linux desktop. If you don't want this, if you're happy enough with Linux programs by themselves, you don't need to bother with the alias. Just include the launching command in the .bashrc file to have the whole thing run automatically on starting Linux.

But if you do want a full desktop as well, you'll need a separate X-server configuration file. The only difference is that this one should use "one big window" (or probably better "fullscreen") instead of multiple windows. You may as well create that now. I think you also need to set the display number to zero. You'll also want to add one other line to your .basrhc file. After the export DISPLAY command :

• export LIBGL_ALWAYS_INDIRECT=1

Unfortunately there's more to this than just altering the X-server. Since GUIs aren't officially supported in WSL yet, it doesn't come with a desktop facility - you'll have to install this yourself.

To do this, it's worth running some commands on Linux to ensure it can find all the latest, correct packages :

• sudo apt update
• sudo apt upgrade
• sudo apt-get update -y

Which might take a few minutes. You can then go and get yourself a desktop. For this I used LXDE. This is easily installed on Linux :

• sudo apt-get install lxde

And then you run it with "startlxde". With your X-server running, you should get a shiny Linux desktop in one big Windows window. It's a little finicky about whether the "one big window" mode really means full screen or hides the taskbar because the window extends beyond the edge of the screen, but it should basically work (fullscreen mode seems more robust to me; clicking on the main window header bar also helps). What you don't want to do is have the two different X-servers running simultaneously. You can either have Linux programs or a Linux desktop, not both, at least not that I've been able to figure out.

And then set yourself a second alias, much like before :

• alias xdesktop="'/mnt/c/Program Files/VcXsrv/xlaunch.exe' -run 'C:\My\Directory\FullDesktop.xlaunch' && startlxde"

That's pretty much it for WSL. It's some work to set up, but when it's done, it's triviality itself. Each time I open a Linux terminal I see the commands to run the GUIs so I never forget, and all I have to do is type them and I'm basically using full Linux under Windows. The actual number of commands you need to do to get everything to work is small - it was figuring out what those should be that's the time consuming bit. Hopefully, with this guide, that should now be ~30 minutes work or less.

What I cannot remember is if I had to configure the GUI in the Linux desktop or not, i.e. because I have a very particular preference for the workspace manager. Also, on startup I get an apparently useless error message about "No session of PID 384", which some Google searching has demonstrated is more difficult to remove than it's worth, so I haven't. 

Finally, I haven't fully figured out how to have the full right-click option on the Linux desktop. The best I've found is :

• Preferences -> Desktop Preferences -> Advanced -> DISABLE "show menus provided by window manager when desktop is clicked"

Which gives me something, though it's limited : I can open a terminal on a selected folder but not just in the current directory. Weird, but despite the limitation this level of functionality is more than sufficient to use it in anger.

EDIT : This all works fine for me on Debian, but Ubuntu may have problems using the .bashrc file. A short guide to fixing this is here.

Astronomy in WSL (these instructions should also work for standard Linux)

There are three tools I just can't do without : miriad, ds9, and kvis. Miriad I use for spectral analysis of HI data cubes (being from an earlier era, it's outstanding at handling even files which are tens of gigabytes in size). Ds9 is a FITS viewer that's trivial to install on Windows, but its "funtools" plugin for optical photometry is more difficult. Kvis is part of the karma package* and is an excellent, lightweight FITS viewer explicitly designed for 3D cubes. I also sometimes use IRAF, Gaia, GILDAS, and various others, but very much less frequently than the big three.

* Not the ESO one.

You might well wonder why I use other FITS viewers given that I've been developing my own since 2012. Two reasons. First, kvis is fast, ultra-well-tested, and is good for making publication-quality plots. It's not exactly feature-rich (in fact, for analysis it's downright dismal), but it more than makes up for that in speed, reliability, and like miriad it's great at handling very large data sets (though not at the tens of gigabytes level). Ds9 for me is all about the aperture photometry, which is just amazingly simple to use. 

Second, while I rely heavily on FRELLED/Blender, this is the only program I've so far been unable to run in WSL. As far as I can gather this is a fundamental limitation of WSL's handling of OpenGL, which I expect will be fixed in due course. But even then I'll still be using kvis : for pure visual inspection, so long as nothing particular complex is required, it beats everything else hands down. It's a great shame development was abandoned.

For now, having access to these tools is such a godsend that I'm still getting used to it. I don't have to put up with Linux's crap any more, I can just get on with stuff in nice, friendly Windows. So let's go through the installation of ds9/funtools, miriad, and karma. Most of the other basic Linux apps tend to be no more complex than the standard sudo apt-get install, so I won't cover them. These more specialist apps require certain compilers and libraries, so it's useful to document these steps that everyone does once, instantly forgets about, and then struggles to remember what the hell they did the next time they try installing stuff.

DS9 with funtools

The main ds9 program is just a simple binary file : download it and create an alias in your .bashrc file to run it however you like. The "funtools" (FITS Users Need Tools) plugin is a bit more involved. There's a "funtools.ds9" file, but I'm not sure this does anything by itself. Unfortunately you have to compile funtools from source. Following these instructions, here's what I did.

First create configuration files :

• sudo ./mkconfigure

These gave me an error "autoconf not installed" but this was easily fixed with :

• sudo apt-get install autoconf

I re-ran the mkconfigure command, which now worked. So then I tried to set the configuration to use a target installation directory :

• sudo ./configure --prefix=[installdir]

I think for [installdir] I used. /usr/local/ds9/funtools. But this gave an error, "configure: error: no acceptable C compiler found in $PATH". This was fixed with :

• sudo apt-get install build-essential

After re-running the configure command, all I had to do was :

• sudo make
• sudo make install

And that's it : funtools was automatically added to the ds9 Analysis menu, with no need to use the .ds9 file at all.

Well... not quite true. Funtools is actually using a different funtools.ds9 file than was compiled. This means it gets a bit finicky about where the "funtools-master" directory is located. I put it in ~/ProgramFiles (it wasn't happy if I put it somewhere more sensible like /usr/local/ds9/funtools). Then in ds9 itself, I set Edit->Preferences->Analysis to have this file autoload on startup. Et voila, aperture photometry on "Windows" :

Karma / kvis

The main problem with installing karma is not that it's difficult, it's that it's sufficiently obscure that finding installation instructions that actually work is a bugger. The main result keeps coming up with this page, which describes how you can use a single rsync command and everything will be as lovely as riding a magical unicorn. Unfortunately this seems to be out of date, since the files are no longer hosted where they're supposed to be. The unicorn has gone missing.

Eventually, I found these instructions which do actually work, which I'll mirror here. It's actually pretty simple, since karma is basically a series of pre-compiled binaries. Basically you download a couple of packages and make some minor adjustments to the .bashrc file.

First you'll need some command-line way to access the files via url. I use wget :

• sudo apt-get install wget

Then created a directory "karmafiles" where I could download and unpack everything (this is just a temporary holding area, the real installation directory comes later). From there :

• wget ftp://ftp.atnf.csiro.au/pub/software/karma/karma-1.7.25-common.tar.bz2
• wget ftp://ftp.atnf.csiro.au/pub/software/karma/karma-1.7.25-amd64_Linux_libc6.3.tar.bz2

Those directories also contain manuals, though those are optional. The main files need to be unpacked into the temporary directory :

• tar -xvf karma-1.7.25-amd64_Linux_libc6.3.tar.bz2
• tar -xvf karma-1.7.25-common.tar.bz2

But Linux complained that bzip2 was not found, though this was easily fixed with :

• sudo apt-get install bzip2 

Now the tar commands should work. They unpack the files to a single common directory "karma-1.7.25" (how they do this I know not, but they do). I moved this to /usr/local/karma : 

• sudo mv karma-1.7.25 /usr/local/karma/

Finally there are a few lines to add to  ~/.bashrc :

• LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/karma/amd64_Linux_libc6.3/lib/
• export LD_LIBRARY_PATH
• export KARMABASE=”/usr/local/karma/amd64_Linux_libc6.3″
• alias kvis="/usr/local/karma/amd64_Linux_libc6.3/bin/./kvis"
• alias kshell="/usr/local/karma/amd64_Linux_libc6.3/bin/./kshell"
• alias kpvslice="/usr/local/karma/amd64_Linux_libc6.3/bin/./kpvslice"

I just put them all at the end, with a comment to indicate that this was stuff for karma.

And that's it. Kvis runs just fine. It even supports its feature of moving the pointer via the cursor keys and freezing the coordinates with the space bar, so long as NumLock is off. It sometimes has a display issue with some of the buttons not showing up correctly, but I think this only happens when I forget to close an old X-server.

Miriad

This is basically the same as for karma. The main installation guide is here, and again I'll mirror those instructions for the sake of my own convenience.

First download the two packages, one common and one OS-specific :

• wget ftp://ftp.atnf.csiro.au/pub/software/miriad/miriad-common.tar.bz2
• wget ftp://ftp.atnf.csiro.au/pub/software/miriad/miriad-linux64.tar.bz2

Unpack them into a holding directory (I called this "miriadfiles") :

• tar -xvf miriad-common.tar.bz2
• tar -xvf miriad-linux64.tar.bz2

Move the common directory this creates to its proper installation location, /usr/local/miriad : 

• sudo mv miriad /usr/local/miriad/

Then we do some procedure to replace the directory in the miriad initialisation file :

• MIR=/usr/local/miriad; export MIR
• cd $MIR
• sed -e "s,@MIRROOT@,$MIR," scripts/MIRRC.in > MIRRC
• sed -e "s,@MIRROOT@,$MIR," scripts/MIRRC.sh.in > MIRRC.sh
• chmod 644 MIRRC*

Apparently this changes the path in the file $MIR/scripts/MIRRC.in from the default "@MIRROOT@" to the correct path, e.g. /usr/local/miriad, and saves the result in the file MIRRC (similar for MIRRC.sh.in). I presume you could also do this manually if you wanted to.

Finally we make some additions to the .bashrc file so we can start miriad anywhere, which is a bit different to simply setting an alias :

• source /usr/local/miriad/MIRRC.sh
• export PATH=/usr/local/miriad/linux/bin:$PATH
• export PATH=$MIRARCHD/bin:$PATH

And that's it ! It just works.

The only - very minor - downside is that we cannot simply run miriad by doing "wsl miriad" from Windows, even though other Linux commands are supposed to work like this. This is likely a minor limitation of WSL for now. Apart from that, miriad appears to be fully-functional. No need to muck about with the notoriously awful PGPLOT library or anything daft like that.

So that's it. Today I managed to (a) insult Linux fans and (b) finally document those annoying installation procedures I've been meaning to document for bloody years. Time to celebrate with a nice cup of tea.