OpenWRT, Two GL.Inet Routers, and Tailscale: Successes and Failures

| Comments

This blog isn’t going to lead to some keen piece of insight or an interesting conclusion. I’ve been messing around with a pair of very different OpenWRT routers from GL.iNet: the GL-MT300N-V2 Mango and the GL-AXT1800 Slate AX.

I learned some things I didn’t know about last week. I learned just a little more about Tailscale. This is mostly just to write down what has worked and what hasn’t worked for me.

GL.iNet Mango and Slate AX 1800

I figured I should write some of these things down both for future me and for anyone else who might be trying to accomplish something similar. Mostly for future me.

How did we get here?

A friend of ours is moving to Ireland. He just wants to watch Jeopardy on his Hulu Live TV subscription. He’s been watching Brian and I monkey with Tailscale on our GL.iNet Mango routers for ages. I know our friend has had his Mango for a while, quite possibly for about as long as Brian and I have had our Mangos, but now he’s been trying to use it to connect his Apple TV to an exit node in the United States.

The story is more complicated than this, but the Mango didn’t work out for him, so he tried the much beefier GL.iNet Slate AX, and he couldn’t make that work. That’s how I wound up having a Slate AX here in my possession. He is currently using a Raspberry Pi 4 with Tailscale as a router to forward his Apple TV to America.

I am able to get my tiny Mango to route traffic of connected devices through one of my Tailscale exit nodes. I am not able to do the same with the much nicer Slate AX router.

Who is Pat blaming?!

I don’t think anyone needs to accept the blame here. I don’t believe that running Tailscale on an OpenWRT router isn’t officially supported. Running Tailscale on the Mango is a bit of a hack because the Mango doesn’t have enough storage to even hold Tailscale.

Not only that, but as I learned long after I started trying to figure this out, the GL.iNet WiFi6 routers aren’t even built upon official upstream OpenWRT!

We can probably blame me for not doing science correctly. I am also only a sample size of one, so just because something is working well for me several times on the day I happen to test it does not always mean it works smoothly next week.

Keeping track of what is going on is problematic. The logs on the OpenWRT routers are ephemeral. Not only that, but I didn’t think I would need to troubleshoot anything on my Mango, so my startup scripts redirect all tailscaled output to /dev/null!

Pat is bad at science

I did a bad job writing things down. I didn’t know I was going to have to. All I really managed to do was write down some things after the fact on our Discord server. Those were usually the weirdest happenings.

We also aren’t helped by the fact that not everything happened at the same time. I updated Tailscale and OpenWRT on my Mango a few weeks ago, and everything there was fine. Then, when I tried booting up the router a few days later, things weren’t going as smoothly there as I thought. Then even more time passed before I got to fart around with the beefier WiFi 6 router.

Science is hard. I was kind of expecting to make things work, wipe everything clean, and make it work again. That last step would have given me all the documentation I needed, but I never made things work on the beefy router.

Connecting my GL.iNet Mango to a Tailscale exit node

First of all, I am cheating when I load Tailscale on my tiny Mango. It only has 16 megabytes of flash, and most of that is taken up by GL.iNet’s OpenWRT firmware. The official Tailscale binaries are bigger than that, so I wound up installing them on a USB flash drive.

It was a rather manual process, and I documented it as best as I could. I had to make a few tweaks in the OpenWRT LuCI GUI to get exit node traffic routing correctly, but once I did, I thought I had it working pretty well. I powered the router off and on several times, watched it succeed every time time, then I packed the Mango away in my laptop bag to use in an emergency.

The Mango is pretty slow. It can only encrypt traffic over my Tailscale link at about 4 megabits per second. That’s way slower than Wireguard in the kernel on this machine, and probably only just barely enough for video streaming. I am guessing Netflix would bump you down to standard definition via a Tailscale exit node on the Mango.

A few weeks later when my friend was having trouble, so I pulled my Mango out of the bag and plugged it in. My Mango didn’t want to connect to my Tailnet unless I killed and restarted tailscaled. Adding a long sleep to my Tailscale script seems to help with this, but it isn’t perfect.

I should also note that I only managed to get the Mango to route traffic through my exit node if the Mango is using Ethernet for its WAN connection. If I set up the Mango to use WiFi as its WAN, it won’t route traffic via the exit node.

I didn’t notice this right away, and I haven’t investigated what is going on here. This is only a problem when trying to route traffic through an exit node. The Mango works fine for me with WiFi WAN as long as it is just a regular node or subnet router.

NOTE: On the Mango, I have to create an interface in LuCI for the tailscale0 interface and make sure it is attached to the WAN firewall rules. This coaxes OpenWRT’s firewall scripting to apply the correct iptables rules for packets to flow.

The GL.iNet Slate AX is built on Qualcomm’s OpenWRT fork

Everything that went wrong for me when attempting to route the GL-AXT1800 through a Tailscale exit node was completely different than on the Mango. I assumed something was just a little different between the newer release of OpenWRT that this router was built on, so I investigated the idea of installing a different version.

I couldn’t downgrade the AXT-1800 to match the Mango. This made sense to me. Why would GL.iNet start crafting their build for a new device with an older version of OpenWRT for a new piece of hardware?

I also noticed that I couldn’t upgrade the Mango to match the Slate. I knew there were official OpenWRT builds for the Mango, so I checked the OpenWRT site for firmware for the Slate, but I couldn’t find it. There wasn’t any open-source firmware for any of GL.iNet’s WiFi6 devices.

I thought I read that there wasn’t yet any OpenWRT support for WiFi6 devices

At least I thought I read this, but GL.iNet sells a few OpenWRT routers with WiFi6 chips. How can that be?! We just figured it out, and GL.iNet doesn’t hide what they’re doing. They spell it out right in the product description.

Qualcomm fork of OpenWRT

OpenWRT doesn’t yet have much support for WiFi6 devices, but Qualcomm has a fork of OpenWRT that supports their latest WiFi6 chipsets.

I haven’t decided precisely how I feel about this, but it is definitely making my troubleshooting more difficult. That is assuming you’re willing to call my ham-fisted attempts at messing about here troubleshooting!

I understand that I don’t truly understand how Tailscale works

I understand well enough how packets get from a Tailscale device at my house to a Tailscale device at another location. What I don’t understand is how packets on my Linux machines find their way into the Tailscale process, but I do understand that Tailscale has its own little IP stack hiding in there.

There are no entries in my routing tables listing any of my Tailscale addresses. They should all be matched by my default gateway, yet they manage to get snagged by Tailscale and routed appropriately.

What works for me with Tailscale on the GL.iNet GL-AXT1800?

I installed the Tailscale OpenWRT package and immediately noticed that it is built with an old enough version of Tailscale that it doesn’t support exit nodes. I cheated and replaced the package’s binaries with the latest Tailscale binaries. It fires right up, connects to my Tailnet, and I can connect to things.

I figured cheating was the right thing to do. It seemed smart to let the OpenWRT Tailscale package install startup scripts and maybe even LuCI GUI configuration bits and pieces for me, and I could just replace the outdated binaries.

What happens when I try to route traffic through an exit node?

When I run the command to route traffic through an exit node, things get weird.

As soon as the exit node is enabled, I can no longer reach the exit node’s IP address, but I can reach other nodes just fine. At least I thought I could reach other nodes just fine.

One time I was accidentally pinging the wrong node in the background while enabling the exit node. I got really excited when it continued to respond, but then I noticed that the round-trip time increased quite a bit. A few seconds later I noticed that I was pinging the wrong address.

You can pass Tailscale the --netfilter-mode=off option to prevent Tailscale from creating any firewall rules. This gave me the same results.

What is the solution?

Our friend’s solution is Tailscale on a Raspberry Pi. That’s a fine solution, but I did something that made me feel a bit dirty. I set up a Wireguard server in a Docker container.

I followed somebody’s guide. I don’t think it was this exact guide, but it was similar. I was disappointed when I saw that I had to hit my page-down key twelve times to get to the end of the documentation. This is an order of magnitude more work that setting up Tailscale and a handful of usable exit nodes.

Not only was it long, but the documentation didn’t work for me. I had to install the wireguard-dkms package on the Docker machine. This makes perfect sense, but it took longer than I’d like to admit for me to figure out that I needed Wireguard support in my host kernel.

The good news is that GL.iNet’s awesome Domino interface makes it extremely easy to connect to a Wireguard or OpenVPN server.

The Domino GUI on the Slate AX is fancier than the tiny Mango!

These are very different travel routers. The Mango cost me $20 two years ago, only has 2.4 GHz WiFi, only a pair of 10/100 Ethernet ports, and it happily boots up when plugged into a 0.5 amp USB hub.

The Slate AX has a beefy heat sink with a little fan. It comes with a 5-volt 4-amp power supply. It has near bleeding-edge WiFi speeds. It is a dense beast of a little machine, and it costs nearly six times more than the Mango. I am not surprised that there are significantly more options in the stock Domino interface.

Their Domino GUI is a really nice feature on both routers. Either will let you do things like connect to WiFi or tether your phone to use as your WAN. You could do this with OpenWRT and LuCI, but you’d have to click dozens of buttons and change so many settings, and you better not mess any of it up!

The Domino GUI lets you do this with just a few clicks. I’m not excited about having this at home, but it is extremely handy on a travel router.

The Slate AX has options to allow you to use Ethernet, WiFi, and a tethered cell phone as a weighted multiport WAN. I haven’t had a chance to test this, but I am impressed that the option is available, and it looks easy to configure. You can do this sort of thing with stock OpenWRT, but you will have to work very hard to do it!

Conclusion

It is a bummer that we couldn’t get a GL-AXT1800 routing through a Tailscale exit node, but I am mostly only bummed out about it because we couldn’t get our friend watching Jeopardy through an OpenWRT router with Tailscale. The things that we are currently able to do with Tailscale on an OpenWRT router are still quite impressive to me.

When I bought the Mango two years ago, loading Tailscale on it was an afterthought, and exit nodes didn’t even exist yet. I just thought it was neat that I could leave my Mango behind and still be able to ssh to it. That option alone has a ton of value to me, and there are dozens of times when something like this would have come in handy over the last twenty years.

Two years later, and my Mango can route traffic through an exit node or route outside traffic to its own local subnet. I am hopeful that things will improve over time from every angle. GL.iNet seems to really want to get the IPQ6000 support ported to upstream OpenWRT. OpenWRT should eventually have a more recent version of Tailscale in their package repositories. And of course Tailscale is constantly improving.

This conclusion has gotten away from me. What I am trying to say is that it sure seems like everyone involved is probably doing a good job.

Comments