Tailscale is Awesome!

| Comments

I noticed a problem last week. Tailscale is at least a small part of nearly everything that runs on my own private network, so it gets mentioned in the vast majority of the blogs I write. I also tend to write a post every time Tailscale drops a new feature and I start using or relying on those new features, but those write-ups are rather specific. I don’t have anything recent to link to that talks about what I actually use Tailscale for on my personal network.

This is the post that I need to write to address this!

What on Earth is Tailscale?

If I just say that Tailscale is a mesh VPN, I feel like everyone should have a pretty good idea what I am saying, but the world has gotten weird. When I first starting using them twenty-something years ago, a VPN was the magical thing that would put your Internet-connected computer on the other side of your company’s firewall, or it might be used to connect two physical locations using an Internet connection.

hacker2

Marketing has made the vast majority of people now think the services that companies like NordVPN are selling is the traditional use of a VPN, even though what these companies are doing is just one of the many things you can do with a VPN connection.

Yeah, yeah. What in the heck Tailscale?

You can shoehorn Tailscale into doing a lot of useful things, but how about we just talk about the basics? The idea is that you install Tailscale on all of your laptops, phones, and servers. Tailscale will work its magic and set up a direct Wireguard VPN connection between each of your devices. As long as they have a decent Internet connection, each of your devices will be able to make a direct network connection to any other device on your personal Tailnet.

You can be on your phone at Starbucks, connect to a test web server that is running on your laptop at home, and that laptop will be able to access a database server that lives at your office. Everything just works.

The best part is how easy it is to get up and running. You could probably already have signed up for a Tailscale account and had two or three machines connected to your new Tailnet in the time it took you to read up to this point. It is that easy to get going.

Tailscale is awesome because you don’t have to jump into the deep end

Tailscale is probably the easiest VPN to set up, and it is really easy to install it on two or three computers just to solve a single problem. Dipping your toe in is a great way to get started!

You can install Tailscale on your phone and your Home Assistant server, and you will immediately be able to access your home automation dashboard from anywhere in the world.

Maybe you need remote access to your NAS. Just install Tailscale on your laptop and your NAS, and you’ll be connected in no time.

If you keep adding one or two machines at a time to your Tailnet to solve small problems, it won’t be long before almost all your computers are part of your Tailnet.

This is definitely how I became so reliant on Tailscale. I started with just a handful of nodes on my Tailnet, and before I knew it, every device I own was on my Tailnet, and at this point almost every network service that I run is only accessible via my Tailnet. Say Tailnet again. Tailnet.

What is Pat doing with Tailscale?

I am going to expand on this, but I think it is best to start with a list:

Some of these things are simple enough that they don’t require their own heading.

Remote access to Home Assistant is kind of basic. The Home Assistant app on my Android phone can always communicate with my Home Assistant server no matter where I am, and I didn’t have to expose Home Assistant to the Internet. I can click a button to warm up my espresso machine when I am driving home, and that is awesome.

The Raspberry Pi-KVM is really cool. It is a do-it-yourself networked keyboard, video, and mouse device. You connect the Pi to a network and plug the appropriate HDMI and USB cables into a server. You can then use a web browser to see everything on that server’s HDMI port and type as though you are in the room with the server.

Tailscale means I can easily leave the Pi-KVM behind at my parents’ house and troubleshoot their problems from the comfort of my desk at home.

I am not using Tailnet Lock

Tailscale has a potential security problem. It is possible for Tailscale to add a new node to your private Tailnet. Up until a few months ago, you just had to trust that the folks at Tailscale wouldn’t add some sort of nefarious network device to your Tailnet.

Tailnet Lock closes that hole. When you enable the lock, you have to manually approve new nodes on your Tailnet. It seems to be implemented in a secure way. You can read more about Tailnet Lock in Tailscale’s documentation.

Tailscale Lock is not yet compatible with node sharing. I rely very heavily on node sharing, so I couldn’t enable Tailscale Lock today even if I wanted to.

I haven’t decided whether I even want to enable it. I don’t think I feel paranoid enough to worry about someone at Tailscale adding a node to my Tailnet.

Self-hosted Seafile has saved me $700 so far!

I wouldn’t have self-hosted Seafile again if I had to run the server on the public Internet. Tailscale lets me access my Seafile server from anywhere in the world, and my little Raspberry Pi server doesn’t accept any connections from the local network. It is only accessible via my Tailscale network.

I don’t have to worry so much about old, broken, exploitable services when they are running on my own personal Tailnet. The entire Internet can’t hammer away at the nginx server in front of my Seafile server looking for a hole. I don’t have to scramble around one morning when I discover that there is a patch for a zero-day exploit against Seafile. I can just remain happy knowing that only five computers in the world can see my Seafile server.

Seafile is syncing a little over six terabytes of data between my workstation, my NAS, and my laptop. The Seafile server also stores a copy of all that data with 90 days of history.

Google charges $100 per year for two terabytes of cloud sync storage. Dropbox charged $120 for the same. I can’t say that I have done the math perfectly here, because I haven’t done a good job of tracking when I actually cross each two-terabyte threshold. I do know that I would be paying $400 to Google or $480 to Dropbox this month if I weren’t hosting my own Seafile server.

I have been slowly spending those savings on local storage. The first $300 went to the Seafile Pi and its 14 TB hard drive. Another $200 or so went to a 12-terabyte drive for my workstation, and last month I added a 14-terabyte drive to my homelab server. If I don’t need to replace any failed hardware, the $400 I save each of the next three or four years will go right into my pocket!

Node sharing is Tailscale’s killer feature

We need to share large video files to produce our various video content. I usually have to send a 20- to 30-gigabyte video file to Jeremy so he can finish up the work on the Create/Invent Podcast, and Brian has to send me about 40 gigabytes of video when we record The Butter, What?! Show. They both have accounts on my Seafile server, and I have shared the server to their Tailnets.

Why is this such a killer feature? If I were doing this the old-fashioned way, I could have sent them Wireguard keys!

With Tailscale, I don’t need to maintain anything. I don’t need to generate keys. I don’t need to ship those keys to Brian or Jeremy. They just have to log in to Tailscale, and Tailscale manages all of that for me.

Brian already had his own Tailnet, but Jeremy was new to Tailscale. I sent him the sharing link. That sent him to the Tailscale website, where he was able to create an account, and he was taken straight from there to the download page. He was able to ping my Seafile server a few minutes later. All I had to do was send him a URL.

If Brian or Jeremy can’t log in to Tailscale, they aren’t going to have to call me.

Geography-agnostic servers are pretty cool!

I made a mistake and created a problem with my Seafile Pi when I as upgrading all my machines to use Tailscale SSH. It was my fault, but I could no longer connect via SSH, and I needed to sit down at the console or pop the boot drive out. I was at Brian Moses’s house one Saturday night for pizza, so I just made sure to bring the Pi home with me.

Seafile was down while I drove, but I plugged the Raspberry Pi in when I got home, so nobody noticed that anything happened. If Brian didn’t see me leave the house with the server, he wouldn’t have noticed that it was in a new location.

I fixed my Tailscale SSH problem, and I 3D-printed a new case with a slot for an OoberLights board. I think the Seafile Pi was on my desk for two weeks before I took it back to Brian’s house.

I think this is so cool. I can move a server to a new location for troubleshooting, and everything that connects to that server will still be able to connect. No configuration changes need to be made anywhere. Everything just works!

Tailscale SSH has been a nice upgrade!

Tailscale keeps adding new features. They are almost always useful features. They tend to be the sort of features that if you’re already using Tailscale, then there isn’t much excuse not to turn them on and try them out. Tailscale SSH is one of those features.

I have an SSH private key on my workstation. I have an SSH private key on my laptop. It is my job to make sure I install their matching public keys on every device I need to connect to via SSH. Then I have to remember to generate new keys on some sort of regular basis.

I always fail at the last part. One of the keys I was using last year was generated in 2013.

Tailscale is already generating a private key on every node to establish secure Wireguard connections. Why not let Tailscale handle all my key management for me? I can use Tailscale’s ACLs to control which machines can connect to other machines. My desktop and laptop can connect anywhere, my internal servers can sometimes connect to each other, but my public servers are never allowed to SSH anywhere.

It is unfortunate, but I still need to maintain SSH keys. I can’t use Tailscale SSH to authenticate to Gitlab, Github, or even Tailscale nodes that have been shared with me. Even so, cutting the number of places where I manage public keys down to four from dozens and dozens is a huge win!

Tailscale’s ACLs

There is a good chance that you won’t need to use Tailscale’s ACLs. I think I was running Tailscale for almost two years before I even considered [setting up any access controls][tacls].

I decided it was time to tag all my nodes and configure ACLs when I added my first vulnerable node to my Tailscale network. I migrated the web server that runs the nginx server for our blogs over to a cheaper Digital Ocean droplet, and at the same time I added Tailscale to the server.

This server is answering requests over the Internet. Anyone sitting anywhere in the world could be banging on this server trying to get a shell. I don’t want them to have a point from which to attack the rest of my Tailnet if they succeed. I have fragile things like Samba servers hiding behind Tailscale!

These are my tags:

  • workstation (machines where I sit at the keyboard)
  • shared (via node sharing)
  • server-ts (servers only accessible via Tailscale)
  • server-dmz (servers accessible via Tailscale or LAN)
  • server-external (servers on the public Internet)

The workstation tag can connect to anything. The server-ts tag can connect to other any of the three server tags. The server-dmz can connect to server-dmz or server-external, and the server-external just can’t connect to anything. The devices that have more exposure can never connect to safer devices.

Tailscale Funnel is solving real problems!

I wanted to yell something like, “I don’t know what I would do without Tailscale’s fun tunnels!” but that would be an exaggeration, if not a total lie. We muddled along just fine without Funnels. Our continuous blog deployments used to have a two- or three-minute delay, but with a Funnel, they happen within a couple of seconds of a new post being pushed to Gitlab.

What is a fun tunnel? If you have a service running on your private network, and you need to expose that service to the public Internet, then you can use a Funnel.

I added a webhook server to our little development server, and I configured Tailscale to point a Funnel from butterwhat.humpback-rooster.ts.net to our development server. Now a Gitlab action can let our development server know that a new commit is available, and it can publish our changes immediately.

Funnels are currently in alpha and have unpublished bandwidth limits. The traffic has to run through Tailscale’s relays, so the bandwidth is much lower than with a direct Tailscale connection. You probably don’t want to stream video over Plex or Jellyfin with a Funnel, but it is probably alright to host a low-traffic web server.

I would prefer to not need a Funnel at all!

My only Funnel is a consequence of not yet having Tailscale when we set up a Git repository so Brian Moses and I could collaborate on Butter, What?! blogs. We needed a repository we could both access, so I set one up for us on Gitlab.

I already had a virtual machine running on my homelab server that was handling Octopress and Jekyll blogs, so it made sense to me to just add one more blog to the mix. That server had a cron job that attempts to pull changes from Gitlab every few minutes, and if there were changes, they would be published to the Internet.

I improved the situation with a Funnel just because I could. The better solution to the problem would be removing Gitlab from the process. I shared the virtual machine with Brian via Tailscale a long time ago. Why aren’t we just pushing our updates directly?

Tailscale has a proxy server and handles Let’s Encrypt certificates for you!

Tailscale makes it easy to create TLS certificates for your devices. They have had this feature for a while, and their Funnel feature wouldn’t function without it.

I am way more excited about the proxy server that landed alongside Funnels. Not only does the proxy work with your Funnels, but it also works just fine on your Tailnet. You can ask Tailscale to create a Let’s Encrypt certificate for you, then have Tailscale proxy connections to an HTTP or HTTPS server for you.

1
2

root@octoprint:~# tailscale cert octoprint.humpback-rooster.ts.net
root@octoprint:~# tailscale serve / proxy 5000

There is no extra software to install. I don’t have to figure out which web server Octoprint uses. I don’t have to ask Google how to install a certificate on that particular web server. I had a proper HTTPS connection to my Octoprint server in less than a minute, and it was awesome.

Tailscale exit nodes might be a replacement for NordVPN and friends

Tailscale lets you designate machines on your Tailnet as exit nodes. You can configure an Android phone to be an exit node, and you can even share your exit nodes with your friends.

What is an exit node? It is a device that you can route all your Internet traffic through. If you’re sitting in a coffee shop on some sketchy WiFi, you can click a button and have Tailscale force off of your traffic through one of your exit nodes. The coffee shop will only be able to see encrypted packets flowing past while your real unencrypted traffic exits via a computer at your house.

I have an exit node in my house, at Brian Moses’s house, and on a Digital Ocean droplet in New York.

Services like NordVPN claim to make you completely anonymous. I can’t speak to how truthful they are about this, but I can definitely tell you that you will not be completely anonymous when using Tailscale exit nodes. Maybe you would be somewhat anonymous if you set up a Tailscale exit node on an old Android phone, connect it to the Starbucks WiFi, then hide the phone somewhere near Starbucks. There is a flaw to this plan, but I am not sure it is a bigger flaw than trusting NordVPN.

You will definitely be hiding the content of your network traffic from the coffee shop, but whatever servers you are connecting to will see the IP address of your exit node. If you’re doing something nefarious, someone can link you to your exit node’s Comcast account.

I can use an exit node while I am on sketchy WiFi, but how often does that happen these days when our phones are fast WiFi hotspots? I am way more likely to use exit nodes to test website functionality from a remote location.

Tailscale on a travel router is neat, but isn’t quite ready

I have a really inexpensive OpenWRT travel router in my laptop bag. It is a GL.iNet Mango that I bought for about $20. They go on sale quite often. I had to shoehorn Tailscale on there because the Mango’s flash storage is ridiculously tiny.

All the recent models from GL.iNet have plenty of storage for Tailscale, and the latest beta release of their custom OpenWRT interface has Tailscale and Zerotier right in the GUI. You can get either up and running with just a few clicks!

The cheapest GL.iNet router that can run this beta firmware seems to be the GL.iNet GL-SFT1200 Opal travel router.

Why would I want the cheapest travel router? I enjoy the idea of having a tiny, USB-powered network device that I can leave behind. The cheaper it is, the less concerned I will be about not being able to recover the device!

I can leave it behind at my parents’ house so I can connect to their network to troubleshoot a problem. I can get permission to leave it at a customer’s site to remotely connect to some of their gear. I am sure you can come up with some use cases, both legitimate and nefarious!

Several of us on our Discord server would really like to be able to connect a Fire TV or Android TV to a travel router, then have the travel router pass all the packets through a Tailscale exit node. We haven’t had a ton of luck. I’ve managed to mostly make it work on the ancient version of OpenWRT that ships on the Mango, but the Mango can only pass VPN traffic at about four megabits per second. I haven’t had any success with newer firmware releases on nicer routers.

Conclusion

I knew that I was using Tailscale for a lot of small yet important things, and when I sat down to write this blog, I knew I would have to write a few paragraphs about each of those things. What I didn’t know was that I would be going way past 3,000 words! Does that mean I should keep the closing paragraphs brief?

Are you curious about Tailscale? Are you having trouble understanding what exactly Tailscale is or what it can do for you? Don’t worry about that. It should only take you a few minutes to get Tailscale up and running. I expect you’ll have a better idea about what is going on and how Tailscale can solve problems for you. I am solving problems with Tailscale that I didn’t even know I had!

Comments