I am Using Tailscale SSH, and Maybe You Should Too!

| Comments

I don’t know if I qualify as an early adopter of Tailscale. My first blog post about it was in September of 2020, and that post said I had Tailscale installed on a few machines for months before I really started using and relying on it. I do know that I very much rely on Tailscale, and I don’t know what I would do without it.

I woke up at around 9:00 p.m. That was really late! I saw dozens and dozens of tweets about Tailscale SSH in my Twitter timeline, and it was the number-two post on Hacker News. I read a little about what it did, and my very first thought was, “Uh-oh! This is scary!” Then I thought it was neat. Then I thought it was scary again.

I made a latte. I played some video games. Then I started setting up Tailscale SSH.

What is Tailscale?

Tailscale is a mesh VPN. You install the Tailscale client on all of your computers, servers, and devices. Then each device will attempt to make a Wireguard VPN connection directly to every other device on your network. It is like having your own virtual LAN, but it can be spread out over the entire planet.

It is also stupid easy to set up. If you aren’t dilly-dallying, I bet it takes less than two minutes to get a new machine onto your Tailnet.

One of my favorite features of Tailscale is their machine-sharing option. I depend on this pretty heavily. It is how I collaborate on blogs with Brian Moses, and it is how I share dozens of gigabytes of Create/Invent Podcast video files with Jeremy Cook.

Why am I brave enough to use Tailscale SSH?

I was worried about this idea at first. If you could hijack my SSH keys or agents, you can wreak havoc on the Butter, What?! media empire! You could post nonsense on patshead.com, butterwhat.com, and creativitycast.com. You could do horrible things to me.

Do I really want to trust a third party with this piece of my security?

The truth is that I already do. I have the Tailscale client installed on a couple dozen machines. Those clients now have SSH servers built right in, but they don’t need SSH servers to be a security risk. Tailscale has had the ability to run arbitrary code on my boxes for years.

I was aware of that when I first decided to use Tailscale. I am downloading executable binary images from a third party. I don’t know what’s inside. I decided that I was going to trust the Tailscale company, and I still do.

As long as there’s an SSH server hiding inside every one of my Tailscale clients, why shouldn’t I use it?!

What advantages will I see over distributing SSH keys manually?

If you’re using Tailscale, but you’re just using password authentication with SSH, I think you should just turn on Tailscale SSH. I don’t feel terribly safe even having password authentication turned on. You’d be better off ditching the passwords and letting Tailscale handle your SSH authentication for you.

First, I am bad at rotating SSH keys. I use a separate SSH private key on each of my workstations. The key on my new 2-in-1 laptop is as fresh as the laptop. The key on my old laptop is from 2021. The key on my desktop is scary old. I haven’t distributed the new laptop’s public key to all my servers yet. I figured I’d sit down and generate a new key for my desktop and kill two birds with one stone.

That is the second problem. Distributing new keys is a pain in the neck. I have one machine that runs Windows 11 now. Some are physical machines. Some aren’t powered on all the time. Some are virtual machines here at home, while others are virtual servers on the Internet.

I usually build a new authorized_keys file with all my current keys, then run a loop that uses scp to drop it in place on each device. If something goes wrong on a device I can’t easily access, it can be a real pain in the tuchus.

Nearly every single server I have is running Tailscale, and each Tailscale client has its own private key for Wireguard. Why do I need to maintain and distribute more keys?

Tailscale SSH authorizes machines instead of users

Tailscale doesn’t know that 18 people have shell accounts on a server. Tailscale just knows that this server and my desktop machine are both on my Tailnet. If you enable Tailscale SSH on both devices, then any of those 18 people would be able to SSH to my desktop computer!

NOTE: I should verify this. I immediately set the action value to accept. If I didn’t do that, it would check that my client is authenticated via the browser. I imagine this would save me, but I SSH so often that it would be quite annoying!

I have split my Tailnet into two different tags to remedy this situation. My desktop and both laptops are now tagged with workstation, and almost everything else is tagged with server. The machines with the workstation tag are computers that no one besides me has credentials for.

1
2
3
4
5
6
7

"ssh": [
  {
      "action": "accept",
      "src":    ["tag:workstation"],
      "dst":    ["tag:server", "tag:workstation"],
      "users":  ["autogroup:nonroot", "root"],
  },

I set the SSH ACL to allow connections from any workstation to any server or workstation. It seems to work as expected. Now all I need to do is start enabling Tailscale SSH on more devices!

From a practical standpoint, is this much different than managing SSH keys yourself?

I don’t know about you, but when I store an SSH private key on a server with multiple uses, it automatically feels like that key is compromised.

What if someone else has root? They can steal the key. What if I accidentally goofed up the permissions? What if someone has access to the backups? What if someone manages to connect to the ssh-agent?

These are the sort of keys I would have out in the world for something like a backup job, or to run jobs that publish sites to production. I try do give them as little access as possible.

Tailscale will let me continue to do that, but it will let me do it in a centralized location. I can set ACLs that say ScaryHost1 and only connect to ScaryHost2 as one particular user, and I won’t even have to log in to either host to allow that to happen or to revoke the access.

Centralized auth management will be awesome!

I am aware that there are systems built specifically to address this issue. I’ve never had much interest in working to implement them, because my SSH keys were already created, secured, and distributed. I never have to start from scratch. I usually just have to sneak one new key into a handful of places every few years.

I am already using Tailscale. Tailscale is already running on 95% of my machines. It doesn’t take much to install Tailscale on the rest.

At that point I am inches away from never having to manage SSH keys again!

What about shared nodes?

Shared nodes are the bee’s knees. Truth be told, I am more than a little uncomfortable with giving Tailscale so much power over my network. In my opinion, machine sharing is the biggest reason to use Tailscale’s service instead of hosting your own Headscale network.

I can click on a machine in my admin console, and it will give me a URL to send to a friend. They click on the URL, sign up for Tailscale, and they’ll be pinging my shared server in 5 minutes. If they already use Tailscale, they’ll be pinging my machine in seconds.

The documentation says that shared nodes don’t support Tailscale SSH, but it doesn’t say what happens if it is enabled. I had some guesses, but I didn’t have an easy way to try it for myself this week, so I figured I should ask. If you use Tailscale SSH on a shared node, anyone who you have shared the machine with will just fail to authenticate.

I am pretty sure Brian pushes to a Git repo on our Butter, What?! staging server.

Should I just continue to use SSH keys for this dev server? Should I turn on Tailscale SSH for my convenience, then make Brian use a different port? Either will work fine, but it looks like I won’t be completely eliminating my SSH keys anytime soon!

Tailscale SSH fits well with how I’ve been using Tailscale

In days gone by, I had a handful of services exposed on the Ethernet and Wi-Fi interfaces on my desktop and laptop. I don’t own many devices that aren’t now running Tailscale, so I’ve been locking more and more things down.

My desktop and laptop don’t have open ports on their physical interfaces. If I take my laptop to a coffee shop or our local makerspace, there’s nothing open for anyone to try to poke at. Except for SSH. I am always brave enough to have passphrase-only SSH open to the world.

I am only just now realizing that I can lock down my NAS, my Octoprint server, and my CNCjs server in the same way. I don’t connect to those from any devices that aren’t already part of my Tailnet!

Tailscale SSH feels like another step on this journey.

I am finally enabling MagicDNS!

This is why I assume I am an early adopter of Tailscale. I was relying on Tailscale for so long that I had things set up before MagicDNS arrived. I have a gigantic hosts file on each of my workstations instead!

I need at least two or three servers to be listed in those hosts files anyway. My virtual machine host, my NAS, and my desktop are connected with 40-gigabit Infiniband. They have to bypass Tailscale to reach even a small fraction of their maximum speed.

I am relying less on my NAS today, and I am working towards eliminating the need for this Infiniband link. Eliminating that hosts file and upgrading to MagicDNS will take me a step farther down that road.

Conclusion

I am looking forward to seeing where Tailscale SSH goes from here.

What do you think? Am I crazy for relying on Tailscale SSH? Or is this one of the best decisions I have ever made? Let me know in the comments, or stop by the Butter, What?! Discord server to chat with me about it!

Comments