Headscale & Tailscale

In early 2024, I discovered Headscale, an open source, self-hosted implementation of the Tailscale control server. I’ve been using it successfully ever since. Over time, I have documented my learning in Obsidian and also created several PDFs to help my friends connect to my Tailnet network. Finally, I decided to compile everything into a series of blog posts, recently revived after 10 years of silence. One topic I find challenging to research is access control via ACL. I emphasize this area because I believe it can be helpful to many people. Additionally, I wrote this blog post in French to remove a language barrier.

VPN Server

My journey into the world of Homelab began in 2017 with the purchase of a Synology 4-bay NAS. I quickly realized the value of securing my server. So I learned to change the ports of my services, expose the minimum of things on the Internet, limit the countries that can access my server, add a reverse proxy and finally add a Synology VPN server to be able to access it from the outside. The aim was to be able to administer my homelab and also use my two Pi-Holes to filter my DNS from outside my local network.

This VPN server also enabled me to connect my second NAS hosted at a friend’s for offline backups. Unfortunately, I was having regular disconnections from this remote NAS. It didn’t know how to reconnect automatically. What’s more, the Synology VPN doesn’t know how to assign an IP to a machine, so if I connected with my personal computer, my remote server risked changing IP if it reconnected… In short, this solution didn’t suit me for a number of reasons.

During this process of securing my remote connections, I made a few attempts with WireGuard. I liked being able to assign a fixed IP. But in the end, I never completed this solution to replace the Synology VPN.

In addition, I wanted to find a solution to secure my friends’ access to certain self-hosted services like Vaultwarden and Mealie. I used Nginx Proxy Manager’s IP filter to limit access. But this isn’t practical, as most of my friends have dynamic IPs and it didn’t work if they were on their phones on the move. What’s more, it made these services visible from the outside, a situation that didn’t sit well with me.

Tailscale

For some time now, I’d been hearing good things about Tailscale, a VPN system with WireGuard that enables direct connections to all network nodes. It manages all IP addresses and configurations automatically. Sounds ideal, but I was having trouble going ahead with it because I didn’t want to delegate access to my homelab to an outside site.

Headscale

Via following some hashtags on Mastodon, I started hearing about Headscale, an open source self-hosting implementation of the Tailscale controller. It allows the use of Tailscale’s many official clients and supports the majority of Tailscale’s features such as permissions via an ACL file. It seemed the best of both worlds.

Installation is relatively easy for someone with a little self-hosting experience. The easiest way is to install it via Docker on a server in your homelab or on a VPS. I chose this first method because I prefer to limit my external dependencies. I chose to run Headscale on a separate VM in my Proxmox for greater security (isolation and backup).

Here’s my Docker Compose that I use with Portainer stacks.

version: '3.9'
services:
  headscale:
    container_name: headscale
    image: headscale/headscale:0.23.0
    restart: unless-stopped
    ports:
      - 8080:8080
      - 9090:9090
    volumes:
      - /data/docker/headscale/config:/etc/headscale
      - /data/docker/headscale/data:/var/lib/headscale
    environment:
      - TZ=America/Montreal
    command: serve

Don’t use the “latest”; after each major update, Headscale has many breaking changes, and the configuration needs to be reviewed. This is the only one of my containers that has not been automatically updated with Watchtower.

Before starting the Docker Compose, you need to create two directories on your Headscale host:

• /data/docker/headscale/config
• /data/docker/headscale/data

You’ll also need a configuration file in the “config” folder. Download the default file for version 0.23.0 and do the fallowing change based on your configuration.

# The url clients will connect to
server_url: https://hs.xxxx.xx

# Address to listen to / bind to on the server
listen_addr: 0.0.0.0:8080

# Address to listen to /metrics
metrics_listen_addr: 0.0.0.0:9090

database:
  sqlite:
    write_ahead_log: true

policy:
    # The mode can be "file" or "database" that defines
    # where the ACL policies are stored and read from.

    # Allows ACLs to be managed in the Headplane interface
    mode: database 

    # If you want to manage the ACL rules via a file and not Headplane.
    path: "/etc/headscale/acl.hujson" 

 dns:
    magic_dns: true
    base_domain: ts.xxxx.xx
    # List of DNS servers to expose to clients.
    nameservers:
      global:
        - 192.168.1.222 # Main PiHole 
        - 192.168.1.223 # Seconday PiHole

Headplane Visual Interface

There are several interfaces available to interact more easily with Headscale. For now, I’ve chosen Headplane that seems to be the most maintained and allows the ACL file to be managed via the graphical interface.

It is necessary to create an API key in a terminal of the machine where Headscale rotates:

docker exec headscale headscale apikeys create

This is what my Docker Compose looks like for a Headscale and Headplane.

version: '3.9'
services:
  headscale:
    container_name: headscale
    image: headscale/headscale:v0.23
    restart: unless-stopped
    ports:
      - 8080:8080
      - 9090:9090
    volumes:
      - /data/docker/headscale/config:/etc/headscale
      - /data/docker/headscale/data:/var/lib/headscale
    environment:
      - TZ=America/Montreal
    command: serve

  headplane:
    container_name: headplane
    image: ghcr.io/tale/headplane:latest
    restart: unless-stopped
    volumes:
      - '/data/docker/headscale/config:/etc/headscale'
      - '/data/docker/headscale/data:/var/lib/headscale'
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
    ports:
      - '3000:3000'
    environment:
      # This is always required for Headplane to work
      COOKIE_SECRET: 'xxxxxxx' # generated by "openssl rand -hex 32"
      HEADSCALE_URL: 'https://hs.xxxx.xxx'
      CONFIG_FILE: '/etc/headscale/config.yaml'
      #ACL: '/etc/headscale/acl.hujson'
      HEADSCALE_INTEGRATION: 'docker'
      HEADSCALE_CONTAINER: 'headscale'
      DISABLE_API_KEY_LOGIN: 'true'
      HOST: '0.0.0.0'
      PORT: '3000'
      TZ: 'America/Montreal'
      ROOT_API_KEY: 'xxxxxxxxxxx' # API key previously generated

I keep things open if another interface turns out to be more complete in the future. Among other things, a visualization of the ACL access rules (see below) will be fantastic:-).

Nginx Proxy Manager Integration

For my part, I use the NPM (Nginx Proxy Manager) reverse proxy. This is the configuration that works in my homelab. You obviously need to have a domain name (via Cloudflare for me), an SSL certificate (generated via Let’s Encrypt) and have a system to update your local public IP address on your register (via the Cloudflare DDNS docker for me). Personally, I use my xxxx.xxx domain with wildcards for all the many subdomains.

To use Headplane (and/or metrics) on the same subdomain, you should add a “Custom Nginx Configuration” in the “advanced” section. I also included a robots.txt file to prevent indexing and other robots, just in case.

location /{
        proxy_pass http://192.168.1.240:8080;
        proxy_redirect http:// https://;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}
location ^~ /metrics/ {
        proxy_pass http://192.168.1.240:9090/metrics;
        proxy_redirect http:// https://;
        allow 192.168.1.0/24;  # limits access to only the local network
        allow 100.64.1.0/24;   # limits access to only the tailnet network
        deny all;
}
location ^~ /admin/ {
        proxy_pass http://192.168.1.240:3000/admin/;
        proxy_redirect http:// https://;        
        allow 192.168.1.0/24;  # limits access to only the local network
        allow 100.64.1.0/24;   # limits access to only the tailnet network
        deny all;
}
location = /robots.txt {
        add_header Content-Type text/plain;
        return 200 "User-agent: *\nDisallow: /\nUser-agent: GPTBot\nDisallow: /\nUser-agent: nibbler\nDisallow: /\n";
}

By default, when you access the URL https://hs.xxxx.xxx/admin, the registry of your domain will give you your public local IP. As a result, the above filtering will not work because NPM will see your public IP and not your local IP. To simply solve this problem, you can add a local DNS record [A/AAAA) to your Pi-Hole(s).

hs.xxxx.xxx = LOCAL.IP.OF.NPM

As seen that I managed nearly 40 subdomains in NPM, there’s a more elegant solution, the wildcard DNS. Unfortunately, this is not yet possible to do this via the user interface, you have to use a terminal on your Pi-hole.

nano /etc/dnsmasq.d/02-my-wildcard-dns.conf

Add this line

address=/xxxx.xxx/LOCAL.IP.OF.NPM

Then restart your Pi-hole and everything should work. You can double-check if it’s working correctly using this command:

dig @IP.OF.YOUR.PIHOLE hs.xxxx.xxx

The result should be

;; ANSWER SECTION:
hs.xxxx.xxx.		0	IN	A	LOCAL.IP.OF.NPM

Be careful, it is very important to do this procedure for all your Pi-holes. I fought days to isolate a major problem in my homelab because of a poorly configured Pi-hole. FYI, if you use Orbital Sync to synchronize your Pi-holes, wildcards are not synchronized.

Create Users

With the terminal of the Headscale machine:

docker exec headscale headscale users create USERNAME

To do this in the Headplane interface, click on “add a new user” in the user page

Adding nodes to Headscale

Here’s how to add Tailscale clients to the Headscale network.

• iOS Installation
• MacOS Installation
• Linux Debian Installation
• Proxmox LXC Installation
• Synology NAS Installation
• Hassio Installation (Home Assistant OS)

Sorry, I don’t have a Windows machine or Android phone at home to do a tutorial.

Exit Node and subnet access

In addition to the Headscale VM, I’ve created another Debian VM to install Tailscale. It performs several functions.

The most important is to be able to access my machines/VM/LXC that are not in Tailscale from outside. I use it regularly and transparently, to the point where I feel as if my homelab is always with me! It’s simply magical.

The exit node allows me to encrypt my internet traffic between my devices and my homelab when I’m on the move. It’s disabled by default; ideally, I’d like Tailscale to enable it automatically when I’m connected to public Wi-Fi. Theoretically, if one of my Tailscale users living in France were to activate an exit node, this could enable me to access geolocated content, such as certain Arte programs. I’ll leave you to imagine other uses :-)

To add routes or exit nodes, use a terminal on our Tailscale node.

tailscale up --advertise-routes=192.168.1.0/24,192.168.3.0/24,192.168.4.0/24 --accept-routes --advertise-exit-node --login-server=https://hs.xxxx.xxx

Roads and exit node must be approved in Headscale on command line

docker exec headscale headscale route list

ID | Node          | Prefix                   | Advertised | Enabled | Primary
2  | tailscale     | 0.0.0.0/0                | false      | false   | -
3  | tailscale     | ::/0                     | false      | false   | -
4  | tailscale     | 192.168.1.0/24           | false      | false   | false
5  | tailscale     | 192.168.3.0/24           | false      | false   | false

docker exec headscale headscale route enable -r 2 3 4 5

ID | Node          | Prefix                   | Advertised | Enabled | Primary
2  | tailscale     | 0.0.0.0/0                | true       | true    | -
3  | tailscale     | ::/0                     | true       | true    | -
4  | tailscale     | 192.168.1.0/24           | true       | true    | true
5  | tailscale     | 192.168.3.0/24           | true       | true    | true

Another method is to do this via the Headplane interface. On the node, click on the 3 dots on the right and choose “Edit route settings of Tailscale”.

Access control via ACL

By default, a conventional VPN server grants full access to the local area network. This poses a significant risk if either my friend or I experience a compromise. In contrast, the Headscale/Tailscale solution enables precise access control for each node through a simple ACL file.

First, separate all the nodes in the group (me, user, server, etc.). Next, create a diagram to outline the connections you want to permit among this group. This diagram will illustrate the connections between my Tailscale nodes. You can use a simple piece of paper for this task. :-)

Tags need to be added to the node to define the group. For my configuration, I created five different tags:

• tag:me | For my personal devices, computer, phone and tablet
• tag:private | For my services, accessible only by me.
• tag:semi-private | For services accessible by my users and me
• tag:user | My users
• tag:remote | My remote NAS for off-site backups.

You can add the tag via this command:

docker exec headscale headscale nodes tag -i <ID> -t tag:<TAGNAME>

More friendly, Headplane lets you add the tag via an interface. Click on the 3 dots to the right of the node, then click on “edit ACL tags.”

These are my ACL rules that correspond to the graph above.

{
  "acls": [
    {
      "action": "accept", 
      "src": ["tag:me"], 
      "dst": [
        "tag:private:*",
        "tag:remote:*",
        "tag:semiprivate:*",
        "tag:me:*",
        "autogroup:internet:*", # access to the Internet
        "192.168.1.0/24:*",     # access to my main vlan
        "192.168.3.0/24:*",     # access to my IoT vlan
        "192.168.4.0/24:*"      # access to my NoT vlan
      ]
    },
    {
      "action": "accept",
      "src": ["tag:private"], 
      "dst": [
        "tag:private:*",
        "tag:remote:*",
        "tag:semiprivate:*"
      ]
    },
    {
      "action": "accept", 
      "src": ["tag:semiprivate"], 
      "dst": [
        "tag:semiprivate:*"
      ]
    },
    {
      "action": "accept", 
      "src": ["tag:remote"], 
      "dst": [
        "tag:private:*"
      ]
    },
    {
      "action": "accept", 
      "src": ["tag:user"], 
      "dst": [
        "100.64.0.15:9000", # Mealie
        "100.64.0.14:8000", # VaulWarden
        "100.64.0.13:3001"  # Uptime Kuma
      ]
    },
  ]
}

Add your ACL rules via the “/etc/headscale/acl.hujson” file or in the Headplane interface.

I still have a lot of things to explore in the ACL world, but as it stands, this setup meets my current expectations.

Pi-Holes

My two Pi-Holes are part of my Tailnet network. Since my personal devices are always connected to Headscale, my DNS servers filter out a lot of tracking, advertising, and other internet annoyances. I chose not to include these Pi-Holes for my users, because I don’t want to manage their DNS traffic, which reveals a lot about their privacy. Unfortunately, it doesn’t seem possible to have different DNS servers per tag in Headscale/Tailscale.

So, as it stands, my users don’t use Tailscale’s DNS and have to connect to my services using IP and port. I hope a future implementation will be more flexible about this. But, if you have a solution to this problem, I’m interested!

What’s Next?

Although my system is complete and stable, there are still a few improvements to consider for 2025.

Headscale beta 0.24.0 offers better OpenID management thanks to OIDC authentication. I’m learning Authentik, a feature I’d like to add soon. I’m going to test it in a new VM so as not to disrupt my current system, which is essential.

For the moment, I’m not using MagicDNS or Tailscale’s split DNS. I create local DNS records in my Pi-Holes to link a name to an IP address (for example, uptime = 192.168.1.231). All services are associated with a sub-domain in Nginx Proxy Manager (like nodered.xxxx.xx -> http://192.168.1.231:3001). This solution allows me to manage all services and systems, whether or not they are part of Tailscale. However, I may be missing a feature that would simplify my homelab. Your comments would be most welcome.

Also, I’d like to explore the option of having my own DERP server to avoid depending on Tailscale’s servers. You can find more information about this on the Tailscale website.

Conclusion

Although at first glance the Headscale/Tailscale solution may seem very technical, it is actually simpler to implement than a VPN server, if you take into account the management of its many disadvantages. This solution is really very robust, and to date I haven’t had any problems. To be exact, I once had a problem with my misconfigured DNS server, which had nothing to do with Tailscale. As usual, the problem is very often between the chair and the keyboard :-).

I recommend you try the Headscale/Tailscale solution, I don’t think you’ll be disappointed! If, like me, you’re seduced by this solution, I invite you to contribute to the project via GitHub, on Discord, or to finance it via Ko-fi. As for Headplane, that’s via GitHub Sponsors.

Feel free to leave a comment if you need clarification or assistance, or if I missed an important point.

P.S. I only wish I had discovered this solution sooner!