Notes on hosting Headscale DERP nodes

If you find yourself in a situation where you have to self host DERP nodes for your Tailnet, there are some important things to keep in mind for a smooth experience.

Why host DERP nodes?

DERP is essentially a proxy that relays traffic between Tailnet nodes that are unable to establish a direct connection due to restrictive firewalls. It is the last resort after the signaling server is unable to help achieve a direct connection through UDP hole punching.

And while Headscale has a built-in DERP server, you might want to host separate nodes due to reasons such as processing power or available bandwidth in a specific region.

Why not use public Tailscale DERP servers?

Public DERP nodes could have some bandwidth or access limits. But the most important reason for self-hosting is blocking - Tailscale DERP servers are blocked by some networks, preventing you from reaching your Tailnet altogether.

Tailscale now offers Peer Relay functionality, which makes individual nodes function as a proxy, but that functionality mainly exists to solve the bandwidth problem of public DERP servers, not the access issue - Peer Relay nodes don't run on TCP 443 and will be blocked in restrictive networks.

Running DERP on a shared server

Similarly to HTTPS, DERP also runs on port 443 by default, but it is not HTTPS. So you can't use regular HTTP proxy server configuration. But TCP proxies can do that. And such proxies can use SNI (Server Name Indication) extension of TLS to support both web traffic and DERP traffic at the same time.

How does it work?

During the TLS handshake a client specifies the hostname and the proxy server decides if it should proxy it as web traffic, TCP traffic (and handle encryption) or do a TCP pass-through, which lets the backend server handle everything, including the TLS part. And this is all that is needed.

SNI pass-through is supported by all major proxy servers like Nginx (using stream mode), HAProxy, Caddy, Traefik, etc. All of those let the proxy inspect SNI without decrypting TLS, then route to appropriate backend.

Pass-through also means that you have to give a valid certificate to the DERP server, it is not enough that your proxy server has it configured.

Example configuration

Setting up Docker on arbitrary nodes is quite easy so I just use Docker Compose and Traefik to get it all running quickly.

If you don't have any other services running on the same host, you can of course just expose the port 443 directly.

networks:
  derp:
    name: derp

  traefik:
    external: true

services:
  derp:
    # https://github.com/Janhouse/tailscaled-derper
    image: janhouse/tailscaled-derper
    restart: unless-stopped
    volumes:
      - ${CERTS}/fullchain.pem:/app/cert.crt
      - ${CERTS}/privkey.pem:/app/cert.key
      - /dev/net/tun:/dev/net/tun
      - derp-data:/var/lib/tailscale
    cap_add:
      - net_admin
    ports:
      #- "0.0.0.0:443:443"
      - "0.0.0.0:3478:3478/udp"
    environment:
      DERP_CERT_MODE: "manual"
      DERP_DOMAIN: "${DOMAIN_NAME}"
      DERP_VERIFY_CLIENTS: "true"
    networks:
      - derp
      - traefik
    labels:
      traefik.docker.network: traefik
      traefik.enable: true
      traefik.http.routers.derp.rule: Host(`${DOMAIN_NAME}`)
      traefik.http.routers.derp.entrypoints: web
      traefik.http.services.derp.loadbalancer.server.port: 443
      traefik.http.routers.derp.service: derp

      traefik.tcp.routers.derp-tcp.rule: HostSNI(`${DOMAIN_NAME}`)
      traefik.tcp.services.derp-tcp.loadBalancer.serversTransport: mytransport@file
      traefik.tcp.services.derp-tcp.loadBalancer.server.port: 443
      traefik.tcp.routers.derp-tcp.entrypoints: websecure
      traefik.tcp.routers.derp-tcp.tls.passthrough: true
      traefik.tcp.routers.derp-tcp.service: derp-tcp

  # docker compose exec derp tailscale up --login-server=https://headscale.example.com --hostname=derp-gb --accept-dns=false
  
  volumes:
    derp-data:

Making the DERP node private

Unless you restrict your DERP server, anyone can use it.

Most likely you don't want that and this is why we pass DERP_VERIFY_CLIENTS: "true" to the container, which makes the DERP server check the list of Tailnet clients within your Tailnet. It gets the list from a running tailscaled service inside the container.

This means that DERP nodes on the Tailnet need to see the full list of other Tailnet nodes, or else node verification will fail. If you don't use ACLs, then it is fine, but if you do, remember to add a rule that allows DERP to ping anything: this will ensure it functions properly.

  "acls": [
    {
      // DERP nodes need to see all other nodes for verification
      "action": "accept",
      "proto": "icmp",
      "src": ["group:derp"],
      "dst": ["autogroup:member:*", "autogroup:tagged:*"]
    }
  ]

To summarize the key things to keep in mind:

• Self-hosted DERP nodes allow you to bypass network restrictions that block public Tailscale servers.
• SNI pass-through allows you to share port 443 with other services on the same server.
• The DERP server needs its own valid TLS certificate since TLS is not terminated at the proxy.
• Set DERP_VERIFY_CLIENTS: true to restrict access to your own Tailnet.
• And if you use ACLs, don't forget to allow ICMP from your DERP nodes — without it, client verification will fail.