Troubleshooting TLS Handshake Failures Between Traefik Ingress and Vault on k3s

I’m running a small k3s cluster for testing and learning Kubernetes and have deployed HashiCorp Vault via Helm with TLS and UI enabled. I configured Traefik as the ingress controller in my helm override file, but when I access the DNS name (vault.home.arpa) that points to my cluster, I get a “bad gateway” error. In the Vault pod logs I see TLS handshake errors like:

BASHCopied
2025-01-29T12:16:18.833Z [INFO]  http: TLS handshake error from 10.42.0.52:46042: remote error: tls: bad certificate
2025-01-29T12:16:19.659Z [INFO]  http: TLS handshake error from 10.42.0.52:58742: remote error: tls: bad certificate
2025-01-29T12:16:19.952Z [INFO]  http: TLS handshake error from 10.42.0.52:58752: remote error: tls: bad certificate
2025-01-29T12:16:20.157Z [INFO]  http: TLS handshake error from 10.42.0.52:58766: remote error: tls: bad certificate

I can access the Vault UI via HTTPS without specifying a port, so it seems the issue lies in communication between the ingress and the Vault service/pod. I prefer not using a LoadBalancer (which works when I list specific ports) as it forces mandatory port usage after the node IP.

Below is my helm override YAML for Vault:

YAMLCopied
global:
  namespace: vault-dev
  enabled: true
  tlsDisable: false

server:
  ingress:
    enabled: true
    ingressClassName: "traefik" # Use Traefik as the ingress controller
    annotations:
      traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
      traefik.ingress.kubernetes.io/router.tls: "true"
    hosts:
      - host: "vault.home.arpa" # Replace with your domain
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vault # Placeholder for the Vault service name
                port:
                  number: 8200
    tls:
      - secretName: ingress-secret # Ensure this secret exists with your SSL certs
        hosts:
          - "vault.home.arpa"

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
    VAULT_SKIP_VERIFY: "true"
  
  volumes:
    - name: userconfig-vault-server-tls
      secret:
        defaultMode: 420
        secretName: vault-server-tls # Matches the ${SECRET_NAME} from above

  volumeMounts:
    - mountPath: /vault/userconfig/vault-server-tls
      name: userconfig-vault-server-tls
      readOnly: true

  standalone:
    enabled: true
    config: |
      ui = true
      listener "tcp" {
        tls_disable = 0
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"

      }

      storage "file" {
        path = "/vault/data"
      }

ui:
  enabled: true
  # serviceType: LoadBalancer

The TLS certificates were created following the example provided in HashiCorp’s documentation for a self-signed CA for the k3s cluster and Vault. I can confirm the certificates exist upon exec’ing into the Vault pod.

My suspicion is that the ingress may not be using the correct Vault certificate (crt/key) as expected. Here’s the running config of the ingress service:

BASHCopied
$ kubectl describe ingress vault
Name:             vault
Labels:           app.kubernetes.io/instance=vault
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=vault
                  helm.sh/chart=vault-0.29.1
Namespace:        vault-dev
Address:          192.168.178.211,192.168.178.212,192.168.178.213
Ingress Class:    traefik
Default backend:  <default>
TLS:
  ingress-secret terminates vault.home.arpa
Rules:
  Host             Path  Backends
  ----             ----  --------
  vault.home.arpa  
                   /   vault:8200 (10.42.3.73:8200)
Annotations:       meta.helm.sh/release-name: vault
                   meta.helm.sh/release-namespace: vault-dev
                   traefik.ingress.kubernetes.io/router.entrypoints: websecure
                   traefik.ingress.kubernetes.io/router.tls: true
Events:            <none>

And the ingress-secret for TLS/KEY:

BASHCopied
$ kubectl describe secrets ingress-secret
Name:         ingress-secret
Namespace:    vault-dev
Labels:       io.portainer.kubernetes.configuration.owner=administrator
              io.portainer.kubernetes.configuration.owner.id=1
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.key:  1708 bytes
tls.crt:  1211 bytes

I’d appreciate any insights or suggestions on where to troubleshoot further, especially regarding ensuring that the correct TLS certificates are being used between Traefik and Vault. Any help is welcome!