Why does my k3s DNS resolution return a self-signed certificate from my home router?

After much, much reading, I learned that it came down to /etc/resolv.conf in the containers.

In /etc/resolv.conf, within the search directive, there was the lan domain.

Also, options ndots: 5 was set.

per man resolv.conf,

PLAINTEXTCopied
search Search list for host-name lookup.
      By default, the search list contains one entry, the local domain name.
      It is determined from the local hostname returned by gethostname(2); the
      local domain name is taken to be everything after the first '.'.
      Finally, if the hostname does not contain a '.', the root domain is
      assumed as the local domain name.

      This may be changed by listing the desired domain search path following
      the search keyword with spaces or tabs separating the names.  Resolver
      queries having fewer than ndots dots (default is 1) in them will be
      attempted using each component of the search path in turn until a match
      is found.  For environments with multiple subdomains please read options
      ndots:n below to avoid man-in-the-middle attacks and unnecessary traffic
      for the root-dns-servers.  Note that this process may be slow and will
      generate a lot of network traffic if the servers for the listed domains
      are not local, and that queries will time out if no server is available
      for one of the domains.

So, any domain name with fewer 5 dots in it, was searching lan and it looks
like my home router 192.168.1.1 gladly reports its IP back for any/all random
*.lan domains, apart from legitimate hosts on the lan such as my laptop:
thinkpad.lan: 192.168.1.47

Obviously google.com has fewer than 5 dots in it, so google.com.lan was
mapping to my home router. Nice.

Additionally, for another layer of context, k3s was using coreDNS; it spun up a service,
configmap with a Corefile, and pod. In the Corefile, all DNS requests were
proxied to /etc/resolv.conf:

PLAINTEXTCopied
.:53 {
    forward . /etc/resolv.conf
}

• CoreDNS ConfigMap Docs
• CoreDNS forward plugin Docs

What finally tipped me off to the issue was host.

I learned about adding a trailing dot to a domain name to signify that it is a
FQDM (fully qualified domain name) to force a DNS lookup against the
nameserver and not do any search business.

I realized that dig google.com without +search was ignoring the
/etc/resolv.conf search directive and that explained why it was not facing the lan problem

BASHCopied
$ kubectl exec -it dnsutils -- host google.com
google.com.lan has address 192.168.1.1

$ kubectl exec -it dnsutils -- host google.com.
google.com has address 142.250.115.101

$ kubectl exec -it dnsutils -- ping -c 1 google.com
PING google.com (192.168.1.1): 56 data bytes
64 bytes from 192.168.1.1: seq=0 ttl=63 time=487.186 ms

$ kubectl exec -it dnsutils -- ping -c 1 google.com.
PING google.com. (142.250.113.101): 56 data bytes
64 bytes from 142.250.113.101: seq=0 ttl=53 time=46.633 ms

$ kubectl exec -it dnsutils -- dig +search google.com

; <<>> DiG 9.16.27 <<>> +search google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59307
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 350fdd3d6b0e9118 (echoed)
;; QUESTION SECTION:
;google.com.lan.                        IN      A

;; ANSWER SECTION:
google.com.lan.         5       IN      A       192.168.1.1

;; Query time: 3 msec
;; SERVER: 10.43.0.10#53(10.43.0.10)
;; WHEN: Sun Feb 02 19:23:33 UTC 2025
;; MSG SIZE  rcvd: 85

$ kubectl exec -it dnsutils -- traceroute google.com
traceroute to google.com (192.168.1.1), 30 hops max, 46 byte packets
  1  10.42.0.1 (10.42.0.1)  0.017 ms  0.014 ms  0.013 ms
  2  192.168.1.1 (192.168.1.1)  4.172 ms  1.658 ms  0.721 ms

$ kubectl exec -it dnsutils -- traceroute google.com.
traceroute to google.com (142.250.114.101), 30 hops max, 46 byte packets
  1  10.42.0.1 (10.42.0.1)  0.017 ms  0.014 ms  0.013 ms
  2  192.168.1.1 (192.168.1.1)  4.172 ms  1.658 ms  0.721 ms
  3  *  *  *
  4  *  *  *
  5  *  *  *
  6  *  *  *
  7  *  *  *
  8  syn-035-146-031-064.res.spectrum.com (35.146.31.64)  82.547 ms  94.083 ms  417.137 ms
  9  lag-63.rcr01hstqtx02.netops.charter.com (24.164.209.122)  21.506 ms  523.018 ms  30.659 ms
10  syn-131-150-063-015.res.spectrum.com (131.150.63.15)  42.605 ms  syn-024-093-037-083.inf.spectrum.com (24.93.37.83)  40.012 ms  syn-131-150-063-015.res.spectrum.com (131.150.63.15)  39.808 ms
11  *  *  *
12  142.251.237.112 (142.251.237.112)  38.418 ms  72.14.237.46 (72.14.237.46)  45.226 ms 142.251.76.36 (142.251.76.36)  43.672 ms
13  108.170.233.119 (108.170.233.119)  28.496 ms 142.250.60.238 (142.250.60.238)  38.363 ms 108.170.228.91 (108.170.228.91)  28.948 ms
14  108.170.228.82 (108.170.228.82)  39.146 ms 108.170.233.119 (108.170.233.119)  80.752 ms 108.170.233.117 (108.170.233.117)  30.590 ms
15  142.251.76.47 (142.251.76.47)  39.273 ms 108.170.229.87 (108.170.229.87)  32.167 ms 142.250.233.171 (142.250.233.171)  34.306 ms
16  142.250.224.11 (142.250.224.11)  36.546 ms 216.239.43.144 (216.239.43.144)  32.527 ms 209.85.252.210 (209.85.252.210)  28.142 ms
17  142.250.224.27 (142.250.224.27)  40.756 ms 142.250.224.25 (142.250.224.25)  33.482 ms 142.250.224.13 (142.250.224.13)  32.428 ms
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  rr-in-f101.1e100.net (142.250.114.101)  30.500 ms  *  36.067 ms
...