U/Surdeus/Incidents/K3s And Traefik Covering The Caddy

1. Incident Overview

On October 18, 2026, the secure web routing pipeline collapsed. Public HTTPS traffic hitting the frontend gateway received a mixture of local plaintext *HTTP 400 Bad Request* errors followed by unexpected *404 Not Found* screens. The target backend service managed by *Caddy* was completely unreachable.

Impact: Total outage of the wiki.mojosa.su domain.
Core Vulnerability: A recent system-wide package and kernel upgrade coupled with an uncleaned Kubernetes (K3s) laboratory environment created a cascaded infrastructure collision.

2. Root Cause Analysis (The Cascade Effect)

The failure was traced to a three-tier chain reaction:

# *Kernel Module Eviction (AmneziaWG):* The OS update pushed a new Linux Kernel (6.12.113+deb13-amd64). The Dynamic Kernel Module Support (DKMS) framework failed to auto-compile the AmneziaWG driver for the new version, leaving the module compiled only on the dead kernel (6.12.96). This completely broke the network interface (wg0), rendering the backend pipeline physically blind.
# *Port Squatting (K3s / Traefik Ghost Pods):* During recent software experimentation, K3s (Lightweight Kubernetes) had been deployed. Due to K3s using the KillMode=process directive by default, stopping the main systemd service left containerized Traefik Ingress workers running natively in background memory. Traefik hijacked port 443 before HAProxy could bind to it.
# *Protocol Header Invalidation:* When troubleshooting the resulting 404s, the proxy protocol was shifted between version formats. Because the downstream Caddy was explicitly configured to parse binary Proxy Protocol v2 (send-proxy-v2), any fallback tests omitting or downgrading this wrapper caused immediate parser drops (Plaintext HTTP 400).

3. Step-by-Step Resolution Protocol

The following precise sequence was executed to fully reclaim the infrastructure and restore the secure socket state:

Phase A: Resuscitating the AmneziaWG Network Tunnel

To resolve the modprobe: FATAL: Module amneziawg not found mismatch, the driver was manually recompiled against the active kernel:

    # 1. Fetch developer tools matching the exact active kernel instance
    sudo apt update && sudo apt install linux-headers-\$(uname -r)

    # 2. Force DKMS to build the module for the active architecture target
    sudo dkms install amneziawg/1.0.0 -k \$(uname -r)

    # 3. Rebuild the system module dependency trees and initialize the driver
    sudo depmod -a
    sudo modprobe amneziawg

Phase B: Purging Rogue Kubernetes Port Listeners

To evict the orphaned Traefik proxies squatting on public ports 80/443, the built-in K3s environmental sweep tool was executed:

# Evict all background containerd runtimes, shims, and network wrappers
sudo /usr/local/bin/k3s-killall.sh

Phase C: Restoring HAProxy Directives & Rekindling Pipeline

Once port 443 was successfully freed from Traefik, HAProxy was locked into pure Layer 4 TCP Passthrough mode, and the vital binary proxy headers were returned to guarantee correct routing into Caddy.

Inside /etc/haproxy/haproxy.cfg:

    frontend https_in
        bind *:443
        mode tcp
        option tcplog
        default_backend https_home

    backend https_home
        mode tcp
        # MANDATORY: send-proxy must be explicit to satisfy Caddy's listener wrappers
        server home_server 10.0.0.2:443 send-proxy

Finally, the entire stack was cleanly re-initialized:

    # 1. Fire up the secure AmneziaWG link
    sudo awg-quick up wg0

    # 2. Restart the reverse proxy to bind to the now vacant port 443
    sudo systemctl restart haproxy

4. Post-Mortem Operational Takeaways

Proxy Protocol Integrity: Do not alter the backend injection parameters (send-proxy-v2) globally when debugging, as modern Caddy listener wrappers treat unexpected packet headers (or missing headers) as invalid protocol data, spawning internal 400/404 responses.
Kubernetes Isolation: Never run test k3s environments directly on a production reverse-proxy host machine unless explicit --disable=traefik parameters are passed to the daemon config, protecting your public ingress routes from hijack risks during updates.

📖:U/Surdeus/Incidents/K3s And Traefik Covering The Caddy