Server Network Troubleshooting

Server Network Troubleshooting

Diagnostic methodology for Linux server network failures, plus the recovery techniques and specific failure patterns that have actually bitten us in production.

This guide is layered: read the Approach section first to anchor your mental model, then jump to the Scenario that matches your symptom, then reach for Recovery Techniques as needed.

Approach: diagnose by OSI layer, bottom-up

Most outages get solved 10x faster when you isolate the broken layer before changing anything. Each layer has its own diagnostic — skip the layered march and you’ll waste hours chasing the wrong thing.

New to “L1 / L2 / L3” shorthand?

The companion iproute2 Mastery primer defines the OSI layers, MAC vs IP, ARP/NDP, gateways, scope, netlink, and EUI-64 in one short section. Skim it once if any term in the table below is unfamiliar.

Layer	What can fail	First diagnostic
L1 — physical (cable, NIC, port)	Cable, NIC, switch port, transceiver	ip link show, switch-port LED, ethtool <iface>
L2 — link / switching (MAC, ARP, VLAN)	ARP, VLAN, MAC table, bridge	arping, ip neigh, tcpdump -nn -e arp
L3 — IP routing (your side)	Address, route, gateway	ip addr, ip route, ping <gw>
L3 — upstream (ISP side)	ISP routing, ARP for gateway	ping 1.1.1.1, traceroute, arping <gw>
L4 — TCP (ports)	Port reachability, firewall	nc -zv <ip> <port>, ss -tlnp
L5-L7 — service (HTTP, DNS, app)	DNS, HTTP, app-level	dig, curl -v, app logs

The discipline: start at L1, work up. Each working layer rules out a huge surface area. Do not change anything until you have a hypothesis that explains every observation.

Scenarios

Scenario: “ping 1.1.1.1 doesn’t work”

This is the most generic symptom. Walk down the list:

# 1. Is the physical link up?
ip -br link show
# Look for state UP and LOWER_UP on your WAN interface. NO-CARRIER = L1 problem.

ethtool eno1 | grep -E 'Link detected|Speed'
# "Link detected: yes" + matching speed = L1 healthy.

# 2. Does the interface have its IP and default route?
ip -4 addr show
ip -4 route
# Look for the right address on the right interface, and a "default via X dev Y" entry.

# 3. Can you reach the gateway at all?
ping -c 3 <gateway-ip>

# 4. If gateway pings: can the gateway route you out?
ping -c 3 1.1.1.1
ping -c 3 8.8.8.8

# 5. Is it actually DNS, not routing?
ping -c 3 example.com    # If this fails but 1.1.1.1 works, it's DNS
cat /etc/resolv.conf

The first failing step tells you the layer.

Scenario: “link is up, gateway pings, but no internet” (the half-dead CPE)

We hit this when Metronet’s edge router went into a “half-dead” state after a power event — link layer fine, ARP fine to local segment, but the gateway stopped routing upstream traffic.

Symptoms:

• Port LED green, ip link show shows UP/LOWER_UP, ethtool reports 1Gbps link
• ping <gateway> may or may not work (ICMP-only response, no forwarding)
• ping 1.1.1.1 → Destination Host Unreachable or just timeouts
• arping <gateway> may show ARP replies, but tcpdump -nn -e arp shows the gateway eventually stops responding to your ARPs

Critical test — does the gateway answer ARP for itself?

arping -c 5 -I vmbr0 <gateway-ip>
# 5/5 responses = gateway is on the segment and answering L2
# 0/5 responses = gateway is dead at L2, even though port LED is lit

Check the ARP cache state:

ip neigh show dev vmbr0
# Look for "<gateway-ip>  INCOMPLETE" — kernel failed to resolve MAC

Sniff ARP on the wire to see who’s talking:

tcpdump -i vmbr0 -nn -e arp
# Your "who-has <gateway> tell <self>" requests should leave every second
# If nothing answers, the gateway is dead upstream of you

Verdict: If arping to the gateway returns 0 responses and ip neigh shows INCOMPLETE while local intra-subnet ARP works fine, the ISP gateway is dead. You can’t fix this from your side — call the ISP, or fail over to a backup WAN if you have one.

Anti-pattern: Do not power-cycle your switch, your server, or restart networking yet. Your end is fine. Bouncing your own gear just adds variables and may take you further from working state.

Scenario: “I can ping the gateway but nothing past it”

Different problem from above. The gateway answers ARP and ICMP, but nothing past it routes.

traceroute 1.1.1.1
# Look for where it stops. If it dies at hop 2 (your ISP's next hop),
# it's an ISP routing issue. Call them.

# Confirm DNS isn't masking the issue
dig +short google.com @1.1.1.1
# If 1.1.1.1 doesn't reply, it's not DNS, it's transit.

Possible causes:

• ISP routing issue past your CPE
• Your ISP NAT’d public IP got dropped from their routing table
• The CPE has authentication tied to your MAC, and the MAC changed (e.g., after a switch swap)

Scenario: “outbound works but external services can’t reach my public IP”

You have working internet, but services on your public IP (e.g., 46.110.152.203:443) time out from the outside.

Most common causes after a WAN swap or outage:

1. Stale ARP at upstream gateway. During an outage we don’t ARP- announce our public IPs upstream. Fix by sending gratuitous ARP:

   for ip in 46.110.152.203 46.110.152.204 46.110.152.205 46.110.152.206; do
       arping -A -c 5 -I vmbr0 "$ip"
   done

   (Sends “I am <ip>, please update your tables.” Harmless if not needed.)

2. Stale conntrack entries pointing replies to the wrong source IP. Especially after a default-route swap during failover:

   conntrack -F
   # Flushes all NAT/connection tracking. Existing TCP connections drop;
   # new ones use current routing correctly.

3. Public IP isn’t actually bound on the interface. Common after service moves or config drift:

   ip -4 addr show vmbr0 | grep 46.110.152
   # If your "public IP" isn't listed, traffic to it doesn't reach you.

4. Asymmetric routing. Traffic arrives via WAN A, replies leave via WAN B. Check whether the source IP in your replies matches the destination IP the client used:

   curl --interface 46.110.152.203 https://ifconfig.me
   # Should return 46.110.152.203 (or whatever ISP-side IP maps to it).
   # If it returns something else, your kernel is sourcing from a
   # different IP, which breaks return-path routing.

Scenario: “I’m locked out — Tailscale doesn’t even work”

You normally SSH via Tailscale (100.x.y.z), but the box has no internet at all, so Tailscale can’t reach DERP relays and your tailnet identity is offline. You can’t SSH the box from anywhere.

The IPv6 link-local recovery technique works without any IP config on either side, as long as you can plug into the same physical L2 segment as the server.

# 1. Plug your laptop into any port on the server's switch
#    (must be on the same untagged VLAN as the server — usually any
#    open port on a "flat" switch will work)

# 2. Find your laptop's Ethernet interface
ifconfig | grep -B1 ether    # macOS
ip -br link                  # Linux

# 3. Derive the server's IPv6 link-local from its MAC
#    Server MAC = e4:43:4b:32:53:0c
#    EUI-64 rule: insert ff:fe in the middle, XOR first byte with 0x02
#    e4 XOR 02 = e6 → fe80::e643:4bff:fe32:530c

# 4. Confirm L2 reachability
ping6 -c 2 -I en7 fe80::e643:4bff:fe32:530c%en7
#                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^
#                   IPv6 link-local address      scope = your laptop iface

# 5. SSH in
ssh root@fe80::e643:4bff:fe32:530c%en7

This works because every Linux Ethernet interface auto-generates a deterministic fe80:: address from its MAC. No DHCP, no static IP, no network config required. As long as both ends are on the same L2 segment and the server’s SSH is listening on :: (or 0.0.0.0 — sshd binds both by default), you’re in.

This bypasses the standard “backup SSH cable” technique that requires both sides to have IPv4 addresses on the same subnet. Especially useful when your public /29 has no free IPs to give a laptop.

Recovery techniques

Atomic default-route swap (failover)

Use ip route replace, not del + add. The kernel processes replace as a single netlink message — there is no zero-default-route window:

# Atomic swap to backup gateway
ip route replace default via 192.168.1.1 dev vmbr0

# Always follow up with conntrack flush to clear stale NAT/state
conntrack -F

# Restart anything that needs to re-establish on the new path
systemctl restart tailscaled cloudflared

ip route del default && ip route add default ... works most of the time, but has a microsecond window where outgoing packets get dropped. Don’t use it for automated failover.

Gratuitous ARP (refresh upstream L2 caches)

When an IP rebinds, comes back after an outage, or moves between hosts, upstream switches and gateways may keep stale IP → MAC mappings for minutes. Force-refresh with:

arping -A -c 5 -I <iface> <ip>
# -A = ARP announcement (gratuitous, not a request)
# Sends "I am <ip>, please update your MAC table"

Pure broadcast, advertise-only — safe to run any time. If you’ve just added a secondary IP, always follow with a gratuitous ARP.

Conntrack flush after route changes

Linux’s connection-tracking table keeps NAT and stateful firewall entries pinned to specific source IPs. After a WAN swap, these entries are wrong — outgoing replies try to use the old source IP and get dropped at the new ISP. Flush:

conntrack -F

Existing TCP connections drop immediately (acceptable trade — they were going to die anyway). New connections use current routing.

DHCP probe on an existing-static interface

When you have a backup ISP available on the same physical segment as your primary WAN, you can grab a DHCP lease from it without disrupting your static config:

# Probe DHCP on vmbr0 (which already has a static 46.110.152.203/29)
dhclient -v -1 vmbr0

# After binding, lease IP is added as a secondary on vmbr0:
ip -4 addr show vmbr0
# inet 46.110.152.203/29 scope global vmbr0
# inet 192.168.1.9/24 scope global dynamic vmbr0   ← new lease

# Default route does NOT automatically swap on Debian. Do it manually:
ip route replace default via 192.168.1.1 dev vmbr0
conntrack -F
systemctl restart tailscaled cloudflared

-1 means “one-shot, give up if no offer in 30s.” Useful for diagnostics — if you get no offer, you know the segment doesn’t have a DHCP server, or it’s not L2-reachable from your interface.

Multi-WAN failover (architecture)

Industry pattern for resilient WAN, summarized:

• Active probing through both paths every N seconds — ICMP to 2-3 well-known anycast targets (1.1.1.1, 8.8.8.8, 9.9.9.9). Not just one target (one being down isn’t a path failure).
• Source-based policy routing so probes can use the other path while the default route is still pinned to the primary. Otherwise you can’t tell when the primary recovers.
• Hysteresis: fail after N consecutive misses (e.g., 3), recover after M consecutive hits (e.g., 5) PLUS a cooldown (e.g., 300s in failover before failback). Asymmetric — harder to recover than to fail — prevents flapping on marginal links.
• Atomic route swap with ip route replace + conntrack -F.
• Service restart on transition (tailscaled, cloudflared, anything that needs to re-establish on the new path).

Reference implementations

• mwan3 (OpenWrt) — the canonical SOHO/SMB multi-WAN. Active probes, policy routing, hysteresis. OpenWrt-only runtime.
• keepalived (Linux HA) — vrrp_script + notify_* hooks are widely used as a generic probe-and-react harness, even when you don’t need VRRP. Battle-tested C daemon.
• FRR (FRRouting) with BFD — BGP-grade sub-second detection. Overkill for SOHO but the gold standard for enterprise.
• systemd-networkd with multipath default — handles link-down only, not “gateway alive but upstream broken.” Insufficient on its own.

Source-based PBR for two-path probing

The trick that makes probing both paths simultaneously possible:

# Create a second routing table for the backup path
echo "200 failover_backup" >> /etc/iproute2/rt_tables

# Add backup-path routes in table 200
ip route add 1.1.1.1/32 via 192.168.1.1 dev vmbr0 table 200
ip route add 8.8.8.8/32 via 192.168.1.1 dev vmbr0 table 200
ip route add 9.9.9.9/32 via 192.168.1.1 dev vmbr0 table 200

# Source-based rule: traffic from the DHCP IP uses table 200
ip rule add from 192.168.1.9 lookup 200 priority 200

# Pin the same targets to primary in the main table
ip route add 1.1.1.1/32 via 46.110.152.201 dev vmbr0
ip route add 8.8.8.8/32 via 46.110.152.201 dev vmbr0
ip route add 9.9.9.9/32 via 46.110.152.201 dev vmbr0

# Now probe primary (default routing)
ping -W 2 -c 1 -I vmbr0 8.8.8.8

# And probe backup (forced via source IP)
ping -W 2 -c 1 -I 192.168.1.9 8.8.8.8

Without this, after failover the probes start using the backup path and you can’t tell when primary recovers.

Local implementation

We run wan-failover.service on the Proxmox host at the johncloud site. It implements all of the above. Code lives in servers/johncloud/wan-failover/ in ai-coding-env. Notifications go to a Signal group and via Brevo SMTP to sac@norequirement.com on every state transition.

CLI:

wan-failover status                 # JSON dump of state + current default route
wan-failover lock primary           # pin to primary, no auto failover
wan-failover lock backup            # pin to backup
wan-failover unlock                 # resume auto failover
wan-failover force-failover         # manual swap to backup
wan-failover force-failback         # manual swap to primary
wan-failover test-notify            # fire synthetic notification, no state change

Use lock backup ahead of planned ISP maintenance windows. Use unlock after.

Tools quick-reference

Tool	What it tells you	Example
ip -br link show	Link state per interface (NO-CARRIER = cable/NIC issue)	ip -br link show
ethtool <iface>	Negotiated speed, duplex, link-detect flag	ethtool eth0
ip -4 addr show	All IPv4 addresses on all interfaces	ip -4 addr show vmbr0
ip -4 route	Routing table — what would ip route get X use?	ip -4 route get 1.1.1.1
ip rule	Policy routing rules (which table for which traffic)	ip rule show
ip neigh	ARP cache. INCOMPLETE = ARP resolution failed	ip neigh show dev vmbr0
arping -c N -I <iface> <ip>	L2 reachability + ARP responder check	arping -c 5 -I vmbr0 192.168.1.1
arping -A -c N -I <iface> <ip>	Gratuitous ARP — announce, don’t ask	arping -A -c 5 -I vmbr0 46.110.152.204
tcpdump -i <iface> -nn -e arp	Sniff ARP requests + responses on the wire	tcpdump -i vmbr0 -nn -e arp
ping -c N -I <iface|src-ip> <target>	L3 reachability with source/iface pinning	ping -c 3 -I vmbr0 1.1.1.1
traceroute -n <target>	Hop-by-hop path; reveals where routing dies	traceroute -n 1.1.1.1
nc -zv -w 3 <ip> <port>	TCP-port reachability (refused vs filtered vs open)	nc -zv -w 3 46.110.152.203 443
ss -tlnp	Listening TCP sockets (which process on which port)	ss -tlnp | grep 8443
dig +short <name> @<ns>	DNS resolution, force specific resolver	dig +short example.com @1.1.1.1
curl -v --interface <ip|iface> <url>	HTTP with source/iface pinning	curl -v --interface 46.110.152.203 https://ifconfig.me
conntrack -L / conntrack -F	List / flush kernel connection-tracking state	conntrack -F
mtr -nr <target>	Continuous traceroute, packet loss per hop	mtr -nr 1.1.1.1

Operational discipline (when remote)

A few rules of thumb when troubleshooting a server you can’t physically reach:

• Tailscale / VPN is your only lifeline. Never run a command that could disrupt the path you’re using to SSH. systemctl restart networking from an SSH session over Metronet is how people lose access. Always verify your SSH path’s resilience before changing related infra.
• Read-only diagnostics first. Snapshot state with ip addr, ip route, ip neigh, ss -tlnp, journalctl before any change. Save the snapshot; you’ll want to diff after.
• Hypothesis before action. Write down what you think is broken and what each command will confirm or refute. If the result doesn’t match the hypothesis, stop and revise — don’t just keep trying things.
• Atomic, reversible changes. ip route replace, not del + add. Snapshot files before editing them. Keep .bak copies. Use systemctl reload (config reread, no restart) when supported.
• Make the recovery path independent of the thing you’re fixing. If you’re debugging the WAN, the recovery path must not use that WAN. Hence: Tailscale (independent of which ISP), IPv6 LLA (independent of any IP config), out-of-band IPMI (independent of the OS).
• Don’t power-cycle equipment you can’t see. Reboots take seconds to seem-fixed and minutes to actually-fixed; if something doesn’t come back, you’re now driving on-site.

See also

• Cloudflare Troubleshooting — Cloudflare-specific
• Linux Server Initial Setup
• Linux Networking with Netplan
• Tailscale — context on the lifeline path
• Tailscale Exit Node — how Tailscale tolerates ISP changes
• Self-Host Cloudflare Tunnels — why outbound tunnels survive WAN swaps