iproute2 Mastery — Linux Networking Beyond ifconfig

iproute2 Mastery — Linux Networking Beyond ifconfig

A reference for ip(8) and friends — the modern Linux networking toolkit. Most online tutorials still teach ifconfig and route; those commands have been deprecated for over a decade and miss the majority of features the kernel exposes. If you’ve ever found yourself wondering how multi-WAN, VPN split-tunneling, or container networking actually works under the hood, this is the toolkit doing it.

Who is this for?

Anyone who runs a Linux server, configures non-trivial networking, or needs to debug network failures past the “is my cable plugged in” stage. Especially useful if you’re moving from ifconfig-era knowledge to modern Linux, or building custom failover / routing / firewalling infrastructure.

Why iproute2 (and not ifconfig)

ifconfig, route, arp, and netstat are part of net-tools — deprecated, unmaintained, missing modern features. iproute2 is what replaced them, ships with the kernel, and exposes the actual data structures the kernel uses.

Old (net-tools)	New (iproute2)	What’s new
ifconfig	ip addr, ip link	Separates link state from L3 addresses
route	ip route	Multiple routing tables, atomic operations, policy rules
arp	ip neigh	IPv6 NDP unified with ARP; richer state
netstat	ss	Faster, modern netlink, BPF-aware
(none)	ip rule	Policy-based routing (PBR)
(none)	ip monitor	Real-time netlink event stream
(none)	tc	Traffic control / QoS

The mental model also changes. Where ifconfig mashed link state and L3 addresses into one command, iproute2 separates concerns:

• Link (L1/L2 state) — ip link
• Address (L3 binding) — ip addr
• Route (L3 path decisions) — ip route
• Rule (which routing table to consult) — ip rule
• Neighbor (ARP/NDP cache) — ip neigh

Each object lives in its own namespace and you query/manipulate them independently. This is also why the same commands work for both IPv4 and IPv6 — just add -4 or -6.

Don’t mix tools

Don’t use ifconfig eth0 192.168.1.10 and then ip addr show eth0 expecting clean state. The old tools can usually read modern kernel state, but they don’t represent it faithfully (e.g., they hide secondary addresses). Pick iproute2 and stay there.

Networking primer — terms used throughout this doc

If any of these terms are new, skim once and the rest of the doc will read cleanly. iproute2 mostly operates at L1-L3; everything below is context.

The OSI / TCP-IP layer model (the “L1/L2/L3” shorthand)

Networking is conventionally split into stacked layers, each solving a different problem. When you see “L2 this” or “L3 that,” this is what they mean:

Layer	Name	Examples	What can fail
L1	Physical	Cable, NIC chip, fiber, wireless	Cable unplugged, broken NIC, dead switch port
L2	Data link	Ethernet frames, MAC addresses, ARP, VLANs, switches	Switch loop, wrong VLAN, broken ARP
L3	Network	IP addresses, routing, ICMP, gateways	Wrong IP, no route, unreachable gateway
L4	Transport	TCP, UDP, ports	Firewall block, app not listening
L5-L7	Session / App	HTTP, SSH, DNS, TLS	Service config, DNS lookup, cert

iproute2 lives at L1-L3. ss covers L4 listening sockets. dig / curl cover L7. The phrase “L3 state” means “the IP addresses, routes, and policy rules the kernel currently has loaded” — exactly what ip addr and ip route show you.

Addresses

• MAC address (L2) — 48-bit ID assigned to every Ethernet NIC at the factory. Format: e4:43:4b:32:53:0c. Switches forward Ethernet frames based on MAC addresses.
• IP address (L3) — Numeric ID used for routing across networks. IPv4 is 32-bit (192.168.1.10), IPv6 is 128-bit (fe80::1).
• CIDR notation — 192.168.1.0/24 means “the network whose first 24 bits are fixed, leaving 8 bits for host IDs.” That’s 256 total addresses, 254 usable (the first is the network number, the last is broadcast). Quick rules of thumb: /24 = 256, /25 = 128, /26 = 64, /27 = 32, /28 = 16, /29 = 8, /30 = 4, /32 = a single host.

Routing concepts

• Gateway (a.k.a. next hop) — Where to send a packet whose destination isn’t on your local network. Usually your router’s IP.
• Default route — Catch-all: “send packets for anywhere I don’t otherwise have a route for, to this gateway.” Most hosts have exactly one.
• Host route — A route for a single IP (/32 for IPv4, /128 for IPv6). Useful to pin specific destinations to specific paths (e.g., always route 8.8.8.8 via ISP A, even if the default goes via ISP B).
• On-link — The destination is on the same L2 segment as you, so no gateway is needed — just ARP for its MAC. Routes added by the kernel for your own subnet are on-link by definition.
• Scope — Linux’s marker for how far a route or address can reach:
  • scope link — only valid on this network segment
  • scope global — routable anywhere, normal traffic
  • scope host — only addressable from this machine (e.g., 127.0.0.1)

L2 resolution protocols

• ARP (Address Resolution Protocol, IPv4) — Broadcasts “who has IP X? tell me your MAC.” Cached in the kernel’s neighbor table (ip neigh). When ARP fails, L2 can’t deliver the frame even if L3 routing is correct.
• NDP (Neighbor Discovery Protocol, IPv6) — IPv6’s equivalent of ARP, plus router advertisements and address autoconfiguration. Same command (ip neigh) shows both.
• Gratuitous ARP — Unsolicited “I am IP X, here is my MAC.” Useful to refresh stale ARP caches at upstream routers when an IP moves or rebinds (e.g., after WAN failover).

Kernel internals

• Netlink — The protocol Linux uses for the kernel to talk to userspace about network state. ip is a netlink client — every ip ... command sends a netlink message and reads the kernel’s reply. “Single netlink message” means atomic from the kernel’s perspective — no intermediate state visible to anything else.
• Network namespace — An isolated copy of the entire network stack (interfaces, routes, firewall). Containers and VMs use these for isolation. Manipulated with ip netns ....
• Bridge — A virtual L2 switch implemented in the kernel. Multiple interfaces can be members of one bridge; the bridge forwards frames between them as if they were ports on a physical switch. Proxmox’s vmbr0 is a bridge.

Other useful terms

• VLAN (Virtual LAN) — Tagged L2 segments that let one physical cable carry multiple logical networks. Identified by 12-bit IDs (1-4094). “Untagged VLAN 1” is the default everywhere.
• MTU (Maximum Transmission Unit) — The largest packet size a link will accept (1500 bytes for standard Ethernet; 9000 for “jumbo frames”).
• ICMP — Control protocol at L3, used by ping and traceroute. Separate from your TCP/UDP traffic — some firewalls block ICMP while permitting TCP, which makes ping lie about reachability.
• EUI-64 — Rule for deriving a 64-bit IPv6 interface identifier from a 48-bit MAC. Used to auto-generate IPv6 link-local addresses (fe80::/64). The procedure: insert ff:fe in the middle of the MAC, then XOR the first byte with 0x02 (to flip the “universal / local” bit). For MAC e4:43:4b:32:53:0c:
  • Insert ff:fe in the middle → e4:43:4b:ff:fe:32:53:0c
  • XOR first byte: e4 XOR 02 = e6 → e6:43:4b:ff:fe:32:53:0c
  • That’s the bottom 64 bits of fe80::e643:4bff:fe32:530c

This rule is the reason every Linux Ethernet interface auto-generates a deterministic fe80:: address from its MAC — and the reason that address is a universal recovery backdoor for any Linux box on the same L2 segment, without any IP config required on either side.

Policy-routing vocabulary (introduced fully later)

• PBR (Policy-Based Routing) — Routing decisions based on something other than just the destination IP. Examples: route based on source IP, firewall mark, or input interface. Implemented via ip rule + multiple routing tables.
• fwmark — A 32-bit “firewall mark” attached to a packet by iptables/nftables. ip rule can match on fwmark to send marked traffic through different routes.
• Reverse-path filter (rp_filter) — Kernel sanity check: drops a packet if the source IP looks like it came in “the wrong interface.” Defaults vary; “strict” mode breaks asymmetric routing and multi-WAN setups. We’ll come back to it.

The five objects

ip link — interfaces (L1/L2 state)

# Compact list (most useful single command)
ip -br link show
#  lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
#  eth0             UP             aa:bb:cc:dd:ee:ff <BROADCAST,MULTICAST,UP,LOWER_UP>
#  eth1             DOWN           aa:bb:cc:dd:ee:f0 <BROADCAST,MULTICAST>

# Full detail
ip link show eth0

# Statistics (RX/TX bytes, errors, drops)
ip -s link show eth0

# Bring an interface up/down
ip link set eth0 up
ip link set eth0 down

# Change MTU
ip link set eth0 mtu 9000

# Change MAC (rare — usually a bad idea)
ip link set eth0 address 02:00:00:11:22:33

Key flags to look for in ip link show:

• UP — the admin state (“operator wants this interface running”)
• LOWER_UP — the carrier state (“physical layer reports link is alive”)
• NO-CARRIER — admin-up but no physical link (cable issue, switch port off)
• BROADCAST,MULTICAST — capability flags, mostly informational

A normal working interface shows <BROADCAST,MULTICAST,UP,LOWER_UP>. If you see UP without LOWER_UP, the OS is trying but the cable/NIC won’t link.

ip addr — IP addresses on interfaces

# All addresses on all interfaces
ip addr show
ip -4 addr show              # IPv4 only
ip -6 addr show              # IPv6 only
ip -br -4 addr show          # compact one-line-per-interface

# One interface
ip addr show eth0

# Add an address
ip addr add 192.168.1.10/24 dev eth0
ip addr add 192.168.1.11/24 dev eth0 label eth0:secondary

# Remove an address
ip addr del 192.168.1.10/24 dev eth0

# Replace (idempotent — adds if missing, no error if present)
ip addr replace 192.168.1.10/24 dev eth0

Multiple addresses on one interface are a first-class concept (not “aliases” like in old ifconfig days). Every address is just another L3 binding on the same L2 interface — they all share the same MAC, and the kernel routes between them via the routing table.

# A "secondary" IP is just another addr add
ip addr add 46.110.152.204/29 dev vmbr0 label vmbr0:jellyfin

Scopes matter for traffic decisions:

• scope global — routable, normal
• scope link — link-local only (e.g., a /29 segment)
• scope host — only addressable from this host (e.g., 127.0.0.1)

You rarely set scope manually; it’s derived from the network. But seeing scope link in ip route output tells you “this network is on-link, no gateway needed.”

ip route — routing table

This is where most people spend most of their time.

# Show the main routing table
ip route show

# IPv4 / IPv6 specifically
ip -4 route show
ip -6 route show

# Just the default route
ip route show default
ip -4 route show default

# Which route would the kernel use for a specific destination?
ip route get 1.1.1.1
ip route get 1.1.1.1 from 192.168.1.10    # with source IP

# Add a route
ip route add 10.0.0.0/24 via 192.168.1.1 dev eth0
ip route add default via 192.168.1.1 dev eth0

# Add a host route (single /32)
ip route add 8.8.8.8/32 via 192.168.1.1 dev eth0

# Remove a route
ip route del 10.0.0.0/24

# Atomic replace (no zero-window!)
ip route replace default via 192.168.1.1 dev eth0

Use ip route replace, not del + add

ip route replace is processed as a single netlink message — there is zero time during which the default route is missing. del followed by add has a microsecond window where the kernel has no route. For production failover, always use replace. Same applies to ip addr replace and ip neigh replace.

Reading a routing-table line:

default via 192.168.1.1 dev eth0 proto static src 192.168.1.10 metric 100

• default — destination (here: 0.0.0.0/0)
• via 192.168.1.1 — next-hop gateway IP
• dev eth0 — send out this interface
• proto static — added by admin (vs dhcp, kernel, boot)
• src 192.168.1.10 — use this source IP when sending via this route
• metric 100 — preference when multiple routes match (lower wins)

proto kernel scope link for a directly-connected network — kernel adds these automatically when you add an address.

ip rule — policy routing rules

Most servers use only the main routing table. But Linux supports up to 255 routing tables, and ip rule decides which table to consult for each packet.

# Show current rules (sorted by priority — lower = higher priority)
ip rule show

# Default rules on a fresh system:
#   0:      from all lookup local      <- system table, do not touch
#   32766:  from all lookup main       <- the normal routing table
#   32767:  from all lookup default    <- "default" table (usually empty)

The kernel walks rules in priority order. First match wins; that table is consulted. If a rule matches but its table has no matching route, the kernel keeps walking.

# Add a rule: "traffic from 10.0.0.5 looks up table 200"
ip rule add from 10.0.0.5 lookup 200 priority 100

# Add a rule with firewall mark (often used with iptables -j MARK)
ip rule add fwmark 0x42 lookup 100 priority 101

# Add a rule for a destination range
ip rule add to 10.0.0.0/8 lookup 200 priority 102

# Remove
ip rule del from 10.0.0.5 lookup 200

When you need policy routing

Most networks don’t need it. You need PBR when:

• Multi-WAN failover — probe targets through both ISPs simultaneously
• VPN split-tunnel — only some traffic uses the VPN
• Container isolation — a container’s traffic uses a different gateway
• Source-IP-based egress — traffic from a specific IP uses a specific path

If you’re doing any of these, you’ll be using ip rule. If you’re not, the main table is enough.

ip neigh — ARP/NDP cache

# Show the neighbor cache (combines ARP for IPv4, NDP for IPv6)
ip neigh show
ip neigh show dev eth0
ip -4 neigh show
ip -6 neigh show

# Add a static entry (rarely needed)
ip neigh add 192.168.1.1 lladdr aa:bb:cc:dd:ee:ff dev eth0

# Remove
ip neigh del 192.168.1.1 dev eth0

# Flush the entire cache
ip neigh flush dev eth0
ip -4 neigh flush all

States to know:

• REACHABLE — recently confirmed, fully valid
• STALE — older but kernel thinks it’s still good
• DELAY / PROBE — kernel is re-confirming
• INCOMPLETE — ARP/NDP was sent but no reply; resolution failed
• FAILED — gave up; subsequent sends will retry resolution
• PERMANENT — manually configured static entry

If ip neigh show 192.168.1.1 reports INCOMPLETE, the kernel can’t resolve the MAC for that IP. Either the host is offline, ARP responses are being blocked, or you’re on the wrong segment. See the network troubleshooting guide for diagnosis.

Multiple routing tables

Routing tables live in /etc/iproute2/rt_tables. The defaults:

255  local
254  main
253  default
0    unspec

You can add named tables for your own use:

echo "200 failover_backup" >> /etc/iproute2/rt_tables
echo "201 vpn"             >> /etc/iproute2/rt_tables

Then use the names in commands:

ip route add 10.0.0.0/24 via 10.0.0.1 dev wg0 table vpn
ip rule  add fwmark 0x42 lookup vpn priority 100
ip route show table vpn

The naming is purely for human readability — under the hood it’s the number that matters.

Pattern: multi-WAN with simultaneous probing

This is the trick that makes proper WAN failover possible — probing the other path while your default route is still pinned to the current one.

/etc/iproute2/rt_tables

# Add:  200 failover_backup

# Backup path's probe targets in table 200
ip route add 1.1.1.1/32 via 192.168.1.1 dev eth0 table failover_backup
ip route add 8.8.8.8/32 via 192.168.1.1 dev eth0 table failover_backup

# Source-based rule: packets sourced from 192.168.1.9 use table 200
ip rule add from 192.168.1.9 lookup failover_backup priority 200

# Now probe primary (uses main table, default route via 46.110.152.201)
ping -W 2 -c 1 1.1.1.1

# And probe backup (forced via source-IP into table 200)
ping -W 2 -c 1 -I 192.168.1.9 1.1.1.1

Both probes go to the same destination via different paths, in parallel. This is how a failover daemon can detect “primary recovered” without first switching the default route to test.

Pattern: split tunnel via fwmark

Mark packets with iptables, route based on the mark:

# iptables: mark all traffic from a specific source port range
iptables -t mangle -A OUTPUT -p tcp --sport 50000:60000 -j MARK --set-mark 0x42

# ip rule: marked traffic uses table 201
ip rule add fwmark 0x42 lookup 201 priority 101

# table 201: default route via VPN
ip route add default via 10.0.0.1 dev wg0 table 201

Now all TCP traffic with source ports 50000-60000 goes through the VPN. Everything else uses the main table.

Common operations cookbook

”Show me everything about how a packet would be routed”

ip route get 1.2.3.4
# 1.2.3.4 via 192.168.1.1 dev eth0 src 192.168.1.10 uid 1000
#     cache

ip route get 1.2.3.4 from 10.0.0.5
# (forces the lookup with that source IP — may hit a different rule/table)

ip route get 1.2.3.4 mark 0x42
# (forces the lookup with that fwmark)

ip route get resolves through rules → tables → routes and tells you exactly which interface, gateway, and source IP would be used. Use it to debug PBR.

”Add a backup IP that survives a route change”

# Add as a secondary on the bridge / interface
ip addr add 46.110.152.206/29 dev vmbr0 label vmbr0:mail

# Tell upstream we own this IP now
arping -A -c 5 -I vmbr0 46.110.152.206

The address persists for the lifetime of the interface UP state.

”Make a route survive reboot”

Runtime ip route add is not persistent. Pick the right persistence mechanism for your distro:

Stack	Where to put it
ifupdown / ifupdown2 (Debian, Proxmox)	/etc/network/interfaces — use post-up, up, pre-down
netplan (Ubuntu)	/etc/netplan/*.yaml under routes:
systemd-networkd	/etc/systemd/network/*.network [Route] sections
NetworkManager	nmcli connection modify ... +ipv4.routes ...
ifcfg / RHEL legacy	/etc/sysconfig/network-scripts/route-eth0

If you want a truly distro-agnostic systemd unit:

/etc/systemd/system/extra-routes.service

[Unit]
Description=Apply extra static routes
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/sbin/ip route add 10.0.0.0/24 via 192.168.1.1 dev eth0
ExecStart=/sbin/ip route add 10.0.1.0/24 via 192.168.1.1 dev eth0
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

“What’s running on this interface right now?”

# Listening sockets (replaces netstat -tlnp)
ss -tlnp
ss -tlnp | grep eth0          # filter by interface name doesn't work; use addr
ss -tlnp '( sport = :22 )'    # listening on port 22

# Established connections
ss -tnp

# UDP listeners
ss -ulnp

ss reads kernel netlink directly — much faster than netstat, which iterates through /proc.

”Watch the routing table change in real time”

ip monitor route     # all route changes
ip monitor link      # link state changes
ip monitor neigh     # ARP/NDP changes
ip monitor all       # everything

Useful when debugging “why did my route disappear?” — start ip monitor in one terminal, reproduce the issue in another.

IPv6 specifics

All the same commands work with -6. Key differences:

ip -6 addr show
# Every Ethernet interface gets an automatic fe80::/64 link-local
# address derived from MAC via EUI-64

ip -6 route show
# IPv6 routes; default is `default via fe80::xx dev eth0`

ip -6 neigh show
# IPv6 neighbor discovery — same INCOMPLETE/REACHABLE states as ARP

Link-local scope: any fe80:: address requires a scope identifier when used as a destination:

ping6 fe80::1234%eth0
ssh user@[fe80::1234%eth0]

The %eth0 tells the kernel which interface to use. Without it, the kernel doesn’t know which link the address refers to (there’s one fe80::/64 per link).

Universal recovery via IPv6 link-local

Because every Linux interface auto-generates a deterministic fe80:: address from its MAC via EUI-64, you can SSH to any Linux box on the same L2 segment without any IP configuration on your side. See the network troubleshooting guide for the technique.

Reverse-path filtering (rp_filter)

A surprising number of multi-WAN problems trace back to rp_filter, which drops packets whose source IP looks like it came in “the wrong interface” (i.e., the reverse path would go out a different interface).

# Check current setting (per-interface and global)
sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.default.rp_filter
sysctl net.ipv4.conf.eth0.rp_filter

# Values:
# 0 = no filtering
# 1 = strict mode (RFC 3704) — drops if RPF check fails
# 2 = loose mode — drops only if no route at all exists

For multi-WAN or asymmetric routing, set to 2 (loose) or 0:

sysctl -w net.ipv4.conf.all.rp_filter=2
sysctl -w net.ipv4.conf.default.rp_filter=2
# Persist in /etc/sysctl.d/99-multiwan.conf

rp_filter and asymmetric routing

If you have two ISPs on one interface (e.g., a static /29 plus DHCP from a backup router both on vmbr0), strict rp_filter=1 may drop return traffic for the secondary IPs because the kernel’s RPF check finds the “wrong” path. Symptoms: outbound works, inbound dies.

Common pitfalls

1. ip route add doesn’t survive reboot. Persist via your distro’s network stack (see “Make a route survive reboot” above) or a oneshot systemd unit.

2. ifconfig lies about secondary addresses. Old ifconfig shows only one address per interface. If you’ve added secondaries with ip addr add, you need ip addr show to see them all.

3. ip route del requires exact match. If you added the route with a specific metric or source, the del must match all of those attributes. Easier: use ip route replace for changes.

4. ip rule add priorities matter. The kernel walks rules in priority order. A new rule at default priority (32766-1) goes between the default rules. Be explicit: priority 200 etc.

5. DHCP clients can clobber your config. dhclient adds routes and addresses based on the lease. After failover, you may need to re-apply your own static routes. The wan-failover daemon at servers/johncloud/wan-failover/ in ai-coding-env handles this explicitly — read it for a real example.

6. Routing changes don’t update existing connections. A TCP connection established before a route change keeps using the old path (until it dies or retransmits). For clean state after a swap, conntrack -F is the canonical move (see network-troubleshooting).

7. You can’t ip route add to a host that’s not on-link without a gateway. If you want ip route add 8.8.8.8/32 dev eth0 (no via), the host must be directly reachable on eth0’s L2 segment. Almost never the case for public IPs.

8. scope link vs scope global. A route with scope link means “this destination is directly on this link, no gateway needed.” Most default-via-gateway routes are scope global. Mixing these up can produce mysterious “Destination Host Unreachable” errors.

Quick reference

Goal	Command
List interfaces	ip -br link show
List IPv4 addresses	ip -4 addr show
List routes (main table)	ip -4 route show
Show default route	ip route show default
What route for destination?	ip route get 1.2.3.4
What route for src+dst?	ip route get 1.2.3.4 from 10.0.0.5
List policy rules	ip rule show
Show ARP cache	ip neigh show
Add an IP	ip addr add 192.168.1.10/24 dev eth0
Add a route	ip route add 10.0.0.0/24 via 192.168.1.1 dev eth0
Atomic-swap default	ip route replace default via X dev Y
Add a rule	ip rule add from 10.0.0.0/24 lookup 200 priority 100
List tables	cat /etc/iproute2/rt_tables
Show table N	ip route show table N
Watch link/route changes	ip monitor all
Listening sockets	ss -tlnp
Flush ARP cache	ip neigh flush all
Bring iface up/down	ip link set eth0 up|down
Change MTU	ip link set eth0 mtu 1500

Going deeper

• LARTC — Linux Advanced Routing & Traffic Control HOWTO — the foundational free book. Slightly dated but still the canonical reference for policy routing and traffic control.
• man 8 ip + man 8 ip-route + man 8 ip-rule — comprehensive and underrated.
• Kernel’s Documentation/networking/ — read the source files for policy_routing.txt, bonding.txt, etc.
• systemd-networkd documentation — modern Linux network configuration uses these abstractions.

See also

• Network Troubleshooting — diagnostic methodology using these tools
• Linux Networking with Netplan — declarative configuration that compiles down to iproute2 state
• Linux Server Initial Setup
• Tailscale — Tailscale uses iproute2 heavily under the hood (creates a tailscale0 interface with its own routes + rules)
• Self-Host Cloudflare Tunnels