1. Introduction
Networking is critical in most areas of contemporary computing. To facilitate external and Local Area Network (LAN) networking, applications often employ the Internet Protocol (IP) in combination with routing. In particular, we ensure different endpoints have unique addresses and we know how to reach them. For the latter, systems rely on routing tables with routing entries.
In this tutorial, we explore kernel errors and common issues that we can encounter while configuring a routing entry. First, we briefly refresh our knowledge about routing tables and entries. Next, we go over the main way to modify routes and add route entries. Then, we show examples with a specific tool. After that, we discuss kernel errors and how they relate to routing changes. Finally, we dive into messages specific to routing table handling.
We tested the code in this tutorial on Debian 12 (Bookworm) with GNU Bash 5.2.15. It should work in most POSIX-compliant environments unless otherwise specified.
2. Routing Tables and Entries
The routing table is a structure that contains fields with data used to guide a packet to its final destination by indicating the next hop in relation to the target IP address:
+--------------+---------------+---------------+-----------+--------+
| Destination | Netmask | Gateway | Interface | Metric |
+--------------+---------------+---------------+-----------+--------+
| 10.66.60.0 | 255.255.255.0 | 10.66.60.1 | eth0 | 1 |
| default | 0.0.0.0 | 10.66.60.1 | eth0 | 0 |
| 192.168.66.0 | 255.255.254.0 | 192.168.66.66 | eth1 | 666 |
+--------------+---------------+---------------+-----------+--------+This structure is part of the Linux kernel, so any tool that modifies it works with the same data. Still, a system can have several routing tables.
Let’s break down the three entries above:
- for any destination in the 10.66.60.0 network with subnet mask 255.255.255.0, we send the packet to 10.66.60.1 over network Interface eth0, which costs 1
- for any destination in the 192.168.66.0 network with subnet mask 255.255.254.0, we send the packet to 192.168.66.66 over network Interface eth1, which costs 666
- for all destinations that don’t match another rule, we send the packet to 10.66.60.1 over network Interface eth0
Yet, to add a routing entry, we need to know the value of at least three fields:
- destination
- gateway
- interface
Under the hood, Linux uses a special service for network and routing control messages.
3. RTNETLINK
RTNETLINK is the Linux routing socket. This communication channel works within the kernel but also between the kernel and userspace programs.
Importantly, we can control many networking aspects of a given system via NETLINK and RTNETLINK:
- network routes
- IP addresses
- link parameters
- neighbor setup
- queueing discipline
- traffic classes
- packet classifiers
The messages are based on the netlink (AF_NETLINK) datagram-oriented service. When it comes to routes, NETLINK_ROUTE netlink_family messages control routing and link updates, as well as all other services from the list:
socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);After opening such a socket, we can send the relevant data in the form of a sockaddr structure. This is what many control tools do in the background.
4. Routing Changes With ip
To begin with, let’s see some routing changes we can perform. Since any tool can modify the same internal routing tables, we choose the ip command as the widely-used standard that employs RTNETLINK instead of its deprecated alternative ifconfig which uses ioctl.
In particular, ip provides the route subcommand for our purposes.
4.1. List Routes
First, we can list all current routes:
$ ip route list
default via 10.66.60.1 dev eth0 onlink
10.66.60.0/24 dev eth0 proto kernel scope link src 10.66.60.66Here, we see a default route through interface eth0 via 10.66.60.1 and another route through the same interface. In the latter, src indicates the IP we assign as the source.
Interestingly, the proto kernel part means the kernel installed the route. Should we add our own, it would be a proto static.
The scope shows whether we have a host or link route or employ a broader universe forwarding.
4.2. Add Route
Now, let’s add a route:
$ ip route add <DESTINATION> via <GATEWAY>In this case, we add a next hop GATEWAY for any packet destined for DESTINATION. Since we don’t specify an interface, the first one after the loopback (lo) is selected.
To explicitly choose an interface, we can add the dev option followed by the name of the interface:
$ ip route add <DESTINATION> via <GATEWAY> <dev> INTERFACEOf course, we can do both of the above with a specific host or a whole network as the DESTINATION.
Furthermore, to add a default route, we use the default keyword as the destination:
$ ip route add default via <GATEWAY> <dev> INTERFACEFinally, we can employ all relevant route fields as needed.
Now, let’s explore how route deletion works.
4.3. Remove Route
Using the ip command, we can also delete routes:
$ ip route delete <DESTINATION>In general, we don’t need to specify much else, but we could narrow our selection down with the INTERFACE or GATEWAY and other fields from earlier.
Naturally, we can also work on the default route:
$ ip route delete defaultSo, how do we modify routes?
4.4. Modify Route
We can use change and replace to modify routes:
$ ip route change default via <NEW_GATEWAY> dev eth0This command should switch to the NEW_GATEWAY as the default. However, the behavior of these subcommands isn’t always consistent across systems.
Actually, to change a route, the kernel just deletes and recreates it. No route modification messages exist, mostly due to the overhead this would involve. So, we can just simulate that when we need the functionality.
As with other operations, working with routes may result in errors. Some of these errors involve cryptic messages.
5. Kernel Errors and RTNETLINK answers:
Many messages that we see as error output from ip route begin with RTNETLINK answers:. This line comes from the ip command source code itself:
[...]
default:
perror("RTNETLINK answers");
}
return len;
}
/* check for any messages returned from kernel */
nl_dump_ext_ack(h, NULL);
return 0;
}
[...]The message after the colon shows the kernel answer to our RTNETLINK request. It relates to an error number from the base internal kernel error list:
+------------+-------+--------------------------------------+
| Error Name | Error | Error Text |
+------------+-------+--------------------------------------+
| EPERM | 1 | Operation not permitted |
| ENOENT | 2 | No such file or directory |
| ESRCH | 3 | No such process |
| EINTR | 4 | Interrupted system call |
| EIO | 5 | I/O error |
| ENXIO | 6 | No such device or address |
| E2BIG | 7 | Argument list too long |
| ENOEXEC | 8 | Exec format error |
| EBADF | 9 | Bad file number |
| ECHILD | 10 | No child processes |
| EAGAIN | 11 | Try again |
| ENOMEM | 12 | Out of memory |
| EACCES | 13 | Permission denied |
| EFAULT | 14 | Bad address |
| ENOTBLK | 15 | Block device required |
| EBUSY | 16 | Device or resource busy |
| EEXIST | 17 | File exists |
| EXDEV | 18 | Cross-device link |
| ENODEV | 19 | No such device |
| ENOTDIR | 20 | Not a directory |
| EISDIR | 21 | Is a directory |
| EINVAL | 22 | Invalid argument |
| ENFILE | 23 | File table overflow |
| EMFILE | 24 | Too many open files |
| ENOTTY | 25 | Not a typewriter |
| ETXTBSY | 26 | Text file busy |
| EFBIG | 27 | File too large |
| ENOSPC | 28 | No space left on device |
| ESPIPE | 29 | Illegal seek |
| EROFS | 30 | Read-only file system |
| EMLINK | 31 | Too many links |
| EPIPE | 32 | Broken pipe |
| EDOM | 33 | Math argument out of domain of func |
| ERANGE | 34 | Math result not representable |
+------------+-------+--------------------------------------+In addition, let’s see some of the expanded internal kernel error list:
+-----------------+---------+-----------------------------------------------+
| Error Name | Error | Error Text |
+-----------------+---------+-----------------------------------------------+
| [...] | [...] | [...] |
| ENONET | 64 | Machine is not on the network |
| ENOPKG | 65 | Package not installed |
| EREMOTE | 66 | Object is remote |
| ENOLINK | 67 | Link has been severed |
| EADV | 68 | Advertise error |
| ESRMNT | 69 | Srmount error |
| ECOMM | 70 | Communication error on send |
| EPROTO | 71 | Protocol error |
| EMULTIHOP | 72 | Multihop attempted |
| EDOTDOT | 73 | RFS specific error |
| EBADMSG | 74 | Not a data message |
| EOVERFLOW | 75 | Value too large for defined data type |
| ENOTUNIQ | 76 | Name not unique on network |
| EBADFD | 77 | File descriptor in bad state |
| EREMCHG | 78 | Remote address changed |
| [...] | [...] | [...] |
| ENOTSOCK | 88 | Socket operation on non-socket |
| EDESTADDRREQ | 89 | Destination address required |
| EMSGSIZE | 90 | Message too long |
| EPROTOTYPE | 91 | Protocol wrong type for socket |
| ENOPROTOOPT | 92 | Protocol not available |
| EPROTONOSUPPORT | 93 | Protocol not supported |
| ESOCKTNOSUPPORT | 94 | Socket type not supported |
| EOPNOTSUPP | 95 | Operation not supported on transport endpoint |
| EPFNOSUPPORT | 96 | Protocol family not supported |
| EAFNOSUPPORT | 97 | Address family not supported by protocol |
| EADDRINUSE | 98 | Address already in use |
| EADDRNOTAVAIL | 99 | Cannot assign requested address |
| ENETDOWN | 100 | Network is down |
| ENETUNREACH | 101 | Network is unreachable |
| ENETRESET | 102 | Network dropped connection because of reset |
| ECONNABORTED | 103 | Software caused connection abort |
| ECONNRESET | 104 | Connection reset by peer |
| ENOBUFS | 105 | No buffer space available |
| EISCONN | 106 | Transport endpoint is already connected |
| ENOTCONN | 107 | Transport endpoint is not connected |
| ESHUTDOWN | 108 | Cannot send after transport endpoint shutdown |
| ETOOMANYREFS | 109 | Too many references: cannot splice |
| ETIMEDOUT | 110 | Connection timed out |
| ECONNREFUSED | 111 | Connection refused |
| EHOSTDOWN | 112 | Host is down |
| EHOSTUNREACH | 113 | No route to host |
| EALREADY | 114 | Operation already in progress |
| EINPROGRESS | 115 | Operation now in progress |
| ESTALE | 116 | Stale file handle |
| EUCLEAN | 117 | Structure needs cleaning |
| ENOTNAM | 118 | Not a XENIX named type file |
| ENAVAIL | 119 | No XENIX semaphores available |
| EISNAM | 120 | Is a named type file |
| EREMOTEIO | 121 | Remote I/O error |
| EDQUOT | 122 | Quota exceeded |
| [...] | [...] | [...] |
+-----------------+---------+-----------------------------------------------+The textual part of the ip route errors can come from different sources. For example, the GNU Lib C glibc errlist.h file has the definitions and so does the new Rust part of the Linux kernel in the rust/kernel/rust.rs source file.
In particular, the error is resolved via the nl_dump_ext_ack() -> mnl_nlmsg_get_payload() function chain. Although we can look through the source code of ip route, the actual message comes from RTNETLINK.
Let’s check some common ones and their usual meaning as it relates to NETLINK routing, i.e., RTNETLINK.
6. Common Route Management Errors
At this point, we have a solid understanding of the error mechanism around route modifications. So, let’s see some concrete examples of errors. For each, we can consult the tables above to get their name and number.
Also, we assume a sample routing table is already in the system:
$ ip route list
default via 10.66.60.1 dev eth0 onlink
10.66.60.0/24 dev eth0 proto kernel scope link src 10.66.60.66Based on the current routes, we get specific results with the commands below.
6.1. RTNETLINK answers: No such process
To reproduce the common RTNETLINK answers: No such process error, we can just attempt to delete a route that doesn’t exist:
$ ip route delete 192.0.2.66
RTNETLINK answers: No such processIn general, No such process in this context represents a mismatch between the available values or entries and the supplied value.
6.2. RTNETLINK answers: Network is unreachable
Unlike other errors, RTNETLINK answers: Network is unreachable is fairly self-explanatory:
$ route add default gw 192.0.2.66
RTNETLINK answers: Network is unreachableIn this case, the host has no way to reach the IP address 192.0.2.66 specified as a default gateway. We used the route tool, as it communicates directly via NETFILTER.
On the other hand, contemporary ip versions have their own version of this error and generate it in a different manner:
$ ip route add default via 192.0.2.66
Error: Nexthop has invalid gatewayIn general, the problem is that we specify a nexthop that our host doesn’t know how to reach.
6.3. RNETLINK answers: File exists
Next up, we have an error due to a conflict between routes:
$ ip route add default via 10.66.60.1
RNETLINK answers: File existsHere, we try adding the same gateway we already have. Because of this, RTNETLINK lets us know that this information already exists in the routing table.
6.4. RTNETLINK answers: Invalid argument
As in any other context, an Invalid argument means our values are incorrect in and of themselves based on the type of data expected:
$ ip route add 66.60.0.10/30 via 10.66.60.1
RNETLINK answers: Invalid argumentHere, again, we see the output of an older ip version. Newer implementations of ip use an internal mechanism for such basic checks instead of querying via RTNETLINK:
$ ip route add 66.60.0.10/30 via 10.66.60.1
Error: Invalid prefix for given prefix length.In both cases, the problem comes down to an incorrect network specification, since applying the subnet mask to the IP doesn’t result in the first network address. In general, most cases with such non-matching type and input result in the Invalid argument error.
6.5. RTNETLINK answers: Operation not permitted
As with other administrative tools, many ip route modification operations:
$ ip route add default via 10.66.60.2
RTNETLINK answers: Operation not permittedIn this case, we have a valid command, but the current user or context doesn’t permit makeing the route checks or modifications.
This can occur in subtle situations like Docker containers, where full permissions aren’t the default. In this specific case, we can include the NET_ADMIN capability via –cap-add:
--cap-add=NET_ADMINWhen it comes to regular environments, tools like sudo and su are also helpful.
6.6. RTNETLINK answers: Permission denied
Generally, Permission denied and Operation not permitted look, sound similar in many situations.
However, in the context of RTNETLINK, there’s sometimes an important difference:
$ ip -family inet6 route add default via 2001:db8:666:666::10 dev eth0
RTNETLINK answers: Permission deniedHere, we attempt to add a default gateway for IPv6 via -family inet6. However, the system has no address of that version. In fact, IPv6 is completely disabled in our case:
$ sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
$ sysctl net.ipv6.conf.default.disable_ipv6
net.ipv6.conf.default.disable_ipv6 = 1A similar situation may occur when there is a [prohibit]ed route.
In summary, Permission denied is the result when we don’t have permission to make a given change outside of privilege limitations.
6.7. Other Errors
Some general errors don’t relate to syntax or values but rather external factors:
- Input/output error: usually a hardware problem
- Numerical result out of range: specifying a very long label for an interface
- No space left on device: no free storage on root device for any changes
- Operation not supported: may occur when attempting an operation like working with IPv6 when it’s disabled
- Network is down: self-explanatory, limited access to the network
Knowing these errors, we can make corrections for each one accordingly.
7. Summary
In this article, we explored routing checks and changes, as well as errors that might result from them.
In conclusion, armed with the fundamental knowledge of error sources and the main meaning of many messages, we can adjust our settings and work around problems.