[dn42] AS4242420263 configuration - Multi-{pop,homing} (Part 3/5)

Posted on Aug 14, 2024

Table of contents:

This article is the 3rd part of my dn42 experiment, it describes the configuration of a second router (“multi-pop”) and the addition of more BGP peering links (“multi-homing”), using an internal peering link between both routers.

As a reminder, all my dn42 routers configurations are backed-up in this github repository. Mandatory credits: this blog post from jlu5.com helped me greatly for this part, also Lan Tian’s blog contains many awesome posts about his dn42 set-up.

Reminder - Network information

ASN	Block	Primary name server	Secondary name server
`AS4242420263`	`fd28:7515:7d51::/48`	`fd28:7515:7d51:a::1`	`fd28:7515:7d51:c::1`
`AS4242420263`	`172.22.144.160/27`	`172.22.144.161`	`172.22.144.177`

I’ve split my blocks in 4 equal size subnets:

Desc.	IPv6 Network	IPv6 Gateway	IPv4 Network	IPv4 Gateway
`gw`	`fd28:7515:7d51:a::/64`	`fd28:7515:7d51:a::1/64`	`172.22.144.160/29`	`172.22.144.161/29`
`gw`	`fd28:7515:7d51:b::/64`	`fd28:7515:7d51:b::1/64`	`172.22.144.168/29`	`172.22.144.169/29`
`gw2`	`fd28:7515:7d51:c::/64`	`fd28:7515:7d51:c::1/64`	`172.22.144.176/29`	`172.22.144.177/29`
`gw2`	`fd28:7515:7d51:d::/64`	`fd28:7515:7d51:d::1/64`	`172.22.144.184/29`	`172.22.144.185/29`

All links and IPs used are visible on this network diagram:

Pre-requisites

I use 2 virtual machines running Debian 12: gw-dn42 and gw2-dn42, each with 384MB of memory, a 10GB disk for the system, 1 (slow) vcpu and a network interface configured with a public IPv6 and a NAT-ed private IPv4.

Desc.	Public IPv6	Private IPv4
`gw-dn42`	`2001:bc8:3feb:100::9`	`192.168.0.9`
`gw2-dn42`	`2001:bc8:3feb:100::6`	`192.168.0.6`

As described in the previous post, on each VM:

set-up the kernel parameters
install wireguard and generate a set of public/private keys

install and configure bird2, without any peering sessions on gw2-dn42. Re-use the configuration of gw-dn42, with the exception of the file /etc/bird/variables.conf where OWNIP and OWNIPv6 must be adapted.

# Configuration variables
#########################

define OWNAS       = 4242420263;

define OWNIP       = 172.22.144.177;
define OWNNET      = 172.22.144.160/27;
define OWNNETSET   = [172.22.144.160/27+];

define OWNNETv6    = fd28:7515:7d51::/48;
define OWNIPv6     = fd28:7515:7d51:c::1;
define OWNNETSETv6 = [fd28:7515:7d51::/48+];

# BGP community
define BANDWIDTH   = 24;   # >= 100Mbps
define LINKTYPE    = 34;   # encrypted with safe vpn solution with PFS (Perfect Forward Secrecy)
define REGION      = 41;   # Europe
define COUNTRY     = 1250; # France

Establish a wireguard tunnel between both servers

Create the file /etc/wireguard/wg-peer-int.conf

on gw-dn42

  [Interface]
  PrivateKey = **gw-dn42 wireguard private key**
  ListenPort = 51850
  PostUp = /sbin/ip addr add dev %i fe80::201/128 peer fe80::202/128
  PostUp = /sbin/ip addr add dev %i 172.22.144.161/29 peer 172.22.144.177/29
  PostUp = /sbin/ip addr add dev %i fd28:7515:7d51:a::1/64 peer fd28:7515:7d51:c::1/64
  Table = off

  [Peer]
  PublicKey = **gw2-dn42 wireguard public key**
  Endpoint = 2001:bc8:3feb:100::6:51850
  PersistentKeepalive = 30
  AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10, ff02::1:6/128

on gw2-dn42

  [Interface]
  PrivateKey = **gw2-dn42 wireguard private key**
  ListenPort = 51850
  PostUp = /sbin/ip addr add dev %i fe80::202/128 peer fe80::201/128
  PostUp = /sbin/ip addr add dev %i 172.22.144.177/29 peer 172.22.144.161/29
  PostUp = /sbin/ip addr add dev %i fd28:7515:7d51:c::1/64 peer fd28:7515:7d51:a::1/64
  Table = off

  [Peer]
  PublicKey = **gw-dn42 wireguard public key**
  Endpoint = 2001:bc8:3feb:100::9:51850
  PersistentKeepalive = 30
  AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10, ff02::1:6/128

Start the tunnel: systemctl enable --now wg-quick@wg-peer-int

Note that:

A link-local IPv6 is set-up on the internal link
the router IPs are not set on a loopback interface anymore, I set them on the internal peering link
in the AllowedIPs lists, ff02::1:6/128 is a special multicast address used by the Babel protocol. If you use OSPF, add ff02::5/128 instead.

Prepare wireguard end-points for future hosts

We will use these end-points to connect extra clients and servers (like the LibreNMS monitoring server in part 5/5).

Create the file /etc/wireguard/wg-hosts.conf

on gw-dn42

  [Interface]
  PrivateKey = **gw-dn42 wireguard private key**
  ListenPort = 51900
  PostUp = /sbin/ip addr add dev %i 172.22.144.169/29
  PostUp = /sbin/ip addr add dev %i fd28:7515:7d51:b::1/64
  Table = off

  [Peer]
  PrivateKey = **gw2-dn42 wireguard public key**
  PersistentKeepalive = 30
  AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10

on gw2-dn42

  [Interface]
  PrivateKey = **gw2-dn42 wireguard private key**
  ListenPort = 51900
  PostUp = /sbin/ip addr add dev %i 172.22.144.185/29
  PostUp = /sbin/ip addr add dev %i fd28:7515:7d51:d::1/64
  Table = off

  [Peer]
  PrivateKey = **gw-dn42 wireguard public key**
  PersistentKeepalive = 30
  AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10

systemctl enable --now wg-quick@wg-hosts

[IGP] Set-up Babel to propagate the internal network routes

We use an IGP to handle the routes with our internal network between gw-dn42 and gw2-dn42. An IGP such as OSPF or Babel will detect the topology of the internal network and will set the routes accordingly. It should never be used to forward routes acquired from BGP, or else be prepared for a very impressive failure.

I’ve finally chosen Babel, which has the great advantage of working out of the box with bird2 with a very simple configuration.

To go further, we could investigate the use of Babel-RTT, which seems well suited for overlay networks such as dn42 if we had more distant gateways to the network in the future.

To enable Babel on the internal network, this section is added to the end of the main bird configuration file /etc/bird/bird.conf on gw-dn42 and gw2-dn42

# Babel - Internal network
##########################

# Include directly connected networks
protocol direct {
    ipv4;
    ipv6;
    interface "wg-hosts";
    interface "wg-peer-int";
};

protocol babel int_babel {
    ipv4 {
        # Only route our internal network using Babel - ignore
        # everything sent from BGP.
        import where is_self_net() && source != RTS_BGP;
        export where is_self_net() && source != RTS_BGP;
    };
    ipv6 {
        import where is_self_net_v6() && source != RTS_BGP;
        export where is_self_net_v6() && source != RTS_BGP;
    };

    interface "wg-peer-int" {
        # Note: Babel's cost metric is slightly different from BGP and OSPF.
        # rxcost specifies the cost for the neighbour to send traffic to us,
        # not the cost to send TO that neighbour. Of course, this won't have
        # any impact if you keep costs consistent on both ends.
        rxcost 123;
    };

};

This is sufficient to share the routes of the network configured on the network interfaces wg-hosts and wg-peer-int.

Bird can reload its configuration gracefully with the command birdc configure

birdc configure
BIRD 2.0.12 ready.
Reading configuration from /etc/bird/bird.conf
Reconfigured

Verify the state of the babel protocol, all subnets should be visible.

birdc show babel entries
BIRD 2.0.12 ready.
int_babel:
Prefix                   Router ID               Metric Seqno  Routes Sources
172.22.144.160/29        00:00:00:00:ac:16:90:a1      0     1       0       0
172.22.144.160/27        00:00:00:00:ac:16:90:a1      0     1       1       0
172.22.144.176/29        00:00:00:00:ac:16:90:b1    123     1       1       1
172.22.144.168/29        00:00:00:00:ac:16:90:a1      0     1       0       0
172.22.144.184/29        00:00:00:00:ac:16:90:b1    123     1       1       1
fd28:7515:7d51::/48      00:00:00:00:ac:16:90:a1      0     1       1       0
fd28:7515:7d51:a::/64    00:00:00:00:ac:16:90:a1      0     1       0       0
fd28:7515:7d51:b::/64    00:00:00:00:ac:16:90:a1      0     1       0       0
fd28:7515:7d51:c::/64    00:00:00:00:ac:16:90:b1    123     1       1       1
fd28:7515:7d51:d::/64    00:00:00:00:ac:16:90:b1    123     1       1       1

Verify that the internal routes are added to the system routing table, in example on gw2-dn42

ip route
...
172.22.144.160/29 dev wg-peer-int proto kernel scope link src 172.22.144.177
172.22.144.160/29 via 172.22.144.161 dev wg-peer-int proto bird src 172.22.144.177 metric 32 onlink
172.22.144.168/29 via 172.22.144.161 dev wg-peer-int proto bird src 172.22.144.177 metric 32 onlink
172.22.144.176/29 dev wg-peer-int proto bird scope link src 172.22.144.177 metric 32
172.22.144.184/29 dev wg-hosts proto kernel scope link src 172.22.144.185
172.22.144.184/29 dev wg-hosts proto bird scope link src 172.22.144.177 metric 32
...

ip -6 route
...
fd28:7515:7d51:a::1 dev wg-peer-int proto kernel metric 256 pref medium
fd28:7515:7d51:a::/64 via fe80::201 dev wg-peer-int proto bird src fd28:7515:7d51:c::1 metric 32 pref medium
fd28:7515:7d51:b::/64 via fe80::201 dev wg-peer-int proto bird src fd28:7515:7d51:c::1 metric 32 pref medium
fd28:7515:7d51:c::/64 dev wg-peer-int proto bird src fd28:7515:7d51:c::1 metric 32 pref medium
fd28:7515:7d51:c::/64 dev wg-peer-int proto kernel metric 256 pref medium
fd28:7515:7d51:d::/64 dev wg-hosts proto bird src fd28:7515:7d51:c::1 metric 32 pref medium
fd28:7515:7d51:d::/64 dev wg-hosts proto kernel metric 256 pref medium
fe80::201 dev wg-peer-int proto kernel metric 256 pref medium
fe80::202 dev wg-peer-int proto kernel metric 256 pref medium
...

[iBGP] Set-up an internal BGP peering link between gw1 & gw2

⚠ iBGP requires gateways to be connected in a full-mesh, so the configuration is more complex with more than 2 gateways, and this is not described here. As an alternative, route reflectors (FRR) can be used, see the burble.dn42 network design as an example <https://dn42.burble.com/network/design/>.

Create a new file /etc/bird/bgp_peers/gw2-dn42 on gw-dn42:

protocol bgp ibgp_gw2 {
    local as OWNAS;
    neighbor fd28:7515:7d51:c::1 as OWNAS;

    ipv4 {
        import where source = RTS_BGP && is_valid_network() && !is_self_net();
        export where source = RTS_BGP && is_valid_network() && !is_self_net();
        next hop self;
        extended next hop on;
    };

    ipv6 {
        import where source = RTS_BGP && is_valid_network_v6() && !is_self_net_v6();
        export where source = RTS_BGP && is_valid_network_v6() && !is_self_net_v6();
        next hop self;
        extended next hop off;
    };
}

Create a new file /etc/bird/bgp_peers/gw-dn42 on gw2-dn42:

protocol bgp ibgp_gw {
    local as OWNAS;
    neighbor fd28:7515:7d51:a::1 as OWNAS;

    ipv4 {
        import where source = RTS_BGP && is_valid_network() && !is_self_net();
        export where source = RTS_BGP && is_valid_network() && !is_self_net();
        next hop self;
        extended next hop on;
    };

    ipv6 {
        import where source = RTS_BGP && is_valid_network_v6() && !is_self_net_v6();
        export where source = RTS_BGP && is_valid_network_v6() && !is_self_net_v6();
        next hop self;
        extended next hop off;
    };
}

Reload the bird configuration on both servers

birdc configure
BIRD 2.0.12 ready.
Reading configuration from /etc/bird/bird.conf
Reconfigured

check the state of the iBGP session

birdc show protocols all ibgp_gw2
BIRD 2.0.12 ready.
Name       Proto      Table      State  Since         Info
ibgp_gw2   BGP        ---        up     2024-07-27    Established
  BGP state:          Established
    Neighbor address: fd28:7515:7d51:c::1
    Neighbor AS:      4242420263
    Local AS:         4242420263
    Neighbor ID:      172.22.144.177
    Local capabilities
      Multiprotocol
        AF announced: ipv4 ipv6
      Route refresh
      Extended next hop
        IPv6 nexthop: ipv4
      Graceful restart
      4-octet AS numbers
      Enhanced refresh
      Long-lived graceful restart
    Neighbor capabilities
      Multiprotocol
        AF announced: ipv4 ipv6
      Route refresh
      Extended next hop
        IPv6 nexthop: ipv4
      Graceful restart
      4-octet AS numbers
      Enhanced refresh
      Long-lived graceful restart
    Session:          internal multihop AS4
    Source address:   fd28:7515:7d51:a::1
    Hold timer:       178.554/240
    Keepalive timer:  57.525/80
  Channel ipv4
    State:          UP
    Table:          master4
    Preference:     100
    Input filter:   (unnamed)
    Output filter:  (unnamed)
    Routes:         474 imported, 723 exported, 22 preferred
    Route change stats:     received   rejected   filtered    ignored   accepted
      Import updates:         279277          0          0      25307     253970
      Import withdraws:        91141          0        ---        246      90895
      Export updates:        1823986      41331          5        ---    1782650
      Export withdraws:         1005        ---        ---        ---      30770
    BGP Next hop:   fd28:7515:7d51:a::1
    IGP IPv4 table: master4
    IGP IPv6 table: master6
  Channel ipv6
    State:          UP
    Table:          master6
    Preference:     100
    Input filter:   (unnamed)
    Output filter:  (unnamed)
    Routes:         448 imported, 736 exported, 18 preferred
    Route change stats:     received   rejected   filtered    ignored   accepted
      Import updates:         168703          0          0      13363     155340
      Import withdraws:        42984          0        ---        145      42839
      Export updates:         230669      60289          5        ---     170375
      Export withdraws:         2343        ---        ---        ---      18928
    BGP Next hop:   fd28:7515:7d51:a::1
    IGP IPv6 table: master6

On gw2-dn42, the external routes should be exchanged and added on the system.

birdc show route all
...
172.20.209.128/26    unicast [ibgp_gw 2024-08-11 from fd28:7515:7d51:a::1] * (100/123) [AS4242420293i]
        via fe80::201 on wg-peer-int
        Type: BGP univ
        BGP.origin: IGP
        BGP.as_path: 4242421588 4242422547 4242420293
        BGP.next_hop: fd28:7515:7d51:a::1
        BGP.local_pref: 100
        BGP.community: (64511,3) (64511,24) (64511,34)
                     unicast [whojk_v6 2024-08-11] (100) [AS4242420293i]
...

ip route get 172.20.209.128
172.20.209.128 via inet6 fe80::201 dev wg-peer-int src 172.22.144.177 uid 1000

[eBGP] Set-up an additional eBGP peering link on `gw2-dn42`

Choose an automated peering service and repeat the guide of the previous section on the second server.

Bird will now choose the best route for each external network, it will either route the traffic directly to a local eBGP peer, or relay to the iBGP peer.

Test in “real conditions”

Connect to burble.dn42 shell servers in different terminals

ssh hcartiaux@shell.fr.burble.dn42  # France
ssh hcartiaux@shell.uk.burble.dn42  # United Kingdom
ssh hcartiaux@shell.de.burble.dn42  # Germany
ssh hcartiaux@shell.lax.burble.dn42 # USA - Los Angeles
ssh hcartiaux@shell.nyc.burble.dn42 # USA - New York

Upon connecting to the shell servers, the motd is printed and you have access to a standard ubuntu server in user mode, using a bash shell.

ssh hcartiaux@shell.lax.burble.dn42
Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-107-generic x86_64)

 System information as of Wed 14 Aug 06:39:42 UTC 2024

  System load:             0.01
  Usage of /:              15.8% of 17.88GB
  Memory usage:            16%
  Swap usage:              0%
  Processes:               124
  Users logged in:         0
  IPv4 address for enp6s0: 169.254.0.199
  IPv6 address for enp6s0: 2a04:bdc7:100:1173:216:3eff:feec:78bd

Expanded Security Maintenance for Applications is not enabled.

3 updates can be applied immediately.
To see these additional updates run: apt list --upgradable

16 additional security updates can be applied with ESM Apps.
Learn more about enabling ESM Apps service at https://ubuntu.com/esm


*** System restart required ***
----------------------------------------------------------------------------

This is shell.lax.burble.dn42
See https://burble.dn42/services/shell/ for more details.

The service here is provided for your benefit.
Abuse just spoils the fun for everyone else and will not be tolerated.

If you have problems, or would like additional packages installed 
you can raise an issue here: https://git.dn42.dev/burble/config-shell

----------------------------------------------------------------------------

Last login: Tue Jul 16 21:43:56 2024 from fd28:7515:7d51:b::2
hcartiaux@shell-lax:~$

Ensure all subnets are reachable from the outside

In IPv6:

hcartiaux@shell-lax:~$ for i in a b c d ; do ping -c1 fd28:7515:7d51:$i::1 ; done
PING fd28:7515:7d51:a::1(fd28:7515:7d51:a::1) 56 data bytes
64 bytes from fd28:7515:7d51:a::1: icmp_seq=1 ttl=61 time=268 ms

--- fd28:7515:7d51:a::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 268.251/268.251/268.251/0.000 ms
PING fd28:7515:7d51:b::1(fd28:7515:7d51:b::1) 56 data bytes
64 bytes from fd28:7515:7d51:b::1: icmp_seq=1 ttl=61 time=268 ms

--- fd28:7515:7d51:b::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 268.435/268.435/268.435/0.000 ms
PING fd28:7515:7d51:c::1(fd28:7515:7d51:c::1) 56 data bytes
64 bytes from fd28:7515:7d51:c::1: icmp_seq=1 ttl=61 time=269 ms

--- fd28:7515:7d51:c::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 268.662/268.662/268.662/0.000 ms
PING fd28:7515:7d51:d::1(fd28:7515:7d51:d::1) 56 data bytes
64 bytes from fd28:7515:7d51:d::1: icmp_seq=1 ttl=61 time=271 ms

--- fd28:7515:7d51:d::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 270.629/270.629/270.629/0.000 ms

In IPv4:

for i in 161 169 177 185 ; do ping -c1 172.22.144.$i ; done
PING 172.22.144.161 (172.22.144.161) 56(84) bytes of data.
64 bytes from 172.22.144.161: icmp_seq=1 ttl=61 time=272 ms

--- 172.22.144.161 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 271.902/271.902/271.902/0.000 ms
PING 172.22.144.169 (172.22.144.169) 56(84) bytes of data.
64 bytes from 172.22.144.169: icmp_seq=1 ttl=61 time=270 ms

--- 172.22.144.169 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 270.120/270.120/270.120/0.000 ms
PING 172.22.144.177 (172.22.144.177) 56(84) bytes of data.
64 bytes from 172.22.144.177: icmp_seq=1 ttl=61 time=278 ms

--- 172.22.144.177 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 278.334/278.334/278.334/0.000 ms
PING 172.22.144.185 (172.22.144.185) 56(84) bytes of data.
64 bytes from 172.22.144.185: icmp_seq=1 ttl=61 time=278 ms

--- 172.22.144.185 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 277.573/277.573/277.573/0.000 ms

Use traceroute to shows that the internal peering link is used. We can compare the routes used from the New York and French shell servers.

hcartiaux@fr-shell:~$ traceroute fd28:7515:7d51:a::1
traceroute to fd28:7515:7d51:a::1 (fd28:7515:7d51:a::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:1016::1 (fd42:4242:2601:1016::1)  0.326 ms  0.252 ms  0.200 ms
 2  fr1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::8)  5.088 ms  5.117 ms  4.930 ms
 3  ns1.hcartiaux.dn42 (fd28:7515:7d51:a::1)  19.082 ms  19.264 ms  19.415 ms
hcartiaux@fr-shell:~$ traceroute fd28:7515:7d51:b::1
traceroute to fd28:7515:7d51:b::1 (fd28:7515:7d51:b::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:1016::1 (fd42:4242:2601:1016::1)  0.304 ms  0.254 ms  0.205 ms
 2  fr1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::8)  5.120 ms  5.040 ms  4.942 ms
 3  fd28:7515:7d51:b::1 (fd28:7515:7d51:b::1)  19.296 ms  19.224 ms  19.208 ms
hcartiaux@fr-shell:~$ traceroute fd28:7515:7d51:c::1
traceroute to fd28:7515:7d51:c::1 (fd28:7515:7d51:c::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:1016::1 (fd42:4242:2601:1016::1)  0.246 ms  0.190 ms  0.263 ms
 2  fr1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::8)  4.956 ms  4.842 ms  5.718 ms
 3  ns1.hcartiaux.dn42 (fd28:7515:7d51:a::1)  19.465 ms  19.399 ms  19.313 ms
 4  ns2.hcartiaux.dn42 (fd28:7515:7d51:c::1)  24.329 ms  24.986 ms  25.295 ms
hcartiaux@fr-shell:~$ traceroute fd28:7515:7d51:d::1
traceroute to fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:1016::1 (fd42:4242:2601:1016::1)  0.331 ms  0.274 ms  0.179 ms
 2  fr1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::8)  5.128 ms  5.068 ms  5.021 ms
 3  ns1.hcartiaux.dn42 (fd28:7515:7d51:a::1)  19.805 ms  19.894 ms  19.782 ms
 4  fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1)  24.606 ms  25.292 ms  25.530 ms

hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:a::1
traceroute to fd28:7515:7d51:a::1 (fd28:7515:7d51:a::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.383 ms  0.209 ms  0.067 ms
 2  uk-lon1.burble.dn42 (fd42:4242:2601:35::1)  74.677 ms  74.619 ms  74.481 ms
 3  uk1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::10)  74.791 ms  74.699 ms  74.552 ms
 4  de2.hub.kioubit.dn42 (fdcf:8538:9ad5:1111::3)  93.114 ms  93.004 ms  93.326 ms
 5  fr1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::8)  200.243 ms  200.156 ms  200.053 ms
 6  ns1.hcartiaux.dn42 (fd28:7515:7d51:a::1)  105.237 ms  105.613 ms  105.493 ms
hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:b::1
traceroute to fd28:7515:7d51:b::1 (fd28:7515:7d51:b::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.609 ms  0.504 ms  0.436 ms
 2  uk-lon1.burble.dn42 (fd42:4242:2601:35::1)  74.784 ms  74.823 ms  74.781 ms
 3  uk1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::10)  74.769 ms  74.761 ms  74.691 ms
 4  de2.hub.kioubit.dn42 (fdcf:8538:9ad5:1111::3)  93.270 ms  93.198 ms  93.163 ms
 5  fr1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::8)  105.829 ms  105.715 ms  105.665 ms
 6  fd28:7515:7d51:b::1 (fd28:7515:7d51:b::1)  104.934 ms  105.271 ms  105.197 ms
hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:c::1
traceroute to fd28:7515:7d51:c::1 (fd28:7515:7d51:c::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.408 ms  0.389 ms  0.377 ms
 2  uk-lon1.burble.dn42 (fd42:4242:2601:35::1)  76.548 ms  76.586 ms  76.578 ms
 3  uk1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::10)  77.308 ms  77.023 ms  76.901 ms
 4  de2.hub.kioubit.dn42 (fdcf:8538:9ad5:1111::3)  95.258 ms  95.102 ms  95.061 ms
 5  fr1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::8)  107.934 ms  107.792 ms  107.655 ms
 6  ns1.hcartiaux.dn42 (fd28:7515:7d51:a::1)  107.380 ms  106.623 ms  106.399 ms
 7  ns2.hcartiaux.dn42 (fd28:7515:7d51:c::1)  111.054 ms  110.117 ms  110.094 ms
hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:d::1
traceroute to fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.433 ms  0.388 ms  0.359 ms
 2  uk-lon1.burble.dn42 (fd42:4242:2601:35::1)  74.509 ms  74.555 ms  74.489 ms
 3  uk1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::10)  74.350 ms  74.318 ms  74.381 ms
 4  de2.hub.kioubit.dn42 (fdcf:8538:9ad5:1111::3)  93.558 ms  93.528 ms  93.484 ms
 5  fr1.edge.kioubit.dn42 (fdcf:8538:9ad5:1111::8)  105.797 ms  105.693 ms  105.602 ms
 6  ns1.hcartiaux.dn42 (fd28:7515:7d51:a::1)  105.411 ms  104.802 ms  104.743 ms
 7  fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1)  109.683 ms  110.220 ms  110.941 ms

My best peering link is kioubit, so most networks tend to always be routed via gw-dn42. If I stop all my peering links on gw-dn42, the packets are routed through gw2-dn42 and my SSH connections to the burble shell servers are preserved.

hcartiaux@fr-shell:~$ traceroute fd28:7515:7d51:a::1
traceroute to fd28:7515:7d51:a::1 (fd28:7515:7d51:a::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:1016::1 (fd42:4242:2601:1016::1)  0.420 ms  0.351 ms  0.296 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  17.051 ms  17.039 ms  16.985 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  24.957 ms  24.915 ms  24.807 ms
 4  ns2.hcartiaux.dn42 (fd28:7515:7d51:c::1)  36.257 ms  36.200 ms  36.134 ms
 5  ns1.hcartiaux.dn42 (fd28:7515:7d51:a::1)  37.738 ms  37.718 ms  37.672 ms
hcartiaux@fr-shell:~$ traceroute fd28:7515:7d51:b::1
traceroute to fd28:7515:7d51:b::1 (fd28:7515:7d51:b::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:1016::1 (fd42:4242:2601:1016::1)  0.189 ms  0.157 ms  0.124 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  16.891 ms  16.892 ms  16.861 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  24.642 ms  24.617 ms  24.590 ms
 4  ns2.hcartiaux.dn42 (fd28:7515:7d51:c::1)  38.741 ms  38.728 ms  38.790 ms
 5  fd28:7515:7d51:b::1 (fd28:7515:7d51:b::1)  38.929 ms  38.908 ms  38.887 ms
hcartiaux@fr-shell:~$ traceroute fd28:7515:7d51:c::1
traceroute to fd28:7515:7d51:c::1 (fd28:7515:7d51:c::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:1016::1 (fd42:4242:2601:1016::1)  0.283 ms  0.213 ms  0.160 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  17.245 ms  17.215 ms  17.126 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  25.082 ms  25.024 ms  24.909 ms
 4  ns2.hcartiaux.dn42 (fd28:7515:7d51:c::1)  36.091 ms  36.042 ms  35.972 ms
hcartiaux@fr-shell:~$ traceroute fd28:7515:7d51:d::1
traceroute to fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:1016::1 (fd42:4242:2601:1016::1)  0.250 ms  0.169 ms  0.273 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  16.985 ms  16.939 ms  16.859 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  24.947 ms  24.937 ms  24.874 ms
 4  fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1)  35.838 ms  35.814 ms  36.341 ms

hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:d::1
traceroute to fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.474 ms  0.103 ms  0.118 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  81.659 ms  81.497 ms  81.348 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  89.165 ms  89.219 ms  89.029 ms
 4  fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1)  100.670 ms  100.547 ms  100.318 ms
hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:a::1
traceroute to fd28:7515:7d51:a::1 (fd28:7515:7d51:a::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.384 ms  0.115 ms  0.108 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  81.624 ms  81.528 ms  81.306 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  89.413 ms  89.346 ms  89.231 ms
 4  ns2.hcartiaux.dn42 (fd28:7515:7d51:c::1)  100.769 ms  100.688 ms  100.572 ms
 5  ns1.hcartiaux.dn42 (fd28:7515:7d51:a::1)  102.738 ms  102.610 ms  102.501 ms
hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:b::1
traceroute to fd28:7515:7d51:b::1 (fd28:7515:7d51:b::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.237 ms  0.177 ms  0.112 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  81.507 ms  81.498 ms  81.345 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  89.343 ms  89.268 ms  89.232 ms
 4  ns2.hcartiaux.dn42 (fd28:7515:7d51:c::1)  101.188 ms  101.224 ms  101.117 ms
 5  fd28:7515:7d51:b::1 (fd28:7515:7d51:b::1)  103.506 ms  103.374 ms  103.315 ms
hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:c::1
traceroute to fd28:7515:7d51:c::1 (fd28:7515:7d51:c::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.347 ms  0.339 ms  0.226 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  81.530 ms  81.391 ms  81.344 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  89.436 ms  89.462 ms  89.399 ms
 4  ns2.hcartiaux.dn42 (fd28:7515:7d51:c::1)  100.748 ms  100.717 ms  101.201 ms
hcartiaux@shell-nyc:~$ traceroute fd28:7515:7d51:d::1
traceroute to fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1), 30 hops max, 80 byte packets
 1  fd42:4242:2601:101d::1 (fd42:4242:2601:101d::1)  0.404 ms  0.215 ms  0.103 ms
 2  tier1.de-fra1.burble.dn42 (fd42:4242:2601:31::1)  81.589 ms  81.465 ms  81.198 ms
 3  de.hujk.dn42 (fd94:dba8:42b0:e::1)  89.200 ms  89.105 ms  89.022 ms
 4  fd28:7515:7d51:d::1 (fd28:7515:7d51:d::1)  100.562 ms  100.520 ms  100.476 ms

Follow-up

You will notice that in the traceroute extracts above, my gateways IPs have a reverse domain name. The next post will describe the configuration of a primary and secondary DNS servers, to host my own hcartiaux.dn42 zone and the reverse zones of my network.

Same, but different - using OpenBSD

You can refer to these 3 blog posts to set-up a similar dn42 environment under OpenBSD:

[dn42] AS4242420263 configuration - Multi-{pop,homing} (Part 3/5)

Reminder - Network information

Pre-requisites

Establish a wireguard tunnel between both servers

Prepare wireguard end-points for future hosts

[IGP] Set-up Babel to propagate the internal network routes

[iBGP] Set-up an internal BGP peering link between gw1 & gw2

[eBGP] Set-up an additional eBGP peering link on gw2-dn42

Test in “real conditions”

Follow-up

Same, but different - using OpenBSD

[eBGP] Set-up an additional eBGP peering link on `gw2-dn42`