[dn42] AS4242420263 under the hood - more POPs, full mesh network and auto-peering service
Table of contents:
This post describes the internals of my current dn42 network, with the extension from 2 to 7 POPs, and last but not least, the creation of a SSH-based auto-peering service. This post will be edited and kept up-to-date in the future.
New POP servers
I’ve used STARDUST1-S cloud instances from Scaleway to create two new POPs and extend my network.
Each instance cost around 1€/month in IPv6-only, opting out of IPv4 reduces cloud operating costs.
These new servers are also managed by Terraform and Ansible.
Additionally, my friend Gyptazy has granted me several instances on his Boxybsd infrastructure.
| Location | Hostname | Host |
|---|---|---|
| Amsterdam (NL) | nl-ams1.flap42.eu | Scaleway, dedibox/libvirt |
| Amsterdam (NL) | nl-ams2.flap42.eu | Scaleway, dedibox/libvirt |
| Paris (FR) | fr-par1.flap42.eu | Scaleway, cloud |
| Warsaw (PL) | pl-waw1.flap42.eu | Scaleway, cloud |
| New-York (US) | us-nyc1.flap42.eu | BoxyBSD |
| Frankfurt (DE) | de-fra1.flap42.eu | BoxyBSD |
| Toronto (CA) | ca-yto1.flap42.eu | BoxyBSD |
Full-mesh network with Wireguard
All network links are created with Wireguard. All configuration files are generated with Ansible by iterating on the inventory. The generated configuration files are backed-up in the git repository dn42-as4242420263, in the directories servers/*/wireguard (in example for fr-par1).
Each POP is assigned with an IPv6 and IPv4 subnet.
| Hostname | IPv6 Network | IPv6 Gateway | IPv4 Network | IPv4 Gateway |
|---|---|---|---|---|
nl-ams1 | fd28:7515:7d51:a::/64 | fd28:7515:7d51:a::1 | 172.22.144.160/29 | 172.22.144.161 |
nl-ams2 | fd28:7515:7d51:b::/64 | fd28:7515:7d51:b::1 | 172.22.144.168/29 | 172.22.144.169 |
fr-par1 | fd28:7515:7d51:c::/64 | fd28:7515:7d51:c::1 | 172.22.144.176/29 | 172.22.144.177 |
pl-waw1 | fd28:7515:7d51:d::/64 | fd28:7515:7d51:d::1 | 172.22.144.184/29 | 172.22.144.185 |
us-nyc1 | fd28:7515:7d51:e::/64 | fd28:7515:7d51:e::1 | 172.22.145.160/30 | 172.22.145.161 |
de-fra1 | fd28:7515:7d51:f::/64 | fd28:7515:7d51:f::1 | 172.22.145.164/30 | 172.22.145.165 |
ca-yto1 | fd28:7515:7d51:9::/64 | fd28:7515:7d51:9::1 | 172.22.145.168/30 | 172.22.145.169 |
Bird configuration
As usual, the whole bird configuration for all POPs is backed-up in the git repository dn42-as4242420263in the directories common/bird and servers/*/bird.
babel(IGP) is configured on all the interfaces of the full-mesh network (wg-int-*). See thebird.conffile here.- All
[i|e]BGPsession files are either written manually or generated by the auto-peering service. They can be found in the directoriesservers/*/bird/bgp_peers.
Auto-peering service
My dn42 POPs feature a new “auto-peering” service written in Python, which consists of a custom shell served via SSH.
This allows dn42 members to request and manage peering sessions via SSH with the following POP: nl-ams2, fr-par1, pl-waw1, us-nyc1, de-fra1, ca-yto1.
This service is free software, distributed on github under MIT license.
It implements:
- a SSH daemon, using Paramiko - it uses the dn42 registry as its source of authentication
- a custom shell implementing these commands:
peer_list- list your existing peering sessions.peer_create- create a new peering session interactivelypeer_remove- remove an existing peering sessionpeer_config- show the configuration for an existing peering sessionpeer_status- print the current status of a peering session.
Under the hood, it uses:
Paramikofor the SSH protocol implementationPython Rich, for the console styles, boxes, tablesSQLite, to store the peering sessions information in a database- A shell script called by a systemd timer every 5 minutes, to read the
SQLitedatabase and maintain the peering sessions configuration files on the system
Users can connect to all servers (except nl-ams1) via SSH, using their maintainer name (without the -MNT suffix) as username, and port 4242.
Example:
$ ssh gyptazy@nl-ams2.flap42.eu -p 4242
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
██ ████ ████
░██ █░░░ █ █░░░ █
███████ ░██ ██████ ██████████ ██████░ ░█░█ ██░█
░░██░░░██ ░██ █████ ░░░░░░██ ░░██░░██░░██ ██░░░░ ███ ░█░█ ░█
░██ ░██ ░██░░░░░ ███████ ░██ ░██ ░██░░█████ █░░ ░█░ ██
░██ ░██ ░██ ██░░░░██ ░██ ░██ ░██ ░░░░░██ █ ░█ ░░
███ ░██ ███ ░░████████ ███ ░██ ░██ ██████ ░██████░ █████
░░░ ░░ ░░░ ░░░░░░░░ ░░░ ░░ ░░ ░░░░░░ ░░░░░░ ░░░░░
█ █████ ██ ████ ██ ████ ██ ████ ████ ████ ████ ████
███ █░░░░ █░█ █░░░ █ █░█ █░░░ █ █░█ █░░░ █ █░░░██ █░░░ █ █░░░ █ █░░░ █
██░██ ░█ █ ░█ ░ ░█ █ ░█ ░ ░█ █ ░█ ░ ░█░█ █░█░ ░█░█ ░ ░ ░█
██ ░░██░██████ ██████ ███ ██████ ███ ██████ ███ ░█ █ ░█ ███ ░█████ ███
███████░░░░░██░░░░░█ █░░ ░░░░░█ █░░ ░░░░░█ █░░ ░██ ░█ █░░ ░█░░░ █ ░░░ █
░█░░░░░█ ░██ ░█ █ ░█ █ ░█ █ ░█ ░█ █ ░█ ░█ █ ░█
░█ ░█ █████ ░█ ░██████ ░█ ░██████ ░█ ░██████░ ████ ░██████░ ████ ░ ████
░ ░░░░░░░ ░ ░░░░░░ ░ ░░░░░░ ░ ░░░░░░ ░░░░ ░░░░░░ ░░░░ ░░░░
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
▶ FlipFlapNet Home page ┄┄┄┄┄┄┄┄┄┄┄┄ https://hcartiaux.github.io/dn42
▶ Pop configuration ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ https://github.com/hcartiaux/dn42-as4242420263
▶ SSH server source ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ https://github.com/hcartiaux/dn42-sshd-autopeer
▶ Contact (mail) ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ hyacinthe@cartiaux.net
▶ Contact (matrix) ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ @hyacinthe:bsd.cafe
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Welcome to Flip Flap Network (AS4242420263) automated peering service
You are connected as GYPTAZY-MNT to nl-ams2.flap42.eu @ AS4242420263
┏━━━━━━━━━━━━━━━━━━━┓
┃ Your AS number(s) ┃
┡━━━━━━━━━━━━━━━━━━━┩
│ 4242423588 │
│ 4242421344 │
└───────────────────┘
Use this shell to configure your BGP peering session.
Type help or ? to list commands.
AS4242420263> help
Documented commands (type help <topic>):
========================================
bye help intro peer_config peer_create peer_list peer_remove peer_status
AS4242420263> peer_list
Your existing peering sessions
┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ ┃ Wireguard ┃ ┃ ┃
┃ ┃ public ┃ ┃ ┃
┃ AS number ┃ key ┃ Endpoint address ┃ Endpoint port ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 4242421344 │ RE+6… │ 2a02:3100:3e6f:d100:250a:3f69:c3d4:8da8 │ 56111 │
└────────────┴───────────┴─────────────────────────────────────────┴───────────────┘
AS4242420263> peer_status
$ # Configuration generator timer
$ systemctl list-timers dn42-genconfig
NEXT LEFT LAST PASSED
UNIT ACTIVATES
Wed 2025-06-04 14:55:37 CEST 58s left Wed 2025-06-04 14:50:37 CEST 4min 1s ago
dn42-genconfig.timer dn42-genconfig.service
1 timers listed.
Pass --all to see loaded but inactive timers, too.
$ # Wireguard interface
$ wg show wg-as4242421344
interface: wg-as4242421344
public key: C3Wlu6y+v84FN/vreuTqL6r5wEtGTMXX5rKgHkxDaTI=
private key: (hidden)
listening port: 52001
peer: RE+****************************************=
endpoint: [2a02:**********************************]:42424
allowed ips: 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10
latest handshake: 15 seconds ago
transfer: 263.94 MiB received, 259.14 MiB sent
persistent keepalive: every 30 seconds
$ # Bird BGP session
$ birdc show protocols all ebgp_as4242421344_v6
BIRD 2.0.12 ready.
Name Proto Table State Since Info
ebgp_as4242421344_v6 BGP --- up 2025-05-28 Established
BGP state: Established
Neighbor address: fe80:263::2:1%wg-as4242421344
Neighbor AS: 4242421344
Local AS: 4242420263
Neighbor ID: 172.22.130.225
...
Channel ipv4
State: UP
Table: master4
Preference: 100
Input filter: (unnamed)
Output filter: (unnamed)
Import limit: 9000
Action: block
Routes: 747 imported, 610 exported, 144 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 849659 0 16 81081 768562
Import withdraws: 826 0 --- 131 695
Export updates: 378711 31602 16 --- 347093
Export withdraws: 783 --- --- --- 20159
BGP Next hop: :: fe80::106
Channel ipv6
State: UP
Table: master6
Preference: 100
Input filter: (unnamed)
Output filter: (unnamed)
Import limit: 9000
Action: block
Routes: 816 imported, 669 exported, 152 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 341761 0 49 11176 330536
Import withdraws: 1265 0 --- 137 1128
Export updates: 446375 22160 16 --- 424199
Export withdraws: 2093 --- --- --- 21418
BGP Next hop: :: fe80:263::1:1
AS4242420263> peer_config
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Link config. ┃ AS4242421344 ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Wg pub key │ RE+****************************************= │
│ Wg Endpoint addr. │ 2a02:********************************** │
│ Wg Endpoint port │ 42424 │
│ Link-local address │ fe80:0263::2:1 │
└────────────────────┴──────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Link config. ┃ AS4242420263 ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Wg pub key │ C3Wlu6y+v84FN/vreuTqL6r5wEtGTMXX5rKgHkxDaTI= │
│ Wg Endpoint addr. │ nl-ams2.flap42.eu │
│ Wg Endpoint port │ 52001 │
│ Link-local address │ fe80:0263::1:1 │
└────────────────────┴──────────────────────────────────────────────┘
Wireguard configuration for AS4242421344
[Interface]
PrivateKey = **REPLACEME**
ListenPort = 42424
PostUp = /sbin/ip addr add dev %i fe80:0263::2:1/128 peer fe80:0263::1:1/128
Table = off
[Peer]
PublicKey = C3Wlu6y+v84FN/vreuTqL6r5wEtGTMXX5rKgHkxDaTI=
Endpoint = nl-ams2.flap42.eu:52001
PersistentKeepalive = 30
AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10
Bird configuration for AS4242421344
protocol bgp flipflap {
local as 4242421344;
neighbor fe80:0263::1:1 as 4242420263;
path metric 1;
interface "wg-peer-flipflap";
ipv4 {
extended next hop on;
import limit 9000 action block;
import table;
};
ipv6 {
extended next hop off;
import limit 9000 action block;
import table;
};
}
AS4242420263> bye
See You, Space Cowboy!