[dn42] AS4242420263 under the hood - more POPs, full mesh network and auto-peering service
Table of contents:
This post describes the internals of my current dn42 network, with the extension from 2 to 4 POPs, and last but not least, the creation of a SSH-based auto-peering service. This post will be edited and kept up-to-date in the future.
New POP servers
I’ve used STARDUST1-S cloud instances from Scaleway to create two new POPs and extend my network.
Each instance cost around 1€/month in IPv6-only, opting out of IPv4 reduces cloud operating costs.
These new servers are also managed by Terraform and Ansible.
| Location | Hostname | Host |
|---|---|---|
| Amsterdam (NL) | nl-ams1.flap42.eu | Scaleway, dedibox/libvirt |
| Amsterdam (NL) | nl-ams2.flap42.eu | Scaleway, dedibox/libvirt |
| Paris (FR) | fr-par1.flap42.eu | Scaleway, cloud |
| Warsaw (PL) | pl-war1.flap42.eu | Scaleway, cloud |
Full-mesh network with Wireguard
All network links are created with Wireguard. All configuration files are generated with Ansible by iterating on the inventory. The generated configuration files are backed-up in the git repository dn42-as4242420263, in the directories servers/*/wireguard (in example for fr-par1).
Each POP is assigned with an IPv6 and IPv4 subnet.
| Hostname | IPv6 Network | IPv6 Gateway | IPv4 Network | IPv4 Gateway |
|---|---|---|---|---|
nl-ams1 | fd28:7515:7d51:a::/64 | fd28:7515:7d51:a::1 | 172.22.144.160/29 | 172.22.144.161 |
nl-ams2 | fd28:7515:7d51:b::/64 | fd28:7515:7d51:b::1 | 172.22.144.168/29 | 172.22.144.169 |
fr-par1 | fd28:7515:7d51:c::/64 | fd28:7515:7d51:c::1 | 172.22.144.176/29 | 172.22.144.177 |
pl-war1 | fd28:7515:7d51:d::/64 | fd28:7515:7d51:d::1 | 172.22.144.184/29 | 172.22.144.185 |
Bird configuration
As usual, the whole bird configuration for all POPs is backed-up in the git repository dn42-as4242420263in the directories common/bird and servers/*/bird.
babel(IGP) is configured on all the interfaces of the full-mesh network (wg-int-*). See thebird.conffile here.- All
[i|e]BGPsession files are either written manually or generated by the auto-peering service. They can be found in the directoriesservers/*/bird/bgp_peers.
Auto-peering service
My dn42 POPs feature a new “auto-peering” service written in Python, which consists of a custom shell served via SSH.
This allows dn42 members to request and manage peering sessions via SSH with the following POP: nl-ams2, fr-par1, pl-war1.
This service is free software, distributed on github under MIT license.
It implements:
- a SSH daemon, using Paramiko - it uses the dn42 registry as its source of authentication
- a custom shell implementing these commands:
peer_list- list your existing peering sessions.peer_create- create a new peering session interactivelypeer_remove- remove an existing peering sessionpeer_config- show the configuration for an existing peering sessionpeer_status- print the current status of a peering session.
Under the hood, it uses:
Paramikofor the SSH protocol implementationPython Rich, for the console styles, boxes, tablesSQLite, to store the peering sessions information in a database- A shell script called by a systemd timer every 5 minutes, to read the
SQLitedatabase and maintain the peering sessions configuration files on the system
Users can connect to all servers (except nl-ams1) via SSH, using their maintainer name with the -MNT suffix as username, and port 4242.
Example:
$ ssh gyptazy@nl-ams2.flap42.eu -p 4242
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
██ ████ ████
░██ █░░░ █ █░░░ █
███████ ░██ ██████ ██████████ ██████░ ░█░█ ██░█
░░██░░░██ ░██ █████ ░░░░░░██ ░░██░░██░░██ ██░░░░ ███ ░█░█ ░█
░██ ░██ ░██░░░░░ ███████ ░██ ░██ ░██░░█████ █░░ ░█░ ██
░██ ░██ ░██ ██░░░░██ ░██ ░██ ░██ ░░░░░██ █ ░█ ░░
███ ░██ ███ ░░████████ ███ ░██ ░██ ██████ ░██████░ █████
░░░ ░░ ░░░ ░░░░░░░░ ░░░ ░░ ░░ ░░░░░░ ░░░░░░ ░░░░░
█ █████ ██ ████ ██ ████ ██ ████ ████ ████ ████ ████
███ █░░░░ █░█ █░░░ █ █░█ █░░░ █ █░█ █░░░ █ █░░░██ █░░░ █ █░░░ █ █░░░ █
██░██ ░█ █ ░█ ░ ░█ █ ░█ ░ ░█ █ ░█ ░ ░█░█ █░█░ ░█░█ ░ ░ ░█
██ ░░██░██████ ██████ ███ ██████ ███ ██████ ███ ░█ █ ░█ ███ ░█████ ███
███████░░░░░██░░░░░█ █░░ ░░░░░█ █░░ ░░░░░█ █░░ ░██ ░█ █░░ ░█░░░ █ ░░░ █
░█░░░░░█ ░██ ░█ █ ░█ █ ░█ █ ░█ ░█ █ ░█ ░█ █ ░█
░█ ░█ █████ ░█ ░██████ ░█ ░██████ ░█ ░██████░ ████ ░██████░ ████ ░ ████
░ ░░░░░░░ ░ ░░░░░░ ░ ░░░░░░ ░ ░░░░░░ ░░░░ ░░░░░░ ░░░░ ░░░░
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
▶ FlipFlapNet Home page ┄┄┄┄┄┄┄┄┄┄┄┄ https://hcartiaux.github.io/dn42
▶ Pop configuration ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ https://github.com/hcartiaux/dn42-as4242420263
▶ SSH server source ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ https://github.com/hcartiaux/dn42-sshd-autopeer
▶ Contact (mail) ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ hyacinthe.cartiaux@gmail.com
▶ Contact (matrix) ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ @hyacinthe:bsd.cafe
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Welcome to Flip Flap Network (AS4242420263) automated peering service
You are connected as GYPTAZY-MNT to nl-ams2.flap42.eu @ AS4242420263
┏━━━━━━━━━━━━━━━━━━━┓
┃ Your AS number(s) ┃
┡━━━━━━━━━━━━━━━━━━━┩
│ 4242423588 │
│ 4242421344 │
└───────────────────┘
Use this shell to configure your BGP peering session.
Type help or ? to list commands.
AS4242420263> help
Documented commands (type help <topic>):
========================================
bye help intro peer_config peer_create peer_list peer_remove peer_status
AS4242420263> peer_list
Your existing peering sessions
┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ ┃ Wireguard ┃ ┃ ┃
┃ ┃ public ┃ ┃ ┃
┃ AS number ┃ key ┃ Endpoint address ┃ Endpoint port ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ 4242421344 │ RE+6… │ 2a02:3100:3e6f:d100:250a:3f69:c3d4:8da8 │ 56111 │
└────────────┴───────────┴─────────────────────────────────────────┴───────────────┘
AS4242420263> peer_status
$ # Configuration generator timer
$ systemctl list-timers dn42-genconfig
NEXT LEFT LAST PASSED
UNIT ACTIVATES
Wed 2025-06-04 14:55:37 CEST 58s left Wed 2025-06-04 14:50:37 CEST 4min 1s ago
dn42-genconfig.timer dn42-genconfig.service
1 timers listed.
Pass --all to see loaded but inactive timers, too.
$ # Wireguard interface
$ wg show wg-as4242421344
interface: wg-as4242421344
public key: C3Wlu6y+v84FN/vreuTqL6r5wEtGTMXX5rKgHkxDaTI=
private key: (hidden)
listening port: 52001
peer: RE+****************************************=
endpoint: [2a02:**********************************]:42424
allowed ips: 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10
latest handshake: 15 seconds ago
transfer: 263.94 MiB received, 259.14 MiB sent
persistent keepalive: every 30 seconds
$ # Bird BGP session
$ birdc show protocols all ebgp_as4242421344_v6
BIRD 2.0.12 ready.
Name Proto Table State Since Info
ebgp_as4242421344_v6 BGP --- up 2025-05-28 Established
BGP state: Established
Neighbor address: fe80:263::2:1%wg-as4242421344
Neighbor AS: 4242421344
Local AS: 4242420263
Neighbor ID: 172.22.130.225
...
Channel ipv4
State: UP
Table: master4
Preference: 100
Input filter: (unnamed)
Output filter: (unnamed)
Import limit: 9000
Action: block
Routes: 747 imported, 610 exported, 144 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 849659 0 16 81081 768562
Import withdraws: 826 0 --- 131 695
Export updates: 378711 31602 16 --- 347093
Export withdraws: 783 --- --- --- 20159
BGP Next hop: :: fe80::106
Channel ipv6
State: UP
Table: master6
Preference: 100
Input filter: (unnamed)
Output filter: (unnamed)
Import limit: 9000
Action: block
Routes: 816 imported, 669 exported, 152 preferred
Route change stats: received rejected filtered ignored accepted
Import updates: 341761 0 49 11176 330536
Import withdraws: 1265 0 --- 137 1128
Export updates: 446375 22160 16 --- 424199
Export withdraws: 2093 --- --- --- 21418
BGP Next hop: :: fe80:263::1:1
AS4242420263> peer_config
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Link config. ┃ AS4242421344 ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Wg pub key │ RE+****************************************= │
│ Wg Endpoint addr. │ 2a02:********************************** │
│ Wg Endpoint port │ 42424 │
│ Link-local address │ fe80:0263::2:1 │
└────────────────────┴──────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Link config. ┃ AS4242420263 ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Wg pub key │ C3Wlu6y+v84FN/vreuTqL6r5wEtGTMXX5rKgHkxDaTI= │
│ Wg Endpoint addr. │ nl-ams2.flap42.eu │
│ Wg Endpoint port │ 52001 │
│ Link-local address │ fe80:0263::1:1 │
└────────────────────┴──────────────────────────────────────────────┘
Wireguard configuration for AS4242421344
[Interface]
PrivateKey = **REPLACEME**
ListenPort = 42424
PostUp = /sbin/ip addr add dev %i fe80:0263::2:1/128 peer fe80:0263::1:1/128
Table = off
[Peer]
PublicKey = C3Wlu6y+v84FN/vreuTqL6r5wEtGTMXX5rKgHkxDaTI=
Endpoint = nl-ams2.flap42.eu:52001
PersistentKeepalive = 30
AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10
Bird configuration for AS4242421344
protocol bgp flipflap {
local as 4242421344;
neighbor fe80:0263::1:1 as 4242420263;
path metric 1;
interface "wg-peer-flipflap";
ipv4 {
extended next hop on;
import limit 9000 action block;
import table;
};
ipv6 {
extended next hop off;
import limit 9000 action block;
import table;
};
}
AS4242420263> bye
See You, Space Cowboy!