[dn42] AS4242420263 configuration - monitoring with LibreNMS (Part 5/5)

Posted on Sep 8, 2024

Table of contents:

This article is the 5th part of my dn42 experiment. The objective is to monitor, graph and visualize the network flows in a network weathermap using LibreNMS.

New server

I’ve set-up a new server on my remote lab with this bit of terraform configuration.

vms_list = {
  "tf-librenms-dn42" = {
    bridge_name     = "vmbr0"
    vm_memory       = 1024
    vm_vcpu         = 1
    vm_disk_size    = 100
    cloud_image_url = "https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2"
    network_interfaces = {
      ens3 = {
        addresses = [
          "192.168.0.5/16",
          "2001:bc8:3feb:100::5/64",
        ]
      }
    }
    system = {
      hostname = "librenms-dn42"
      packages = ["wget"]
    }
    users = {
      "hcartiaux" = {
        shell               = "/bin/bash"
        sudo                = "ALL=(ALL) NOPASSWD:ALL"
        hashed_passwd       = "!"
        lock_passwd         = true
        ssh_authorized_keys = ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICtAyNLxu6GqPOLQutasE70tPMOSF+WS9LmK6kldYwPD hcartiaux@hc-xps13"]
      }
    }
  }
...
}

dn42 IPs

`librenms.hcartiaux.dn42`	IPv4	IPv6
IPs	`172.22.144.186/29`	`fd28:7515:7d51:d::2/64`
Net.	`172.22.144.184/29`	`fd28:7515:7d51:d::/64`
GW.	`172.22.144.185/29`	`fd28:7515:7d51:d::1/64`

The complete network information are given in part 3

Wireguard wg-quick configuration

Set-up wireguard as per instructions in part 2. The librenms server is connected to the gw2-dn42 router, via the wg-hosts wireguard end point.

This is the content of /etc/wireguard/wg0.conf

[Interface]
PrivateKey = **REDACTED**
PostUp = /sbin/ip addr add dev %i 172.22.144.186/29 peer 172.22.144.185/29
PostUp = /sbin/ip addr add dev %i fd28:7515:7d51:d::2/64 peer fd28:7515:7d51:d::1/64

[Peer]
PublicKey = t3JlqfswkNdZPDenJNO07Nz4W9Hop1YQbT/LB+Uomnw=
Endpoint = [2001:bc8:3feb:100::6]:51900
AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10
PersistentKeepalive = 30

Note the AllowedIPs value, which contains all networks in use on dn42. Static routes to the networks listed in AllowedIPs are added when the interface goes up.

systemctl enable --now wg-quick@wg0

For ease of use, I’ve also connected my archlinux laptop to my dn42 network, via the wg-hosts wireguard end point on gw-dn42. This is the wg-quick configuration:

[Interface]
PrivateKey = **REDACTED**
PostUp = /sbin/ip addr add dev %i 172.22.144.170/29 peer 172.22.144.169/29
PostUp = /sbin/ip addr add dev %i fd28:7515:7d51:b::2/64 peer fd28:7515:7d51:b::1/64
DNS = fd28:7515:7d51:c::1,fd28:7515:7d51:a::1

[Peer]
PublicKey = 8JNlIxV5BTOxNBB2wDs/A5HSvzcZxSLbIEVzz7b94Qc= 
Endpoint = gw-dn42.nbsdn.fr.eu.org:51900
AllowedIPs = 172.16.0.0/12, 10.0.0.0/8, fd00::/8, fe80::/10
PersistentKeepalive = 30

Note that we use the DNS servers configured in the previous part. For the archlinux specificities, as usual, read the famous wiki.

LibreNMS installation

Strictly follow the instructions from the libreNMS documentation. LibreNMS could be installed using the docker container but it makes it more difficult to install additional plugins.
Install systemd-cron to manage traditional crontabs as systemd timers, there is no cron daemon installed by default on the Debian cloud image.

apt install systemd-cron
sudo systemctl enable cron.target
systemctl list-timers 
NEXT                         LEFT                  LAST                         PASSED               UNIT                           ACTIVATES                       
Sat 2024-09-07 18:15:00 CEST 40s left              Sat 2024-09-07 18:10:00 CEST 4min 19s ago         cron-librenms-librenms-1.timer cron-librenms-librenms-1.service
Sat 2024-09-07 18:15:00 CEST 40s left              Sat 2024-09-07 18:10:00 CEST 4min 19s ago         cron-librenms-librenms-2.timer cron-librenms-librenms-2.service
Sat 2024-09-07 18:15:00 CEST 40s left              Sat 2024-09-07 18:14:00 CEST 19s ago              cron-librenms-librenms-3.timer cron-librenms-librenms-3.service
Sat 2024-09-07 18:15:00 CEST 40s left              Sat 2024-09-07 18:10:00 CEST 4min 19s ago         cron-librenms-librenms-4.timer cron-librenms-librenms-4.service
Sat 2024-09-07 18:15:00 CEST 40s left              Sat 2024-09-07 18:10:00 CEST 4min 19s ago         cron-librenms-librenms-6.timer cron-librenms-librenms-6.service
Sat 2024-09-07 18:15:00 CEST 40s left              Sat 2024-09-07 18:10:00 CEST 4min 19s ago         cron-librenms-librenms-8.timer cron-librenms-librenms-8.service
Sat 2024-09-07 18:15:00 CEST 40s left              Sat 2024-09-07 18:14:00 CEST 19s ago              librenms-scheduler.timer       librenms-scheduler.service
Sat 2024-09-07 18:17:00 CEST 2min 40s left         Sat 2024-09-07 17:17:00 CEST 57min ago            cron-hourly.timer              cron-hourly.target
Sat 2024-09-07 18:33:00 CEST 18min left            Sat 2024-09-07 12:33:00 CEST 5h 41min ago         cron-librenms-librenms-0.timer cron-librenms-librenms-0.service
Sat 2024-09-07 18:39:00 CEST 24min left            Sat 2024-09-07 18:09:00 CEST 5min ago             cron-php-root-0.timer          cron-php-root-0.service
Sat 2024-09-07 18:39:00 CEST 24min left            Sat 2024-09-07 18:09:00 CEST 5min ago             phpsessionclean.timer          phpsessionclean.service
Sat 2024-09-07 19:01:00 CEST 46min left            Sat 2024-09-07 18:01:00 CEST 13min ago            cron-librenms-librenms-5.timer cron-librenms-librenms-5.service
Sat 2024-09-07 22:42:01 CEST 4h 27min left         Sat 2024-09-07 15:54:16 CEST 2h 20min ago         apt-daily.timer                apt-daily.service
Sun 2024-09-08 00:00:00 CEST 5h 45min left         Sat 2024-09-07 00:00:00 CEST 18h ago              dpkg-db-backup.timer           dpkg-db-backup.service
Sun 2024-09-08 00:19:00 CEST 6h left               Sat 2024-09-07 00:19:00 CEST 17h ago              cron-librenms-librenms-7.timer cron-librenms-librenms-7.service
Sun 2024-09-08 01:25:42 CEST 7h left               Sat 2024-09-07 01:25:42 CEST 16h ago              systemd-tmpfiles-clean.timer   systemd-tmpfiles-clean.service
Sun 2024-09-08 03:10:08 CEST 8h left               Sun 2024-09-01 03:10:31 CEST 6 days ago           e2scrub_all.timer              e2scrub_all.service
Sun 2024-09-08 06:25:00 CEST 12h left              Sat 2024-09-07 06:25:00 CEST 11h ago              cron-daily.timer               cron-daily.target
Sun 2024-09-08 06:58:04 CEST 12h left              Sat 2024-09-07 06:24:34 CEST 11h ago              apt-daily-upgrade.timer        apt-daily-upgrade.service
Sun 2024-09-08 08:25:48 CEST 14h left              Sat 2024-09-07 10:00:00 CEST 8h ago               man-db.timer                   man-db.service
Mon 2024-09-09 01:28:03 CEST 1 day 7h left         Mon 2024-09-02 00:11:20 CEST 5 days ago           fstrim.timer                   fstrim.service
Mon 2024-09-09 06:47:00 CEST 1 day 12h left        Mon 2024-09-02 06:47:00 CEST 5 days ago           cron-weekly.timer              cron-weekly.target
Tue 2024-10-01 06:52:00 CEST 3 weeks 2 days left   Sun 2024-09-01 06:52:00 CEST 6 days ago           cron-monthly.timer             cron-monthly.target
Wed 2025-01-01 00:00:00 CET  3 months 23 days left Sat 2024-06-22 23:16:42 CEST 2 months 15 days ago cron-yearly.timer              cron-yearly.target

24 timers listed.
Pass --all to see loaded but inactive timers, too.

In order to finish the installation, you’ll need to use the web installer.

You can use ssh port redirection and open the web interface in your default browser:

ssh -L1080:localhost:80 librenms-dn42 &
xdg-open http://localhost:1080/install

Or simply use a direct connection and use the domain name configured in the previous part.

xdg-open 'http://librenms.hcartiaux.dn42/install'

Network-Weathermap plugin

Once again, strictly follow the instructions from the LibreNMS documentation to install PHP Weathermap as a plugin. Do not forget to install php-pear or the plugin will miss mandatory PHP libraries.

apt install php-pear

If you want to force the regeneration of your weathermap(s), use this command:

su - librenms -c "/opt/librenms/html/plugins/Weathermap/map-poller.php"

SNMP daemons

I’ve used ansible to set-up snmpd identically on all my servers (gw-dn42, gw2-dn42 and librenms-dn42). This is the relevant inventory extract:

snmpd_com2sec:     dn42
snmpd_syslocation: Rack D25, Room 1 1, Block E, Position 1, Server 4, fr-par2 - DC5, Paris, France
snmpd_syscontact:  Hyacinthe Cartiaux <hyacinthe.cartiaux@gmail.com>

As a result, the content of the file /etc/snmpd/snmpd.conf can be reduced to these few lines:

syslocation Rack D25, Room 1 1, Block E, Position 1, Server 4, fr-par2 - DC5, Paris, France
syscontact  Hyacinthe Cartiaux <hyacinthe.cartiaux@gmail.com>

extend distro /usr/bin/distro

com2sec readonly default dn42
group MyROGroup v2c readonly
view all included .1 80
access MyROGroup "" any noauth exact all none none

includeDir /etc/snmp/snmpd.conf.d

LibreNMS configuration

Head to the librenms web interface and enter the admin credentials set-up previously.

xdg-open 'http://librenms.hcartiaux.dn42/'

Add all the servers (in the top menu, “Devices” > “Add devices”), fill the IP and the SNMP community name for each of them.

The data should be quickly polled (“Devices” > “All devices” > “Server”)

Weathermap Creation

Open the web editor (“Overview” > “Plugins” > “Weathermap”). Create a new map or open an existing one.

Draft a map, add all the nodes and create the links (“Add Node” and “Add Link” in the top menu).

Click on the nodes and links to set the data source.

Customize (and backup!) the file /opt/librenms/html/plugins/Weathermap/configs/dn42-as4242420263.conf. Alignment and placement is a lot easier to edit manually, and many features are simply not included in the GUI. Consult the reference here for a full description of all the configuration directives.
Come back to LibreNMS, the new weathermap is linked in the top menu (“Overview” > “Plugin” > “Weathermap” > “[DN42] Weathermap - AS4242420263”)

Experiments

Let’s generate traffic with iperf3 and visualize it on the weathermap:

Start an iperf3 server: iperf3 -s
Connect to the server: iperf3 -c <server ip or domain> (bandwidth can be restricted to 10Mbps by adding -b 10M)

I will use 3 hosts:

my laptop connected to wg-hosts on gw-dn42
the librenms server connected to wg-hosts on gw2-dn42
the public shell server shell.fr.burble.dn42

From my laptop to `librenms.hcartiaux.dn42`

We can see the path used by our packets, through the internal peering (wg-peer-int)
Since all links are wireguard tunnels, the usage of the links to Internet is the sum of the usage of all wireguard end points.

From my laptop to `shell.fr.burble.dn42`

From my laptop, connected to gw-dn42, the traffic to shell.fr.burble.dn42 is routed via AS4242421588.

From LibreNMS to `shell.fr.burble.dn42`

From the librenms server, connected to gw2-dn42, the traffic to shell.fr.burble.dn42 is routed via AS4242423035.

Conclusion

I realize LibreNMS and PHP Weathermap look old school compared to Grafana and its fancy Network Weathermap plugin, but it works and it’s simple enough for my use case. Also, more work is required to secure the set-up described here, but that’s another topic!

This is the end of the adventure… for the moment. I’m open to peering !