Author SHA1 Message Date
Ruakij c36f269406 journal_upload: a node leaves its own NetBird address out of /etc/hosts, Traefik sees it as the cilium_host address 2026-10-24 16:04:06 +00:00
Ruakij 19a62a43ca common: remove fail2ban where CrowdSec agent and bouncer run, stop installing it 2026-10-24 15:57:34 +00:00
Ruakij fd58ee2c16 journal_upload: ship the journal to VictoriaLogs with systemd-journal-upload over NetBird 2026-10-24 15:55:22 +00:00
Ruakij d5610a5b34 host_firewall: refresh the Spamhaus sets by rule set reload, flush and refill let unlisted addresses match 2026-10-24 14:53:10 +00:00
Ruakij ebb1377470 host_firewall: block Spamhaus DROP, DROPv6 and ASN-DROP networks in, out and forward 2026-10-24 14:31:00 +00:00
Ruakij 5e544cdfb5 host_firewall: optional nf_log_all_netns sysctl so the firewall logs of LXC guests reach the host kernel log 2026-10-24 13:43:12 +00:00
Ruakij 88c30eb252 host_firewall: log dropped inbound packets with a global rate limit and reject outbound TCP 25 except on the mail host 2026-10-24 12:14:47 +00:00
Ruakij 41dbce3641 host_firewall: admin.tcp ports answer the admin set alone, never the geo sets 2026-10-24 08:40:31 +00:00
Ruakij 81e7a9dc6a crowdsec_range_bouncer: enforce CrowdSec range decisions in nftables interval sets, the firewall bouncer stores only the network address of a range 2026-10-24 08:10:34 +00:00
Ruakij 5cac169224 Give system.slice and the k3s unit CPUWeight 1000 so the control plane wins CPU against pods 2026-10-23 21:14:13 +00:00
Ruakij 60a55e6e49 host_exporters: install the node exporter without recommends, smartd and nvmf-autoconnect failed on hugo 2026-10-23 21:04:05 +00:00
Ruakij c5b51777c6 Ignore the missing geo timer unit of a host without geo sets in check mode 2026-10-23 20:54:31 +00:00
Ruakij a4648e0ea7 Join a fresh host to NetBird idempotently with the keyring, the management URL and the interface blacklist from the vars, and edit the blacklist and log level of a joined one 2026-10-23 20:54:30 +00:00
Ruakij a64d082c0c Add a host_exporters role for the node and smartctl exporters and a host_firewall scrapers rule that opens exporter ports on wt0 to named source addresses 2026-10-23 20:44:59 +00:00
Ruakij 381e1b285c Add an unattended_upgrades role that installs only Debian and Ubuntu security updates, with no automatic reboot 2026-10-22 09:00:33 +00:00
Ruakij bcdc460297 Copy per-host files of kopia_backup from a files list, empty by default 2026-10-22 08:25:43 +00:00
Ruakij bd24f01110 Let a host name its CrowdSec machine at the LAPI with crowdsec.machine_name, the short hostname of doris is dev 2026-10-21 06:10:01 +00:00
Ruakij 1ede3d3329 Read the fail2ban sshd jail from the journal, Debian 12+ has no auth.log and the jail failed to start 2026-10-13 16:36:08 +00:00
Ruakij deb3521bfa Add a proxmox_no_nag role that hides the subscription dialog and reapplies it from an apt hook after every widget-toolkit update 2026-10-13 15:55:56 +00:00
Ruakij c8cac22d12 Run the cilium status check in check mode too and never report it as changed 2026-10-09 14:37:30 +00:00
Ruakij 6fe03c5f70 Skip the cilium CLI download when it is installed already 2026-10-09 14:37:30 +00:00
Ruakij 15b4e54bd5 Read resolv.conf for the k3s nameservers in check mode too 2026-10-09 14:37:30 +00:00
Ruakij c016459922 Write the k3s registries.yaml only on a semantic change 2026-10-09 14:37:29 +00:00
Ruakij 4d1725f17f Deploy a registries.yaml from registries_config on k3s servers and restart k3s on its change 2026-10-09 14:37:29 +00:00
Ruakij 80658c632f Add the cilium network plugin to the kubernetes role, disabling servicelb, and collect the disabled k3s components in one disable list 2026-10-09 14:37:29 +00:00
Ruakij 60b67f748e Install the k3s servers one at a time with retries from the latest channel, enable the service and link the kubeconfig to ~/.kube/config 2026-10-09 14:37:28 +00:00
Ruakij 4a76aa3a2e Set node-name, node-external-ip and labels_extra in the k3s config and config_extra for agents 2026-10-09 14:37:27 +00:00
Ruakij e9f1153482 Add clean_resolv_conf to the kubernetes role to give k3s a resolv.conf with only the nameserver lines 2026-10-09 14:37:27 +00:00
Ruakij ca1fcc16b6 Fall back to the default IPv4 and IPv6 address when the k3s node IP interface has none 2026-10-09 14:37:27 +00:00
Ruakij fb94fd5997 Deploy the calico operator v3.28.0 2026-10-09 14:37:26 +00:00
Ruakij 713d38d77d Shrink the default IPv6 cluster CIDR of k3s to fd42::/112 2026-10-09 14:37:26 +00:00
Ruakij bc71066de1 Run the k3s install of the kubernetes role only on target_host when it is set 2026-10-09 14:37:26 +00:00
Ruakij 5eae9ca348 Drop the docker dependency and the helm install from the kubernetes role 2026-10-09 14:37:25 +00:00
Ruakij 844e017f9e Add a tuned role that installs tuned and sets the throughput-performance profile 2026-10-09 14:37:25 +00:00
Ruakij 381f3ab7c9 Start the wireguard-ipv6-converter again whenever its bound service starts via a drop-in and always restart it on deploy 2026-10-09 14:37:24 +00:00
Ruakij 6390e6288b Disable SSH password auth in the common role only when sshd is active 2026-10-09 14:37:24 +00:00
Ruakij 13d60c4f9b Install nmon and fail2ban and upgrade packages on every distribution in the common role 2026-10-09 14:37:24 +00:00
Ruakij cb09f77b68 Enable TCP BBR with the fq qdisc on kernels from 4.9 in the common role 2026-10-09 14:37:23 +00:00
Ruakij 84fd52b678 Generate a random IPv6 stable secret once per host for the default interface in the common role 2026-10-09 14:37:23 +00:00
Ruakij c7eceae087 Ensure the .bashrc exists, expand aliases after sudo and color grep in the common role aliases 2026-10-09 14:37:23 +00:00
Ruakij 23ed82b113 Own the k3s config.yaml in the kubernetes role template, write it only on a semantic change and run the installer only when the pinned k3s version differs 2026-10-09 14:13:21 +00:00
Ruakij 37e21ead36 Expose the etcd metrics on the node IP of the etcd members of k3s 2026-10-09 12:56:24 +00:00
Ruakij 2798853153 Raise the inotify limits of the k3s nodes to 65536 instances and 4194304 watches, except in LXC 2026-10-09 12:55:46 +00:00
Ruakij e9ba272d70 Manage the kubelet image garbage collection drop-in and its config-dir kubelet-arg in the kubernetes role 2026-10-09 12:54:41 +00:00
Ruakij b62992f21e Cap the systemd journal at 512M on disk, 128M in memory and two weeks in the common role 2026-10-09 12:53:44 +00:00
Ruakij fd73949412 Whitelist the public_ips of every inventory host in crowdsec and keep a LAPI allowlist equal to the whitelist 2026-10-08 09:19:51 +00:00
Ruakij d345ccf585 Add a pre_command to kopia_backup for host dump scripts and install gpg for the apt key 2026-10-08 08:51:23 +00:00
Ruakij b45a07da56 Add forward_guard to host_firewall for a forward chain that drops traffic to guarded guests outside the allowed interfaces, ports and ICMP types 2026-10-08 08:48:40 +00:00
Ruakij 9ded6814c3 Pin the fstrim timer and the monthly ZFS trim and scrub per host in the common role to keep them out of the backup window 2026-10-07 22:42:08 +00:00
Ruakij a9a60e3106 Keep the clock on systemd-timesyncd and the timezone on UTC in the common role 2026-10-07 21:22:01 +00:00
Ruakij 5da89fdf29 Run the kopia_backup timer at 03:30 UTC by default since the hosts run in different timezones 2026-10-07 21:19:09 +00:00
Ruakij 9e2e6cbb99 Add IPv4 port forwards and masquerade to host_firewall in an ip host_nat table 2026-10-07 21:10:35 +00:00
Ruakij c9d2334906 Add a kopia_backup role for the daily Kopia snapshot script and timer against an existing S3 repository 2026-10-07 21:00:25 +00:00
Ruakij c87af201aa Add a crowdsec role for agents and the nftables bouncer against a central LAPI 2026-10-07 11:02:59 +00:00
Ruakij cfe87002c5 Replace the nftables role with host_firewall, one atomic inet host input table with zones, geo sets and lockout rollback 2026-10-07 11:02:59 +00:00
Ruakij 27e9d4a1ab Merge branch 'role_nftables' 2023-06-21 17:35:09 +02:00
Ruakij 755f9b2e1a Initial role 2023-05-21 20:57:43 +02:00
Ruakij 753f456ef3 Merge branch 'role_kubernetes-k3s' 2023-04-14 10:21:06 +02:00
Ruakij 3fe288f6a5 Merge branch 'role_wireguard-ipv6-converter' 2023-04-12 20:40:01 +02:00
Ruakij ab9220d042 Add pause & gather_facts at end when service was started 2023-04-12 20:39:49 +02:00
Ruakij 15ad7920d4 Merge branch 'role_wireguard-ipv6-converter' 2023-04-12 20:24:27 +02:00
Ruakij 7012e1ea2b Separate enable and re-/start service for fine-control 2023-04-12 20:23:08 +02:00
Ruakij 13ebd48c5d Add register for task download 2023-04-12 20:22:34 +02:00
Ruakij 1a76b94a46 Add download-version 2023-04-12 14:58:43 +02:00
Ruakij 1b765689e6 Make sure daemon is reloaded before service-start 2023-04-12 14:50:14 +02:00
Ruakij 5b607df2de Add checks if var is set 2023-04-12 14:47:02 +02:00
Ruakij 521b76453a Add start to naming 2023-04-12 14:45:55 +02:00
Ruakij 7f503d983a Fix download-location 2023-04-12 14:45:43 +02:00
Ruakij b143d9c848 Fix checks 2023-04-12 14:45:31 +02:00
Ruakij f95bcbc38d change defaults 2023-04-12 14:44:10 +02:00
Ruakij 248a3c08b8 Initial role-data 2023-04-12 13:36:14 +02:00
Ruakij f2c86dc22d Merge branch 'role_kubernetes-k3s' 2023-04-05 20:20:03 +02:00
Ruakij a79f2cac8a Merge branch 'role_netbird_client' 2023-04-05 20:19:44 +02:00
Ruakij fd302e4ebc Move regather facts to join when changed 2023-03-19 13:47:01 +01:00
Ruakij b5729caa0e Add wait for interface to come up 2023-03-19 13:46:44 +01:00
Ruakij dca40ed835 Remove throttle 2023-03-19 13:46:21 +01:00
Ruakij 95ddd04a86 Fix join command 2023-03-19 10:39:06 +01:00
Ruakij 911bc47acb Initial role stuff 2023-03-17 15:58:49 +01:00
Ruakij f3e381aca3 Move netmaker to deprecated 2023-01-18 13:23:51 +01:00
Ruakij 233eadaf40 Merge branch 'role_netmaker_server' 2023-01-10 09:47:34 +01:00
Ruakij 109a09052d Update to 0.17.1
This also sets up tls-termination for mosquitto
2023-01-10 09:46:55 +01:00
Ruakij 4ea9492ca3 Change hos group-name 2023-01-10 09:46:01 +01:00
Ruakij e5ebc2ad5f Merge branch 'role_common' 2022-11-02 16:41:10 +01:00
Ruakij 98c51c6fc1 Merge branch 'role_kubernetes-k3s' 2022-11-02 16:40:52 +01:00
Ruakij 6b59bf6c75 Merge branch 'role_netmaker' 2022-11-02 16:40:36 +01:00
Ruakij 1b2af7cf6c Merge branch 'role_netmaker_server' 2022-11-02 16:40:25 +01:00
Ruakij d9cf3d2066 Re-gatherfacts at the end for other plays 2022-11-02 16:38:13 +01:00
Ruakij f42bce9b6b Add changed-detection 2022-11-02 16:37:53 +01:00
Ruakij 975746e7d7 Add IPv6-network to common 2022-11-02 10:27:06 +01:00
Ruakij a27ca2c37a Create dedicated docs-folder and move files there 2022-10-27 01:24:30 +02:00
Ruakij 247fdec7ae fixme: hotfix for multi-master netmaker-server
netmaker doesnt handle concurrent joins to different server-nodes well and will duplicate addresses
2022-10-27 01:23:45 +02:00
107 changed files with 2733 additions and 131 deletions
+3
View File
@@ -0,0 +1,3 @@
---
dependencies: []
#- role: docker
@@ -2,4 +2,6 @@
when: "netclient.join_network_token is defined"
command: "netclient join -t {{ netclient.join_network_token }}"
failed_when: command.rc != 0
changed_when: "'starting wireguard' in command.stdout"
register: command
throttle: 1
+8
View File
@@ -0,0 +1,8 @@
- import_tasks: ./certs.yml
- import_tasks: ./install.yml
- import_tasks: ./join-network.yml
- name: Gather facts to get changes
ansible.builtin.gather_facts:
@@ -30,7 +30,7 @@ component netmaker_server {
component nm_api
nm_api -down- nm_api_http
ng_http --( nm_api_http
nm_api -up-( ng_TLS : db-connection to rqlite-master
nm_api .up.( ng_TLS : db-connection to rqlite-master
nm_api --( mq_plain
}
@@ -1,11 +1,11 @@
per_listener_settings false
listener 8883
protocol websockets
allow_anonymous false
certfile /certs/node.crt
keyfile /certs/node.key
listener 1883
protocol websockets
allow_anonymous false
plugin /usr/lib/mosquitto_dynamic_security.so
@@ -30,7 +30,7 @@
headers:
Authorization: 'Bearer {{ netmaker_creds.master_key }}'
Content-Type: application/json
when: "inventory_hostname == groups['netmaker'][0]"
when: "inventory_hostname == groups['netmaker_server'][0]"
register: default_mesh
until: "default_mesh is not failed"
retries: 2
@@ -50,7 +50,7 @@
headers:
Authorization: 'Bearer {{ netmaker_creds.master_key }}'
Content-Type: application/json
when: "inventory_hostname == groups['netmaker'][0]"
when: "inventory_hostname == groups['netmaker_server'][0]"
register: default_mesh_key
until: "default_mesh_key is not failed"
retries: 2
@@ -33,15 +33,15 @@ services:
-auth /config.json
{% if inventory_hostname != groups['netmaker'][0] %}
{% if inventory_hostname != groups['netmaker_server'][0] %}
-join-as netmaker
-join https://{{ netmaker_rqlite.http_host }}.{{ groups['netmaker'][0] }}:{{ netmaker_nginx.advertise_port }}
-join https://{{ netmaker_rqlite.http_host }}.{{ groups['netmaker_server'][0] }}:{{ netmaker_nginx.advertise_port }}
{% endif %}
"
# FIXME: /\ \/ Change http -> https
netmaker: # The Primary Server for running Netmaker
image: gravitl/netmaker:v0.16.1
image: gravitl/netmaker:v0.17.1
depends_on:
- rqlite
cap_add:
@@ -104,7 +104,7 @@ services:
- "51821-51830:51821-51830/udp" # wireguard ports
netmaker-ui: # The Netmaker UI Component
image: gravitl/netmaker-ui:v0.16.1
image: gravitl/netmaker-ui:v0.17.1
depends_on:
- netmaker
links:
@@ -120,7 +120,6 @@ services:
- ./mosquitto/config:/mosquitto/config
- ./mosquitto/data:/mosquitto/data
- ./mosquitto/logs:/mosquitto/log
- "./certs:/certs:ro"
depends_on:
- netmaker
command: ["/mosquitto/config/wait.sh"]
@@ -6,7 +6,7 @@ stream{
{{ netmaker_ui.host }}.{{ netmaker_base_domain }} 127.0.0.1:8443;
{{ netmaker_api.host }}.{{ netmaker_base_domain }} 127.0.0.1:8443;
{{ netmaker_broker.tls_host }}.{{ netmaker_base_domain }} mosquitto:8883; # todo: tls-terminate?
{{ netmaker_broker.tls_host }}.{{ netmaker_base_domain }} 127.0.0.1:8443;
{{ netmaker_rqlite.http_host }}.{{ ansible_facts.nodename }} 127.0.0.1:8443;
{{ netmaker_rqlite.cluster_host }}.{{ ansible_facts.nodename }} rqlite:4002;
@@ -4,6 +4,8 @@ map $host $proxy_name {
{{ netmaker_ui.host }}.{{ netmaker_base_domain }} netmaker-ui:80;
{{ netmaker_api.host }}.{{ netmaker_base_domain }} netmaker:8081;
{{ netmaker_broker.tls_host }}.{{ netmaker_base_domain }} mosquitto:8883;
{{ netmaker_rqlite.http_host }}.{{ ansible_facts.nodename }} rqlite:4001;
default 444;
+14
View File
@@ -0,0 +1,14 @@
ipv6_stable_secret_seed: "@JW7rQ^P3*f^KRkX3744DZQJb3DfMgZJJMtGc!Jmy!ECBqb^QzAMYLHD8dLvf8m%2bRSRFhgY#SXzAgnxWTvHv#ti6b5PngR^k2gHbQW*%dJ9sdeKM$H77Gymh&xamNN"
# Times of the heavy disk jobs, for hosts on a shared HDD pool that keep them out of the backup window.
disk_maintenance:
# systemd OnCalendar for fstrim.timer; unset keeps the distribution schedule.
fstrim_on_calendar:
# Cron "minute hour" of the monthly ZFS trim and scrub from zfsutils-linux; unset keeps 00:24.
zfs_cron_time:
# journald.conf [Journal] settings of a drop-in, in this order; null leaves journald untouched.
journald_size_cap:
SystemMaxUse: 512M
RuntimeMaxUse: 128M
MaxRetentionSec: 2week
+5
View File
@@ -1,2 +1,7 @@
- name: reload_sysctl
command: sysctl --system
- name: restart_journald
systemd:
name: systemd-journald
state: restarted
+21
View File
@@ -1,3 +1,9 @@
- name: Ensure .bashrc exists
file:
path: "{{ ansible_facts.env.HOME }}/.bashrc"
state: touch
mode: "0644"
- name: General aliases
blockinfile:
path: "{{ ansible_facts.env.HOME }}/.bashrc"
@@ -8,11 +14,26 @@
alias syslog="tail -f --lines=100 /var/log/syslog"
alias cp="rsync -hlAXEptgoDS --numeric-ids --info=progress2"
# We can tell bash to check the next word after the alias (i.e sudo) by adding a space to the end of the alias value. https://askubuntu.com/questions/22037/aliases-not-available-when-using-sudo
alias sudo='sudo '
- name: ls aliases and colors
blockinfile:
path: "{{ ansible_facts.env.HOME }}/.bashrc"
marker: "# {mark} ANSIBLE MANAGED BLOCK | ls aliases and colors"
block: |
# enable color support of ls and also add handy aliases
if [ -x /usr/bin/dircolors ]; then
test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)"
alias ls='ls --color=auto'
#alias dir='dir --color=auto'
#alias vdir='vdir --color=auto'
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'
fi
export LS_OPTIONS='--color=auto'
eval "`dircolors`"
alias ls='ls $LS_OPTIONS'
+35
View File
@@ -0,0 +1,35 @@
- name: Pin the fstrim timer
when: disk_maintenance.fstrim_on_calendar | default(none) is not none
block:
- name: Create the fstrim timer drop-in directory
file:
path: /etc/systemd/system/fstrim.timer.d
state: directory
- name: Set the fstrim timer schedule
copy:
dest: /etc/systemd/system/fstrim.timer.d/schedule.conf
content: |
[Timer]
OnCalendar=
OnCalendar={{ disk_maintenance.fstrim_on_calendar }}
RandomizedDelaySec=0
# A catch-up run after a boot or a timezone change would land at any time.
Persistent=false
register: fstrim_timer
- name: Restart the fstrim timer
when: fstrim_timer.changed
systemd:
name: fstrim.timer
daemon_reload: yes
state: restarted
- name: Schedule the monthly ZFS trim and scrub
when: disk_maintenance.zfs_cron_time | default(none) is not none
lineinfile:
path: /etc/cron.d/zfsutils-linux
regexp: '^\S+ \S+ (\S+ \* \* root .*/usr/lib/zfs-linux/{{ item }};.*)$'
line: '{{ disk_maintenance.zfs_cron_time }} \1'
backrefs: yes
loop: [trim, scrub]
+27
View File
@@ -0,0 +1,27 @@
# CrowdSec reads the same logs and bans through its bouncers. fail2ban is
# removed only where both are running, so a host never ends up without a ban
# mechanism.
- name: Check the CrowdSec services
command: systemctl is-active crowdsec crowdsec-firewall-bouncer
register: crowdsec_active
changed_when: false
failed_when: false
check_mode: false
tags: fail2ban
- name: Stop and disable fail2ban
systemd:
name: fail2ban
state: stopped
enabled: false
when: crowdsec_active.rc == 0
failed_when: false
tags: fail2ban
- name: Remove fail2ban
apt:
name: fail2ban
state: absent
purge: true
when: crowdsec_active.rc == 0
tags: fail2ban
+17
View File
@@ -0,0 +1,17 @@
- name: Cap the journal size
when: journald_size_cap | default(none) is not none
block:
- name: Create the journald drop-in directory
file:
path: /etc/systemd/journald.conf.d
state: directory
- name: Set the journal size cap
copy:
dest: /etc/systemd/journald.conf.d/10-size-cap.conf
content: |
[Journal]
{% for key, value in journald_size_cap.items() %}
{{ key }}={{ value }}
{% endfor %}
notify: restart_journald
Regular → Executable
+11
View File
@@ -1,7 +1,18 @@
- import_tasks: ./packages.yml
- import_tasks: ./fail2ban.yml
- import_tasks: ./ssh.yml
- import_tasks: ./packages.yml
- import_tasks: ./timesync.yml
- import_tasks: ./aliases.yml
- import_tasks: ./networking.yml
- import_tasks: ./disk_maintenance.yml
- import_tasks: ./journald.yml
tags: journald
+59
View File
@@ -0,0 +1,59 @@
- name: Set sysctl settings for ip-forwarding
copy:
dest: "/etc/sysctl.d/ip-forwarding.conf"
content: |
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
notify: reload_sysctl
- name: Check if the sysctl configuration file exists
stat:
path: "/etc/sysctl.d/ipv6-slaac-address-generation.conf"
register: sysctl_conf
- block:
# - name: Generate IPv6 stable-secret segments
# set_fact:
# stable_secret_segments: "{{ stable_secret_segments | default() + '%0x' | format(range(0, 2**(16*2)) | random(seed=ipv6_stable_secret_seed+inventory_hostname+(stable_secret_segments | default()))) }}"
# loop: "{{ range(4) }}"
# - name: Format IPv6 stable-secret segments
# set_fact:
# ipv6_stable_secret: "{{ stable_secret_segments | regex_replace('(.{4})(?=.)', '\\1:') }}"
- name: Generate random IPv6 stable-secret
set_fact:
ipv6_stable_secret: "{{ 'fd' + (lookup('password', '/dev/null length=30 chars=hex')).split('.')[0][:7] + '::' + (lookup('password', '/dev/null length=4 chars=hex')) }}"
- name: Set sysctl settings for ipv6-address-generation
copy:
dest: "/etc/sysctl.d/ipv6-slaac-address-generation.conf"
content: |
{% if ansible_facts.default_ipv6.interface is defined %}
net.ipv6.conf.{{ ansible_facts.default_ipv6.interface }}.addr_gen_mode = 2
net.ipv6.conf.{{ ansible_facts.default_ipv6.interface }}.stable_secret = {{ ipv6_stable_secret }}
{% else %}
net.ipv6.conf.{{ ansible_facts.default_ipv4.interface }}.addr_gen_mode = 2
net.ipv6.conf.{{ ansible_facts.default_ipv4.interface }}.stable_secret = {{ ipv6_stable_secret }}
{% endif %}
notify: reload_sysctl
when:
- ansible_facts.default_ipv6 is defined
- not sysctl_conf.stat.exists
- name: Set sysctl settings to override ipv6-slaac with enabled forwarding
copy:
dest: "/etc/sysctl.d/ipv6-slaac-override.conf"
content: |
net.ipv6.conf.all.accept_ra = 2
notify: reload_sysctl
- name: Set sysctl settings for TCP BBR
copy:
dest: "/etc/sysctl.d/tcp-bbr.conf"
content: |
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
when:
- ansible_kernel is version('4.9', '>=')
notify: reload_sysctl
+1 -2
View File
@@ -2,7 +2,6 @@
apt:
update_cache: yes
upgrade: yes
when: ansible_facts.distribution == "Debian"
- name: Install Packages
package:
@@ -11,10 +10,10 @@
- htop
- iotop
- slurm
- nmon
- sudo
- screen
- curl
- rsync
- zstd
state: latest
when: ansible_facts.distribution == "Debian"
+10 -2
View File
@@ -1,12 +1,20 @@
- name: Disable SSH password auth
- name: Check if SSH daemon is active
systemd:
name: sshd
state: started
register: sshd_status
- block:
- name: Disable SSH password auth
lineinfile:
dest: /etc/ssh/sshd_config
regexp: '^PasswordAuthentication\s*yes'
line: "PasswordAuthentication no"
register: sshd_config
- name: Restart SSH daemon
- name: Restart SSH daemon
service:
name: sshd
state: restarted
when: sshd_config.changed
when: sshd_status.status.ActiveState == 'active'
+19
View File
@@ -0,0 +1,19 @@
# Containers share the host kernel clock and cannot discipline it themselves.
- name: Configure time synchronisation
when: ansible_facts['virtualization_role'] != 'guest' or ansible_facts['virtualization_type'] not in ['lxc', 'docker', 'podman', 'container']
block:
- name: Install systemd-timesyncd
package:
name: systemd-timesyncd
state: present
- name: Enable systemd-timesyncd
systemd:
name: systemd-timesyncd
enabled: yes
state: started
# Logs and local-time timers line up across hosts.
- name: Set the timezone to UTC
community.general.timezone:
name: Etc/UTC
+28
View File
@@ -0,0 +1,28 @@
---
crowdsec:
# Central LAPI; hosts run only the agent and the bouncer.
lapi_url: ""
registration_token: ""
bouncer_api_key: ""
# Machine name at the LAPI; empty is the short hostname.
machine_name: ""
# name -> hub collection, merged across group_vars/host_vars (null drops one).
collections: {}
# name -> hub scenario outside the collections.
scenarios: {}
# path below /etc/crowdsec -> content, for local parsers and scenarios.
local_files: {}
# name -> CIDR (single addresses as /32 or /128) never acted upon. Joined by
# the public_ips (same format) of every inventory host, then whitelisted by
# this agent and, when lapi_allowlist is set, allowlisted on the LAPI.
whitelist: {}
# LAPI allowlist kept equal to the whitelist; it also drops alerts on ranges
# holding a whitelisted address, which a parser whitelist cannot.
lapi_allowlist: ""
# Host whose k3s kubectl reaches the LAPI pod (crowdsec/crowdsec-lapi).
lapi_kubectl_host: ""
# name -> acquisition datasource, written to /etc/crowdsec/acquis.d/<name>.yaml.
acquisition: {}
# packagecloud has no trixie suite yet; the bookworm build is static Go.
apt_suite_map:
trixie: bookworm
+10
View File
@@ -0,0 +1,10 @@
---
- name: Restart crowdsec
ansible.builtin.systemd:
name: crowdsec.service
state: restarted
- name: Restart crowdsec-firewall-bouncer
ansible.builtin.systemd:
name: crowdsec-firewall-bouncer.service
state: restarted
+223
View File
@@ -0,0 +1,223 @@
---
- name: Require LAPI settings
ansible.builtin.assert:
that:
- crowdsec.lapi_url | length > 0
- crowdsec.registration_token | length > 0
- crowdsec.bouncer_api_key | length > 0
fail_msg: crowdsec.lapi_url, crowdsec.registration_token and crowdsec.bouncer_api_key must be set
- name: Create /etc/apt/keyrings
ansible.builtin.file:
path: /etc/apt/keyrings
state: directory
mode: "0755"
- name: Add the CrowdSec apt key
ansible.builtin.get_url:
url: https://packagecloud.io/crowdsec/crowdsec/gpgkey
dest: /etc/apt/keyrings/crowdsec.asc
mode: "0644"
- name: Add the CrowdSec apt repository
ansible.builtin.copy:
dest: /etc/apt/sources.list.d/crowdsec.list
mode: "0644"
content: >
deb [signed-by=/etc/apt/keyrings/crowdsec.asc]
https://packagecloud.io/crowdsec/crowdsec/{{ ansible_distribution | lower }}/
{{ crowdsec.apt_suite_map[ansible_distribution_release] | default(ansible_distribution_release) }} main
register: crowdsec_repo
# The postinst asks these; without them it registers a local machine for a
# local LAPI and enrols in the central CAPI.
- name: Preseed the package for a remote LAPI
ansible.builtin.debconf:
name: crowdsec
question: "{{ item.question }}"
vtype: "{{ item.vtype }}"
value: "{{ item.value }}"
loop:
- { question: crowdsec/lapi, vtype: boolean, value: "false" }
- { question: crowdsec/capi, vtype: boolean, value: "false" }
- { question: crowdsec/lapi_host, vtype: string, value: "{{ crowdsec.lapi_url | urlsplit('netloc') }}" }
- name: Create the CrowdSec config directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
mode: "0755"
loop:
- /etc/crowdsec
- /etc/crowdsec/acquis.d
- /etc/crowdsec/bouncers
- /etc/crowdsec/parsers/s02-enrich
- /etc/systemd/system/crowdsec.service.d
- /etc/systemd/system/crowdsec-firewall-bouncer.service.d
- /etc/systemd/system/nftables.service.d
- name: Disable the local API server
ansible.builtin.copy:
dest: /etc/crowdsec/config.yaml.local
mode: "0644"
content: |
api:
server:
enable: false
notify: Restart crowdsec
# The postinst starts the agent before it is registered; an empty credentials
# file then makes the start a no-op instead of a failed package install.
- name: Start the agent only once registered
ansible.builtin.copy:
dest: /etc/systemd/system/crowdsec.service.d/registered.conf
mode: "0644"
content: |
[Unit]
ConditionFileNotEmpty=/etc/crowdsec/local_api_credentials.yaml
register: crowdsec_dropin
# Start, reload and stop of nftables.service flush the whole ruleset with the
# bouncer tables; a restart recreates them with all decisions. Ordering after
# nftables.service makes the queued restarts wait for its job to finish.
- name: Restart the bouncer after nftables.service flushes the ruleset
ansible.builtin.copy:
dest: "{{ item.dest }}"
mode: "0644"
content: "{{ item.content }}"
loop:
- dest: /etc/systemd/system/nftables.service.d/crowdsec-firewall-bouncer.conf
content: |
[Service]
ExecStartPost=-/bin/systemctl --no-block try-restart crowdsec-firewall-bouncer.service
ExecReload=-/bin/systemctl --no-block try-restart crowdsec-firewall-bouncer.service
ExecStopPost=-/bin/systemctl --no-block try-restart crowdsec-firewall-bouncer.service
- dest: /etc/systemd/system/crowdsec-firewall-bouncer.service.d/nftables.conf
content: |
[Unit]
After=nftables.service
loop_control:
label: "{{ item.dest }}"
register: crowdsec_bouncer_dropins
# Present before the package install, so the postinst finds an API key and
# does not try to create one on a local LAPI.
- name: Configure the firewall bouncer
ansible.builtin.template:
src: crowdsec-firewall-bouncer.yaml.local.j2
dest: /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.local
mode: "0600"
notify: Restart crowdsec-firewall-bouncer
- name: Install the parser whitelist
ansible.builtin.template:
src: whitelist.yaml.j2
dest: /etc/crowdsec/parsers/s02-enrich/host-whitelist.yaml
mode: "0644"
notify: Restart crowdsec
- name: Install the acquisition files
ansible.builtin.copy:
dest: "/etc/crowdsec/acquis.d/{{ item.key }}.yaml"
mode: "0644"
content: "{{ item.value | to_nice_yaml }}"
loop: "{{ crowdsec.acquisition | dict2items | rejectattr('value', 'none') }}"
notify: Restart crowdsec
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: true
when: crowdsec_dropin is changed or crowdsec_bouncer_dropins is changed
- name: Install the agent and the nftables bouncer
ansible.builtin.apt:
name: [crowdsec, crowdsec-firewall-bouncer-nftables]
state: present
update_cache: "{{ crowdsec_repo is changed }}"
# The postinst otherwise runs service detection, which needs credentials that only the registration below creates
environment:
CROWDSEC_SETUP_UNATTENDED_DISABLE: "1"
- name: Read the agent credentials
ansible.builtin.command: grep -q '^login:' /etc/crowdsec/local_api_credentials.yaml
register: crowdsec_registered
changed_when: false
failed_when: false
check_mode: false
- name: Register the agent at the LAPI
ansible.builtin.command: >-
cscli lapi register --url {{ crowdsec.lapi_url }}
--token {{ crowdsec.registration_token }} --machine {{ crowdsec.machine_name | default(ansible_hostname, true) }}
when: crowdsec_registered.rc != 0
no_log: true
notify: Restart crowdsec
- name: List the installed hub items
ansible.builtin.command: cscli hub list -o json
register: crowdsec_hub_json
changed_when: false
# cscli is missing in check mode before the first install.
failed_when: crowdsec_hub_json.rc != 0 and not ansible_check_mode
check_mode: false
- name: Install the hub items
ansible.builtin.command: "cscli {{ item }} install {{ crowdsec_missing | join(' ') }}"
vars:
crowdsec_wanted: "{{ crowdsec[item] | dict2items | rejectattr('value', 'none') | map(attribute='value') | unique }}"
crowdsec_present: "{{ (crowdsec_hub_json.stdout | default('{}', true) | from_json)[item] | default([], true) | map(attribute='name') }}"
crowdsec_missing: "{{ crowdsec_wanted | difference(crowdsec_present) }}"
loop: [collections, scenarios]
when: crowdsec_missing | length > 0
notify: Restart crowdsec
- name: Install the local parsers and scenarios
ansible.builtin.copy:
dest: "/etc/crowdsec/{{ item.key }}"
mode: "0644"
content: "{{ item.value }}"
loop: "{{ crowdsec.local_files | dict2items | rejectattr('value', 'none') }}"
loop_control:
label: "{{ item.key }}"
notify: Restart crowdsec
- name: Enable the agent and the bouncer
ansible.builtin.systemd:
name: "{{ item }}"
enabled: true
state: started
loop: [crowdsec.service, crowdsec-firewall-bouncer.service]
- name: Sync the LAPI allowlist
when: crowdsec.lapi_allowlist | length > 0
run_once: true
delegate_to: "{{ crowdsec.lapi_kubectl_host }}"
vars:
crowdsec_lapi_cscli: k3s kubectl -n crowdsec exec deploy/crowdsec-lapi -c crowdsec-lapi -- cscli
block:
- name: Read the LAPI allowlist
ansible.builtin.command: "{{ crowdsec_lapi_cscli }} allowlists inspect {{ crowdsec.lapi_allowlist }} -o json"
register: crowdsec_allowlist_json
changed_when: false
failed_when: false
check_mode: false
- name: Create the LAPI allowlist
ansible.builtin.command: >-
{{ crowdsec_lapi_cscli }} allowlists create {{ crowdsec.lapi_allowlist }}
-d "Internal ranges and the public addresses of all hosts, managed by Ansible"
when: crowdsec_allowlist_json.rc != 0
# cscli stores single addresses without their /32 or /128.
- name: Update the LAPI allowlist
ansible.builtin.command: "{{ crowdsec_lapi_cscli }} allowlists {{ item.action }} {{ crowdsec.lapi_allowlist }} {{ item.cidrs | join(' ') }}"
vars:
crowdsec_allowlist_wanted: "{{ crowdsec_whitelist | map('regex_replace', '/(32|128)$', '') | list }}"
crowdsec_allowlist_present: >-
{{ ((crowdsec_allowlist_json.stdout | default('{}', true) | from_json).get('items') or []) | map(attribute='value') | list }}
loop:
- { action: add, cidrs: "{{ crowdsec_allowlist_wanted | difference(crowdsec_allowlist_present) }}" }
- { action: remove, cidrs: "{{ crowdsec_allowlist_present | difference(crowdsec_allowlist_wanted) }}" }
loop_control:
label: "{{ item.action }} {{ item.cidrs | join(' ') }}"
when: item.cidrs | length > 0
@@ -0,0 +1,8 @@
# {{ ansible_managed }}
api_url: {{ crowdsec.lapi_url }}/
api_key: {{ crowdsec.bouncer_api_key }}
mode: nftables
# forward covers Docker-published ports and Proxmox guests.
nftables_hooks:
- input
- forward
+9
View File
@@ -0,0 +1,9 @@
# {{ ansible_managed }}
name: local/host-whitelist
description: Internal ranges and the public addresses of all hosts
whitelist:
reason: internal or admin source
cidr:
{% for cidr in crowdsec_whitelist %}
- "{{ cidr }}"
{% endfor %}
+4
View File
@@ -0,0 +1,4 @@
---
crowdsec_whitelist: >-
{{ (crowdsec.whitelist | dict2items | rejectattr('value', 'none') | map(attribute='value') | list
+ groups['all'] | map('extract', hostvars, 'public_ips') | select('defined') | flatten) | unique }}
@@ -0,0 +1,15 @@
[Unit]
Description=Mirror the CrowdSec range ban decisions into the nftables table inet crowdsec_ranges
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/crowdsec-range-bouncer
TimeoutStartSec=60
StateDirectory=crowdsec-range-bouncer
StateDirectoryMode=0700
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
@@ -0,0 +1,10 @@
[Unit]
Description=Mirror the CrowdSec range ban decisions into nftables every minute
[Timer]
OnBootSec=30s
OnUnitActiveSec=60s
AccuracySec=5s
[Install]
WantedBy=timers.target
+7
View File
@@ -0,0 +1,7 @@
---
- name: Run the range bouncer
ansible.builtin.systemd:
name: crowdsec-range-bouncer.service
state: started
# Check mode does not install the unit.
ignore_errors: "{{ ansible_check_mode }}"
+53
View File
@@ -0,0 +1,53 @@
---
# crowdsec-firewall-bouncer in nftables mode stores only the network address of
# a range decision (CIDR scope "Range") in a plain address set. This role keeps
# the ranges in interval sets of its own table, next to the firewall bouncer,
# which keeps handling the single addresses.
- name: Require LAPI settings
ansible.builtin.assert:
that:
- crowdsec.lapi_url | length > 0
- crowdsec.range_bouncer_api_key | default('') | length > 0
fail_msg: crowdsec.lapi_url and crowdsec.range_bouncer_api_key must be set
- name: Install python3 and nftables
ansible.builtin.apt:
name: [python3, nftables]
state: present
- name: Configure the range bouncer
ansible.builtin.template:
src: crowdsec-range-bouncer.json.j2
dest: /etc/crowdsec-range-bouncer.json
mode: "0600"
no_log: true
notify: Run the range bouncer
- name: Install the range bouncer
ansible.builtin.template:
src: crowdsec-range-bouncer.py.j2
dest: /usr/local/sbin/crowdsec-range-bouncer
mode: "0755"
notify: Run the range bouncer
- name: Install the systemd units
ansible.builtin.copy:
src: "{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop: [crowdsec-range-bouncer.service, crowdsec-range-bouncer.timer]
register: crowdsec_range_bouncer_units
notify: Run the range bouncer
- name: Enable the range bouncer timer
ansible.builtin.systemd:
name: crowdsec-range-bouncer.timer
enabled: true
state: started
daemon_reload: "{{ crowdsec_range_bouncer_units is changed }}"
# Check mode does not install the unit.
ignore_errors: "{{ ansible_check_mode }}"
# The first run has to succeed before the play moves on to the next host.
- name: Apply the changes now
ansible.builtin.meta: flush_handlers
@@ -0,0 +1 @@
{{ {'lapi_url': crowdsec.lapi_url, 'api_key': crowdsec.range_bouncer_api_key, 'allow': crowdsec_range_bouncer_allow} | to_nice_json }}
@@ -0,0 +1,124 @@
#!/usr/bin/python3
# {{ ansible_managed }}
"""Mirror the CrowdSec range ban decisions into the nftables table inet crowdsec_ranges.
Stateless: every run fetches all range decisions from the LAPI, builds the
whole table and loads it in one transaction. The table is only touched when
its text changed or the table is gone (nftables.service flushes the whole
ruleset on reload). Any error leaves the current table in place.
"""
import hashlib
import ipaddress
import json
import os
import subprocess
import sys
import urllib.request
CONFIG = "/etc/crowdsec-range-bouncer.json"
STATE_DIR = os.environ.get("STATE_DIRECTORY", "/var/lib/crowdsec-range-bouncer")
HASH_FILE = os.path.join(STATE_DIR, "last.sha256")
NFT = "/usr/sbin/nft"
TABLE = "crowdsec_ranges"
def fetch_ranges(lapi_url, api_key):
"""Return the networks of all ban decisions with scope Range."""
request = urllib.request.Request(
lapi_url.rstrip("/") + "/v1/decisions?scopes=range",
headers={"X-Api-Key": api_key, "User-Agent": "crowdsec-range-bouncer"},
)
with urllib.request.urlopen(request, timeout=20) as response:
decisions = json.load(response)
# The LAPI answers a literal null when nothing matches.
if decisions is None:
return []
if not isinstance(decisions, list):
raise ValueError("unexpected LAPI answer: %s" % type(decisions).__name__)
return [
ipaddress.ip_network(d["value"], strict=False)
for d in decisions
if d["type"] == "ban" and d["scope"].lower() == "range"
]
def collapse(networks, version):
"""Merge overlapping and adjacent networks: interval set elements must not overlap."""
return [str(n) for n in ipaddress.collapse_addresses(n for n in networks if n.version == version)]
def set_text(name, address_type, elements):
text = "\tset %s {\n\t\ttype %s\n\t\tflags interval\n" % (name, address_type)
if elements:
text += "\t\telements = {\n\t\t\t%s\n\t\t}\n" % ",\n\t\t\t".join(elements)
return text + "\t}\n"
def table_text(allow, block):
# The accept of an allowed address only ends the chain of this table:
# host_firewall, Docker, Cilium and the CrowdSec firewall bouncer still decide.
# Same hooks and priority as the firewall bouncer: input, and forward for
# published Docker ports and guests.
rules = (
"\t\tip saddr @allow4 accept\n"
"\t\tip6 saddr @allow6 accept\n"
"\t\tip saddr @block4 counter drop\n"
"\t\tip6 saddr @block6 counter drop\n"
)
return (
"table inet %(t)s {}\n"
"delete table inet %(t)s\n"
"table inet %(t)s {\n"
% {"t": TABLE}
+ set_text("allow4", "ipv4_addr", collapse(allow, 4))
+ set_text("allow6", "ipv6_addr", collapse(allow, 6))
+ set_text("block4", "ipv4_addr", collapse(block, 4))
+ set_text("block6", "ipv6_addr", collapse(block, 6))
+ "\tchain input {\n\t\ttype filter hook input priority -10; policy accept;\n"
+ "\t\tiifname \"lo\" accept\n"
+ rules
+ "\t}\n"
+ "\tchain forward {\n\t\ttype filter hook forward priority -10; policy accept;\n"
+ rules
+ "\t}\n}\n"
)
def table_exists():
result = subprocess.run(
[NFT, "list", "table", "inet", TABLE], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
)
return result.returncode == 0
def main():
with open(CONFIG) as f:
config = json.load(f)
allow = [ipaddress.ip_network(v, strict=False) for v in config["allow"]]
block = fetch_ranges(config["lapi_url"], config["api_key"])
text = table_text(allow, block)
digest = hashlib.sha256(text.encode()).hexdigest()
try:
with open(HASH_FILE) as f:
last = f.read().strip()
except FileNotFoundError:
last = ""
if digest == last and table_exists():
return
subprocess.run([NFT, "-f", "-"], input=text.encode(), check=True)
tmp = HASH_FILE + ".tmp"
with open(tmp, "w") as f:
f.write(digest + "\n")
os.replace(tmp, HASH_FILE)
print("loaded %d IPv4 and %d IPv6 ranges from %d decisions" % (
len(collapse(block, 4)), len(collapse(block, 6)), len(block)))
if __name__ == "__main__":
try:
main()
except Exception as e:
print("error, keeping the current table: %s: %s" % (type(e).__name__, e), file=sys.stderr)
sys.exit(1)
+6
View File
@@ -0,0 +1,6 @@
---
# Same sources as the crowdsec role (roles/crowdsec/vars/main.yml), plus loopback.
crowdsec_range_bouncer_allow: >-
{{ (crowdsec.whitelist | default({}) | dict2items | rejectattr('value', 'none') | map(attribute='value') | list
+ groups['all'] | map('extract', hostvars, 'public_ips') | select('defined') | flatten
+ ['127.0.0.0/8', '::1/128']) | unique }}
+4
View File
@@ -0,0 +1,4 @@
---
# Adds prometheus-smartctl-exporter (trixie-backports) next to the node exporter
# on hosts with physical disks.
host_exporters_smart: false
+71
View File
@@ -0,0 +1,71 @@
---
# Both exporters listen on all addresses; host_firewall limits who reaches them.
# Its recommends (smartmontools, nvme-cli, ipmitool) start units that fail on hosts without such hardware.
- name: Install the node exporter
ansible.builtin.apt:
name: prometheus-node-exporter
install_recommends: false
state: present
- name: Enable the node exporter
ansible.builtin.systemd:
name: prometheus-node-exporter
enabled: true
state: started
# Check mode does not install the package.
ignore_errors: "{{ ansible_check_mode }}"
- name: Install the smartctl exporter
when: host_exporters_smart
block:
- name: Look for an existing backports source
ansible.builtin.shell: >-
grep -rEqs '^[^#]*{{ ansible_distribution_release }}-backports'
--include='*.list' --include='*.sources' --exclude=host-exporters-backports.sources /etc/apt
register: host_exporters_backports
changed_when: false
failed_when: host_exporters_backports.rc > 1
check_mode: false
# Backports stay at priority 100, so only packages named with default_release come from there.
- name: Add the backports source
ansible.builtin.copy:
dest: /etc/apt/sources.list.d/host-exporters-backports.sources
mode: "0644"
content: |
Types: deb
URIs: http://deb.debian.org/debian
Suites: {{ ansible_distribution_release }}-backports
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
register: host_exporters_source
when: host_exporters_backports.rc == 1
# The module rejects default_release while the release is missing from its package cache.
- name: Look for the backports release in the package lists
ansible.builtin.shell: apt-cache policy | grep -q 'n={{ ansible_distribution_release }}-backports'
register: host_exporters_lists
changed_when: false
failed_when: false
check_mode: false
- name: Refresh the package lists for the backports source
ansible.builtin.apt:
update_cache: true
when: host_exporters_lists.rc != 0
- name: Install the smartctl exporter from backports
ansible.builtin.apt:
name: prometheus-smartctl-exporter
default_release: "{{ ansible_distribution_release }}-backports"
state: present
# Check mode does not write the backports source.
ignore_errors: "{{ ansible_check_mode }}"
# The packaged unit runs as root, which smartctl needs for the disks.
- name: Enable the smartctl exporter
ansible.builtin.systemd:
name: smartctl_exporter
enabled: true
state: started
ignore_errors: "{{ ansible_check_mode }}"
+71
View File
@@ -0,0 +1,71 @@
---
# Port and address maps are name -> value so group_vars and host_vars levels
# merge into one map (hash_behaviour = merge); a null value drops an entry.
# Ports are numbers or "first-last" ranges.
firewall:
public:
tcp: {}
udp: {}
# Accepted only from the geo sets or the admin set.
geo:
tcp: {}
# Accepted on wt0 before the public zone is evaluated.
vpn:
tcp: {}
udp: {}
# tcp: ports accepted from the admin set alone, never from the geo sets.
admin:
tcp: {}
v4: {}
v6: {}
# Metrics scrapers: on wt0 only these source addresses reach the listed tcp
# ports (the vpn zone applies to every peer). Rendered only when both are set.
scrapers:
tcp: {}
v4: {}
v6: {}
# Interface names (nft wildcards allowed) whose traffic is trusted.
internal_ifaces: {}
# Guest bridges (Proxmox vmbr*): DHCP, router solicitations and the public
# zone only, never the geo ports.
guest_ifaces: {}
# Kubernetes ring: pod interfaces, and node NetBird IPs / pod CIDRs on wt0.
cluster:
ifaces: {}
v4: {}
v6: {}
# IPv4 port forwards: name -> {daddr, to, tcp: ports, udp: ports}, ports as in
# the maps above. Forwarded traffic passes the forward hook, not input.
dnat: {}
# IPv4 source NAT: name -> nft match, e.g. 'oifname "vmbr0"'.
masquerade: {}
# Forwarded traffic to guarded guests: name -> {ifaces, v4, v6}, where v4 and
# v6 are {daddr, tcp: ports, icmp: icmp or icmpv6 types}. Replies and traffic
# in on ifaces pass, the listed ports and types pass, the rest to daddr drops;
# traffic to other addresses is not filtered.
forward_guard: {}
ssh_rate_limit: "10/minute burst 5 packets"
# Kernel log of dropped inbound packets (fw-in-drop: ) and of rejected
# outbound mail (fw-out-reject: ), read from the journal by CrowdSec. Each
# rate is one limit shared by all matching packets.
log:
in_rate: "5/second burst 30 packets"
out_rate: "6/minute burst 5 packets"
# Lets the kernel log the nft log statements of container network
# namespaces (LXC guests) into the host kernel log.
all_netns: false
# Shared by the Spamhaus drop (fw-spamhaus-in: ) and reject (fw-spamhaus-out: ) chains, one limit each.
spamhaus_rate: "6/minute burst 10 packets"
# Spamhaus DROP, DROPv6 and ASN-DROP: traffic from the listed networks drops
# (input, forward), traffic to them is rejected (output, forward). The sets
# are refreshed daily by nft-host-spamhaus.
spamhaus: true
# Reject outbound TCP 25; the mail group turns it off.
block_smtp_out: true
# EU-27 plus CH, NO, GB, IS, LI (ipdeny country codes). Empty leaves the geo
# ports to the admin set; a load peaks at about 60 MB of kernel memory, too much for a 1 GB host.
geo_countries: [at, be, bg, hr, cy, cz, dk, ee, fi, fr, de, gr, hu, ie, it, lv, lt, lu, mt, nl, pl, pt, ro, sk, si, es, se, ch, "no", gb, is, li]
rollback_seconds: 120
# name -> shell command that has to exit 0 through the new rule set before the
# rollback is disarmed; retried for a quarter of the rollback window.
confirm: {}
+8
View File
@@ -0,0 +1,8 @@
[Unit]
Description=Refresh the geo sets of the host firewall
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/nft-host-geo
+10
View File
@@ -0,0 +1,10 @@
[Unit]
Description=Daily refresh of the host firewall geo sets
[Timer]
OnCalendar=daily
RandomizedDelaySec=6h
Persistent=true
[Install]
WantedBy=timers.target
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
# Started by a transient timer before a rule set load; Ansible stops the timer
# once the new rule set is confirmed.
cd /etc/nftables.d || exit 1
mv -f host.nft host.nft.rolled-back
if [ -s host.nft.good ]; then
cp host.nft.good host.nft && nft -f host.nft && exit 0
fi
nft delete table inet host
+167
View File
@@ -0,0 +1,167 @@
#!/usr/bin/python3
# Managed by Ansible (role host_firewall), do not edit.
"""Rebuilds the Spamhaus sets spamhaus_v4 and spamhaus_v6 of table inet host.
Sources are the Spamhaus DROP and DROPv6 prefix lists and the ASN-DROP list,
whose ASNs are expanded to prefixes with the iptoasn bulk files (one download
per address family). Any failed download or sanity check exits non-zero before
anything changes, so the previous sets stay in place.
"""
import gzip
import http.client
import ipaddress
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
import urllib.request
PERSISTED = "/etc/nftables.d/host-spamhaus.nft"
SPAMHAUS = "https://www.spamhaus.org/drop/"
IPTOASN = "https://iptoasn.com/data/"
USER_AGENT = "ruekov-host-firewall (nft-host-spamhaus)"
# Lower bounds that catch an empty or truncated answer.
MIN_RECORDS = {"drop_v4": 800, "drop_v6": 30, "asndrop": 200}
MIN_TSV_LINES = {4: 200000, 6: 20000}
MIN_ASN_PREFIXES_V4 = 100
# A prefix this wide is a parse error, not a bad network.
MIN_PREFIXLEN = {4: 8, 6: 16}
def fail(message):
print(f"nft-host-spamhaus: {message}", file=sys.stderr)
sys.exit(1)
def fetch(url, dest):
error = None
for attempt in range(3):
try:
request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
with urllib.request.urlopen(request, timeout=120) as response, open(dest, "wb") as out:
shutil.copyfileobj(response, out)
return
except (OSError, http.client.HTTPException) as e:
error = e
time.sleep(5 * (attempt + 1))
fail(f"download of {url} failed: {error}")
def spamhaus_records(name, tmp):
path = os.path.join(tmp, name + ".json")
fetch(f"{SPAMHAUS}{name}.json", path)
records, meta = [], None
with open(path, encoding="utf-8") as f:
for line in f:
if not line.strip():
continue
record = json.loads(line)
if record.get("type") == "metadata":
meta = record
else:
records.append(record)
if meta is None or meta.get("records") != len(records):
fail(f"{name}: record count does not match the metadata record")
if len(records) < MIN_RECORDS[name]:
fail(f"{name}: only {len(records)} records")
return records
def asn_prefixes(asns, family, tmp):
path = os.path.join(tmp, f"ip2asn-v{family}.tsv.gz")
fetch(f"{IPTOASN}ip2asn-v{family}.tsv.gz", path)
networks, lines = [], 0
with gzip.open(path, "rt", encoding="utf-8") as f:
for line in f:
lines += 1
fields = line.split("\t", 3)
if len(fields) < 3 or fields[2] not in asns:
continue
networks.extend(ipaddress.summarize_address_range(
ipaddress.ip_address(fields[0]), ipaddress.ip_address(fields[1])))
if lines < MIN_TSV_LINES[family]:
fail(f"ip2asn-v{family}: only {lines} lines")
return networks
def merge(family, *groups):
networks, skipped = [], 0
for group in groups:
for network in group:
network = ipaddress.ip_network(network)
if network.prefixlen < MIN_PREFIXLEN[family]:
fail(f"suspiciously wide prefix {network}")
if not network.is_global:
skipped += 1
continue
networks.append(network)
return list(ipaddress.collapse_addresses(networks)), skipped
def nft(*args):
return subprocess.run(["nft", *args], capture_output=True, text=True)
def main():
with tempfile.TemporaryDirectory() as tmp:
drop4 = [r["cidr"] for r in spamhaus_records("drop_v4", tmp)]
drop6 = [r["cidr"] for r in spamhaus_records("drop_v6", tmp)]
asns = {str(r["asn"]) for r in spamhaus_records("asndrop", tmp)}
asn4 = asn_prefixes(asns, 4, tmp)
asn6 = asn_prefixes(asns, 6, tmp)
if len(asn4) < MIN_ASN_PREFIXES_V4:
fail(f"only {len(asn4)} IPv4 prefixes for {len(asns)} ASNs")
sets = {}
for family, name, drop, asn in ((4, "spamhaus_v4", drop4, asn4), (6, "spamhaus_v6", drop6, asn6)):
sets[name], skipped = merge(family, drop, asn)
print(f"{name}: {len(drop)} DROP + {len(asn)} ASN-DROP prefixes -> "
f"{len(sets[name])} merged, {skipped} non-global skipped")
print(f"ASN-DROP: {len(asns)} ASNs")
def elements(networks):
return ",\n\t\t".join(str(n) for n in networks) + ","
persisted = ["# Generated by nft-host-spamhaus, do not edit.", "table inet host {"]
for name, kind in (("spamhaus_v4", "ipv4_addr"), ("spamhaus_v6", "ipv6_addr")):
persisted.append(f"\tset {name} {{ type {kind}; flags interval; auto-merge; elements = {{\n"
f"\t\t{elements(sets[name])}\n\t}} }}")
persisted.append("}\n")
candidate = os.path.join(tmp, "host-spamhaus.nft")
with open(candidate, "w", encoding="utf-8") as f:
f.write("\n".join(persisted))
checked = nft("-c", "-f", candidate)
if checked.returncode != 0:
fail(f"nft rejects the sets: {checked.stderr.strip()}")
previous = None
if os.path.exists(PERSISTED):
previous = os.path.join(tmp, "previous.nft")
shutil.copyfile(PERSISTED, previous)
shutil.copyfile(candidate, PERSISTED + ".new")
os.chmod(PERSISTED + ".new", 0o644)
os.replace(PERSISTED + ".new", PERSISTED)
# A reload replaces the whole table, sets included, in one transaction.
# Flushing and refilling a running interval set instead lets packets of
# unlisted addresses match while the transaction commits.
active = subprocess.run(["systemctl", "-q", "is-active", "nft-host.service"]).returncode == 0
if active:
reloaded = subprocess.run(["systemctl", "reload", "nft-host.service"], capture_output=True, text=True)
if reloaded.returncode != 0:
if previous:
shutil.copyfile(previous, PERSISTED)
fail(f"reloading the rule set failed, kept the previous sets: {reloaded.stderr.strip()}")
print("reloaded the rule set" if active else "persisted only")
try:
main()
except SystemExit:
raise
except Exception as e: # keep the previous sets on any surprise
fail(f"{type(e).__name__}: {e}")
@@ -0,0 +1,8 @@
[Unit]
Description=Refresh the Spamhaus sets of the host firewall
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/nft-host-spamhaus
@@ -0,0 +1,10 @@
[Unit]
Description=Daily refresh of the host firewall Spamhaus sets
[Timer]
OnCalendar=daily
RandomizedDelaySec=6h
Persistent=true
[Install]
WantedBy=timers.target
+22
View File
@@ -0,0 +1,22 @@
[Unit]
Description=Host input firewall and NAT (nftables tables inet host and ip host_nat)
# Same early-boot placement as nftables.service. Its "flush ruleset" on start,
# reload and stop would wipe this table, so load after it and follow all three.
DefaultDependencies=no
Wants=network-pre.target
Before=network-pre.target shutdown.target
After=nftables.service
PartOf=nftables.service
ReloadPropagatedFrom=nftables.service
Conflicts=shutdown.target
ConditionPathExists=/etc/nftables.d/host.nft
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nft -f /etc/nftables.d/host.nft
ExecReload=/usr/sbin/nft -f /etc/nftables.d/host.nft
ExecStop=-/usr/sbin/nft delete table inet host
[Install]
WantedBy=sysinit.target
+234
View File
@@ -0,0 +1,234 @@
---
- name: Install nftables and curl
ansible.builtin.apt:
name: [nftables, curl]
state: present
- name: Check that this node is part of its own cluster ring
ansible.builtin.assert:
that: ansible_wt0.ipv4.address in (firewall.cluster.v4 | dict2items | map(attribute='value') | list)
fail_msg: "wt0 address {{ ansible_wt0.ipv4.address }} is missing from firewall.cluster.v4"
when: firewall.cluster.v4 | default({}, true) | length > 0 and ansible_wt0.ipv4.address is defined
- name: Remove the include block of the old nftables role
ansible.builtin.blockinfile:
path: /etc/nftables.conf
marker: "# {mark} ANSIBLE MANAGED BLOCK - nftables"
state: absent
- name: Remove the rule files of the old nftables role
ansible.builtin.file:
path: /etc/nftables/ansible-managed
state: absent
- name: Create /etc/nftables.d
ansible.builtin.file:
path: /etc/nftables.d
state: directory
mode: "0755"
# The kernel logs nft log statements of other network namespaces (LXC guests)
# only with this sysctl; the lines land in the host kernel log.
- name: Log the firewalls of container network namespaces
ansible.posix.sysctl:
name: net.netfilter.nf_log_all_netns
value: "1"
sysctl_file: /etc/sysctl.d/90-nf-log-all-netns.conf
sysctl_set: true
reload: false
when: firewall.log.all_netns
- name: Install the systemd units
ansible.builtin.copy:
src: "{{ item }}"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop: [nft-host.service, nft-host-geo.service, nft-host-geo.timer, nft-host-spamhaus.service, nft-host-spamhaus.timer]
register: firewall_units
- name: Install the rollback script
ansible.builtin.copy:
src: nft-host-rollback
dest: /usr/local/sbin/nft-host-rollback
mode: "0755"
- name: Geo sets
when: firewall.geo_countries | length > 0
block:
- name: Install the geo set updater
ansible.builtin.template:
src: nft-host-geo.j2
dest: /usr/local/sbin/nft-host-geo
mode: "0755"
- name: Build the geo sets on first run
ansible.builtin.command: /usr/local/sbin/nft-host-geo
args:
creates: /etc/nftables.d/host-geo.nft
- name: Read the geo sets file
ansible.builtin.stat:
path: /etc/nftables.d/host-geo.nft
register: firewall_geo
# Without geo sets the public SSH port would only answer NetBird and the admin set.
- name: Require non-empty geo sets
ansible.builtin.assert:
that: firewall_geo.stat.exists and firewall_geo.stat.size > 0
fail_msg: /etc/nftables.d/host-geo.nft is missing or empty, refusing to apply the firewall
# Check mode neither installs the updater nor builds the sets.
when: not ansible_check_mode
- name: Enable the geo set timer
ansible.builtin.systemd:
name: nft-host-geo.timer
enabled: true
state: started
daemon_reload: "{{ firewall_units is changed }}"
# Check mode does not install the unit.
ignore_errors: "{{ ansible_check_mode }}"
- name: Stop the geo set refresh
when: firewall.geo_countries | length == 0
block:
- name: Disable the geo set timer
ansible.builtin.systemd:
name: nft-host-geo.timer
enabled: false
state: stopped
daemon_reload: "{{ firewall_units is changed }}"
# Check mode does not install the unit.
ignore_errors: "{{ ansible_check_mode }}"
- name: Remove the geo sets file
ansible.builtin.file:
path: /etc/nftables.d/host-geo.nft
state: absent
- name: Spamhaus sets
when: firewall.spamhaus
block:
- name: Install the Spamhaus set updater
ansible.builtin.copy:
src: nft-host-spamhaus
dest: /usr/local/sbin/nft-host-spamhaus
mode: "0755"
# The rule set includes this file, so a missing one would fail the load at boot.
- name: Build the Spamhaus sets on first run
ansible.builtin.command: /usr/local/sbin/nft-host-spamhaus
args:
creates: /etc/nftables.d/host-spamhaus.nft
- name: Read the Spamhaus sets file
ansible.builtin.stat:
path: /etc/nftables.d/host-spamhaus.nft
register: firewall_spamhaus
- name: Require non-empty Spamhaus sets
ansible.builtin.assert:
that: firewall_spamhaus.stat.exists and firewall_spamhaus.stat.size > 0
fail_msg: /etc/nftables.d/host-spamhaus.nft is missing or empty, refusing to apply the firewall
# Check mode neither installs the updater nor builds the sets.
when: not ansible_check_mode
- name: Enable the Spamhaus set timer
ansible.builtin.systemd:
name: nft-host-spamhaus.timer
enabled: true
state: started
daemon_reload: "{{ firewall_units is changed }}"
# Check mode does not install the unit.
ignore_errors: "{{ ansible_check_mode }}"
- name: Stop the Spamhaus set refresh
when: not firewall.spamhaus
block:
- name: Disable the Spamhaus set timer
ansible.builtin.systemd:
name: nft-host-spamhaus.timer
enabled: false
state: stopped
daemon_reload: "{{ firewall_units is changed }}"
# Check mode does not install the unit.
ignore_errors: "{{ ansible_check_mode }}"
- name: Remove the Spamhaus sets file
ansible.builtin.file:
path: /etc/nftables.d/host-spamhaus.nft
state: absent
- name: Render the rule set
ansible.builtin.template:
src: host.nft.j2
dest: /etc/nftables.d/host.nft
mode: "0644"
validate: nft -c -f %s
register: firewall_rules
- name: Read the firewall unit state
ansible.builtin.command: systemctl is-active nft-host.service
register: firewall_active
changed_when: false
failed_when: false
check_mode: false
- name: Apply the rule set with a rollback timer
when:
- firewall_rules is changed or firewall_units is changed or firewall_active.stdout != 'active'
- not ansible_check_mode
block:
# A failed run of the transient unit keeps its name taken for systemd-run.
- name: Clear a leftover rollback timer
ansible.builtin.shell: >-
systemctl stop nft-host-rollback.timer;
systemctl reset-failed nft-host-rollback.timer nft-host-rollback.service;
true
changed_when: false
# Unless disarmed below, this moves the rejected rule file aside so a reboot
# does not load it again, and reloads the last confirmed one; without one
# it drops the table.
- name: Arm the rollback
ansible.builtin.command: systemd-run --on-active={{ firewall.rollback_seconds }} --unit=nft-host-rollback /usr/local/sbin/nft-host-rollback
changed_when: false
- name: Load the rule set
ansible.builtin.systemd:
name: nft-host.service
enabled: true
state: reloaded
daemon_reload: true
# A reused SSH control socket would hide a lockout, so test a fresh connection.
- name: Drop the SSH connection
ansible.builtin.meta: reset_connection
- name: Reconnect through the new rule set
ansible.builtin.wait_for_connection:
timeout: "{{ (firewall.rollback_seconds / 2) | int }}"
- name: Check the services behind the new rule set
ansible.builtin.shell: "{{ item.value }}"
loop: "{{ firewall.confirm | default({}, true) | dict2items | rejectattr('value', 'none') }}"
loop_control:
label: "{{ item.key }}"
register: firewall_confirm
until: firewall_confirm.rc == 0
retries: "{{ (firewall.rollback_seconds / 20) | int }}"
delay: 5
changed_when: false
# Fails when the timer already fired, so a slow check cannot pass for a rolled-back host.
- name: Disarm the rollback
ansible.builtin.shell: >-
test "$(systemctl show --property=SubState --value nft-host-rollback.timer)" = waiting &&
systemctl stop nft-host-rollback.timer
changed_when: true
- name: Keep the rule set as the last confirmed one
ansible.builtin.copy:
src: /etc/nftables.d/host.nft
dest: /etc/nftables.d/host.nft.good
remote_src: true
mode: "0644"
+251
View File
@@ -0,0 +1,251 @@
#!/usr/sbin/nft -f
# {{ ansible_managed }}
{% macro list(map) -%}
{{ map | default({}, true) | dict2items | rejectattr('value', 'none') | map(attribute='value') | map('string') | unique | join(', ') }}
{%- endmacro %}
{% macro addr_set(name, type, map) -%}
set {{ name }} {
type {{ type }}; flags interval; auto-merge;
{% if list(map) %}
elements = { {{ list(map) }} }
{% endif %}
}
{%- endmacro %}
{% set cluster = firewall.cluster | default({}, true) %}
{% set cluster_ifaces = cluster.ifaces | default({}, true) | dict2items | rejectattr('value', 'none') | map(attribute='value') | unique | list %}
{% set internal_ifaces = firewall.internal_ifaces | default({}, true) | dict2items | rejectattr('value', 'none') | map(attribute='value') | unique | list %}
{% set guest_ifaces = firewall.guest_ifaces | default({}, true) | dict2items | rejectattr('value', 'none') | map(attribute='value') | unique | list %}
{% set guest_match = 'iifname { "' ~ (guest_ifaces | join('", "')) ~ '" }' %}
{% set dnat = firewall.dnat | default({}, true) | dict2items | rejectattr('value', 'none') | map(attribute='value') | list %}
{% set masquerade = firewall.masquerade | default({}, true) | dict2items | rejectattr('value', 'none') | map(attribute='value') | unique | list %}
{% set scrapers = firewall.scrapers | default({}, true) %}
{% set forward_guard = firewall.forward_guard | default({}, true) | dict2items | rejectattr('value', 'none') | list %}
# Declare-then-delete makes every load one atomic transaction that replaces
# only these tables; Cilium, Docker, NetBird and CrowdSec tables stay untouched.
table inet host {}
delete table inet host
table ip host_nat {}
delete table ip host_nat
{% if firewall.geo_countries %}
include "/etc/nftables.d/host-geo.nft"
{% endif %}
{% if firewall.spamhaus %}
include "/etc/nftables.d/host-spamhaus.nft"
{% endif %}
table inet host {
{{ addr_set('admin_v4', 'ipv4_addr', firewall.admin.v4) }}
{{ addr_set('admin_v6', 'ipv6_addr', firewall.admin.v6) }}
{{ addr_set('cluster_v4', 'ipv4_addr', cluster.v4) }}
{{ addr_set('cluster_v6', 'ipv6_addr', cluster.v6) }}
{{ addr_set('scrapers_v4', 'ipv4_addr', scrapers.v4) }}
{{ addr_set('scrapers_v6', 'ipv6_addr', scrapers.v6) }}
set ssh_meter_v4 {
type ipv4_addr; flags dynamic; timeout 10m; size 65535;
}
set ssh_meter_v6 {
type ipv6_addr; flags dynamic; timeout 10m; size 65535;
}
chain input {
type filter hook input priority filter; policy drop;
{% if firewall.spamhaus %}
# Before the established rule: outbound is blocked too, so listed
# addresses have no legitimate replies.
ip saddr @spamhaus_v4 jump spamhaus_in
ip6 saddr @spamhaus_v6 jump spamhaus_in
{% endif %}
ct state established,related accept
ct state invalid jump drop_log
iif lo accept
icmp type { echo-request, destination-unreachable, time-exceeded, parameter-problem } accept
icmpv6 type { echo-request, destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld-listener-query } accept
udp sport 67 udp dport 68 accept
meta nfproto ipv6 udp sport 547 udp dport 546 accept
{% if cluster_ifaces or list(cluster.v4) or list(cluster.v6) %}
# ponytail: the cluster ring trusts every port between nodes and pods;
# per-port lists (6443, 10250, 2379-2380, 4240, 8472, ...) are the upgrade path.
{% for iface in cluster_ifaces %}
iifname "{{ iface }}" accept
{% endfor %}
iifname "wt0" ip saddr @cluster_v4 accept
iifname "wt0" ip6 saddr @cluster_v6 accept
{% endif %}
{% if internal_ifaces %}
# ponytail: internal bridges are trusted on every port; per-port lists are the upgrade path.
{% for iface in internal_ifaces %}
iifname "{{ iface }}" accept
{% endfor %}
{% endif %}
{% if guest_ifaces %}
# Pods on k3s guests egress through the node IP, so guest bridges get
# only DHCP and router solicitations beyond the public zone.
{{ guest_match }} udp dport 67 accept
{{ guest_match }} icmpv6 type nd-router-solicit accept
{% endif %}
# VPN peers get the vpn ports, then everything public.
iifname "wt0" jump vpn
jump public
# Broadcasts and multicasts are never scans; they stay out of the log.
meta pkttype { broadcast, multicast } drop
jump drop_log
}
# A single rule makes the limit global: a flood logs at the limit and the
# rest drops silently. The kernel log line has the iptables LOG fields that
# the CrowdSec iptables-logs parser reads.
chain drop_log {
limit rate {{ firewall.log.in_rate }} log prefix "fw-in-drop: "
drop
}
{% if firewall.spamhaus %}
# Own limits, so a listed scanner cannot use up the budget of fw-in-drop.
# CrowdSec whitelists both prefixes: the lists already ban these networks.
chain spamhaus_in {
limit rate {{ firewall.log.spamhaus_rate }} log prefix "fw-spamhaus-in: "
drop
}
chain spamhaus_out {
limit rate {{ firewall.log.spamhaus_rate }} log prefix "fw-spamhaus-out: "
reject
}
{% endif %}
{% if firewall.block_smtp_out or firewall.spamhaus %}
# Outbound mail goes through the mail host only. Loopback stays open for a
# local MTA; the log lines carry OUT=, which CrowdSec does not count as a scan.
chain output {
type filter hook output priority filter; policy accept;
{% if firewall.spamhaus %}
ip daddr @spamhaus_v4 jump spamhaus_out
ip6 daddr @spamhaus_v6 jump spamhaus_out
{% endif %}
{% if firewall.block_smtp_out %}
oifname != "lo" tcp dport 25 limit rate {{ firewall.log.out_rate }} log prefix "fw-out-reject: "
oifname != "lo" tcp dport 25 reject with tcp reset
{% endif %}
}
{% endif %}
chain vpn {
{% if list(scrapers.tcp) %}
{% if list(scrapers.v4) %}
ip saddr @scrapers_v4 tcp dport { {{ list(scrapers.tcp) }} } accept
{% endif %}
{% if list(scrapers.v6) %}
ip6 saddr @scrapers_v6 tcp dport { {{ list(scrapers.tcp) }} } accept
{% endif %}
{% endif %}
{% if list(firewall.vpn.tcp) %}
tcp dport { {{ list(firewall.vpn.tcp) }} } accept
{% endif %}
{% if list(firewall.vpn.udp) %}
udp dport { {{ list(firewall.vpn.udp) }} } accept
{% endif %}
}
chain public {
{% if list(firewall.public.tcp) %}
tcp dport { {{ list(firewall.public.tcp) }} } accept
{% endif %}
{% if list(firewall.public.udp) %}
udp dport { {{ list(firewall.public.udp) }} } accept
{% endif %}
{% if list(firewall.geo.tcp) %}
tcp dport { {{ list(firewall.geo.tcp) }} } jump geo
{% endif %}
{% if list(firewall.admin.tcp) %}
tcp dport { {{ list(firewall.admin.tcp) }} } jump admin
{% endif %}
}
{% if list(firewall.admin.tcp) %}
chain admin {
ip saddr @admin_v4 accept
ip6 saddr @admin_v6 accept
}
{% endif %}
chain geo {
{% if guest_ifaces %}
# Guest IPv6 addresses sit inside the geo sets.
{{ guest_match }} return
{% endif %}
ip saddr @admin_v4 accept
ip6 saddr @admin_v6 accept
tcp dport 22 ct state new update @ssh_meter_v4 { ip saddr limit rate over {{ firewall.ssh_rate_limit }} } jump drop_log
# One IPv6 client usually holds a whole /64.
tcp dport 22 ct state new update @ssh_meter_v6 { ip6 saddr & ffff:ffff:ffff:ffff:: limit rate over {{ firewall.ssh_rate_limit }} } jump drop_log
{% if firewall.geo_countries %}
ip saddr @geo_v4 accept
ip6 saddr @geo_v6 accept
{% endif %}
}
{% if forward_guard or firewall.spamhaus %}
chain forward {
type filter hook forward priority filter; policy accept;
{% if firewall.spamhaus %}
ip saddr @spamhaus_v4 jump spamhaus_in
ip6 saddr @spamhaus_v6 jump spamhaus_in
ip daddr @spamhaus_v4 jump spamhaus_out
ip6 daddr @spamhaus_v6 jump spamhaus_out
{% endif %}
{% for guard in forward_guard %}
# {{ guard.key }}
{% set ifaces = guard.value.ifaces | default({}, true) | dict2items | rejectattr('value', 'none') | map(attribute='value') | unique | list %}
{% for fam, l3, icmp in [('v4', 'ip', 'icmp'), ('v6', 'ip6', 'icmpv6')] if list((guard.value[fam] | default({}, true)).daddr) %}
{% set zone = guard.value[fam] %}
{% set match = l3 ~ ' daddr { ' ~ list(zone.daddr) ~ ' }' %}
{{ match }} ct state established,related accept
{% if ifaces %}
{{ match }} iifname { "{{ ifaces | join('", "') }}" } accept
{% endif %}
{% if list(zone.tcp) %}
{{ match }} tcp dport { {{ list(zone.tcp) }} } accept
{% endif %}
{% if list(zone.icmp) %}
{{ match }} {{ icmp }} type { {{ list(zone.icmp) }} } accept
{% endif %}
{{ match }} jump drop_log
{% endfor %}
{% endfor %}
}
{% endif %}
}
{% if dnat or masquerade %}
table ip host_nat {
chain prerouting {
type nat hook prerouting priority dstnat; policy accept;
{% for fwd in dnat %}
{% for proto in ['tcp', 'udp'] if list(fwd[proto]) %}
ip daddr {{ fwd.daddr }} {{ proto }} dport { {{ list(fwd[proto]) }} } dnat to {{ fwd.to }}
{% endfor %}
{% endfor %}
}
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
{% for match in masquerade %}
{{ match }} masquerade
{% endfor %}
}
}
{% endif %}
+46
View File
@@ -0,0 +1,46 @@
#!/bin/sh
# {{ ansible_managed }}
# Rebuilds the geo sets of table inet host from ipdeny. Any failed download or
# check exits before the swap, so the previous file stays in place.
set -eu
out=/etc/nftables.d/host-geo.nft
countries="{{ firewall.geo_countries | join(' ') }}"
prefix='^[0-9a-f.:]+/[0-9]+$'
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
fetch() {
curl -fsS --retry 3 --max-time 120 -o "$2" "$1"
grep -Eq "$prefix" "$2" || { echo "no prefixes in $1" >&2; exit 1; }
}
for cc in $countries; do
fetch "https://www.ipdeny.com/ipblocks/data/aggregated/$cc-aggregated.zone" "$tmp/$cc.v4"
fetch "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/$cc-aggregated.zone" "$tmp/$cc.v6"
done
elements() {
cat "$@" | grep -E "$prefix" | sed 's/$/,/'
}
{
echo "# Generated by $0, do not edit."
echo 'table inet host {'
echo ' set geo_v4 { type ipv4_addr; flags interval; auto-merge; elements = {'
elements "$tmp"/*.v4
echo ' } }'
echo ' set geo_v6 { type ipv6_addr; flags interval; auto-merge; elements = {'
elements "$tmp"/*.v6
echo ' } }'
echo '}'
} >"$tmp/host-geo.nft"
nft -c -f "$tmp/host-geo.nft"
install -m 0644 "$tmp/host-geo.nft" "$out.new"
mv -f "$out.new" "$out"
if systemctl -q is-active nft-host.service; then
systemctl reload nft-host.service
fi
+12
View File
@@ -0,0 +1,12 @@
---
journal_upload:
# Ingest route of VictoriaLogs (insert path only, NetBird sources, basic auth per host).
# The public DNS answers with the cluster's public address, where the route rejects
# every source outside NetBird; /etc/hosts pins the name to the NetBird addresses below.
host: ""
path: /insert/journald
# NetBird addresses of the Traefik nodes. The client tries them in this order when one is down.
addresses: []
# Basic auth credential; the user is the host's entry in the Secret vl-ingest-users.
user: "{{ inventory_hostname }}"
password: ""
+6
View File
@@ -0,0 +1,6 @@
---
- name: Restart systemd-journal-upload
ansible.builtin.systemd:
name: systemd-journal-upload
state: restarted
ignore_errors: "{{ ansible_check_mode }}"
+58
View File
@@ -0,0 +1,58 @@
---
# Ships the whole journal to VictoriaLogs. The uploader keeps the cursor of the last uploaded
# entry in its state directory, so a restart resumes where it stopped.
- name: Require the endpoint and the credential
ansible.builtin.assert:
that:
- journal_upload.host | length > 0
- journal_upload.addresses | length > 0
- journal_upload.user | length > 0
- journal_upload.password | length > 0
fail_msg: journal_upload.host, addresses, user and password must be set
- name: Install the journal uploader
ansible.builtin.apt:
name: systemd-journal-remote
install_recommends: false
state: present
# A node reaching its own address arrives at Traefik from its cilium_host address, which the
# NetBird allowlist rejects; it takes the other nodes.
- name: Pin the ingest name to the NetBird addresses
ansible.builtin.blockinfile:
path: /etc/hosts
marker: "# {mark} journal_upload"
block: |
{% for address in journal_upload.addresses | reject('in', ansible_facts.all_ipv4_addresses) %}
{{ address }} {{ journal_upload.host }}
{% endfor %}
notify: Restart systemd-journal-upload
- name: Create the uploader config directory
ansible.builtin.file:
path: /etc/systemd/journal-upload.conf.d
state: directory
mode: "0755"
# The service runs as a dynamic user, which reads the file after the manager dropped privileges.
- name: Configure the upload URL
ansible.builtin.copy:
dest: /etc/systemd/journal-upload.conf.d/10-victorialogs.conf
mode: "0644"
content: |
[Upload]
URL=https://{{ journal_upload.user }}:{{ journal_upload.password }}@{{ journal_upload.host }}{{ journal_upload.path }}
# No client certificate; the server certificate is checked against the system CAs.
ServerKeyFile=-
ServerCertificateFile=-
TrustedCertificateFile=/etc/ssl/certs/ca-certificates.crt
notify: Restart systemd-journal-upload
no_log: true
- name: Enable the journal uploader
ansible.builtin.systemd:
name: systemd-journal-upload
enabled: true
state: started
# Check mode does not install the package.
ignore_errors: "{{ ansible_check_mode }}"
+33
View File
@@ -0,0 +1,33 @@
---
kopia_backup:
# S3 repository; it is created, and its policies and maintenance are set, outside this role.
endpoint: tier1.backups.ruekov.eu
bucket: ""
access_key: ""
secret_key: ""
password: ""
# Snapshot owner is root@<hostname>; several hosts can share one repository.
hostname: "{{ inventory_hostname_short }}"
# Kopia defaults when unset.
content_cache_size_mb:
metadata_cache_size_mb:
# Paths to snapshot; missing ones are skipped at run time.
sources: []
# source path -> ignore rules, replacing the policy rules on every run (null drops one).
ignore: {}
# Command run first, e.g. a host script writing dumps into a source; a failure fails the run, the snapshot still runs.
pre_command: ""
# Database dumps written to /var/backups/dumps before the snapshot, which then includes that directory.
# Each item: file, container, command (run with sh -c inside the container, output is the dump).
dumps: []
# Files copied to the host as they are, each { src: local path, dest, mode }; owned by root.
files: []
# Snapshot the newest k3s etcd snapshot, staged in /var/backups/etcd.
etcd_snapshot: false
# Explicit UTC: the hosts run in different timezones.
on_calendar: "*-*-* 03:30 UTC"
randomized_delay: ""
# Extra units the service starts after, e.g. docker.service for dumps.
after: []
# Nice=10 and idle IO scheduling.
idle_priority: true
+104
View File
@@ -0,0 +1,104 @@
---
- name: Require the repository settings
ansible.builtin.assert:
that:
- kopia_backup.bucket | length > 0
- kopia_backup.access_key | length > 0
- kopia_backup.secret_key | length > 0
- kopia_backup.password | length > 0
- kopia_backup.sources | length > 0
fail_msg: kopia_backup.bucket, access_key, secret_key, password and sources must be set
- name: Create /etc/apt/keyrings
ansible.builtin.file:
path: /etc/apt/keyrings
state: directory
mode: "0755"
- name: Install gpg for the apt key
ansible.builtin.apt:
name: gpg
state: present
- name: Add the Kopia apt key
ansible.builtin.shell: curl -fsSL https://kopia.io/signing-key | gpg --dearmor -o /etc/apt/keyrings/kopia-keyring.gpg
args:
creates: /etc/apt/keyrings/kopia-keyring.gpg
- name: Add the Kopia apt repository
ansible.builtin.copy:
dest: /etc/apt/sources.list.d/kopia.list
mode: "0644"
content: |
deb [signed-by=/etc/apt/keyrings/kopia-keyring.gpg] http://packages.kopia.io/apt/ stable main
register: kopia_backup_repo
- name: Install Kopia
ansible.builtin.apt:
name: kopia
state: present
update_cache: "{{ kopia_backup_repo is changed }}"
- name: Read the repository connection
ansible.builtin.command: kopia repository status
environment:
KOPIA_CHECK_FOR_UPDATES: "false"
register: kopia_backup_status
changed_when: false
failed_when: false
check_mode: false
- name: Connect the repository
ansible.builtin.command:
argv: "{{ argv + (cache if kopia_backup.content_cache_size_mb else []) }}"
vars:
argv:
- kopia
- repository
- connect
- s3
- --endpoint={{ kopia_backup.endpoint }}
- --bucket={{ kopia_backup.bucket }}
- --override-hostname={{ kopia_backup.hostname }}
- --override-username=root
cache:
- --content-cache-size-mb={{ kopia_backup.content_cache_size_mb }}
- --metadata-cache-size-mb={{ kopia_backup.metadata_cache_size_mb }}
# Secrets through the environment, flags would show them in the process list
environment:
AWS_ACCESS_KEY_ID: "{{ kopia_backup.access_key }}"
AWS_SECRET_ACCESS_KEY: "{{ kopia_backup.secret_key }}"
KOPIA_PASSWORD: "{{ kopia_backup.password }}"
KOPIA_CHECK_FOR_UPDATES: "false"
when: kopia_backup_status.rc != 0
no_log: true
- name: Install the files of the host
ansible.builtin.copy:
src: "{{ item.src }}"
dest: "{{ item.dest }}"
mode: "{{ item.mode }}"
owner: root
group: root
loop: "{{ kopia_backup.files }}"
- name: Install the backup script
ansible.builtin.template:
src: kopia-backup.j2
dest: /usr/local/sbin/kopia-backup
mode: "0755"
- name: Install the systemd units
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
loop: [kopia-backup.service, kopia-backup.timer]
register: kopia_backup_units
- name: Enable the timer
ansible.builtin.systemd:
name: kopia-backup.timer
enabled: true
state: started
daemon_reload: "{{ kopia_backup_units is changed }}"
+58
View File
@@ -0,0 +1,58 @@
#!/bin/bash
# {{ ansible_managed }}
# Daily Kopia backup of {{ kopia_backup.hostname }} into the bucket {{ kopia_backup.bucket }} on {{ kopia_backup.endpoint }}.
set -uo pipefail
export KOPIA_CHECK_FOR_UPDATES=false
failed=0
sources=()
{% if kopia_backup.pre_command %}
{{ kopia_backup.pre_command }} || { echo "pre_command failed" >&2; failed=1; }
{% endif %}
{% if kopia_backup.etcd_snapshot %}
# Only the latest etcd snapshot; the live etcd dir is not consistent on disk and older snapshots only add churn
etcd=/var/backups/etcd
latest=$(ls -t /var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-* 2>/dev/null | head -1)
if [ -n "$latest" ]; then
install -d -m700 "$etcd"
rm -f "$etcd/etcd-snapshot"
ln "$latest" "$etcd/etcd-snapshot" 2>/dev/null || cp -p "$latest" "$etcd/etcd-snapshot" || failed=1
fi
{% endif %}
{% if kopia_backup.dumps %}
dumps=/var/backups/dumps
install -d -m700 "$dumps"
dump() { # file, container, command run inside the container
if docker exec "$2" sh -c "$3" > "$dumps/$1.tmp"; then
mv "$dumps/$1.tmp" "$dumps/$1"
else
echo "dump of $1 failed" >&2; rm -f "$dumps/$1.tmp"; failed=1
fi
}
{% for d in kopia_backup.dumps %}
dump {{ d.file | quote }} {{ d.container | quote }} {{ d.command | quote }}
{% endfor %}
{% endif %}
{% set ignores = kopia_backup.ignore | dict2items | rejectattr('value', 'none') | list %}
{% if ignores %}
# Ignore rules of a source, replaced as a whole (--clear-ignore runs after --add-ignore within one call).
# Kopia sorts the rules, so negations (!) cannot be relied on to come after the rule they override.
ignore() {
local path=$1; shift
kopia policy set "$path" --clear-ignore >/dev/null 2>&1 &&
kopia policy set "$path" "${@/#/--add-ignore=}" >/dev/null 2>&1 ||
{ echo "setting the ignore rules of $path failed" >&2; failed=1; }
}
{% for i in ignores %}
ignore {{ i.key | quote }} {{ i.value | map('quote') | join(' ') }}
{% endfor %}
{% endif %}
for p in {{ kopia_backup.sources | map('quote') | join(' ') }}{{ ' "$etcd"' if kopia_backup.etcd_snapshot }}{{ ' "$dumps"' if kopia_backup.dumps }}; do
[ -d "$p" ] && sources+=("$p")
done
kopia snapshot create "${sources[@]}" || failed=1
exit $failed
@@ -0,0 +1,14 @@
# {{ ansible_managed }}
[Unit]
Description=Kopia backup to {{ kopia_backup.endpoint }}
After={{ (['network-online.target'] + kopia_backup.after) | join(' ') }}
Wants=network-online.target
[Service]
Type=oneshot
Environment=HOME=/root
{% if kopia_backup.idle_priority %}
Nice=10
IOSchedulingClass=idle
{% endif %}
ExecStart=/usr/local/sbin/kopia-backup
@@ -0,0 +1,13 @@
# {{ ansible_managed }}
[Unit]
Description=Daily Kopia backup to {{ kopia_backup.endpoint }}
[Timer]
OnCalendar={{ kopia_backup.on_calendar }}
{% if kopia_backup.randomized_delay %}
RandomizedDelaySec={{ kopia_backup.randomized_delay }}
{% endif %}
Persistent=true
[Install]
WantedBy=timers.target
+27 -3
View File
@@ -7,7 +7,7 @@ kubernetes:
service_cidr: 10.43.0.0/16
ipv6:
# Minimum: /120
cluster_cidr: fd42::/56
cluster_cidr: fd42::/112
service_cidr: fd43::/112
# Interface to grab node-IPv4/v6 from
@@ -19,7 +19,11 @@ kubernetes:
token: <shared token for nodes to join>
network:
# One of [flannel, calico]
# Copies the current /etc/resolv.conf whilst only keeping "nameserver" lines
# can be used to remove filter
clean_resolv_conf: false
# One of [none, flannel, calico, cilium]
plugin: calico
# Helper for networking
@@ -29,9 +33,18 @@ kubernetes:
# Helpful, when nodeIp_interface is a wireguard-interface
routingtabletowg: false
# One of [traefik-ingress]
# One of [none, traefik-ingress, nginx-ingress]
ingress_controller: traefik-ingress
# k3s version to install, e.g. v1.37.1+k3s1. An installed k3s is replaced only
# when this is set and differs; empty installs the latest when k3s is missing.
version: ""
# Extra config of the nodes running etcd, left out with disable-etcd
etcd_config_extra:
# Without it etcd serves metrics on 127.0.0.1 only, out of reach of vmagent
etcd-expose-metrics: true
config_extra:
# etcd-tuning
# heartbeat: 0.5-1.5x of rtt
@@ -39,3 +52,14 @@ kubernetes:
etcd-arg:
heartbeat-interval: 500
election-timeout: 5000
registries_config:
# Defines the names and endpoints of registries
# When embedded-registry is enabled, every registry needs to be defined here to allow this node to distribute those images.
#mirrors:
# Enable distribution for all registries
#"*": {}
# Defines authentication and extra settings for the registries
#config:
+2 -2
View File
@@ -1,3 +1,3 @@
---
dependencies:
- role: docker
#dependencies:
# - role: docker
+36
View File
@@ -0,0 +1,36 @@
# kubelet gives kubepods.slice a cpu.weight from its allocatable CPUs (313 on
# 8 cores), while system.slice and k3s.service keep the default 100. Under load
# the pods then outweigh etcd, apiserver and kubelet. Weight 1000 lets the
# control plane win that contention.
- name: Set the CPUWeight of system.slice and the k3s unit
vars:
k3s_unit: "{{ 'k3s' if type == 'server' else 'k3s-agent' }}.service"
block:
- name: Create the drop-in directories
ansible.builtin.file:
path: "/etc/systemd/system/{{ item }}.d"
state: directory
mode: "0755"
loop:
- system.slice
- "{{ k3s_unit }}"
- name: Write the CPUWeight drop-ins
ansible.builtin.copy:
dest: "/etc/systemd/system/{{ item.unit }}.d/50-cpu-weight.conf"
mode: "0644"
content: |
[{{ item.section }}]
CPUWeight=1000
loop:
- unit: system.slice
section: Slice
- unit: "{{ k3s_unit }}"
section: Service
register: cpu_weight
# Only daemon-reload: the weight applies to the running k3s unit without a restart
- name: Reload systemd for the CPUWeight drop-ins
ansible.builtin.systemd_service:
daemon_reload: true
when: cpu_weight is changed
+42 -4
View File
@@ -2,26 +2,64 @@
ansible.builtin.file:
path: /etc/rancher/k3s/
state: directory
mode: '0755'
mode: "0755"
- name: Read the live k3s config
slurp:
src: /etc/rancher/k3s/config.yaml
register: live_config
failed_when: false
# A rewrite restarts k3s, so the file is written only when k3s would read other
# settings: the live files differ from the rendering in comments, order and
# whitespace.
- name: Deploy k3s config
ansible.builtin.template:
src: k3s/{{ type }}/config.yaml.jinja2
dest: /etc/rancher/k3s/config.yaml
register: config
when: >-
live_config.content is not defined or
(live_config.content | b64decode | from_yaml) != (lookup('template', 'k3s/' ~ type ~ '/config.yaml.jinja2') | from_yaml)
- name: Read the installed k3s version
command: k3s --version
register: k3s_installed
changed_when: false
failed_when: false
check_mode: false
- name: Download install-script
get_url:
url: https://get.k3s.io
dest: /root/k3s_install.sh
mode: '744'
mode: "744"
# todo: update when file changed?
- name: Read the live registries.yaml
slurp:
src: /etc/rancher/k3s/registries.yaml
register: live_registries
failed_when: false
when: "type == 'server'"
# Written only on a change k3s would see, like config.yaml
- name: Deploy registries.yaml
ansible.builtin.template:
src: k3s/server/registries.yaml.j2
dest: /etc/rancher/k3s/registries.yaml
register: registries_config
when:
- "type == 'server'"
- >-
live_registries.content is not defined or
(live_registries.content | b64decode | from_yaml) != (lookup('template', 'k3s/server/registries.yaml.j2') | from_yaml)
- import_tasks: ./install/server/setup_network.yml
when: "type == 'server'"
- import_tasks: ./install/server/install_helm.yml
when: "type == 'server'"
#- import_tasks: ./install/server/install_helm.yml
# when: "type == 'server'"
- import_tasks: ./install/server/install_k3s.yml
when: "type == 'server'"
@@ -1,5 +1,8 @@
- name: Install K3s agent
command: /root/k3s_install.sh {{ type }}
environment:
INSTALL_K3S_VERSION: "{{ kubernetes.version }}"
when: k3s_installed.rc != 0 or (kubernetes.version != '' and ('k3s version ' ~ kubernetes.version ~ ' ') not in k3s_installed.stdout)
register: command
changed_when: "'No change detected' not in command.stdout"
until: "command is not failed"
+29 -34
View File
@@ -1,55 +1,50 @@
- name: Install K3s-server for 1st-node
- block:
- name: Install K3s-server
command: /root/k3s_install.sh {{ type }}
when: "inventory_hostname == groups['kubernetes'][0]"
register: command
changed_when: "'No change detected' not in command.stdout"
- name: Make sure service is started / restarted on config change
service:
name: k3s
state: "{{ 'restarted' if not command.changed and config.changed else 'started' }}"
when: "inventory_hostname == groups['kubernetes'][0]"
- name: Waiting for K3s-server to accept connections
ansible.builtin.wait_for:
host: "{{ inventory_hostname }}"
port: 6443
state: started
when: "inventory_hostname == groups['kubernetes'][0]"
- name: Install K3s-server for other nodes
command: /root/k3s_install.sh {{ type }}
when: "inventory_hostname != groups['kubernetes'][0]"
environment:
INSTALL_K3S_CHANNEL: latest
INSTALL_K3S_VERSION: "{{ kubernetes.version }}"
when: k3s_installed.rc != 0 or (kubernetes.version != '' and ('k3s version ' ~ kubernetes.version ~ ' ') not in k3s_installed.stdout)
register: command
changed_when: "'No change detected' not in command.stdout"
until: "command is not failed"
retries: 2
delay: 10
- name: Make sure service is started / restarted on config change
- name: Make sure service is started / restarted on config change
service:
name: k3s
state: "{{ 'restarted' if not command.changed and config.changed else 'started' }}"
when: "inventory_hostname != groups['kubernetes'][0]"
enabled: true
state: "{{ 'restarted' if not command.changed and (config.changed or registries_config.changed) else 'started' }}"
- name: Waiting for K3s-server to accept connections on other nodes
- name: Waiting for K3s-server to accept connections
ansible.builtin.wait_for:
host: "{{ inventory_hostname }}"
port: 6443
state: started
when: "inventory_hostname != groups['kubernetes'][0]"
throttle: 1
#- name: Add Kubernetes environment-vars to /etc/profile.d/
# blockinfile:
# path: /etc/profile.d/k3s-bin.sh
# marker: "# {mark} ANSIBLE MANAGED BLOCK | k3s"
# block: |
# export KUBECONFIG="/etc/rancher/k3s/k3s.yaml"
# create: true
- name: Ensure the .kube directory exists
file:
path: "~/.kube"
state: directory
mode: "0755"
- name: Create symbolic link for k3s.yaml
file:
src: "/etc/rancher/k3s/k3s.yaml"
dest: "~/.kube/config"
state: link
- name: Deploy calico
import_tasks: ./roles/kubernetes/tasks/install/server/network-plugin/deploy_calico.yml
when: "kubernetes.network.plugin == 'calico'"
when:
- "kubernetes.network.plugin == 'calico'"
- name: Deploy cilium
import_tasks: ./roles/kubernetes/tasks/install/server/network-plugin/deploy_cilium.yml
when:
- "kubernetes.network.plugin == 'cilium'"
- name: Deploy network-helpers
import_tasks: ./roles/kubernetes/tasks/install/server/network-plugin/deploy_network_helper.yml
@@ -1,5 +1,5 @@
- name: Deploy calico operator
command: kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/tigera-operator.yaml
command: kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml
register: command
changed_when: "'created' in command.stdout"
run_once: true
@@ -0,0 +1,71 @@
- name: Check if already installed in cluster
command: "cilium status"
register: cilium_status
ignore_errors: true
changed_when: false
check_mode: false
run_once: true
- name: Check if the Cilium CLI is installed
stat:
path: /usr/local/bin/cilium
register: cilium_cli
- name: Get Cilium CLI version
ansible.builtin.command:
argv:
- curl
- -s
- https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt
register: cilium_cli_version
check_mode: no
changed_when: false
run_once: true
- name: Determine CLI architecture
set_fact:
cli_arch: "{{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}"
run_once: true
- name: Download and verify Cilium CLI
block:
- name: Download Cilium CLI tar.gz file
get_url:
url: "https://github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_version.stdout }}/cilium-linux-{{ cli_arch }}.tar.gz"
dest: "/tmp/cilium-linux-{{ cli_arch }}.tar.gz"
checksum: "sha256:{{ lookup('url', 'https://github.com/cilium/cilium-cli/releases/download/' + cilium_cli_version.stdout + '/cilium-linux-' + cli_arch + '.tar.gz.sha256sum', wantlist=True)[0].split(' ')[0] }}"
mode: '0755'
- name: Extract Cilium CLI tar.gz to /usr/local/bin
unarchive:
src: "/tmp/cilium-linux-{{ cli_arch }}.tar.gz"
dest: "/usr/local/bin"
remote_src: yes
#extra_opts: ["--strip-components=1"]
mode: 777
# An installed CLI is left alone, not upgraded to the latest release
when: cilium_cli_version.stdout is defined and not cilium_cli.stat.exists
become: yes
- name: Clean up temporary files
file:
path: "/tmp/cilium-linux-{{ cli_arch }}.tar.gz"
state: absent
# Install cilium via cli
- block:
- name: Install cilium
command: >
cilium install
--set
imap.mode=kubernetes,
ipv6.enabled=true,
#ipam.operator.clusterPoolIPv4PodCIDRList={{ kubernetes.ipPool.ipv4.cluster_cidr }},
#ipam.operator.clusterPoolIPv6PodCIDRList={{ kubernetes.ipPool.ipv6.cluster_cidr }}
- name: Wait for cilium
command: cilium status --wait
changed_when: false
run_once: true
when:
- cilium_status.rc != 0
+6 -1
View File
@@ -1,4 +1,9 @@
- import_tasks: ./prerequisites.yml
- import_tasks: ./install.yml
- import_tasks: ./node.yml
- import_tasks: ./cpu_weight.yml
tags: cpu_weight
- import_tasks: ./install.yml
when: target_host is not defined or inventory_hostname == target_host
+32
View File
@@ -0,0 +1,32 @@
- name: Kubelet image garbage collection
tags: kubelet_gc
block:
- name: Create the kubelet drop-in directory
file:
path: /etc/rancher/k3s/kubelet.conf.d
state: directory
mode: "0755"
- name: Set the image garbage collection thresholds
copy:
dest: /etc/rancher/k3s/kubelet.conf.d/10-image-gc.conf
mode: "0644"
content: |
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
imageGCHighThresholdPercent: 70
imageGCLowThresholdPercent: 55
imageMaximumGCAge: 168h
# Every pod watching its config files takes inotify instances and watches from
# the per-user pool. An LXC container inherits these from its host. Only this
# file is loaded: sysctl --system would also apply distribution defaults such as
# rp_filter that differ from the live values Cilium relies on.
- name: Raise the inotify limits
tags: inotify
when: ansible_facts['virtualization_type'] != 'lxc'
ansible.posix.sysctl:
name: "{{ item.key }}"
value: "{{ item.value }}"
sysctl_file: /etc/sysctl.d/90-inotify.conf
loop: "{{ {'fs.inotify.max_user_instances': 65536, 'fs.inotify.max_user_watches': 4194304} | dict2items }}"
+22 -3
View File
@@ -25,7 +25,7 @@
# For Longhorn:
- nfs-common
- open-iscsi
state: latest
state: present
- import_tasks: ./prerequisites/containerd.yml
@@ -35,8 +35,27 @@
- name: Getting nodeIp-data from interface
set_fact:
nodeip_ipv4: "{{ ansible_facts[ interface ].ipv4.address }}"
nodeip_ipv6: "{{ ansible_facts[ interface ].ipv6[0].address if ansible_facts[ interface ].ipv6 is defined }}"
nodeip_ipv4: >-
{{ (ansible_facts[interface].ipv4.address)
if (ansible_facts is defined and ansible_facts.get(interface) is defined and ansible_facts[interface].ipv4 is defined)
else (ansible_default_ipv4.address | default('')) }}
nodeip_ipv6: >-
{{ (ansible_facts[interface].ipv6[0].address)
if (ansible_facts is defined and ansible_facts.get(interface) is defined and ansible_facts[interface].ipv6 is defined and (ansible_facts[interface].ipv6 | length) > 0)
else (ansible_default_ipv6.address | default('')) }}
- block:
- name: Read resolv.conf
shell: "grep '^nameserver' /etc/resolv.conf"
register: nameserver_line
changed_when: false
check_mode: false
- name: Write /etc/resolv-k3s.conf with only nameservers
copy:
content: "{{ nameserver_line.stdout }}"
dest: /etc/resolv-k3s.conf
when: kubernetes.network.clean_resolv_conf is true
- name: Run handlers to reload configurations
meta: flush_handlers
@@ -1,12 +1,37 @@
{% set disabled_components = [] %}
## Base ##
node-name: {{ ansible_fqdn }}
# Join cluster
server: https://{{ hostvars[groups['kubernetes'][0]]['nodeip_ipv4'] }}:6443
token: '{{ kubernetes.token }}'
# Networking
{% if ansible_default_ipv6 is defined %}
node-external-ip: {{ ansible_default_ipv4.address }},{{ ansible_default_ipv6.address }}
{% else %}
node-external-ip: {{ ansible_default_ipv4.address }}
{% endif %}
{% if kubernetes.network.clean_resolv_conf %}
resolv-conf: /etc/resolv-k3s.conf
{% endif %}
{% if nodeip_ipv6 != "" and kubernetes.ipPool.ipv6 is defined %}
node-ip: {{ nodeip_ipv4 }},{{ nodeip_ipv6 }}
{% else %}
node-ip: {{ nodeip_ipv4 }}
{% endif %}
# Network-plugin
{% if kubernetes.network.plugin == "flannel" %}
{% else %}
{% if kubernetes.network.plugin == "cilium" %}
# Cilium requires servicelb to be disabled when replacing kube-proxy
{% set _ = disabled_components.append('servicelb') %}
{% endif %}
{% endif %}
## Label
# Region & DC
node-label:
@@ -16,3 +41,13 @@ node-label:
{% if zone is defined %}
- topology.kubernetes.io/zone={{ zone }}
{% endif %}
{% for item in kubernetes.labels_extra %}
- {{ item }}
{% endfor %}
# Disabled components
{% if disabled_components %}
disable: {{ disabled_components | join(',') }}
{% endif %}
{{ kubernetes.config_extra | to_yaml }}
@@ -1,8 +1,12 @@
{% set disabled_components = [] %}
## Base ##
node-name: {{ ansible_fqdn }}
{% if inventory_hostname == groups['kubernetes'][0] %}
# Initialize with internal etcd
cluster-init: true
{% else %}
# Join cluster
server: https://{{ hostvars[groups['kubernetes'][0]]['nodeip_ipv4'] }}:6443
{% endif %}
@@ -11,6 +15,16 @@ tls-san:
- {{ kubernetes.control_plane.dns_name }}
# Networking
{% if ansible_default_ipv6 is defined and ansible_default_ipv6.address is defined %}
node-external-ip: {{ ansible_default_ipv4.address }},{{ ansible_default_ipv6.address }}
{% else %}
node-external-ip: {{ ansible_default_ipv4.address }}
{% endif %}
{% if kubernetes.network.clean_resolv_conf %}
resolv-conf: /etc/resolv-k3s.conf
{% endif %}
{% if nodeip_ipv6 != "" and kubernetes.ipPool.ipv6 is defined %}
node-ip: {{ nodeip_ipv4 }},{{ nodeip_ipv6 }}
cluster-cidr: {{ kubernetes.ipPool.ipv4.cluster_cidr }},{{ kubernetes.ipPool.ipv6.cluster_cidr }}
@@ -21,6 +35,11 @@ cluster-cidr: {{ kubernetes.ipPool.ipv4.cluster_cidr }}
service-cidr: {{ kubernetes.ipPool.ipv4.service_cidr }}
{% endif %}
# Pin the apiserver advertise-address to the mesh IP: node-external-ip above
# otherwise gets picked, leaking a LAN-only address into the default/kubernetes
# Endpoints object that other nodes cannot route to.
advertise-address: {{ nodeip_ipv4 }}
egress-selector-mode: disabled
# Network-plugin
@@ -29,11 +48,15 @@ flannel-backend: vxlan
{% else %}
disable-network-policy: true
flannel-backend: none
{% if kubernetes.network.plugin == "cilium" %}
# Cilium requires servicelb to be disabled when replacing kube-proxy
{% set _ = disabled_components.append('servicelb') %}
{% endif %}
{% endif %}
# Ingress-plugin
{% if kubernetes.ingress_controller != "traefik-ingress" %}
disable: traefik
{% set _ = disabled_components.append('traefik') %}
{% endif %}
## Label
@@ -45,5 +68,24 @@ node-label:
{% if zone is defined %}
- topology.kubernetes.io/zone={{ zone }}
{% endif %}
{% if kubernetes.labels_extra is defined %}
{% for item in kubernetes.labels_extra %}
- {{ item }}
{% endfor %}
{% endif %}
# Disabled components
{% if disabled_components %}
disable: {{ disabled_components | join(',') }}
{% endif %}
{{ kubernetes.config_extra | to_yaml }}
# Image garbage collection tuning (drop-in KubeletConfiguration)
kubelet-arg:
- config-dir=/etc/rancher/k3s/kubelet.conf.d
{% if not kubernetes.config_extra['disable-etcd'] | default(false) %}
# etcd members only
{{ kubernetes.etcd_config_extra | to_nice_yaml(indent=2) }}
{% endif %}
@@ -0,0 +1 @@
{{ kubernetes.registries_config | to_nice_yaml(indent=2) }}
@@ -0,0 +1,11 @@
netbird_client:
# Key and url to join a network
# leave empty to ignore
join_network:
setup_key:
management_url:
# Interface name prefixes appended to IFaceBlackList in /var/lib/netbird/default.json
iface_blacklist_extra: []
# Overrides the --log-level of the netbird service when set
log_level:
+5
View File
@@ -0,0 +1,5 @@
- name: restart_netbird
systemd:
name: netbird
state: restarted
daemon_reload: true
+49
View File
@@ -0,0 +1,49 @@
- name: Read netbird config
slurp:
src: /var/lib/netbird/default.json
register: netbird_config_file
when: netbird_client.iface_blacklist_extra | default([]) | length > 0
tags: netbird_client_config
- name: Add the extra interface prefixes to IFaceBlackList
when: netbird_config_file.content is defined
vars:
netbird_config: "{{ netbird_config_file.content | b64decode | from_json }}"
netbird_blacklist_missing: "{{ netbird_client.iface_blacklist_extra | difference(netbird_config.IFaceBlackList | default([])) }}"
tags: netbird_client_config
block:
# netbird writes its in-memory config back on exit, so edit only while it is stopped
- name: Stop netbird to edit its config
systemd:
name: netbird
state: stopped
when: netbird_blacklist_missing | length > 0
notify: restart_netbird
register: netbird_stopped
- name: Re-read netbird config after stop
slurp:
src: /var/lib/netbird/default.json
register: netbird_config_stopped
when: netbird_stopped is changed
- name: Write IFaceBlackList
copy:
dest: /var/lib/netbird/default.json
content: "{{ cfg | combine({'IFaceBlackList': cfg.IFaceBlackList | default([]) + netbird_client.iface_blacklist_extra | difference(cfg.IFaceBlackList | default([]))}) | to_nice_json(indent=4, sort_keys=False) }}\n"
mode: "0600"
vars:
cfg: "{{ netbird_config_stopped.content | b64decode | from_json }}"
when: netbird_stopped is changed
- name: Set netbird log level
copy:
dest: /etc/systemd/system/netbird.service.d/20-log-level.conf
content: |
[Service]
ExecStart=
ExecStart=/usr/bin/netbird service run --log-level {{ netbird_client.log_level }} --daemon-addr unix:///var/run/netbird.sock --log-file /var/log/netbird/client.log
mode: "0644"
when: netbird_client.log_level | default('') != ''
notify: restart_netbird
tags: netbird_client_config
+28
View File
@@ -0,0 +1,28 @@
- name: Install Packages
package:
name:
- ca-certificates
- curl
- gnupg
# Same keyring path the hand-installed hosts use, so the repository line is identical.
- name: Install the netbird repository keyring
ansible.builtin.shell:
cmd: set -o pipefail && curl -fsSL https://pkgs.netbird.io/debian/public.key | gpg --dearmor -o /usr/share/keyrings/netbird-archive-keyring.gpg
executable: /bin/bash
creates: /usr/share/keyrings/netbird-archive-keyring.gpg
- name: Add netbird-repository
apt_repository:
repo: "deb [signed-by=/usr/share/keyrings/netbird-archive-keyring.gpg] https://pkgs.netbird.io/debian stable main"
state: present
filename: netbird
update_cache: yes
# present, not latest: an upgrade restarts the daemon and drops the tunnels of the node.
- name: Install wireguard & netbird
package:
name:
- wireguard
- netbird
state: present
+37
View File
@@ -0,0 +1,37 @@
- name: Read the netbird status
when: "netbird_client.join_network.setup_key | default('') | length > 0"
command: netbird status -d
register: netbird_status
changed_when: false
failed_when: false
check_mode: false
# Joined means connected to the management server of the vars; without a management_url any server counts.
- name: Join netbird-network
when:
- "netbird_client.join_network.setup_key | default('') | length > 0"
- netbird_status.stdout is not search(netbird_joined_pattern, multiline=True)
vars:
netbird_management_url: "{{ netbird_client.join_network.management_url | default('') }}"
netbird_joined_pattern: "^Management: Connected{{ (' to ' ~ (netbird_management_url | regex_escape) ~ '$') if netbird_management_url != '' else '' }}"
netbird_blacklist_extra: "{{ netbird_client.iface_blacklist_extra | default([]) }}"
netbird_up_argv: >-
{{ ['netbird', 'up']
+ (['--management-url', netbird_management_url] if netbird_management_url != '' else [])
+ ['--setup-key', netbird_client.join_network.setup_key]
+ (['--extra-iface-blacklist', netbird_blacklist_extra | join(',')] if netbird_blacklist_extra | length > 0 else []) }}
command:
argv: "{{ netbird_up_argv }}"
no_log: true
changed_when: true
register: join
- name: Wait for netbird-interface to exist
wait_for:
path: "/sys/class/net/wt0"
state: present
when: join is changed
- name: Gather facts to get changes
ansible.builtin.gather_facts:
when: join is changed
@@ -1,5 +1,5 @@
- import_tasks: ./certs.yml
- import_tasks: ./install.yml
- import_tasks: ./join-network.yml
- import_tasks: ./config.yml
-3
View File
@@ -1,3 +0,0 @@
---
dependencies:
- role: docker
+1
View File
@@ -0,0 +1 @@
DPkg::Post-Invoke { "/usr/local/sbin/pve-no-nag"; };
+7
View File
@@ -0,0 +1,7 @@
#!/bin/sh
# Disables the "No valid subscription" dialog of the Proxmox web UI. Called by the apt hook
# 86pve-no-nag, because every proxmox-widget-toolkit update restores the file. Without a match
# (changed upstream code) the dialog returns and nothing breaks.
f=/usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
[ -f "$f" ] && sed -i 's/checked_command: function (orig_cmd) {$/& orig_cmd(); return;/' "$f"
exit 0
+15
View File
@@ -0,0 +1,15 @@
- name: Install the subscription nag remover
copy:
src: pve-no-nag
dest: /usr/local/sbin/pve-no-nag
mode: "0755"
- name: Run it after every dpkg run
copy:
src: 86pve-no-nag
dest: /etc/apt/apt.conf.d/86pve-no-nag
mode: "0644"
- name: Remove the subscription nag
command: /usr/local/sbin/pve-no-nag
changed_when: false
+2
View File
@@ -0,0 +1,2 @@
tuned:
profile:
+6
View File
@@ -0,0 +1,6 @@
- name: Install Packages
package:
name:
- tuned
- tuned-utils
- tuned-utils-systemtap
+5
View File
@@ -0,0 +1,5 @@
- import_tasks: ./install-packages.yml
- import_tasks: ./setup-service.yml
- import_tasks: ./set-profile.yml
+5
View File
@@ -0,0 +1,5 @@
- name: Set profile
command: tuned-adm profile throughput-performance
register: command
failed_when:
- "command.rc != 0"
+5
View File
@@ -0,0 +1,5 @@
- name: Make sure service is started
service:
name: tuned
enabled: true
state: "started"
+16
View File
@@ -0,0 +1,16 @@
---
unattended_upgrades:
# Security updates only. Third-party repositories (Docker, Kopia, CrowdSec, NetBird, Proxmox, GitLab) stay out:
# an update of docker-ce or a pve package restarts every container or guest.
# Keyed by ansible_distribution; Debian lists go to Origins-Pattern, Ubuntu lists to Allowed-Origins.
origins_pattern:
Debian:
- origin=Debian,codename=${distro_codename}-security,label=Debian-Security
allowed_origins:
Ubuntu:
- ${distro_id}:${distro_codename}-security
- ${distro_id}ESMApps:${distro_codename}-apps-security
- ${distro_id}ESM:${distro_codename}-infra-security
# Recipient of the report mail; unset sends none, as no host has an outgoing mail setup.
mail:
mail_report: only-on-error
+33
View File
@@ -0,0 +1,33 @@
---
- name: Require an allowed origin for the distribution
ansible.builtin.assert:
that:
- (unattended_upgrades.origins_pattern[ansible_distribution] | default([]) + unattended_upgrades.allowed_origins[ansible_distribution] | default([])) | length > 0
fail_msg: unattended_upgrades has no origins for {{ ansible_distribution }}
- name: Install unattended-upgrades
ansible.builtin.apt:
name: unattended-upgrades
state: present
- name: Enable the daily package list update and upgrade
ansible.builtin.template:
src: 20auto-upgrades.j2
dest: /etc/apt/apt.conf.d/20auto-upgrades
mode: "0644"
# Separate file, the package owns 50unattended-upgrades.
- name: Configure unattended-upgrades
ansible.builtin.template:
src: 52unattended-upgrades-local.j2
dest: /etc/apt/apt.conf.d/52unattended-upgrades-local
mode: "0644"
- name: Enable the apt timers
ansible.builtin.systemd:
name: "{{ item }}"
enabled: true
state: started
loop:
- apt-daily.timer
- apt-daily-upgrade.timer
@@ -0,0 +1,4 @@
// {{ ansible_managed }}
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::AutocleanInterval "7";

Some files were not shown because too many files have changed in this diff Show More