David's Space

Tag: Home Lab

  • Backing it up: Ansible + Borg = Excellence

    With Ansible and Borg I have been able to automate my backup configuration into just a few host variables per service.

    I spent the better part of two weeks working on my Ansible role to set up a backup system in a way that works for me. I am happy to report that it has been a success. I will be posting the content of my files in here, I do have a git repo but I am not in a position to publish it as I want to avoid accidentally leaking something. So instead I will share the contents of all the files to help anybody else looking for inspiration or help. I am by no means a pro, but I get by with Ansible.

    Timeline

    • Started simple and tested at each step. Below is a list of things I did and tested before moving on.
      • Install Borg, create directories with correct permission, copy files, copy files with some echos to verify variables.
      • Only when all that was working did I create the templates and send them to hosts as backup scripts.
    • Added more quality of life features after I got the basics down.
    • Added status pings to my Uptime Kuma instance.
    • Modified pre and post hook commands to be a list and iterate instead of a single item so I can dump a database as well as stop/start containers.
    • Added logrotate into the mix to keep my logs from expanding forever.
    • Added ability to exclude files, this was for Trilium Notes and any other services that include built-in backups and/or log files that I don’t need to backup.
    • Tested and realized that I needed to make modifications to the ability to exclude files because I structured the Jinja If and For statements incorrectly.

    The Setup

    Here is a quick overview of the parts of my home lab that pertain to backups.

    The borg server is running on TrueNAS in a container. That took some work, I tried working with the Borg Warehouse community application but I could not get permissions working correctly. I am glad I did not get it working, it would have required much more manual configuration. Instead I am running Borg as a container. Generic Debian or Ubuntu image (whatever you want to use works too), and then I mounted a BorgBackup dataset into my user’s home directory. The reason I use a specific dataset is so I can also run a Cloud Task on it in TrueNAS and have that as the offsite backup.

    My servers are all connected via Tailscale and my Tailnet serves as the backbone for connecting and controlling everything. It also serves to transfer data to services that aren’t expose to the internet, such as a Borg backup server. So my VPSes run backups over the Tailnet. Same with other servers I manage for friends, at least until they get their own NAS.

    My cloud tasks from TrueNAS currently backup to OneDrive and Google Drive. Only reason is that I have the family plan on both and I am not paying for either, so I may as well take advantage while I can. Eventually I would like to move it to a storage VPS. Anyways, I am not terribly concerned since all the Borg backups are encrypted. Yes I know that these companies are still not to be trusted and can hold onto data for an era of quantum computing to break all the encryption, but for now this is sufficient for me.

    The Role

    Role Defaults

    This establishes default options for the role in case a variable is not specified, and it is not crucial to be defined per-host. This way we do not run into a failure over a benign variable being.

    ---
# file: roles/borgbackup/defaults/main.yaml

borgbackup_backup_server: false
borgbackup_init_repo: true
borgbackup_cron:
  hour: 4
  minute: 25

    Tasks

    Below is the main task list file that everything else is orchestrated from.

    Task files below will be in alphabetical order following main.yaml. Explanations will be provided for each file where necessary.

    ---
# file: roles/borgbackup/tasks/main.yaml

- name: Install borgbackup packages
  ansible.builtin.apt:
    name: "{{ borgbackup_install_packages }}"
    state: present
  become: true

- name: Setup clients and all their backup targets
  ansible.builtin.include_tasks: client_setup.yaml
  when:
    - borgbackup_backup_server is false
    - borgbackup_configs is defined
  loop: "{{ borgbackup_configs | dict2items }}"

- name: Init the borg repo
  ansible.builtin.include_tasks: init_repo.yaml
  when:
    - borgbackup_backup_server is false
    - borgbackup_configs is defined
  loop: "{{ borgbackup_configs | dict2items }}"

- name: Configure backups on hosts
  ansible.builtin.include_tasks: roles/borgbackup/tasks/configure_backups.yaml
  when:
    - borgbackup_backup_server is false
    - borgbackup_configs is defined

    Templates

    Variables

    ---
# file: roles/borgbackup/vars/main.yaml

borgbackup_install_packages:
  - borgbackup
  - logrotate

    I also have a variable you’ve probably seen my_user. It is a host variable that represents the user that I connect to my servers with and run my services from. My Ansible setup role creates a separate ansible user to run Ansible commands from. Make sure to either configure a my_user variable, or set it to the value of your user.

    Host Variables

    This is how I configure each backup per host, in their own host file. Here is an example of a few different backups.

    borg_repokey: REPO_PASSPHRASE

borgbackup_configs:
    first-backup:
        src_path:
            - /path/to/.borg/tmp/first-backup-db.sql
            - /path/to/first/backup
        pre_hook:
            - >-
                docker exec mariadb mariadb-dump --password= --databases 
                > /path/to/.borg/tmp/first-backup-db.sql
        status_url: https://kuma.domain.com
        borg_repo_server: borgbackup.server.hostname
    second-backup:
        src_path:
            - /path/to/second/backup
            - /path/to/.borg/tmp/second-backup-db.sql
        pre_hook:
            - >-
                docker exec mariadb mariadb-dump --password= --databases 
                > /path/to/.borg/tmp/second-backup-db.sql
        status_url: https://kuma.domain.com
        borg_repo_server: borgbackup.server.hostname
    third-backup:
        src_path:
            - /path/to/third/backup
        exclude_patterns:
            - /path/to/third/backup/logs
        pre_hook:
            - docker stop container
        post_hook:
            - docker start container
        status_url: https://kuma.domain.com
        borg_repo_server: borgbackup.server.hostname

    What did I learn?

    So very much. I got a better grasp of whitespace adjustments that YAML and Jinja can make, the various modifiers for that such as - and >-. I also learned a lot about shell scripting. More about lists and YAML handling in general too. I wish I could list it better but pretty much this whole process was learning.

    What’s Next?

    I am currently happy with where the role is now, I have added a few services already and they have been happily backing up without any need for interference. Everything is logged so I can monitor it.

    I think what I have built should cover 99% of my use-cases, including creating a repo/adding an SSH key for a service such as Nextcloud AIO.

    There will inevitably be some tweaking as I use the role and find edge-cases that need a little extra added. I look forward to that, it is fun, like trying to add new pieces to a puzzle.

    Sources & Inspiration

    One last thing

    I will be writing about this in the future, but I am venturing back into pyinfra now that I have had some time with Ansible. If I do move forward with pyinfra again I will be writing up a post with details why and what I like more and all that fun stuff.

    I will of course be porting my borg role over into a deploy, and I will also create a similar write-up for that too. Stay tuned!