Enable gitea dump on tent with custom rotating backups #234

abonerib · 2026-03-05T11:44:42+01:00

abonerib commented

2026-03-05 11:44:42 +01:00

closes #233

I opted for copying the systemd units with different names instead of overriding the ones from the NixOS module in case upstream changes in NixOS silently break our overrides (e.g. they rename the systemd-unit)

I tested it in weasel, and it seems to work (although I did not have any repos or users):

$ sudo su gitea -c "nix run jungle#unzip -- -l /vault/gitea/gitea-dump-Thu.zip"
Archive:  /vault/gitea/gitea-dump-Thu.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
    65732  03-05-2026 11:29   gitea-db.sql
     1170  03-05-2026 11:07   app.ini
        0  03-05-2026 11:07   custom/conf/
       43  03-05-2026 11:05   custom/conf/oauth2_jwt_secret
     1170  03-05-2026 11:07   custom/conf/app.ini
       64  03-05-2026 11:05   custom/conf/secret_key
      105  03-05-2026 11:05   custom/conf/internal_token
        0  03-05-2026 11:05   data/queues/
        0  03-05-2026 11:07   data/queues/common/
       16  03-05-2026 11:07   data/queues/common/CURRENT
      601  03-05-2026 11:07   data/queues/common/000002.ldb
      191  03-05-2026 11:07   data/queues/common/MANIFEST-000004
       16  03-05-2026 11:07   data/queues/common/CURRENT.bak
      946  03-05-2026 11:07   data/queues/common/LOG
        0  03-05-2026 11:07   data/queues/common/000003.log
        0  03-05-2026 11:05   data/queues/common/LOCK
  2113536  03-05-2026 11:05   data/gitea.db
        0  03-05-2026 11:05   data/jwt/
     3268  03-05-2026 11:05   data/jwt/private.pem
        0  03-05-2026 11:05   data/actions_log/
        0  03-05-2026 11:05   data/home/
      379  03-05-2026 11:05   data/home/.gitconfig
        0  03-05-2026 11:05   data/repo-avatars/
        0  03-05-2026 11:05   data/indexers/
        0  03-05-2026 11:05   data/indexers/issues.bleve/
       42  03-05-2026 11:05   data/indexers/issues.bleve/index_meta.json
        0  03-05-2026 11:05   data/indexers/issues.bleve/store/
    65536  03-05-2026 11:07   data/indexers/issues.bleve/store/root.bolt
       13  03-05-2026 11:05   data/indexers/issues.bleve/rupture_meta.json
        0  03-05-2026 11:05   data/actions_artifacts/
        0  03-05-2026 11:05   data/avatars/
---------                     -------
  2252828                     31 files

Caveats:

dumping to stdout and redirecting to the file solves the error of gitea dump not allowing overrides, but if the command fails, we risk overriding a valid backup with garbage
The restore process is manual and they don't seem to address how incompatibilities in the DB across gitea versions are handled?

closes #233 I opted for copying the systemd units with different names instead of overriding the ones from the NixOS module in case upstream changes in NixOS silently break our overrides (e.g. they rename the systemd-unit) I tested it in weasel, and it seems to work (although I did not have any repos or users): ``` $ sudo su gitea -c "nix run jungle#unzip -- -l /vault/gitea/gitea-dump-Thu.zip" Archive: /vault/gitea/gitea-dump-Thu.zip Length Date Time Name --------- ---------- ----- ---- 65732 03-05-2026 11:29 gitea-db.sql 1170 03-05-2026 11:07 app.ini 0 03-05-2026 11:07 custom/conf/ 43 03-05-2026 11:05 custom/conf/oauth2_jwt_secret 1170 03-05-2026 11:07 custom/conf/app.ini 64 03-05-2026 11:05 custom/conf/secret_key 105 03-05-2026 11:05 custom/conf/internal_token 0 03-05-2026 11:05 data/queues/ 0 03-05-2026 11:07 data/queues/common/ 16 03-05-2026 11:07 data/queues/common/CURRENT 601 03-05-2026 11:07 data/queues/common/000002.ldb 191 03-05-2026 11:07 data/queues/common/MANIFEST-000004 16 03-05-2026 11:07 data/queues/common/CURRENT.bak 946 03-05-2026 11:07 data/queues/common/LOG 0 03-05-2026 11:07 data/queues/common/000003.log 0 03-05-2026 11:05 data/queues/common/LOCK 2113536 03-05-2026 11:05 data/gitea.db 0 03-05-2026 11:05 data/jwt/ 3268 03-05-2026 11:05 data/jwt/private.pem 0 03-05-2026 11:05 data/actions_log/ 0 03-05-2026 11:05 data/home/ 379 03-05-2026 11:05 data/home/.gitconfig 0 03-05-2026 11:05 data/repo-avatars/ 0 03-05-2026 11:05 data/indexers/ 0 03-05-2026 11:05 data/indexers/issues.bleve/ 42 03-05-2026 11:05 data/indexers/issues.bleve/index_meta.json 0 03-05-2026 11:05 data/indexers/issues.bleve/store/ 65536 03-05-2026 11:07 data/indexers/issues.bleve/store/root.bolt 13 03-05-2026 11:05 data/indexers/issues.bleve/rupture_meta.json 0 03-05-2026 11:05 data/actions_artifacts/ 0 03-05-2026 11:05 data/avatars/ --------- ------- 2252828 31 files ``` ### Caveats: - dumping to stdout and redirecting to the file solves the error of gitea dump not allowing overrides, but if the command fails, we risk overriding a valid backup with garbage - The restore process is [manual][1] and they don't seem to address how incompatibilities in the DB across gitea versions are handled? [1]: https://docs.gitea.com/next/administration/backup-and-restore#restore-command-restore

abonerib added 2 commits 2026-03-05 11:44:42 +01:00

Enable rotating gitea backups 3a73eae98d

Override files in rotating gitea dump service

CI / build:all (pull_request) Successful in 56m29s

Details

CI / build:cross (pull_request) Successful in 1h5m24s

Details

11684d2dc9

rarias commented

2026-03-05 12:06:56 +01:00

Thanks for taking a look into this! I also think is a good idea to use our custom service.

dumping to stdout and redirecting to the file solves the error of gitea dump not allowing overrides, but if the command fails, we risk overriding a valid backup with garbage

We can use an atomic mv command like this (notice date only runs once):

set -e
name="gitea-dump-$(date +%a).${cfg.dump.type}"
${exe} dump --type ${cfg.dump.type} --file - > "$name.tmp"
mv "$name.tmp" "$name"

If gitea dump fails, you only end up with garbage in the .tmp file, but the mv doesn't run due to the set -e.

The restore process is manual and they don't seem to address how incompatibilities in the DB across gitea versions are handled?

Gitea performs the upgrade automatically of the DB from previous versions on start. We cannot restore a previous backup into a new Gitea version, but I think this will be okay for our cases.

I would need to test the restore process in a real scenario. If you already have a working Gitea instance on weasel perhaps that would be the easiest place to test:

Create a test repository and write a couple of test issues with some test images or attached files.
Make a backup.
Delete the repo from Gitea.
Restore the backup and check that the repo is still there as well as the issues with the attached images.

Thanks for taking a look into this! I also think is a good idea to use our custom service. > dumping to stdout and redirecting to the file solves the error of gitea dump not allowing overrides, but if the command fails, we risk overriding a valid backup with garbage We can use an atomic `mv` command like this (notice date only runs once): ```sh set -e name="gitea-dump-$(date +%a).${cfg.dump.type}" ${exe} dump --type ${cfg.dump.type} --file - > "$name.tmp" mv "$name.tmp" "$name" ``` If gitea dump fails, you only end up with garbage in the .tmp file, but the mv doesn't run due to the `set -e`. > The restore process is [manual](https://docs.gitea.com/next/administration/backup-and-restore#restore-command-restore) and they don't seem to address how incompatibilities in the DB across gitea versions are handled? Gitea performs the upgrade automatically of the DB from previous versions on start. We cannot restore a previous backup into a new Gitea version, but I think this will be okay for our cases. I would need to test the restore process in a real scenario. If you already have a working Gitea instance on weasel perhaps that would be the easiest place to test: 1. Create a test repository and write a couple of test issues with some test images or attached files. 2. Make a backup. 3. Delete the repo from Gitea. 4. Restore the backup and check that the repo is still there as well as the issues with the attached images.

abonerib added 1 commit 2026-03-05 12:18:23 +01:00

More robust gitea dump override

CI / build:all (pull_request) Successful in 22m52s

Details

CI / build:cross (pull_request) Successful in 31m45s

Details

05403dbd9b

abonerib force-pushed gitea-dump from 05403dbd9b to bdf7e16c0d

2026-03-05 13:49:11 +01:00

Compare

abonerib commented

2026-03-05 13:58:48 +01:00

I would need to test the restore process in a real scenario. If you already have a working Gitea instance on weasel perhaps that would be the easiest place to test:

Create a test repository and write a couple of test issues with some test images or attached files.

Make a backup.

Delete the repo from Gitea.

Restore the backup and check that the repo is still there as well as the issues with the attached images.

Ran these checks in weasel and all seems to work. Feel free to poke around in weasel:3000. Here is the nuke script:

systemctl start gitea-dump-rotating
systemctl stop gitea
rm -rf /var/lib/gitea
systemd-tmpfiles --create # recereate /var/lib/gitea
systemctl start gitea # gitea should be empty

Maybe we could try to restore the hut / tent Gitea instance into weasel (here is the config: weasel-gitea-test)

(My Gitea does not have CI logs / artifacts, so I did not test anything regarding that)

The gitea doctor check command has found no issues (also passed with --all):

$ /nix/store/rv8p6j7k1mqhvbqjqv5wpijbbgpm1hmj-gitea-1.25.2/bin/gitea doctor check  --config /var/lib/gitea/custom/conf/app.ini

[1] Check paths and basic configuration
 - [I] Configuration File Path:    "/var/lib/gitea/custom/conf/app.ini"
 - [I] Repository Root Path:       "/var/lib/gitea/repositories"
 - [I] Data Root Path:             "/var/lib/gitea/data"
 - [I] Custom File Root Path:      "/var/lib/gitea/custom"
 - [I] Work directory:             "/var/lib/gitea"
 - [I] Log Root Path:              "/var/lib/gitea/log"
 - [I] Static File Root Path:      "/nix/store/9klv03ypx81fg2ff8fc9n63xzfs1z367-gitea-1.25.2-data"
OK

[2] Check Database Version
 - [I] Expected database version: 323
OK

[3] Check if user with wrong type exist
OK

[4] Check if OpenSSH authorized_keys file is up-to-date
OK

[5] Synchronize repo HEADs
 - [I] All 2 repos have their HEADs in the correct state
OK

All done (checks: 5).

Quirks of the backup

In the nix module, the sql database is saved in /var/lib/gitea/data/gitea.db which is then backed up inside the zip file. This is the whole db, instead of the sql commands that gitea saves in the root of the zip at gitea-db.sql. By copying data we already have the database for free, so we can skip the import of the db from the restore process.

$ ls -lh /var/lib/gitea/data/gitea.db gitea-db.sql
-rw------- 1 gitea gitea 128K Mar  5 12:44 gitea-db.sql
-rw-r----- 1 gitea gitea 2,1M Mar  5 13:16 /var/lib/gitea/data/gitea.db

$ file /var/lib/gitea/data/gitea.db gitea-db.sql
/var/lib/gitea/data/gitea.db: SQLite 3.x database, last written using SQLite version 3050004, file counter 605, database pages 521, cookie 0x1e2, schema 4, UTF-8, version-valid-for 605
gitea-db.sql:                 Unicode text, UTF-8 text, with very long lines (9266)

I think keeping it this way is fine, since our production db is probably not that big, but I might be wrong.

The same happens with app.ini which is duplicated in the custom folder.

> I would need to test the restore process in a real scenario. If you already have a working Gitea instance on weasel perhaps that would be the easiest place to test: > > 1. Create a test repository and write a couple of test issues with some test images or attached files. > 2. Make a backup. > 3. Delete the repo from Gitea. > 4. Restore the backup and check that the repo is still there as well as the issues with the attached images. Ran these checks in weasel and all seems to work. Feel free to poke around in `weasel:3000`. Here is the nuke script: ```bash systemctl start gitea-dump-rotating systemctl stop gitea rm -rf /var/lib/gitea systemd-tmpfiles --create # recereate /var/lib/gitea systemctl start gitea # gitea should be empty ``` Maybe we could try to restore the `hut` / `tent` Gitea instance into `weasel` (here is the config: [weasel-gitea-test][1]) (My Gitea does not have CI logs / artifacts, so I did not test anything regarding that) The `gitea doctor check` command has found no issues (also passed with `--all`): ``` $ /nix/store/rv8p6j7k1mqhvbqjqv5wpijbbgpm1hmj-gitea-1.25.2/bin/gitea doctor check --config /var/lib/gitea/custom/conf/app.ini [1] Check paths and basic configuration - [I] Configuration File Path: "/var/lib/gitea/custom/conf/app.ini" - [I] Repository Root Path: "/var/lib/gitea/repositories" - [I] Data Root Path: "/var/lib/gitea/data" - [I] Custom File Root Path: "/var/lib/gitea/custom" - [I] Work directory: "/var/lib/gitea" - [I] Log Root Path: "/var/lib/gitea/log" - [I] Static File Root Path: "/nix/store/9klv03ypx81fg2ff8fc9n63xzfs1z367-gitea-1.25.2-data" OK [2] Check Database Version - [I] Expected database version: 323 OK [3] Check if user with wrong type exist OK [4] Check if OpenSSH authorized_keys file is up-to-date OK [5] Synchronize repo HEADs - [I] All 2 repos have their HEADs in the correct state OK All done (checks: 5). ``` ### Quirks of the backup In the nix module, the sql database is saved in `/var/lib/gitea/data/gitea.db` which is then backed up inside the zip file. This is the whole db, instead of the sql commands that gitea saves in the root of the zip at `gitea-db.sql`. By copying `data` we already have the database for free, so we can skip the import of the db from the restore process. ``` $ ls -lh /var/lib/gitea/data/gitea.db gitea-db.sql -rw------- 1 gitea gitea 128K Mar 5 12:44 gitea-db.sql -rw-r----- 1 gitea gitea 2,1M Mar 5 13:16 /var/lib/gitea/data/gitea.db $ file /var/lib/gitea/data/gitea.db gitea-db.sql /var/lib/gitea/data/gitea.db: SQLite 3.x database, last written using SQLite version 3050004, file counter 605, database pages 521, cookie 0x1e2, schema 4, UTF-8, version-valid-for 605 gitea-db.sql: Unicode text, UTF-8 text, with very long lines (9266) ``` I think keeping it this way is fine, since our production db is probably not that big, but I might be wrong. The same happens with `app.ini` which is duplicated in the `custom` folder. [1]: https://jungle.bsc.es/git/abonerib/jungle/src/branch/weasel-gitea-test

abonerib force-pushed gitea-dump from bdf7e16c0d to 83579c729a

2026-03-05 15:39:25 +01:00

Compare

rarias force-pushed gitea-dump from 83579c729a to ef0268b87a

2026-03-06 12:50:37 +01:00

Compare

rarias commented

2026-03-06 12:58:38 +01:00

Thanks for taking a look. I renamed the service to just "gitea-backup" and placed it inside a subdirectory so it follows the already existing backups.

I ran this on tent with the current Gitea data and added an extra step to also store the backup in Ceph, so we have:

3 redundant copies in the 3 local SSD disks in SW raid in tent
3 redundant copies in Ceph spread across the disks in the two storage nodes (three disks must fail for data loss).

I think this would be enough. Here is how it looks like after two backups:

tent% sudo ls -l /{vault,ceph}/backup/gitea/gitea-dump-{Thu,Fri}.zip
-rw-r--r-- 1 gitea gitea 808255271 Mar  6 04:31 /ceph/backup/gitea/gitea-dump-Fri.zip
-rw-r--r-- 1 gitea gitea 807948772 Mar  5 20:05 /ceph/backup/gitea/gitea-dump-Thu.zip
-rw-r--r-- 1 gitea gitea 808255271 Mar  6 04:31 /vault/backup/gitea/gitea-dump-Fri.zip
-rw-r--r-- 1 gitea gitea 807948772 Mar  5 16:43 /vault/backup/gitea/gitea-dump-Thu.zip
tent% sudo md5sum /{vault,ceph}/backup/gitea/gitea-dump-{Thu,Fri}.zip
951468b1c0d2a255a2203966a05e938e  /vault/backup/gitea/gitea-dump-Thu.zip
11329a3a9d48ef156baa3e30569ae9e5  /vault/backup/gitea/gitea-dump-Fri.zip
951468b1c0d2a255a2203966a05e938e  /ceph/backup/gitea/gitea-dump-Thu.zip
11329a3a9d48ef156baa3e30569ae9e5  /ceph/backup/gitea/gitea-dump-Fri.zip

I think we have enough space to keep the 7 backups without problems.

Regarding the DB, I would leave it as-is for now.

Thanks for taking a look. I renamed the service to just "gitea-backup" and placed it inside a subdirectory so it follows the already existing backups. I ran this on tent with the current Gitea data and added an extra step to also store the backup in Ceph, so we have: - 3 redundant copies in the 3 local SSD disks in SW raid in tent - 3 redundant copies in Ceph spread across the disks in the two storage nodes (three disks must fail for data loss). I think this would be enough. Here is how it looks like after two backups: ``` tent% sudo ls -l /{vault,ceph}/backup/gitea/gitea-dump-{Thu,Fri}.zip -rw-r--r-- 1 gitea gitea 808255271 Mar 6 04:31 /ceph/backup/gitea/gitea-dump-Fri.zip -rw-r--r-- 1 gitea gitea 807948772 Mar 5 20:05 /ceph/backup/gitea/gitea-dump-Thu.zip -rw-r--r-- 1 gitea gitea 808255271 Mar 6 04:31 /vault/backup/gitea/gitea-dump-Fri.zip -rw-r--r-- 1 gitea gitea 807948772 Mar 5 16:43 /vault/backup/gitea/gitea-dump-Thu.zip tent% sudo md5sum /{vault,ceph}/backup/gitea/gitea-dump-{Thu,Fri}.zip 951468b1c0d2a255a2203966a05e938e /vault/backup/gitea/gitea-dump-Thu.zip 11329a3a9d48ef156baa3e30569ae9e5 /vault/backup/gitea/gitea-dump-Fri.zip 951468b1c0d2a255a2203966a05e938e /ceph/backup/gitea/gitea-dump-Thu.zip 11329a3a9d48ef156baa3e30569ae9e5 /ceph/backup/gitea/gitea-dump-Fri.zip ``` I think we have enough space to keep the 7 backups without problems. Regarding the DB, I would leave it as-is for now.

👍 1

rarias changed title from ~~WIP: Enable gitea dump on tent with custom rotating backups~~ to Enable gitea dump on tent with custom rotating backups

2026-03-06 12:59:05 +01:00

rarias force-pushed gitea-dump from ef0268b87a to 8728095799

2026-03-09 08:54:02 +01:00

Compare

rarias approved these changes 2026-03-09 09:02:42 +01:00

abonerib reviewed 2026-03-09 10:48:14 +01:00

m/tent/gitea.nix Outdated

						
				@@ -29,0 +74,4 @@

				    description = "Update timer for gitea-backup";

				    partOf = [ "gitea-backup.service" ];

				    wantedBy = [ "timers.target" ];

				    timerConfig.OnCalendar = config.services.gitea.dump.interval;