Ad meliora

Mar 16, 2024

Upgrading my Gitea/Woodpecker Setup

OUTLINE

REASONS

So I am using gitea-1.19.4 as well as Woodpecker-0.15.x. The current version(s) of gitea and woodpecker are 1.21.x and 2.3.x respectively.

I wanted to upgrade to these versions because of:

(a) Gitea 1.21.x comes with Gitea Actions enabled by default. THis is similar to Github Actions which I want to learn as we use that at $DAYJOB.

(b) I want to upgrade to a new version every year. This is to make upgrades manageable and repeatable so by doing it more frequently, I would have a runbook for how to upgrade.

(c) These versions (1.19.x, 0.x.x) are no longer actively maintained, which could mean that they are stable but not receiving security fixes which is something I want to avoid.

(d) Woodpecker 2.3.x adds the ability to use Forego which is a community fork of Gitea and runs Codeberg. I may switch to Forego in the future as I may want to hack on it and contribute back to it.

PLAN OF ACTION

Just persuing the Gitea changelong and Woodpecker changelog gives me the sense that a lot has changed between versions. This might mean that the upgrade might be seamless or not. So my plan of action is this:

(a) Copy over my data from the two instances to my local computer

(b) Upgrade Gitea. (Ideally, I would have loved to resize the instances but I am too lazy to do so)

(c) Upgrade Woodpecker (Ideally, I would have loved to resize the instances but I am too lazy to do so)

(d) Test, If botched, then recreate new instances (what a pain in the ass, that would be)

GITEA

For the Gitea instance, I would mainly be upgrading to 1.21.x and with the Woodpecker instance, I would be upgrading to 2.3.x. For Gitea,you would:

  • Backup database
  • Backup Gitea config
  • Backup Gitea data files in APP_DATA_PATH
  • Backup Gitea external storage (eg: S3/MinIO or other storages if used)

The steps I have used are taken from here and here.

(a) Define some environment variables

$ export GITEABIN="/usr/local/bin/gitea"
$ export GITEACONF="/etc/gitea/app.ini"
$ export GITEAUSER="git"
$ export GITEAWORKPATH="/var/lib/gitea"

(a) Get current version

$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" --version | cut -d ' ' -f 3

(b) Get gitea version to install

$ export GITEAVERSION=$(curl --connect-timeout 10 -sL https://dl.gitea.com/gitea/version.json | jq -r .latest.version)

(c) Download version from (b) and uncompress

$ binname="gitea-${GITEAVERSION}-linux-amd64"
$ binurl="https://dl.gitea.com/gitea/${giteaversion}/${binname}.xz"
$ curl --connect-timeout 10 --silent --show-error --fail --location -O "$binurl{,.sha256,.asc}"
$ sha256sum -c "${binname}.xz.sha256"
$ rm "${binname}".xz.{sha256,asc}
$ xz --decompress --force "${binname}.xz"

(d) Flush Gitea queues so that you can properly backup

$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" manager flush-queues

(e) Stop gitea

$ sudo systemctl stop gitea

(f) Dump database using pg_dump

$ pg_dump -U $USER $DATABASE > gitea-db.sql

(g) Dump the gitea config and files

$ mkdir temp
$ chmod 777 temp # necessary otherwise the dump fails
$ cd temp
$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" dump --verbose

(h) Move the downloaded gitea binary to the new location

$ sudo cp -f "${GITEABIN}" "${GITEABIN}.bak"
$ sudo mv -f "${binname}" "${GITEABIN}"

(i) Restart Gitea

$ sudo chmod +x "${GITEABIN}"
$ sudo systemctl start gitea
$ sudo systemctl status gitea

To restore as taken from here

$ sudo systemctl stop gitea
$ sudo cp -f "${GITEABIN}.bak" "${GITEABIN}"
$ cd temp
$ unzip gitea-dump-*.zip
$ cd gitea-dump-*
$ sudo mv app.ini "${GITEACONF}"
$ sudo mv data/* "${GITEAWORKPATH}/data/"
$ sudo mv log/* "${GITEAWORKPATH}/log/"
$ sudo mv repos/* "${GITEAWORKPATH}/gitea-repositories/"
$ sudo chown -R gitea:gitea /etc/gitea/conf/app.ini "${GITEAWORKPATH}"
$ cp ../gitea-db.sql .
$ psql -U $USER -d $DATABASE < gitea-db.sql
$ sudo systemctl start gitea
$ sudo systemctl stop gitea
# Regenerate repo hooks
$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" admin regenerate hooks

WOODPECKER

(a) Stop agent and server

$ sudo systemctl stop woodpecker
$ sudo systemctl stop woodpecker-agent

(b) Backup database with pg_dump

$ pg_dump -U woodpecker woodpeckerdb > woodpecker-db.sql
$ pg_dump -U $USER $DATABASE > woodpecker-db.sql
$ sudo cp /etc/woodpecker-agent.conf woodpecker-agent.conf.bak
$ sudo cp /etc/woodpecker.conf woodpecker.conf.bak

(c) Download the binaries

$ woodpeckerversion=2.3.0
$ binurl="https://github.com/woodpecker-ci/woodpecker/releases/download/v${woodpeckerversion}/"
$ curl --connect-timeout 10 --silent --show-error --fail --location -O "${binurl}woodpecker{,-agent,-cli,-server}_${woodpeckerversion}_amd64.deb"
$ ls
woodpecker-cli_2.3.0_amd64.deb 
woodpecker-agent_2.3.0_amd64.deb
woodpecker-server_2.3.0_amd64.deb

(d) Install the new ones

$ sudo dpkg -i ./woodpecker-*.deb

(e) Restart woodpecker

$ sudo systemctl start woodpecker
$ sudo systemctl start woodpecker-agent

(f) Delete back up files

$ sudo rm /etc/woodpecker-agent.conf.bak /etc/woodpecker.conf.bak

Unfortunately I did not find any restore functions for woodpecker. So I did it blindly.

TESTING

As part of the upgrade of the Woodpecker instance from 0.15.x to 2.3.x, the pipeline key in my woodpecker ymls was changed to steps as the pipeline key is deprecated in 2.3.x. So when I tried to test this change in the upgrade, I got this error message

2024/03/17 13:51:54 ...s/webhook/webhook.go:130:handler() [E] Unable to deliver webhook task[265]: unable to deliver webhook task[265] in https://<website-name>/api/hook?access_token=<token> due to error in http client: Post "https://<website-name>/api/hook?access_token=<token>": dial tcp <website-name>: webhook can only call allowed HTTP servers (check your webhook.ALLOWED_HOST_LIST setting), deny '<website-name>(x.x.x.x)'

So I googled the error and saw this. Basically, they reduced the scope of the webook delivery for security reasons by mandating another setting be made in the app.ini file. So I did the easy thing and added this setting below:

[webhook]
ALLOWED_HOST_LIST = *

Now that did not work. So I had to reconstruct my test repository by

  1. deleting my test repository I had on my Woodpecker instance
  2. deregistering the OAUTH application on the Gitea instance
  3. registered a new OAUTH application to give me a new client id and secret
  4. re-adding my test repository to my Woodpecker instance

When I then pushed a new commit, the builds were picked up. Now, I was not happy with this setting so I looked again at the PR in the link and saw that I could provide the website name for my instance and the ipaddress with the CIDR as values for the ALLOWED_HOST_LIST setting. So to get the ip address with the CIDR setting, I ran ip s a and picked the first non-loopback entry. So the new setting was:

[webhook]
ALLOWED_HOST_LIST = <website-name>,x.x.x.x/x

That worked. I also found this blog post that also does what I did to reduce the attack surface area for the webhooks.

OBSERVATIONS

  • On the whole, the upgrade of Gitea was very smooth as there was an ability to restore if I failed. Both softwares though were easy to upgrade. I initially panicked during the Woodpecker upgrade after receiving errors like this. After I read this, I saw that those errors did not affect the running of my Woodpecker instance.

  • Based on the issues I encountered in testing, I will do the following

    • run the upgrades in a local docker-compose instance to iron out all the kinks.
    • read all the PRs between the versions to determine what changed.
posted at 09:52  ·   ·  cicd  til