Ad meliora

Aug 01, 2024

Productionizing a golang binary

OUTLINE

Introduction

So after developing my golang project, a vanity url mapper for my golang packages, I needed a way to deploy it from my local machine to my server. So this means I would run it as a systemd1 service. For this, I would make the systemd service run it,

(a) either locally on my server filesystem, or

(b) in a docker container.

Types of systemd service

A systemd service is normally defined in a unit file. The syntax and description of the unit file can be found by running man systemd.unit on any Linux system that supports systemd. So there are conventionally two types of systemd services

(a) System service: This service is enabled for all users that log into the system. To do this, you would normally run this set of commands:

$ sudo cp <SERVICE_NAME>.service /etc/systemd/system/
$ sudo systemctl daemon-rDeploying to run on the filesystemeload
$ sudo systemctl enable <SERVICE_NAME>.service

(b) User service: The service is enabled for only the logged-in user. To do this, you would normally run this set of commands:

$ touch /home/<$USER>/.config/systemd/user/<SERVICE_NAME>.service
$ systemctl --user reload
$ systemctl --user enable --now <SERVICE_NAME>

Enabling the service will then symlink it into the appropriate .wants subdirectory, and it will run only when that user is logged in. This means that if that user is not logged in, systemd might terminate the service2.

The systemd user instance is started after the first login of a user and killed after the last session of the user is closed. Sometimes it may be useful to start it right after boot, and keep the systemd user instance running after the last session closes, for instance to have some user process running without any open session

This is bad for long-running transcations/services. So to do that, you would run this

$ loginctl enable-linger $USER

This disconnects that logged-in user specific systemd instance from that logged-in user session.

Attempt One

I did initially not know the difference between the types of systemd services. So I ran these steps:

(a) Created a sample service

[Unit]
Description=A custom url mapper for go packages!
After=network-online.target

[Service]
ExecStart=/opt/gocustomurls -conf /home/vagrant/.config/config.json
SyslogIdentifier=gocustomurls
StandardError=journal
Type=exec

[Install]
WantedBy=multi-user.target

(b) Ran these steps

$ sudo cp gocustomurls.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl start gocustomurls.service
$ sudo systemctl status gocustomurls.service

So I got this error

× gocustomurls.service - GocustomUrls. A custom url mapper for go packages!
     Loaded: loaded (/etc/systemd/system/gocustomurls.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Fri 2024-07-19 03:31:19 UTC; 54s ago
   Duration: 23ms
    Process: 3027 ExecStart=/opt/gocustomurls -conf /home/vagrant/.config/config.json (code=exited, status=1/FAILURE)
   Main PID: 3027 (code=exited, status=1/FAILURE)
        CPU: 6ms
$ http --body "http://localhost:7070/x/touche?go-get=1"
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
        <meta name="go-import" content="scale.dev/x/migrate git https://codeberg.org/Gusted/mCaptcha.git">
        <meta name="go-source" content="scale.dev/x/migrate https://codeberg.org/Gusted/mCaptcha.git https://codeberg.org/Gusted/mCaptcha.git/tree/main{/dir} https://codeberg.org/Gusted/mCaptcha.git/blob/main{/dir}/{file}#L{line}">                   
    </head>
</html>

Jul 19 03:31:19 fedoramachine systemd[1]: Starting gocustomurls.service - GocustomUrls. A custom url mapper for go packages!...
Jul 19 03:31:19 fedoramachine systemd[1]: Started gocustomurls.service - GocustomUrls. A custom url mapper for go packages!.
Jul 19 03:31:19 fedoramachine gocustomurls[3027]: 2024/07/19 03:31:19 File: => /home/vagrant/.config/config.json
Jul 19 03:31:19 fedoramachine gocustomurls[3027]: 2024/07/19 03:31:19 Ok: => false
Jul 19 03:31:19 fedoramachine gocustomurls[3027]: 2024/07/19 03:31:19 Warning, generating default config file
Jul 19 03:31:19 fedoramachine gocustomurls[3027]: 2024/07/19 03:31:19 neither $XDG_CONFIG_HOME nor $HOME are defined
Jul 19 03:31:19 fedoramachine systemd[1]: gocustomurls.service: Main process exited, code=exited, status=1/FAILURE
Jul 19 03:31:19 fedoramachine systemd[1]: gocustomurls.service: Failed with result 'exit-code'.

Basically the error is that the $HOME environment variable is empty. This is consistent with some of the properties of system services which are:

  • Their spawned processes inherit NO environment (e.g., in a shell script run by the service, the $HOME environment variable will actually be empty)

  • They run as root by default. As a result, they have root permissions.

Attempt Two

So I tried to fix some of the issues with my first attempt by creating a user service. I went through these steps:

(a) Created a non-root user.

Debian

$ sudo adduser \
    --system \ # Non-expiring accounts
    --shell /bin/bash \ # the login shell
    --comment 'Go Custom Urls Service' \ # removes finger info, making the command non-interactive 
    --group \ # creates an identically named group as its primary group
    --disabled-password --home /home/$USER $USER

Fedora

$ sudo useradd --system --shell /bin/bash --comment 'Go Custom Urls Service' --home-dir /home/gourls -m gourls

(b) Unlocked the account by deleting the password

$ sudo passwd -d gourls // to unlock the account

(c) Fixed the permissions. I used this resource3 and this resource4 to generate the required permissions.

$ sudo chown -R gourls:gourls /home/gourls
$ sudo chmod -R 770 /home/gourls

(d) Modified the servcie unit above to change the invocation of the service to a user throught the use of the User= and Group= directives. 

[Unit]
Description=GocustomUrls. A custom url mapper for go packages!
After=network-online.target

[Service]
User=gols
Group=gols
ExecStart=/home/gols/gocustomurls -conf /home/gols/.config/gocustomurls/config.json
SyslogIdentifier=gocustomurls
StandardError=journal
Type=simple
WorkingDirectory=/home/gols
Environment=USER=gols HOME=/home/gols


[Install]
WantedBy=multi-user.target

(e) Copied the unit file to the config folder.

$ sudo mkdir -p /home/gols/.config/systemd/user
$ sudo chmod -R 770 /home/gols
$ sudo cp gocustomurls.service /home/gols/.config/systemd/user/
$ sudo chown -R gols:gols /home/gols

(f) Reloaded systemd services

$ su gols
$ systemctl --user daemon-reload
Failed to connect to bus: Permission denied

So after googling this error, I was told to check if my XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS was set correctly. So checking,

$ id
uid=993(gols) gid=992(gols) groups=992(gols) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
$ echo $XDG_RUNTIME_DIR
/run/user/1000
$ echo $DBUS_SESSION_BUS_ADDRESS
unix:path=/run/user/1000/bus

As you can see above, the variables XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS are configured with the incorrect user id. So I tried to export them in the user's ~/.bash_profile

$ cat ~/.bash_profile
...
export XDG_RUNTIME_DIR="/run/user/$UID"
export DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/bus

This did not help. So after googling, I came across this resource5 and tried that

$ systemctl -M gols@ --user daemon-reload
Failed to connect to bus: Permission denied

So after further googling, I came across this resource6. From this resource, there are basically two requirements for systemctl --user command to work which are:

  • The user instance of the systemd daemon must be alive, and

  • systemctl must be able to find it through certain path variables.

For the first condition, the loginctl enable-linger command enables that to happen by starting a user instance of systemd even if the user is not logged in. For the second condition, su or su --login does not set the required variables. To do that, you would normally ssh into that user's session. This is obviously a non-starter for me. But there is a new command called machinectl7 that obviates that need. So armed with this information, I was ready for another attempt.

Attempt Three

So machinectl7 is available from systemd-container (Debian/Fedora). So I ran these steps based on this resource8 and this resource5:

$ sudo dnf install systemd-container
$ sudo loginctl enable-linger gols
$ sudo machinectl shell --uid=gols
Connected to the local host. Press ^] three times within 1s to exit session.
$ exit
$ sudo systemctl -M gourls@ --user daemon-reload
$ su gopls
$ echo $XDG_RUNTIME_DIR
/run/user/993
$ echo $DBUS_SESSION_BUS_ADDRESS
unix:path=/run/user/993/bus

In order to the command sudo systemctl -M gourls@ --user daemon-reload to be successful, the variables from the previous attempt must be exported in ~/.bash_profile. Okay, I then ran a journalctl command to check the status of the failed service

$ journalctl --user -u gocustomurls.service
Hint: You are currently not seeing messages from the system.
      Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.
      Pass -q to turn off this notice.
No journal files were opened due to insufficient permissions.

So I added the gourls user to the systemd-journal group

$ sudo usermod -a -G systemd-journal gols
$ sudo systemctl -M gols@ --user list-units | grep "gocustom" # list all the units and find gocustom

So far so good. Restarting the service produces an error

$ sudo systemctl -M gourls@ --user restart gocustomurls.service
$ su gols
$ journalctl --user -u gocustomurls.service
Jul 19 07:19:03 fedoramachine systemd[749]: Started gocustomurls.service - GocustomUrls. A custom url mapper for go packages!.
Jul 19 07:19:03 fedoramachine (stomurls)[3291]: gocustomurls.service: Failed to determine supplementary groups: Operation not permitted
Jul 19 07:19:03 fedoramachine systemd[749]: gocustomurls.service: Main process exited, code=exited, status=216/GROUP
Jul 19 07:19:03 fedoramachine systemd[749]: gocustomurls.service: Failed with result 'exit-code'.

Based on the error above, it seemed that the group was incorrect. This observation was also confirmed after viewing this9. So I removed the User= , Group= and Environment= directoves from the unit file and restarted the service.

[Unit]
Description=GocustomUrls. A custom url mapper for go packages!
After=network-online.target

[Service]
ExecStart=/home/gols/gocustomurls -conf /home/gols/.config/gocustomurls/config.json
SyslogIdentifier=gocustomurls
StandardError=journal
WorkingDirectory=/home/gols
Type=simple

[Install]
WantedBy=multi-user.target

$ sudo cp gocustomurls.service /home/gols/.config/systemd/user/
$ sudo chown -R gols:gols /home/gols
$ sudo machinectl shell --uid=gols
Connected to the local host. Press ^] three times within 1s to exit session.
$ exit
logout
Connection to the local host terminated.
$ sudo systemctl -M gols@ --user daemon-reload
$ sudo systemctl -M gols@ --user restart gocustomurls.service
$ sudo systemctl -M gols@ --user status gocustomurls.service
× gocustomurls.service - GocustomUrls. A custom url mapper for go packages!
     Loaded: loaded (/home/gols/.config/systemd/user/gocustomurls.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Fri 2024-07-19 10:02:31 UTC; 8min ago
   Duration: 1ms
    Process: 2919 ExecStart=/home/gols/gocustomurls -conf /home/gols/.config/gocustomurls/config.json (code=exited, status=203/EXEC)
   Main PID: 2919 (code=exited, status=203/EXEC)
        CPU: 1ms

The error was because of a missing routes.json. The binary that the gocustomurls.service runs, returns an exit code if routes.json is not found. Adding a rules.json to the folder that is read by the gocustomurls.service returns a success

$ sudo cp rules.json /home/gols/.config/gocustomcurls/
$ sudo systemctl -M gols@ --user daemon-reload
$ sudo systemctl -M gols@ --user restart gocustomurls.service
$ sudo systemctl -M gols@ --user status gocustomurls.service

● gocustomurls.service - GocustomUrls. A custom url mapper for go packages!
     Loaded: loaded (/home/gols/.config/systemd/user/gocustomurls.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sat 2024-07-20 05:16:31 UTC; 28s ago
   Main PID: 3158
      Tasks: 6 (limit: 2319)
     Memory: 1.7M (peak: 2.0M)
        CPU: 4ms
     CGroup: /user.slice/user-993.slice/user@993.service/app.slice/gocustomurls.service
             └─3158 /home/gols/gocustomurls -conf /home/gols/.config/gocustomurls/config.json

Testing with httpie results in success.

$ sudo dnf install httpie
$ http --body "http://localhost:7070/x/touche?go-get=1"
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
        <meta name="go-import" content="scale.dev/x/migrate git https://codeberg.org/Gusted/mCaptcha.git">
        <meta name="go-source" content="scale.dev/x/migrate https://codeberg.org/Gusted/mCaptcha.git https://codeberg.org/Gusted/mCaptcha.git/tree/main{/dir} https://codeberg.org/Gusted/mCaptcha.git/blob/main{/dir}/{file}#L{line}">                   
    </head>
</html>

Attempt Four

So I was not happy with the amount of configuration needed to get this working. So I decided to just write a system service instead because it seemed like the path of least resistance. So I ran these commends:

$ echo $PATH
/home/vagrant/.asdf/shims:/home/vagrant/.asdf/bin:/home/vagrant/.local/bin:/home/vagrant/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
$ sudo go <gocustomurls_binary> /usr/local/bin # to put in the PATH
$ sudo mkdir -p /var/lib/$USER
$ sudo cp gocustomurls.service /etc/systemd/system/
$ sudo mkdir -p /var/lib/$USER
$ sudo useradd --system --comment 'Go Custom Urls Service' --no-create-home $USER
useradd: failed to reset the lastlog entry of UID 992: No such file or directory
$ sudo passwd -d $USER
passwd: password changed.
$ getent passwd $USER
$USER:x:992:991:Go Custom Urls Service:/home/$USER:/bin/bash
$ sudo cp config.json /var/lib/$USER
$ sudo cp rules.json /var/lib/$USER
$ sudo ls /var/lib/$USER/
config.json  rules.json
$ sudo chmod -R 770 /var/lib/$USER
$ sudo chown -R $USER:$USER /var/lib/$USER
$ sudo systemctl daemon-reload
$ sudo systemctl start gocustomurls.service
$ sudo systemctl status gocustomurls.service
● gocustomurls.service - GocustomUrls. A custom url mapper for go packages!
     Loaded: loaded (/etc/systemd/system/gocustomurls.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sat 2024-07-20 06:52:09 UTC; 23s ago
   Main PID: 4020 (gocustomurls)
      Tasks: 6 (limit: 2319)
     Memory: 7.1M (peak: 7.5M)
        CPU: 14ms
     CGroup: /system.slice/gocustomurls.service
             └─4020 /usr/local/bin/gocustomurls -conf /var/lib/gourls/config.json
$ http --body "http://localhost:7070/x/touche?go-get=1"
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
        <meta name="go-import" content="scale.dev/x/migrate git https://codeberg.org/Gusted/mCaptcha.git">
        <meta name="go-source" content="scale.dev/x/migrate https://codeberg.org/Gusted/mCaptcha.git https://codeberg.org/Gusted/mCaptcha.git/tree/main{/dir} https://codeberg.org/Gusted/mCaptcha.git/blob/main{/dir}/{file}#L{line}">                   
    </head>
</html>

Attempt Five (Docker)

So this application generates logs and I wanted to view the logs outside the container. So the application logs generated as I run the container would have to have the same user as the user invoking the docker command to avoid me having to use sudo to view it. Based on the above, the sample docker file would be something like this

# syntax=docker/dockerfile:1
FROM golang:1.20-alpine AS build

WORKDIR /app

RUN apk --update add --no-cache git bash make alpine-sdk g++ && \
    git clone https://git.iratusmachina.com/iratusmachina/gocustomurls.git --branch feature/add-log-rotation gurls && \
    cd gurls && make build

FROM alpine AS run

ARG USERNAME
ARG UID
ARG GID
ARG PORT

RUN mkdir -p /home/$USERNAME

COPY --from=build /app/gurls/artifacts/gocustomurls /home/$USERNAME/gocustomurls

RUN addgroup -g ${GID} ${USERNAME} && \
    adduser --uid ${UID} -G ${USERNAME} --gecos "" --disabled-password --home "/home/$USERNAME" --no-create-home $USERNAME && \
    chmod -R 755 /home/$USERNAME && \
    chown -R ${UID}:${GID} /home/$USERNAME

ENV HOME=/home/$USERNAME

WORKDIR $HOME

EXPOSE $PORT

USER $USERNAME

CMD ["sh","-c", "/home/${USERNAME}/gocustomurls", "-conf", "${HOME}/config.json"] # based on this link https://github.com/moby/moby/issues/5509#issuecomment-1962309549

So running these set of commands produces an error:

$ docker build -t testimage:alpine --build-arg UID=$(id -u) --build-arg GID=$(id -g) --build-arg USERNAME=$(whoami) --build-arg PORT=7070 --no-cache .
...
=> => naming to docker.io/library/testimage:alpine
$ docker run -d -p 7070:7070 -v ${PWD}:/home/$(whoami) testimage:alpine
d7f5011b4e7f6317757a3ee8fe76fe710f8fd5ed1ac6a6e5c32ad5393c78f4c0
$ http --body "http://localhost:7070/x/touche?go-get=1"

http: error: ConnectionError: HTTPConnectionPool(host='localhost', port=7070): Max retries exceeded with url: /x/touche?go-get=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f986fc73410>: Failed to establish a new connection: [Errno 111] Connection refused')) while doing a GET request to URL: http://localhost:7070/x/touche?go-get=1

So checking the logs,

$ docker container ls -a
CONTAINER ID   IMAGE              COMMAND                  CREATED          STATUS                        PORTS     NAMES
d7f5011b4e7f   testimage:alpine   "sh -c /home/${USERN…"   56 seconds ago   Exited (127) 55 seconds ago             frosty_leakey

$ docker logs d7f5011b4e7f
-conf: line 0: /home//gocustomurls: not found

So I attempted to fix the above error by putting the CMD command in a start.sh file by doing this:

# syntax=docker/dockerfile:1
...
RUN echo "${HOME}/gocustomurls -conf ${HOME}/config.json" > start.sh \
    && chown ${UID}:${GID} start.sh \
    && chmod u+x start.sh

EXPOSE $PORT

USER $USERNAME

CMD ["sh", "-c", "./start.sh" ]

This also produced an error, after building and running the container as shown below:

$ docker logs e9abe2bed4b1
sh: ./gocustomurls: not found

This perplexed me. So I decided to print the environment with printenv and run ls in the start.sh file. So after doing that, building and re-running the container, I got this:

$ docker logs 98dd740dc727
HOSTNAME=98dd740dc727
SHLVL=2
HOME=/home/vagrant
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
total 12K    
drwxr-xr-x    2 vagrant  vagrant       61 Jul 21 21:30 .
drwxr-xr-x    1 root     root          21 Aug  3 14:06 ..
-rw-r--r--    1 vagrant  vagrant     2.4K Aug  3 14:06 Dockerfile
-rw-r--r--    1 vagrant  vagrant      131 Jul 21 21:18 config.json
-rw-r--r--    1 vagrant  vagrant      471 Jul 21 18:40 rules.json

This shows that the gocustomurls binary was being deleted by the time I wanted to run a container from the image. This was confirmed here 12. So I instead used the approach shown here 13. There is also using a data-only volume but that is for another day. So after reading 13 and 14, I came up with new Dockerfile.

# syntax=docker/dockerfile:1.4
FROM golang:1.20-alpine AS build

WORKDIR /app

RUN <<EOF
apk --update add --no-cache git bash make alpine-sdk g++
git clone https://git.iratusmachina.com/iratusmachina/gocustomurls.git --branch feature/add-log-rotation gurls
cd gurls
make build
EOF

FROM alpine AS run

ARG USERNAME
ARG UID
ARG GID
ARG PORT

RUN <<EOF
addgroup --gid ${GID} ${USERNAME}
adduser --uid ${UID} --ingroup ${USERNAME} --gecos "" --disabled-password --home "/home/$USERNAME" $USERNAME
chmod -R 755 /home/$USERNAME
chown -R ${UID}:${GID} /home/$USERNAME
EOF

COPY --from=build /app/gurls/artifacts/gocustomurls /home/$USERNAME/gocustomurls

RUN <<EOF
mkdir -p /var/lib/apptemp
cp /home/$USERNAME/gocustomurls /var/lib/apptemp/gocustomurls
chown -R ${UID}:${GID} /home/$USERNAME
chown -R ${UID}:${GID} /var/lib/apptemp
ls -lah /home/$USERNAME
EOF


ENV HOME=/home/$USERNAME

COPY <<EOF start.sh
    #!/usr/bin/env sh
    printenv
    cd "${HOME}"
    ls -lah .
    if [ ! -f "${HOME}/gocustomurls" ]
    then
        cp /var/lib/apptemp/gocustomurls ${HOME}/gocustomurls
        rm /var/lib/apptemp/gocustomurls
    fi
    ${HOME}/gocustomurls -conf ${HOME}/config.json
EOF

RUN <<EOF
chown ${UID}:${GID} start.sh
chmod u+x start.sh
EOF

EXPOSE $PORT

USER $USERNAME

CMD ["sh", "-c", "./start.sh" ]

I also changed the run command from

$ docker run -d -p 7070:7070 -v ${PWD}:/home/$(whoami) testimage:alpine

to 

$ docker run -d -p 7070:7070 -v ${PWD}/otherfiles:/home/$(whoami) testimage:alpine

This allows me to exclude the Dockerfile from the image. I also added journald logging to the run command so I can use journalctl to view the logs. Running with this addition is shown below:

$ docker run --log-driver=journald -d -p 7070:7070 -v ${PWD}/otherfiles:/home/$(whoami) testimage:alpine
$ sudo journalctl CONTAINER_NAME=vigilant_mclaren
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: HOSTNAME=561944ae05ca
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: SHLVL=2
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: HOME=/home/vagrant
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: PWD=/
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: total 9M     
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: drwxr-xr-x    2 vagrant  vagrant       63 Aug  3 14:36 .
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: drwxr-xr-x    1 root     root          21 Aug  3 14:29 ..
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: -rw-r--r--    1 vagrant  vagrant      131 Jul 21 21:18 config.json
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: -rwxr-xr-x    1 vagrant  vagrant     9.3M Aug  3 14:31 gocustomurls
Aug 03 14:37:19 fedoramachine 561944ae05ca[1228]: -rw-r--r--    1 vagrant  vagrant      471 Jul 21 18:40 rules.json

This works. So I wanted to see if I could run this Dockerfile from a non-root, non-login account. After moving the requiste files, I ran the build command

bash-5.2$ docker build -t testimage:alpine --build-arg UID=$(id -u) --build-arg GID=$(id -g) --build-arg USERNAME=$(whoami) --build-arg PORT=7070 --no-cache --progress=plain .
ERROR: mkdir /home/golsuser: permission denied

I am using the --progress=plain option to allow me to use echo during the build process. This error is because docker binary cannot find the .docker directory for this user. Normally, this folder is auto-created when you run docker build. So to remove this error, I ran this instead.

bash-5.2$ HOME=${PWD} docker build -t testimage:alpine --build-arg UID=$(id -u) --build-arg GID=$(id -g) --build-arg USERNAME=$(whoami) --build-arg PORT=7070 --no-cache --progress=plain .

This worked. Now I normally run my dockerfiles using Docker Compose. So a sample compose file for the Dockerfile is shown below:

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - USERNAME=${USERNAME}
        - UID=${USER_ID}
        - GID=${GROUP_ID}
        - PORT=${PORT:-7070}
      labels:
        - "maintainer=iratusmachina"
    image: appimage/gocustomurls
    logging:
      driver: journald
    container_name: app_cont
    ports:
      - "7070:7070"
    volumes:
      - ${PWD}/otherfiles:/home/${USERNAME}

I also wanted to run the compose file using systemd. So the systemd unit file is shown below:

[Unit]
Description=GocustomUrls. A custom url mapper for go packages!
After=docker.service
PartOf=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/var/lib/golsuser
User=golsuser
Group=golsuser
ExecStart=/bin/bash -c "HOME=/var/lib/golsuser USER_ID=$(id -u) GROUP_ID=$(id -g) USERNAME=$(whoami) docker compose --file /var/lib/golsuser/docker-compose.yml up --detach --force-recreate"
ExecStop=/bin/bash -c "HOME=/var/lib/golsuser USER_ID=$(id -u) GROUP_ID=$(id -g) USERNAME=$(whoami) docker compose --file /var/lib/golsuser/docker-compose.yml down -v"
SyslogIdentifier=gurls
StandardError=journal

[Install]
WantedBy=multi-user.target

So running this service produced this output:

$ sudo systemctl start gurls.service
$ sudo systemctl status gurls.service

● gurls.service - GocustomUrls. A custom url mapper for go packages!
     Loaded: loaded (/etc/systemd/system/gurls.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Sat 2024-08-03 18:08:46 UTC; 50s ago
    Process: 24117 ExecStart=/bin/bash -c HOME=/var/lib/golsuser USER_ID=$(id -u) GROUP_ID=$(id -g) USERNAME=$(whoami) docker compose --file /var/lib/golsuser/docker-compose.yml up --detach --force-recreate (code=exited, status=0/SUCCESS)
   Main PID: 24117 (code=exited, status=0/SUCCESS)
        CPU: 125ms

Aug 03 18:08:45 fedoramachine gurls[24132]: #17 DONE 0.0s
Aug 03 18:08:45 fedoramachine gurls[24132]: #18 [app] resolving provenance for metadata file
Aug 03 18:08:45 fedoramachine gurls[24132]: #18 DONE 0.0s
Aug 03 18:08:45 fedoramachine gurls[24132]:  Network golsuser_default  Creating
Aug 03 18:08:45 fedoramachine gurls[24132]:  Network golsuser_default  Created
Aug 03 18:08:45 fedoramachine gurls[24132]:  Container app_cont  Creating
Aug 03 18:08:45 fedoramachine gurls[24132]:  Container app_cont  Created
Aug 03 18:08:45 fedoramachine gurls[24132]:  Container app_cont  Starting
Aug 03 18:08:46 fedoramachine gurls[24132]:  Container app_cont  Started
Aug 03 18:08:46 fedoramachine systemd[1]: Finished gurls.service - GocustomUrls. A custom url mapper for go packages!.

$ sudo systemctl stop gurls.service
$ sudo systemctl status gurls.service

○ gurls.service - GocustomUrls. A custom url mapper for go packages!
     Loaded: loaded (/etc/systemd/system/gurls.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)

Aug 03 18:08:46 fedoramachine systemd[1]: Finished gurls.service - GocustomUrls. A custom url mapper for go packages!.
Aug 03 18:10:10 fedoramachine systemd[1]: Stopping gurls.service - GocustomUrls. A custom url mapper for go packages!...
Aug 03 18:10:10 fedoramachine gurls[24402]:  Container app_cont  Stopping
Aug 03 18:10:20 fedoramachine gurls[24402]:  Container app_cont  Stopped
Aug 03 18:10:20 fedoramachine gurls[24402]:  Container app_cont  Removing
Aug 03 18:10:20 fedoramachine gurls[24402]:  Container app_cont  Removed
Aug 03 18:10:20 fedoramachine gurls[24402]:  Network golsuser_default  Removing
Aug 03 18:10:20 fedoramachine gurls[24402]:  Network golsuser_default  Removed
Aug 03 18:10:20 fedoramachine systemd[1]: gurls.service: Deactivated successfully.
Aug 03 18:10:20 fedoramachine systemd[1]: Stopped gurls.service - GocustomUrls. A custom url mapper for go packages!.

Conclusion

This was a fun yet frustruating experience in trying to piece together how to run services with systemd. I am glad that I went through it so I can reference it next time. I decided to go with the systemd approach (attempt four). With go artifacts being a single binary, the overhead of managing Docker was not worth it for me. If I wanted to remove the systemd service, I could use the commands from this resource10.

  1. Systemd ↩

  2. Systemd stopping user services ↩

  3. Chmod calculator ↩

  4. Other chmod calculator ↩

  5. Usage of machinectl ↩↩

  6. Interplay between su and systemd ↩

  7. Machinectl ↩↩

  8. Machinectl invocation ↩

  9. Group exit status code ↩

  10. Removing a systemd service ↩

  11. Using a start.sh ↩

  12. Bind mounts overwriting files ↩

  13. Get around bind mounts ↩↩

  14. Heredocs ↩

posted at 09:52  ·   ·  cicd  debugging

Mar 16, 2024

Upgrading my Gitea/Woodpecker Setup

OUTLINE

REASONS

So I am using gitea-1.19.4 as well as Woodpecker-0.15.x. The current version(s) of gitea and woodpecker are 1.21.x and 2.3.x respectively.

I wanted to upgrade to these versions because of:

(a) Gitea 1.21.x comes with Gitea Actions enabled by default. THis is similar to Github Actions which I want to learn as we use that at $DAYJOB.

(b) I want to upgrade to a new version every year. This is to make upgrades manageable and repeatable so by doing it more frequently, I would have a runbook for how to upgrade.

(c) These versions (1.19.x, 0.x.x) are no longer actively maintained, which could mean that they are stable but not receiving security fixes which is something I want to avoid.

(d) Woodpecker 2.3.x adds the ability to use Forego which is a community fork of Gitea and runs Codeberg. I may switch to Forego in the future as I may want to hack on it and contribute back to it.

PLAN OF ACTION

Just persuing the Gitea changelong and Woodpecker changelog gives me the sense that a lot has changed between versions. This might mean that the upgrade might be seamless or not. So my plan of action is this:

(a) Copy over my data from the two instances to my local computer

(b) Upgrade Gitea. (Ideally, I would have loved to resize the instances but I am too lazy to do so)

(c) Upgrade Woodpecker (Ideally, I would have loved to resize the instances but I am too lazy to do so)

(d) Test, If botched, then recreate new instances (what a pain in the ass, that would be)

GITEA

For the Gitea instance, I would mainly be upgrading to 1.21.x and with the Woodpecker instance, I would be upgrading to 2.3.x. For Gitea,you would:

  • Backup database
  • Backup Gitea config
  • Backup Gitea data files in APP_DATA_PATH
  • Backup Gitea external storage (eg: S3/MinIO or other storages if used)

The steps I have used are taken from here and here.

(a) Define some environment variables

$ export GITEABIN="/usr/local/bin/gitea"
$ export GITEACONF="/etc/gitea/app.ini"
$ export GITEAUSER="git"
$ export GITEAWORKPATH="/var/lib/gitea"

(a) Get current version

$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" --version | cut -d ' ' -f 3

(b) Get gitea version to install

$ export GITEAVERSION=$(curl --connect-timeout 10 -sL https://dl.gitea.com/gitea/version.json | jq -r .latest.version)

(c) Download version from (b) and uncompress

$ binname="gitea-${GITEAVERSION}-linux-amd64"
$ binurl="https://dl.gitea.com/gitea/${giteaversion}/${binname}.xz"
$ curl --connect-timeout 10 --silent --show-error --fail --location -O "$binurl{,.sha256,.asc}"
$ sha256sum -c "${binname}.xz.sha256"
$ rm "${binname}".xz.{sha256,asc}
$ xz --decompress --force "${binname}.xz"

(d) Flush Gitea queues so that you can properly backup

$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" manager flush-queues

(e) Stop gitea

$ sudo systemctl stop gitea

(f) Dump database using pg_dump

$ pg_dump -U $USER $DATABASE > gitea-db.sql

(g) Dump the gitea config and files

$ mkdir temp
$ chmod 777 temp # necessary otherwise the dump fails
$ cd temp
$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" dump --verbose

(h) Move the downloaded gitea binary to the new location

$ sudo cp -f "${GITEABIN}" "${GITEABIN}.bak"
$ sudo mv -f "${binname}" "${GITEABIN}"

(i) Restart Gitea

$ sudo chmod +x "${GITEABIN}"
$ sudo systemctl start gitea
$ sudo systemctl status gitea

To restore as taken from here

$ sudo systemctl stop gitea
$ sudo cp -f "${GITEABIN}.bak" "${GITEABIN}"
$ cd temp
$ unzip gitea-dump-*.zip
$ cd gitea-dump-*
$ sudo mv app.ini "${GITEACONF}"
$ sudo mv data/* "${GITEAWORKPATH}/data/"
$ sudo mv log/* "${GITEAWORKPATH}/log/"
$ sudo mv repos/* "${GITEAWORKPATH}/gitea-repositories/"
$ sudo chown -R gitea:gitea /etc/gitea/conf/app.ini "${GITEAWORKPATH}"
$ cp ../gitea-db.sql .
$ psql -U $USER -d $DATABASE < gitea-db.sql
$ sudo systemctl start gitea
$ sudo systemctl stop gitea
# Regenerate repo hooks
$ sudo --user "${GITEAUSER}" "${GITEABIN}" --config "${GITEACONF}" --work-path "${GITEAWORKPATH}" admin regenerate hooks

WOODPECKER

(a) Stop agent and server

$ sudo systemctl stop woodpecker
$ sudo systemctl stop woodpecker-agent

(b) Backup database with pg_dump

$ pg_dump -U woodpecker woodpeckerdb > woodpecker-db.sql
$ pg_dump -U $USER $DATABASE > woodpecker-db.sql
$ sudo cp /etc/woodpecker-agent.conf woodpecker-agent.conf.bak
$ sudo cp /etc/woodpecker.conf woodpecker.conf.bak

(c) Download the binaries

$ woodpeckerversion=2.3.0
$ binurl="https://github.com/woodpecker-ci/woodpecker/releases/download/v${woodpeckerversion}/"
$ curl --connect-timeout 10 --silent --show-error --fail --location -O "${binurl}woodpecker{,-agent,-cli,-server}_${woodpeckerversion}_amd64.deb"
$ ls
woodpecker-cli_2.3.0_amd64.deb 
woodpecker-agent_2.3.0_amd64.deb
woodpecker-server_2.3.0_amd64.deb

(d) Install the new ones

$ sudo dpkg -i ./woodpecker-*.deb

(e) Restart woodpecker

$ sudo systemctl start woodpecker
$ sudo systemctl start woodpecker-agent

(f) Delete back up files

$ sudo rm /etc/woodpecker-agent.conf.bak /etc/woodpecker.conf.bak

Unfortunately I did not find any restore functions for woodpecker. So I did it blindly.

TESTING

As part of the upgrade of the Woodpecker instance from 0.15.x to 2.3.x, the pipeline key in my woodpecker ymls was changed to steps as the pipeline key is deprecated in 2.3.x. So when I tried to test this change in the upgrade, I got this error message

2024/03/17 13:51:54 ...s/webhook/webhook.go:130:handler() [E] Unable to deliver webhook task[265]: unable to deliver webhook task[265] in https://<website-name>/api/hook?access_token=<token> due to error in http client: Post "https://<website-name>/api/hook?access_token=<token>": dial tcp <website-name>: webhook can only call allowed HTTP servers (check your webhook.ALLOWED_HOST_LIST setting), deny '<website-name>(x.x.x.x)'

So I googled the error and saw this. Basically, they reduced the scope of the webook delivery for security reasons by mandating another setting be made in the app.ini file. So I did the easy thing and added this setting below:

[webhook]
ALLOWED_HOST_LIST = *

Now that did not work. So I had to reconstruct my test repository by

  1. deleting my test repository I had on my Woodpecker instance
  2. deregistering the OAUTH application on the Gitea instance
  3. registered a new OAUTH application to give me a new client id and secret
  4. re-adding my test repository to my Woodpecker instance

When I then pushed a new commit, the builds were picked up. Now, I was not happy with this setting so I looked again at the PR in the link and saw that I could provide the website name for my instance and the ipaddress with the CIDR as values for the ALLOWED_HOST_LIST setting. So to get the ip address with the CIDR setting, I ran ip s a and picked the first non-loopback entry. So the new setting was:

[webhook]
ALLOWED_HOST_LIST = <website-name>,x.x.x.x/x

That worked. I also found this blog post that also does what I did to reduce the attack surface area for the webhooks.

OBSERVATIONS

  • On the whole, the upgrade of Gitea was very smooth as there was an ability to restore if I failed. Both softwares though were easy to upgrade. I initially panicked during the Woodpecker upgrade after receiving errors like this. After I read this, I saw that those errors did not affect the running of my Woodpecker instance.

  • Based on the issues I encountered in testing, I will do the following

    • run the upgrades in a local docker-compose instance to iron out all the kinks.
    • read all the PRs between the versions to determine what changed.
posted at 09:52  ·   ·  cicd  til