Goodbye, Jenkins! Hello, SourceHut!

Posted by Matt Soukup at 02:30 PM MST on 2023-08-27

From the moment I laid eyes on it, I was in love.

No JavaScript. Minimalist styling. Open Source tooling. Founded by an impassioned, idealistic thought leader and hacker.

I pored over the blog and docs in glee. Could SourceHut.org become the new home for Magic in the Browser?

History

Since its inception, MITB was built on a self-hosted instance of Jenkins on DigitalOcean. While Jenkins was the primary purpose, the box tripled as a database replication target and email server. Being a build server, though, required heftier machine specs, and given that builds were relatively rare, it was a lot of wasted resource.

In 2021, DigitalOcean went public, and the following year, prices increased by 20%. It begged the question: was DigitalOcean's simpler pricing model and superior UI still worth the squeeze compared to other cloud providers?

AWS and Azure are the cloud providers that I came to know in my professional career. I had limited exposure to Azure but enough to know I hated the UI and the inflated price. I'd also just taken steps to de-Microsoft my life and wasn't about to succumb to the wickedness and snares of Bill Gates. AWS afforded many opportunities for cost savings such as prepaying in the form of "reserved instances" and stopping instances to avoid incurring cost.

Ultimately, all servers were transferred from DigitalOcean to AWS and that became the catalyst for rethinking the build strategy. The Jenkins instance was expensive and Jenkins itself left much to be desired.

Jenkins

In my opinion, Jenkins time has passed. While it is very extensible, there's a minimum baseline that needs to arrive out-of-the-box for the purposes of building software. Configuration is a trying task- pipelines are an amalgam of plugins rather than a first-class citizens. The community seems to be waning. Plugins are outdated. Blue Ocean failed. I also really despise the Groovy build syntax. Shell commands feel more at home. I wish Jenkins the best and will continue rooting for it, but it's time for me to move on.

I believe there are better alternatives today. JetBrains TeamCity is also free and open source but has the backing of a company. Its pipeline modeling is far superior. I was not looking to run an instance, though, and the price of its cloud service is not amenable to MITB's scale.

SourceHut

SourceHut is the hacker's forge. Here are just a few of its strengths:

Simplicity - It has the features it needs and those features are built right.
Speed - It is the fastest git server and web UI I have ever used.
No Walled Garden - Open Source tools that work over email and do not require registration.
Mission - Open Source. No ads. No tracking. No outside investors.
Builds - Shell-based build specifications submitted over UI or API as build manifests. You can log in to the build server itself to inspect build failures!

In SourceHut, named build jobs are not defined like other platforms. Instead, builds are triggered ad-hoc by submitting yaml-based build manifests. SourceHut lacks the familiar notion of a pipeline but offers enough functionality to emulate one. With my strong desire to migrate to this platform, I treated this as a puzzle to be solved rather than a hurdle. This is how I went about things.

Each repository can define a top-level .build.yml build manifest, which automatically runs on each check-in. Here is the build manifest for mitb-front, the frontend repository for MITB, which builds, tests, and if the branch is master, deploys to staging and runs functional tests.

image: ubuntu/22.04
arch: amd64
repositories:
  nodesource: https://deb.nodesource.com/node_20.x jammy main 1655A0AB68576280
  google-chrome: http://dl.google.com/linux/chrome/deb/ stable main 7721F63BD38B4796
packages:
  - nodejs
  - google-chrome-stable
secrets:
  - <redacted>
  - <redacted>
sources:
  - git@git.sr.ht:~msoukup/mitb-front
  - git@git.sr.ht:~msoukup/mitb-infra
tasks:
  - init: |
      cd mitb-front
      echo "export CURRENT=$(git rev-parse HEAD)" | tee -a ~/.buildenv > /dev/null
      echo "export MASTER=$(git rev-parse master)" | tee -a ~/.buildenv > /dev/null
  - install: |
      cd mitb-front
      npm i
  - build: |
      cd mitb-front
      npm run build
  - test: |
      cd mitb-front
      npm t
  - deploy_stage: |
      if [ "${CURRENT}" = "${MASTER}" ]; then
        mitb-infra/pipeline/deploy-mitb-front.sh stage mitb-front/build/mitb-front.tar.gz
      else
        echo "Skipping deploy stage"
      fi
  - func_test_stage: |
      if [ "${CURRENT}" = "${MASTER}" ] && [ -f ~/.mitb-functional-tests ]; then
        mitb-infra/pipeline/run-func-tests.sh
      else
        echo "Skipping functional test stage"
      fi
artifacts:
  - mitb-front/build/mitb-front.tar.gz
triggers:
  - action: email
    condition: failure
    to: <redacted>

Any other build manifest has to be submitted through the UI or GraphQL API. Shell scripts are the natural way to call the GraphQL API, but we'd like to keep the build manifests themselves in yaml files. The following script, build-runner.sh, accepts a path to a build manifest and optionally an envsubst shell-format for substituting environment variables in that manifest.

#!/bin/bash

if [ $# -ne 1 ] && [ $# -ne 2 ]; then
  echo "Usage: $0 build-manifest-path [envsubst SHELL-FORMAT]"
  exit 1
fi

BUILD_MANIFEST=$1
SHELL_FORMAT="'$2'"

if [ -n "${SHELL_FORMAT}" ]; then
  BUILD_MANIFEST_CONTENT=$(cat ${BUILD_MANIFEST} | envsubst ${SHELL_FORMAT})
else
  BUILD_MANIFEST_CONTENT=$(cat ${BUILD_MANIFEST})
fi

GRAPHQL="mutation {
  submit(manifest: """${BUILD_MANIFEST_CONTENT}""", secrets: true, visibility: PRIVATE) {
    id
    created
    status
    owner {
      canonicalName
    }
  }
}"

STRINGIFIED_JSON=$(echo "${GRAPHQL}" | jq --raw-input --slurp)

RESPONSE=$(curl -s 
--oauth2-bearer "${OAUTH_TOKEN}" 
-H 'Content-Type: application/json' 
-d "{
"query": ${STRINGIFIED_JSON}
}" https://builds.sr.ht/query || exit 1)

JOB_ID=$(echo "${RESPONSE}" | jq .data.submit.id)
USERNAME=$(echo "${RESPONSE}" | jq --raw-output .data.submit.owner.canonicalName)

echo "${RESPONSE}"
echo "https://builds.sr.ht/${USERNAME}/job/${JOB_ID}"

In addition to the build.yml present at the top-level of each repository, I've created build manifests for the following additional tasks:

Run functional tests on any environment
Release to production

Give SourceHut a try! You'll be glad you did.

« Back to Blog