What I Learned Adding Packit CI to a Fedora Package

The task

I was assigned a seemingly simple ticket: add a gating test to console-login-helper-messages (CLHM). It's a small package - shell scripts that show SSH keys, IP addresses, and failed systemd units on the login terminal. No compilation, no binaries, just shell scripts and systemd units. The work happened in PR #136.

"Add a gating test" turned into adding TMT tests, Packit CI, an in-tree spec file, CentOS Stream build targets, a propose_downstream job, and eventually tagging a release. Here's everything I learned along the way.

What are all these things

If you're new to Fedora packaging (like I was), the number of tools and systems is overwhelming. Here's the landscape as I understand it now.

Upstream is the GitHub repo where developers write code. This is the source of truth for the code itself.

Dist-git is Fedora's packaging repo at src.fedoraproject.org. It doesn't contain source code - it contains the recipe for building an RPM: a spec file, patches, and a pointer to an upstream tarball. Think of upstream as the ingredients and dist-git as the recipe card. CentOS Stream has its own dist-git on gitlab.com/redhat/centos-stream/rpms/.

Koji is Fedora's build system. When someone merges to dist-git, Koji picks up the spec and tarball and produces an RPM. For Red Hat, it's called Brew.

COPR is a community build service - like a personal Koji. Upload source code, it builds an RPM and hosts it in a repo anyone can dnf install from. Packit uses COPR to build RPMs from pull requests.

TMT (Test Management Tool) is the framework that runs tests. It uses FMF (Flexible Metadata Format) - YAML files with .fmf extension that describe what to test and how. OSCI uses TMT to gate RPMs before they enter RHEL/RHCOS composes.

Packit is the glue. It's a GitHub App that bridges upstream repos and Fedora packaging. On pull requests, it can rebuild the RPM from your PR source (via COPR) and run TMT tests against it (via Testing Farm). On releases, it can sync the spec file to dist-git automatically.

Testing Farm is the infrastructure that runs TMT tests. It spins up a real Fedora (or CentOS Stream) VM, installs your COPR-built RPM, and runs the tests.

The TMT smoke test

The first thing reviewers asked for was a smoke test. For a shell-script package like CLHM, this is intentionally minimal:

# tests/tmt/tests/core/core.fmf
summary: check clhm package is installed
tag:
  - smoke
test: |
  set -xeuo pipefail
  rpm -q console-login-helper-messages

That's it - rpm -q checks if the package installed. It sounds trivial, but an RPM can build successfully and still fail to install (broken dependencies, file conflicts, scriptlet errors). This is the same pattern ignition, afterburn, and coreos-installer use.

The test has two companion plan files:

# tests/tmt/plans/main.fmf - installs the package before tests run
prepare:
  - name: Install console-login-helper-messages package
    how: install
    package: console-login-helper-messages

# tests/tmt/plans/smoke.fmf - discovers and runs smoke-tagged tests
summary: Basic smoke test
discover:
  how: fmf
  filter: "tag: smoke"
execute:
  how: tmt

main.fmf installs the package. smoke.fmf finds tests tagged smoke and runs them. They work together.

How Packit tests your PR's actual code

Here's the part that confused me at first. When Packit's tests job runs on Testing Farm, the test VM is a stock Fedora machine. But the test is supposed to verify your PR's code, not the stock Fedora package. How?

When copr_build finishes, Testing Farm automatically adds the COPR repo (where your PR's RPM was just built) as a package source on the test VM. So when main.fmf says how: install, dnf sees two sources:

1. The official Fedora repo (stock package, v0.22.0)
2. The COPR repo (PR-built package, v0.22.0-1.pr136.xxx)

The COPR version has a higher release string, so dnf picks it. That's it - no configuration needed. Packit and Testing Farm handle it behind the scenes.

The spec file saga

This was the hardest part, and where I learned the most.

In-tree vs fetch from dist-git

There are two approaches for getting a spec file in Packit:

Fetch from dist-git - a post-upstream-clone action downloads the spec from src.fedoraproject.org before each build. This is what afterburn and ignition do. The dist-git spec is the single source of truth.

In-tree spec - the spec file lives in the upstream repo. This is what bootupd does. Spec changes are testable in PRs, and propose_downstream syncs the spec to dist-git on releases.

I started with the fetch approach, but the dist-git spec didn't match the current source - a file had been removed upstream but was still listed in the spec's %files section. Every build failed. So when a reviewer suggested moving the spec in-tree, I did.

Things that broke

%autochangelog on CentOS Stream 9. I replaced the 200-line manual changelog with %autochangelog, which Packit expands automatically. But I also added srpm_build_deps: rpmautospec-rpm-macros thinking it was needed - and that broke all builds because the package doesn't exist in COPR mock chroots. The fix: just remove srpm_build_deps. Packit handles %autochangelog natively. I verified this by checking bootc's config - they use %autochangelog with no srpm_build_deps.

Patch file not found in SOURCES. The dist-git has a patch that reverts /run/issue.d paths to /etc/issue.d for RHEL builds. When I added it as Patch0: in the spec, rpmbuild looked for it in its SOURCES/ directory - but Packit only puts the spec and the tarball there, not extra files from the repo. The error:

error: Bad file: /builddir/build/SOURCES/0001-revert-issue-to-etc.patch: No such file or directory

The workaround is to skip Patch0: and apply the patch manually in %prep:

%prep
%autosetup -p1
%if 0%{?rhel}
patch --forward -p1 < rpm/0001-revert-issue-to-etc.patch || true
%endif

The --forward flag skips hunks that are already applied (part of the patch was already merged upstream), and || true prevents the build from failing on skipped hunks. The patch lives inside the extracted source tree at rpm/, so no SOURCES directory needed.

Double-apply. The dist-git patch had two commits. The first one (restoring a tmpfiles config) was already merged upstream via PR #134. Applying the full patch would conflict on those already-applied hunks. The --forward flag solved this too - it detected the applied hunks and skipped them, applying only the new ones.

In the end, the reviewers decided to drop the patch from upstream entirely. It's only needed for production RHEL builds, which happen from dist-git where it already lives. The Packit builds are for CI testing - the smoke test runs rpm -q, which doesn't care about issue.d paths.

The final .packit.yaml

After all the iterations, the config ended up clean:

upstream_package_name: console-login-helper-messages
upstream_tag_template: v{version}

downstream_package_name: console-login-helper-messages
specfile_path: rpm/console-login-helper-messages.spec

jobs:
  - job: copr_build
    trigger: pull_request
    targets:
      - fedora-rawhide-x86_64
      - centos-stream-10-x86_64
      - centos-stream-9-x86_64

  - job: tests
    trigger: pull_request
    targets:
      - fedora-rawhide-x86_64
      - centos-stream-10-x86_64
      - centos-stream-9-x86_64
    tmt_plan: /tests/tmt/plans/smoke

  - job: propose_downstream
    trigger: release
    dist_git_branches:
      - fedora-all

Three jobs: build the RPM on three targets, test it on three targets, sync to dist-git on release. Only x86_64 because CLHM is noarch - different architectures would run the exact same test.

Tagging a release

Once the PR was merged and reviewed, it was time to test the full pipeline. CLHM has a RELEASE.txt with the procedure:

LASTTAG='v0.22.0'
VERSION='0.23.0'
NEWTAG="v${VERSION}"

# Generate release notes from git history
git shortlog --no-merges --pretty=format:"%h %s" -e ${LASTTAG}..HEAD

# Create a GPG-signed tag
git tag --file release-notes.txt -s ${NEWTAG}

# Verify the signature
git tag --verify ${NEWTAG}

# Push to upstream (not your fork!)
git push upstream ${NEWTAG}

A few pitfalls I hit: - git push --tags pushes to whatever remote is default, which was my fork. Use git push upstream v0.23.0 explicitly. - You need to git fetch upstream --tags first so the previous tag (v0.22.0) exists locally for the shortlog range. - Heredoc pasting in zsh can break on multiline commands - set variables separately, run git shortlog separately, then create the notes file.

propose_downstream in action

Within minutes of pushing the tag, Packit's propose_downstream kicked in. Three pull requests appeared automatically on src.fedoraproject.org:

• #22 - Update rawhide to upstream release 0.23.0
• #21 - Update f43 to upstream release 0.23.0
• #20 - Update f44 to upstream release 0.23.0

The full pipeline, end to end:

PR merged → release tagged → Packit propose_downstream →
dist-git PRs opened on rawhide, f43, f44

Seeing those PRs appear automatically on the Packit dashboard was the most satisfying moment of the whole process. The spec file I'd been fighting with for days was now being synced to three Fedora branches without any manual work.

What I'd do differently

Start with the in-tree spec from the beginning. The fetch-from-dist-git approach only works if the dist-git spec matches the current source. If there's any drift (and there usually is), you spend more time debugging spec mismatches than you would maintaining an in-tree copy.

Don't add srpm_build_deps for %autochangelog. Packit handles it natively. This one cost me a build cycle on all three targets.

Check if patches have already-applied hunks before adding Patch0:. If upstream has merged any part of a dist-git patch, you'll hit conflicts. Either split the patch or use patch --forward.

Read the build logs carefully. Every error I hit told me exactly what was wrong and where it was looking. The path in the error message (/builddir/build/SOURCES/) told me rpmbuild expected the patch in SOURCES, not in the source tree. The fix followed from understanding the error.

The takeaway

What looked like "add a gating test" turned into a crash course in Fedora packaging, RPM spec files, Packit configuration, and release management. The individual pieces aren't hard - TMT tests are just YAML, Packit config is just YAML, spec files are just macros. The hard part is understanding how they all connect, and finding out which things break only when you try them in a real build environment.

If you're adding Packit to a Fedora package for the first time, I hope this saves you a few build cycles.