windows

CI for Windows installer and macOS bundle generation: KDE Binary Factory

Kevin Funk

• 5 min read

For some time now the KDE community has had a separate Continuous Integration system running which repeatedly generates Windows installers and macOS app bundles (DMG) for a specific subset of KDE projects.

For a starter, all the KDevelop on Windows releases (read: 32-bit and 64-bit NSIS-based installers) are nowadays generated on this CI and the binary blob which falls out of this process is used as official Windows release installer on the KDevelop website.

So, what exactly does KDE's Binary Factory do, and how does it work -- and why was it created to begin with?

Motivation

With the move to KF5 it became easier to get KDE applications running on non-Linux systems. With the very-much needed split of kdelibs into smaller components, it is now easier to pick & choose what to use on these platforms -- so arguably it is easier to cut out unwanted dependencies (think of DBus, KDE service daemons, ...) on these platforms and stop packaging them at all.

Still, the process of getting a reproducible build of KDE project X on either Microsoft Windows and/or macOS has always been a daunting task. Lots of energy went into KDE's Craft (an open source meta build system and package manager, primarily but not only focused on C++ projects) lately which also helped streamlining this process.

It has never been that easy to generate installers or app bundles for any KDE project (where Craft has a recipe for) using Craft. The missing bit was automating all this by using Craft on Continuous Integration system.

Introducing the KDE Binary Factory

KDE's Binary Factory is a Continuous Integration system primarily created for generating Microsoft Windows installers and macOS app bundles. It has no relation to KDE's CI system other than by sharing some of the machines of the CI workers. The jobs on the Binary Factory are mostly generated manually -- adding projects is very easy though.

How does it work?

First of all, the Binary Factory doesn't really know how to build a particular project. The logic for that -- i.e. which dependencies need to be there, which CMake arguments should be used etc. pp. -- is all stored only in Craft blueprints.

The Binary Factory has a set of projects for which Jenkins will trigger jobs on a nightly basis. It does not do much more than calling Craft like this each night:

# exemplary Windows job for kbruch

# rebuild kbruch
python "C:/Packaging/craftroot32/craft/bin/craft.py"  -v --fetch --unpack --compile --install --qmerge  kbruch

# after that, package kbruch (create a Windows installer)
python "C:/Packaging/craftroot32/craft/bin/craft.py"  -v  --package kbruch

If there would be missing dependencies for kbruch, or dependencies out of date, Craft would automatically install or update the depedencies, resp.

After the package has been created, Jenkins is instructed to archive the result. You can easily grab the freshly generated installer on the job page, for instance: https://binary-factory.kde.org/job/kbruch-stable-win32/ which usually shows something alike:

Last Successful Artifacts
	kbruch-17.08.3-windows-msvc2017_32-cl.exe	32.65 MB	view

One click there and the download of the installer starts.

How do I add my pet project?

If you do want your project to be on the Binary Factory, add a recipe (documentation here) and then notify me.

I'd urge you to try building your pet project on Windows yourself first, via Craft, so we don't need to play ping-pong with the CI too much. You (the "project owner") are responsible that the Craft blueprint for your project is up-to-date and works.

Now what to do with the installer/bundle?

The Binary Factory generates Windows installers and macOS bundles at this point. Now what I would not like to see (what already happened) is that individual projects just link to the Binary Factory job pages and tell people 'here, this is the official release of project X'. You shouldn't just link to untested binaries.

Instead I'd like to establish this workflow for project owners:

If there's a release of, say, Kate:

  1. Project owner waits for or triggers a build of Kate on the Binary Factory
  2. Project owner verifies that the installer/bundle works(!)
  3. Project owner uploads the install/bundle on the KDE FTP into the correct project folder
  4. Project owner then adds links to the newly uploaded files on the FTP to the project homepage

Note: Of course linking to the Binary Factory to point users to 'nightly builds of project X' is of course fine -- we do that for KDevelop, too.

Why should I use the Binary Factory?

It has some nice benefits:

  • Automated nightly installer/bundle generation for your project, if set up properly
  • Automated signing of installers/bundles with a KDE-wide code signing certificate
    • Avoids false-positives in AV scanners, warnings by Windows SmartScreen, etc. pp.
  • No need to run a Windows or macOS CI yourself, for package generation
  • Kept up-to-date implicitly via Craft features
    • I.e. right now we're using Qt 5.9.1, KF5 5.37.0
  • It's all there already -- you just need to use it!

Future plans

Start generating macOS bundles

We have a macOS worker set up, but unfortunately the DMG package generation for macOS is still somewhat broken in Craft. We need to sit down and work on this. I don't think there's a lot work left; we can actually create macOS bundles just fine for KDevelop, but there are a few problems with missing shared libraries and/or wrongly set up library metadata (RPATH, etc.).

For the time being, the main focus is on providing Windows installers, but we'll have a look into providing macOS bundles soon-ish now.

Start releasing Windows installers

A couple of KDE projects would be really nice to have on Windows, and actually quite a few installers generated by the Binary Factory are already usable!

One prime example, Filelight:

Setting up a job on the Binary Factory for generating the installer for Filelight literally took me a few hours (for preparing the Craft blueprints, injecting the job in Jenkins -- that's it). Now we just need to link the installer on the official Filelight homepage.

Final words

I'm happy to see this finally working in an automated way. At least for the KDevelop team, the automated installer generation for the KDevelop installers has been a major step into the right direction. After a couple attempts of doing that on personal machines where we literally always managed to break our setup, we now have a clean, automated process of generating them on a isolated machine.

I hope the Binary Factory can be useful to other KDE projects as well. I'm happy to help you guys out to set up jobs on the CI so your personal projects can be built.