Reproducible Builds in November 2024
Welcome to the November 2024 report from the Reproducible Builds project!
Our monthly reports outline what we’ve been up to over the past month and highlight items of news from elsewhere in the world of software supply-chain security where relevant. As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.
Table of contents:
- Reproducible Builds mourns the passing of Lunar
- Introducing reproduce.debian.net
- New landing page design
- SBOMs for Python packages
- Debian updates
- Reproducible builds by default in Maven 4
- PyPI now supports digital attestations
- “Dependency Challenges in OSS Package Registries”
- Zig programming language demonstrated reproducible
- Website updates
- Upstream patches
- Misc development news
- Reproducibility testing framework
Reproducible Builds mourns the passing of Lunar
The Reproducible Builds community sadly announced it has lost its founding member, Lunar. Jérémy Bobbio aka ‘Lunar’ passed away on Friday November 8th in palliative care in Rennes, France.
Lunar was instrumental in starting the Reproducible Builds project in 2013 as a loose initiative within the Debian project. He was the author of our earliest status reports and many of our key tools in use today are based on his design. Lunar’s creativity, insight and kindness were often noted.
You can view our full tribute elsewhere on our website. He will be greatly missed.
Introducing reproduce.debian.net
In happier news, this month saw the introduction of reproduce.debian.net. Announced at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project.
rebuilderd is our server designed monitor the official package repositories of Linux distributions and attempts to reproduce the observed results there.
In November, reproduce.debian.net began rebuilding Debian unstable on the amd64
architecture, but throughout the MiniDebConf, it had attempted to rebuild 66% of the official archive. From this, it could be determined that it is currently possible to bit-for-bit reproduce and corroborate approximately 78% of the actual binaries distributed by Debian — that is, using the .buildinfo
files hosted by Debian itself.
reproduce.debian.net also contains instructions how to setup one’s own rebuilderd instance, and we very much invite everyone with a machine to spare to setup their own version and to share the results. Whilst rebuilderd is still in development, it has been used to reproduce Arch Linux since 2019. We are especially looking for installations targeting Debian architectures other than i386
and amd64
.
New landing page design
As part of a very productive partnership with the Sovereign Tech Fund and Neighbourhoodie, we are pleased to unveil our new homepage/landing page.
We are very happy with our collaboration with both STF and Neighbourhoodie (including many changes not directly related to the website), and look forward to working with them in the future.
SBOMs for Python packages
The Python Software Foundation has announced a new “cross-functional project for SBOMs and Python packages”. Seth Michael Larson writes that the project is “specifically looking to solve these issues”:
- Enable Python users that require SBOM documents (likely due to regulations like CRA or SSDF) to self-serve using existing SBOM generation tools.
- Solve the “phantom dependency” problem, where non-Python software is bundled in Python packages but not recorded in any metadata. This makes the job of software composition analysis (SCA) tools difficult or impossible.
- Make the adoption work by relevant projects such as build backends, auditwheel-esque tools, as minimal as possible. Empower users who are interested in having better SBOM data for the Python projects they are using to be able to contribute engineering time towards that goal.
A GitHub repository for the initiative is available, and there are a number of queries, comments and remarks on Seth’s Discourse forum post.
Debian updates
There was significant development within Debian this month. Firstly, at the recent MiniDebConf in Toulouse, France, Holger Levsen gave a Debian-specific talk on rebuilding packages distributed from ftp.debian.org
— that is to say, how to reproduce the results from the official Debian build servers:
Holger described the talk as follows:
For more than ten years, the Reproducible Builds project has worked towards reproducible builds of many projects, and for ten years now we have build Debian packages twice—with maximal variations applied—to see if they can be build reproducible still.
Since about a month, we’ve also been rebuilding trying to exactly match the builds being distributed via
ftp.debian.org
. This talk will describe the setup and the lessons learned so far, and why the results currently are what they are (spoiler: they are less than 30% reproducible), and what we can do to fix that.
The Debian Project Leader, Andreas Tille, was present at the talk and remarked later in his Bits from the DPL update that:
It might be unfair to single out a specific talk from Toulouse, but I’d like to highlight the one on reproducible builds. Beyond its technical focus, the talk also addressed the recent loss of Lunar, whom we mourn deeply. It served as a tribute to Lunar’s contributions and legacy. Personally, I’ve encountered packages maintained by Lunar and bugs he had filed. I believe that taking over his packages and addressing the bugs he reported is a meaningful way to honor his memory and acknowledge the value of his work.
Holger’s slides and video in .webm
format are available.
Next, rebuilderd is the server to monitor package repositories of Linux distributions and attempt to reproduce the observed results. This month, version 0.21.0 released, most notably with improved support for binNMUs by Jochen Sprickerhof and updating the rebuilderd-debian.sh
integration to the latest debrebuild
version by Holger Levsen. There has also been significant work to get the rebuilderd
package into the Debian archive, in particular, both rust-rebuilderd-common
version 0.20.0-1
and rust-rust-lzma
version 0.6.0-1
were packaged by kpcyrd and uploaded by Holger Levsen.
Related to this, Holger Levsen submitted three additional issues against rebuilderd as well:
rebuildctl
should be more verbose when encountering issues. […]- Please add an option to used randomised queues. […]
- Scheduling and re-scheduling multiple packages at once. […]
… and lastly, Jochen Sprickerhof submitted one an issue requested that rebuilderd downloads the source package in addition to the .buildinfo
file […] and kpcyrd also submitted and fixed an issue surrounding dependencies and clarifying the license […]
Separate to this, back in 2018, Chris Lamb filed a bug report against the sphinx-gallery
package as it generates unreproducible content in various ways. This month, however, Dmitry Shachnev finally closed the bug, listing the multiple sub-issues that were part of the problem and how they were resolved.
Elsewhere, Roland Clobus posted to our mailing list this month, asking for input on a bug in Debian’s ca-certificates-java
package. The issue is that the Java key management tools embed timestamps in its output, and this output ends up in the /etc/ssl/certs/java/cacerts
file on the generated ISO images. A discussion resulted from Roland’s post suggesting some short- and medium-term solutions to the problem.
Holger Levsen uploaded some packages with reproducibility-related changes:
-
devscripts
versions 2.24.3, 2.24.4 and 2.24.5 were uploaded, including several fixes for thedebrebuild
anddebootsnap
and scripts. -
cdbs
version 0.4.167 uploaded in order to dropdh_buildinfo
support, asdpkg
has generated.buildinfo
files since 2016 and the results ofdh_buildinfo
are typically unreproducible. Related to this a mass bug filing by Helmut Grohne intended to remove the obsolete and deprecateddh-buildinfo
package from the archive. At the time of writing, this still affects 311 packages in Debian unstable.
Lastly, 12 reviews of Debian packages were added, 5 were updated and 21 were removed this month adding to our knowledge about identified issues in Debian.
Reproducible builds by default in Maven 4
On our mailing list this month, Hervé Boutemy reported the latest release of Maven (4.0.0-beta-5
) has reproducible builds enabled by default. In his mailing list post, Hervé mentions that “this story started during our Reproducible Builds summit in Hamburg”, where he created the upstream issue that builds on a “multi-year” effort to have Maven builds configured for reproducibility.
PyPI now supports digital attestations
Elsewhere in the Python ecosystem and as reported on LWN and elsewhere, the Python Package Index (PyPI) has announced that it has finalised support for PEP 740 (“Index support for digital attestations”).
Trail of Bits, who performed much of the development work, has an in-depth blog post about the work and its adoption, as well as what is left undone:
One thing is notably missing from all of this work: downstream verification. […]
This isn’t an acceptable end state (cryptographic attestations have defensive properties only insofar as they’re actually verified), so we’re looking into ways to bring verification to individual installing clients. In particular, we’re currently working on a plugin architecture for
pip
that will enable users to load verification logic directly into theirpip install
flows.
There was an in-depth discussion on LWN’s announcement page, as well as on Hacker News.
“Dependency Challenges in OSS Package Registries”
At BENEVOL, the Belgium-Netherlands Software Evolution workshop in Namur, Belgium, Tom Mens and Alexandre Decan presented their paper, “An Overview and Catalogue of Dependency Challenges in Open Source Software Package Registries”.
The abstract of their paper is as follows:
While open-source software has enabled significant levels of reuse to speed up software development, it has also given rise to the dreadful dependency hell that all software practitioners face on a regular basis. This article provides a catalogue of dependency-related challenges that come with relying on OSS packages or libraries. The catalogue is based on the scientific literature on empirical research that has been conducted to understand, quantify and overcome these challenges. […]
A PDF of the paper is available online.
Zig programming language demonstrated reproducible
Motiejus Jakšty posted an interesting and practical blog post on his successful attempt to reproduce the Zig programming language without using the pre-compiled binaries checked into the repository, and despite the circular dependency inherent in its bootstrapping process.
As a summary, Motiejus concludes that:
I can now confidently say (and you can also check, you don’t need to trust me) that there is nothing hiding in
zig1.wasm
[the checked-in binary] that hasn’t been checked-in as a source file.
The full post is full of practical details, and includes a few open questions.
Website updates
Notwithstanding the significant change to the landing page (screenshot above), there were an enormous number of changes made to our website this month. This included:
-
Alex Feyerke and Mariano Giménez:
-
Bernhard M. Wiedemann:
- Update the “System images” page to document the
e2fsprogs
approach. […]
- Update the “System images” page to document the
-
Chris Lamb:
- Cachebust every CSS file per-release. […]
- Replace some inline markdown with HTML. […]
- Use spaces on the “Publications” page. […]
- Add a news article about the passing of Lunar. […][…][…][…]
- Add a black memorial band to the top of the page. […]
-
FC (Fay) Stegerman:
- Replace more inline markdown with HTML on the “Success stories” page. […]
- Add some links, fix some other links and correct some spelling errors on the “Tools” page. […]
-
Holger Levsen:
-
Julia Krüger:
- Add a new “Stripping of unreproducible information page to the documentation. […]
-
Ninette Adhikari & hulkoba:
-
Philip Rinn:
- Import 47 historical weekly reports. […]
-
hulkoba:
- Add
alt
text to almost all images (!). […][…] - Fix a number of links on the “Talks”. […][…]
- Avoid so-called ‘ghost’ buttons by not using
<button>
elements as links, as the affordance of a<button>
implies an action with (potentially) a side effect. […][…] - Center the sponsor logos on the homepage. […]
-
Move publications and generate them instead from a
data.yml
file with an improved layout. […][…] -
Make a large number of small but impactful stylisting changes. […][…][…][…]
- Expand the “Tools” to include a number of missing tools, fix some styling issues and fix a number of stale/broken links. […][…][…][…][…][…]
- Add
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
-
Bernhard M. Wiedemann:
clisp
(fix contributed by Bruno Haible)conky
(date-related issue)emacs-auctex
(date-relatedgzip
issue)javadoc
(filesystem ordering issue)jboss-websocket-1.0-api
(embedsuname -r
)lcms2
(CPU issue)LiE
(ASLR-related issue)make_ext4fs
(toolchain-related issue for for VM images)obs-build
(issue when running builds with certain CPU types or core numbers)perl-Time-modules
(fails to build far in the future)python-bson
(fails to build far in the future)python-exiv2
(fails to build far in the future)python-moto
(date-relatedgzip
issue)python-pyhanko-certvalidator
(fails to build far in the future)python-python-gvm
(concurrency-related issue)python310
(fails to build far in the future)python313
(fails to build far in the future)reproducible-faketools
(toolchain for emacs)shadowsocks-rust
(date-related issue)swipl
(fails to build far in the future)
-
Chris Lamb:
- #1087330 filed against
python-pydash
. - #1087485 filed against
fritzconnection
. - #1087486 filed against
tracy
. - #1088238 filed against
rust-broot
. - #1088353 filed against
python-aiovlc
. - #1088742 filed against
python-aiohomekit
.
- #1087330 filed against
-
James Addison:
Misc development news
-
Bernhard M. Wiedemann published another report for the openSUSE distribution.
-
Martin Abente Lahaye updated diffoscope to fix a crash when
objdump
is missing. […] -
On our mailing list, Jan-Benedict Glaw announced the publication of the fifth NetBSD Reproducibility Report
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In November, a number of changes were made by Holger Levsen, including:
-
reproduce.debian.net-related changes:
- Create and introduce a new reproduce.debian.net service and subdomain […]
- Make a large number of documentation changes relevant to
rebuilderd
. […][…][…][…][…] - Explain a temporary workaround for a specific issue in
rebuilderd
. […] - Setup another
rebuilderd
instance on theo4
node and update installation documentation to match. […][…] - Make a number of helpful/cosmetic changes to the interface, such as clarifying terms and adding links. […][…][…][…][…]
- Deploy configuration to the
/opt
and/var
directories. […][…] - Add an infancy (or ‘alpha’) disclaimer. […][…]
- Add more notes to the temporary
rebuilderd
documentation. […] - Commit an nginx configuration file for reproduce.debian.net’s “Stats” page. […]
- Commit a
rebuilder-worker.conf
configuration for theo5
node. […]
-
Debian-related changes:
-
Misc changes:
- Adapt the
update_jdn.sh
script for new Debian trixie systems. […] - Stop installing the PostgreSQL database engine on the
o4
ando5
nodes. […] - Prevent accidental reboots of the
o4
node because of a long-running job owned byjosch
. […][…]
- Adapt the
In addition, Mattia Rizzolo addressed a number of issues with reproduce.debian.net […][…][…][…]. And lastly, both Holger Levsen […][…][…][…] and Vagrant Cascadian […][…][…][…] performed node maintenance.
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
-
IRC:
#reproducible-builds
onirc.oftc.net
. -
Mastodon: @reproducible_builds@fosstodon.org
-
Mailing list:
rb-general@lists.reproducible-builds.org
-
Twitter: @ReproBuilds