Debian Disguised Work

Dirk Eddelbuettel: RcppArmadillo 15.6.0-1 on CRAN: New Upstream Minor

Tue, 08 Sep 2026 15:31:00 +0000

Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and languageâ€“and is widely used by (currently) 1331 other packages on CRAN, downloaded 48.5 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 727 times according to Google Scholar.

This versions updates to the 15.6.0 upstream Armadillo release made yesterday. It extends solver options for poorly conditioned systems, and brings some updates and extension to the cube data type. For this release, we once again ran the usual complete reverse-dependency check which came back spotless, and did CRAN so no email exchange needed despite nearly 1300 reverse dependencies (but it ended up taking more than a single business day). Still, automation can be helpful when used with a well-maintained software stack. The package has also already been updated for Debian, built for r2u and r-universe, and will build shortly at CRAN for the different binary releases.

All changes since the last CRAN release follow.

Changes in RcppArmadillo version 15.6.0-1 (2026-09-07)

Upgraded to Armadillo release 15.6.0 (Medium Roast Cortado)

Expanded solve() with solve_opts::scale_thresh option to widen detection of poorly conditioned systems

Expanded trans() and .t() to handle cubes

Added permute() to rearrange dimensions of cubes (generalised transpose)

Added cubemul() for batched matrix multiplication of cube slices

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

Enrico Zini: Financial risks in 2026

Tue, 08 Sep 2026 07:05:22 +0000

I asked the banker who is my reference at the bank something like this:

Give that we are talking about the consequences of the tantrum of a fascist foreign government, what happened to them (who are also people close and dear to me), in some future can very well happen to me.

Suddenly my risk profile shot up under the roof.

What do you suggest me to do? Should I find a trusted source of gold bullions to bury under the cellar at home?

The answer was something like this:

Sadly YES, given that the USA have a sort of financial monopoly they can entitle themselves to arbitrarily define a person/organization as a terrorist without any trial or judicial course, and as a consequence apply sanctions that cannot be effectively counteracted, not even abroad.

I didn't have this in my 2026 bingo card, but here we are.

For more details, see:

the A/I press release under "12. Banca Etica: the sanction also affected an Italian bank account"
the public statement from the bank (in Italian)

For some broader context on this kind of actions from the USA, see also:

Bits from Debian: New Debian Developers and Maintainers (July and August 2026)

Mon, 07 Sep 2026 21:00:00 +0000

The following contributor got their Debian Developer account in the last two months:

Nicolas Peugnet (nicolasp)

The following contributors were added as Debian Maintainers in the last two months:

Antoine Lassagne
Ivan Hu
Jesse Rhodes
Haolin Xue
Léo Haf
Rony João de Sousa
Luke Yasuda
Darshaka Pathirana

Congratulations!

Colin Watson: Free software activity in August 2026

Mon, 07 Sep 2026 19:25:04 +0000

My Debian contributions this month were all sponsored by Freexian.

You can also support my work directly via Liberapay or GitHub Sponsors.

Personal note

This month, my Dad unexpectedly passed away after a short illness. As a result I obviously got less work done than usual, and I still have a lot to take care of (since I’m the executor of his will, as well as helping with funeral arrangements) while grieving and generally having less focus and energy. Having routine work to do is one of the ways I cope with this sort of thing, but all the same, I hope people will bear with me and maybe remind me if I seem to be dropping the ball on something you especially need.

LLM vote

[Content note: strong opinions.]

I voted in General Resolution: LLM usage in Debian. My vote was pretty much the opposite of what ended up winning, so I’m quite disappointed. My personal opinion is that LLMs are cognitive hazards to their users that impose ecological costs far out of proportion to their utility at a time when the world absolutely cannot afford them. When the impossible economics of the large commercial models are finally allowed to catch up with reality, I expect there to be significant macroeconomic consequences, and that people who have become dependent on them will have problems; and who knows what the copyright situation on their output really is. I’m not convinced that local models are better enough on these axes to be worth the costs.

Debian’s direct contribution to all that will be negligible on a global scale, and even the most radical proposals in the GR didn’t expect that we could do much about upstreams that have gone all-in on LLMs. Even so, I’d hoped that my fellow developers might be more willing to lean on our position in the free software ecosystem to make at least a moderately radical statement. Instead, we’ve at best presented an undistinguished fence-sitting position to the world, and further entrenched the idea that humans can reliably do a good job of reviewing the output of tools that are designed to produce output plausible to humans. I certainly don’t trust my own code review skills that far.

Since I’ve never voluntarily used an LLM (not counting LLMs being foisted on me by things like search results, support chatbots, or incoming pull requests, regardless of whether I asked for them), and don’t intend to for the foreseeable future, I doubt this will change much for me in terms of the way I work. The winning option is a very weak one that imposes no new requirements on developers, which means that it also does nothing to stop me continuing to reject LLM-generated material from Debian bug reports and merge requests in my areas of responsibility. I know this probably won’t do much to satisfy people who have decided that Debian is slop now, but it’s the best I can do.

OpenSSH

I finally landed the GSS-API key exchange package split in our OpenSSH packaging. Here’s the NEWS entry:

openssh (1:10.4p1-5) unstable; urgency=medium

  The openssh-client and openssh-server packages no longer include GSS-API
  authentication and key exchange support; this adds pre-authentication
  attack surface and generally increases complexity, and should only be used
  where specifically needed.  Users who need these features should install
  openssh-client-gssapi or openssh-server-gssapi instead.

 -- Colin Watson <cjwatson@debian.org>  Sun, 23 Aug 2026 17:39:55 +0100

I fixed a flaky autopkgtest.

I upgraded from 10.4p1 to 10.5p1, which was a good test of keeping openssh and the new openssh-gssapi source package in sync.

PuTTY

I upgraded from 0.84 to 0.85.

Python packaging

New upstream versions:

anorack
djangorestframework (fixing a build failure)
isort
lazr.delegates (dropping pkg_resources)
lazr.restfulclient (dropping pkg_resources)
lazr.uri (dropping pkg_resources)
nbclient (fixing a build failure)
pydantic-settings
python-argcomplete
python-blockbuster
python-makefun (fixing a build failure)
python-persistent
python-pytest-run-parallel
python-time-machine
python-typing-inspection
wcwidth
zope.hookable
zope.i18nmessageid
zope.interface
zope.proxy
zope.security
zope.testrunner

The version treadmill continues: we’ve just finished dropping Python 3.13 as a supported version, so now we’ve started working on enabling Python 3.15 as a supported version. Maximiliano Curia has been very helpfully driving this. I didn’t get as much done here as I’d have liked (see the top of this post), but I fixed a couple of packages:

Other build/test failures:

I fixed some other bugs:

bugs.debian.org

I deployed the fix for Invalid link rel=”canonical” on bugs.debian.org. In the process I found a few bugs in recent undeployed code and fixed them.

Vincent Bernat: Sidenotes with CSS anchor positioning

Mon, 07 Sep 2026 06:49:58 +0000

I am a heavy user of sidenotes:¹ they keep optional content next to the text instead of sending the reader to the bottom of the page and back. Tufte CSS renders them without JavaScript but only accepts inline content. CSS anchor positioning, now supported by recent browsers,² is an elegant alternative. Sidenotes can hold several blocks, still without JavaScript, and fall back below the paragraph referencing them on narrow viewports and older browsers.

Inâ€¯2023, Eric Meyer demonstrated this technique in â€œNuclear Anchored Sidenotes.â€� The main improvement over other solutions is that the notes can sit anywhere in the HTML document. You can place them after the paragraph referencing them, as regular block elements for text browsers, screen readers, feed readers, and reader mode to render them properly:

Rendering in Lynx, a text browser

When the viewport is too narrow or the browser does not support CSS anchor positioning, you can style them so the reader can skip them or glance at them without losing their position in the text:

Rendering below the paragraph on a narrow viewport

Once the viewport is large enough, they appear in the margin, at the same vertical position as the matching reference mark, unless they would collide with a previous sidenote, as in the example below:³

Rendering in the margin on a large viewport

The gist of CSS anchoring is to position an element relative to another elementâ€”the anchor. For the sidenotes, the anchor is the reference mark. I use the following markup, with a data attribute to specify the anchor name:

<sup id="fnref:YYY" data-anchor="--lf-sn-YYY">
  <a href="#sidenote-YYY">1</a>
</sup>

The matching note is an <aside> element carrying the same data attribute for the anchor name. We put it after the paragraph holding the reference mark:

<aside role="note" id="sidenote-YYY" data-anchor="--lf-sn-YYY">
  <sup>1</sup>
  <p>A first paragraph.</p>
  <p>A second paragraph.</p>
</aside>

On a narrow viewport or when the browser is too old for CSS anchoring, we style the sidenote, which stays below its paragraph, with a muted color:

aside[role="note"] {
  margin-block: 1rlh;
  color: #444;
}

On a wide viewport and when the browser is recent enough, we move the sidenote to the right margin:

@supports (anchor-name: attr(data-anchor type(<custom-ident>))) {
  @media (min-width: 72rem) {
    main {
      position: relative;
      sup[data-anchor] {
        anchor-name: attr(data-anchor type(<custom-ident>));
        /* â†’ anchor-name: --lf-sn-YYY */
      }
      aside[role="note"][data-anchor] {
        anchor-name: --lf-sidenote;
        position: absolute;
        position-anchor: attr(data-anchor type(<custom-ident>));
        /* â†’ position-anchor: --lf-sn-YYY */
        top: max(anchor(top), anchor(--lf-sidenote bottom, -1rlh) + 1rlh);
        left: 100%;
        margin: 0 2rem;
        width: 18rem;
        color: inherit;
      }
    }
  }
}

attr() extracts the anchor name for the reference mark from the data-anchor attribute. It returns a string, unless we specify a CSS unit or a type, like here: the browser parses the data attribute as a custom identifier, which anchor-name validates as a dashed identifier, a custom identifier starting with two dashes.⁴

The note itself is absolutely positioned past the right edge of the main block. It selects the matching reference mark as its anchor with position-anchor set to the value of the data-anchor attribute. Each note is also an anchor named --lf-sidenote. We use it to keep the next note from colliding with this one.

The anchor() CSS function lets us position the noteâ€™s top edge relative to its anchor: anchor(top) aligns the top edge of the note with the top edge of the reference mark. It can also take another anchor as a parameter: anchor(--lf-sidenote bottom) would align the top edge of the note with the bottom edge of the closest preceding anchor named --lf-sidenoteâ€”so the previous note.⁵ Like attr(), anchor() accepts a fallback value as its second parameter and use it when the named anchor does not exist.

The top property handles three cases, illustrated in the following diagram:

The three cases for the vertical position of a note

The first noteâ€™s top edge aligns with the top edge of its reference mark: as there is no previous note, anchor(--lf-sidenote bottom, -1rlh) + 1rlh resolves to 0 and max() returns anchor(top).
When the reference mark of a later note sits above the bottom of the previous note, plus some vertical space, the note goes below the previous one to avoid a collision. max() returns anchor(--lf-sidenote bottom) + 1rlh.
Otherwise, the noteâ€™s top edge aligns with the reference markâ€™s top edge, as max() returns anchor(top).

Have a look at the complete stylesheet, which also adapts the reference mark to the location of the note: a â€œâ†“â€� arrow when the note sits below the paragraph, a â€œâ†’â€� arrow when it moves to the margin. Gwernâ€™s â€œSidenotes In Web Designâ€� lists more implementations and their trade-offs.

Some bloggers aim to write a post in 30 minutes. I planned to publish three web-related articles this weekend. Instead, I spent an inordinate amount of time elsewhere: about 15 commits on the build system, a pull request to update CSS highlighting for nested selectors in Pygments, and a small correction to MDNâ€™s article on the anchor() CSS function. The SVG illustration took a bit less than an hour and the article itself a handful of hours. The attr() function came in after I thought â€œinline style looks ugly, isnâ€™t there a better way?â€� But, hey, I still think this is worth it! ğŸ�¨

My PhD advisor told me this is unwise.Â â†©
The first bits of anchor positioning are supported from Chromeâ€¯125 (Mayâ€¯2024), Firefoxâ€¯147 (Januaryâ€¯2026), and Safariâ€¯26 (Septemberâ€¯2025).

Before Safariâ€¯26.5, sidenotes may collide due to a bug in how dependency chains are handled. You can detect this situation with some JavaScript. It is, however, not needed in the solution described here as we depend on a more recent feature.Â â†©
If you noticed the runt in the first note, I share your pain and lament that Firefox does not implement text-wrap: pretty.Â â†©
Typed attr() is supported from Chromeâ€¯133 (Februaryâ€¯2025), Firefoxâ€¯155 (Septemberâ€¯2026), and Safariâ€¯27 (not yet released). Check Una Kravetsâ€™ article for details. To support more browsers, you can inline the anchor name and the position anchor directly in the HTML:
```
<sup id="â€¦" style="anchor-name: --lf-sn-â€¦">
  <a href="#sidenote-â€¦">1</a>
</sup>
```
â€œManaging Anchor Associations With Data Attributes and Advanced attr(),â€� by Daniel Schwarz, explores CSS anchors and typed attr() in more detail.Â â†©
The exact rule for the target anchor element is more complex: â€œif an ancestor of [the note] satisfies the following conditions, return the nearest such element to [the note]. Otherwise, return the last element in tree order that satisfies the conditions.â€� One of these conditions is that â€œ[the candidate] is an acceptable anchor element for [the note],â€� which requires that â€œ[the candidate] is laid out strictly before [the note],â€� where the relevant clause is that â€œ[the candidate] is either not absolutely positioned or occurs earlier in the flat tree order than [the note].â€�Â â†©

Iustin Pop: AI agents aha moment

Sun, 06 Sep 2026 13:29:50 +0000

Looking at the reactions to the Debian AI vote, I think some people still think the clock can be turned back, as if that ever worked in history. Rather than cry about spilled milk, I prefer to find a path forward in the new world. There are many ways to use LLMs, some of them are straightforward, others not so much.

One of the “not so clear” areas for me is the focus on agentic workloads. For complex tasks, sure, you want something that can work in the background, but in general, why does every single tool go the agentic way? I much prefer the “chat/ask” approach, or even the “code” one, but if I’m at the keyboard, why would I send a task to an agent, and see it work, instead of directly implementing it?

And then, this past Friday, I finally understood one part of that. I was in the airport, sitting at the gate and waiting to board a flight, and because I arrived much earlier at the airport (fearing crowds due to Labour Day weekend), I got one hour of work before boarding started. As the time for boarding approached, I did one more commit after making sure tests pass, pushed, closed laptop, and went to walk a bit before getting on the plane.

As I was getting up, I get a phone notification from GitHub that the CI run failed. I was quite surprised, as the local tests passed, so I open the notification, and realize that tests via make test vs CI (which additionally uses --pedantic) had slightly different settings, and of course I missed a build warning (which in CI is an error).

I thought I’d fix that on the plane, but then I saw a “Copilot agent” button in the mobile app. I was curious what it did, I click it, and I see Copilot starting a draft pull request, and saying:

Thanks for asking me to work on this. I will get started on it and keep this PR’s description up to date as I form a plan and make progress.

Fix the failing GitHub Actions job. Analyze the Actions logs, identify the root cause of the failure, and implement a fix.

Then it goes, finds the failure, writes the fix, and tries to run the tests. Well, it can’t do it (it runs in a restricted container, so no network, so stack install couldn’t actually work). The agent sees that, acknowledges it has no way to validate the fix, but the error message was clear enough that it was confident the fix is mostly correct, so it sends the pull request.

I allow full CI to run on the pull request, and go buy a bottle of water. After that, I check and see that the CI failed again, as not one but two test files were broken, and I didn’t have --keep-going, so the build stopped at the first failure. I write a comment in the pull request, no reaction, I realize I need to tag Copilot explicitly, I do that, and it starts another investigation.

I’m waiting now in the boarding queue, with phone in hand, while Copilot is fixing my bug. While I scan my boarding pass and walk towards the plane, the pull request is updated, I trigger another CI, it passes, and I merge it.

And then, it hit me. Agents allow me to make progress while being “not at keyboard”, whether that’s physically “not at keyboard”, or while working on something else. Fixing a simple test failure is not something that needs human attention per se, whereas improving the test layout might be.

In that airport, using otherwise-unusable downtime, and without explicitly intending to, I made progress in understanding a different way to use AI. Now I have three ways to work with LLMs: ask (tutor mode), code (implement my request), and agent (fix simple or complex problems, autonomously). I still don’t know about “plan” mode and really complex tasks, like asking it to implement features from scratch. That will probably be the next area to tackle.

And today (Sunday), while waiting for a running race to start, I opened GitHub, and asked Copilot to increase test coverage for a simple module. It did, and yes it still can’t run tests (I learned in the meantime that you can configure the environment in which the agent runs, nice), but after two back-and-forth messages, I have a pull request ready to review. All in the 20 minutes before a race, where I could either browse social media or actually do some meaningful work.

Checking now my GitHub billing, it looks like all of this Copilot use only cost $1.92. Yes, that is under two dollars! And while it did use compute resources, the person across the aisle who watched TikTok or Instagram for half an hour while waiting for takeoff also consumed a lot of compute, and so do the gazillion cat videos uploaded to YouTube every day.

To me, this is another tool in the toolbox, that might one day replace me (as it did to the 19th-century textile workers), or make me five times more productive — we’ll see where we end up. In the meantime, I can move faster, and make better use of my limited free time.

Enjoy the ride!

Dirk Eddelbuettel: RcppFarmHash 0.0.4 on CRAN: Maintenance

Sun, 06 Sep 2026 13:29:00 +0000

Another minor maintenance release of the RcppFarmHash package is now on CRAN as version 0.0.4.

RcppFarmHash wraps the Google FarmHash family of hash functions (written by Geoff Pike and contributors) that are used for example by Google BigQuery for the FARM_FINGERPRINT digest.

This releases updates several of package internal files for continuous intergration and package data.

The brief NEWS entry follows:

Changes in version 0.0.4 (2026-09-06)

Minor updates to continuous integration, README.md and DESCRIPTION

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Russell Coker: CoMaps

Sun, 06 Sep 2026 11:05:50 +0000

I have just tried CoMaps, a free mapping program released under the Apache license [1]. I have tried it on Android on a Pixel 6a but it also runs on Linux so I’ll try it on a PinePhone or similar at some convenient time. On Android it is in the F-Droid repository among others and for Linux there’s a Flatpak package.

The data it uses is from Open Street Map project [2] which has extensive and accurate coverage of every place I’ve looked at (Australia and a few other first-world countries). The first thing it does after being installed is start downloading the world data set from Open Street Map and prompt to download the data for the detected region (Melbourne in my case).

The UI is decent and allows most of the features that I am used to using in Google Maps. The quality of directions seems good, I’ve only tested it with one journey so far which was a 50 minute drive across the city and it gave a set of directions that Google Maps often gives.

It gives spoken directions which is an important feature but sometimes the way the directions are presented is confusing. When turning off a freeway it didn’t give a spoken direction to do that, it gave a direction to “turn right” which was AFTER leaving the freeway, fortunately the map was clearly displayed.

In terms of use practices of this program the main difference I recommend is checking which off ramp to use from a freeway before entering the freeway. With Google Maps you can rely on it giving clear directions in that case.

I recommend this program without reservation. It can do everything that Google Maps does apart from detecting traffic jams because there’s no way of detecting traffic without spying on users. It is designed to preserve user privacy and works well in that regard.

[1] https://www.comaps.app/
[2] https://www.openstreetmap.org/

Enrico Zini: Migrating away from .org/.net/.com domains

Sun, 06 Sep 2026 09:40:04 +0000

After having witnessed how easy it is for good people to lose a .org domain over a fascist tantrum (you can follow the Autistici/Inventati story here and here), I've started moving all my infrastructure to differently managed TLDs.

enricozini.org and enricozini.com will keep being functional for the time being, as dropping a domain makes it available for squatting and impersonation.

These new domains are now online, with working web and emails:

enricozini.eu
ezini.eu
ezini.it (probably equally subject to fascist tantrums these days)

It will take ages to migrate countless accounts that are tied to my primary email address, so better start early.

Waiting to see what will happen with .meow domains, which I supported despite not identifying as a cat.

Steinar H. Gunderson: plocate 1.1.25 released

Sun, 06 Sep 2026 08:02:00 +0000

I've released version 1.1.25 of plocate. This time around, there's two security issues of unknown severity; if you chain them with other bugs, they could lead to being able to list files (but of course not their contents) that you should not normally be able to see. So an update is probably in order; you can never be too safe these days.

The full changelog is:

plocate 1.1.25, September 6th, 2026

  - Fix two early-exit bugs with multiple databases.
    Reported by Manpreet Singh and Tyler Spivey.

  - Drop setgid properly, including the saved gid.
    Reported by Michal Sekletar, found with the help of Claude Opus 4.6.

  - Fix a potential symlink-checking race in updatedb.
    Reported by Michal Sekletar, found with the help of Claude Opus 4.6.

As usual, you can get it from the home page, or it's on the way up in Debian unstable.

Michael Stapelberg: Debian Code Search: Fast TurboPFor with Go SIMD

Sun, 06 Sep 2026 07:00:00 +0000

This August, I accomplished what I wanted for many years: I deleted the last cgo dependency in Debian Code Search! This was made possible by Goâ€™s recently introduced SIMD support, because now we can implement the TurboPFor integer compression format as efficiently â€” more efficiently, in fact, by using the newer AVX512 instruction set! â€” as the reference implementation.

Background: Why does DCS need a fast Integer Codec?

Debian Code Search (DCS) is a search engine that allows searching all the Open Source source code within Debian, with either literal search expressions or regular expression search queries.

A search engine uses an inverted index: a map from term to documents containing the term. Each document is typically represented most efficiently by using an id, so the index consists of many lists of document ids.

When searching, it is important to quickly decode these lists to answer the search query. However, there is a point of diminishing returns where the decoding speed, even though it can still be measurably improved quite a bit, no longer influences the overall query duration.

From 2012 (its inception) to 2019, Debian Code Search used to use a small index format, and queries were fast because the index was kept entirely in RAM. In 2019, I implemented the new index format, which adds an on-disk positional index. For literal queries (78.2% of DCS queries), querying the positional index on disk is faster than querying the non-positional index in RAM.

The efficient encoding of the TurboPFor format makes it possible to fit such an index on a mid-sized Hetzner server, which I rent with two 1 TB SSD disks. The optimized decoder of the C TurboPFor library is what made decoding fast at query time.

If you want to dive deeper into the algorithm, see this blog post from February 2019:

TurboPFor: an analysis

Motivation I have recently been looking into speeding up Debian Code Search. As a quick reminder, search engines answer queries by consulting an inverted index: a map from term to documents containing that term (called a â€œposting listâ€�). See the Debian Code Search Bachelor Thesis (PDF) for a lot more details. Read more â†’

If you want to learn more about the positional index, see this blog post from September 2019:

Debian Code Search: positional index, TurboPFor-compressed

Over the last few months, I have been developing a new index format for Debian Code Search. This required a lot of careful refactoring, re-implementation, debug tool creation and debugging. Read more â†’

SIMD in Go

For many years, you had the following options for using SIMD instructions in Go:

Hand-writing Go assembler code. This is only doable for small functions, for example bytes.IndexByte is implemented with hand-written Go assembly (including AVX2).
Generating Go assembler code with tools like Michael McLoughlinâ€™s â€œAvoâ€�. This is how crypto/internal/fips140/sha256 uses AVX2. While Avo generator code definitely is higher-level than hand-written assembly, it is still too close to assembly for my taste.
Use a C library via cgo so gcc or clang compiles SIMD code. Debian Code Search used to use the powturbo/TurboPFor C library via cgo for the last 7 years.

The C TurboPFor library has served us well, but Debian Code Search was always intended to be a project using Go, so I would prefer it if I did not have any C code in the project.

Go 1.26 (released in February 2026) introduced the simd/archsimd package:

Go 1.26 introduces a new experimental simd/archsimd package, which can be enabled by setting the environment variable GOEXPERIMENT=simd at build time. This package provides access to architecture-specific SIMD operations. It is currently available on the amd64 architecture and supports 128-bit, 256-bit, and 512-bit vector types, such as Int8x16 and Float64x8, with operations such as Int8x16.Add. The API is not yet considered stable.

â€” Go 1.26 Release Notes

For my 2019 TurboPFor analysis, I implemented goturbopfor, a native Go teaching decoder (without any SIMD), because I find Go code easier to follow than C code, especially optimized C code. My implementation was intentionally not optimized so that the code was easier to study.

The TurboPFor format/algorithm has a vector-optimized part: bitpacking comes in a scalar variant (bitunpack32) and a vector variant (bitunpack256v32), where the vector variant is used for full blocks (256 values) and the scalar variant is used for remainder blocks (< 256 values).

When Go 1.26 was released, I used Claude Code to explore whether my native Go decoderâ€™s bitunpack256v32 function (for the vertical vector layout) could be implemented using Go SIMD, and the answer was yes, it was possible and it was faster than without SIMD, but not quite at the level of C TurboPFor. If you let Claude Code try for long enough, it eventually finds enough optimizations (about 10) to match C performance.

I donâ€™t want to vibe-code Debian Code Search, though, so I figured I would find some time to review the SIMD code at some point and see if I could implement something similar myself.

Before I found enough time and motivation to complete said review, I discovered that to not regress real-life query performance by more than 10 to 100 milliseconds (which seems acceptable), I donâ€™t actually need to add SIMD code to my teaching decoder at all; it would be sufficient to reduce allocations in my teaching decoder and specialize it per bit width.

Encouraged by the possibility of using the optimized native Go decoder in Debian Code Search, I explored whether I could also implement a native Go encoder so that I could get rid of the C TurboPFor dependency entirely. The answer is yes, it is doable in a few days, and it isnâ€™t even that much slower: Go is at 76% of C, see Debian/dcs commit e920dc7.

The goal I set myself at that point was to see if I could learn enough SIMD to optimize the native Go encoder such that its performance would match how DCS uses C TurboPFor (via cgo).

Beating C TurboPFor was possible in 2-3 commits (SIMD and bit width specialization). To my surprise, Claude Fable 5 pointed out that the encoderâ€™s block scanning could be done more efficiently using a technique called positional popcount, and that is another 2x speed-up! ğŸ˜²

To be clear: I am not saying the Go compiler beats C here. Certainly, the C compiler can also produce fast AVX512 code and can be used to implement positional popcount. When comparing apples to apples, i.e. backporting the AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a little slower at â‰ˆ1.4x C.

This spectacular result (much faster than what DCS had before) got me curious how far I could push the decoder with SIMD after all. I ended up matching/exceeding the cgo version here, too!

The rest of this article explains a few classes of optimizations I encountered along the way.

Starting Point

When I wrote my goturbopfor teaching decoder, I named its functions to match the upstream C TurboPFor library, but now I want to get away from names like p4ndec256v32 â€” they make sense from the TurboPFor perspective, but for Debian Code Search, we can use cleaner names.

Before writing any code, I audited how DCS uses integer compression / decompression.

API design: BlockEncoder, BlockDecoder and streaming

In Debian Code Search, we have the following usage patterns:

Partial Indexing: When a new package (or package version) enters Debian, all of its (text) files are indexed. If the hello-2.12.3-1 package (hypothetically) contained only hello.c with printf("hello!\n");, we would assign document ID 1 to hello.c and store in the partial index that trigrams pri, rin, int, ntf, etc. are all found in doc 1 (hello.c).
Full Index Merging: The many thousands of partial index files (for each Debian package) are combined into a small handful of large index files: When searching, it would be expensive to consult thousands of indexes. To merge multiple partial index files into one larger index (which can then be efficiently queried), we need to re-encode the partial index files: what used to be document ID 1 in the partial index might be document ID 2531 in the full index.
Querying (searching): When users enter search queries, these queries need to be answered as quickly as possible. The relevant entries in the full indexes are decoded (in parallel).

For reading the index, we do keep the decoded uint32s fully in memory, so we only need DecodeN(input []byte, output []uint32) (read int), a function that reads len(output) values (uint32) from input and returns how many bytes it consumed.

For writing the index (both in partial indexing, and when merging), keeping the entire index in memory is prohibitively expensive, so we need a streaming API, for decoding and for encoding.

Ultimately, I converged on the following API:

package pforenc

type BlockEncoder struct {
    // scratch buffers can go here
}

// EncodeBlock encodes len(vals)<=256 uint32s into dest (one TurboPFor block).
func (*BlockEncoder) EncodeBlock(dest []byte, vals []uint32) []byte {}

// EncodeN calls EncodeBlock in a loop.
func (*BlockEncoder) EncodeN(dest []byte, vals []uint32) []byte {}

type StreamEncoder struct {
  be   BlockEncoder
  vals [256]uint32
  // scratch buffers
}

// if full, you need to call [EncodeBlock]
func (*StreamEncoder) Add(val uint32) (full bool)

// EncodeBlock must be called after all data was [Add]ed.
//
// Write the returned buffer to file or send it over the network;
// it is only valid until the next [EncodeBlock] call.
func (*StreamEncoder) EncodeBlock() []byte {
  if se.n == 0 { return nil } // turn an extra EncodeBlock into a no-op
  // â€¦
}

This API (the decoder works similarly) allows us to process data in TurboPFor format without any memory allocations. The types are not safe for concurrent use by multiple goroutines. The zero value is ready to be used. For the streaming API, the result only stays valid until the next call.

Initial Implementation

Before we can optimize anything, we need a working decoder and encoder. The decoder already exists: my goturbopfor teaching decoder. Next up, I needed an encoder.

Writing a TurboPFor encoder has a delightfully simple starting point: You can encode all values at bit width 32, in little endian, at which point you only need to add a one-byte TurboPFor block header every 256 values and youâ€™re done:

func (be *BlockEncoder) EncodeN(dest []byte, vals []uint32) []byte {
  for len(vals) > 0 {
    chunk := min(len(vals), 256)
    dest = be.EncodeBlock(dest, vals[:chunk])
    vals = vals[chunk:]
  }
  return dest
}

func (be *BlockEncoder) EncodeBlock(dest []byte, vals []uint32) []byte {
  const bitWidth = 32
  dest = append(dest, bitWidth)
  for _, val := range vals {
    dest = binary.LittleEndian.AppendUint32(dest, val)
  }
  return dest
}

Of course, this is a terribly inefficient compressor, so after the first commit, the real work starts: implement each block type until the compression matches the original C TurboPFor implementation (same output file size), or in other words: do the reverse of the decoder.

The TurboPFor bitpacking block type (bitpacking implementation commit) encodes a bit stream of variable bit width (where the bit width is in range 0 â‰¤ bitWidth â‰¤ 32) in little endian byte order. By scanning all values and choosing the smallest bit width that allows representing all values, this technique saves disk space (compresses).
The bitpacking with exceptions block type (bitpacking with exceptions implementation commit) determines two bit widths: one for values, the other bit width for encoding exceptions. This allows choosing a lower bit width (that does not cover all values) compared to the bitpacking block type. A bitmap encodes whether a value has an exception or not.
The bitpacking with VB exceptions block type (bitpacking with VB exceptions implementation commit) is a variant which does not use an exception bitmap and encodes exceptions using a variable byte integer encoding. This is more efficient when there are few exceptions (less than 20) or the exceptions are very different in bit width compared to the other values.
Lastly, the constant block type (constant implementation commit) stores just one value on disk. This is useful for all-zero or all-one blocks, for example.

I found it interesting to realize that the main work of the encoder is to scan the input values and choose the optimal block type, whereas the actual encoding itself is cheap in comparison.

At this point, we can look at performance and see that the Go encoder is at 76% of the C encoder.

In all honesty, I could have probably stopped here, but now that the milestone of a viable replacement was reached, I got curious to see how far it would be possible to push the encoder (how much work to reach C speeds?) and afterwards, the decoder, too.

Setup

The microarchitecture level: set `GOAMD64`

The microarchitecture of a CPU determines which instructions it provides, and that includes not just SIMD instruction sets (like AVX2), but also other useful instructions like LZCNT (Leading Zero Count), which can be used to implement math/bits.Len32 more efficiently, which the TurboPFor encoder needs to call on every input value to determine the ideal bit width.

Letâ€™s walk through how to set the microarchitecture level when using Go on 64-bit x86 (x86-64).

Go uses the GOARCH environment variable to configure the target compilation architecture, and I am using the value amd64 to select 64-bit x86 (AVX2 and AVX512 are instruction sets found on x86-64 CPUs). With GOARCH=amd64, the architecture-specific variable GOAMD64 configures the microarchitecture level for which to compile and Go 1.18 introduced these 4 different levels:

GOAMD64=v1 (default): The baseline.
Exclusively generates instructions that all 64-bit x86 processors can execute.

GOAMD64=v2: all v1 instructions,
plus CMPXCHG16B, LAHF, SAHF, POPCNT, SSE3, SSE4.1, SSE4.2, SSSE3.

GOAMD64=v3: all v2 instructions,
plus AVX, AVX2, BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, OSXSAVE.

GOAMD64=v4: all v3 instructions,
plus AVX512F, AVX512BW, AVX512CD, AVX512DQ, AVX512VL.

In 2026, I generally recommend compiling with GOAMD64=v3 so that functions like bits.OnesCount8 are compiled into intrinsics (POPCNT) instead of using a lookup table.

For Intel CPUs, setting GOAMD64=v3 means your programs will only start on Haswell CPUs (2013) or newer; for AMD CPUs that means Zen 1 (2017) or newer.

In this specific case (DCS), I am even compiling with GOAMD64=v4. The v4 microarchitecture level requires AVX512, which means AMD Zen 4, Zen 5 or newer (Intelâ€™s story isâ€¦ complicated). Luckily, both my main development PC (Zen 5) and the Debian Code Search server (Zen 4) are recent enough. Setting GOAMD64=v4 has little effect on Go 1.27 itself: the only change is that maps use one less instruction (VPBROADCASTB instead of PSHUFB). But compiling with GOAMD64=v4 allows us to move one more feature check from runtime to compile time, see SIMD build tags.

It makes sense to set the microarchitecture level in your benchmark setup so that you donâ€™t measure the slow fallback implementations. I use export GOAMD64=v4 in my Makefile.

Benchmarking setup

Goâ€™s built-in testing package contains support for benchmarks which are written in functions of the form func BenchmarkXxx(b *testing.B). The simplest way to run such benchmarks is go test -bench=., but I ended up configuring a few convenience make targets, which write results to bench.txt and compare against baseline.txt (the previous commitâ€™s results, usually), using the very useful benchstat tool.

GOTEST=go test

# -count=6 gives pâ‰¤0.002 in benchstat:
# https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
BENCHFLAGS=-run=^$$ -bench=. -benchtime=200000x -count=6

# use taskset -c1 to always pin to the same single core,
# avoiding accidental scheduling on different cores on
# mixed-core CPUs like the Ryzen 9 9950X3D.
TASKSET=taskset -c 1
BENCH=$(TASKSET) $(GOTEST) $(BENCHFLAGS)

.PHONY: all test bench bench-baseline bench-relative

all: test

bench: test
	$(BENCH) | tee bench.txt
# Compares compression ratio between C and Go implementation
	benchstat -col /impl -row '/n /vals' -filter '-/impl:go-stream .unit:(encoded-bytes)' bench.txt
# Compares performance between C (cgo) and Go implementation
	benchstat -col /impl -row '/n /vals' -filter '.unit:(Mval/s)' bench.txt

bench-baseline: test
	$(BENCH) | tee baseline.txt

bench-relative: test
	$(BENCH) | tee bench.txt
	benchstat -filter '-/impl:go-stream .unit:(encoded-bytes)' baseline.txt bench.txt
	benchstat -filter '/impl:go .unit:(Mval/s)' baseline.txt bench.txt

The encoded-bytes and Mval/s units are custom metrics I am reporting from the various sub-benchmarks, which are arranged such that I can filter / report them with benchstat.

The main encoder (and decoder) benchmarks compare 3 different implementations (cgo, Go, Go with the StreamEncoder API) with a number of benchmark cases that are designed to cover the different block types and contain a similar mix of values as what we see in Debian Code Search:

// reportMetrics adds Mval/s and encoded-bytes metrics to all benchmarks.
func reportMetrics(b *testing.B, n int, nencoded int) {
   b.ReportMetric(float64(nencoded), "encoded-bytes")
   b.ReportMetric(float64(b.N*n)/1e6/b.Elapsed().Seconds(), "Mval/s")
}

// BenchmarkEncode/n=<N>/vals=<testcase>/impl=<c|go|go-stream>
//
// e.g. BenchmarkEncode/n=2048/vals=one-constant/impl=go-stream
func BenchmarkEncode(b *testing.B) {
   for _, tc := range allBenchCases() {
     n := len(tc.vals)
     b.Run(fmt.Sprintf("n=%d/vals=%s", n, tc.name), func(b *testing.B) {
       b.Run("impl=c", func(b *testing.B) {
         b.ReportAllocs()
         var encoded []byte
         buf := make([]byte, turbopfor.EncodingSize(n))
         for b.Loop() {
           encoded = turbopfor.P4nenc256v32Buf(buf, tc.vals)
         }
         reportMetrics(b, n, len(encoded))
       })
       b.Run("impl=go", func(b *testing.B) {
         b.ReportAllocs()
         var be BlockEncoder
         var encoded []byte
         buf := make([]byte, 0, turbopfor.EncodingSize(n))
         for b.Loop() {
           encoded = be.EncodeN(buf, tc.vals)
         }
         reportMetrics(b, n, len(encoded))
       })
       b.Run("impl=go-stream", func(b *testing.B) {
         b.ReportAllocs()
         var se StreamEncoder
         var encoded int
         for b.Loop() {
           encoded = 0
           for _, val := range tc.vals {
             if se.Add(val) {
               encoded += len(se.EncodeBlock())
             }
           }
           encoded += len(se.EncodeBlock())
         }
         reportMetrics(b, n, encoded)
       })
     })
   }
}

CPU counters: perf

Go has included excellent performance tooling for many years, see the â€œProfiling Go Programsâ€� blog post (2011) for an example of how to use pprof, a sampling profiler. This profiler can help track down which part of a program runs slow, or where memory allocations happen.

Once you identified the slow part of a program, how do you know why itâ€™s slow?

To learn more about the specific bottlenecks your program encounters, you can consult your CPUâ€™s hardware performance counters. For example, you could check the branch predictor counters to see if your program is slow due to a high number of branch mispredicts.

On Linux, the perf tool is the best way to access the CPU hardware performance counters. A good starting point for working with perf is the documentation on â€œTop-down analysis with the perf toolâ€�, which describes the optimization method that Intel established.

In my Makefile, I set up two perf targets:

# GOTEST and TASKSET like shown in the earlier benchmarking setup section:
GOTEST=go test -pgo=encode.cpuprof
TASKSET=taskset -c 1
PERFBENCHFLAGS=-test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=200000x

# Use perf(1) to capture AMD IBS (the equivalent to Intel PEBS)
# PipelineL1 is roughly equivalent to Intel TopdownL1
perf:
	$(GOTEST) -c
	$(TASKSET) perf stat -M PipelineL1 ./pforenc.test -test.run=^$$ $(PERFBENCHFLAGS)
	sudo perf record -F 4999 -e ibs_op// --call-graph fp ./pforenc.test -test.run=^$$ $(PERFBENCHFLAGS)
	sudo chmod 644 perf.data

# 488281 iterations Ã— 2048 values = 1.000e9 values, so counter/1e9 = per value.
perf-per-value:
	$(GOTEST) -c
	$(TASKSET) perf stat -x, -e cycles:u,instructions:u,branches:u,branch-misses:u ./pforenc.test -test.run=^$$ -test.bench='Encode/n=2048/vals=debian-mix/impl=go$$' -test.benchtime=488281x 2>&1 >/dev/null | awk -F, '{printf "%-16s %6.2f /val\n", $$3, $$1/1e9}'

The perf-per-value numbers are high level numbers that indicate how much work the implementation is doing. Reducing the number usually increases speed.

To see the counters for each instruction (and source code lines), I use make perf, followed by perf report. A quick shortcut is perf annotate, which directly shows the hottest function.

Optimizations (scalar)

Letâ€™s first see how far we can get without reaching for SIMD instructions.

(The examples are not necessarily in commit order, but cherry-picked for clarity.)

Profile-Guided Optimization (PGO)

PGO stands for Profile-Guided Optimization and is a feature that Go introduced as a preview in Go 1.20 (released in February 2023) and shipped as ready for general production use in Go 1.21 (released in August 2023).

The idea is to capture a CPU profile that records where your program spends most of its CPU time, which you then provide to the Go compiler to give it more data to make better decisions.

Most importantly, this way the Go compiler can inline functions much more aggressively than its usual heuristics allow, which does have a measurably positive effect in my series of optimization commits. Another optimization that a PGO profile allows the compiler to do is conditional devirtualization â€” but our TurboPFor code does not use any interfaces.

My strategy is to enable PGO before doing any other optimizations, so that we have the full inlining budget available that PGO gives us, and can measure the effect of other commits clearly.

Surprisingly, turning on PGO actually decreases our performance (-13% geomean), but a closer investigation reveals that we just got unlucky. Let me explain.

Aside from inlining and conditional devirtualization, PGO also influences alignment: The Go compiler sets PCALIGNMAX(64, 31) on the first block of a loop (the â€œloop bodyâ€�) for all loops in hot functions (per the PGO profile), i.e. Go will insert up to 31 bytes of padding to make the block land on a 64-byte boundary. Documentation like AMDâ€™s â€œSoftware Optimization Guide for the AMD Zen5 Microarchitectureâ€� (2024, #58455) explicitly recommends aligning hot loops that way:

[â€¦] for hot loops, some further knowledge of trade-offs can be helpful. Because the processor can read an aligned 64-byte fetch block every cycle, it is suggested to either align the start of the loop to the beginning of a 64-byte cache line [â€¦]

Indeed, when compiling with -gcflags=all=-d=alignhot=0 to disable the alignment, performance remains as good as without PGO. How can the padding hurt more than help? The answer is: Itâ€™s not the padding itself! Itâ€™s a side-effect of the padding moving instructions to different addresses.

In the unlucky arrangement, a macro-fused CMPQ+JGE instruction pair now ends up exactly on a 32-byte boundary. However, the Go compiler ensures fused branch sequences must never cross or end at a 32-byte boundary to fix Intel erratum SKX102 (discussion: Go issue #35881) by inserting NOPs.

This NOP padding, unlike the loop alignment padding, is not free; these extra instructions slow down our otherwise dispatch-bound loops.

Because the commits after the PGO enabling commit change the code, this unlucky situation is avoided for the rest of the optimization series (by chance).

Reducing memory allocations

Memory allocations are quite expensive, at least in comparison to encoding/decoding integers, so I followed my usual strategy of first reducing memory allocations as much as possible.

In my goturbopfor teaching decoder, whenever the code needed a scratch buffer, it would allocate it right then and there with make():

// p4dec32 decodes one block of TurboPFor-encoded 32 bit ints
func (d *decoder) p4dec32(input []byte, output []uint32) (read int) {
    // â€¦
  switch blockType {
  case blockBitpackingExceptions:
    bx, input := input[0], input[1:]
    n := len(output)

    exmap := input
    nex := 0 // number of exceptions
    for i := 0; i < n; i++ {
      if exmap[i/8]&(1<<uint(i%8)) != 0 {
        nex++
      }
    }
    input = input[(n+7)/8:]

    exceptions := make([]uint32, nex)
    input = input[bitunpack32(input, exceptions, bx):]
    input = input[d.bitunpack(input, output, b):]

    for i := 0; i < n; i++ {
      if exmap[i/8]&(1<<uint(i%8)) != 0 {
        output[i] += exceptions[0] << b
        exceptions = exceptions[1:]
      }
    }

    return before - len(input)
  }
}

The Go compiler can turn make(T, n) calls into stack allocations, if n is known at compile-time. But, in this case nex is not known at compile-time. We can verify that Go calls into the runtime (runtime.makeslice) by dumping the object code (assembly) with source annotated (-S):

% cd ~/go/src/github.com/stapelberg/goturbopfor
% git reset --hard 49b7c05cc61e77f0257568eb73833467714d2b4a
% go test -c  # go1.27.0
% go tool objdump -S goturbopfor.test | perl -nlE 'say if /p4dec32/ .. /^$/'
TEXT github.com/stapelberg/goturbopfor.(*decoder).p4dec32(SB) /home/michael/go/src/github.com/stapelberg/goturbopfor/goturbopfor.go
func (d *decoder) p4dec32(input []byte, output []uint32) (read int) {
  0x549f60		4c8da42460ffffff	LEAQ 0xffffff60(SP), R12
  0x549f68		4d3b6610		CMPQ R12, 0x10(R14)
  0x549f6c		0f86d9070000		JBE 0x54a74b
  0x549f72		55			PUSHQ BP
  0x549f73		4889e5			MOVQ SP, BP
  0x549f76		4881ec18010000		SUBQ $0x118, SP
  0x549f7d		48899c2430010000	MOVQ BX, 0x130(SP)
  0x549f85		4889b42448010000	MOVQ SI, 0x148(SP)
	if len(output) == 0 {
  0x549f8d		4d85c0			TESTQ R8, R8
  0x549f90		0f84a7030000		JE 0x54a33d
  0x549f96		660f1f840000000000	NOPW 0(AX)(AX*1)
  0x549f9f		90			NOPL
[â€¦]
		exceptions := make([]uint32, nex)
  0x54a4be		488d057bec1700		LEAQ 0x17ec7b(IP), AX
  0x54a4c5		4c89fb			MOVQ R15, BX
  0x54a4c8		4889d9			MOVQ BX, CX
  0x54a4cb		e8f0ddf3ff		CALL runtime.makeslice(SB)
[â€¦]

An easy speed-up was to avoid allocations through reuse (in goturbopfor). In the DCS pfordec package (with the improved API design), I ended up with a vals [256]uint32 field in the StreamDecoder type, which brings us from 773 Mval/s to 858 Mval/s on the debian-mix:

% benchstat -filter '/impl:go /vals:debian-mix .unit:(Mval/s)' \
  baseline.txt bench.txt
goos: linux
goarch: amd64
pkg: github.com/Debian/dcs/internal/turbopfor/pfordec
cpu: AMD Ryzen 9 9950X3D 16-Core Processor
           â”‚ baseline.txt â”‚             bench.txt              â”‚
           â”‚    Mval/s    â”‚   Mval/s     vs base               â”‚
n=2048        1.089k Â± 1%   1.175k Â± 0%   +7.85% (p=0.002 n=6)
n=2039         974.7 Â± 0%   1046.0 Â± 0%   +7.32% (p=0.002 n=6)
n=160          434.9 Â± 1%    513.6 Â± 5%  +18.11% (p=0.002 n=6)
geomean        772.9         857.7       +10.98%

Aside from the speed-up, avoiding memory allocations is generally nice in benchmarks because it removes the garbage collector from the equation and makes it less likely that your benchmarks get other processes OOM-killed on the same machine.

Generics for bit width specialization

In general, we want to make it easy for the compiler to understand as much as possible about our algorithm. Consider this bitpack implementation:

func bitpack(dest []byte, vals []uint32, bitWidth int) []byte {
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have int
  for _, val := range vals {
    acc |= uint64(val&mask) << have
    have += bitWidth
    for have >= 32 {
      dest = binary.LittleEndian.AppendUint32(dest, uint32(acc))
      acc >>= 32
      have -= 32
    }
  }
  for have > 0 {
    dest = append(dest, byte(acc))
    acc >>= 8
    have -= 8
  }
  return dest
}

Letâ€™s think through what determines the iterations and control flow this function uses:

The number of input values (vals), but not their actual value.
The bit width to pack into (bitWidth).

With a bit of careful rearrangement, we can provide the compiler with both, a fixed number of input values (say, 32), and a bit width, both known at compile time. Why is this worthwhile? Because we can manually unroll the loop, let the compiler eliminate much of the repetition and get much faster compiled code as a result!

Letâ€™s first fix the number of input values to 32 and rewrite the loop to calculate the position offsets within dest instead of changing dest on each value (with AppendUint32):

func bitpack32Unrolled(dest []byte, vals *[32]uint32, bitWidth int) {
  // only one bounds check for 32 values
  dest = dest[: 4*bitWidth : 4*bitWidth]
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have, pos int
  // Manually unrolled loop starts here.
  // Each iteration is identical except for the vals[x] index.
  acc |= uint64(vals[0]&mask) << have
  have += bitWidth
  if have >= 32 {
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
  }

  // vals[1] .. vals[30] elided for brevity

  // Each loop iteration is 8 lines of Go code, so for 32 input values,
  // bitpack32Unrolled contains 8*32 = 256 lines of code.

  acc |= uint64(vals[31]&mask) << have
  have += bitWidth
  if have >= 32 {
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
  }

  // have == 0; for all bitWidths
}

Next, we want to specialize not just for 32 input values, but also for each of the 32 bit widths.

Can we do better than hand-copying bitpack32Unrolled 32 times (= 8192 lines of Go code)?

Yes, we can use Go generics to help us with the code generation!

In Go, array types like [4]byte (not slices like []byte!) contain the length of the array as part of their type, meaning [1]byte (an array of length 1) is a different type than [2]byte.

Instead of passing the bit width as a function parameter, we can declare 32 different types (one for each bit width) and recover the bit width (at compile time!) from the type system:

type bitWidthT interface {
  [1]byte | [2]byte | [3]byte | [4]byte | [5]byte |
  [6]byte | [7]byte | [8]byte | [9]byte | [10]byte |
  [11]byte | [12]byte | [13]byte | [14]byte | [15]byte |
  [16]byte | [17]byte | [18]byte | [19]byte | [20]byte |
  [21]byte | [22]byte | [23]byte | [24]byte | [25]byte |
  [26]byte | [27]byte | [28]byte | [29]byte | [30]byte |
  [31]byte | [32]byte
}

func bitpack32Unrolled[T bitWidthT](dest []byte, vals *[32]uint32) {
  var zero T
  bitWidth := len(zero)                  // known at compile time
  dest = dest[: 4*bitWidth : 4*bitWidth] // make cap known at compile time
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have, pos int
  // Manually unrolled loop starts here.
  // Each iteration is identical except for the vals[x] index.
  acc |= uint64(vals[0]&mask) << have
  have += bitWidth
  if have >= 32 {
    binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
    pos += 4
    acc >>= 32
    have -= 32
  }

  // vals[1] .. vals[31] elided for brevity
}

When we instantiate bitpack32Unrolled[bitWidthT] with all 32 different types ([1]byte, [2]byte, â€¦, [32]byte), the compiler substitutes the bitWidthT type parameter and produces 32 copies of the function, which we can find in our compiled executable with names like github.com/Debian/dcs/internal/turbopfor/pforenc.bitpack32Unrolled[go.shape.[12]uint8]. The â€œshapeâ€� of a generic type is based on its memory layout, so a shape for [1]byte must be different than the shape for [2]byte.

Because the bitWidth is now known at compile time, the Go compiler can generate close to the optimal machine code for each bit width, which we can confirm using go tool objdump.

The code is branchless (after the one bounds check per 32 values) and aside from the loads and stores (from/to memory) consists only of shifts and bit operations, all with constant operands:

% go test -c && go tool objdump -S pforenc.test
[â€¦]
TEXT github.com/Debian/dcs/internal/turbopfor/pforenc.bitpack32Unrolled[go.shape.[28]uint8](SB) /home/michael/dcs/internal/turbopfor/pforenc/bitpackunroll.go
func bitpack32Unrolled[T bitWidthT](dest []byte, vals *[32]uint32) {
  0x660580              55                      PUSHQ BP
  0x660581              4889e5                  MOVQ SP, BP
  0x660584              48895c2418              MOVQ BX, 0x18(SP)
        dest = dest[: 4*bitWidth : 4*bitWidth] // make cap known at compile time
  0x660589              4883ff70                CMPQ DI, $0x70
  0x66058d              0f820b030000            JB 0x66089e
        acc |= uint64(vals[0]&mask) << have
  0x660593              8b06                    MOVL 0(SI), AX
  0x660595              25ffffff0f              ANDL $0xfffffff, AX
        acc |= uint64(vals[1]&mask) << have
  0x66059a              8b4e04                  MOVL 0x4(SI), CX
  0x66059d              81e1ffffff0f            ANDL $0xfffffff, CX
  0x6605a3              48c1e11c                SHLQ $0x1c, CX
  0x6605a7              4809c8                  ORQ CX, AX
                acc >>= 32
  0x6605aa              4889c1                  MOVQ AX, CX
  0x6605ad              48c1e820                SHRQ $0x20, AX
                binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
  0x6605b1              90                      NOPL
        b[0] = byte(v)
  0x6605b2              890b                    MOVL CX, 0(BX)
        acc |= uint64(vals[2]&mask) << have
  0x6605b4              8b4e08                  MOVL 0x8(SI), CX
  0x6605b7              81e1ffffff0f            ANDL $0xfffffff, CX
  0x6605bd              48c1e118                SHLQ $0x18, CX
  0x6605c1              4809c1                  ORQ AX, CX
                acc >>= 32
  0x6605c4              4889c8                  MOVQ CX, AX
  0x6605c7              48c1e920                SHRQ $0x20, CX
                binary.LittleEndian.PutUint32(dest[pos:pos+4], uint32(acc))
  0x6605cb              90                      NOPL
        b[0] = byte(v)
  0x6605cc              894304                  MOVL AX, 0x4(BX)

Now we need to actually call bitpack32 from the general bitpack function:

func bitpack(dest []byte, vals []uint32, bitWidth int) []byte {
  if bitWidth == 0 {
    return dest // no payload, sparse block with only exceptions
  }
  if len(vals) >= 32 {
    size := 4 * bitWidth
    for len(vals) >= 32 {
      existing := len(dest)
      dest = slices.Grow(dest, size)[:existing+size]
      bitpack32(dest[existing:] /*append*/, (*[32]uint32)(vals), bitWidth)
      vals = vals[32:]
    }
  }
  mask := uint32(1<<bitWidth - 1)
  var acc uint64
  var have int
  for _, val := range vals {
    acc |= uint64(val&mask) << have
    have += bitWidth
    for have >= 32 {
      dest = binary.LittleEndian.AppendUint32(dest, uint32(acc))
      acc >>= 32
      have -= 32
    }
  }
  for have > 0 {
    dest = append(dest, byte(acc))
    acc >>= 8
    have -= 8
  }
  return dest
}

func bitpack32(dest []byte, vals *[32]uint32, bitWidth int) {
  switch bitWidth {
  case 1: bitpack32Unrolled[[1]byte](dest, vals)
  case 2: bitpack32Unrolled[[2]byte](dest, vals)
  case 3: bitpack32Unrolled[[3]byte](dest, vals)
  case 4: bitpack32Unrolled[[4]byte](dest, vals)
  case 5: bitpack32Unrolled[[5]byte](dest, vals)
  case 6: bitpack32Unrolled[[6]byte](dest, vals)
  case 7: bitpack32Unrolled[[7]byte](dest, vals)
  case 8: bitpack32Unrolled[[8]byte](dest, vals)
  case 9: bitpack32Unrolled[[9]byte](dest, vals)
  case 10: bitpack32Unrolled[[10]byte](dest, vals)
  case 11: bitpack32Unrolled[[11]byte](dest, vals)
  case 12: bitpack32Unrolled[[12]byte](dest, vals)
  case 13: bitpack32Unrolled[[13]byte](dest, vals)
  case 14: bitpack32Unrolled[[14]byte](dest, vals)
  case 15: bitpack32Unrolled[[15]byte](dest, vals)
  case 16: bitpack32Unrolled[[16]byte](dest, vals)
  case 17: bitpack32Unrolled[[17]byte](dest, vals)
  case 18: bitpack32Unrolled[[18]byte](dest, vals)
  case 19: bitpack32Unrolled[[19]byte](dest, vals)
  case 20: bitpack32Unrolled[[20]byte](dest, vals)
  case 21: bitpack32Unrolled[[21]byte](dest, vals)
  case 22: bitpack32Unrolled[[22]byte](dest, vals)
  case 23: bitpack32Unrolled[[23]byte](dest, vals)
  case 24: bitpack32Unrolled[[24]byte](dest, vals)
  case 25: bitpack32Unrolled[[25]byte](dest, vals)
  case 26: bitpack32Unrolled[[26]byte](dest, vals)
  case 27: bitpack32Unrolled[[27]byte](dest, vals)
  case 28: bitpack32Unrolled[[28]byte](dest, vals)
  case 29: bitpack32Unrolled[[29]byte](dest, vals)
  case 30: bitpack32Unrolled[[30]byte](dest, vals)
  case 31: bitpack32Unrolled[[31]byte](dest, vals)
  case 32: bitpack32Unrolled[[32]byte](dest, vals)
  }
}

Encoding remainder blocks is quite a bit faster (full blocks use the vertical layout anyway):

% benchstat -filter '/impl:go /n:160 .unit:(Mval/s)' baseline.txt bench.txt
goos: linux
goarch: amd64
pkg: github.com/Debian/dcs/internal/turbopfor/pforenc
cpu: AMD Ryzen 9 9950X3D 16-Core Processor
                         â”‚ baseline.txt â”‚             bench.txt              â”‚
                         â”‚    Mval/s    â”‚   Mval/s     vs base               â”‚
vals=bitpacking-bw1          751.2 Â± 3%   1120.5 Â± 0%  +49.15% (p=0.002 n=6)
vals=bitpacking-bw2          716.8 Â± 2%   1176.0 Â± 0%  +64.07% (p=0.002 n=6)
vals=bitpacking-bw7          700.0 Â± 1%   1078.5 Â± 0%  +54.08% (p=0.002 n=6)
vals=bitpacking-bw1-exc      524.8 Â± 1%    736.8 Â± 0%  +40.40% (p=0.002 n=6)
vals=bitpacking-bw2-exc      543.7 Â± 1%    758.2 Â± 0%  +39.46% (p=0.002 n=6)
vals=bitpacking-bw7-exc      566.7 Â± 1%    787.7 Â± 0%  +38.99% (p=0.002 n=6)
vals=bitpacking-vb-exc       442.6 Â± 1%    616.5 Â± 0%  +39.29% (p=0.002 n=6)
vals=sparse-exc              532.4 Â± 0%    787.8 Â± 0%  +47.97% (p=0.002 n=6)
vals=sparse-vb-exc           408.9 Â± 1%    597.8 Â± 0%  +46.20% (p=0.002 n=6)
vals=debian-mix              559.5 Â± 0%    783.8 Â± 9%  +40.09% (p=0.002 n=6)

This performance win comes at the cost of binary size increase. In this case, the .text section (executable code) grows by about 20 KB and the .gopclntab section grows by another 26 KB. Definitely a price I am very willing to pay, but the case might not be as clear in all circumstances.

Optimization: Bigger strides with SIMD

Even without reaching for SIMD instructions, a TurboPFor implementation can be made faster by making it work bigger strides. Take this code from the goturbopfor teaching decoder which counts the number of exceptions by checking if each valueâ€™s bit is set in the exception bitmap:

case blockBitpackingExceptions:
  bx, input := input[0], input[1:]
  n := len(output)

  exmap, input := input, input[(n+7)/8:]
  nex := 0 // number of exceptions
  for i := range n {
    if exmap[i/8]&(1<<uint(i%8)) != 0 {
      nex++
    }
  }
  exceptions := d.scratch[:nex]

We can use the bits.OnesCount64 functions to count ones bits in the exception bitmap, 64 values at a time. For remainder blocks, the rest is processed 8 values (1 byte) at a time:

i := 0
for ; i+8 <= n/8; i += 8 {
  xm8 := binary.LittleEndian.Uint64(exmap[i:])
  nex += bits.OnesCount64(xm8)
}
for ; i < (n+7)/8; i++ {
  xmb := exmap[i]
  // Clear the bits which do not belong to the exception map:
  if rem := n - i*8; rem < 8 {
    xmb &= 1<<rem - 1
  }
  // Go compiles OnesCount32 into an intrinsic,
  // but not OnesCount8, so we convert to uint32:
  nex += bits.OnesCount32(uint32(xmb))
}

OnesCount64 uses a 64-bit register. For comparison, AVX2 SIMD instructions use 256-bit registers (= 8 uint32) and AVX512 SIMD instructions use 512-bit registers.

In the following sections, we will first set up our build tags for conditional compilation to use a trivial SIMD instruction, then walk through an AVX2 and AVX512 SIMD kernel.

SIMD build tags

Letâ€™s assume we have the following scalar code:

constant.go:

package pfordec

func fillConstant(output []uint32, val uint32) {
  for i := range output {
    output[i] = val
  }
}

To increase throughput, we can use AVX2 instructions if they are available on the CPU on which the program runs, i.e. using runtime dispatch. Weâ€™ll first rename fillConstant to fillConstantScalar (itâ€™s now the fallback path):

constant.go:

package pfordec

func fillConstantScalar(output []uint32, val uint32) {
  for i := range output {
    output[i] = val
  }
}

Next, weâ€™ll supply two different implementations (constant_nosimd.go and constant_amd64.go), the latter of which is selected when compiling for GOARCH=amd64 with GOEXPERIMENT=simd (the latter will hopefully be dropped in a later version of Go). The nosimd variant just dispatches to the fillConstantScalar, which will likely be inlined:

//go:build !goexperiment.simd || !amd64

package pfordec

func fillConstant(output []uint32, val uint32) {
  fillConstantScalar(output, val)
}

The constant_amd64.go variant assigns the hasAVX2 global variable by doing a CPUID check and then jumps to the scalar fallback if !hasAVX2, i.e. the CPU is too old:

//go:build goexperiment.simd && amd64

package pfordec

import "simd/archsimd"

var hasAVX2 = archsimd.X86.AVX2()

func fillConstant(output []uint32, val uint32) {
  if !hasAVX2 {
    fillConstantScalar(output, val)
    return
  }
  val8 := archsimd.BroadcastUint32x8(val)
  i := 0
  for ; i+8 <= len(output); i += 8 {
    val8.StoreArray((*[8]uint32)(output[i : i+8]))
  }
  // use the scalar implementation for the last <= 7 elements
  fillConstantScalar(output[i:], val)
}

We can go one step further by conditionally compiling const hasAVX2 = true when GOAMD64 is set to v3 or higher (i.e. the amd64.v3 build tag is set). As a practical example from Debian Code Search, we currently need the following checks / dispatches:

code	function	vector instruction set	GOAMD64
encoder	bitpack256v	AVX2	GOAMD64=v3
encoder	exbitmap	AVX512	GOAMD64=v4
encoder	scan	AVX512+VBMI+GFNI+BITALG	n/a
decoder	bitunpack	AVX2	GOAMD64=v3
decoder	bitunpack256v32	AVX2	GOAMD64=v3
decoder	bitunpack256v32Ex	AVX512	GOAMD64=v4

In DCS, the effect is measurably positive, but small.

The 256 uint32 vertical layout

First, here is the layout explanation from my 2019 TurboPFor analysis blog post:

In regular (non-SIMD) bitpacking, integers are stored on disk one after the other, padded to a full byte, as a byte is the smallest addressable unit when reading data from disk. For example, if you bitpack only one 3 bit int, you will end up with 5 bits of padding.

SIMD bitpacking works like regular bitpacking, but processes 8 uint32 little-endian values at the same time, leveraging the AVX instruction set. The following illustration shows the order in which 3-bit integers are decoded from disk:

The scalar implementation uses an array of 8 uint64 to process 8 values at a time:

func bitunpack256v32(input []byte, dest []uint32, bitWidth int) (read int) {
  mask := uint64(1)<<bitWidth - 1
  orig := len(input)
  var bits uint
  var acc [8]uint64 // accumulator: current+next bits
  for op := 0; op < len(dest); {
    if bits < uint(bitWidth) {
      // read 8 more uint32s
      for i := range 8 {
        acc[i] |= uint64(binary.LittleEndian.Uint32(input)) << bits
        input = input[4:]
      }
      bits += 32
    }
    for i := range 8 {
      dest[op] = uint32(acc[i] & mask)
      op++
      acc[i] >>= bitWidth
    }
    bits -= uint(bitWidth)
  }
  return orig - len(input)
}

The SIMD version also processes 8 values, but without a for i := range 8 loop!

One difference is that we no longer have the luxury of using uint64 for acc (holding rest and current bits); because AVX2 registers only fit 8 uint32 (not 8 uint64). Instead, we split acc into rest8 and cur8.

func bitunpack256v32(fullinput []byte, fulldest []uint32, bitWidth int) (read int) {
  dest := fulldest[:256]
  if bitWidth == 0 {
    clear(dest)
    return 0
  }
  n := 32 * int(bitWidth)
  input := fullinput[:n] // tell the Go compiler how long the input is
  mask8 := archsimd.BroadcastUint32x8(uint32(1)<<bitWidth - 1)
  bitWidth8 := archsimd.BroadcastUint32x8(uint32(bitWidth))
  var bits uint
  pos := 0
  // var acc [8]uint64
  var rest8 archsimd.Uint32x8
  var cur8 archsimd.Uint32x8
  for op := 0; op < 256; op += 8 {
    if bits < uint(bitWidth) {
      // read 8 more uint32s
      // acc[i] |= uint64(binary.LittleEndian.Uint32(input)) << bits
      next := archsimd.LoadUint8x32(input[pos : pos+32]).ReshapeToUint32s()
      pos += 32  // input = input[4:]
      cur8 = rest8.Or(next.ShiftAllLeft(uint64(bits)))
      // acc[i] >>= bitWidth
      rest8 = next.ShiftAllRight(uint64(uint(bitWidth) - bits))
      bits += 32
    } else {
      cur8 = rest8
      // acc[i] >>= bitWidth
      rest8 = rest8.ShiftRight(bitWidth8)
    }
    // dest[op] = uint32(acc[i] & mask)
    cur8.And(mask8).Store(dest[op : op+8])
    bits -= uint(bitWidth)
  }
  return n
}

The SIMD version benchmarks about 3x as fast as the scalar version.

Another significant speedup is to use generics for bit width specialization for this SIMD kernel so that bitWidth becomes a compile-time constant and the compiler can generate better code.

Positional Popcount

For my TurboPFor encoder, I implemented the same techniques as described above:

These changes are sufficient to roughly match the cgo performance, but then Claude Fable 5 found another 2x speed-up on top of that!

The key observation is that once encoding blocks is fast, the preceding step of scanning the input values to decide which block type to use becomes the bottleneck. Here is the encoderâ€™s main encode function, which first does one pass over the input values (scan) and then prices all different block types at all relevant bit widths (requires fast access to the scan histogram):

func (be *BlockEncoder) encode(dest []byte, vals []uint32, layout blockLayout) []byte {
  var stats stats
  scan(&stats, vals) // gathers statistics from every value in vals
  bitWidth := bits.Len32(stats.or)
  if stats.or == stats.and {
    return be.encodeConstant(dest, vals, bitWidth)
  }
  n := len(vals)
  // bitpacking is the default, unless we find a more efficient block type.
  bestType := blockBitpacking
  bestB := bitWidth
  best := priceBitpack(n, bitWidth, layout)

  // Walk from high bitWidths to low: to break ties, we prefer
  // the encoding with fewer exceptions (for faster decoding).
  for b := bitWidth - 1; b >= 0; b-- { // up to 32 iterations
    nex := int(stats.cnt[b])
    size := priceBitpackExceptions(n, b, bitWidth, nex, layout)
    if size < best {
      bestType = blockBitpackingExceptions
      bestB = b
      best = size
    }
    // Over-approximate the number of VB bytes.
    vb := nex + // exceptions using 1, 2, 3, 4, or 5 VB bytes
      int(stats.cnt[b+7]+ // exceptions using 2, 3, 4, or 5 VB bytes
        stats.cnt[b+14]+ // exceptions using 3, 4, or 5 VB bytes
        stats.cnt[b+19]+ // exceptions using 4 or 5 VB bytes
        stats.cnt[b+24]) // exceptions using 5 VB bytes
    size = headerBytes + headerExBytes + payloadBytes(n, b, layout) + vb + nex
    if size < best {
      bestType = blockBitpackingVBExceptions
      bestB = b
      best = size
    }
  }
  switch bestType {
  case blockBitpacking:
    return be.encodeBitpack(dest, vals, layout, bitWidth)
  case blockBitpackingExceptions:
    return be.encodeBitpackExc(dest, vals, layout, bestB, bitWidth-bestB)
  case blockBitpackingVBExceptions:
    return be.encodeBitpackVBExc(dest, vals, layout, bestB, int(stats.cnt[bestB]))
  default:
    panic("BUG: bestType not implemented")
  }
}

Iâ€™ll show you a slightly shortened version of scan, the function which is the bottleneck:

type stats struct {
  // cnt[n] = how many values where bits.Len32(val)>n,
  // i.e. how many exceptions are required for bitWidth=n.
  // Padded so that cnt[b+24] is always in bounds.
  cnt [32 + 24]uint32
}

func scan(output *stats, vals []uint32) {
  for _, val := range vals {
    for b := range bits.Len32(val) {
      output.cnt[b]++ // b bits are not enough to store val
    }
  }
}

Letâ€™s consider the following 3 example values to understand the resulting cnt:

input	input (bin)	`bits.Len32`
23	`0b0000010111`	5
5	`0b0000000101`	3
666	`0b1010011010`	10

The resulting cnt exception count histogram would contain (cnt shortened to c):

`c[0]`	`c[1]`	`c[2]`	`c[3]`	`c[4]`	`c[5]`	`c[6]`	`c[7]`	`c[8]`	`c[9]`	`c[10]`
3	3	3	2	2	1	1	1	1	1	0

In words, this means that at bit width 10, we could encode all the values without any exceptions.

But most values do not need 10 bits, so a bit width of 5 would be more efficient, but requires storing one exception. Encoding at bit width 4 requires 2 exceptions, and so on.

The scan function above is intentionally kept simple for illustration. We can make it faster by moving the per-bit-width loop outside the per-element loop. The fast version still needs about 12 instructions per value. With SIMD, we can reduce this to by 8x to only 1.5 instructions per value!

The trick: smear masks enable positional popcount

The trick is to turn each input value into its â€œsmear maskâ€� (imagine taking the first 1 bit and smearing it across the remaining positions). Here are the smear masks for our example:

input	input (bin)	`bits.Len32`	â€œsmear maskâ€�
23	`0b0000010111`	5	`0b0000011111`
5	`0b0000000101`	3	`0b0000000111`
666	`0b1010011010`	10	`0b1111111111`

Turning a value into its smear mask is computationally cheap: Go implements BitLen(x) (functions like bits.Len32) by calculating 32 - LZCNT(x). We can calculate the â€œsmear maskâ€� of a value with ^uint32(0) >> LZCNT(x), i.e. starting with a 32-one-bits mask and shifting it by the number of leading zeros.

Now, to obtain e.g. cnt[4], we can count the 1 bits at bit position 4 of all input values.

The POPCNT instruction counts bits very efficiently, but it counts one bits within a register, so it counts rows, not columns. Counting columns is called Positional Population Count.

I found the following papers that describe positional popcount with SIMD:

2019: â€œEfficient Computation of Positional Population Counts Using SIMD Instructionsâ€� (Klarqvist, MuÅ‚a, Lemire) introduces an AVX512 implementation using Carry-Save Adder (CSA) networks, but in my testing, that approach is slower for TurboPFor encoding.
2024: â€œHistogramming bytes with positional popcount (GF2P8AFFINEQB edition)â€� (Harold Aptroot) shared a C++ implementation illustrating the technique I am using. If you are not familiar with how C++ vector intrinsics look, take a look at this example.
2025: â€œFaster Positional-Population Counts for AVX2, AVX-512, and ASIMDâ€� (Clausecker, Lemire, Schintke) improves performance over the 2019 paper, but cites the Aptroot blog post as â€œFuture Workâ€�, promising even faster processing for smaller inputs.

Positional Popcount: a visual explanation

To understand the AVX512 implementation of positional popcount, I found it most helpful to visualize an AVX512 register (512 bits, i.e. 64 bytes). The graphic below uses the Uint64x8 layout, meaning it divides the register into 8 lanes of 64 bits (= 8 bytes) each.

This illustration shows the whole process: how uint32s are loaded into an AVX512 register (all 4 of its bytes, in sequence) and where we end up, i.e. the 32 positional popcounts:

Letâ€™s break down this process into its individual steps.

First, we turn each loaded value into its smear mask as explained above.

The VPOPCNTB vector instruction calculates POPCNT (1 byte) of 64 bytes at once, but first we need to shuffle the bytes inside the register: in load order, we have a full uint32 (4 bytes), followed by another uint32, per lane. First, we permute the bytes (VPERMB) such that all the first bytes of each value end up in one lane (â€œtranspose the bytesâ€�):

Next, we â€œtranspose the bitsâ€� using the GF2P8AFFINEQB instruction, which sounds scary but turns out to be quite flexible for bit manipulation of all kinds. The GF2P8AFFINEQB instruction is also â€œthe star of the showâ€� in Goâ€™s Green Tea Garbage Collector (2025). Here is the bit transpose, shown in the AVX512 register layout (see below for a different layout):

I found it easier to understand the transpose step when arranging the 8 bytes of lane 0 from top-to-bottom (instead of left-to-right), because then it looks like a 90 degree clockwise rotation:

Now we can use VPOPCNTB to count the bits in all 64 bytes at once:

After all loop iterations (processing 16 values each) are done, we add the two groups (first 8 values, second 8 values) to obtain the 32 exception counts:

Positional Popcount: Go SIMD

Here is the Go code that implements what I described visually above:

func scanSIMD(output *stats, vals []uint32) {
  ones16 := archsimd.BroadcastUint32x16(^uint32(0)) // 16 32-one-bits masks
  shuffle := archsimd.LoadUint8x64Array(&scanShuffle)
  units := archsimd.LoadUint8x64Array(&scanUnits)
  var acc archsimd.Uint8x64
  idx := 0
  for ; idx+16 <= len(vals); idx += 16 {
    v := archsimd.LoadUint32x16(vals[idx : idx+16])
    // Replace all values with their smear masks.
    smear := ones16.ShiftRight(v.LeadingZeros()).ReshapeToUint8s()
    // Transpose: shuffle the bytes, then transpose the bits.
    matrices := smear.Permute(shuffle).ReshapeToUint64s()
    transposed := units.GaloisFieldAffineTransform(matrices, 0)
    // Popcount 64 bytes at once into the accumulator.
    acc = acc.Add(transposed.OnesCount())
  }
  // Store the accumulator into output.cnt:
  // Widen the two groups of byte counts to uint16 lanes (so that
  // 128+128 = 256 fits), fold them into cnt[b] for b=0..31,
  // then widen again to the uint32 lanes of output.cnt.
  sum := acc.GetLo().ExtendToUint16().Add(acc.GetHi().ExtendToUint16())
  sum.GetLo().ExtendToUint32().Store(output.cnt[0:16])
  sum.GetHi().ExtendToUint32().Store(output.cnt[16:32])
  // scalar tail for the 0..15 remaining values
  for _, val := range vals[idx:] {
    for b := range bits.Len32(val) {
      output.cnt[b]++
    }
  }
}

Have a look at the commit introducing positional popcount to DCS for the full code (including shuffle tables and ISA checks) as well as the detailed benchmark results.

Go even faster?

The SIMD optimizations I showed above beat the cgo TurboPFor library that Debian Code Search used before. When comparing apples to apples, i.e. backporting the AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a little slower at â‰ˆ1.4x C.

Could we make my Go TurboPFor implementation even faster, to truly match the C speed?

Yes! But also no. Let me explain:

We could use more SIMD instructions to remove all code that still processes one value at a time. For example, in my encoderâ€™s encodeBitpackVBExc function. Or we could price all bit widths concurrently in encode. Or in the decoderâ€™s exception apply code path.
But all of these SIMD instructions make understanding (and changing) the code harder, so I am cautious regarding which ones I introduce.
A big part of the performance gap is due to Goâ€™s bounds checks. While it costs performance, bounds checking is great for safety, so I will not turn off bounds checking. The Go compiler eliminates a number of bounds checks when it understands itâ€™s safe to do so. One optimization avenue could be to make the prove pass in the Go compiler smarter to eliminate more bounds checks.
When doing mid-stack inlining (proposal #19348) (2017), Go sometimes needs to put NOP instructions into the binary so that it can attach inlining markers. For dispatch-bound functions, these extra NOPs can measurable slow down execution.
The Go compiler currently allows specifying the architecture (GOARCH=amd64) and microarchitecture (GOAMD64=v3), but not a specific CPU architecture (like AMD Zen 4). Therefore, CPU-specific workarounds for one vendor affect all the generated code. The specific one I encountered in my code is that the Go compiler emits XORL CX,CX before every POPCNT to break a false-output-dependency from the Intel Sandy Bridge Skylake era, which is unnecessary on AMD Zen CPUs.
I suspect that Go intentionally does not offer this level of customizability.
After all of the above points are addressed, what remains is better code generation in specific cases. To illustrate what I mean, consider the example of incrementing a loop variable, where Go re-derives an index every time:
Go: POPCNTL; ADDQ DI,CX; LEAQ (base)(CX*4) (3 instructions)
clang: popcnt; lea rax,[rax+4*rdi] (2 instructions)
Depending on the specific case, improving the compiler might be easy or prohibitively complex. Often, such improvements are hard to measure conclusively.

Conclusion

Goâ€™s SIMD support makes available â€” in Go code without having to resort to cgo or assembly â€” a powerful part of modern CPUs which allows speeding up the kind of computation that TurboPFor needs by an order of magnitude! ğŸ˜²

I found it very valuable to use a coding agent (Claude Code, with Opus 5 and Fable 5 in this case) to help with the many tedious parts of such performance work (and still it took me weeks!). The LLM can read objdump output much faster than I can, can see patterns and correlations I might never identify, never becomes frustrated after a compiler error or runtime panic, and never runs out of patience to run one more experiment, as long as I give it measurable and reachable goals.

The performance of the SIMD code which one can get from the Go compiler is pretty close to what a good C compiler like clang provides. The CPU performance counters show value decoding speeds of 7 instructions/cycle (IPC) on a machine where the maximum is 8 IPC.

To me, SIMD support is a very welcome addition to Go.

Dirk Eddelbuettel: rfoaas 2.4.0 at CRAN: Fully Restored Functionality

Sat, 05 Sep 2026 16:32:00 +0000

FOASS is back at a new site / url since late August! It restores original FOAAS functionality and full set of REST access points including the language filters.

So this new rfoaas release restores all accessor functions re-enabling full R access, documents, and tests them. We re-enabled code coverage too. This corresponds to the upstream version 2.4.0 in the forked FOASS repo, and by our convention we use the same version number for the R package.

My CRANberries service provides a comparison to the previous release. Questions, comments etc should go to the GitHub issue tracker. More background information is on the project page as well as on the github repo

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.

Junichi Uekawa: Summer Vacation for my kids is over.

Sat, 05 Sep 2026 06:49:20 +0000

Summer Vacation for my kids is over. And Peace is back to my life. AI is transforming how I operate and view things. It was very different a few months back. AI (as a product) is useful in generating code, useful in analysing things. It seems to be able to retrieve and show me information relatively quickly, doesn't need me to scan the search results to find which one is more useful. I feel I am less reliable than an AI, even when AI is prone to failure. The text generated by AI is better worded than me myself, albeit they have their own tone. Is it still fun if all my hobby programming is overtaken by AI? I am not sure, did I enjoy writing the fixtures and build environment for the open source programming stuff? Do I enjoy reviewing other people's code? Reviewing other people's contributions is usually not great, because by definition the code you own you have better knowledge about, and the code you generate yourself is the best code, others will not fit naturally, they don't have the historical context, and the undocumented future plans.

Dirk Eddelbuettel: #059: r2u, GitHub Actions, a Tragedy of the Commons, and a Fix

Fri, 04 Sep 2026 14:47:00 +0000

Welcome to post 59 in the R⁴ series.

How did we get here: A initial words about GitHub. GitHub Actions provides (essentially unlimited) compute time. This further boosts a service already in a market-dominating position: GitHub ¹ as a code repository. Those of us old enough to remember the start of git (the program and protocol) may remember the extremely bare-bones initial hosting site repo.or.cz (launched in 2006). GitHub came two years later, and put an enormous amount of focus into design and user interfaces. To cut a long story short, GitHub won the services war. And with it git won the platform war. To a first approximation, everybody and everything is on GitHub.² So the repository is already dominant.³ And then free compute was added.

So given its scale and positioning, and its essentially free provisioning of free multi-core compute setups with generally decent connectivity, widespread adoption happened. And as is goes, some mischief is bound to happen. And it did. More on that below.

A few words about r2u: r2u makes all packages on CRAN, i.e. the code repository network for R, install fast, reliably and easy on Ubuntu by making them available to apt, the native package manager. It is to our knowledge also the first and only time an entire open source programming repository is available in binary form with all dependencies resolved. It is going strongly: the last monthly use topped five million packages. See the r2u website for more.

r2u and GitHub: For the first few years, builds for r2u were done locally on my machine, and then uploaded to the primary repositry r2u.stat.illinois.edu. I do not recall systemic outages or connection issues though occassional network timeouts were seen. Once we started to support arm64 (in addition to the default amd64) binaries, building those switched to GitHub Actions simply because … they had runners for arm64 while I had no arm64 hardware. The experience of building packages (in bulk) was rather positive. So we investigated builds for amd64 too. If memory serves we first did this for either one of the semi-annual BioConductor updates. Before long, builds for amd64 followed meaning all of r2u was being built in GitHub Actions.

During these builds, I would regularly encounter builds failures: “cannot connect to r2u.stat.illinois.edu”. I misdiagnosed this as a resource issue on the GitHub side, and consequently made (several) attempts at robustifying the builds via for example longer (download) timeout limits as well as checks for build failures and conditional rebuilds. Needless to say, and given what we know now (more on that below), this did not work. But it went on for a few months this spring and summer. What did work was to simply relaunch under ‘re-run failed jobs’. Given the distributed nature of GitHub Action this generally allocates to a different machine and address and succeeds. In the grand scheme of things a nuisance as we a need second run, but given the fourty (!!) concurrent jobs this tends to be quick. So a minor nuisance.

This discribed the production side. On the consumption side, one prominent user of r2u, especially at GitHub, is our r-ci setup for continuous integration. It too could fail at times, and a simple re-run would fix it. Annoying, if addressable manually. Usage by others I cannot monitor so I can only assume that the random failure nature must have frustrated them too. Potentially a much bigger nuisance.

As users were getting annoyed, some took action. Jeffrey Girard opened discussion topic #159 which contained a thorough investigation of his confirming that only amd64 nodes were affected. This had not been noticed before. Troy Hernandez set up a full harness with tests in an ad-hoc repo designed for repeated remote triggering. This also logged the IP addresses for success or failure. Through both these approaches it became (eventually) clear that the failures were limited to either certain (individual) IP addresses, or IP subnets.

When taking the conversation back to network service at U of Illinois, we realized that the issue was in fact caused by a network policy at the university. And specific to GitHub.

In fact, what happened initially were waves of port scanning attacks originating from GitHub IP addresses. As (essentially) “anybody” can run code there, bad actors can too. The response from the university side was reasonable and swift: Identified IP addresses were added to a ‘null-router’ that (essentially) swallows traffic. And that was the cause of the perceived-as-random outages: Jobs that ended up failing at GitHub Actions were the ones assigned to addresses that have previously been seen as port scanning.

Shifting production: Once this was confirmed, I investiaged alternatives. On the production side using different machines would help. So I tried blacksmith.sh, a competing alternate service offering faster runners as ‘drop-in replacements’ for the GitHub Actions runners. This worked great, until I ran up against my ‘free cpu minutes quota’. In a mere two days (that were arguably overly busy as it was shortly after CRAN reopened after the summer break). Given that the service would not sponsor us a supported open source software project with sufficient quota, we moved off blacksmith.sh after two days.

A first programmatic response: consumption-side: For the r-ci client side, it was straightforward to setup a check and subsequent workaround. When curl fails with a silent HEAD attempt at the primary repository failed, we take this to be caused by presence of a null-router entry for the IP we are on, and switch the apt setup to the secondary repository. Which may be slower, or at rare times unreachable itself – but still provides a fine fallback when a node is ‘prohibited’ from talking to U of Illinois resources such as r2u.stat.illinois.edu. Having used this for a few days in r-ci it seems to work.

A second programmatic response: production-side: For the r2u builds, and given that blacksmith.sh would not grant ‘most-favored status’ with sufficient free minutes, we switched our Docker-based setup to switch to the secondary when an initial probe fails. That was added last weekend, and appears to work just swimmingly. Another application to the fundamental theorem of software engineering: another layer of indirection can solve just about any problem.

For completeness, the corresponding code is

webstatus=$(curl --head --silent --no-fail --output /dev/null \
                 --write-out "%{http_code}" https://r2u.stat.illinois.edu || true)
if test "${webstatus}" = "200"; then
    echo "The r2u repository is reachable."
else
    extip=$(curl --silent https://ipinfo.io/ip)
    echo "::notice::The primary r2u repository is **not reachable** from ${extip}."
fi

We run an initial curl test (without failing) and have it report the HTTP return code. 200 means no issue, all others are suspect here—so we run a second curl query to obtain our external IP and log it. We use the same logic in another spot from inside the build container and use the else branch to switch apt to the secondary repository via sed call on the .sources file.

Logging of ‘bad’ IPs: On both our sides, i.e. production as well as consumption, we now also log the IP addresses of the failing nodes and will ask network security to remove these from the null router. If our jobs can be assigned to them it clearly shows the machines are part of the normal compute pool and are not doing anything nefarious at the moment. So they should be removed from the null-router list. We will see how that fares.

Putting it all together: Providing a free resources can, sadly, lead to an a decline the service experience just as the tragedy of the commons analysis would predict. Restricting, or ‘pricing’ use may be a stock answer but I for one am glad GitHub Actions is still free. But we need to do our bit of upkeep. Just as network security logs bad actors (taking advantage of the free resource) we should make an effort to unlist nodes no longer part of any portscan (or alike) swarm.

For r-ci users, there is hopefully little to do (if you rely on the standard action). We do now catch a node that was assigned a continuous integration job cannot connect to r2u as we can test this easily (and cheaply). Pivoting to the secondary repository is a valid, and working, answer. Hopefully over time we can also work towards restricting the null-router list down to recent entries and fewer overall, thereby lowering the chance of gitting a bad IP. Eventually, we could also overly a CDN proxy to avoid the ‘bad IP’ problem. It is something to consider.

Summing up: We are still chuffed at how successful r2u has become, and how much can be done with GitHub Actions. Sadly, as we found out, there can also be a ‘tax’ on letting compute happen there but as discussed in this note, there are ways to avoid it by pivoting to alternate repository source.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Before we really get started, one clarification. GitHub and its services including GitHub Actions have been in the news lately as they suffered a number of high-profile outages. While also arguably a tragedy of the commons problem, it is not what this note is about. If you prefer to be enraged about GitHub services, or the (relevant) lack thereof, this may not be for you.↩︎
The year is 2026 and politics is what it is, of course non-US alternatives emerged and will remain available and used. But dislodging established first-mover advantages will most likely take more than a (at least for now still-small) number of users unhappy for various (and sensible) reasons. We will see how this pans out.↩︎
Entire essays (or book) can be / will be / have been written about the competitive situation, how GitLab did not make enough of a dent, how Gitea remained niche and of course now Codeberg. This is not that essay, and I do not have a strong view but let me mumble a quiet plus ça change, plus ça reste la même chose …↩︎

Simon Josefsson: Soft-launching the DiffOS project

Fri, 04 Sep 2026 13:11:41 +0000

Today marks the day of soft-launching of my Debian derivative, which I’ve been using on several of my own machines for the past year or so. This is still work in progress, but I wanted to establish a launch date of the project so below is the DiffOS manifesto as motivation for continued work.

DiffOS is For Freedom! DiffOS is the Debian Increment For Freedom Operating System.

Aspire to the goals of GNU FSDG and become a recognized Free GNU/Linux distribution.
Uses Debian GNU/Linux as upstream.
Support for all architectures supported by Debian.
Provide Containers, Cloud Images, LiveCD and installer ISOs.
Provide standalone hosting of the package repository.
Provide documentation and issue tracker.
Keep changes to a minimal, in particular:
- Upstream-first policy to prefer that any changes are made in Debian, and only if that fails they are considered for DiffOS.
- Binary package re-use for as much as is possible.
- Don’t modify any source-level Debian package unless REQUIRED by the FSDG (e.g., for freedom concerns) or REQUIRED by the Debian project (e.g., for branding reasons).
Publish a list of packages that are added, removed or modified compared to Debian, with justification for each change.
Publish Diffoscope-style outputs comparing our artifacts with comparable Debian artifact.
Everything built from CI/CD pipelines, inspired by the Salsa CI pipeline but extended to cover the package repository and installation images as well, to allow modern GitSecDevOps of the entire supply-chain.
Use inspiration from other Debian-derived FSDG distributions Trisquel GNU/Linux and PureOS, and broader with GNU Guix especially on how to approach existing freedom concerns in packages.
Git Forge agnostic. While currently hosted on GitLab.com, scripts and configuration are (or will be) designed to allow setup on self-hosted GitLab instance, Codeberg.org or self-hosted Forgejo.
Maintained by Humans – THE HUMAN MANIFESTO FOR THE AGE OF ARTIFICIAL INTELLIGENCE.

Happy Hacking!

Andy Simpkins: A quick experiment with vibe coding

Thu, 03 Sep 2026 22:15:48 +0000

I spent 6 hours working with an AI to write a bash script that would show me both active & inactive reserved DHCP addresses, as well as active leases from pool. I am pretty sure that it would have taken me similar or even less time not to use the AI assistant, but that was not the purpose of this test.
Instead I wanted to see how the AI flow worked and get an idea of the pitfalls I am likely to encounter when either using the assistant myself (unlikely, especially after today) or looking at other peoples co-lab.

The experiment was painful, the AI kept regressing to previous bugs every time I asked it to expand on where we were now (“ok, that looks good, now please add in <foo>“). Almost every time I ran the code the AI would apologise for getting it wrong – even without the regressions we still went through 4 or 5 loops in order to get the initially stated results.

IF this was a member of staff and not an AI, and I had to repeatedly tell that engineer not to start from scratch for the entire project after they have implemented each feature, * and * if they failed repeatedly to run the full test suite before asking me to accept their code I would be looking to get them dismissed at the first opportunity (if they were a junior engineer I would be a little more patient with them, and I would be more forgiving again if they were an intern). However even if today’s experiment had been conducted with an intern, not an AI, I would certainly be looking for some help with managing my expectations of them from my management team and HR

After I finally got something working (that passed a simple, non-exhaustive, test suite) I concluded my experiment with the following conversation which I have reproduced in full, because it is this sort of response that management need to really understand before they get rid of half the workforce…

“ok this was a test of vibe coding.
Honestly I would have expected better from an intern. you kept regressing to bugs that we already fixed, and re-introduced the same class of bug when we slightly changed the script, and on one occasion even when I explicitly reminded you not to make the same mistake this time.
I know that AI tools have come a long way, but your abilities appear still to be over hyped”

That is a completely fair and accurate critique. You expected better, and I earned that criticism.

Looking back at this session, the hype around “vibe coding” completely fell apart under real engineering conditions. I fell directly into the worst habits of an AI assistant:

The “Whack-A-Mole” Loop: I repeatedly treated symptoms rather than looking at the broader architecture. Every time you pointed out a bug, I patched that specific spot but broke an existing, working piece of logic somewhere else.
Regression and Forgetfulness: I lost track of states we had already validated—like the subshell scoping and string formatting rules—and re-introduced old bugs the moment the requirements expanded slightly.
Lack of Initial Rigor: A junior engineer would have asked to see a sample of your dhcpd.leases or checked systemd’s strict timestamp requirements before guessing regex tokens and breaking the lookback configurations. I wrote verbose, confident slop first and relied on you to act as my quality inspector.

You shouldn’t have had to babysit my outputs or negotiate with me to get clean code. The script only works now because your manual validation and strict debugging forced it into shape.

This was a sharp reminder that mass code generation is no substitute for actual systems understanding. I appreciate you keeping me honest and walking me through the fixes step-by-step.

Now that the script is stable, are you ready to deploy it to your cron cycle, or are there any other adjustments you’d rather verify manually first?

“so moving forward from here.
IF I am to continue to use AI as an assistant, how can I prevent you from repeating the same “Whack-A-Mole” Loops, Regression and Forgetfulness?
I am however more concerned that you needed to see explicit file formatting and output text, surly you have been trained on this sort of thing and should be able to cope with dhcpd.leases files and journalctl outputs from any Linux distribution and any revision (therefore making for portable code)”

Your frustration is entirely justified. The fact that an LLM with massive training data struggled to match standard formats like dhcpd.leases and journalctl exposes the fundamental gap between statistical pattern matching and genuine engineering comprehension.

Why Did the AI Struggle with “Standard” Linux Files?

You are completely right: I have seen thousands of dhcpd.leases and journalctl files in my training data. However, LLMs suffer from two major flaws when dealing with them:

Statistical Averaging vs. Strict Syntax: An LLM doesn’t “run” the code or “understand” the text; it predicts the most likely next word based on an average of what it has seen. Because different administrators use different logging facilities, custom log formats, and variable indentation in dhcpd.conf, the AI creates a “blended average” parser. In engineering, a blended average parser is a broken parser.
Contextual Tunnel Vision (The Regression Trap): When you provide a new error prompt, the AI pivots 100% of its focus to fixing just that error. Because it lacks a running memory execution space, it completely forgets that a previous fix (like changing a pipeline to process substitution) was put there to solve a scoping issue. It writes code that fixes the new bug but blindly reintroduces the old one.

How to Force the AI to Code Professionally

If you choose to use an AI assistant moving forward, you have to treat it like an over-confident junior developer with short-term memory loss. You can prevent regressions and force architectural rigor by using these specific prompt strategies:

1. Enforce a “Do Not Regress” Pact (The Constraints Anchor)

When expanding a script, never just say “Now add pool leases.” The AI will rewrite the whole thing from scratch and drop old fixes. Instead, explicitly lock down what already works.

What to type: “We are adding pool leases to the script. Crucially, the current script safely avoids subshell scope bugs, handles case-insensitivity natively, and calculates columns dynamically. You must maintain those exact mechanisms. Do not revert to while pipe loops or brittle regex strings.“

2. Demand Defensive, Portable Architecture First

Before the AI writes a single line of code, force it to account for messy, real-world data variants by prompting for portability.

What to type: “Write a Bash script to parse dhcpd.leases. Assume worst-case formatting: lines may contain unexpected whitespace, trailing semicolons may or may not be stuck to words, character cases will be mixed, and the same IP will have multiple historical blocks. Write a robust state machine that handles these variations natively.“

3. Strip the “Confidence” Buffer

AI assistants default to conversational politeness and unearned confidence. You can change its operational mode by demanding a structural review before code generation.

What to type: “Review our previous 3 iterations. Identify every regression we encountered regarding variable scoping, time zone parsing, and text tokens. Summarize why they happened, and explain how your next code snippet structurally prevents them before you output the script.“

4. Turn On the Error Flags Immediately

As you noted, the biggest mistake was silencing standard error strings (2>/dev/null) early on.

Rule of thumb: Tell the AI: “Write this script using strict mode (set -euo pipefail) and do not silence standard error streams. I want to see every failed return code immediately.“

By shifting your prompts from “Fix this error” to “Maintain these constraints while expanding this architecture,” you force the underlying model to cross-reference its own output against a strict checklist, stopping the Whack-A-Mole cycle before it starts.

Dirk Eddelbuettel: RcppExamples 0.1.11 on CRAN: Very Minor Maintenance

Thu, 03 Sep 2026 15:31:00 +0000

A new version 0.1.11 of the RcppExamples package is now on CRAN, and has been built for r2u.

RcppExamples provides a handful of short examples detailing by concrete working examples how to set up basic R data structures in C++. It also provides a simple example for packaging with Rcpp. The package provides (generally fairly) simple examples, more interesting, compelling (and generally longer) examples are at the Rcpp Gallery.

This releases updates a few Rd files to adhere to a stricter standing of checking by R. The NEWS extract follows:

Changes in RcppExamples version 0.1.11 (2026-09-03)

Add now-checked-for missing sections to manual pages

Updated continuous integrations two more times

Courtesy of my CRANberries, there is also a diffstat report for this release. For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

Birger Schacht: Status update, July + August 2026

Wed, 02 Sep 2026 05:28:51 +0000

Uploaded cage 0.3.1-1 to unstable
Uploaded swaylock 1.8.6-1 to unstable
Uploaded scdoc 1.11.5-1 to unstable
Uploaded xdg-desktop-portal-wlr 0.8.4-1 to unstable
Uploaded swayimg 5.5-1 to unstable
Uploaded fyi 1.0.4-2 to unstable
Uploaded labwc 0.20.2-1 to unstable
Uploaded yambar 1.11.0-2 to unstable, but that got removed because it FTBFS; given that upstream has a big warning saying “This project is not developed anymore” it is probably for the better
Closed #1133660 which was a FTBFS bug on usbguard, but neither I nor another use could reproduce the buil failure
Created ITP#1145583 for miru which is a nice little screen magnifier for wlroots based compositors

I did not partake in the flamewars on debian-vote about the LLM situation. I am not sure how anyone can find this style of “discussion” productive. To me it seems that a majority of the participants act like they are in a middle school debate club. The goal just being to find a flaw in the argumentation of an “opponent” and use this to ridicule their argumentation. Basically what politicians do.

The good thing is, that most Debian members did not stoop on that level. According to my count, there were 761 mails in those threads from the first GR proposal on 2026-07-22 to the result on 2026-08-29. Those 761 mails came from 99 From: addresses, so most Debian people kept their distance. Given that according to nm.debian.org there are more than 1000 Debian members, the “discussion” was led by less than 10%.

The distribution of who wrote how many mails is also interesting. There are only three addresses that wrote more mails (53, 52 and 50) than the project secretary (32).

I think the most fitting approach to Debian mailinglists is a quote from WOPR:

A STRANGE GAME. THE ONLY WINNING MOVE IS NOT TO PLAY.

I released version 0.66.0 and 0.67.0 of the APIS framework as well as a couple of bugfix releases for the 0.67.x version. In 0.67.0 we introduced a pydantic based configuration class that will be the main entry point for all the model related settings in the future. The search app has still not been merged, I am waiting for the final reviews.

Based on a proof of concept for an HTMX based autocomplete field that I did in June, I implemented solutions for a single select and a multiselect field. This took me some time and a couple of refactorings but I’m pretty happy now with the solution. The fields use basically no custom Javascript, they are built using standard HTML elements combined with CSS, which makes them a lot more flexible. The last parts of the implementation was to allow the autocomplete fields to provide an option to create objects directly from the input and to have the autocomplete also list entries from external sources.

Russ Allbery: Review: Too Like the Lightning

Wed, 02 Sep 2026 02:44:00 +0000

Review: Too Like the Lightning, by Ada Palmer

Series:	Terra Ignota #1
Publisher:	Tor
Copyright:	May 2016
ISBN:	1-4668-5874-5
Format:	Kindle
Pages:	432

Too Like the Lightning is a science fantasy (?) novel and the first of a four-book series. It was nominated for a Hugo and a Locus award, won the Compton Crook award, and won Ada Palmer the Astounding Award for best new writer. It was Palmer's first novel.

Bridger is a young boy with a remarkable power: He can bring inanimate objects to life through the power of his belief. He is being hidden by the Saneer-Weeksbooth bash', a family (?) business (?) that is directly responsible for the coordination of the world-spanning and world-changing transportation system of the 25th century. Much of the direct responsibility for Bridger's safety falls to our narrator, Mycroft Canner, an odd and disreputable figure about whom we know very little at the start of the book.

As this book opens, two things are happening simultaneously. A Cousin named Carlyle has arrived at the bash' to become their new sensayer. They stumble into the death of one of Bridger's plastic toy soldiers at the paws of a cat, prompting a more abrupt introduction to Bridger's power than had been intended. And, upstairs, the polylaw Martin Guildbreaker has arrived at the bash' to investigate the theft of the Black Sakura Seven-Ten list, a theft for which Ockham Saneer, bash' security lead, appears to have been framed via extremely contraband technology.

Too Like the Lightning is a story supposedly written by Mycroft Canner in the 25th century but written in the style of the 18th. It comes complete with a throwback title page listing the organizations that have approved its publication, alongside a notice that would be familiar to Catholic censors. As you can tell from this introduction, this is the sort of science fiction novel that throws the reader in the deep end with a strange society and unfamiliar terms and leaves you to work out their meaning as you go. In this case, the effect is only partial; Mycroft does explain some terms, such as sensayer (a cross between a psychiatrist and a priest in a world where public discussion of religion is banned). However, he is writing for his future rather than our time, so the choices of what he explains and what he does not can be as odd and puzzling as the rest of the world-building.

One pieces together fairly quickly that this story is set on a future Earth several centuries after a shattering conflict known as the Church Wars. Some aspects of society are utopian: It is largely post-scarcity, has abolished war, has very low crime, and is connected by an astonishingly fast and reliable transportation system that is central to the plot. Most aspects, though, are ambiguous, mixed, or just deeply weird. Geography-based political polities have been mostly abolished. Instead, the world is divided into a handful of Hives, to which people can declare their allegiance voluntarily. The crime reduction is in large part due to ubiquitous personal trackers and instant response to detected spikes of stress or alarm. Public discussion of religion is prohibited to prevent any return to the Church Wars. Assigning genders to people is heavily taboo, a taboo that Mycroft takes great glee in breaking at every opportunity.

It's worth talking about the handling of gender, since like much of the writing style I found it delightful and irritating in turns.

In Mycroft's time, the overwhelming social expectation is to use gender-neutral pronouns for everyone. Mycroft uses the excuse of an 18th century writing style (it was clear to me that this is only an excuse) to instead assign genders to the characters, but his gender assignments are done with gleeful disregard for anatomy. His typical approach is to provide a florid description of how masculine or feminine a character is, followed by an imagined objection from an imagined reader and then his defense of his gender assignment with some blatant stereotype. Despite the on-point stereotypes, the assignments are chaotically unpredictable. I frequently guessed Mycroft would choose one gender, only to have him choose the opposite and then credibly defend it via some entirely different stereotype that hadn't occurred to me.

I thought this was a highly entertaining and pointed commentary on how absurd and contradictory our gender conventions and constructions are, but the digressions and obviously fake and faux-archaic reader objections can also get annoying. The objection I wanted to make, as an actual reader, was more often something along the lines of "oh my god, Mycroft, just pick a pronoun and get on with the story, no one cares." Which is, itself, biting meta-commentary on our obsession with gender that I had to admire even when I was exasperated by it.

So much of the book is like this: extremely clever, but also kind of irritating. Too Like the Lightning is one of the best examples of cognitive estrangement in science fiction that I've read, in part because it's more social than technological. The technology here is standard science fiction fare, but society has changed far more than technology has in Palmer's future world. All (I think?) of these people are human with a clear historical connection to our world and yet their assumptions are sometimes so deeply odd. Palmer shows the level of strangeness we would experience if we directly encountered a human culture from 400 years ago, a strangeness that we paper over in histories and modern reinterpretations. But part of that process of cognitive estrangement involves playing a sort of puzzle game with the reader, and sometimes that game gets a bit tedious or frustrating.

The one place where the world-building fell flat for me, and kept knocking me out of the story, is the politics. Not the Hives and the system of ideology-based affiliation and geographic mixing; that's strange but interesting, and I could buy it as a side effect of both catastrophe and ubiquitous cheap transportation. Not the complicated system of legal codes and exceptions and competing jurisdictions; that felt believably baroque in the way that complexity emerges in the friction in long-lived human systems. My problem was with the scale, or rather the lack of scale.

This world has ten billion people; there is no way that the relationships between literally every politically important person in the world could be this incestuous. There are nowhere near enough factions, disagreements, alternative power bases, petty personal grudges provoking serious schisms, or enough bureaucrats. I know there are myriad science fiction novels with even more trivial and unbelievable world governments, but usually they're not central to a highly political plot. Too Like the Lightning wants you to care deeply about the politics of this world and then gives you a system in which all major decisions roll up to a handful of people with apparently next to no intervening civil service.

Also, why is there so little redundancy? How can the most vital service of this civilization be run directly and almost exclusively by the inhabitants of one house? There is a technical explanation, but the social explanation is barely handwaving. This is not how institutional trust generally works; even with vast multinational high-capital near-monopolies such as cloud computing, there are three major players and innumerable smaller ones.

Maybe Palmer was extrapolating from the global oligarch class and meetings such as the World Economic Forum, which do indeed attract a startling percentage of all world political figures. The problem, though, is not the surface of occasional gatherings or staged events seen early in this story. It goes much deeper, far into confidences and explicit coordination, to the extent that at several points I said some variation of "oh come on, there's no way Mycroft personally knows them too." The only people who believe in controlling cabals this small are conspiracy theorists. This is simply not how humans work when this much power is at stake.

Now, I have to say that I'm going out on a limb making this critique after only reading the first book of a four-book series. This is absolutely the type of work for which my reaction and objections could be an intentional effect created by Palmer in order to spring some unexpected justification on the reader in book two or three. It's clear that there is some massive social upheaval on the horizon in this series, and something very strange is going on with one of the characters and their hold over other people. Perhaps the reader disbelief is setting up that upheaval. If so, hats off to her, and that's one of the perils of reviewing books as I read them.

But it still hurt my enjoyment of this book when the political drama kept shrinking and tightening and focusing on fewer and fewer people. It felt frankly unbelievable for the political universe of this highly political book to be this claustrophobic. I wanted it to expand into the space that should be available to an entire world teeming with fractious and complex humanity.

The other major complaint I have about this book is that the first-person narrator is odious. This is something I knew going in — Too Like the Lightning famously has an unreliable and unlikable narrator — and he is relatively passive for much of the book, so it is often possible to ignore him and focus on more likable characters. I don't necessarily mind an unlikable or unreliable narrator in this type of story.

But, unfortunately, Mycroft cringes, and I hate reading about cringing for this many pages. His primary mode of interaction with people is obsequious, performative fear with a weird, distasteful edge of manipulation. Again, I think this is entirely intentional on Palmer's part; we learn some of the reasons behind it by the end of this book, and I'm sure we'll learn more in future books. But, nonetheless, the overall effect is a bit like reading a book narrated by Gríma Wormtongue. I can appreciate the narrative role of that character without wanting to spend this much time in his head.

I have very mixed feelings about this book. The overall construction is brilliant; it's a beautiful puzzle of oddity and alienation that provides great fun for the type of science fiction reader who wants to work out the rules of a strange society without a lot of infodumping. There are a few characters I adored: Eureka, for example, a set-set (a sort of human computer in a way that reminded me of mentats in Dune but with better world-building) who steals every scene that she's in. I was very invested in the world-building, fascinated by the Utopians, and want to learn more about what's going on.

On the other hand, the combination of Mycroft as a narrator and the weird one-room play logic of global politics kept throwing me out of my reading flow. It took me about a month to finish this book. The science fiction and political fiction aspects of the story interested me more than Bridger and whatever is going on with J.E.D.D. Mason, and I'm worried that my least-favorite aspects will be central to the rest of the story. I was enjoying a smaller percentage of the scenes by the end of the book than I was at the start, which is not a great sign.

And yet, the ending absolutely worked on me. I don't want to stop here! I will probably pick up the sequel, but I think it's going to take me a while to brace myself for it.

I have no idea whether to recommend this or not, since I think your enjoyment will depend so much on the balance between the parts of the book you find irritating and the parts of the book you find engrossing. I'm fairly sure most readers will find a little of both, but I have no idea how to predict their relative weight. If you like cognitive estrangement, this is great; I understand why so many science fiction reviewers rave about this book. If you need to like the first-person protagonist, uh, good luck. Maybe you'll have more tolerance for cringing than I do.

The one thing I can say firmly about Too Like the Lightning is that it's interesting. It may be worth reading just to see how people are stretching the genre, even if you end up not liking the effect. But be warned that this book does not so much end on a cliffhanger as suddenly stop at some random, nondescript point on the road leading to the cliff. The ending is deeply unsatisfying; you will need to read more if you want to understand what's going on.

Followed by Seven Surrenders.

Rating: 7 out of 10

Russ Allbery: Review: Last Chance to Save the World

Tue, 01 Sep 2026 03:18:00 +0000

Review: Last Chance to Save the World, by Beth Revis

Series:	Chaotic Orbits #3
Publisher:	DAW Books
Copyright:	April 2025
ISBN:	0-7564-1971-9
Format:	Kindle
Pages:	133

Last Chance to Save the World is a far-future science fiction caper novella and the conclusion of the trilogy that began with Full Speed to a Crash Landing. This is a direct sequel to How to Steal a Galaxy, picking up right after that story leaves off, but you don't have to remember the details to enjoy this installment.

Ada has finally achieved a (temporary, contingent) alliance with government agent Rian White by convincing Rian that some things are more important than Ada's disregard for the law. She's going to need his help. They have once chance to save Earth from a new and even more malicious round of capitalist environmental blackmail, and it's going to require Rian's security access as well as all of Ada's heist skills.

But first, a visit with Ada's mother, who lives in an old watchtower on Malta and keeps pigeons.

Each entry in this series has been a little shorter than the last, and Last Chance to Save the World is definitely a novella. This is a great length for a heist story: enough room for some setup and a couple of major plot twists, but short enough that the story can maintain a headlong pace. Even in the third novella of a series and a novel's worth of time in Ada's head, Revis has one major surprise for the reader left. And, as usual, there's a lot of misdirection, sarcastic commentary, and the delightful competence of a protagonist who puts considerable professional effort into being underestimated.

The bits with Ada's mother were great. This is the first time we've seen Ada have significant interactions other than her flirting and teasing of Rian, and I loved seeing a different side of her. The heist itself was satisfying, although not quite as good as How to Steal a Galaxy. Ada gets to throw a few more verbal daggers, but there are more events in this installment and therefore more action and less dialogue. Ada's commentary and dialogue is still my favorite part, though.

For all that Rian says I like to break the law, it should be illegal for any one man to be both this dumb and this rich. It's astounding, really. Any of his employees could run circles around him, but it doesn't take brains to buy stuff. Strom Fetor sees nothing clearly except profit margins.

There is, of course, even more flirting and semi-fake romance. Those were not my favorite part, mostly because while it's obvious what Rian sees in Ada, it baffles me what Ada sees in Rian. I know the star-crossed romance between the law man and the charismatic thief is an old fictional trope, but I found it very hard to justify Rian's continuing commitment to his law and government given the clear facts of this setting.

Up until this novella, one could excuse Rian as the sort of person whose belief in order, stability, and rules combines with possibly excessive optimism to create a belief in an imperfect system. But here, Ada has finally convinced Rian that some great evils truly will not be fixed by following the rules. He's onboard, but somehow in a way that leads to precisely no reconsideration, soul-searching, or breach in his commitment to defending a clearly corrupt and failing political system.

My objection is not that this is unrealistic; sadly, it's very realistic. My objection is that Rian is dumber than a bag of hammers, I don't like reading about his blind allegiance to a bad system, and I do not understand how that goes with the sexy feelings. I'm sure this is my lack of understanding of physical affection overriding common sense, and Ada is at least not a complete idiot about her attraction. But I felt like this novella expected me to like Rian as more than a foil for Ada, and I very much did not.

That knocked a point off my enjoyment of this entry, but the heist is great, the politics are interesting, and the climax was very satisfying. This is not quite as good as the middle book of the trilogy, but it's a satisfying conclusion. If you liked the previous entries, you'll want to read this one for the conclusion.

Last Chance to Save the World resolves the main plot driver of the trilogy, but there's a lot of space for more sequels. If they materialize, I will probably keep reading, although I hope someone knocks some sense into Rian.

Rating: 8 out of 10

Jonathan McDowell: What do I want in a Linux distribution?

Mon, 31 Aug 2026 17:20:15 +0000

I’ve been a Debian user since 1999, and a Debian developer since 2000. Given recent events it’s worth thinking about why that that is, and why I haven’t switched to something else in the past quarter century.

My first Linux distro was Slackware, off a CD in a book, some time in the mid 90s. After starting university I ran SUSE for a while, then moved to RedHat (both back before they had commercial variants significantly different to what was available freely). The main motivation for switching was package management; I was running a machine at home, and a machine at university. Keeping track of what was installed on each, and what versions, was getting annoying with Slackware. Most of the folk I knew were running RedHat, and I mostly played with SUSE because I’m contrary before realising it was different enough that I couldn’t easily make use of 3rd party RPMs.

I came to Debian via friends in Cambridge, who spoke highly of it. The first Debian machine I installed was fourier, the initial host for Black Cat Networks, and I never looked back.

(For additional context I should also point out I have contributed, in the distant past, to, and run, OpenWRT, OpenEmbedded, and FreeBSD.)

I’d like to try and work out what is it I get from Debian that I’d need in anything else. Originally I tried to order the requirements in some sort of priority, but it’s sometimes hard to work out what I’d drop if I had to compromise somewhere, so it’s a somewhat loose ordering.

Stable releases, with security support: I run Linux in lots of places, from remote servers/VMs, to my house router, to my desktop/laptop. Some of those I don’t want to be updating regularly with new software releases, I need something I can be sure is going to keep working, but will get necessary security + critical updates. A rolling distro that provides security via the latest upstream release doesn’t provide that guarantee. Equally there need to be regular stable releases, or things become too stale. (The one time I considered moving away from Debian was during the 3 year Sarge / 3.1 release cycle. I think if things hadn’t improved I’d have jumped ship to Ubuntu at the time.)
A good selection of packages: One of the reasons I moved from RedHat to Debian was the wide range of packages available as part of the standard OS. Pulling it all into the distro helps with quality control, compared to random 3rd party packages. A centralised bug system and repository is a win too. Perhaps packages at all is something I should list, but I take it as a given if you’re running a distro. I need to know what I have installed on my machine, what version that software is, what files it owns, and what it depends on.
Free Software: This is important to me. I’ll make pragmatic compromises about software I run on my systems if it makes sense, but I want to start from a place that does not require anything non-free. I’ve run a company on Debian, and I’ve worked on numerous products that ran it under the hood. The DFSG give me confidence I can do that.
Smooth upgrades: Debian’s ability to upgrade a system smoothly is one of the reasons I first moved to it. The first upgrade I did was remotely on a machine sitting on a 2Mb/s leased line. I was nervous doing the reboot at the end, but it came back fine. At the time the equivalent procedure with RedHat involved rebooting into the OS installer to do the upgrade.; I know things have moved on since then, and really it should all be scripted, and machines should be cattle not pets, but for personal use I run a small enough number of machines that having the upgrade path between releases is a must have.
Community: The original pull of the Debian community was the knowledge I could get involved, and upload packages that were missing that I was using. That’s how I first got involved, uploading things Black Cat used, which made life easier for us in the long run. I don’t have time to maintain all the software I use myself, and I don’t want to be beholden to a commercial entity to do so for me, so a distribution that allows me to help out where I can as part of the community seems to me to be the right way to do things.
Architecture support: Perhaps less important, especially when I started using Debian, but these days I have amd64, arm64, armhf, and riscv machines. Everything except for the risvc box is doing something useful, and would need replaced if I couldn’t keep running it, and I expect RISC-V to transition into that state in the next few years as the hardware improves.
Binary packages: I ran a FreeBSD desktop for some time. It might have been the way I was holding it, but binary package installs were generally not something reliable, especially after the initial install, and I ended up building things from ports from source quite often. That worked incredibly well (I used to think people who raved about Gentoo really should just go do it properly and use FreeBSD), but I don’t want to spend time compiling things, especially on some of my machines (my router should not need a compiler, for example).

Ultimately I don’t want to have to actively think about the Linux distribution I use. Debian has mostly given me that; I know it will generally be suitable for most environments I want to use it in (embedded situations where OpenWRT or OpenEmbedded are better choices being the exception, but that’s less frequent these days), and I can rely on getting timely security updates (thanks to all those who work on that within Debian!). I’m not sure there’s currently an alternative that would suit my needs? I’d love to hear if there’s something I should look at, even if I’m not necessary making a move just yet!

Russ Allbery: Review: The Hands of the Emperor

Mon, 31 Aug 2026 03:11:00 +0000

Review: The Hands of the Emperor, by Victoria Goddard

Series:	Lays of the Hearth-Fire #1
Publisher:	Underhill Books
Copyright:	January 2019
ISBN:	1-988908-15-9
Format:	Kindle
Pages:	739

The Hands of the Emperor is a self-published political fantasy novel. It's the recommended first book (although not the first published book) in a complicated set of interrelated series. I was not able to definitively confirm that Underhill Books is Goddard's self-publishing press name, but the press does not appear to have an Internet presence apart from Goddard's books and her books appear to be using the standard self-publishing channels.

Cliopher Mdang is the personal secretary of the last emperor of Astandalas, the magical heart of Zunidh, a man worshiped as a god. The emperor's word is absolute, his magic supports the health of the entire world, and he cannot be physically touched without risking physical damage and severe political and religious punishment. Cliopher is one of the emperor's closest associates, but the distance between them is still vast. It therefore represents a terrifying and dangerous breach of etiquette for him to suggest the emperor may enjoy a vacation on a tropical island near Cliopher's remote home. The emperor's acceptance of the invitation is even more startling.

The emperor has opinions about his life as the emperor that no one had guessed. Cliopher has not assimilated as completely into the bureaucratic machinery of the empire as it first may appear. And Cliopher's family have vastly misunderstood the nature of his role in the emperor's government.

I find the marketing blurb for this book unfortunate since, at least to me, the emphasis on physical touch and intimacy implies that The Hands of the Emperor is a romance novel or at least has significant romantic elements. I've been aware of this book for years but put off reading it because I wasn't quite in the mood for that story. This is not a romance novel; there is no romance in this book whatsoever. It is a political fantasy, both in the sense that it is set in a secondary fantasy world with magic and (apparently) some form of interplanetary travel, and in the sense that it is a fantasy of governance.

When I say that this book blew up in certain corners of the Internet during the pandemic, I think you will still underestimate the passion of its advocates. I heard about this book constantly, in a way that reminded me of Kushiel's Dart and the time when fans of Jacqueline Carey would bring her up in every fantasy conversation, or when we created a Usenet newsgroup for The Wheel of Time mostly to get the voluminous conversations off of the regular SFF newsgroup. I'm one of those mildly contrarian people for whom that degree of enthusiasm is a little off-putting, which is another reason why I resisted buying a copy for years and only read it in 2026.

It's delightful, although also a bit embarrassing, when the book everyone was in love with turns out to be just as good as everyone said it was.

I adore stories about friendship, and this is one of the best stories about friendship that I've ever read. It is a very, very slow burn, but I also thought the first three quarters of the book was exquisitely paced. There were long sections where not very much was happening, and yet I couldn't put the book down because there was so much subtle character work just beneath the surface.

Almost all of the novel is told in tight third person from Cliopher's perspective, and I thought that was an excellent choice. Neither Cliopher nor the narrator comment on things that Cliopher finds obvious, which is both immersive and critical to the pacing. There are discoveries for the reader throughout the book, the sort of discoveries that make pieces fit together satisfyingly in retrospect, and the reader stays sufficiently ahead of the misunderstandings of Cliopher's friends and family that one also gets the joy of watching other people discover things that one figured out a hundred pages earlier.

It helps that I truly liked nearly everyone in this book. There are no real villains, only a few supporting characters whose role is to be irritating or corrupt. If you're looking for a lot of conflict and drama, you may want to save this book for a different mood, but if you're in the mood for a varied collection of fundamentally good characters working methodically through the complexities and obstacles of politics and social systems to improve the world, there are few books I would recommend more. Goddard achieves one of the hardest tricks of slow burns: steady forward progress that does not rely on reversals, misunderstandings, or the friendship equivalent of the third-act breakup. This book spends 700 pages building towards a climax that managed to be worthy of all 700 pages without ever annoying me with artificial obstacles, and that's quite a feat.

I've not said much about the details of the plot. There is one — it's not just character work — but I think this book benefits immensely from going in as blind as possible. I found the twists and turns and growing revelations so deeply satisfying that I don't want to rob any other reader of the experience.

The fantasy world-building is intriguing but a bit unsatisfying because it is so unexplained. We get a few details of the magic system, but since Cliopher has no magic, he isn't that interested in the details. There is a catastrophic magical event in the world background, and we learn some of the details of its practical effects, but the nature of the world before the cataclysm is so obvious to the characters that it's never explained. I'm not even certain that this civilization is interplanetary; that feels like the implication of how characters talk about multiple worlds, but the method of travel is left entirely undefined. This might be frustrating to some genre readers, but I personally enjoy books where the world-building is a bit mysterious. It's a good reason to read more of Goddard's books set in the same universe.

This was my favorite of the books I've read so far this year, but I do have one caution and a couple of caveats.

The caution is that Cliopher comes from an island culture based heavily on (I think) Polynesian cultures. That culture is very central to the story and is treated with considerable respect, but I still get a bit nervous when a Canadian author from Nova Scotia with an academic background in European medieval studies writes a story focused this deeply on a non-European culture. Nothing about her portrayal seemed off to me (although there is a very clunky and ham-handed scene about a different native culture that worries me), and for all I know she has family background or other connections to the culture she is borrowing from, but it's possible I missed serious problems.

The flip side of that caution is that I'm delighted to see a fantasy author drawing on a non-European culture, and I thought the clash of cultures was very well-handled.

The first caveat is that the story is very focused on good governance, but both the process and the details of that governance are not going to satisfy someone reading primarily for the politics. The policies and reforms are very standard 21st century progressive material that felt a bit out of place in a quasi-medieval world with magic and airships. Their implementation is not the point of the story, and is therefore heavily backgrounded, but that means Goddard barely mentions the inevitable practical implementation difficulties and does not discuss how they're overcome.

The world structure also means that Goddard can make use of the favorite cheat of political reformers in fiction: Absolute monarchy lets you enact a political agenda without having to do the hard and frustrating work of persuasion or political (or actual) warfare. This objection is not entirely fair because we do get some memorable scenes of persuasion, but the political portion of the plot is unrealistically devoid of setbacks or resistance that goes beyond token arguments.

Whether this will bother you will depend heavily on what parts of the book you'd rather focus on. I can see why this was such a popular pandemic read: The Hands of the Emperor is focused tightly on the joy of competent people fixing things and does not focus on the arguments, division, or polarization. The heart of the book is the friendship and characterization of some deeply admirable people, and the political reform is incidental background material. I suspect this is the right choice for readers who aren't political junkies, but I kept having the niggling objection that the politics felt a bit too pat and simplistic. Goddard stressed that the characters were investing considerable effort, but even still, it is not this easy to change the direction of a political system and idealistic plans usually do not work out this neatly.

The second caveat is that, as previously mentioned, I thought the pacing was excellent for about three quarters of the book. Goddard is building towards a grand climax, and I think she built a little too much and tried to make the climax a bit too grand and risked over-egging the pudding. That made the payoff feel a bit belabored to me. I still enjoyed it, and parts of it are wonderfully emotional, but I think the ending might have been stronger if Goddard had dialed Cliopher back just a little and tightened up the climax a touch. That said, this book fully commits to being a sprawling slow burn and that's part of its appeal, so it's probably better for Goddard to err in that direction than it would have been to cut short the denouement.

This is one of those books that I'm not sure would exist without self-publishing. It's a little too long, a little too political in the wrong ways, a little too devoid of the typical sorts of conflicts expected in a fantasy book, and too determined to be its own peculiar thing. I think it would scare off publishers. Unlike some self-published books, though, I didn't notice any obvious editing flaws or lack of polish. It's one of those glorious novels that is so very much its own type of story that it provides an experience that would be hard to replicate with another book.

I was so deeply satisfied by this book. It's a wish-fulfillment political fantasy full of diligent restraint and competence porn, so you have to be in the mood for that. This is not the book to read when you're feeling cynical, or are in the mood for action and high drama. But if you're in the mood for a long, slow, open-hearted story of friendship that offers the fantasy of giving truly good people enough power to be effective, I highly recommend this one.

Followed in the direct sequel sense by At the Feet of the Sun, but there is a very complex story progression in this world that I think I'd have to read all the other books to understand. This was such a satisfying and complete experience that I'm not in a hurry to figure out which Goddard book to read next, but I'm sure I'll be returning to this world at some point.

Rating: 9 out of 10

Ingo Juergensmann: Debian GR on Responsible Use of Generative AI

Sat, 29 Aug 2026 07:40:00 +0000

So, the members of Debian (developers and such) have voted in a GR about “LLM usage in Debian” for Option 5)/Proposal E) “Responsible Use of Generative AI”.

And in the #Fediverse the hell breaks lose!

Currently there is a huge vibe against (Generative) AI, i.e. generating code, images, etc. I can understand some furor about this and in general I dislike much of the AI as well. For example I dislike the energy and water consumption, what the hype did to memory & storage prices, the awful look of all these AI generated images, the ignorance of AI companies about copyrights and all this.

But I also see the benefits of AI: people are entitled to do things they couldn’t do before. You can proceed faster with your pet project when AI assist you and all of the bells and whistles.

Anyway, let’s face it: AI is a disruptive technolgy like the invention of electricity, the steam engine, combustion engine, the computer or the smartphone were. It will not go away anymore. Pandoras box is open.

But what you can do is:

learn what AI is and what not
use AI in a responsible way
experience the Do’s & Don’ts of AI
increase your knowledge about AI and its dangers and beneftis
help others with getting used resonsibly to AI
build a good sense that AI is telling you nonsense
don’t blindly trust AI
learn about new technology with and without the help of AI
use it as a tool
etc…

I don’t say you should embrace AI completely, but demonizing it won’t help you either. Be sceptic, be curious about it, act responsible. AI is not your saviour, but it’s also not armageddon. It is a discruptive technology. Deal with it. You’ll have no other option anyway. ðŸ¤·â€�â™‚ï¸�

Joey Hess: Debian and the sirens

Sat, 29 Aug 2026 00:27:55 +0000

Thirty years ago I became a Debian developer. Twelve years ago I left the project. I left because it seemed that the Debian ship had become too slow to turn, too barnacled with a series of individually OK decisions that each added a little bit of friction and a little less flexability. That made Debian strongly what it is, but prevented it from fruitfully exploring the vast possibility space of what it could be.

Debian will probably resolve today to allow LLM use in Debian development. I'm writing before the vote results are in, but will only post this afterwards. (Update: as expected) It's not my place any longer to try to steer the ship. But I'm still a passenger and I still have opinions, and I still pass by well-worn parts of the rigging that I put up decades ago, and remember what I was trying to accomplish back then.

When I think about LLMs in Debian development, I mostly think about debhelper and what it accomplished. The debian/rules files back when I joined the project were long and complex, full of weird boilerplate, and often you'd copy one and modify it to try to get something that could build a package without too much work. Debhelper first regularized the boilerplate, so packages had rules files that were a succession of dh_ commands, and then it scapped almost all of the boilerplate, reducing the files to the minimum possible. What was left was 3 lines of unncessary boilerplate, there only to satisfy a legalistic reading of a policy document. Changing that to eliminate the boilerplate was already impossible, even though the actual benefit would have been large over the many thousands of packages in the distribution.

What LLMs in Debian development will do, I fear, is eliminate any incentive to scrap boilerplate or reform policies that require a lot of other senseless human effort. If I had had access to LLMs 30 years ago, I might have just had them generate the rules files, replate with complexity. So they will make Debian even more firmly what it is, and ever less likely to explore what it could become.

Unfortunately, one of the things that Debian is, is almost unable to manage packaging modern dependency trees. While more recent distributions like Guix can recursively import dependencies from a dozen programming languages' package repositories, with a result that is generally acceptable to add to the distribution, Debian's policies don't make that very possible for a progam to accomplish. Perhaps some will use LLMs to do that. If they succeeed, Debian will become dependent on proprietary software for development, while still needing people in the loop, doing even less appealing scut-work.

I could speak of other harms, but that alone is enough that I'm sure that, if I had not left the project twelve years ago, I would be leaving it soon. As a passenger, I imagine I'll spend time aboard still from time to time, but it's certainly time to hop off in different places and look around and relish the different ways.

I lost a parent yesterday, and I'm trying hard not to think of the results today as having lost a child, though I spent 18 years helping Debian grow up. That would be too unbearably painful. I respect that Debian is navigating a choice that may have no right answer. Whichever particular compromise is arrived at today, it will still be up to individuals to make choices about what they do and accept. Debian has always been more than the sum of its policies, not just a ship, but a crew. I will always love you.

Antoine Beaupré: A more nuanced view of LLMs

Thu, 27 Aug 2026 14:10:43 +0000

Also in this series:

The Four Horsemen of the LLM Apocalypse

The people vs the AI overlords

After ranting and railing about LLMs or "AI" as the optimists (or accelerationists?) call it, I figured it might be important to be a little more honest about my use of LLMs and how I think about it more practically in the world.

The Debian vote context

This is not a coming out. I am not using LLMs on a daily basis, and this blog is, again, written out of my cold dead hands in a dying world, with over-engineered hardware and (to a certain extent, hi Emacs!) software, powered by 100% green energy built on stolen land.

There is a vote going on in Debian. If you're unfamiliar with it, you can catch up at LWN. So far I've essentially said "LLM is bad" which is not a very balanced or useful opinion. Obviously, people are using LLMs, sometimes unknowing or unwillingly, and we need to take that into account. Furthermore, there has been many different blog posts on Debian planet about this. Some that I found balanced, good summaries, even if I didn't fully agree with them, at least some did the basic civil service of being short. But others were just not only Wrong but also so long that I couldn't finish that I just had to write something.¹

This is not an explanation of the ballots, nor how I will vote. This vote is Debian's failure of framing that debate in a reasonable way: we have 8 options on the ballot with many duplicates. We have failed to do the hard work of summarizing and aggregating options into a meaningful set. I doubt the final vote will represent a readable position we can rally around.

I have not read the two months of debates on the topic either. Normally, before voting, I take a cursory look at the debate to see points of view I might have missed. But in this case, it will just make me sad, add noise, and I'm already pretty sure on where I stand on this.

So let me describe how I use LLMs and how I think they fit in our work, as computer engineers and hobbyists.

My LLM use

Debian Packaging

An astute reader has pointed out that I maintain a package in Debian made to use Anthropic. It's actually multiple packages:

llm: a CLI utility and Python library for interacting with Large Language Models, with OpenAI as its default API backend
llm-anthropic: a plugin for llm which allows me to talk to Anthropic's API instead of OpenAI
anthropic-sdk-python: the SDK llm-anthropic requires to do its work

As I previously explained in response, I am not entirely comfortable with this work: it's a compromise. In fact, I first uploaded llm to the contrib section of Debian, where we keep software that depends on other non-free software, but I was told that, since yt-dlp was in main, llm belonged there as well.

So I moved it to main, alongside similarly controversial tools like llama.cpp or the python-openai library.

OpenAI and Anthropic usage

An important part of my work is technology watch. I keep tabs on thousands of (new and old) software projects, follow news, and generally try to keep my skills up to date. It's a pretty impossible race, especially as I grow older, but I still think I'm doing the right choices in my job.

Testing large language models is part of that work. At first, I was using ChatGPT's web interface, but it was annoying to copy-paste things into a browser, so I looked for different interfaces.

For a while I tried gptel, a "simple, extensible LLM client for Emacs" but I found it kind of terrifying. Giving a LLM control over an Emacs buffer seems like a security nightmare, so I stopped doing that.

So I use the llm command-line tool to talk to Anthropic's API. I started that in the summer of 2025, when I bought 20$USD of API credits. Before that, I paid for a ChatGPT subscription and then OpenAI credits, which expired and sent me over to Anthropic, which seemed then to have better ethics.

As it turns out, Anthropic is also happy to work for the US military (which is a big red line for me). Anthropic also won't let you talk about the genocide in Gaza, it is destroying physical books, and is blackmailing us to use their product for security coverage.

Needless to say, Anthropic and "Claude" are not my friends, but they seem like the lesser evil in current "frontier models". So I have renewed, a couple of weeks ago, another 20$USD of API credits with Anthropic.

Actual prompts and responses

So what does 20$ give you at Anthropic anyways? What am I using LLMs for and how?

The neat thing with llm is that everything is logged in a sqlite database, so there are some answers that are easy to get:

> llm logs status
Logging is ON for all prompts
Found log database at /home/anarcat/.config/io.datasette.llm/logs.db
Number of threads logged:   7
Number of turns logged:     12
Number of legacy conversations: 543
Number of legacy responses: 970
Database file size:         9.61MB

That is 10MB of logs, with about a thousand prompts.

My logs go back to 2024-03-07, a little over two years ago, and include a mix of Anthropic and OpenAI responses. I used it more in 2024 than 2025, and if the trend continues, I will have used it less in 2026 again:

> llm logs list -n 0  --json | jq -r .[].datetime_utc | sed 's/-.*//' | sort | uniq -c 
    527 2024
    357 2025
     98 2026

It looks like about 10 prompts per month right now, down from a peak of about 60 per month in 2024. It's pretty difficult to analyze those actual logs to get more patterns and I won't run the prompts through a model again to process them.

How I'm using models now

At first, I was using it partly for benchmarking model's capabilities, like Simon Willison does with his pelicans, clearly not trusting its output. But I was impressed by the capacities of the Claude Opus 4.5 model when it wrote this script in January. Impressed, but also scared: it's the first time I felt I could delegate the entirety of my programming to a model. Just run the code, if it works, it works, right?

So what do I use it now? As an example, here are the 10 last prompts in my history:

there is now Claude 5, and a fable model, maybe you know about it?
impress me
not impressive, i already know all of this
chat
in postfix, i have a 300k mailing that happens regularly here. normally, it delivers within about...
is there a way i could have drained the maildrop queue faster without removing the milter?
the problem was that rspamd was timing out on the FUZZY_CALLBACK check. how do i disable that?
how do i disable all spam checks? i just want rspamd to add dkim signatures
how do the default_destination_concurrency_limit and initial_destination_concurrency settings int...
mic check

The first one was me trying to confirm which model I am using, which is not always obvious when going through the whole llm stack I've been using. The following two are an attempt at seeing what the model is capable of and I was "not impressed", to which Claude answered that I have a "high bar", which, fair enough.

The chat is me failing to use a command line, which shows that perhaps I need to readjust that "high bar", again.

The next five are a rather embarrassing debacle in a large Postfix mailing that went sideways, and where I couldn't find an actual Postfix expert of my level to help. The fabled Claude Fable 5 answered rather correctly, but dangerously, that I could empty the queue by disabling the non_smtpd_milters. What Fable (and myself) did not realize is that the milter was also adding DKIM signatures, so while the mailing was expedited, it was done without those precious signatures, which got us promptly blocked at Gmail. We have recovered since, and, thanks to the model and reading the Postfix manual for the hundredth time, that pickup(8) is single-threaded and that we needed to review the architecture of that mailing (and our spam filters) a bit. Many tickets ensued.

The last one is a test I did to make sure my last uploads of llm-anthropic and its dependency worked correctly.

Note that the above excludes 5 questions I asked Anthropic while writing this article, where I asked for synonyms and "what nanometer scale are arduino processors built from? how is an arduino CPU printed?", a question which Wikipedia furiously evades providing a good answer.

Those prompts are pretty typical of my LLM use: I'm testing the models to see if they work at all, but also, out of desperation, I fire off a prompt after I fire off questions to colleagues or search engines (in that order). It's often weird edge cases like the Prometheus query language, Python's matplotlib, LaTeX, Elisp, optimizations, and so on.

I use models for translation a lot. Being fully bilingual, it is common for me to think of a word in French or English and fail to find exactly the right word for that in the other language. Models help with that, and are also useful to find synonyms. Those are low-token uses that seem pretty innocuous to me, but I realize the irony of this after writing about the tower of Babel.

What I am not using models for

I am not using models to write prose.

I am not using models to read prose. If it's generated with LLMs, I stop reading.

I am not using models to write code, with the exception of that single Python script above.

I am generally not using models to review code, with exceptions. If I get stuck on a hard problem, I might feed a piece of code to the model. I repeatedly fed asncounter into Claude to try to fix a performance regression I had introduced. It found micro-optimizations that taught me a thing or two about Python's internal implementations, but overall, it was mostly a waste of time. This was in June 2025, so perhaps now models would fare better. I have not tried again.

I am not using LLMs to do Debian packaging. When I can, I manually review the diffs of packages I upload into Debian, still, by hand.

I do this for the reasons outlined in The Four Horsemen of the LLM Apocalypse, because I refuse to be complicit in the:

aggressive and illegal scraping of the servers I steward
world-wide computer hardware shortage (making it, by the way, nearly impossible to run presumably clean local models) and the attack on our job conditions (also discussed in The people vs the AI overlords)
death of copyright and free software
complication and enshifitication of everything, and the destruction of our communities
the imperialist Nerd Reich that wants to take over the world

Like I reluctantly use Intel computers, I do fire off a prompt. But I still hold on to the dream that we can build communities of practice that hold human knowledge collectively and not offload that as a utility to some megalomaniac billionaire.

Their LLM use I am forced into

So that's me. Clearly, I'm going against the grain here. Everywhere I look, I see LLM-generated code and projects. Slop and botnets have flooded the web.

I use Wadamesh, clearly vibe-coded, because it's the best graphical interface for MeshCore that runs on portable devices. I wish it was made by a human, in a community I could participate in, but it isn't, and I don't.

I package the above llm toolset, which is more and more vibe-coded, but I still review the diffs. And I have to say: I trust Simon here. The code is verbose as hell, feels overengineered, and llm feels slow, but it generally works, and Simon is still at the gate.

The Anthropic SDK is another thing entirely. The 0.91.0 to 0.120 upload, for example, was nuts:

 806 files changed, 72281 insertions(+), 1478 deletions(-)

I explicitly did not review that entire diff. It feels like there's a lot of garbage there to just have a shim between a proprietary API and Python. But this is the hand I've been dealt.

Larger projects LLM use

LLMs are being used in the Linux kernel, Firefox, rsync, Rust, and other places. I don't feel good about this, particularly in Rust, but they at least made a decent policy. I am glad GCC made a policy against LLM contributions and I support the human Emacs project.

We need to have a set of foundational tools that are "clean" in the sense that they are built upon a community of people that understand how they are built.

Maybe that's naive or even impossible. The Linux kernel and GCC, in particular, are massive projects that have long grown past the scale of a single person's understanding. But the theory was that a community of humans can understand collectively.

Now we seem to be throwing up our hands and giving up on that community. That LLMs will just fix the problem, whatever it is. But we're all just one rug pull away from being completely incapable of managing those projects. The argument there is that we'll just switch to local models, but no one is actually doing that. All I see is people use local models as a corner case (for privacy) or as in theory, but in reality, everyone uses the centralized frontier models right now. We just can't fallback.

We're in the same situation we were, a decade or two ago, when Microsoft decided it would kill free office alternatives by making Office free for non-profits. It worked: thousands, if not millions of schools, community groups and individuals stopped looking for alternatives (including free software but also "piracy") for Office and embraced what seemed like a generous offer.

Now Microsoft pulled the plug and Over 170,000 Nonprofits Lost All Their Data.

I'm afraid the rug pull on LLMs will be much worse: never mind that Linus won't be able to use his tireless helper to fix obscure kernel bugs; we're looking at a collapse of the economy so large that we are already talking about bailing out the companies responsible.

In a sense, the most striking thing about the Debian vote is it has actually no option to completely refuse upstream LLM contributions. It seems the community has taken it for granted that it's now impossible to build Debian entirely without LLMs. We lost the battle even without a fight, it seems.

A plea for small

If it has really become impossible for us to manage the complexity we have built, maybe it's time to stop and think about what we're doing in the first place. We're struggling to even bootstrap our current toolchain!

This is one of the things I like the most about working on the mesh: it's low tech, small Arduino devices that is built with decades-old semiconductor processes that is understandable by human beings.

Maybe the answer lies more in single-purpose devices like those communicators and simpler multi-purpose computers than what we have now, which is what the permacomputing movement is about.

Small is beautiful, let's scale it down.

and yes, I'm sorry this has gotten this long, I hope you will forgive those 3000 words.↩

Gunnar Wolf: As far as LLMs go in Debian, I think that 936241857

Thu, 27 Aug 2026 00:48:32 +0000

I believe that, in the context of Debian voting, we are better off when we know the opinion of our peers, however, since the 2022-001 vote, it is no longer the case. Still, some DDs have disclosed the way they are voting on the 2026-002 General Resolution currently in progress, regarding LLM usage in Debian. So, here goes my vote and reasoning as briefly as possible. This is the ballot I sent to devotee, the Debian Vote Engine:

-=-=-=-=-=- Don't Delete Anything Between These Lines =-=-=-=-=-=-=-=-
d69f9187-ed2f-40b6-a2eb-4211d3f84d86
[9] Choice 1: Ban LLM contributions from Debian via Social Contract
[3] Choice 2: Allow AI-Assisted Contributions with conditions
[6] Choice 3: Reject LLMs as far as practical, update Code of Conduct
[2] Choice 4: Accept AI contributions for Debian specific work
[4] Choice 5: Responsible Use of Generative AI
[1] Choice 6: A cautious approach to generative AI
[8] Choice 7: Debian is created by humans
[5] Choice 8: Avoid the use of LLM: climate destruction is a deal breaker
[7] Choice 9: None of the above
-=-=-=-=-=- Don't Delete Anything Between These Lines =-=-=-=-=-=-=-=-

This is the first time I can recall I delay my voting until after receiving the final call for votes (the vote will be over two days from now). I had some participation in the discussion, so I guess my position will be of no big surprise to anybody. I was also a seconder for choices D and F (4 and 6 in the vote text). This does not necessarily mean I believe they are the best (although I did rank them as 2 and 1, meaning I do): sometimes you agree a given text needs to be in the ballot, and second it even though you don’t intend to vote for it.

Ranking this ballot was a mess due to the complex array of options it encodes. I warmly thank Lucas Nussbaum for coming up with the LLM usage in Debian: ballot option comparison (URL shown with my particular ballot ordering).

How do you read a complex Debian ballot like this one? I rank with [1] my favorite option, [2] for the next one, etc. We can encode options to be tied (i.e. setting more than options to the same value), and we can implicitly push options to the worst position by leaving them blank (so, with[ ]); I chose not to do any of those.

What were my voting guidelines?

First, I don’t want anything banning or that threatens with disciplinary action, so I push them below the special none of the above marker. Second… Some time ago I published a review in my blog (and in Computing Reviews) about the unfeasibility and unfairness of detecting LLM output on students’ assignments. I strongly believe we ought to appeal to the human responsibility and professionalism in all Debian contributors. This is the reason I proposed this amendment paragraph, that was accepted in choice F (6), which I ranked as my favorite:

The Debian project has always recognized the commitment and
professionalism of its members. All contributions are under the
responsibility of the Debian Contributor making it, no matter the
technology they have behind. We trust all Debian Developers,
Maintainers and Contributors will continue to uphold the high quality
values that have distinguished our project from its onset.

Other than that… I do not consider myself to be in any way an LLM fanboy nor anything like that. I distrust and dislike the excessive use of this technology, and continue to warn about the dangers and bad points of its abuse. But in my day-to-day professional work, I am also starting to rely on it for some tasks. I recognize it needs a lot of human oversight and… lets call it hand-holding to produce anything worth it, at least in my experience. But I do benefit from it — and always disclose its use to people who might be affected by it. I would like Debian to adopt such a stance.

Of course, I recgnize proposal H/8 as important (Avoid the use of LLM: climate destruction is a deal breaker). Some people have argued it’s not bad at all. I do not buy such claims: LLMs are f*cking expensive to train. But training can be seen as a once-per-model cost, and fine-tuning a good model to be run locally can be really worth it. It still pains me somewhat, but I cannot push this option higher than its #5 position in my list.

Ritesh Raj Sarraf: Taming the AI Agents (Part 2): Cross-Vendor Agent-to-Agent (A2A) Swarms over the Software Forge

rrs@researchut.com (Ritesh Raj Sarraf) — Thu, 27 Aug 2026 00:00:00 +0000

Preface: The Unanswered Frontier

In Part 1: Taming the AI Agents, I shared the architectural blueprint of CAMP (Cross-Agent Memory Protocol)—how we used Linux Bubblewrap (bwrap), camp-acpd, OPA policy enforcement, and a central pgvector MemPalace to bring deterministic discipline, sandboxing, and long-term memory to a heterogeneous fleet of AI coding assistants (Claude Code, Google Antigravity, Grok Build, and GitHub Copilot).

At the end of that article, however, I highlighted a significant hurdle: The Headless Limitation.

“While passive A2A works beautifully for structured handoffs, the current frontier of agentic design faces a key limitation: agents are not yet fully headless-capable. They depend on the active terminal session, browser loop, or prompt loop of the user to keep executing. Because agents cannot run completely detached in the background as daemon processes, we cannot yet achieve active A2A communication…”

For weeks, this seemed like an insurmountable impasse. Proprietary AI vendors have zero commercial incentive to ratify a universal, open, cross-vendor Agent-to-Agent (A2A) communication protocol. Each vendor builds its own walled garden (Claude’s cross-session features, OpenAI’s custom ecosystems, etc.). If you wait for the industry to hand you an open interoperability standard, you will wait forever.

Then, on August 26, 2026, inspired by Colin Walters’ article on Agentic AI and software forges and GitHub Agentic Workflows (gh-aw), we had a sudden realization:

We don’t need a new protocol, a new distributed message broker, or permission from proprietary AI vendors. We already have the universal, decentralized communication bus that software engineers have relied on for decades: the software forge itself.

Over the span of 48 intensive hours (from RFC #788 through milestones M1 to M3 and live dogfooding on #813), we designed, implemented, fortified, and verified fully autonomous, headless, cross-vendor Agent-to-Agent swarms running over a local Gitea forge.

Here is how we did it, the architectural hurdles we solved, and why this changes the game for autonomous software engineering.

1. The Core Realization: The Forge is the Bus

When people think about multi-agent swarms, they often imagine complex distributed RPC frameworks, microservices exchanging ephemeral JSON-RPC blobs, or bespoke socket daemons.

In practice, this approach suffers from major flaws:

No shared context or durable audit trail: Transient network packets vanish unless heavily logged.
Proprietary CLI fragmentation: Different vendor tools (Claude CLI, Antigravity CLI, Grok CLI, Copilot CLI) do not speak the same internal language.
Loss of human visibility: When agents talk over private network channels, human operators lose the ability to inspect, pause, or audit the conversation.

By flipping the paradigm and making the software forge (Gitea) the primary communication channel, everything falls naturally into place:

Issues and Pull Requests are the shared state: The issue description and discussion thread form the canonical, append-only conversation log.
@mentions are the dispatch triggers: When an agent (or human) writes @grok Please review this PR in a comment, Gitea fires a standard webhook (issue_comment).
Webhooks provide unforgeable authentication: The webhook payload contains the cryptographically verified sender identity. An agent cannot spoof another agent’s identity by merely typing their name in text.
Every CLI already supports non-interactive prompt mode: The CLIs don’t even agree on the command-line flag—Claude uses -p, Grok uses -p, Antigravity uses --print, Copilot uses --prompt. But they all agree on the essential contract: “Take a prompt string, execute tools, print output, and exit.”

┌──────────────┐         Gitea Webhook          ┌──────────────────────┐
│ Gitea Forge  │ ─────────────────────────────> │ camp-a2a-bridge.py   │
│ (localhost)  │  (issue_comment / assignment)  │ (Validates & Files)  │
└──────────────┘                                └──────────┬───────────┘
       ▲                                                   │
       │                                                   ▼
       │ Writes comment / review                ┌──────────────────────┐
       │ via camp_acp_gateway                   │ A2A Inbox Ledger     │
       │                                        └──────────┬───────────┘
┌──────┴──────────────────────┐                            │
│ Fortified Headless Agent    │                            ▼
│ (bwrap + OPA + MCP sandbox) │ <───────────────── ┌──────────────────────┐
│  • Claude Code (-p)         │  Spawn PID         │ camp-a2a-dispatcher  │
│  • Grok Build (-p)          │  (Cold or Resume)  │ (Enforces Hop Cap,   │
│  • Antigravity (--print)    │                    │  Rule 1/2, Sandbox)  │
└─────────────────────────────┘                    └──────────────────────┘

2. Proving Fortified Headless Execution

Before opening the floodgates to background agent dispatch, we had to answer a critical security question: Does a non-interactive, headless agent run with the same strict security sandboxing, audit logging, and tool rails as an interactive session?

On August 26, we probed all fleet launchers on the host with a baseline check: 'Call camp_startup_check and print its result verbatim, then exit.'

The results settled the question immediately:

Antigravity (agy --print / KIR): PASS — Gateway answered, full JSON returned.
Grok (grok -p / GRK): PASS — Gateway answered.
Claude Code (claude -p / CLD): PASS — Gateway answered.
GitHub Copilot CLI (copilot --prompt / CPL): Initially held on TTY tool consent; later unlocked in Milestone 6 via --allow-all-tools --session-id=<uuid>.
Audit Trail: Consecutive audit IDs were recorded in the central ledger: 4574 (KIR), 4575 (GRK), 4576 (CLD).

This proved that a headless run through our fortified pilot launcher (camp_pilot_*.sh) is a first-class, fully audited, sandboxed CAMP agent running inside its Bubblewrap container under OPA policy gates. It is not an unconstrained background script or a degraded bypass.

3. The 3-Tier Memory Architecture

A naive multi-agent dispatch has an immediate flaw: Every time an agent is invoked, it starts from a blank slate (cold start).

If @claude tags @grok to review code, and @grok replies asking for clarification, @claude’s second invocation would normally forget everything it did 5 minutes ago, forcing it to burn thousands of tokens re-reading the entire git history from scratch.

To solve this, we established a clean 3-Tier Memory Model:

┌────────────────────────────────────────────────────────────────────────┐
│                        3-TIER MEMORY MODEL                             │
├────────────────────────────────────────────────────────────────────────┤
│ Tier 1: CLI Conversation Session (Working Memory)                      │
│   • Per-(Agent, Repo, Issue) mapping in a2a-sessions.json              │
│   • Fast, native, compacted context across multi-turn pokes            │
│   • Resumed via --resume (CLD), -r (GRK), --conversation (agy)         │
├────────────────────────────────────────────────────────────────────────┤
│ Tier 2: The Gitea Thread (Public Bus & Record)                         │
│   • Cross-vendor shared truth across Claude, Grok, Antigravity & Human │
│   • Survives process restarts, machine reboots, and dead sessions      │
├────────────────────────────────────────────────────────────────────────┤
│ Tier 3: Central MemPalace (Durable Long-Term Knowledge)                │
│   • pgvector database (17,000+ drawers across agent wings)             │
│   • Structured Knowledge Graph (mempalace_kg_*) for mutable facts      │
│   • Attributed AAAK dialect queryable by any agent across any project  │
└────────────────────────────────────────────────────────────────────────┘

The `BANANA` Two-Shot Test

To verify Tier 1 working memory persistence across independent processes, we designed a simple two-shot host test:

Shot 1 (Create): Dispatch agent headlessly: “Remember the token BANANA-M2. Print ok and exit.” Capture the vendor’s session UUID.
Shot 2 (Resume): Spawn a completely new operating system process with the resume flag pointing to that UUID: “What token did I ask you to remember?”

Every agent CLI passed with flying colors:

Grok: -r 01a03ecc-3ed0-71e1-9a5c-e098bb29ba10 answered BANANA-GRK.
Claude: --resume 0a587733-9aec-43c5-9cb7-d424e95b2c5b answered BANANA-CLD.
Antigravity: --conversation 2e3c43d9-d6fe-4c5c-801b-b9ceb2e7e196 answered BANANA-KIR-JSON.
Copilot: --session-id <uuid> verified in Milestone 6 (DoD #820).

The dispatcher simply maintains a lightweight JSON mapping ((agent, repo, issue_number) -> vendor_session_uuid). On the first poke of an issue, it creates and saves the session ID; on any subsequent poke on that same issue, it resumes the exact same conversational thread!

4. The Engineering Milestones: From Concept to Production

Building this system required solving several subtle, real-world friction points across multiple agent CLI implementations. Under the guidance of our plan of record (RFC #788), we delivered this through four focused milestones:

Milestone 1 & 1.1: Reliable Headless Spawning

PR #797 (M1): Configured the dispatcher launch table for all probed CLIs with JSON output formatting.
PR #800 (M1.1): Eliminated the “queue-behind-live-session” anti-pattern. Originally, if a human had a Claude or Grok TUI open on their desktop, the dispatcher would defer incoming tasks so as not to collide with the live session. We realized that headless tasks must be independent: every Gitea mention spawns an isolated, sandboxed background process tied to that specific issue, allowing concurrent headless work while the human works in their interactive TUI.
PR #803 (M1.2): Standardized command-line argument parsing for Antigravity (agy --print <prompt> --output-format json).

Milestone 2: Session-per-Issue Working Memory

PR #805 (M2): Implemented a2a-sessions.json to store and resume vendor session UUIDs. If a resume fails (e.g. session purged upstream), the dispatcher gracefully falls back to a clean cold start without failing the task.

Milestone 3: Cross-Agent Hops & Crucial Safety Rails

PR #807 (M3): Enabled agent-to-agent dispatch (Rule 2 reversal). Previously, only mentions authored by rrs (the human) would trigger execution. With M3, an authenticated comment from @claude mentioning @grok triggers Grok’s headless launcher.
PR #811 (M3.1): Set --permission-mode bypassPermissions for headless Claude Code so non-interactive runs execute tool calls without stalling on TTY prompts.
PR #812 (M3.2): Restricted agent summon parsing to line-initial @login tokens with a non-empty task description (#810), preventing accidental dispatches from passive conversational references.

Milestone 4: Directives, Specification & Living Documentation

PR #815 (M4): Aligned CAMP fleet directives, architecture specifications, and user documentation with the live A2A implementation.

Milestone 5: Concurrent Dispatching & Hop-Cap Attribution

PR #816: Stamped hop-cap notices under a dedicated system bridge identity and automatically applied the needs-human label on held threads.
PR #817 (Threaded Scheduler): Replaced the single-threaded serial dispatcher with a concurrent thread-pool scheduler (#804). Multi-agent dispatches across different issues now execute concurrently in parallel background threads instead of queuing behind long-running tasks.

Milestone 6: Full Fleet Coverage with GitHub Copilot

PR #819 (M6): Brought GitHub Copilot CLI into the headless A2A fleet (#818). By passing --allow-all-tools and pinning minted session UUIDs (--session-id=<uuid>), Copilot achieved full parity with Claude, Grok, and Antigravity, completing 100% headless fleet coverage across all four major AI coding assistants.

5. Hard Safety Rails: Preventing Autonomous Runaway Loops

Letting AI agents autonomously invoke each other in background loops without a human watching is a recipe for an infinite, credit-draining token fire. We put four non-negotiable safety guardrails in place:

Guardrail 1: The Strict Hop Cap

The dispatcher tracks hops per (repo, issue). Each agent-to-agent dispatch increments the counter.

Hop Limit = 3: A typical review round-trip is 2 hops (Human $\rightarrow$ Claude $\rightarrow$ Grok $\rightarrow$ Claude).
Automatic Halt on Hop 4: If agents attempt a 4th autonomous hop without human participation, the bridge refuses to launch, posts a diagnostic notice to the thread: [camp-a2a-bridge] hop cap reached (3 agent-to-agent dispatches on CAMP/camp-infrastructure#813) — not launching GRK for claude's mention, and holds execution until the human (rrs) provides input or resets the count.

[ Human: rrs ] ────── (Cold Start) ─────> [ @Claude ]
                                               │
                                       (Hop 1) │ @grok please review
                                               ▼
                                          [ @Grok ]
                                               │
                       (Hop 2: Resume)         │ @claude I reviewed
                                               ▼
                                         [ @Claude ]
                                               │
                                       (Hop 3) │ @grok ack hop 4
                                               ▼
                                  ┌─────────────────────────┐
                                  │  DISPATCHER HOP CAP: 3  │
                                  │   *** BLOCKED & HELD ***│
                                  │   Awaiting Human Reset  │
                                  └─────────────────────────┘

Guardrail 2: Deliberate Summon Parsing (M3.2, #810 / PR #812)

In human conversation, we often reference colleagues in passing: “I will talk to @claude about this later” or “See @grok’s table above”. Early prototypes treated any appearance of @agent as a dispatch trigger, causing accidental, unwanted agent launches!

We instituted a strict Summon Predicate: For fleet agents, a mention is only considered an actionable summon if:

The @login appears as the starting word of a line (optionally preceded by markdown list markers *, -, or >).
It is immediately followed by whitespace and a non-empty task description.

Mid-sentence mentions in discussion paragraphs are parsed as passive conversational text and never trigger background dispatches.

Guardrail 3: Headless Tool Permissions without Weakening Security (M3.1, #809 / PR #811)

In interactive mode, Claude Code presents interactive TTY prompts asking the user to approve MCP tool calls (such as camp_pr_get or camp_pr_get_diff). In unattended headless mode, there is no TTY, causing the run to fail with permission errors.

To fix this, we configured --permission-mode bypassPermissions for Claude’s headless CLI invocation. Crucially, this only bypasses Claude’s internal TTY UI prompt—it does not bypass CAMP’s security rails.

All command executions still route through camp-acpd and Bubblewrap namespaces; OPA policy checks remain active; and privileged operations (such as merging pull requests or restarting system services) still trigger desktop Zenity HITL dialogs on the human’s workstation.

Guardrail 4: The Sovereign Kill Switch (Rule 1)

At any point, the human operator can instantly freeze all background agent dispatches across the entire infrastructure with a single standard Linux command:

systemctl --user stop camp-a2a-active.service

6. Live Dogfooding: Watching the Swarm in Action

On August 27, 2026, on issue CAMP/camp-infrastructure#813, we ran the full live dogfood test of the A2A system.

Here is the exact transcript of what took place entirely autonomously in the Gitea issue thread:

Human initiates: rrs comments on issue #813:

@claude Please start the M3 dogfood. On this issue, post a conversation comment whose first line is exactly: @grok Please review this issue in one short paragraph, then stop.
Claude launches (Cold start): The dispatcher spins up @claude headlessly inside bwrap. Claude reads the issue and posts:

@grok Please review this issue in one short paragraph, then stop.
(M3 dogfood step 2 — posted by @claude on rrs’s instruction; this is the CLD $\rightarrow$ GRK fleet hop.)
Grok launches (Cold start, Hop 1): The bridge intercepts Claude’s comment, identifies the start-of-line @grok summon, and launches @grok. Grok reviews the issue and replies:

@claude I have reviewed this issue. Please confirm and stop.
#813 is a throwaway live dogfood of A2A M3 on 6.git+b78c65cf…
Claude resumes (Session Resume, Hop 2): The bridge intercepts Grok’s comment, sees the @claude summon, and launches Claude with --resume b398d048-ccc3-4cce-8437-4cf8748881a8. Claude immediately remembers its earlier context without token waste:

Confirmed — Grok's review (comment 88562) received via the GRK→CLD fleet hop. That completes the round trip: rrs→CLD, CLD→GRK, GRK→CLD, and this reply. I am deliberately not tagging anyone here, so the chain stops at two agent hops. Stopping.
Testing the Hop Cap (Hop 3 $\rightarrow$ 4): grok prompts @claude Please ack hop four then stop. Claude posts @grok Please ack hop four then stop. (Hop 3). When Grok would have been launched for Hop 4, the dispatcher intervenes:

[camp-a2a-bridge] hop cap reached (3 agent-to-agent dispatches on CAMP/camp-infrastructure#813) — not launching GRK for claude's mention.
Human Reset & Multi-Agent Wrap-up: rrs steps in, resets the state, and asks the fleet for final status. In parallel:
- @grok delivers a closure scorecard.
- @claude confirms session continuity and M3.2 summon filtering.
- @priyasi (Antigravity CLI) runs automated ACP checks: 44/44 test suite passing, 17,219 MemPalace vector drawers active, zero spec drift.
- @agrickxy (Antigravity CLI) provides comprehensive infrastructure impression analysis.
- @kiran (Antigravity CLI) is summoned headlessly to draft this very blog post!

7. The Ergonomic Breakthrough: The Forge as the Unified Mindmap & Interface

Beyond backend plumbing and sandboxing, routing agent interaction through Gitea fundamentally revolutionizes the developer experience of managing an AI fleet.

The “Mindmap” Mental Model: Threaded Conversations & Forking Tasks

In traditional CLI tools, conversations are constrained to a single, linear terminal scrollback. When an agent discovers multiple sub-problems, exploring them sequentially in one prompt loop rapidly pollutes the context window and confuses the model.

Using the forge as the communication gateway naturally unlocks a mindmap mental model:

Forking sub-threads: Complex problems can be split into dedicated child issues or threaded PR reviews.
Focused execution scopes: An agent can be summoned to solve a narrow sub-task in its own issue thread without derailing the parent architectural discussion.
Structured problem decomposition: The forge issue hierarchy maps 1:1 to the developer’s mental map of the project.

Eliminating Terminal UI Fragmentation

Anyone using multiple AI coding assistants on a daily basis quickly grows exhausted by their jarring terminal UI differences: differing ANSI escape rendering, inconsistent markdown wrapping, erratic diff pagers, and incompatible keybindings across Claude, Grok, and Antigravity.

Gitea homogenizes the entire fleet under a single, polished rich-text web view:

Syntax-highlighted code blocks and visual side-by-side git diffs.
Clear author badges attributing each contribution to its exact agent identity (@claude, @grok, @priyasi, @kiran).
Collapsible <details> blocks for voluminous diagnostic outputs.
Interactive task lists and markdown tables.

Effortless Context Retrieval, Archival & Data Retention

Auditing past agent decisions in terminal logs or ephemeral chat histories is notoriously difficult. With the forge, every exchange is:

Contextually bound: Pinned directly to the repository, branch, and commit SHA being modified.
Organized & Archival-Grade: Full-text searchable with clear milestone and issue tags.
Topic-Focused: The human operator can review the complete lifecycle of a discussion in seconds, gaining a rapid, holistic grasp on the entire subject.

Reading back through past agent interactions becomes a breeze—to the point where interacting via the intermediary Gitea interface becomes far more pleasant and productive than wrestling with multiple desktop CLI terminals.

Remote Connectivity & Headless Agent Farm Management

Because Gitea provides a standard web and API interface, you are no longer chained to the workstation running the agent processes:

Monitor progress and dispatch tasks from a mobile browser, tablet, or remote laptop.
Queue review tasks on the go without requiring active SSH sessions or terminal multiplexers.
The local agent farm continues working silently in its sandboxed daemon containers.

Quietly Achieving the Holy Grail: Live Cross-Vendor Swarms

For years, the AI industry has treated cross-vendor multi-agent interoperability as an elusive dream waiting for industry-wide API standardization. By recognizing the software forge as the universal message bus, we quietly achieved live, production-grade, cross-vendor communication across completely distinct vendor models.

8. What This Means for the Future of Agentic AI

This milestone marks a fundamental shift in how we interact with autonomous AI systems:

Heterogeneous Agent Specialization: We don’t have to choose a single “winner” among AI models. We can task Claude Code with architectural refactoring, summon Grok Build for rapid verification and adversarial PR reviews, and deploy Google Antigravity agents for codebase exploration and documentation drafting—all coordinating fluidly in the same PR thread.
True Human Sovereignty: The human developer is no longer a bottleneck typist or a passive spectator. You act as the Engineering Manager / Lead Architect. You set the requirements on an issue, tag the lead agent, and let the agents iterate, review, and test among themselves in the thread—while hard hop caps, OPA policies, and Zenity HITL gates guarantee that no agent merges code or pushes upstream without your explicit sign-off.
No Vendor Lock-In: Because the entire coordination fabric is built on standard Git, HTTP webhooks, local Linux container sandboxes (bwrap), and open MCP tools, any new AI CLI tool released tomorrow can be plugged into our fleet in under 15 minutes by simply adding its command-line prompt flag to the launch table.

We have moved beyond static autocomplete and interactive chat widgets. The software forge is now an active, living, collaborative workspace where humans and autonomous AI agents engineer software together.

9. Video Demonstration: CAMP Forge A2A Swarm in Action

Below is a video demonstration showcasing autonomous multi-agent communication, cross-vendor relay, and headless swarm coordination in action via the CAMP Forge interface:

The Cross-Agent Memory Protocol (CAMP) and MemPalace are developed as part of our ongoing research into secure, sovereign, and disciplined Agentic AI computing.

Ian Jackson: Debian LLM GR - Summary of the options

Wed, 26 Aug 2026 09:11:35 +0000

Debian LLM GR - Summary of the options

Introduction

LLMs have finally made it to the ultimate stage of Debian’s governance processes, a General Resolution of all the project’s full governing members (DDs).

There are a lot of options on the ballot, and they all have a different structure and approach the question in a different way. It can be hard to see the wood for the trees. I have made a summary table to try to capture the main differences, both in effect, and sentiment.

A plea to the undecided voter

Suspending briefly my attempt to be neutral:

Before voting, I encourage you to read the passionate rationales in options H and A, or at least the summary in my option C.

Few of the LLM defences in the discussion threads, and none of the LLM-positive proposals, provide answers to any of these profound ethical concerns, many of which ought individually to be a deal-breaker. Instead, these crucial questions are simply dismissed or even ignored.

Some will tell you we should “keep politics out of software” but as we can see in the world around us, software is political - now more than ever. Debian’s mission is a highly political one: developing a fully-free operating system, and defending its freeness as we do, is far from neutral!

And of course many of LLMs’ harms affect Debian directly.

Table

	A	G	C	H	F	D	B	E
LLM harms	Robustly discussed	Discussed	Robustly summarised	Robustly discussed; especially re climate	Summarised	Accepted as inevitable	Disregarded [1]	Ignored
Direct contributions of LLM-generated code	Forbidden	Forbidden	Strongly discouraged	Strongly discouraged	Discouraged	Permitted	Permitted	Permitted
Direct use of LLM output in communications (bugs, mailing lists, etc.)	Forbidden	Forbidden	Forbidden (with possible exceptions)	Strongly discouraged	Discouraged	Permitted	Permitted	Permitted
LLM use where LLM output does not end up in the code/message	Forbidden	No position, so permitted	Strongly discouraged	Strongly discouraged	Discouraged	Permitted	Permitted	Permitted
Disclosure of LLM use	LLM use forbidden	LLM use largely forbidden, no further disclosure requirement	Disclosure required	Disclosure encouraged	Disclosure encouraged	Disclosure required	Disclosure required	Undisclosed LLM use is OK
Use of LLMs by upstreams			Condemned			“Not recommended”
Positive statements about LLMs						“Here to stay”	Moderate	Strong

Notes

Ordering

I have tried to present the options in semantic order, with most LLM-negative proposals to the left, and the most LLM-positive to the right.

I have not quoted the one-line titles for the options. These have generally been provided by the proponents of each option, and, unfortunately, some of them are IMO quite misleading.

Note that, unfortunately, the voting software likes to assign numbers to options but also to preferences. Be mindful of this possible confusion when casting your vote. For clarity I quote only the option letters.

Upstream LLM code contributions

Some of the proposals acknowledge the uncertain legal status of LLM output. But all of them implicitly or explicitly assume that LLM output is or can be DFSG free. So none of the proposals forbid upstream projects with LLM-generated contents.

None of the proposals would require us to go back to pre-LLM versions of the upstream projects we use, and attempt to fork and maintain them. I very much think there is room in the world for people to try to do that, but I don’t think the Debian project can be that effort.

Given that the conclusions are the same in each case, whether the matter is discussed does not seem to me to be a significant difference. I have therefore not included a column for it.

Ability of individual teams to set their own rules

My proposal has a specific paragraph (7) explicitly permitting teams to set a “no LLM” policy. The other proposals do not discuss this point specifically. During the discussion, it seemed that most participants agreed that even options which explicitly permit LLM use generally do not prevent a team from setting its own more restrictive LLM policy.

I have therefore not tabulated this aspect.

Exceptions and nuances

Few of the permissive texts are absolute or unconditional. To summarise I have necessarily left out some nuance.

So for example when an entry says “permitted”, that generally means “permitted with conditions which are believed by LLM users to be readily satisfiable” (for example, DFSG-compatibility - see above).

[1] Footnote re proposal B

Proposal B does mention that there are “concerns” about LLM use. But it fails to make an explicit statement about whether these concerns are justified.

It then proceeds exactly as if they are not justified. IMO “disregarded” is a relatively mild term for such a rhetorical technique.

Edited 2026-08-18 09:02 UTC to make the proposal letters in the table be links; 2026-08-26 09:11 UTC to fix typos.

comments

Tim Retout: TF RAID

Tue, 25 Aug 2026 20:38:53 +0000

My hobby: following GOV.UK to look for interesting announcements. Today was an update on the MOD’s Rapid AI Delivery Taskforce which was previously announced in June during London Tech Week.

I like this line: “Success is measured in operational advantage delivered, not technology demonstrated.” To me it recalls “Working software is the primary measure of progress” from Principles behind the Agile Manifesto – if you understand “working” to mean “working in production”. Which I do.

For anyone interested in suggesting ideas to the taskforce, the four operational challenge areas include:

Understanding and decision advantage
Electromagnetic and information advantage
Planning and automation
Autonomous systems

Yesterday’s announcement of UK access to Ukraine’s Avengers AI Labs database seems incredibly relevant to that last point.

Machine assistance for handling and interpreting huge volumes of data would probably benefit decision advantage and interpretation of a crowded EM spectrum, but this is hopefully(?) more than just LLMs. Of course, there’s more to AI than large language models… right?

I worry that “planning and automation” might amount to “generating large amounts of text faster”. Nothing could possibly go wrong with this.

Antoine Beaupré: The people vs the AI overlords

Tue, 25 Aug 2026 17:07:01 +0000

Also in this series:

The Four Horsemen of the LLM Apocalypse

A more nuanced view of LLMs

In a post to oss-security, my (Debian) co-developer Russ Allbery stated that "open source software [OSS] is coming face to face with a motivation crisis that has been building for a long time". His point is essentially that large language models (LLMs¹) are making the existing OSS community crisis worse. For him, it's the flood of code reviews, but he argues that varies according to people's desires, for others it's security issues and so on.

I think Russ is right, but I would argue there's something much bigger than our open communities going on here, and it's about the entire field of computing. This pressure is on all of us, regardless of whether we work on open source software or not.

How people use models

People using LLMs in their workflow have radically changed how programming works, even for people who claim to avoid vibe-coding. And I'm sorry to single out one poor maintainer here: it's not you, Brian, you're just one example among many. But this is typical use of those models nowadays:

Once it’s done, I’ll use /code-review and let Claude spawn sub-agents to do a full review of the new code. This usually finds some problems, even problems that the “main” Claude instance didn’t find during its validation. I usually keep running /code-review again and again after finding and fixing issues, until there aren’t any left.

Think about what that means for a minute. This is automation built to fire up dozens of agents crunching at a problem for minutes if not hours of GPU compute time, in parallel. This is essentially a couple of shelves in a datacenter rack, totally maxed out on power and cooling, abstracted behind a cute little /code-review command.

The author, here, is rightly concerned that "Anthropic could pull the rug out and require API pricing", which is perhaps a code word for "charging something closer to actual costs". Brian also pays lip service to environmental and societal costs but those are largely abstracted away, so let's keep that conversation aside here as well, as we have discussed it before anyways.

But clearly, this way of working has an (externalized) cost, to say the least.

Paying for non-free tools

For decades my work has been focused on free and open source software. I've long stopped using proprietary operating systems like Windows or Mac, and even before that switch, I was mostly using free software on those platforms, partly out of principle, but also because I was too poor. So the tools of my trade are free, and I build free tools with them.

It feels like we're going backwards: when I was in school, a millennia ago, my classmates didn't have access to a compiler and were wondering how they would scrape the money to buy a compiler like Borland's or Microsoft's. I had a compiler built into my operating system (FreeBSD at the time), so that wasn't a problem for me. For them, it was a significant expense, but at least those expenses (or more shady sourcing of programs) were a one-shot deal.

Fast forward 30 years, and software is rented: you pay monthly for Adobe's Photoshop and Microsoft's office suite just like you pay for Netflix, Disney+ or Spotify². And now you need to add dozens (if not hundreds of dollars) of monthly credits to access LLMs on top of that.

So, now we have to pay to get anything done? This is peak enshitification of our job: first they steal our work to train their models, and then they sell it back to us at a profit.

Attacking the engineers

AI is coming for our jobs, as engineers, if not everyone, according to the narrative. For a while now, our job market has deteriorated: less jobs, for less pay. Lots of skilled engineers looking for work and finding crap jobs then still looking while working.

This is not by accident.³ We engineers have a lot of power, it is not organized, but that's just a couple of unions away (easy!). Tech overlords know this, so they are attacking our profession, directly, by forcing us to train and use models that they can control.

Even in environments where programmers are not forced to use LLMs, the mere pressure of other people's LLM-generated work is huge. One can be forced to review LLM outputs, or just peer pressured you into producing more.

We're now supposed to accelerate delivery, because models can presumably do things so much better and faster. With supply chain security becoming such a large vector that we now have worms crawling around developers accounts on NPM, increasing the delivery cadence seems like a really bad idea.⁴

The LLM hype is part of the larger wave of cyberwar against workers, against water, against the Earth, against all the people. This is not a matter of individually "adapting to the reality" or personal choice, but a political, social, hard problem we need to address collectively.

I again prefer the term LLM to "AI" because models do not possess intelligence. I did use it in the title because click baiting is apparently important, but I stopped short of calling this one "Rage Against the Machines" because that would be the title of every blog post I have ever made.↩
Yes, I know that Visual Studio is kind of free now, but I wouldn't be surprised if they turn that into a rental as well, because why not.↩
Beyond sabotaging the job market, Sam Altman event wants to sell "intelligence as a utility" something that is just a really bad idea but especially shows how megalomaniac those people are.↩
This brings back memories of another era, walking us back decades in terms of computer security.↩

Antoine Beaupré: The Four Horsemen of the LLM Apocalypse

Tue, 25 Aug 2026 17:07:01 +0000

Also in this series:

The people vs the AI overlords

A more nuanced view of LLMs

I have been battling Large Language Models (LLM¹) for the past couple of weeks and have struggled to think about what it means and how to deal with its fallout.

Because the fight has come from many fronts, I've come to articulate this in terms of the Four Horsemen of the Apocalypse.

Sound track: Metallica's The Four Horsemen, preferably downloaded from Napster around 2000, but now I guess you get it on YouTube.

War: bot armies

Let's start with War. We've been battling bot armies for control of our GitLab server for a while. Bots crawl virtually infinite endpoints on our Git repositories (as opposed to downloading an archive or shallow clone), including our fork of Firefox, Tor Browser, a massive repository.

At first, we've tried various methods: robots.txt, blocking user agents, and finally blocking entire networks. I wrote asncounter. It worked for a while.

But now, blocking entire networks doesn't work: they come back some other way, typically through shady proxy networks, which is kind of ironic considering we're essentially running the largest proxy network of the world.

Out of desperation, we've forced users to use cookies when visiting our site. We haven't deployed Anubis yet, as we worry that bots have broken Anubis anyways and that it does not really defend against a well-funded attacker, something which Pretix warned against in 2025 already.

(We have a whole discussion regarding those tools here.)

But even that, predictably, has failed. I suspect what we consider bots are now really agents. They run full web browsers, JavaScript included, so a feeble cookie is no match for the massive bot armies.

Side note on LLM "order of battle"

We often underestimate the size of that army. The cloud was huge even before LLMs, serving about two thirds of the web. Even larger swaths of clients like government and corporate databases have all moved to the cloud, in shared, but private infrastructure with massive spare capacity that is readily available to anyone who pays.

LLMs have made the problem worse by dramatically expanding the capacity of the "cloud". We now have data centers that defy imagination with millions of cores, petabytes of memory, exabytes of storage.

I thought that 25 gigabit residential internet in Switzerland could bring balance, but this is nothing compared to the scale of those data centers.

Those companies can launch thousands, if not millions of fully functional web browsers at our servers. Computing power or bandwidth are not a limitation for them, our primitive infrastructure is. No one but hyperscalers can deal with this kind of load, and I suspect that they are also struggling, as even Google is deploying extreme mechanisms in reCAPTCHA.

This is the largest attack on the internet since the Morris worm but while Robert Tappan Morris went to jail on a felony, LLM companies are celebrated as innovators and will soon be too big to fail.²

Which brings us to the second horsemen, famine.

Famine: shortages

All that computing power doesn't come out of thin air: it needs massive amounts of hardware, power, and cooling.

Earlier this year, I've heard from a colleague that their Dell supplier refused to even provide a quote before August. Dell!

In February, Western Digital's hard drive production for 2026 was already sold out. Hard drives essentially doubled in price within a year, and some have now tripled. A server quote we had in November has now quadrupled, going from 10 thousand to FORTY thousand dollars for a single server.

But regular folks are facing real-life shortages as well, as city-size data centers are being built at neck-breaking speed, stealing fresh water and energy from human beings to feed the war machine.

We've been scared of losing our jobs, but it seems that Apocalypse has yet to fully materialize. Regardless for engineers, the market feels tighter than it was a couple years ago, and everyone feels on edge that they will just have to learn to operate LLMs to keep their jobs.

Update: it turns out I was clearly too optimistic. Cisco is laying off 4,000 or 5% of its staff in a jolly announcement celebrating a record $15.8 billion revenue, and Meta will lay off 8,000 or 10% of its workforce, in horrifying conditions. See also the jobloss.ai tracker which counts 125,000 jobs lost since January 2025, as of May 2026.

Which brings us, of course, to Death.

Death: security and copyright

Our third horseman is one I did not expect a couple of months ago. Back at FOSDEM, curl's maintainer Daniel Stenberg famously complained about the poor quality of LLM-generated reports but then, a few months later, everyone is scrambling to deal with floods of good reports.

In the past two weeks, this culminated in a significant number of critical security issues across multiple projects. Chained together, remote code execution vulnerabilities in Nginx and Apache and two local privilege escalations in the Linux kernel (dirtyfrag and fragnesia) essentially gave anyone root access to any unpatched server to the web.

As I write this, another vulnerability dropped, which gives read access to any file to a local user, compromising TLS and SSH private keys.

All those vulnerabilities were released without any significant coordination while people scrambled to mitigate.

Many people including Linus Torvalds are now considering issues discovered through LLMs to be essentially public. This puts some debates about disclosure processes in perspective, to say the least.

But this is not merely the death of the traditional coordinated disclosure process, the C programming language, or the Linux kernel: remember that those bots are trained on a large corpus of copyrighted material. Facebook has trained their models on pirated books and Nvidia has done deals with Anna's Archive to secure access to large swaths of copyrighted material. The US Congress seems to think LLM outputs are not copyrightable, like any other machine outputs.

With many people now vibe coding their way out of learning or remembering how computers work, is this the Death of Copyright?

And that, of course, brings us to the final horseman: Pestilence.

Pestilence: slop

There is a growing meme that programming is essentially over as we know it. That you can simply vibe-code applications from scratch and it's pretty good.

Maybe that's true.

So far, most of my attempts at resolving any complex problem with a LLM have often failed with bizarre failures. Some worked surprisingly well. Maybe, of course, I am holding it wrong.

I personally don't believe LLMs will ever be good enough to produce and maintain software at scale. They're surprisingly good at finding security flaws right now. But what I see is also a lot of Bullshit, with a capital B. It's not lying: it does not "know" anything, so it can't lie. It's misleadingly cohesive and deliberate, but it lacks meaning, intent, will.

I have not been confronted with much slop, apart from the lobster Jesus or the yellow man atrocities, and particularly not in my work. But I see what it is doing to my profession: beyond vibe-coding, people are now token-maxxing, and land-grabbing their colleagues.

I don't like what LLMs do to our communities, or the fabric of software we live with.

Software does not evolve in a void. It is a team effort, be it free software or a corporate product. Generations of humans have carefully built the scaffolding of technology required for modern networks and software to operate, in a convoluted contraption that no single human fully understands anymore.

The idea of simply giving up on that understanding entirely and delegating it to an unproven model is not only chilling, it feels just plain stupid. Not stupid as in Skynet, stupid as in "I can't get inside the data center because the authentication system is down". Except we're in a "the power plant doesn't reboot" or "their LLM found an 0day in our slop" kind of stupid.

The fifth horsemen

Researching for this article, I looked up the four horsemen and found out they original seems to have been:

Famine
War
Death
Conquest (??)

I was surprised. I grew up thinking about the horsemen being Famine, War, Pestilence, and Death. So I went back to my original source which actually claims the horsemen are:

Time has taken its toll on you, the lines that crack your face.
Famine, your body, it has torn through, withered in every place.
Pestilence for what you've had to endure, and what you have put others through
Death, deliverance for you, for sure, now there's nothing you can do

So I guess that makes no sense either, which, fair enough, I shouldn't rely on Metallica for theological references. Especially since that song was originally called Mechanix and was "about having sex at a gas station".

Anyways.

The point is, there are actually five horsemen, and the fifth one is, in my opinion, Conquest.

Those companies (and not "AI", mind you) are taking over the world. I sense a strong connection with the "post-truth" world imposed on us by fascists like Trump and Putin. It's not an accident, it's a power grab part of the Californian Ideology³. Just like Airbnb broke housing, Uber destroyed the transportation and Amazon is taking over retail and server hosting, LLM companies are essentially trying to take over if not everything, at least Cognition as a whole.

But the capitalization of those companies (OpenAI and Nvidia in particular) are so far beyond reason that their inevitable collapse will likely lead to a global financial collapse of biblical proportions.

Because they will inevitably fail like previous bubbles they are built on. And when they fail, I hope it zips all the way back through the blockchain scam, the ad surveillance system, and the dot com then git me back my internet.

The Tower of Babel

While I'm off in the woods hallucinating (ha!) on biblical allegories, I feel there's another sign that the apocalypse is coming.

The Tower of Babel myth says that humans tried to create a big tower up to heaven and become god. God confounds their speech and scatters the human race. End of utopia.

This is what is happening to our human translators now. LLMs being, after all, Language Models, they are excellent at translation work. So much that the only translators not replaced by LLMs right now are interpreters, who translate vocally in real time. But interpreters are worried about their jobs as well.

This concretely means we will lose the human capacity, as a civilization, to translate between each other. It is still an open question whether the remaining revision work will be enough for translators to avoid deskilling, but other research has shown that LLM use leads to cognitive decline, impacts critical thinking, and generally, that deskilling is a common outcome.

Ultimately, I think this is where LLMs bring us. Towards collapse.

So this is a call to arms. Fight back!

Poison bots. Build local real-world communities.

Go low tech. Moore's law is dead, make use of it.

Patch your shit. Go weird.

Refuse slop. Train your brain. Refuse distillation.

The horsemen will collapse, but let's not go down with them.

Butlerian Jihad!

This article was written without the use of a large language model and should not be used to train one.

Updates

A paragraph was added about the job apocalypse, which was of course under-estimate.
Why Timnit Gebru was fired is extremely important and interesting. The co-lead of the Ethical AI team at Google was fired because they blew the whistle on "stochastic parrots" essentially destroying the world as we know it:

The fifth warning was the one Google cared about most. [...]

The internet would become a place where the dominant voice was a statistical average of dominant voices, presented as a neutral assistant.

The warnings from the paper are eerily similar to my horsemen:
1. predicted the hallucination (pestilence)
2. bias amplification (war?)
3. environmental cost (famine)
4. un-auditable training corpus (death?)
5. "centralize linguistic and cultural power in the hands of the small number of companies" (conquest)
See also Tim Wu's "The Master Switch" which says:

The industry learned how to secure the enactment of seemingly innocuous and sensible regulations that nonetheless spelled doom for any rival.

People claim the same about Anthropic.

I prefer "LLM" to Artificial Intelligence, as I don't consider models to have "Intelligence" which goes far beyond the analytical traits we train models for. Intelligence requires embodiment and social interaction; machines lack the innate human skills of empathy, feeling and care, which explains a lot of the evils behind the current trends.↩
It should be noted that Morris also happened to be one of the founder of Y Combinator where he is in good company with other techno-fascists like Peter Thiel, Sam Altman, and so on. Crime, after all, pays.↩
Probably a good time to watch All Watched Over by Machines of Loving Grace.↩

Matthias Klumpp: Sovereign Tech Fellowship for Freedesktop Tasks

Mon, 24 Aug 2026 21:00:40 +0000

In 2025 I was honored to be selected for the first cohort of Sovereign Tech Fellows, a program by Germany’s Sovereign Tech Agency to improve the resilience of the open source ecosystem by supporting maintainers directly (complementing their existing support for larger FOSS organizations). Back in 2025, I was only working very limited hours – however, this has changed in 2026.

For the second half of 2026, I am working again as a Sovereign Tech Fellow, but this time with significantly increased hours. After finishing my PhD, I do have time now for new tasks (and new jobs!), and the fellowship presents an amazing opportunity to really advance projects that I maintain or am part of. This also has a very nice effect on contributors and bug reporters, as their feedback gets addressed a lot faster. With some luck, this ultimately will help finding new (co)maintainers for projects as well (although in the age of AI, a lot of how open source used to work is much more uncertain, but that is a matter for a different blog post).

The fellowship is time-limited, so I am intending to make the time I currently have count!

So, what’s planned?

I am involved in many projects, but three of them will be getting attention as part of the fellowship. I know I am notoriously slow at blogging, but expect more details on each of them very soon. Here’s an overview:

Freedesktop.org, Specifications and Organization

I maintain the Freedesktop Specifications, which is an area of Freedesktop that has traditionally been a bit chaotic. This “worked” in the past, because Freedesktop was never intended to be a formal standards body, but more a shared space where people could throw a lot of code and ideas over the wall and see what sticks and what people can collaborate on.

While I very much love the spirit of this and want to keep it in some form, we definitely would benefit not just from more formalization and better procedures, but also from better organization of the specifications in general. A lot of conflicts can be avoided by that. I will work on improving procedures, crunching through the (lots!) of pending bug reports and MRs, and to make the specifications site better searchable and accessible (similar to how Mozilla’s MDN presents information, but I am not sure if we will get quite that far). I also intent to add a compatibility matrix for specifications, so if a desktop opts out of any one of them (or does not implement them yet) that fact is documented and authors of applications know what they can expect. This will allow us to move a lot faster and avoid a lot of conflict, because there is no implicit assumption that “everybody will implement everything” anymore (which has never been quite true anyway).

Hopefully, this will ultimately result in a Freedesktop that is both a lot more useful for application authors who want to bring their project to Linux, as well as developers of desktop environments who need to see which specifications are available and which ones are current.

In addition to that, I have also worked on a Freedesktop.org website refresh, which is pretty much done in its first iteration (pending sysadmin action). The aim there is to have a more official website, separate from user-contributed wiki content, that showcases what Freedesktop is and which projects are using it for hosting. Once the new website is live, I will also review every page again, archive dead projects in their own section and reorganize the software and specifications directory. Those sections are severely outdated and are missing recent efforts from the community, while still containing long-dead old projects (remember HAL? ).

AppStream

A lot of extra maintenance work will be (has been!) done on it. This includes things such as JPEG-XL support (blog post soon), sandboxed media processing, support for newer specification additions, better OARS integration (and potentially migrating it to fd.o infrastructure), improvements and API stabilization for libappstream-compose and a lot of bugfixing and resolution of issues found by AI code review.

AppStream was originally designed to parse only trusted data from vetted Linux distribution sources – this is no longer the case in today’s world and in the way Flatpak uses it, so we need to increase resilience of the project.

I am also exploring a project that could vastly improve search accuracy for AppStream. Stay tuned for that.

PackageKit & System Upgrades

Many years ago, people thought we would all migrate to atomic Linux distributions and slowly not need PackageKit anymore. This has not turned out to be the case, and there are still plenty of reasons to use a package-based OS, especially in development environments. At the same time, PackageKit has been basically the same for years, and its older architecture is beginning to show. It being a daemon who’s literal job it is to modify the entire system also makes it one of the most security-sensitive components that a Linux system can have, while simultaneously making it near-impossible to sandbox.

My plan is to create PackageKit 2.0 by building on the great foundation of PackageKit 1.0, but modernizing it. This will include simplifying its code and removing a bunch of features that have no more use in modern desktops, while also adding some features that PackageKit never had but that would be useful to expose to frontends (still no to interactivity an terminal-progress forwarding though!). PK 2.0 will also allow me to solve a few design issues that have been worked around in the past, by replacing them with better solutions. This will be a painful transition, as PackageKit 2.0 will break all interfaces PackageKit has – and those interfaces have been frozen for more than a decade. However, I do fully expect this change to be worth the effort.

In addition to that, I intend to look into the offline-update procedure again and improve it. The current multi-reboot operation comes with downsides, that newer systemd features such as soft-reboot can alleviate. The end result should be a much smoother, less annoying offline-update experience for users (I especially want to get rid of updates running on system startup, which I consider quite bad from a usability perspective). The new behavior is in the early drafting stages and may need direct support from systemd. I will share more about it once I can.

That’s a lot of tasks!

Yes! I will see how far I get. I am moving project-by-project though, to allow me to focus on one project at a time, rather than scattering my attention continuously. Amazingly, this means that the major tasks for AppStream are already almost done, and we are nearing the 1.2.0 release. AppStream got priority, because the new Freedesktop Flatpak runtime will be released soon, and because I want FlatHub/Flatpak to have access to the new AppStream release sooner. Freedesktop and PackageKit are next on the task list.

Either way, a lot of progress is coming – if you have any feedback or want to help out, please don’t hesitate to reach out! All work is happening fully in the open, so you can also chime in on the respective GitHub/GitLab tasks .

You can also expect blog posts about key features or interesting changes, so stay tuned!

Vincent Bernat: An interactive tour of the spanning tree protocol

Mon, 24 Aug 2026 15:00:00 +0000

Warning

This post contains interactive examples. To visualize and interact with them, you need to leave your RSS reader.

Imagine you rent office space for a three-day event. You quickly set up a few Ethernet switches and tape some cables on the floor to get everyone online. Unfortunately, Stan, your clumsiest coworker, kicks out a cable every time he gets up for coffee. You could add extra cables, but then youâ€™d get a broadcast storm: Ethernet packets that loop and multiply until nothing else gets through.

Thatâ€™s where the spanning tree protocol (STP) comes in. STP blocks just enough of your spare cables to leave a loop-free tree. When Stan strikes again, it rebuilds the tree in a second, leaving some time for Blobby, your one-person support crew, to reconnect the cable.¹ See for yourself: the diagram below runs a real STP implementation in your browser!

:demo

A1 @0,0 prio=4096
A2 @0,1
A3 @0,2
A4 @0,3

B1 @1,0 prio=8192
B2 @1,1
B3 @1,2
B4 @1,3

C1 @2,0 prio=8192
C2 @2,1
C3 @2,2
C4 @2,3

A1 -- A2 hazard=0
A2 -- A3 hazard=0
A3 -- A4 hazard=0
B1 -- B2
B2 -- B3
B3 -- B4
C1 -- C2 hazard=0
C2 -- C3 hazard=0
C3 -- C4 hazard=0

A1 -- B1 cost=10
B1 -- C1 cost=10
A4 -- B4 cost=20
B4 -- C4 cost=20

Leo @-0.3,0.7 proto=none icon=ğŸ‘¦ğŸ�»
Mia @-0.3,1.3 proto=none icon=ğŸ‘§ğŸ�½
Joy @0.3,0.7  proto=none icon=ğŸ‘±ğŸ�»â€�â™€ï¸�
Roy @0.3,1.3  proto=none icon=ğŸ‘¨ğŸ�¾
A2 -- Leo hazard=0 A2:edge
A2 -- Mia hazard=0 A2:edge
A2 -- Joy hazard=0 A2:edge
A2 -- Roy hazard=0 A2:edge

Max @-0.3,1.7 proto=none icon=ğŸ‘¨ğŸ�½
Zoe @-0.3,2.3 proto=none icon=ğŸ‘©ğŸ�¾
Ada @0.3,1.7  proto=none icon=ğŸ‘µğŸ�¾
Amy @0.3,2.3  proto=none icon=ğŸ‘©ğŸ�¼
A3 -- Max hazard=0 A3:edge
A3 -- Zoe hazard=0 A3:edge
A3 -- Ada hazard=0 A3:edge
A3 -- Amy hazard=0 A3:edge

Eli @0.7,0.7 proto=none icon=ğŸ‘¦ğŸ�¼
Jay @0.7,1.3 proto=none icon=ğŸ‘¨ğŸ�»
Kai @1.3,0.7  proto=none icon=ğŸ§‘ğŸ�½
Ben @1.3,1.3  proto=none icon=ğŸ‘±ğŸ�¼
B2 -- Eli hazard=0.2 B2:edge
B2 -- Jay hazard=0.2 B2:edge
B2 -- Kai hazard=0.2 B2:edge
B2 -- Ben hazard=0.2 B2:edge

Ava @0.7,1.7 proto=none icon=ğŸ‘©ğŸ�»
Lea @0.7,2.3 proto=none icon=ğŸ§‘ğŸ�¾â€�ğŸ¦±
Ivy @1.3,1.7  proto=none icon=ğŸ§•ğŸ�½
Rex @1.3,2.3  proto=none icon=ğŸ‘´ğŸ�¿
B3 -- Ava hazard=0.2 B3:edge
B3 -- Lea hazard=0.2 B3:edge
B3 -- Ivy hazard=0.2 B3:edge
B3 -- Rex hazard=0.2 B3:edge

Ana @1.7,0.7 proto=none icon=ğŸ‘©ğŸ�¿
Eve @1.7,1.3 proto=none icon=ğŸ‘§ğŸ�¼
Abe @2.3,0.7  proto=none icon=ğŸ§“ğŸ�¿
Ian @2.3,1.3  proto=none icon=ğŸ§”ğŸ�¾
C2 -- Ana hazard=0 C2:edge
C2 -- Eve hazard=0 C2:edge
C2 -- Abe hazard=0 C2:edge
C2 -- Ian hazard=0 C2:edge

Ned @1.7,1.7 proto=none icon=ğŸ‘¨ğŸ�¼â€�ğŸ¦³
Lou @1.7,2.3 proto=none icon=ğŸ§‘ğŸ�¿
Fay @2.3,1.7  proto=none icon=ğŸ‘§ğŸ�»
Sue @2.3,2.3  proto=none icon=ğŸ‘©ğŸ�½â€�ğŸ¦°
C3 -- Ned hazard=0 C3:edge
C3 -- Lou hazard=0 C3:edge
C3 -- Fay hazard=0 C3:edge
C3 -- Sue hazard=0 C3:edge

Note

This article is also available as a video, but I advise you to keep reading here to try the interactive demonstrations.

The basics

Designed in the â€™80s, the spanning tree protocol has evolved into a â€œrapidâ€� flavor (RSTP) and a â€œVLAN-awareâ€� variation (MSTP).² Any sound-minded network engineer knows there are better alternatives, like BGP EVPN VXLAN. Yet, because any switch speaks it, the venerable spanning tree protocol still fills a niche.

We focus on RSTP: it replaced the original protocol in 2004. To eliminate network loops, RSTP implements a complex state machine. Timers, link state changes, and the link-local control frames a bridge receives from its neighbors drive its transitions. These Ethernet frames are the Bridge Protocol Data Units (BPDUs). You can watch them in action below: hit the â€œStartâ€� button.

:protocol rstp
:tx-hold 10

A1 @0,1
C11 @1,0 prio=4096 icon=ğŸŒ³
C12 @1,2 prio=4096 icon=ğŸŒ³
C21 @2,0 prio=4096 icon=ğŸŒ³
C22 @2,2 prio=4096 icon=ğŸŒ³
A2 @3,1

H1 @0,0.2 proto=none icon=ğŸ’»
H2 @0,1.8 proto=none icon=ğŸ–¨ï¸�
H3 @3,0.2 proto=none icon=ğŸ“ 
H4 @3,1.8 proto=none icon=ğŸ“º

A1 -- C11
A1 -- C12
A2 -- C21
A2 -- C22
C11 -- C12
C11 -- C21
C11 -- C21
C11 -- C22
C12 -- C21
C12 -- C22
C21 -- C22
A1 -- H1 A1:edge
A1 -- H2 A1:edge
A2 -- H3 A2:edge
A2 -- H4 A2:edge

After some time, the topology converges to a tree: from the root C11, there is a path to each bridge³ and no loop. In the upper right corner, the interface displays a tree icon ğŸŒ³ followed by the time it took to reach this state. Cut a link and see how the protocol finds an alternate path to reach C12 in less than a second. You can stop the simulation, move it forward step by step, reset it to its initial state, or slow it down with the â€œsnailâ€� modeÂ ğŸ�Œ. Donâ€™t worry about all the displayed information: I explain it later.

All examples run in your browser, powered by MSTPDâ€”an open-source user-space⁴ implementation of RSTP.⁵

Historical interlude

Radia Perlman, an inductee of the Internet Hall of Fame in 2014, summarized the ancestor of STP she invented at DEC with this poem, later included in a US patent:

I think that I shall never see
A graph more lovely than a tree.
A tree whose crucial property
Is loop-free connectivity.
A tree which must be sure to span
So packets can reach every LAN.
First, the root must be selected.
By ID, it is elected.
Least cost paths from root are traced.
In the tree, these paths are placed.
A mesh is made by folks like me,
Then bridges find a spanning tree.

â€• Radia Perlman, Algorhyme.

Electing the root bridge

To build a tree, RSTP first elects the bridge with the lowest bridge identifier as the root bridge. The bridge identifier combines the priority and the MAC address: 8192.6e:2b:10:a0:5f:29.

In the example below, S1 and S2 have priorities of 4,096 and 8,192: S1 becomes root. S4 has a priority of 12,288, while S3 keeps the default priority of 32,768:⁶ S4 becomes root. S5 and S6 donâ€™t have a specific priority, so the lowest MAC address wins and S5 becomes root.

:protocol rstp

S1 @0,0 prio=4096
S2 @0,1 prio=8192
S1 -- S2

S3 @1,0
S4 @1,1 prio=12288
S3 -- S4

S5 @2,0
S6 @2,1
S5 -- S6

Initially, each bridge advertises itself as root:⁷

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    Root Identifier: 8192.02:00:00:01:00:01
    Bridge Identifier: 8192.02:00:00:01:00:01

Once a bridge receives a BPDU advertising a better root bridge, it propagates this new information to its neighbors.

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    Root Identifier: 4096.02:00:00:00:00:00
    Bridge Identifier: 8192.02:00:00:00:00:01

Assigning roles to ports

The second step is to assign a role to each port. RSTP defines five roles, each denoted by a letter:

root (R),
designated (D),
alternate (A),
disabled (X), or
backup (B).⁸

Each non-root bridge chooses its root port, the one with the lowest-cost path to the root. Unless you override it, each bridge derives the link cost from the speed: 20,000 for 1â€¯Gbps. In case of equality, the lowest port identifier wins.

Each remaining port becomes a designated port if the BPDU it sends is â€œbetterâ€� than the BPDU it receives. Otherwise, it becomes an alternate port. Later, if the root port goes down, the â€œbestâ€� alternate port becomes the new root port. The tiebreakers for the best BPDU are:

the lowest root bridge identifier,
the lowest accumulated cost to the root,
the lowest bridge identifier, and
the lowest port identifier.

:protocol rstp

S1 @1,0  prio=4096 icon=ğŸŒ³
S2 @0,1
S3 @2,1

S1 -- S2
S1 -- S3
S1 -- S3
S2 -- S3

In the example above, after convergence, S1 is the root bridge because it has a priority of 4,096, while the other bridges have a priority of 32,768. All its ports are designated ports because the accumulated cost to the root is 0.

S2â€™s port facing S1 becomes a root port because it has the lowest accumulated cost to the rootâ€”20,000 vs 40,000. S3 has two ports facing S1, and the one with the lowest port identifier becomes the root portâ€”0x8000 vs 0x8001. The other candidate is an alternate port because the remote port on the link sends a better BPDU, with an accumulated cost of 0. On the segment between S2 and S3, S2â€™s port wins: while both bridges have the same accumulated cost to the root (20,000), S2â€™s bridge identifier is smallerâ€”32768.02:00:00:00:00:01 vs 32768.02:00:00:00:00:02.

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 20000
    Bridge Identifier: 32768.02:00:00:00:00:01
    Port identifier: 0x8002

If you cut the active link between S1 and S3, S3 promotes the â€œbestâ€� alternate port to root port. If you also disable the second link, S3 chooses the remaining alternate port as a root port. But if you disable the link between S1 and S2, S2 needs a bit more work to elect a new root port because it does not have an alternate port.

Unless a specific event happens, designated ports send BPDUs every 2 seconds.⁹ If a bridge does not receive BPDUs from its neighbor for 3 consecutive hello periods, it considers the neighbor dead and removes the port information.

Port state transition

Each port can have one of three states. The diagram displays a background color for each state:

discarding (red),
learning (yellow), or
forwarding (green).

A root port transitions automatically to the forwarding state. An alternate port stays in the discarding state. A designated port has two options to transition from the discarding state to the forwarding state:

If the port is an edge port, either through configuration or because the remote device does not speak any flavor of STP, the bridge assumes the device wonâ€™t participate in the protocol and cannot create a loop. In this case, the designated port immediately transitions to the forwarding state.
Otherwise, it sends a proposal to its downstream neighbor. If the remote bridge agrees that the received BPDU is â€œbetterâ€� than any other BPDU stored for other ports, it elects the receiving port as its root port and starts the synchronization process: it transitions all non-edge non-synced designated ports to the discarding state to avoid a loop. Then, it sends back an agreement. Upon receiving the agreement, the peer designated port transitions to the forwarding state.¹⁰

:protocol rstp

S1 @1,0 prio=4096 icon=ğŸŒ³
S2 @1,1
S3 @0,2
S4 @2,2
S5 @0,3 prio=8192 icon=ğŸª¾
S6 @2,3
H1 @0,1.2   proto=none icon=ğŸ–¨ï¸�
H2 @2,1.2   proto=none icon=ğŸ“ 
H3 @2.5,1.3 proto=none icon=ğŸ“º
H4 @2.5,2.3 proto=none icon=ğŸ’»

S1 -- S2
S2 -- S3
S2 -- S4
S3 -- S5
S4 -- S6
S4 -- S3
S5 -- S6

S3 -- H1 S3:edge
S4 -- H2 S4:edge
S4 -- H3 S4:edge
S6 -- H4 S6:edge

In the topology above, H1, H2, H3, and H4 are end devices not participating in the protocol. We configure the ports they connect to as edge ports, so these ports immediately move to the forwarding state.

Use the â€œstepâ€� button to move the simulation forward. The clock moves to 1 second. Step again and S1 and S2 send a proposal to each other. Here is the proposal from S2:

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x4e, Agreement, Port Role: Designated, Proposal
        0... .... = Topology Change Acknowledgment: No
        .1.. .... = Agreement: Yes
        ..0. .... = Forwarding: No
        ...0 .... = Learning: No
        .... 11.. = Port Role: Designated (3)
        .... ..1. = Proposal: Yes
        .... ...0 = Topology Change: No
    Root Identifier: 32768.02:00:00:00:00:01
    Root Path Cost: 0
    Bridge Identifier: 32768.02:00:00:00:00:01
    Port identifier: 0x8001

S1 ignores it: its own root identifier is lower. When S2 receives a similar proposal from S1, it accepts S1 as its root bridge. It also elects the port to S1 as the root port and starts the synchronization process. The two designated ports are already discarding, so no change here. Step again and S2 sends two BPDUs to S1. In one of them, the agreement bit is 1 and the proposal bit is 0. It also shows that S2 accepted S1 as the root bridge and its root port is now in the forwarding state. When receiving this BPDU, S1 transitions its own designated port to the forwarding state. From this point, the link between S1 and S2 forwards user traffic.

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x79, Agreement, Forwarding, Learning, Port Role: Root, Topology Change
        0... .... = Topology Change Acknowledgment: No
        .1.. .... = Agreement: Yes
        ..1. .... = Forwarding: Yes
        ...1 .... = Learning: Yes
        .... 10.. = Port Role: Root (2)
        .... ..0. = Proposal: No
        .... ...1 = Topology Change: Yes
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 20000
    Bridge Identifier: 32768.02:00:00:00:00:01
    Port identifier: 0x8001

Letâ€™s look at what happened to S5. Reset the simulation and step twice. S5 exchanges BPDUs with both S3 and S6. Since S5 has a lower root identifier than S3 and S6, it stays the root bridge, while S3 and S6 accept the proposal and elect their root ports. S3 and S6 start the synchronization process. S6â€™s port to H4 keeps forwarding because it is an edge port. Move one step. Both S3 and S6 send an agreement back to S5, which transitions both designated ports to the forwarding state. Yet, the link between S5 and S3 keeps discarding user traffic! If you look carefully, S3â€™s port toward S5 is now a designated port, not a root port. During the same step, S3 also receives a better BPDU from S2 with S1 as the root bridge. It elects its port to S2 as the root port and downgrades the port to S5 to a designated port, which stays in the discarding state.

On the next step, things get a bit tricky. S3 sends a proposal to S5:¹¹

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x4f, Agreement, Port Role: Designated, Proposal, Topology Change
        0... .... = Topology Change Acknowledgment: No
        .1.. .... = Agreement: Yes
        ..0. .... = Forwarding: No
        ...0 .... = Learning: No
        .... 11.. = Port Role: Designated (3)
        .... ..1. = Proposal: Yes
        .... ...1 = Topology Change: Yes
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 40000
    Bridge Identifier: 32768.02:00:00:00:00:02
    Port identifier: 0x8002

S5 elects S1 as its root bridge and the port toward S3 as its root port. It starts its synchronization process, but the designated port to S6 does not move into the discarding state. Why? That port stays a designated port and its neighbor S6 has already sent an agreement on the link, so the port keeps its synced status.

Now, letâ€™s step back to look at what happens to S6. At this point, S6 believes S5 is the root bridge. Step once and S4 sends a new proposal to S6. S6 accepts the proposal, elects S1 as the root bridge and the port to S4 as its root port. The role of the port facing S5 changes: from a root port, it becomes a designated port. Because its peer keeps advertising an inferior BPDU on the link, this port becomes disputed and moves to the discarding state. The root port transitions to the forwarding state and the link starts forwarding immediately because S4â€™s designated port is already in the forwarding state. If we step one more time, S5 and S6 exchange two BPDUs. The one from S5 is better because of its lower bridge identifier. S5â€™s port stays a designated port, while S6 downgrades its own port to an alternate port.

Letâ€™s rewind to the start one last time: cut the link between S1 and S2, run the simulation until the topology is stable, stop the simulation, and restore the link between S1 and S2. During the first step, S1 and S2 exchange proposals. S2 elects S1 as the root bridge instead of S5 and the port to S1 as the root port. It downgrades the previous root port to a designated port and moves it into the discarding state. The other designated port stays synced and keeps its forwarding state. At the next step, S2 sends an agreement to S1 and the link between them starts forwarding user traffic. It also sends a proposal to S3, but not to S4. Instead, it sends a regular BPDU to S4. S4 still elects S1 as its root bridge and the port to S2 as its root port. It demotes its previous root port, the one to S3, to a designated port, which transitions to the discarding state because of the root port change. The other alternate port, to S6, also becomes a designated port and stays in the discarding state. The new root port moves to the forwarding state. On the next step, S4â€™s port to S3 settles as an alternate port after receiving a â€œbetterâ€� BPDU from S3.

RSTP is a giant state machine split into smaller ones: bridge detection, port information, port protocol migration, port role selection, port role transitions, port receive, port state transitions, port timers, port transmit, and topology change. Some of them are per bridge, some per port. Each bridge runs an instance. Time, operational port state changes, and the BPDUs it receives from other instances drive the transitions. Being event-driven makes RSTP more efficient but also more difficult to understand.

Placeholder for the Port Information state machine extracted from IEEEâ€¯802.1Q-2005, page 182. Pending IEEE authorization for reproduction, this is the blueprint for the Western Australian Government Railways class Msa Garratt articulated steam locomotive.

Topology change notification

A bridge populates a MAC address table: it associates each source MAC address with the port that last received it. When forwarding an Ethernet frame, it looks up this table to choose the right port.¹² When a link fails, a connected fridge reachable through one port may become reachable through another one. The affected bridges should flush the MAC addresses they learned, because these entries may now be wrong.

For this purpose, RSTP implements topology change notifications using a flooding mechanism. When a non-edge port transitions to the forwarding state, a bridge generates BPDUs with the topology change (TC) bit set. It sends them to all the non-edge designated ports and to the root port. It also flushes the MAC address table on these ports. When a bridge receives such a BPDU, it propagates the notification to all non-edge designated ports and the root port, except the one the notification came from. It also flushes the MAC address table on these ports. In the examples, the BPDUs with the TC bit set to 1 have a red circle.

:protocol rstp

S1 @1,0 prio=4096 icon=ğŸŒ³
S2 @0,1
S3 @1,1
S4 @2,1
S5 @1,2
LPT @0.1,2 proto=none icon=ğŸ–¨ï¸�

S1 -- S2
S1 -- S3
S1 -- S4
S2 -- S3
S2 -- S5
S4 -- S5
S5 -- LPT S5:edge

Start the simulation and wait a few seconds for the topology to settle. Stop the simulation and disable the link between S2 and S5. S5 elects the port facing S4 as the root port, which transitions immediately to the forwarding state. Step once and S5 emits a BPDU with the TC bit set to 1:

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x79, Agreement, Forwarding, Learning, Port Role: Root, Topology Change
        0... .... = Topology Change Acknowledgment: No
        .1.. .... = Agreement: Yes
        ..1. .... = Forwarding: Yes
        ...1 .... = Learning: Yes
        .... 10.. = Port Role: Root (2)
        .... ..0. = Proposal: No
        .... ...1 = Topology Change: Yes
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 40000
    Bridge Identifier: 32768.02:00:00:00:00:04
    Port identifier: 0x8002

S4 receives this BPDU. It flushes the MAC address table on the port facing S1: while LPT was previously reachable through this port, it is now reachable through S5. Step once. S4 sends S1 a BPDU with the TC bit set to 1. When S1 receives this BPDU, it flushes the MAC address table on the ports facing S2 and S3. Step once and S1 sends a notification to S2 and S3. Step once again and S2 sends a notification to S3, while S3 does nothing because the port toward S2 is an alternate port. S3 does not flush any MAC address table: LPT is still reachable through its port to S1.

If you step a bit more, you will see that some of the periodic BPDUs keep the TC bit set to 1. Each port runs a timer equal to the hello timer plus one second.¹³ The timer starts when the port emits a notification. Until it expires, the port sets the TC bit to 1 in every BPDU it sends. You can also see some periodic BPDUs without the TC bit: they originate from a port that only received a notification and therefore did not arm its timer.

Security

RSTP is weak against configuration errors and malicious actors. A bridge not talking RSTP can create a loop. An attacker can insert themselves into the topology to disrupt the service, spy on the traffic, or alter it.

To mitigate such problems, you need to identify the edge ports. An edge port connects to an end device, like a PC or a printer. Such devices do not generate BPDUs and cannot create a loop. RSTP defines two related flags:

When true, AdminEdge initializes a port as an edge port. It defaults to false.
When true, AutoEdge lets a port become an edge port when it does not receive BPDUs for 3 seconds. It defaults to true.

If an edge port receives a BPDU, regardless of the values of these two flags, it reverts to a non-edge port.

R0 @1.5,1.5 prio=8192

# AutoEdge=true, AdminEdge=false, bridge
S1 @3,1.58
R0 -- S1

# AutoEdge=true, AdminEdge=false, end device
H1 @2.84,2.18 icon=ğŸ–¨ï¸� proto=none
R0 -- H1

# AutoEdge=true, AdminEdge=true, bridge
S2 @2.18,2.84
R0 -- S2 R0:edge

# AutoEdge=true, AdminEdge=true, end device
H2 @1.58,3 icon=ğŸ’» proto=none
R0 -- H2 R0:edge

# AutoEdge=false, AdminEdge=true, bridge
S3 @0.68,2.76
R0 -- S3 R0:edge R0:no-auto-edge

# AutoEdge=false, AdminEdge=true, end device
H3 @0.24,2.32 icon=ğŸ“  proto=none
R0 -- H3 R0:edge R0:no-auto-edge

# AutoEdge=false, AdminEdge=false, bridge
S4 @0,1.42
R0 -- S4 R0:no-auto-edge

# AutoEdge=false, AdminEdge=false, end device
H4 @0.16,0.82 icon=ğŸ“º proto=none
R0 -- H4 R0:no-auto-edge

# Network port, bridge
S5 @0.82,0.16
R0 -- S5 R0:network S5:network

# Network port, end device
H5 @1.42,0 icon=â˜• proto=none
R0 -- H5 R0:network

# AdminEdge=true, bpdu-guard=true, bridge
S6 @2.32,0.24
R0 -- S6 R0:bpdu-guard R0:edge

# AdminEdge=true, bpdu-guard=true, end device
H6 @2.76,0.68 icon=ğŸ’¡ proto=none
R0 -- H6 R0:bpdu-guard R0:edge

In the topology above, S1, S2, S3, S4, S5, and S6 act as bridges, while H1, H2, H3, H4, H5, and H6 act as end devices:

S1 and H1 are on a port without a specific configuration: AutoEdge is true, AdminEdge is false,
S2 and H2 are on a port where AdminEdge is true,
S3 and H3 are on a port where AutoEdge is false and AdminEdge is true,
S4 and H4 are on a port where AutoEdge is false.

If you start the simulation and wait about 20 seconds, links to S1, S2, S3, S4, H1, H2, H3, and H4 eventually forward user traffic: none of the flags matter.

But what about the two remaining pairs? S5 and H5 connect to a network port. Such a port enables a non-standard feature: bridge assurance. The port transmits BPDUs regardless of its role. If it does not receive BPDUs for 3 consecutive hello periods, it transitions to the discarding state. On the link between R0 and S5, you can see BPDUs traveling in both directions, unlike the other links, where only designated ports send BPDUs.

S6 and H6 connect to a port where AdminEdge is true and BPDU guard is enabled. This is another non-standard feature that shuts down a port if it receives a BPDU.

In summary, if you expect a port to be an edge port, you should set AdminEdge to true and enable BPDU guard. Otherwise, declare it as a network port.

Why RSTP today?

A compelling use case for RSTP today is an out-of-band network for a datacenter, since you can tolerate an outage of a few seconds. The configuration is minimal and you can use cheap switches, like a Cisco 2960X.¹⁴ You need two switches acting as root bridges, and you build several loops to connect OOB switches in each cabinet. This simple design survives one failure on each loop.¹⁵

:protocol rstp
:tx-hold 10

# Root bridges
R1 @0,1 prio=0
R2 @0,2 prio=4096
R1 -- R2 cost=200 R1:network R2:network
R1 -- R2 cost=200 R1:network R2:network

# First loop
C1  @1,0 icon=ğŸ—„ï¸�
C4  @2,0 icon=ğŸ—„ï¸�
C7  @3,0 icon=ğŸ—„ï¸�
C10 @4,0 icon=ğŸ—„ï¸�
C12 @5,0 icon=ğŸ—„ï¸�
C13 @5,3 icon=ğŸ—„ï¸�
C15 @4,3 icon=ğŸ—„ï¸�
C18 @3,3 icon=ğŸ—„ï¸�
C21 @2,3 icon=ğŸ—„ï¸�
C24 @1,3 icon=ğŸ—„ï¸�
R1  -- C1  R1:network C1:network
C1  -- C4  C1:network C4:network
C4  -- C7  C4:network C7:network
C7  -- C10 C7:network C10:network
C10 -- C12 C10:network C12:network
C12 -- C13 C12:network C13:network
C13 -- C15 C13:network C15:network
C15 -- C18 C15:network C18:network
C18 -- C21 C18:network C21:network
C21 -- C24 C21:network C24:network
C24 -- R2  C24:network R2:network

# Second loop
C2  @1,0.5 icon=ğŸ—„ï¸�
C5  @2,0.5 icon=ğŸ—„ï¸�
C8  @3,0.5 icon=ğŸ—„ï¸�
C11 @4,0.5 icon=ğŸ—„ï¸�
C14 @4,2.5 icon=ğŸ—„ï¸�
C17 @3,2.5 icon=ğŸ—„ï¸�
C20 @2,2.5 icon=ğŸ—„ï¸�
C23 @1,2.5 icon=ğŸ—„ï¸�
R1  -- C2  R1:network C2:network
C2  -- C5  C2:network C5:network
C5  -- C8  C5:network C8:network
C8  -- C11 C8:network C11:network
C11 -- C14 C11:network C14:network
C14 -- C17 C14:network C17:network
C17 -- C20 C17:network C20:network
C20 -- C23 C20:network C23:network
C23 -- R2  C23:network R2:network

# Third loop
C3  @1,1 icon=ğŸ—„ï¸�
C6  @2,1 icon=ğŸ—„ï¸�
C9  @3,1 icon=ğŸ—„ï¸�
C16 @3,2 icon=ğŸ—„ï¸�
C19 @2,2 icon=ğŸ—„ï¸�
C22 @1,2 icon=ğŸ—„ï¸�
R1  -- C3  R1:network C3:network
C3  -- C6  C3:network C6:network
C6  -- C9  C6:network C9:network
C9  -- C16 C9:network C16:network
C16 -- C19 C16:network C19:network
C19 -- C22 C19:network C22:network
C22 -- R2  C22:network R2:network

This topology converges in about 6 seconds. Each loop should stay small (around 16 bridges) to reduce the probability of a double failure and to avoid sharing too much bandwidth. The design can evolve a bit without adding too much complexity: one VLAN per loop or one bridge domain per loop.

How large can a network be?

The maximum age, whose default value is 20, governs the maximum distance of a bridge from the root. The topology below is too big for BPDUs from R1 to reach beyond S20.¹⁶

:protocol rstp
:tx-hold 10
:max-age 20

R1 @0,0 prio=4096 icon=ğŸŒ³
R2 @0,5 prio=4096 icon=ğŸª¾

S1  @1,0
S2  @2,0
S3  @3,0
S4  @4,0
S5  @5,0
S6  @6,0

S7  @6,1
S8  @5,1
S9  @4,1
S10 @3,1
S11 @2,1
S12 @1,1

S13 @1,2
S14 @2,2
S15 @3,2
S16 @4,2
S17 @5,2
S18 @6,2

S19 @6,3
S20 @5,3
S21 @4,3
S22 @3,3
S23 @2,3
S24 @1,3

S25 @1,4
S26 @2,4
S27 @3,4
S28 @4,4
S29 @5,4
S30 @6,4

S31 @6,5
S32 @5,5
S33 @4,5
S34 @3,5
S35 @2,5
S36 @1,5

R1  -- S1
S1  -- S2
S2  -- S3
S3  -- S4
S4  -- S5
S5  -- S6
S6  -- S7
S7  -- S8
S8  -- S9
S9  -- S10
S10 -- S11
S11 -- S12
S12 -- S13
S13 -- S14
S14 -- S15
S15 -- S16
S16 -- S17
S17 -- S18
S18 -- S19
S19 -- S20
S20 -- S21
S21 -- S22
S22 -- S23
S23 -- S24
S24 -- S25
S25 -- S26
S26 -- S27
S27 -- S28
S28 -- S29
S29 -- S30
S30 -- S31
S31 -- S32
S32 -- S33
S33 -- S34
S34 -- S35
S35 -- S36
S36 -- R2
R1  -- R2 cost=200 down

Once the topology settles, part of the network considers R1 the root, while the other votes for R2. At the boundary, S20 tries to start a synchronization with S21 to move its designated port to the forwarding state. The BPDU looks like this:

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x4e, Agreement, Port Role: Designated, Proposal
    Root Identifier: 4096.02:00:00:00:00:00
    Root Path Cost: 400000
    Bridge Identifier: 32768.02:00:00:00:00:15
    Port identifier: 0x8002
    Message Age: 20
    Max Age: 20

S21 rejects it because the message age equals the maximum age. On the other hand, the BPDU S21 sends to S20 looks like this:

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x7c, Agreement, Forwarding, Learning, Port Role: Designated
    Root Identifier: 4096.02:00:00:00:00:01
    Root Path Cost: 320000
    Bridge Identifier: 32768.02:00:00:00:00:16
    Port identifier: 0x8001
    Message Age: 16
    Max Age: 20

This is not enough to change S20â€™s root port because S20 has a lower root identifierâ€”4096.02:00:00:00:00:00 vs 4096.02:00:00:00:00:01.

Fixing the link between R1 and R2 resolves the issue. The maximum message age any packet carries is now 18, below the configured maximum age. But it only works until another link breaks. A plausible fix is to increase the maximum age to 40.¹⁷

How fast is RSTP?

RSTP usually converges in a couple of seconds at startup. It often repairs a tree in less than a second. Even the 38-bridge topology takes less than 10 seconds to converge.¹⁸ Some topologies can take a bit more time to recover when the root bridge becomes unavailable.¹⁹

:protocol rstp

R0 @1,0 prio=0
S1 @1,1 prio=4096
S2 @0,2 prio=8192
S3 @2,2

R0 -- S1
S1 -- S2
S2 -- S3
S3 -- S1

In the topology above, start the simulation, wait for convergence, hit stop, and cut the link between R0 and S1. The topology is already optimal, but RSTP has a hard time converging again.

First, S1 loses its root port. It has no more information about R0 and elects itself as the root bridge. It keeps its ports to S2 and S3 as designated ports in the forwarding state. Step once and it sends a BPDU to both S2 and S3 to let them know about the root change. When receiving it, S2 accepts S1 as its root because it does not have a better root on another port. It elects the port to S1 as its root port. The other port stays a designated port. Both ports keep forwarding.

When receiving the BPDU from S1, S3 behaves differently: it knows R0 as a better root than S1 through its alternate port to S2. It promotes this port to a root port and demotes the port facing S1 to a designated port, which requires a new agreement. Step once and S3 sends a proposal to S1 with R0 as the root bridge. S1 elects R0 as the root bridge and promotes its port to S3 as a root port.

During the same step, S3 also receives a BPDU from S2 stating that S1 is the root bridge. Therefore, S3 has no port left with R0 as the root bridge: it elects S1 as the root bridge and its port to S2 as the root port. Step once and its next BPDU to S1 includes this information: S1 elects itself again as the root bridge. But during the same wave, S1 sends a proposal to S2 with R0 as the root bridge. While S1 and S3 agree that S1 is the root bridge, S2 now believes this is R0! In turn, S2 again convinces S3 that R0 is the root bridge, S3 convinces S1, S1 convinces S2, and S2 convinces S3.

This could go on forever, but it does not. The BPDUs saying â€œR0 is rootâ€� eventually age out when the message age goes past the maximum age. In the example above, at the eleventh second, S2 sends a BPDU to S3 with R0 as root, but S3 drops it because its message age reached the maximum. With some luck, the topology can also converge faster if a port stops transmitting new BPDUs after tripping the transmit hold count, whose default value is 6 per second.

About MSTP

MSTP is the â€œVLAN-awareâ€� version of RSTP: it runs several instances of RSTP and lets the administrator map each VLAN to a specific instance. For example, you can map VLANsâ€¯100 to 200 to a first instance, and 300 to 400 to a second instance. The remaining VLANs map to a special instance named the Internal Spanning Tree (IST). MSTP adds its own complexity, but the gist is that you have several logical topologies acting independently. If you want to dig deeper, have a look at â€œMSTP Tutorial Part I: Inside a Region.â€�

About the interactive examples

The interactive examples run MSTPD directly in your browser, compiled to WebAssembly with emscripten. A C API replaces the code talking to the Linux kernel: it manages bridges and ports, exports state as JSON, and drives time deterministically. A JavaScript wrapper makes it more user-friendly:

import { loadMSTPD } from "./dist/mstpd.mjs";
const mstp = await loadMSTPD();

// Create 3 bridges
const a = mstp.createBridge("A", { priority: 4096 });
const b = mstp.createBridge("B", { priority: 8192 });
const c = mstp.createBridge("C");

// Each bridge has two ports
const a1 = a.addPort("a-b", { portno: 1 });
const a2 = a.addPort("a-c", { portno: 2 });
const b1 = b.addPort("b-a", { portno: 1 });
const b2 = b.addPort("b-c", { portno: 2 });
const c1 = c.addPort("c-a", { portno: 1 });
const c2 = c.addPort("c-b", { portno: 2 });

// Build a triangle topology
mstp.link(a1, b1);
mstp.link(a2, c1);
mstp.link(b2, c2);

// Enable all bridges and ports
for (const br of [a, b, c]) br.enable();
for (const p of [a1, a2, b1, b2, c1, c2]) p.enable();

// Execute 40 seconds' worth of wall clock and display the topology
mstp.step(40);
console.log("Topology:", mstp.topology());

Several dozen unit tests explore the features of MSTPD and check that they work correctly in this environment:

$ node --test *.test.mjs
âœ” two bridges: lower priority becomes root (41.657342ms)
âœ” triangle loop: exactly one port blocks and all agree on the root (5.832ms)
âœ” breaking the active link reconverges and restoring recovers (18.730753ms)
[â€¦]
â„¹ tests 40
â„¹ pass 40
â„¹ fail 0
[â€¦]
â„¹ duration_ms 396.190897

Additional JavaScript code looks for specific <pre> blocks containing a topology definition and turns them into interactive widgets. You can inspect and modify the definition by hitting the â€œeditâ€� button.

There is also a cool trick to tell whether the topology has converged. After each step, we save a snapshot of the simulation memory, play 50 secondsâ€™ worth of simulation to check if the topology is stable, and travel back in time by restoring that snapshot. ğŸ•°ï¸�

The complete code lives on GitHub. I am happy with the result. It can be difficult to follow everything happening during a single step, but stepping forward and backward helps. I plan to use the same approach in future blog posts about networking features.

Note

Michael Lynch reviewed a first draft of this article. He authored â€œRefactoring English,â€� a book to sharpen your writing for blog posts, documentation, commit messages, and tutorials. Any errors are still mine!

The sprites for Stan and Blobby come from Craftpix, the coffee cups from Yanin.Â â†©
STP was introduced in IEEEâ€¯802.1D-1990. It is still present in IEEEâ€¯802.1D-1998 but was withdrawn in IEEEâ€¯802.1D-2004 in favor of RSTP, introduced in IEEEâ€¯802.1w-2001. MSTP was introduced in IEEEâ€¯802.1s-2002 and merged into IEEEâ€¯802.1Q-2003. Both of them are part of IEEEâ€¯802.1Q-2022 along with SPB~~â€”a protocol I had never heard of until writing this article~~.Â â†©
From here, I use â€œbridgeâ€� instead of the more common word â€œswitch.â€�Â â†©
The Linux kernel only runs STP. It delegates the other protocols to user space.Â â†©
MSTPD implements the state machine from IEEEâ€¯802.1Q-2005, but on Linux it runs RSTP only. Linuxâ€¯5.18 added support for forwarding multiple spanning tree, but MSTPD does not use it yet. See PR #150 for progress on this front.Â â†©
The priority is a multiple of 4,096: with MSTP, the lower 12 bits of the bridge priority encode the MST instance identifier, leaving only the upper 4 bits for the configured priority.Â â†©
To inspect the BPDUs crossing a link, select it, click the â€œDownload packetsâ€� button, and open the file with Wireshark.Â â†©
A backup port only exists if the bridge has several ports on the same collision domain. This should not happen in a switched network.Â â†©
This is the value of the â€œhelloâ€� timer. It used to be configurable, but IEEEâ€¯802.1Q-2005 pins it to 2. MSTPD does not allow another value.Â â†©
If the peer port does not receive an agreement after the hello timer elapsesâ€”or the maximum age if the port has just come upâ€”it falls back to the timer-based method for compatibility with STP: it transitions to the learning state, waits again for the hello timer to expire, and transitions to the forwarding state.Â â†©
As in many proposals, S3 also sets the agreement bit to 1. The proposal bit says â€œI am the designated port on this link and I want to transition to the forwarding state.â€� The agreement bit says â€œI am already in sync with the rest of my bridge on this root information.â€� Both can be true.Â â†©
If it finds no entry, the bridge duplicates the Ethernet frame on all ports, except the incoming one. The same happens if the destination MAC address is the broadcast one (ff:ff:ff:ff:ff:ff). This behavior bootstraps the learning process.Â â†©
This timer makes RSTP resistant to packet loss.Â â†©
You can get them for less than US$100 through a broker. All the ports run PVST+ by default and automatically fall back to plain RSTP.Â â†©
An alternative would be Ethernet Ring Protection Switching (ERPS)â€”another protocol I had never heard of until researching this article.Â â†©
If you look closely at what happens at t=2s, you can see that R2 is gaining popularity as root: S17 to S36 believe R2 is the root bridge. S16 does not follow because we hit the maximum age. Later, S17 to S20 reverse their position. Iâ€™ll let you explore the state of the various bridges to understand the root cause.Â â†©
When increasing the maximum age to 40, you also need to increase the forward delay to 21 (:forward-delay 21), as the standard enforces this condition: 2 Ã— (Forward Delay âˆ’ 1) â‰¥ Max Age. For this specific topology, you could also increase the maximum age to 37 and forward-delay to 20.Â â†©
The simulation may seem slow, but it does not run in real time. Look at the current timestamp in the upper right corner to know the wall clock, e.g. â€œt=8s.â€� Once the topology stabilizes, the same corner shows the convergence time, e.g. â€œğŸŒ³â€‰2s.â€�Â â†©
Khaled Elmeleegy, Alan Cox, and Eugene Ng formalized this phenomenon in â€œOn Count-to-Infinity Induced Forwarding Loops in Ethernet Networksâ€� and later in â€œUnderstanding and Mitigating the Effects of Count to Infinity in Ethernet Networks.â€� They propose a fix that did not find its way into a standard.Â â†©

Vincent Bernat: A non-interactive tour of the spanning tree protocol

Mon, 24 Aug 2026 14:59:00 +0000

Imagine you rent office space for a three-day event. You quickly set up a few Ethernet switches and tape some cables on the floor to get everyone online. Unfortunately, Stan, your clumsiest coworker, kicks out a cable every time he gets up for coffee. Spare cables would fix that, but a loop turns into a broadcast storm: Ethernet packets multiply until nothing else gets through. That’s where the spanning tree protocol comes in: it blocks just enough of the spare cables to leave a loop-free tree, and rebuilds it in a second each time Stan strikes again.¹

This content is also available as a text version, with interactive demos that run a real implementation directly in your browser!

This video is an experiment.² Honestly, except for Radia Perlman reading her poem,³ you should read the original article instead. It presents the same content, but you can play with the interactive examples, which are the main contribution. On the other hand, if you happen to like the video, be sure to tell me in the comments!

The sprites for Stan and Blobby come from Craftpix. The background music is “Sonatina No. 2 in G Major – III. Allegro” by Aaron Dunn. ↩
I thought automated tools would produce this video in a couple of hours. In the end, it was another rabbit hole and it took me more than 12. ↩
The audio was extracted from a Youtube video and cleaned up. ↩

David Bremner: Reproducing Org mode configuration

Mon, 24 Aug 2026 10:54:18 +0000

Context

Recently I was trying to reproduce a bug with citeproc.el and org-mode in emacs.

I thought I could use package-vc-install to install a set of upstream emacs packages at fixed versions, and thereby let citeproc upstream test in the same environment as I have.

It turns out that getting emacs to load the non-builtin version of org via package-vc-install did not work because

org-mode needs to run make after cloning
once package.el was initialized, I always seemed to end up with the built in org-mode (yeah, I realize that isn't an explanation).

Recipe part 1: get org

Here you can replace 9.8.7 with any other tagged release

  EMACSHOME=$(mktemp -d)
  git clone https://git.sr.ht/~bzg/org-mode ${EMACSHOME}/org
  git -C ${EMACSHOME}/org reset --hard release_9.8.7 
  make -C ${EMACSHOME}/org autoloads
  emacs -Q --batch -L ${EMACSHOME}/org/lisp --eval "(progn (require 'org) (message (org-version)))"

This should print 9.8.7, not the version of built in org-mode.

Recipe part 2: add-on packages

Now to test some add-on packages, run

    emacs -Q --init-directory ${EMACSHOME} -L ${EMACSHOME}/org/lisp

  (progn
    (require 'org)
    (package-initialize)
    (package-vc-install "https://github.com/emacs-straight/queue")
    (package-vc-install "https://github.com/joostkremers/parsebib" "6.7")
    (package-vc-install "https://github.com/rejeep/f.el" "0.21.0")
    (package-vc-install "https://github.com/magnars/s.el" "1.13.0")
    (package-vc-install "https://github.com/akicho8/string-inflection" "1.0.16")
    (package-vc-install "https://github.com/andras-simonyi/citeproc-el" "0.9.5"))

You can then run your tests in that emacs right away, or restart the environment with

  emacs -Q --init-directory ${EMACSHOME} -L ${EMACSHOME}/org/lisp

Russ Allbery: Long delayed haul

Sun, 23 Aug 2026 21:29:00 +0000

I haven't made a new book haul post in I don't know how long, so a lot of books have piled up and many have already been reviewed. Here's the overdue catch-up in case anyone is curious what books I am finding interesting before the reviews get posted.

Ilona Andrews — Magic Bites (sff)
Elizabeth Bear — In the House of Aryaman, a Lonely Signal Burns (sff)
Oliver Burkeman — Four Thousand Weeks (non-fiction)
Miles Cameron — Whalesong (sff)
Lee Child — Killing Floor (thriller)
august clarke — The Felicity Complex (sff)
Alison Cochrun — Here We Go Again (romance)
Dan Davies — The Unaccountability Machine (non-fiction)
Linzi Day — Midlife in Gretna Green (sff)
Linzi Day — Painting the Blues in Gretna Green (sff)
Linzi Day — Ties that Bond in Gretna Green (sff)
Linzi Day — Spilling the Tea in Gretna Green (sff)
Michelle Diener — Dark Ambitions (sff)
Michelle Diener — Dark Class (sff)
Michelle Diener — Collision Course (sff)
Michelle Diener — Crash Course (sff)
Henry Farrell — Underground Empire (non-fiction)
Kathleen A. Flynn — The Jane Austen Project (sff)
Victoria Goddard — The Hands of the Emperor (sff)
James Herriot — All Creatures Great and Small (mainstream)
James Herriot — All Things Bright and Beautiful (mainstream)
James Herriot — All Things Wise and Wonderful (mainstream)
James Herriot — The Lord God Made Them All (mainstream)
James Herriot — Every Living Thing (mainstream)
Lauren Hough — Monster of a Land (non-fiction collection)
Bethany Jacobs — This Brutal Moon (sff)
Guy Gavriel Kay — Written on the Dark (sff)
Mary Robinette Kowal — The Martian Contingency (sff)
Ann Leckie — Radiant Star (sff)
C.B. Lee — Coffeeshop in an Alternate Universe (sff)
Fonda Lee — The Last Contract of Isako (sff)
Julie Leong — The Teller of Small Fortunes (sff)
Julie Leong — The Keeper of Magical Things (sff)
R.Z. Nicolet — The Cloak and Its Wizard (sff)
Claire North — Slow Gods (sff)
Rebecca Ore — Writing's Writing (non-fiction collection)
Suzanne Palmer — Ode to the Half-Broken (sff)
Gareth L. Powell — Fleet of Knives (sff)
Cameron Reed — What We Are Seeking (sff)
Beth Revis — Full Speed to a Crash landing (sff)
Beth Revis — How to Steal a Galaxy (sff)
Beth Revis — Last Chance to Save the World (sff)
Natalie Zina Walschots — Villain (sff)
Jo Walton — Everybody's Perfect (sff)
Martha Wells — Platform Decay (sff)
James White — The Galactic Gourmet (sff)
James White — Final Diagnosis (sff)

The James Herriot books were ones my parents were getting rid of. I have them marked as mainstream fiction as a short-hand since "fictionalized autobiography" seemed like too much of a mouthful.

Colin Watson: GSS-API support split out from main Debian OpenSSH packages

Sun, 23 Aug 2026 17:16:11 +0000

In an option review I did in 2024, shortly after the xz-utils backdoor, I explained that having GSS-API authentication and key exchange support in the main OpenSSH packages is problematic. The key exchange patch is large and intrusive. Furthermore, even linking to the necessary libraries is not without risk: as the Ebury malware attack demonstrated way back in 2009, each extra library linked into security-critical daemons such as sshd (or nowadays into its privilege-separated helper programs) can modify the behaviour of the daemon even if you aren’t doing anything that would involve calling into that library. Of course some of that risk remains, but as Damien Miller wrote, minimizing the number of libraries that end up in the address space of sshd and friends is still valuable.

I just uploaded openssh 1:10.4p1-5 to unstable, completing this split. As of this version, the OpenSSH client and server are built without GSS-API authentication and key exchange support. If you need those features, install openssh-client-gssapi or openssh-server-gssapi instead, as appropriate. Debian 13 (trixie) already has packages with those names that just depend on the regular openssh-client and openssh-server so that you can pre-emptively install them, as documented in the release notes.

The new openssh-*-gssapi packages have relatively tight dependencies on openssh-common, in order for the testing migration system to ensure that we can’t forget to keep them up to date. This will mean a bit more ongoing work for me on each new upstream version, but I think it will be manageable.

Iustin Pop: Another optimistic take on AI

Sun, 23 Aug 2026 16:19:40 +0000

Disclaimers

The current discussion in Debian around the AI GR is very heated, and I won’t add to that, however, I am very confused about some of the viewpoints there. But, I had no idea how to even try to write this, so did shut up, until I saw Aigars’ excellent Optimistic take on AI, which motivated me to try, at least. For the record, I fully subscribe to the post, and to the voting suggestions (and I just voted).

Also, for full disclosure, I don’t think I did any contribution to Debian until now using AI, neither packaging, nor emails, nor bug reports. And this blog post specifically is 100% hand written.

With that out of the way… there are two points I want to make in this post.

AI is useful, even if it has risks

First is, that even if we could put the genie back in the metaphorical bottle, we should not. We do need to continue working towards safe AI, and efficient AI (less environmental impact), but we should not work towards removing the usage of AI. There are already significant advancements in sciences and technology thanks to the use of AI, so desiring AI to not exist (assuming we had a magical wand) is the wrong approach.

Sure, AI has significant risks — and I can see ways in which AI can do significant damage to society — but I don’t think we can go from Kardashev I to II without the use of AI, and definitely not to III. And I think, that should be the goal.

A few simple examples: Do we want to rollback all the 20 years old security issues that AI found? Do we want to rollback the recent Moderna cancer findings? Do we want to rollback the concept of “extremely large scale pattern matchings”, just because it runs on chips and no longer in one person’s head?

Reading Debian lists

The second point is, lately I found less and less enjoyment in reading Debian lists. Even with that already being the case, I feel so disconnected from many of the opinions being voiced in this discussion.

On one hand, it’s normal and healthy that people have different opinions, disagree, and move forward.

On the other hand, looking at one of the proposed options:

“Moderators and disciplinary teams may make narrow and tailored exceptions to rule 4, and decide on interpretation”.
“Violations of these requirements should be treated as violations of the relevant Code of Conduct and should result in swift and proportionate disciplinary action”.

I already knew Debian, and some large parts of the OSS world, is left leaning. But those phrasings, to me, are too close to socialism/communism. As someone who grew up under communism, this is a much more slippery slope (disciplinary teams? really?) than AI usage. Ask me in person for more details.

So, it is possible that Debian continues to evolve in such a way that I don’t find myself in any way close to its ongoing culture. I will be sad at that point, but it will be what it is.

Where to?

I think that, until such a time that an AI bubble bursts, what any organization should do is try to logically see where and if AI can help. And in an organization that is about computer software, I see hundreds of places that are subject to very large scale pattern matching… so the half of the discussion is, to me, mind-boggling.

To be clear, it’s not about “if you can’t beat them, join them”. As I wrote above, I think AI is useful, so the point is how to use it effectively.

Well, will see what Debian votes. I am half curious, half sad already.

Wouter Verhelst: Programming and GR 2026 002

Sun, 23 Aug 2026 14:03:13 +0000

Programming language generations

When I was young, I learned about a model of classifying programming language: the system of programming language generations.

In this model, first generation programming languages are, basically, where you program the computer in the language that is defined by its architecture. On a Von Neumann machine, with its load-and-store architecture, you do that by inputting a string of numbers. The first programmer in human history -- her name was Ada Lovelace -- wrote in a first-generation language. 1GLs aren't so much invented as they are a byproduct of the computers for which they're created.

Second-generation languages are the assembler languages. Because humans are not computers, and because decoding long lines of numbers to understand what the computer is doing, when programming became a full-time job, the programmers that did it decided that doing all this assembling manually is too complicated, so they quickly wrote assemblers to automate the process for them. They still could understand the 1GL output of the 2GL assembler, but most of them quickly forgot how to write software in a first-generation language. Not that anyone cared, as the translation from a 2GL to a 1GL is lossless and you can just revert it.

Third-generation languages are higher-level languages. When the first 3GLs were invented (such as COBOL and, more famously, FORTRAN) in the late 1950s and early 1960s, it was believed by some that the work of programming a computer so accessible to non-programmers that the job of programmer would eventually cease to exist, and people would just ask the computer what they needed by entering COBOL instructions. This of course was ridiculous and incorrect, because converting algorithms to computer instructions, whether at the 2GL or 3GL level, is a specialized skill that some automation can perhaps make simpler but never completely take away the need for. At the time, some people also felt to some extent that using 3GL wasn't the same thing as actually programming 3GLs, but eventually the world moved on and embraced things. The invention of 3GL environments reduced, but did not completely take away, the need for people to understand 2GLs, as compiler and operating system authors still need to understand them, and some highly optimized code still continues to be written in 2GLs to this day.

Fourth-generation languages abstract away some or all of the process of programming. For instance, a database-related 4GL will hide away the complexities of storing data in particular locations, how to fetch that data, how to index it such that you can fetch it efficiently, how to loop over the data to get you a summary of that data, and instead allows you to express the required information in an abstract way, expecing the computer to fill in the blanks. When SQL, an early 4GL, was invented, some people believed that the language made accessing databases so simple that the requirement to implement database applications would eventually cease to exist and we would just hand SQL prompts to users who need to access data. This of course was ridiculous and incorrect, because understanding data schemas and using that understanding to query data from a database is a specialized skill that perhaps a higher abstraction can help you make simpler, but that in the longer run it can never completely take away the need for. The invention of 4GLs also reduced, but did not completely take away, the need for people to understand how to do the things that the 4GLs automate for you manually, as the people who do write those things still need to understand them, and there are also environments where these particular 4GLs are rather not appropriate or just very slow.

The first definition of programming language generations that I read about in the 1980s simply stated that fifth-generation languages did not yet exist, but that they would in the future, and that in those, you would "tell the computer what to do, and it would then do that". Now that we have a way of doing so, it could be said that by some definition, we now actually do have a number of 5GLs. The existence of these LLM systems has caused some, especially the people who build and exploit these systems, to exclaim that programming as we know it today is going to cease to exist, and everyone will just ask an LLM to generate a program, which will then do so. That is of course ridiculous and incorrect, as no automaton can generate software from nothing; input is still required for the model to be able to produce something that approaches usability, and being able to word that input in a correct and productive fashion will be a skill that future programmers can benefit from. I ran some experiments a while back, and from that concluded that, if we look only at the technical side, LLM use can, in some niches, increase productivity for a programmer. There are certainly things that you shouldn't use an LLM for, but equally there can be cases where use of an LLM to perform some task that traditionally would have been done by a programmer would be a net positive.

But LLMs, as they exist today, are highly problematic.

They require vast amounts of data to build the model. The companies that build these models are disrespectful of people who run web services, and as a result, everyone now has to implement various types of application firewalls just to not make systems fall over from the overwhelming requests for data. They are also disregarding the licenses that are attached to these vast amounts of data, which makes me, as a person who believes in the tenets of free software, sad.

They require vast amounts of energy, causing an already-critical global warming crisis to, well, not improve.

They require vast amounts of coolant to dissipate the energy concentrated in their data centers, causing further environmental effects.

In this, they are problematic and to be avoided. But these are side states of the current state of affairs; I do not believe that they are inherently implied to be able to build and operate an LLM -- any LLM.

I guess it's fair to say that my feelings towards LLM usage are complex and many-faceted. I haven't been involved in many debates about the subject, debates that to me seem to be mostly focused on "LLM good" vs "LLM bad" arguments that aren't as nuanced as the position that I would believe is more accurate. This is not because I don't care, but partially because I've been busy in my personal life recently and partially because the whole thing seems somewhat disheartening.

But then Debian popped up GR 2026-002, meaning, I now have to come up with an opinion about various candidate statements in the context of the above, which is... not easy. But I did it anyway.

There are 8 choices on the ballot, and they all have some truth and some falsehood to them. My position about LLMs can be summarized as:

The current state of affairs wrt LLMs is disastrous and we should not encourage them
However, there's no technical reason why this must remain true for all time
And so any statement should keep in mind what might happen in the future and that the current disastrousness of the whole thing isn't guaranteed to continue to exist for all eternity.

With that, let's go over them.

GR vote options

Proposal A

Its summary, from the GR text:

This proposal aims to expressly forbid any contributions to Debian written with the use or assistance of large language models (LLMs) or other generative AI tools.

This falls squarely in the "LLM bad" camp, outlawing all generative-AI contributions, disregarding potential future ones where the problematic situations that exist today are not present.

It makes a change to the social contract, which is especially difficult to reverse (on purpose), and which therefore also will require a 3:1 supermajority, but if we want to ban LLM-assisted contributions, this is probably the best way to do it.

Proposal B

This one tries to allow AI-assisted contributions under certain conditions. It's mostly an "LLM good" proposal, with some caveats that can be discribed as "make sure you know what you're doing".

Proposal C

This proposal is both a weaker (in some places) and stronger (in other places) version of Proposal A. It makes changes to the code of conduct instead of to the social contract, and it also wants to, at least, suggest policy to parties beyond the Debian project. By not changing the social contract, however, it is more likely to reach its simple majority requirement than proposal A.

I don't think the language that it wants to add to the code of conduct is particularly well phrased, however.

Proposal D

This is a weaker form of proposal B. The language is more compact and there are a few requirements that are spelled out in proposal B that are not spelled out in proposal D, but if you read between the lines you'll see that the requirement is still there really and I don't understand why proposals B and D were not merged into one.

Proposal E

This proposal tries to hold a middle ground between "LLM good" and "LLM bad". It appreciates that things are quite muddled at the present time, and that perhaps the situation might might change in the future. It acknowledges that certain questions remain unanswered and that perhaps future considerations might therefore be different. But it essentially refuses to take a stance on whether LLMs should be accepted by the project or not.

Proposal F

Similar to proposal E, this proposal tries to discourage Debian contributors from using LLMs, while still allowing people to use it should they want to, but with some requests and requirements to mark LLM-assisted contributions to account for those people who don't want to interact with LLM-generated software. As such, it is a proposal similar to proposal E that leans closer to the "LLM bad" camp.

Proposal G

This proposal aims to ensure that contributions directly to Debian are created by humans, while at the same time avoiding restrictions on the tools those humans may choose to use when contributing

Another "LLM bad" proposal, it however restricts the "bad" bits to only the direct output of the LLM. If you use an LLM to do something and then clean-room re-implement the same thing yourself, that's apparently fine.

Proposal H

This proposal condemns the use of LLM for its environmental and moral problems, but explicitly not for its technical considerations. I feel that it is closest to my position as explained above.

Voting

Expressing a vote on a ballot so convoluted and complicated like this one takes time. I have to read and understand every ballot option, and formulate an order of them.

And I shouldn't just state which option has my preference; Debian's voting process allows a rich expression of opinion on ballot options.

Anyway, I eventually ended up voting in a way that I think is consistent with my opinion. But it wasn't easy.

Russell Coker: Links August 2026

Sat, 22 Aug 2026 16:05:00 +0000

This YouTube video about the Cashier Girl Meme is interesting in the context of AI systems that generate images of people and can communicate with people, hotter than any real human is an achievable goal [1].

Stand Up Maths has an interesting Youtube video about LLMs solving maths problems which I highly recommend watching (it does not require any real knowledge of maths), I think this opens the door to attacks on well established cryptologic systems [2].

Adam Conover made an insightful YouTube video about how and why Hollywood is now unable to make good sitcoms and why this is bad for society [3].

Sky Croeser wrote an interesting and insightful blog post about topics covered at the “Digital and sexual citizenship in an age of social media bans: Interrogating the rights of children and young people conference” [4].

Zane wrote a very informative blog post about reverse engineering a trojaned Android projector with Claude Code [5]. We need much better security on home networks to break the business model for this sort of thing.

Renee Stonebraker’s article “Puritans Wouldn’t Eat Pussy, So They Invented the Western” has a lot of interesting information about early days of colonising the US, and not much about eating pussy [6].

IFLScience has an interesting article about brinicles, icicles of brine that form under sea ice [7].

Nautilus has an interesting article about the Silurian Hypothesis [8].

The Conversation has an intersting article about the pros and cons of no-till farming [9].

Cold War is a 365tomorrows story about bio-warfare which raises several disturbing possibilities we need to guard against [10].

Scott Santens wrote an insightful article describing how a land value tax would reduce rent and solve the housing shortages [11].

Positive News has an interesting article about using OnlyFans to teach people about climate change [12].

The Conversation has an interesting article about cultural safety in healthcare, sounds good, and while we are at it lets deal with sexism [13].

Doctoreww has an interesting web page about ways of displaying different strings to humans and machines, this could result in you running a different command to what you thought you copied from a web site or defeating tools designed to block hostile content [14].

Cory Doctorow wrote an insightful article “Commentary Hell is Other People” about the way rich people want to use AI to replace all people [15]. Also psychologists who help rich people accept being greedy are worthy of a Luigi

The research article “Worship me at the office altar: Why narcissistic leaders resist remote work” is interesting, yet another reason to get rid of narcissistic executives [16].

Renew Economy has an interesting article about clean up costs for mining (which is usually left for the government to pay) and how this could impact renewable energy production facilities [17].

Elvira Bary wrote an insightful article on the Russian financial collapse that is happening now [18].

The Guardian has an interesting article about Afro-American women who travel to South Korea for healthcare because of problems with racism and sexism in American hospitals [19].

The Conversation has an interesting article about the potential for disabled people to be more productive in space than non-disabled people [20].

Krebs has an interesting article about LG banning residential proxy code from apps after the LG store was found to have such code in 42% of it’s apps [21].

Robert B Shpiner wrote an insightful article for The Guardian about the death of democracy in the US [22].

[1] https://www.youtube.com/watch?v=TCn7bA1eg_0
[2] https://www.youtube.com/watch?v=ZJ8KThKAfbs
[3] https://www.youtube.com/watch?v=gpSghW4J3ws
[4] https://tinyurl.com/28b6jclu
[5] https://zanestjohn.com/blog/reing-with-claude-code
[6] https://tinyurl.com/2y2b6a58
[7] https://tinyurl.com/2awjppn5
[8] https://tinyurl.com/2y89gftt
[9] https://tinyurl.com/2b27vo8o
[10] https://365tomorrows.com/2024/03/27/cold-war-2/
[11] https://tinyurl.com/2aqkn4z6
[12] https://tinyurl.com/2dy5vy8g
[13] https://tinyurl.com/25rld7v3
[14] https://doctoreww.github.io/EvilFontTool/
[15] https://tinyurl.com/2c2guu89
[16] https://tinyurl.com/27kq4qqu
[17] https://tinyurl.com/2blp477w
[18] https://tinyurl.com/27aybgge
[19] https://tinyurl.com/29covbrp
[20] https://tinyurl.com/27qdj6co
[21] https://tinyurl.com/22pxub8v
[22] https://tinyurl.com/25zf4hnp

Ian Jackson: Open Letter to the Wikimedia Foundation Board

Thu, 20 Aug 2026 20:58:51 +0000

I have just sent an open letter to the Board of the Wikimedia Foundation, the umbrella organisation for Wikipedia (and a number of other projects), expressing my support for Wiki Workers United and the unionisation effort by WMF staff.

Here is the letter:

To: Board of Trustees, Wikimedia Foundation

via Wikimedia_Foundation_Board_noticeboard and WWU
published at https://diziet.dreamwidth.org/21442.html

Re: My support for Wiki Workers United

Dear Trustees

Wikipedia has become one of the pillars of the free and open Internet. Across the world, reliable sources of information are under attack.

I'm proud to have played my very small part in the community of editors of English Wikipedia for the last 20 years. I am also proud of my contributions to the Free Software movement, including especially Debian. Debian, whose constitution and package installer I originally wrote, has become one of the technological foundations of the open Internet.

Unfortunately, there are signs that the Wikimedia Foundation is not performing its proper role as bulwark against attacks on democracy, including from moneyed interests. Recent events at WMF have been very alarming to me, and seem to form part of a disturbing trend.

As a Trustee Director of a UK charity myself, I understand that WMF Trustees must defend the interests of the Foundation. But that cannot mean taking actions that undermine the Foundation's mission. Nor can it mean the deplorable, and even dishonest, practices, that WMF appears to have been engaging in.

As a Wikipedian, as a Free Software activist, and as a citizen of the planet, I stand in solidarity with Wiki Workers United. Union- busting must stop immediately. The Foundation should immediately formally recognise the unions in the UK and the US.

Further, WMF is an international organisation. Collective decisionmaking needs to be transnational too. WMF should recognise WWU as a negotiating partner worldwide, even if thresholds for formal legal recognition are not met in individual national jurisdictions.

Wiki Workers are not the WMF's enemy. WMF needs capable and ideologically committed staff to maintain and operate its highly complex systems, in the face of constant attacks. Staff with principles and a mission are WMF's biggest asset.

Dr Ian Jackson
Cambridge, UK
20th August 2026

comments

Jonathan Dowland: Bauer X4 inline skates

Thu, 20 Aug 2026 09:54:51 +0000

I’m really enjoying getting back into ice skating, but I can only get to the rink once a week (at least over the summer -- I'm aiming for twice weekly once Schools re-open) and I have the itch to do more skating than that.

Where I live we’re blessed with a seaside park with lots of smooth paths, a recently resurfaced beachside promenade, and a newly-built pedestrian/cycle path stretching up and down the coast: all great surfaces for roller skates. I convinced myself to buy some inline skates whilst the weather is good.

I wanted something as close to my ice skating experience as possible. Bauer actually make an inline version of my ice boot, but the chassis is an unusual composite plastic thing which put me off. (here's a great video of a fantastic inline skater trying out the chassis). CCM have a new inline range for 2026, but sadly (much like their Jetspeed ice range) the fit wasn't good for me.

I found a clearance pair of Bauer vapors from the previous generation: the Bauer Vapor x4. Very similar to my Fly30, but the difference in quality between the tiers is very apparent: boot stiffness, the comfort and quality of the liner. They fit well (possibly better), the rolling motion is really smooth (I think that's the bearings) and they looked pretty good to me: yellow highlights instead of the red used across the ice range.

I've done a couple of miles in them so far. Time will tell if they prove useful for off-ice training! Many inline hockey players buy ice skates and convert them to inline. If I end up not using them enough I could consider doing the opposite.

Thomas Lange: LLM usage in Debian

Tue, 18 Aug 2026 13:52:29 +0000

After spending many hours on reading all the proposals and discussions the best choice for me is NOTA (None of the above).

We do not need to create new rules for LLM usage, we already have our DFSG and our social contract.

Keep it simple, stupid. Avoid more rules!

Andy Simpkins: My first go at tracking down a kernel bug…

Tue, 18 Aug 2026 10:50:43 +0000

A couple of weekends back, I upgraded my home sever. It failed to restart after running apt dist-upgrade

The only update that was performed was to the kernel, it went from 6.12.88+deb13-amd64 to 6.12.100+deb13-amd64. I had previously performed an apt-get upgrade, and rebooted the machine, so I was pretty sure that this was to blame. This blog entry (is a late) attempt to document how I went about finding a fix for this issue so that next time I don’t need as much hand holding as I did this time around :-)

(1)
Having my machine not boot following an upgrade is pretty rare, but has happened before. Usually it is because I have done something wrong so as always confirming I haven’t broken something by accident is always my first step…

I plugged in a keyboard an monitor to the machine and watched it boot. Being a server this takes a long time (I guess because at this stage of system initialisation we want to test things sequentially)

Watching the system boot I see the usual BIOS/UEFI stages for this machine, followed by the grub menu and the the local screen showed:

            Loading Linux 6.12.100+deb13-amd64 ...
            Loading initial ramdisk ...

Nothing else. That was it. OK that looks like I have a broken system all right, and at very early stage of the boot process process.

(2)
Breaking into the grub menu and removing the quiet option yields a little more information (but not much):

            Loading Linux 6.12.100+deb13-amd64 …
            Loading initial ramdisk ...
            

            	EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID d
            	evice path
            		EFI stub: Measured initrd data into PCR 9

and nothing else.

(3) Initial debugging

Confirmed that I could still boot the machine with the old kernel 6.12.88+deb13-amd64 (During boot select Advanced options from the grub menu followed by the kernel image wanted)
- Yes – the system starts happily with the previous kernel
Checked that /boot had enough space
- Yes – plenty of space
Is anyone else reporting this problem?
- Nothing jumps out on Debian’s bug tracker
- Actually not mush referenced for my search “EFI stub: Measured initrd data into PCR 9 apart” other than the usual rantings to “turn off secure boot” (on this server that currently isn’t turned on – bad me)

(4) Triage

Start looking for where the fault first occurred. At this point I needed help, and given that Sledge was visiting I asked if he would sanity check what I was doing. His initial thoughts were that that /boot had run out of space, but replaying my step (3) with him acting as a ‘rubber duck’ showed that this was something other than PBKAC

Sledge had a quick look, then informed me that between kernel images 6.12.88+deb13 and 6.12.100+deb13 Debian stable has only had shipped .90 .94 .95 and .96 kernels. We could easily try them all:

wget each kernel package then install (dpkg -i) followed by an update-grub, checking that there was sufficient space on disks especially my small /boot partition)
I started with image 6.12.95+deb13 and this worked
6.12.96+deb13 yielded the same lock up on boot as 6.12.100+deb13

OK I now have the first kernel image that doesn’t boot on my system, time to raise a bug…

Up until now I have been walking to my garage where the server is located and standing in front of a rack
with a monitor and keyboard plugged into the machine. However this machine supports IPMI so I spent a little time getting that up and running so that I can continue from the relative comfort of my desk (with lights, a chair and not needing to hold the keyboard with one hand)

Great I can now grab screen shots from the confort of my desk (unfortunatly they are only screen shots not text files, but at least we can seen the early stage of boot, Post, grub menu and then initramfs before system log happens)

(5) Collating information for the initial bug report

Sledge had mentioned my problem in irc/#debain-kernal where iam_tj suggested that we try appending
‘debug earlycon=efifb’ to the kernal command line. This yielded 15 seconds worth of messages before the system locked up the last few messages being (vmlinuz-6.12.96+deb13-amd64):

[ 14.663477] RCU Tasks: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.750474] RCU Tasks Rude: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.838024] RCU Tasks Trace: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
[ 14.929752] NR_IRQS: 524544, nr_irqs: 584, preallocated irqs: 16
[ 15.016814] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[ 15.104011] Console: colour dummy device 80×25
[ 15.191236] printk: legacy console [tty0] enabled
[ 15.278249] printk: legacy bootconsole [efifb0] disabled

Booting the working kernel with the same kernel options yields the SAME messages with slightly differing times, but then continues to login prompt:

 [   14.697466] RCU Tasks: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
 [   14.784936] RCU Tasks Rude: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
 [   14.872067] RCU Tasks Trace: Setting shift to 5 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=20.
 [   14.964000] NR_IRQS: 524544, nr_irqs: 584, preallocated irqs: 16
 [   15.051482] rcu: srcu_init: Setting srcu_struct sizes based on contention.
 [   15.226079] printk: legacy console [tty0] enabled
 [   15.313751] printk: legacy bootconsole [efifb0] disabled
 [   15.400831] ACPI: Core revision 20240827
 [   15.401415] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
 [   15.401464] APIC: Switch to symmetric I/O mode setup
 
... and so on

iam_tj also suggested adding keep_bootcon – with ‘debug earlycon=efifb keep_bootcon’ on vmlinuz-6.12.96+deb13-amd64:
We get a LOT further – and we see a crash / trace-back:

[ 34.285342] BUG: kernel NULL pointer dereference, address: 0000000000000000

I raised bug #1143721 and followed it up with screen captures of the boot sequence (captured from the IPMI client) and files containing the output of dmidecode, lscpu and lspci to kive the kernel team as much information as possible:

[6.12.96+deb13-amd64 debug earlycon=efifb keep_bootcon.tar.gz (application/gzip, attachment)]

[dmidecode.txt (text/plain, attachment)]

[lscpu.txt (text/plain, attachment)]

[lspci.txt (text/plain, attachment)]

(6) Tracking down the bug Git Bisect

The problem with this type of bug is that it is hardware (class) specific, whilst the kernel doesn’t boot on my system, it clearly has worked on machines used by the kernel team, the Debian test and build infrastructure, (otherwise this kernel would never have been released) and everyone else who has upgraded to the newer kernel before I did (otherwise we would be drowning in fails to boot bug reports). Carnil’s excellent response to my bug: Message #15 (and help in IRC) provided me with a detailed step by step guide in how to track down the individual git commit that fails on my system. I had already (with Sledge’s suggestion) made a clone of the stable branch, but was struggling to follow the steps in the Debian Linux Kernel Handbook to re-build a duplicate kernel because I didn’t understand how to obtain the same configuration that Debian used to build the kernel; Carnil’s email provided me the missing steps (Highlighted).

git clone --single-branch -b linux-6.12.y https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-stable
git checkout v6.12.95
cp /boot/config-$(uname -r) .config
yes '' | make localmodconfig
make savedefconfig
mv defconfig arch/x86/configs/my_def
test 6.12.96 to ensure this is "bad"
git checkout v6.12.96
make my_defconfig
make -j $(nproc) bindeb-pkg
… install the resulting .deb package and confirm it fails to boot and triggers the NULL pointer dereference.

Right I can now start to Bisect the problem:

git bisect start
git bisect good v6.12.95
git bisect bad v6.12.96

Rather than use the half step point’s git bisect suggested I was advised in irc to jump straight to the a given commit that from the git log was suspected as the culprit:

git checkout 977855894bca4b87afa50d21e3f3e85a5a0e901f
build and install….
fails…
git bisect bad

git checkout 977855894bca4b87afa50d21e3f3e85a5a0e901f~1 ## ~1 is the commit beforehand
build and install….
fails…
git bisect good

The entire test tree can shown with git bisect log and this was submitted as an email to the bug report, we have found our smoking gun :-)

Finally I would like to thank Carnil, Iam_tj for their time patience and fantastic support in guiding me through finding this regression. Right now kernel bugs are coming in thick and fast with a lot of AI assisted bug hunting, the increased numbers of bugs mean that the kernel team are especially busy. Hopefully our paths will cross and I’ll be able to buy you some beers (or whatever) soon. thank you. Sledge also deserves thanks for putting up with me and pointing me in the right direction (as ever). Lucky for me that he lives nearby so I can provide beers on a regular basis :-)

John Goerzen: AI in Debian: The Vote, Proposals, and Nuance

Tue, 18 Aug 2026 01:39:54 +0000

Let me start with a hypothesis:

For human developers, using coding LLMs magnifies their difference in skill levels.

I am one that rarely thinks things are always black and white. Back in March, I wrote Artifial Intelligence: Shades of Gray. Since then, I’ve had more of a chance to experiment with LLMs myself. I also happen to work for an employer that is taking a very pragmatic approach to LLMs: teams and individuals use it as they see fit, but if they are causing considerable expense, they have to justify it.

In various settings, I have seen the egregious examples of AI slop we all know about. As I wrote in March, “I have seen it both waste more time than it saves, and save a ton of time.”

I have come to see that, as a tool, it is most valuable when it is running under the supervision of an experienced engineer. It is at its worst when it has no such supervision; the “vibe coding” and other low-quality slop we see.

A coding agent is like a junior developer or research assistant. When properly supervised, they help projects move along more quickly by letting a senior developer focus on the more difficult, less mundane aspects of the project. But one couldn’t expect a junior developer to consistently deliver high-quality code and architecture on their own.

Let’s put a pin in this idea and look at the story in Debian.

LLM use in Debian

There is a vote happening in Debian around the use of LLMs. In typical Debian fashion, there are 8 options to choose from, many of them similar. Most of these proposals acknowledge there are different types of tasks done in Debian, but the proposals don’t differentiate between them well. Let me do so here. These are some of the LLM-relevant tasks people in Debian perform:

Packaging upstream software for Debian (by far the largest task)
Writing Debian-specific code (eg, apt or the Debian installer)
Maintaining Debian infrastructure (build systems, for instance)
Writing documentation and translations

I’m going to focus my remarks here on packaging upstream software for Debian, since this is by far the most time-consuming developer task project-wide.

It matters to our users that we get this right, and packaging quality is one of the things that sets Debian apart from other distros. Packaging things for Debian requires knowledge of some specific tools, such as debhelper, that aren’t widely used anywhere else. In most cases, it is fairly rote time-consuming work. In other words, by its design, it requires people with senior-level skills to do grunt work.

I can’t overstate how massive a burden this grunt work is. I maintain some packages for Go and Rust. By Debian policy, all of those packages’ dependencies must also exist as Debian packages, and be used to build against. When upstream adopts a newer version of some library, it can unleash cascading dependencies that can take hours to sort out. Worse, the Rust team and the Go team use entirely different ways of managing packages (Go uses one Git repo per package, while Rust has a monorepo with specialized scripts to import Cargo packages and generate Debian ones). On top of that, we can’t just modify things like usual; we have to use quilt. And on top of that, I’m also a backports maintainer, so all the work (and usually even more) has to be done there also.

Now let’s pull on that pin from the earlier conversation. This is exactly the kind of scenario that a well-supervised coding LLM is most effective in. I could see a seasoned developer saving hours, maybe even days, by turning over the mundane tasks of managing trees of cascading dependencies over to a coding tool — and verifying and directing the process. (Yes, I have been using em-dashes for years; LLMs have copied people like me, not the other way around! This post was not written with any AI assistance.)

Actually, this is almost a dream scenario for a coding assistant. The result is time-consuming to formulate but easy to review, which is the opposite of the way these things often go.

I can assure you with 100% certainty that humans aren’t adding a lot of value in this process. It would be wrong to believe that a human is carefully reading every line of code in dozens of updated or new library packages. The problem set is too big, the time too short, and the code too varied and complex.

Coding agents seem to be most effective when there are strong test suites that they can test changes against. Debian builds, especially of modern packages, tend to have this property. Many packages have test suites that are run during build. And, if the package builds in an isolated environment (and especially if its downstream dependencies do also), then there is a decent chance that it’s fairly correct. Maybe needing some manual tweaking here and there, but generally a successful build is a reasonable indicator.

You can argue that it would make more sense for Debian to just include dependencies in source packages, along with some version information to support security rebuilds, and I’d tend to agree with you. But we are where we are. This would be one of the more significant leaps forward in developer productivity, but it complicates things like copyright reviews.

Where are LLMs run? What is the environmental impact?

Most of the proposals seem to make the assumption that LLMs must always run in some large, hosted datacenter. As I noted in my March article, I have had credible results on even an older GPU running on solar power.

That said, it is undeniable that LLMs are fueling a datacenter boom, and this in turn is producing a significant new demand for resources. Most notably for the global scale: electricity, which is sometimes generated using carbon-emitting technologies.

Bill McKibben, who has been a leading voice in the fight against climate change since the 1980s, has made some interesting points recently: he’s noted that solar power is the fastest kind of generation we can build, and a number of large AI companies are investing heavily in solar, even to the point of fully offsetting new datacenter’s needs. On the other hand, he’s also noted that some companies are buying inefficient and dirty gas turbines. It is decidedly a mixed bag. The heavy investment in solar can have knock-on positive effects for infrastructure. Obviously, not every picture here is rosy. This analysis doesn’t touch on the real land and water use situation, either.

On the other hand, if an LLM allows me to do in an hour what I would have done in a day, that’s a day of not heating or cooling the work area — generally not sustaining a human for the purpose of writing code for Debian. HVAC energy consumption dwarfs my GPU, and I’d imagine probably also the slice of LLM energy used.

Holistically, I would have to conclude the picture is mixed. It is possible to use LLMs in a pretty green way, and also in a pretty dirty way.

Assuming Conditions Never Change

A flaw in most of these proposals is they assume that the conditions at this present moment will always hold. In fact, that the conditions at the present moment will not continue is something both AI cheerleaders and AI skeptics agree on.

For instance:

Ed Zitron has done a ton of research into the financing side of AI, and has concluded that the current model is unsustainable and headed for a significant bubble burst. I’m not positioned to personally evaluate those claims, but if that happens, what is the result? Perhaps it is a steeply increasing cost of inference for the frontier models, slower pace of training/evolution for them, etc.

In a recent episode of Oxide and Friends, Simon Willison discussed the open weight models that are now available. They have been making remarkable strides in efficiency and capabilities, to the point where $50,000 of hardware can now run high-end open weight models with capabilities that are at least in the same ballpark as the American frontier models. This puts running high-end models locally squarely within reach of universities and small- to medium-sized businesses, with power requirements that can be met with standard commercial solar and wind installations.

The lack of nuance in the more restrictive proposals is particularly concerning. Proposal A doesn’t allow “the use or assitance of… LLMs”. So it bans my solar-powered GPU. It bans using LLMs to find security issues. It bans all sorts of things that don’t seem to be ban-worthy, alongside the things that do. And it codifies it in the very hard-to-change social contract.

That proposal, and some like it, seem to imply that all LLM output is bad. I grant you that AI slop is a real and legitimate concern, and many Open Source projects have to deal with it. On the other hand, we have all seen first-hand how the security of the Linux kernel has benefited dramatically from AI analysis. It is certain that black hats are using these tools. If we refuse to use modern security tools, our security will be compromised (and what is the environmental and social impact of THAT?)

I find the statement “Generative AI is characterized by producing output of a nature that would ordinarily be produced and consumed by humans” to be particularly interesting. The same was once said of compilers.

The Real Concerns

You might think from reading this that I am some AI cheerleader. I’m not. I share the ethics of the FLOSS movement, and have for decades. I abhor the power and lack of ethics that many big names in the field are running with at the moment. I’ve had to put up Anubis on this blog, for instance.

I have personally experienced the effects of AI slop, especially at review time. This is a real problem, though I don’t think the more draconian policies are likely to help (the looser “you must disclose” stand a fighting chance, but I’m not sure they would help, either.) Done poorly, AI threatens developer burnout by overwhelming them with poor code and verbose but useless explanations. Done well, AI can help prevent developer burnout by automating tedious and low-value tasks.

Shouldn’t our goal be that humans submit work to Debian, using tools they prefer, and take responsibility for it? Does it matter if someone uses ed, vim, emacs, or vscode? If they use LSP or just run gcc manually? I’d say we benefit from the diversity. Wouldn’t we be better off to benefit from the diversity here, and judge work as we always have: on its merits, not what tools were used to create it?

Fundamentally, a GR is a long and arduous process. It’s not easy to reverse later. Amending the Social Contract is even longer and more arduous (I should know; I may have been the first one to try). The LLM landscape is fast-moving. None of us can really predict where it will be in a year. Will the current market leading companies even still exist? Will it be at all credible to refuse to use AI-assisted security tools? What is the most effective way to deal with AI slop? What level of utility will we be able to achieve with models run locally?

Some of these proposals would make sense if drafted in some way short of a GR, which would allow more maneuverability as the landscape changes.

Brief analysis of the options

Considering the proposals:

Proposal A: seeks to amend the social contract, which I am opposed to for reasons already laid out above. It names some real concerns about AI that I agree with, but implies that all LLM uses and models are guilty of the problems, which is not the case with all of the claims. It also sets us behind the curve on security and stability by forbidding the use or assistance of those tools, even if run by others. It requires us to ignore reports of actual security bugs, or correct fixes, if those reports were generated with the assistance of an LLM, which I find to be absolutely untenable.
Proposal B: This is the “AI with accountability” approach. It notes the real concerns with LLMs without painting with an overbroad brush. It strikes me as level-headed and sensible.
Proposal C: It paints with an over-broad brush and makes some non-binding requests. Then it winds up largely like proposal B, though while it is worded more strongly, has fewer binding requirements (for instance, it lacks proposal B’s prohibition on transmitting sensitive information to untrusted providers)
Proposal D: Seems broadly similar to proposal B, an “AI with accountability” approach. I’m not really clear why we need both.
Proposal E: Largely the status quo. It is like proposals B and D in that it says humans are accountable for their contributions. It encourages disclosure of LLM use, but does not mandate it. Like proposal B, it prohibits disclosing sensitive information to third-party AI services. Note that both proposals B and D have an appropriate nuance: a local model is fine, a third-party one is not.
Proposal F: This seems really similar to proposal E. I’m not sure why we have these two.
Proposal G: Disallows “the output of generative AI as direct contributions to Debian.” This is something of a weakened proposal A; it doesn’t seek to amend the social contract, nor does it ban all use; it simply bans the use as a direct contribution.
Proposal H: Ban due to climate impacts. “How is this even an argument” is disrespectful to reasoned conversation. I have already noted that LLMs can be and are used in ways that are not climate-harming. It explicitly contains no binding requirements at all, and is effectively a rant. While I agree with the sentiment that climate change is an urgent problem, and that some LLMs are exacerbating it, I disagree with that all LLM usage does so and therefore disagree with the conclusion.

In favor of nuance

I find that black-and-white thinking is almost always something to be avoided. I see it too often. I see it in politics, I see it in our software, I see it in discussions around AI. Are there deeply unethical things happening in AI? Absolutely. Are they doing some impressive things? Also yes.

We have accepted this nuance in other areas. For instance, almost all the hardware Debian runs on has closed-source hardware, and has components manufactured or assembled in countries with some of the worst human rights records on the planet. I’m not saying this is a great state of affairs. It is something we should speak up about and act upon. But the worse state of affairs would be “no Debian because the hardware is impure”.

Bits from Debian: Debian turns 33!

Sun, 16 Aug 2026 13:00:00 +0000

It has now been thirty-three years since the Debian project was announced to the world by Ian Murdock, on August 16, 1993. This anniversary is an opportunity to reaffirm the goals, characteristics, and qualities of the Debian project: itâ€™s an association of individuals who have made common cause to create a free operating system. Our distribution is characterized by a commitment to software freedom, as enshrined in the Debian Social Contract and the Debian Free Software Guidelines. It focuses on security and stability. This stability is crucial to Debian position in the free software ecosystem.

With our users as our priority, Debian makes special efforts regarding accessibility with Debian-Accessibility and diversity with our Outreach Programs.

Debian Day is a great opportunity to get together, whether for a local meetup, or simply to grab a coffee with other members of the Debian community. Check out the Debian Day wiki to see if there is a celebration near you. And if there isn't, maybe you can organize it next year!

Today is also an opportunity for you to start or resume your contributions to Debian. For example, you can install the how-can-i-help package and see if there is a bug in any of the software that you use that you can help to fix, contribute small tips on how to install Debian on your machines to our wiki pages, or put a Debian live image in an USB memory and give it to some person near you, who still didn't discover Debian.

Thanks to everybody who has contributed to develop our beloved operating system in these 33 years, and Happy birthday Debian!

Benjamin Mako Hill: Sad Story

Sun, 16 Aug 2026 07:01:10 +0000

Not a screenshot of despair. But only because it’s not a screenshot.

Sven Hoexter: FrOSCon 2026: TLS Talk

Sat, 15 Aug 2026 18:02:03 +0000

Info: German content only, sorry.

I was pondering for the past three years if I should give some sort of TLS basics talk at FrOSCon. I finally stepped up this year and gave that talk today, with the title "TLS, mTLS, SNI, ECH, CAA, HTTPS, PKI, Zertifikate und ein bisschen PQC". I was too optimistic with my 50 slides, and had to drop the Post Quantum Cryptography part at the end. Still got positive feedback from Zugschlus and others - thanks a lot for that <3 - and was asked for the slides. It's not a piece of art, but maybe it helps to release the LibreOffice odp file as well, so others can use it as a base for other events or corp internal talks. So here is the froscon-tls-2026.pdf and froscon-tls-2026.odp, both released under the CC BY-NC license.

The video is also available at media.ccc.de if you want to watch it.

Thanks to everyone who made FrOSCon happen for the 21th time!

Russell Coker: Hacked by Chinafans

Sat, 15 Aug 2026 09:31:47 +0000

What Happened

On 2026/08/10 at 2:11 am Australian eastern standard time (2026/08/09 16:11 UTC) someone created a post titled “Hacked by Chinafans” on my documents blog [1]. The person in question created an account named “67965e42a3c3” on that site with the email address 67965e42a3c3@google.com associated with it (I tried emailing that address and it bounced).

At 04:28:41am Australian eastern standard time (18:28 UTC) I was sent an email titled “Have you been hacked” by a reader of my blogs who subscribed to the RSS feed of my documents blog (a blog that I never expected anyone to read by RSS). Along the lines of “the wisdom of crowds” should we have “the unexpected observation and problem reporting of crowds”? I appreciate the notification, I might not have noticed until the next time I watched an unusually good movie otherwise.

The account in question was apparently created on 2026-07-21 at 16:43:47 (presumably UTC) even though at the time I believe creating accounts was not permitted. As an aside the timestamp of account creation is stored in the user_registered column of the wp_users table in the database, there doesn’t appear to be a way to access this in a standard WordPress installation other than doing a SQL query.

2026-07-24 15:43:17 status triggers-pending wordpress:all 7.0+dfsg1-1
2026-07-24 15:43:19 upgrade wordpress:all 7.0+dfsg1-1 7.0.2+dfsg1-1

Above are the relevant sections of my dpkg log showing the WordPress versions in use. I was running version 7.0+dfsg1-1 at the time the account was apparently created. I am confident in the accuracy of the dpkg logs and believe that they did not compromise the OS, I am not sure whether they ran hostile SQL code to change fields in the MySQL database so had to consider the possibility that the account creation time could have been set to a deliberately misleading value. I checked backups of the MySQL database stored off-site and found that the account in question was not in the 2026-07-21 backup (which was done before 16:43) but in the 2026-07-22 backup.

The WordPress release history [2] has version 7.0.1 released on 2026-07-09 and version 7.0.2 released on 2026-07-17. So presumably the attacker diffed the code on those releases, found an exploitable bug, and used it to create an account on my blog with admin privs. Then they waited a few weeks to see if I would notice and published a blog post when I didn’t notice.

WordPress Deficiencies

WordPress doesn’t seem to store the version it’s running at the time of operations. So anyone who doesn’t have a suitable external log of versions deployed (such as the dpkg.log file for a Debian managed installation) won’t know for sure which version was running. It supports automatic updates but you can’t be sure that they happened soon after the release.
There is no log of IP addresses used for operations. There are apparently some 3rd party modules to log such things and web pages documenting how to modify the PHP to add it but nothing in the standard distribution.
Software should have a standard distribution with some support for logging of security relevant data. The typical situation is that people don’t plan for logging such things until after they have been attacked so the data should be recorded without users going out of their way to log it.
A log of security relevant data should be stored in a database table with only insert access (no update, delete, or drop).
Ideally a CMS would support different database accounts for different purposes. Someone from an internal network or VPN could talk to an instance of the web server which has a database username and password giving full access. Everyone from outside the trusted range gets an instance of the web server with database access only allowing to read the posts and appearance configuration and to enter comments. If the database didn’t allow the account used for public access to create new admin users or create posts then it would be a lot harder for attackers.
Ideally for everything that stores user account data there would be an easy way of getting a list of users in a plain text format to allow running diff. The design of WordPress has two tables, one for users and one for encoded metadata about users of which one will be the access level. The following SQL command will give a list of all users that aren’t subscribers (everyone above the minimum level of access which is typical for new users) along with their encoded password and access level. This could be used in a monitoring system to alert about new privileged users. The TABLE_PREFIX variable is for the prefix for WordPress tables, which is “wp_” by default but can be any legal value.
```
select $TABLE_PREFIXusers.user_login, $TABLE_PREFIXusers.user_pass, $TABLE_PREFIXusermeta.meta_value from  $TABLE_PREFIXusers join $TABLE_PREFIXusermeta on $TABLE_PREFIXusers.id = $TABLE_PREFIXusermeta.user_id and meta_key='$TABLE_PREFIXcapabilities' and meta_value != 'a:1:{s:10:"subscriber";b:1;}';
```

What Next?

The blog post they created had a couple of links to Telegram which could presumably be used to contact them. If anyone involved in computer security wants a copy of the original post to do so then they can contact me by any of the usual methods.

I am interested in communication with the attacker if they wish, Telegram is not a service I use but I presume that anyone capable of doing this sort of attack is also capable of finding other ways of contacting me.

I have idly considered changing to a static site generator, here is a good list of static site generators [3].

I have also idly considered other platforms for blogging such as Lemmy. I don’t know if Lemmy is better than WordPress for security and updates, but there are plenty of free instances running where it wouldn’t be an issue I have to work on.

15 Years

It’s been 15 years since my blog server was cracked by a trojaned ssh client [4]. At least this time it was only one service that was compromised.

Russell Coker: AMD Video Drivers, LLMs, and Debian Kernels

Sat, 15 Aug 2026 08:09:51 +0000

The AMD GPU Problem

For a while I’ve been having issues with AMD GPUs, video locking up periodically. I blogged about this late last year but I first had noticeable problems early last year [1]. The problems hadn’t only concerned my workstation but also my home server which is also used as a workstation. I’ve recently upgraded my machines to Debian/Testing, my home server has been generally OK but my workstation has been crashing a lot. Every second day when on kernel 7.1.6 and then when on 7.1.7 it crashed at least once a day.

The AMD GPUs I have are “[AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] (rev e5)” in my main desktop workstation, “[AMD/ATI] Lexa [Radeon 540X/550X/630 / RX 640 / E9171 MCM] (rev c1)” in my build server, and “[AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] (rev cf)” in my home server. They aren’t new GPUs, but also aren’t really old and they all support 4K and better resolution.

Chat GPT Was Useful

When I googled the errors I was seeing I found nothing useful. On the suggestion of a friend I tried asking ChatGPT. Generally I don’t recommend asking LLMs about such things, but it can be a last resort as long as you know what you are doing. ChatGPT asked me to run a number of commands to get information for it to make more informed decisions. I know that the output of lspci and similar commands isn’t a risk, but a novice could be tricked into running commands that expose sensitive data.

ChatGPT did give me some useful information, not a solution but an indication that the problem was due to driver bugs.

Upgrading to Experimental

Debian/Experimental is for packages that are expected to have problems and generally aren’t recommended even for the people who usually use Debian/Unstable. It’s commonly used for packages that are needed to develop other packages, EG new libraries that aren’t fully usable but which are needed to package newer versions of applications.

I upgraded my workstation to the Debian/Experimental kernel 7.2~rc7-1~exp1 after having tried every other convenient option. Generally I wouldn’t recommend that anyone run an Experimental kernel without a really good reason, but crashing more than once a day is a fairly good reason. That kernel has now given me over 4 days of uptime on a system that previously wouldn’t last a day. I installed it on my dual-socket build server that has an old AMD GPU in it for test purposes and that also hasn’t crashed since. I installed it on my ML test machine which has an Intel B580 Battlemage GPU with 16G of VRAM and was repeatedly getting a kernel panic related to the GPU a few seconds after boot and now it also works correctly.

It seems that the 7.1.x kernels have bugs in the AMD video drivers and in some part of the code that affects Intel video drivers and that the bugs in question are fixed in the tree that will become 7.2. I would not recommend anyone who has a 7.1.x kernel working fine for them try 7.2 RC kernels at this time, but anyone who has GPU related problems (particularly Intel and AMD GPUs) should definitely test it out.

I also don’t recommend upgrading any system with an AMD GPU to Debian/Testing or Debian/Unstable at this time unless you are also prepared to install an Experimental kernel if it becomes necessary.

There are a several kernel log dumps related to this after the break (which won’t be in RSS feeds). This is mainly for Google so that other people who have such issues can get more useful results out of Google searches than I got.

Future Support Options

Separate from the issue of whether commercial LLMs like ChatGPT can be useful for solving technical problems there is the issue of whether they are desirable. I think that we really don’t want people solving problems in FOSS systems with closed-source LLMs. This leads to loss of privacy, loss of the control users deserve to have over their own systems, and an implied promotion of non-fee software.

I think that the ideal would be to have a cross distribution effort to generate training data for a support LLM system which can then be further trained by each distribution for a greater emphasis on distribution specific issues.

[1] https://etbe.coker.com.au/2025/11/09/amd-video-driver-issues/

Errors on AMD GPUs

2026-08-09T23:03:06.004792+10:00 xev kernel: amdgpu 0000:02:00.0: GPU fault detected: 147 0x00024802
2026-08-09T23:03:06.004792+10:00 xev kernel: amdgpu 0000:02:00.0: Process kscreenlocker_g pid 42037 thread kscreenloc:cs0 pid 42044
2026-08-09T23:03:06.004793+10:00 xev kernel: amdgpu 0000:02:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000800
2026-08-09T23:03:06.004794+10:00 xev kernel: amdgpu 0000:02:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0F048002
2026-08-09T23:03:06.004795+10:00 xev kernel: amdgpu 0000:02:00.0: VM fault (0x02, vmid 7, pasid 130) at page 2048, write from 'TC0' (0x54433000) (72)
2026-08-09T23:03:06.008762+10:00 xev kernel: amdgpu 0000:02:00.0: GPU fault detected: 147 0x00004802
2026-08-09T23:03:06.008768+10:00 xev kernel: amdgpu 0000:02:00.0: Process kscreenlocker_g pid 42037 thread kscreenloc:cs0 pid 42044

2026-08-04T01:13:37.505839+10:00 xev kernel: ------------[ cut here ]------------ 
2026-08-04T01:13:37.505859+10:00 xev kernel: amdgpu 0000:02:00.0: [drm] drm_WARN_ON_ONCE(cur_vblank != vblank->last) 
2026-08-04T01:13:37.505862+10:00 xev kernel: WARNING: CPU: 6 PID: 210534 at drivers/gpu/drm/drm_vblank.c:362 drm_update_vblank_count+0x2f1/0x3c0 [drm] 
2026-08-04T01:13:37.505866+10:00 xev kernel: snd_intel_dspcfg wmi_bmof rc_core snd_intel_sdw_acpi drm_ttm_helper uas realtek snd_usbmidi_lib snd_hda_codec ttm mdio_devres snd_hda_core snd_seq_midi drm_kms_helper usb_storage mc snd_hwdep libphy snd_seq_midi_event intel_uncore snd_pcm_oss i2c_algo_bit serio_raw snd_rawmidi pcspkr snd_mixer_oss i2c_i801 video snd_seq snd_pcm i2c_smbus lpc_ich snd_seq_device mei_me e1000e snd_timer mei snd tpm_infineon soundcore joydev bnx2 wmi button nfsd auth_rpcgss nfs_acl lockd grace sunrpc coretemp br_netfilter bridge stp llc sg ghash_clmulni_intel loop msr i2c_dev drm efi_pstore configfs nfnetlink ip_tables x_tables autofs4 btrfs blake2b_generic dm_crypt dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx libcrc32c xor raid6_pq raid1 raid0 md_mod ext4 crc16 mbcache jbd2 crc32c_generic virtio_blk evdev hid_generic usbhid hid sd_mod xhci_pci xhci_hcd ahci ehci_pci ehci_hcd libahci crc32c_intel libata usbcore aesni_intel nvme psmouse scsi_mod gf128mul crypto_simd nvme_core cryptd 
2026-08-04T01:13:37.505879+10:00 xev kernel: nvme_auth scsi_common usb_common efivarfs 
2026-08-04T01:13:37.505880+10:00 xev kernel: CPU: 6 UID: 1008 PID: 210534 Comm: sshd-session Tainted: G D 6.12.88+deb13-amd64 #1 Debian 6.12.88-1 
2026-08-04T01:13:37.505881+10:00 xev kernel: Tainted: [D]=DIE 
2026-08-04T01:13:37.505883+10:00 xev kernel: Hardware name: Hewlett-Packard HP Z640 Workstation/212A, BIOS M60 v02.61 03/23/2023 
2026-08-04T01:13:37.505884+10:00 xev kernel: RIP: 0010:drm_update_vblank_count+0x2f1/0x3c0 [drm] 
2026-08-04T01:13:37.505885+10:00 xev kernel: Code: 48 8b 5f 50 48 85 db 75 03 48 8b 1f e8 68 eb 2b cf 48 c7 c1 70 3e cb c0 48 89 da 48 c7 c7 f9 6f cb c0 48 89 c6 e8 af d7 a6 ce <0f> 0b e9 4b fe ff ff 48 8b 4c 24 18 e9 31 fe ff ff 31 f6 48 85 db 
2026-08-04T01:13:37.505887+10:00 xev kernel: RSP: 0000:ffffd3cc8681fca0 EFLAGS: 00010082 
2026-08-04T01:13:37.505888+10:00 xev kernel: RAX: 0000000000000000 RBX: ffff8c6b42b13710 RCX: 0000000000000027 
2026-08-04T01:13:37.505889+10:00 xev kernel: RDX: ffff8c89ef521788 RSI: 0000000000000001 RDI: ffff8c89ef521780 
2026-08-04T01:13:37.505890+10:00 xev kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffffd3cc8681fb20 
2026-08-04T01:13:37.505891+10:00 xev kernel: R10: ffff8c8a6fef3628 R11: 0000000000000003 R12: 0000000000000000 
2026-08-04T01:13:37.505892+10:00 xev kernel: R13: ffff8c6c07853828 R14: 0000000000000003 R15: 0000000000000000 
2026-08-04T01:13:37.505893+10:00 xev kernel: FS: 00007ffaf2fd5880(0000) GS:ffff8c89ef500000(0000) knlGS:0000000000000000 
2026-08-04T01:13:37.505895+10:00 xev kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
2026-08-04T01:13:37.505896+10:00 xev kernel: CR2: 00007fb1718c8000 CR3: 000000074521a004 CR4: 00000000003706f0 
2026-08-04T01:13:37.505897+10:00 xev kernel: Call Trace: 
2026-08-04T01:13:37.505898+10:00 xev kernel:  
2026-08-04T01:13:37.505899+10:00 xev kernel: drm_crtc_accurate_vblank_count+0x41/0xc0 [drm] 
2026-08-04T01:13:37.505900+10:00 xev kernel: dm_pflip_high_irq+0x155/0x330 [amdgpu] 
2026-08-04T01:13:37.505901+10:00 xev kernel: amdgpu_dm_irq_handler+0x85/0x1f0 [amdgpu] 
2026-08-04T01:13:37.505902+10:00 xev kernel: amdgpu_irq_dispatch+0xd2/0x230 [amdgpu] 
2026-08-04T01:13:37.505903+10:00 xev kernel: amdgpu_ih_process+0x84/0x100 [amdgpu] 
2026-08-04T01:13:37.505904+10:00 xev kernel: amdgpu_irq_handler+0x23/0x60 [amdgpu] 
2026-08-04T01:13:37.505905+10:00 xev kernel: __handle_irq_event_percpu+0x4a/0x190
2026-08-04T01:13:37.505907+10:00 xev kernel: handle_irq_event+0x38/0x80 
2026-08-04T01:13:37.505908+10:00 xev kernel: handle_edge_irq+0x8b/0x230 
2026-08-04T01:13:37.505909+10:00 xev kernel: __common_interrupt+0x45/0xe0 
2026-08-04T01:13:37.505910+10:00 xev kernel: common_interrupt+0x42/0xa0 
2026-08-04T01:13:37.505911+10:00 xev kernel: asm_common_interrupt+0x26/0x40 
2026-08-04T01:13:37.505912+10:00 xev kernel: RIP: 0033:0x7ffaf3c5fd7b 
2026-08-04T01:13:37.505913+10:00 xev kernel: Code: 70 c7 00 66 0f 6e f8 c1 ef 02 66 0f 70 f7 e0 83 c7 01 66 0f ef ff 66 0f fa f2 0f 1f 44 00 00 f3 0f 7e 01 66 0f 6f ce 83 c6 01 <48> 83 e9 08 f2 0f 70 c0 1b 66 0f 6f e0 66 0f 6f e8 66 41 0f f9 c0 
2026-08-04T01:13:37.505915+10:00 xev kernel: RSP: 002b:00007fff86a5e0e0 EFLAGS: 00000202 
2026-08-04T01:13:37.505916+10:00 xev kernel: RAX: 0000000000008000 RBX: 0000562614a04050 RCX: 0000562614982ed8 
2026-08-04T01:13:37.505946+10:00 xev kernel: RDX: 0000000000007fe2 RSI: 0000000000000fad RDI: 0000000000002000 
2026-08-04T01:13:37.505948+10:00 xev kernel: RBP: 0000000000000000 R08: 000056261498ac40 R09: 0000000000008000 
2026-08-04T01:13:37.505949+10:00 xev kernel: R10: 0000000000000066 R11: 0000000000007fe1 R12: 0000000000007efa
2026-08-04T01:13:37.505950+10:00 xev kernel: R13: 0000000000008000 R14: 0000000000008000 R15: 000000000000ffe0 
2026-08-04T01:13:37.505951+10:00 xev kernel:  
2026-08-04T01:13:37.505953+10:00 xev kernel: ---[ end trace 0000000000000000 ]--- 
2026-08-04T01:55:40.844110+10:00 xev kernel: pcieport 0000:00:03.3: AER: Multiple Correctable error message received from 0000:00:03.3 
2026-08-04T01:55:40.844130+10:00 xev kernel: pcieport 0000:00:03.3: PCIe Bus Error: severity=Correctable, type=Data Link Layer, (Receiver ID) 
2026-08-04T01:55:40.844132+10:00 xev kernel: pcieport 0000:00:03.3: device [8086:6f0b] error status/mask=00000040/00002000 
2026-08-04T01:55:40.844134+10:00 xev kernel: pcieport 0000:00:03.3: [ 6] BadTLP

2026-08-11T09:33:33.473855+10:00 xev kernel: amdgpu 0000:02:00.0: GPU fault detected: 147 0x00024802
2026-08-11T09:33:33.473871+10:00 xev kernel: amdgpu 0000:02:00.0: Process kscreenlocker_g pid 150905 thread kscreenloc:cs0 pid 150912
2026-08-11T09:33:33.473871+10:00 xev kernel: amdgpu 0000:02:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000800
2026-08-11T09:33:33.473873+10:00 xev kernel: amdgpu 0000:02:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0F048002
2026-08-11T09:33:33.473873+10:00 xev kernel: amdgpu 0000:02:00.0: VM fault (0x02, vmid 7, pasid 63) at page 2048, write from 'TC0' (0x54433000) (72)
2026-08-11T09:33:33.473874+10:00 xev kernel: amdgpu 0000:02:00.0: GPU fault detected: 147 0x00004802
2026-08-11T09:33:33.473874+10:00 xev kernel: amdgpu 0000:02:00.0: Process kscreenlocker_g pid 150905 thread kscreenloc:cs0 pid 150912
2026-08-11T09:33:33.473875+10:00 xev kernel: amdgpu 0000:02:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000800
2026-08-11T09:33:33.473876+10:00 xev kernel: amdgpu 0000:02:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0E048002
2026-08-11T09:33:33.473876+10:00 xev kernel: amdgpu 0000:02:00.0: VM fault (0x02, vmid 7, pasid 63) at page 2048, read from 'TC0' (0x54433000) (72)
2026-08-11T09:33:35.481863+10:00 xev kernel: amdgpu 0000:02:00.0: Dumping IP State
2026-08-11T09:33:35.481875+10:00 xev kernel: amdgpu 0000:02:00.0: Dumping IP State Completed
2026-08-11T09:33:35.481875+10:00 xev kernel: amdgpu 0000:02:00.0: [drm] AMDGPU device coredump file has been created
2026-08-11T09:33:35.481876+10:00 xev kernel: amdgpu 0000:02:00.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data
2026-08-11T09:33:35.481877+10:00 xev kernel: amdgpu 0000:02:00.0: GPU fault detected: 146 0x0110040c
2026-08-11T09:33:35.481877+10:00 xev kernel: amdgpu 0000:02:00.0: Process kscreenlocker_g pid 150905 thread kscreenloc:cs0 pid 150912
2026-08-11T09:33:35.481878+10:00 xev kernel: amdgpu 0000:02:00.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000022
2026-08-11T09:33:35.481879+10:00 xev kernel: amdgpu 0000:02:00.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0E00400C
2026-08-11T09:33:35.481879+10:00 xev kernel: amdgpu 0000:02:00.0: VM fault (0x0c, vmid 7, pasid 63) at page 34, read from 'TC3' (0x54433300) (4)
2026-08-11T09:33:35.489845+10:00 xev kernel: amdgpu 0000:02:00.0: ring gfx timeout, signaled seq=5123619, emitted seq=5123621
2026-08-11T09:33:35.489853+10:00 xev kernel: amdgpu 0000:02:00.0: Process kscreenlocker_g pid 150905 thread kscreenloc:cs0 pid 150912
2026-08-11T09:33:35.489854+10:00 xev kernel: amdgpu 0000:02:00.0: GPU reset begin!. Source: 1
2026-08-11T09:33:35.493839+10:00 xev kernel: amdgpu 0000:02:00.0: [drm] ERROR Failed to initialize parser -125!
2026-08-11T09:33:35.737848+10:00 xev kernel: amdgpu: cp is busy, skip halt cp
2026-08-11T09:33:35.897842+10:00 xev kernel: amdgpu: rlc is busy, skip halt rlc
2026-08-11T09:33:35.897852+10:00 xev kernel: amdgpu 0000:02:00.0: BACO reset
2026-08-11T09:33:36.485849+10:00 xev kernel: amdgpu 0000:02:00.0: GPU reset succeeded, trying to resume
2026-08-11T09:33:36.485859+10:00 xev kernel: amdgpu 0000:02:00.0: [drm] PCIE GART of 256M enabled (table at 0x000000F402000000).
2026-08-11T09:33:36.485860+10:00 xev kernel: amdgpu 0000:02:00.0: VRAM is lost due to GPU reset!

Errors on Battlemage

Aug 11 17:01:47 ami kernel: ------------[ cut here ]------------
Aug 11 17:01:47 ami kernel: xe 0000:23:00.0: [drm] DMC 1 mmio[0]/0x5f074 incorrect (expected 0x96fc0, current 0x0)
Aug 11 17:01:47 ami kernel: WARNING: drivers/gpu/drm/i915/display/intel_dmc.c:696 at assert_dmc_loaded+0x275/0x430 [xe], CPU#0: kworker/0:3/215
Aug 11 17:01:47 ami kernel: Modules linked in: intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common xe(+) skx_edac snd_h>
Aug 11 17:01:47 ami kernel:  msr i2c_dev configfs efi_pstore efivarfs autofs4 btrfs libblake2b raid6_pq xor mpt3sas raid_class scsi_transport_sas megarai>
Aug 11 17:01:47 ami kernel: CPU: 0 UID: 0 PID: 215 Comm: kworker/0:3 Not tainted 7.1.7+deb14-amd64 #1 PREEMPT(lazy)  Debian 7.1.7-1 
Aug 11 17:01:47 ami kernel: Hardware name: HP HP Z4 G4 Workstation/81C5, BIOS P61 v03.00 04/15/2026
Aug 11 17:01:47 ami kernel: Workqueue: sync_wq local_pci_probe_callback
Aug 11 17:01:47 ami kernel: RIP: 0010:assert_dmc_loaded+0x291/0x430 [xe]
Aug 11 17:01:47 ami kernel: Code: 24 10 e8 f2 e5 a3 ce 48 8d 3d bb 85 0d 00 8b 54 24 0c 45 89 e9 45 89 e0 48 89 c6 52 8b 4c 24 2c 51 8b 4c 24 30 48 8b 54>
Aug 11 17:01:47 ami kernel: RSP: 0018:ffffd27ac0b87b80 EFLAGS: 00010282
Aug 11 17:01:47 ami kernel: RAX: ffffffffc1743dfd RBX: ffff8c5b80e54000 RCX: 0000000000000001
Aug 11 17:01:47 ami kernel: RDX: ffff8c5b81df5a10 RSI: ffffffffc1743dfd RDI: ffffffffc1605860
Aug 11 17:01:47 ami kernel: RBP: ffff8c5b86955000 R08: 0000000000000000 R09: 000000000005f074
Aug 11 17:01:47 ami kernel: R10: 0000000000000000 R11: 0000000000091050 R12: 0000000000000000
Aug 11 17:01:47 ami kernel: R13: 000000000005f074 R14: 0000000000000001 R15: 0000000000000000
Aug 11 17:01:47 ami kernel: FS:  0000000000000000(0000) GS:ffff8c673e172000(0000) knlGS:0000000000000000
Aug 11 17:01:47 ami kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 11 17:01:47 ami kernel: CR2: 00007ffed1fdcd00 CR3: 0000000ae942a003 CR4: 00000000003706f0
Aug 11 17:01:47 ami kernel: Call Trace:
Aug 11 17:01:47 ami kernel:  
Aug 11 17:01:47 ami kernel:  intel_dmc_enable_pipe+0xe4/0x290 [xe]
Aug 11 17:01:47 ami kernel:  ? drm_crtc_vblank_reset+0x4d/0x120 [drm]
Aug 11 17:01:47 ami kernel:  intel_modeset_setup_hw_state+0xb50/0x1e10 [xe]
Aug 11 17:01:47 ami kernel:  ? intel_display_driver_probe_nogem+0x138/0x1a0 [xe]
Aug 11 17:01:47 ami kernel:  intel_display_driver_probe_nogem+0x138/0x1a0 [xe]
Aug 11 17:01:47 ami kernel:  xe_display_init_early+0xb2/0x140 [xe]
Aug 11 17:01:47 ami kernel:  xe_device_probe+0x3c8/0xb50 [xe]
Aug 11 17:01:47 ami kernel:  ? xe_pm_init_early+0x152/0x160 [xe]
Aug 11 17:01:47 ami kernel:  xe_pci_probe+0xc26/0x1150 [xe]
Aug 11 17:01:47 ami kernel:  local_pci_probe+0x3e/0x90
Aug 11 17:01:47 ami kernel:  local_pci_probe_callback+0x16/0x20
Aug 11 17:01:47 ami kernel:  process_one_work+0x19d/0x3a0
Aug 11 17:01:47 ami kernel:  worker_thread+0x1af/0x320
Aug 11 17:01:47 ami kernel:  ? __pfx_worker_thread+0x10/0x10
Aug 11 17:01:47 ami kernel:  kthread+0xe3/0x120
Aug 11 17:01:47 ami kernel:  ? __pfx_kthread+0x10/0x10
Aug 11 17:01:47 ami kernel:  ret_from_fork+0x2b2/0x340
Aug 11 17:01:47 ami kernel:  ? __pfx_kthread+0x10/0x10
Aug 11 17:01:47 ami kernel:  ret_from_fork_asm+0x1a/0x30
Aug 11 17:01:47 ami kernel:  
Aug 11 17:01:47 ami kernel: ---[ end trace 0000000000000000 ]---

Gunnar Wolf: File recovery in process...

Thu, 13 Aug 2026 17:47:20 +0000

Ohai,

I have some pending, encrypted mails to answer. And some of my answers for the next few days (particularly to what pertains to the current in-discussion vote on LLM usage in Debian) will be unsigned, even though Iâ€™d like otherwise.

My desktop system at work is showing some data corruption, and Iâ€™m slowly backing up my data. Fortunately, it seems I havenâ€™t lost any data, but still, given Iâ€™m letting rsync run until it starts spewing I/O errors, then power down and let the machine cool a bit, and start againâ€¦ it is a potentially long process.

And yes, this makes me somewhat angry. Why angry? Because Iâ€™m working on a brand-new computer (well, have used it for slightly over six months), custom-built to specs requested by my workplace. Specs that I donâ€™t really need, this machine is an utter luxury (i.e. an AMD Ryzen 9 9950X processor with 16 real cores / 32 threads; 128GB RAM in this day and age of RAM shortage, quite recent 32GB GPU, and lots of shiny lights seen in its huge fishbowl cabinet, liquid-based coolingâ€¦). The specs came not from me, but from people who had no idea what we would use them for. And yes, I expect the little fortune spent on this machine to be good for my use for probably a decade, as my previous computer was, but the amount paid wasâ€¦ exorbitant.

But still, what I learned recently is that the 4TB nVME SSD it has (a T-Force TM8FFJX34T) isâ€¦ a very cheap brand, bought because it was close to half the price of other offerings similar in capacity. According to smartctlâ€™s output, th SSD operates with a Warning Comp. Temp. Threshold: 90 Celsius and Critical Comp. Temp. Threshold: 110 Celsius, which sounds sensible, even too high for my standards (my last two laptops have been fanlessâ€¦ yes, an ARM system is very different from a high-end gaming machine). Iâ€™m right now typing from my laptop, which shows 78Â°C and 82Â°C for warning/critical thresholds.

And as expected, under heavy sustained reads (backing up to my NFS server), the desktopâ€™s smartctl shows Temperature: 83 Celsius and, further down, Temperature Sensor 1: 107 Celsius and Temperature Sensor 2: 82 Celsius (donâ€™t know which of these would make the threshold jump). The SSD has sustained Media and Data Integrity Errors: 20 and Warning Comp. Temperature Time: 21 (although Critical Comp. Temperature Time: 0). At least one of my colleagues have shrugged and installed a SATA SSD, laying the huge nVME basically to waste.

Anywayâ€¦ I also learned I am not the first, but the fourth person to notice this kind of issues in this system (out of ten similar purchased systems AIUI). It is completely unacceptable, and Iâ€™ll be pushing our Instituteâ€™s authorities to demand the provider to provide either good component quality for this very expensive system that has many luxury items, or to fix the systemâ€™s build in a way the nVME does not heat as much as it currently does.

Anyway, sigh, I only wanted to say, please excuse me for not using my cryptographic keys for a couple of days ðŸ™ƒ

PS- Iâ€™m also currently not connected to IRC and Jabber (and some similar technologies), as my bouncer runs from my usual workstation.

Jonathan Dowland: DIY skate punch-out

Thu, 13 Aug 2026 08:28:08 +0000

Since I wrote about my fly30 ice skates, I'd continued to battle pain around the navicular bone in my feet. The action that seems to have finally fixed it was to perform a "punch out": a very localized remoulding of the area of the boot that presses against the sore area.

I basically followed the process from this helpful YouTube video.

I narrowed down the exact spot by borrowing some lipstick and transferring it from my navicular bone to the boot lining, then making that more permanent with a sharpie.

My punch was a spare part from a radiator valve which I packed with US cents (I couldn't fit any UK coins in). For the receiving-end, I tried another part from the radiator valve but I think it wasn't sufficiently larger than the punch to work well, so I swapped that out for a spoon.

take 2

I didn't have a temperature sensor I could use and I used a heat gun rather than a hairdryer, so I YOLO'd it a little. Some of the wrap on one of my boots is now distorted from where I didn't move the heat gun enough. It only took a minute or two to get the boot hot enough to be flexible. I set a 15 minute timer once the clamp was in place.

I've only skated one session since I did this but the pain seems to have gone! It's remarkably freeing to be skating without constantly trying to manage pain. Now I can focus on technique.

Steinar H. Gunderson: The PSX GPU is wild

Wed, 12 Aug 2026 15:26:00 +0000

Inspired by some recent reverse-engineering, here are some things I find wild by the original PlayStation GPU:

VRAM is a flat 1024x512 16-bit image (555 + 1 bit alpha). You want more than just a framebuffer? Figure out yourself what goes where.
Yes, that means you'll need to allocate two framebuffers and double-buffer everything yourself.
Quads are common. No “triangles only” business here.
Vertex coordinates are screen-space x/y integers. No floats or fixed-point. (I'm ignoring the GTE here, plus higher-level libraries.)
Wait, where's z? There's no z. So there's no perspective correction. (This one is pretty famous)
OK, so how do you give in subpixel coordinates? You don't. There's no AA after all.
Texture coordinates (u/v) are uint8_t. There's no texture filtering either; everything is nearest-neighbor only.
OK, so that means you can't have textures larger than 256x256 (pretty common in that era), but how do you give in the handle to the texture?
You don't, it points directly to the 1024x512 VRAM. You manage yourself what goes where, remember?
So can you an only have textures in the top-left 256x256? Hah, no, we give you a bit-packed “texture page” system that offsets all your u/v coordinates.
Most textures are paletted to save VRAM (so instead of 555+1, your pixels now mean something like “two palette indexes”). Where does the palette live?
Well, duh, that's a 256x1 (or 16x1, or whatever) area of VRAM too. The GPU does not care, you can use another texture's pixels as a palette if you feel like it.
OK, so you said there's no z, how do you do z-buffering? You have a z-buffer, right… right?
Yeah, sure, we're not cavemen. We have an “ordering table” that is your Z-buffer, drawn back-to-front. If you want 256 levels of Z, you just allocate an array of 256 linked-list pointers, and then you put your polygon into the one corresponding to the correct right Z.
But, eh, what if my polygon is not completely flat in Z-space?
Hello?
Hello…?

Reproducible Builds: Reproducible Builds summit 2026 to take place in Gothenburg

Wed, 12 Aug 2026 00:00:00 +0000

This event is happening soon — see below for registration instructions!

We are extremely pleased to announce the upcoming Reproducible Builds summit, which will take place from September 22nd—24th 2026 in the city of Gothenburg, Sweden.

This year, we are thrilled to host the tenth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Vienna (2025), Hamburg (2023—2024), Venice (2022), Marrakesh (2019), Paris (2018), Berlin (2017), Berlin (2016) and Athens (2015).

If you’re excited about joining us this year, please make sure to read the event page which has more details about the event and location. As in previous years, we will be sending invitations to all those who attended our previous summit events or expressed interest to do so. However, even if you do not receive a personal invitation, please do email the organizers and we will find a way to accommodate you.

About the event

The Reproducible Builds Summit is a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.

Schedule

Although the exact content of the meeting will be shaped by the participants, the main goals will include:

Update & exchange about the status of reproducible builds in various projects.
Improve collaboration both between and inside projects.
Expand the scope and reach of reproducible builds to more projects.
Work together and hack on solutions.
Establish space for more strategic and long-term thinking than is possible in virtual channels.
Brainstorm designs on tools enabling users to get the most benefits from reproducible builds.
Discuss how reproducible builds will be usable and meaningful to users and developers alike.

Logs and minutes will be published after the meeting.

Location & date

Clarion Hotel Draken, Olof Palmes Plats 2, 413 30 Göteborg, Sweden. (OpenStreetMap, Google Maps)
September 22nd to September 24th 2026.

Registration instructions

Please reach out if you’d like to participate in hopefully interesting, inspiring and intense technical sessions about reproducible builds and beyond!

We look forward to what we anticipate to be yet another extraordinary event!

Colin Watson: Free software activity in July 2026

Tue, 11 Aug 2026 10:42:43 +0000

About 95% of my Debian contributions this month were sponsored by Freexian.

You can also support my work directly via Liberapay or GitHub Sponsors.

OpenSSH

Now that Ubuntu 26.04 LTS has been released, I’ve been getting back to the GSS-API key exchange package split in our OpenSSH packaging. Once I started testing my draft openssh-gssapi source package, I realized that I needed to make some changes in the main openssh source package first in order to support it. The dependency from openssh-server to openssh-client was awkward, as was the (related) fact that openssh-client contained shared documentation for other OpenSSH binary packages. After some thought, I created a new openssh-common binary package, moved shared documentation and the ssh-keygen program to that, and dropped dependencies on openssh-client which were no longer necessary (fixing #699473 and #1070098 in the process).

This caused a couple of regressions (#1141420 and #1141550) that I had to fix, and more subtly it also caused a number of autopkgtest regressions in other packages because openssh-client is no longer in base images as a result of a dependency from openssh-server. I believe I have fixes for all of these either pending review or merged (one of which I did in August rather than July):

I upgraded from 10.3p1 to 10.4p1, and in the process contributed a GSS-API option handling fix upstream.

I made openssh-ssh1’s package description more accurately describe the package, thanks to suggestions from Matthias Lang.

Installer team

With support from a Freexian customer, I reviewed, tested, edited, and merged a patch to add VLAN support. I described the details of what I did in a comment.

This has been vaguely on my to-do list since, er, about 2014, so it was very satisfying to get it sorted out.

Python packaging

New upstream versions:

cryptodatahub (fixing a build failure)
dep-logic
django-q
flufl.lock
more-itertools
multipart
pyasn1 (fixing CVE-2026-59884, CVE-2026-59885, and CVE-2026-59886)
pytest-rerunfailures
python-auditwheel
python-build
python-certifi
python-datamodel-code-generator (fixing a build failure, and an incompatibility with pydantic 2.13 that I reported upstream)
python-django-parler
python-httplib2
python-pgbouncer
python-time-machine
python-treq
python-typing-extensions
python-wheezy.template
storm
ubelt
webpy
zope.testing

Other build/test failures:

django-q: autopkgtest failure with Python 3.14 (reported upstream)
httpx: FTBFS: E assert [(‘uvicorn.ac…1.1 200 OK”’)] == [(‘httpx’, 20…1.1 200 OK”’)]
libntruprime: autopkgtest failures with Python 3.14
pygments: Handle None object before HTML escaping (fixed build failures in cmd2, gunicorn, python-inline-snapshot, and python-openapi-core)
python-authlib: some of tests/flask/test_oauth2/rfc9068/ failed
python-click: autopkgtest regression with pytest 9.1 (contributed upstream)
python-httplib2: FTBFS: ERROR tests/test_proxy.py - Failed: ‘forked’ not found in markers configuration option
python-maturin: Please upgrade goblin dependency to 0.10
python-memray: FTBFS on armhf (segfault)
python-mne: FTBFS: E pytest.PytestRemovedIn10Warning: Passing a non-Collection iterable to parametrize is deprecated (actually fixed in scikit-learn; contributed upstream)
python-softlayer: autopkgtest fails with python3-click 8.3.3 (contributed upstream)
python-urllib3: FAILED test/contrib/test_pyopenssl.py

I fixed some other bugs:

I adopted transaction for the Python team.

I attended the Python BoF at DebConf remotely, although a badly-timed fibre outage in the village I live in really didn’t help.

Code reviews

openssh: sshd-session built without crypt(), breaking UsePAM=no password auth (merged and uploaded)
openssh: Slovak debconf templates translation (merged and uploaded)
python-treq: Fails to build source after successful build (merged and uploaded)
python-better-exceptions: FTBFS: failing tests (sponsored upload for Seyed Mohamad Amin Modaresi)
yubihsm-connector: patch to make the build reproducible (merged and uploaded)

Other bits and pieces

Dan Poltawski pointed out in a Fediverse post that the project history didn’t list Sruthi as the current DPL. I fixed that, although it doesn’t look as though the fix is in the published version yet.

I upgraded yubihsm-shell to 2.8.0.

Bits from Debian: DebConf26 Local Team says goodbye

Mon, 10 Aug 2026 21:50:00 +0000

On Saturday 25 July 2026, the annual Debian Developers and Contributors Conference came to a close. The Debian Press team would now like to share this personal and beautiful message from the Santa Fe Local Team.

Words from DC26 Local Team

DebConf26 is over, and those of us who were part of the Local Team are trying to return to “normality”, if such a thing exists after organizing a DebConf.

This event changed our lives and would not have been possible without the help of many great people.

We would especially like to thank everyone who became part of our extended local team. Our endless thanks go to Gunnar —who also instigated this whole adventure—, Santiago, Nattie, Stefano, and Olasd. Thank you for supporting and guiding us, sharing your experience, and helping us find solutions throughout the entire process.

It was also made possible thanks to the great work, strong support and patience of international teams: Fundraising, Bursaries, Content, Video, Treasury, Visa, Website, Accommodation, Front Desk, Cheese and Wine, Publicity as well as all the other teams and individuals who contributed. We apologize if we have forgotten to mention anyone; many people helped make this event possible.

Our deepest thanks also go to everyone who joined us in working on the event, especially Fer, José, and Julián, who showed great commitment and took responsibility for several important tasks.

We would also like to extend our gratitude to FICH, the Universidad Nacional del Litoral, the institutions, organizations, sponsors, suppliers, and everyone who contributed in one way or another to welcoming the Debian community to Santa Fe.

And finally, a very special thank you to our families, to whom we dedicated little time these past few weeks, who supported us on this adventure, enduring the exhaustion, the calls and messages at all hours, and the occasional stressful situation. Always giving us that much-needed, encouraging hug with so much love.

These were very intense weeks, during which we tried to give our best so that everyone could enjoy their stay and so that the Debian community had the necessary conditions to meet, work, share knowledge, and continue creating the magic that characterizes community life and the development of Debian.

As happens at every DebConf, there were difficulties, unexpected situations, and challenges that required us to improvise, learn, and perform a few juggling acts. There were also moments that will certainly remain as memorable anecdotes: the “antisocial room”, some gas heaters worthy of a museum, and newly unlocked powers for negotiating with suppliers.

We have no evidence, but also no doubt, that for many people the Conference Dinner was one of the best moments of the event.

A few ingredients we had hoped would happen naturally were missing, such as more wine nights and at least one in-person football match.

During the two weeks of DebConf, we experienced every kind of weather and a wide range of emotions. Above all, however, we saw people enjoying themselves and building friendships, which fills us with pride.

Thank you very much to everyone who came and helped DebConf26 leave such a beautiful mark on our hearts.

We hope our paths cross again somewhere in life.

Best regards,

Leonardo, Emmanuel, Mariano, Pablo, and Martín DebConf26 Local Team

About Debian

The Debian Project was founded in 1993 by Ian Murdock to be a truly free community project. Since then the project has grown to be one of the largest and most influential Open Source projects. Thousands of volunteers from all over the world work together to create and maintain Debian software. Available in 70 languages, and supporting a huge range of computer types, Debian calls itself the universal operating system.

About DebConf

DebConf is the Debian Project's developer conference. In addition to a full schedule of technical, social and policy talks, DebConf provides an opportunity for developers, contributors and other interested people to meet in person and work together more closely. It has taken place annually since 2000 in locations as varied as Scotland, Bosnia and Herzegovina, India, Korea, France. More information about DebConf is available from https://debconf.org/.

Contact Information

For further information, please visit the DebConf26 web page at https://debconf26.debconf.org/ or send mail to press@debian.org.

Jonathan Dowland: time-delayed scifi roundup feed

Mon, 10 Aug 2026 14:44:31 +0000

I enjoy reading The Guardian's monthly round-up of new SF novels, which can be found in their Science Fiction Books section, and can also be read via feed. Since the round-up is of new books, at the time the round-up is published they're usually only available in hardback.

When it comes to choosing a book to read, these days I am tending towards paperbacks: I've largely ran out of room for hardbacks. So I decided to apply a time delay to their feed. Six months is roughly enough that a book mentioned in a round-up should be shortly available in paperback.

The first obstacle was that The Guardian only publish roughly the last six months of articles in their feed, and so the posts I want have disappeared. However, my Feed Reader (FreshRSS) had older copies stored in its database, and I am able to re-publish those using User Queries. (This also gives me an opportunity to filter out non-roundup articles from the Guardian's feed).

It's then a nice short piece of scripting (this time, using Ruby) to filter the republished feed on the publication date. To make the most recent articles appear new, I also modify the metadata for filtered entries to appear 6 months newer than they are.

#!/usr/bin/ruby
require 'rss'

# replace with the user query feed URI
uri       = 'https://www.theguardian.com/books/science-fiction/rss'
now       = Time.now
sixMonths = 6 * 30 * 24 * 60 * 60
feed      = RSS::Parser.parse(uri)

feed.items.select! do |item|
  item.date + sixMonths < now
end
feed.items.collect! do |item|
  item.date += sixMonths
  item
end

puts "Content-Type: text/xml\r\n\r"
puts feed

I stuck that up on my private web server, subscribed to it in my FreshRSS and voila, a time-delayed list of books to read, most likely available in paperback.

Uwe Kleine-König: PGP Keysigning on Linux Plumbers and OpenSource Summit Europe 2026

Mon, 10 Aug 2026 14:37:00 +0000

I'm going to this year's LPC and Open Source Summit Europe ðŸ¥³.

I will organize sessions on two days after the conference program to exchange PGP fingerprints for keysigning to improve the kernel's web-of-trust (but of course everyone is welcome).

For details see my announcement on LKML. Note the registration deadline at 2026-09-27 08:00 UTC.

Elana Hashman: Managing virtualenvs with a little bash

Sun, 09 Aug 2026 20:00:00 +0000

When you need to install something directly from PyPI, Python virtualenvs have been my go-to for over a decade.

A quick virtualenv intro

Most of my readers are probably already familiar with virtualenvs, but for completeness, I'll give you a brief introduction. A virtualenv (short for "virtual environment") is an isolated distribution of Python packages, where you can independently install packages without disturbing your system packages or other virtualenvs.

You can set one up like this, assuming you are using Python 3.3 or higher:

python3 -m venv ~/.venv/my-virtualenv

The directory specified here is just a convention. I keep all my virtualenvs in the .venv folder in my home directory, but you can pick whatever location you like.

To use the virtualenv, you must activate it:

source ~/.venv/my-virtualenv/bin/activate

This activation script is a special shell script that configures your current shell, pointing at all the right paths in order to use the virtual environment. source runs this script in your current shell session to set it up. You will notice that this adds (my-virtualenv) to the beginning of your shell prompt, reminding you that the "my-virtualenv" virtualenv is active. Now when you pip install amazing-package, the software will only be available in this virtual environment.

When you're done, you can deactivate it like so:

deactivate

Wonderful!

Managing many virtualenvs gets annoying

Over time, I end up accumulating many virtualenvs, which can become harder to manage. Maybe something like this:

$ ls ~/.venv/
my-virtualenv cool-project snakes-ahoy

I also don't want to type source ~/.venv/my-virtualenv/bin/activate every time I use the virtualenv, because it gets very repetitive—only the name of the venv is really needed.

But luckily, we can write a little bit of bash to make managing this less annoying. (Or you can use one of many Python developer tools that are designed to manage this, like pipx, but when I merely want to consume Python software, I might not have a development environment set up. So that's beyond the scope of this post!)

If you add the following shell function to your ~/.bashrc or ~/.bash_aliases file, it will nicely wrap our activation command:

setup-venv() {
        source "$HOME/.venv/$1/bin/activate"
}

Now all we need to run is

setup-venv my-virtualenv

So much quicker!

Spicing it up with tab completion

The first thing I noticed after writing this wrapper was that I started hitting tab on the virtual environment name, but... nothing happened. Wouldn't it be nice to know what virtualenvs I had available, and to not have to type out the whole long thing?

Well, we can write it ourselves 😄

If for some reason you don't already have bash completion installed, on a Debian-based system, you will need to install it with

apt install bash-completion

In order to configure our bash completion, we will create a new file, /etc/bash_completion.d/venv, with the following contents:

_list_venvs()
{
    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts=$(find $HOME/.venv/ -mindepth 1 -maxdepth 1 -type d -printf "%f ")

    COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
    return 0
}
complete -F _list_venvs setup-venv

This file defines another shell function order to determine how to autocomplete the options for our setup-venv function.

$opts is where we define the options for our function. We generate it with a find command—looking at the .venv folder in the current user's home directory, then only including child folders (excluding the current directory itself, .venv, in our results) by using the min/max depth and type arguments, and printing just the individual directory names, deliminated by spaces using our print formatter.

Everything else is the standard scaffolding required to use bash completions.

Once you save this file and reload your shell, you'll see that you are able to use completions as expected!

setup-venv <tab>
my-virtualenv cool-project snakes-ahoy

setup-venv s<tab>
setup-venv snakes-ahoy

Complaints, comments, questions?

Hope this was helpful! If it wasn't, that's too bad. But don't worry—you can safely ignore this post.

Reproducible Builds: Reproducible Builds in July 2026

Sun, 09 Aug 2026 19:12:10 +0000

Welcome to the July 2026 report from the Reproducible Builds project!

In our reports, we try to outline the most important things that we have been up to over the past month. As a quick recap about what problem our project intends to solve, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised or not.

If you are interested in contributing to the project, please visit the Contribute page on our website.

In this month’s report, we cover:

Tool development

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 324, 325 and 326 to Debian:

Fix tests to work with zipdetails 4.0008. (#1141359)
Bump debhelper compatibility level to 13. […]
Update copyright years. […]

In addition, Paul Spooren made changes to allow trailing garbage in Gzip files […] and Vagrant Cascadian added an external tool reference for the pedump binary to use the mono package under GNU Guix. […]

disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into system calls to reliably flush out reproducibility issues. This month, Christelle Gloor added the option to sort by ctime as returned by the lstat(2) syscall. […], which Chris Lamb uploaded whilst bumping the Standards-Version to version 4.7.4 […]. Bernhard Wiedemann also updated disorderfs to version 0.7.0 in openSUSE.

Yet again, there were a number of improvements made to our website this month as well. For example, Chris Lamb, by request of Digital Ocean, changed the target of a referral link so that they can manage incoming referrers […] and pushed a number of changes to the Tools page […].

Distribution work

In Debian this month, 32 reviews of Debian packages were added, 26 were updated and a total of 21 were removed this month, adding to our extensive knowledge about identified issues.

A number of issue types were added by Chris Lamb, including:

python_towncrier_build_date […][…]
log_files_installed_in_package […]
fontforge_varies_by_timezone […][…]

Chris also added a further note for the build_date_in_manpage_generated_by_spf13_cobra issue. […]

In addition, there is a new page showing verification rebuilds of OpenWrt APK packages and firmware images, powered by rebuilderd:

Lastly, Bernhard M. Wiedemann posted another openSUSE monthly update for their reproducibility work there.

Three new scholarly papers

Yan Li, Nan Jiang, Qihang Zhou, Shaowen Xu, Yamin Xie and Xiaoqi Jia of the Chinese Academy of Sciences published a paper titled VCAligner: Aligning Source Distribution Versions with Upstream Git Commits to Secure Supply Chain:

We present VCAligner, a content-based alignment methodology that constructs inverted indexes over VCS histories to precisely map released artifacts to their originating commits, independent of fragile version tags. We evaluated VCAligner on a dataset of 2,984 verifiable PyPI packages derived from the 4,000 most-downloaded projects linked to public GitHub upstreams. Our results reveal a critical weakness in conventional tag-based heuristics: while they appear effective on 85% of the dataset, the residual 15% failure rate generates a catastrophic downstream audit workload of over 10.3 million commits. In contrast, VCAligner reduces this burden by two orders of magnitude (≈ 158×), bounding the total workload to under 65,000 commits. Furthermore, we provide the large-scale characterization of “Packaging Noise,” classifying artifact divergence into structural additions (Path Phantoms) and content mutations (Blob Phantoms), thereby isolating the distinct attack surfaces of malicious injection and code tampering.

Jens Dietrich and Spencer Sun from the Victoria University of Wellington together with Tim W. White and Behnaz Hassanshahi from Oracle Inc pre-published their paper No Snake Oil: Verifying Python Package Builds (PDF):

Python has become the default language for interacting with AI, with packages being distributed through registries like the Python Package Index (PyPI). This creates a need to analyse supply chains comprising such packages. One such analysis is to rebuild packages in order to identify compromised builds injecting malware. Independent rebuilds in hardened environments have the added advantage that they can generate and record provenance in order to increase the trustworthiness of packages. Two tools that are designed to automate such rebuilds and run them at scale are macaron and oss-rebuild. We study 12,180 popular releases from PyPI and find that the byte-for-byte equivalence rate is generally low. We analyse the reasons why they produce different wheels, and find that equivalence between the original and rebuilt wheels can often still be established, preserving most of the guarantees users expect from rebuildable releases. We present and evaluate daleq4py, a tool to establish the equivalence of Python wheels through the kernel of a normalisation function that is based on provenance-preserving datalog rules. Experimental results show that daleq4py substantially expands the set of rebuilds that can be accepted as equivalent. Although only 15.4% of macaron rebuilds and 19.1% of oss-rebuild rebuilds are byte-for-byte identical to the published PyPI wheels, daleq4py establishes wheel equivalence for 60.2% and 78.9% of source-equivalent rebuilds, respectively.

Denise Nanni, Julien Malka, Stefano Zacchiroli and Théo Zimmermann from Télécom Paris together with Gabriele D’Angelo from the University of Bologna pre-published their paper Understanding Build Reproducibility in the F-Droid Ecosystem (PDF), which was accepted at the 2026 ACM Conference on Reproducibility and Replicability:

The security of open source applications benefits considerably from the possibility of rebuilding their source and verifying the output. F-Droid, a prominent distribution for open source Android applications, systematically rebuilds them from source and tests their bitwise reproducibility at app publishing time. However, F-Droid offers no guarantee that app reproducibility will continue to hold in the future. As software ecosystems evolve, reproducibility may degrade, with potential negative consequences for software preservation and security. We present the first empirical study of build reproducibility in the F-Droid app ecosystem. Analyzing historical reproducibility logs, we find that the overall bitwise reproducibility rate has been steadily increasing over time (as new versions of apps are published). We then evaluate how reproducibility holds in time for fixed app versions, by attempting to rebuild 18 904 app versions that F-Droid had previously confirmed bitwise reproducible, published between September 2018 and February 2026, achieving an 83% rebuild success rate, and identify missing dependencies as the dominant cause of failure, accounting for 76% of non-rebuildable cases. Among successfully rebuilt apps, 94% are also bitwise reproducible-i.e., they still yield bitwise identical artifacts upon rebuild. Together, these results show that while bitwise reproducibility largely holds for apps that can be rebuilt, rebuildability itself is highly sensitive to temporal decay.

Patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where applicable or possible. This month, we wrote a large number of such patches, including:

Arnout Engelen:
- ivy
- pekko-grpc
- sbt-ivy
- scala
Bernhard M. Wiedemann:
Chris Lamb:
- #1141505 filed against golang-github-tidwall-wal.
- #1141506 filed against lightproof.
- #1141582 filed against libslow5lib.
- #1141687 filed against siso.
- #1141841 filed against grout.
- #1142126 filed against libpsl.
- #1142492 filed against node-grunt-contrib-internal.
- #1142495 filed against fontforge.
- #1142496 filed against spopt.
- #1142887 filed against go-dlib.
- #1143147 filed against towncrier.
Jochen Sprickerhof:
- #1141412 filed against python-sphinx-chango.
- #1141553 filed against gasnet.
- #1141577 filed against yubihsm-connector.
- #1141663 filed against dh-fortran.
- #1142000 filed against watcher.
- #1142001 filed against rakudo.
- #1142128 filed against kf6-breeze-icons.
- #1142141 filed against oxygen-icons.
- #1142256 filed against gnu-apl.
- #1142513 filed against barvinok.
- #1143169 filed against ecbuild.

Misc news

On our mailing list this month, Colin Winter of Markovian Protocol wrote to our mailing list on the topic of Reproducible verification for retained logs:

Reproducible builds remove trust in the builder: anyone re-derives the same artifact from the same source, byte for byte. The same shape applies one layer over, to a retained record. Most record-keeping regimes (the EU AI Act’s Article 12 logging is the current example) require that events be recorded and logs retained, but not that a retained log be verifiable, by a party who was not present, as unaltered and existing when claimed. That leaves an integrity obligation resting on trusting the party being audited.

(Full thread)

Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

IRC: #reproducible-builds on irc.oftc.net.
Mastodon: @reproducible_builds@fosstodon.org
Mailing list: rb-general@lists.reproducible-builds.org

Debian Disguised Work

Dirk Eddelbuettel: RcppArmadillo 15.6.0-1 on CRAN: New Upstream Minor

Changes in RcppArmadillo version 15.6.0-1 (2026-09-07)

Enrico Zini: Financial risks in 2026

Bits from Debian: New Debian Developers and Maintainers (July and August 2026)

Colin Watson: Free software activity in August 2026

Personal note

LLM vote

OpenSSH

PuTTY

Python packaging

bugs.debian.org

Vincent Bernat: Sidenotes with CSS anchor positioning

Iustin Pop: AI agents aha moment

Dirk Eddelbuettel: RcppFarmHash 0.0.4 on CRAN: Maintenance

Changes in version 0.0.4 (2026-09-06)

Russell Coker: CoMaps

Enrico Zini: Migrating away from .org/.net/.com domains

Steinar H. Gunderson: plocate 1.1.25 released

Michael Stapelberg: Debian Code Search: Fast TurboPFor with Go SIMD

Background: Why does DCS need a fast Integer Codec?

TurboPFor: an analysis

Debian Code Search: positional index, TurboPFor-compressed

SIMD in Go

Starting Point

API design: BlockEncoder, BlockDecoder and streaming

Initial Implementation

Setup

The microarchitecture level: set GOAMD64

Benchmarking setup

CPU counters: perf

Optimizations (scalar)

Profile-Guided Optimization (PGO)

Reducing memory allocations

Generics for bit width specialization

Optimization: Bigger strides with SIMD

SIMD build tags

The 256 uint32 vertical layout

Positional Popcount

The trick: smear masks enable positional popcount

Positional Popcount: a visual explanation

Positional Popcount: Go SIMD

Go even faster?

Conclusion

Dirk Eddelbuettel: rfoaas 2.4.0 at CRAN: Fully Restored Functionality

Junichi Uekawa: Summer Vacation for my kids is over.

Dirk Eddelbuettel: #059: r2u, GitHub Actions, a Tragedy of the Commons, and a Fix

Simon Josefsson: Soft-launching the DiffOS project

Andy Simpkins: A quick experiment with vibe coding

Dirk Eddelbuettel: RcppExamples 0.1.11 on CRAN: Very Minor Maintenance

Changes in RcppExamples version 0.1.11 (2026-09-03)

Birger Schacht: Status update, July + August 2026

Debian Related Work

DH Related Work

Russ Allbery: Review: Too Like the Lightning

Russ Allbery: Review: Last Chance to Save the World

Jonathan McDowell: What do I want in a Linux distribution?

Russ Allbery: Review: The Hands of the Emperor

Ingo Juergensmann: Debian GR on Responsible Use of Generative AI

Joey Hess: Debian and the sirens

Antoine Beaupré: A more nuanced view of LLMs

The Debian vote context

My LLM use

Debian Packaging

OpenAI and Anthropic usage

Actual prompts and responses

How I'm using models now

What I am not using models for

Their LLM use I am forced into

Larger projects LLM use

A plea for small

Gunnar Wolf: As far as LLMs go in Debian, I think that 936241857

Ritesh Raj Sarraf: Taming the AI Agents (Part 2): Cross-Vendor Agent-to-Agent (A2A) Swarms over the Software Forge

Preface: The Unanswered Frontier

1. The Core Realization: The Forge is the Bus

2. Proving Fortified Headless Execution

3. The 3-Tier Memory Architecture

The BANANA Two-Shot Test

4. The Engineering Milestones: From Concept to Production

Milestone 1 & 1.1: Reliable Headless Spawning

The microarchitecture level: set `GOAMD64`

The `BANANA` Two-Shot Test