Summer of Code: new wanna-build

Getting rid of Packages-arch-specific

2009-05-02T11:45:00.000+02:00

I did a fair bit of (not GSoC-related) Packages-arch-specific maintenance in the pretty recent past. This control file, which even predates build-dependencies being introduced into source packages, specifies which packages are not deemed missing on some architectures and which are thus excluded from autobuilding. Interestingly Ubuntu is also using it for their autobuilding system, which is considerably newer than ours.

As a first step in the cleanup I removed all the entries that were in there solely because the sources in question depend on some architecture-specific packages. This can be solved by automatic dep-waiting directly in wanna-build nowadays. I still think that the whole concept of this file is flawed, though. Basically it works around a misfeature of dpkg-source, which fails to accurately document the architecture set of the source package.

A little example: if you have a package that builds two binary packages, of which one is arch:all and one arch:i386. Now dpkg-source will unhelpfully assume that the package is buildable on every architecture and put "Architecture: any" into the dsc. Now we have no way anymore to get the original information that there are two packages with the architecture set 'all', 'i386'.

If we look at the `debian/control' in the source package however, we can find the information as we need it. If we build a set of all architectures mentioned in there we can determine correctly where it needs building (with the presence of "any" being extrapolated to all architecture we are currently building for).

The practical result of this exercise if we take 'kfreebsd-7' as an other example: The package in question is arch:any according to its dsc. If we build the set of all architectures mentioned we get "all kfreebsd-i386 kfreebsd-amd64". No need for exclusion in a manually-maintained file anymore.

So I patched dpkg-source today (#526617) to collect the necessary information and put it into the dsc. It looks like policy does indeed allow this behaviour and I hope that I did not miss something. I still wonder why this was not fixed within the last decade. Sadly we will still need to continue checking packages which specify arch:any for quite a while as we cannot distinguish packages with properly built lists from packages built with an older dpkg-source.

Even despite this information collection packages could, of course, still fail to build on some architectures because they got their `debian/rules' wrong, despite their `debian/control' mentioning any-ness. But that can be solved with a bunch of bug reports. All in all we should be able to drop this control file hopefully soon.

In other news I tried to determine packages in need of autobuilding in pure Python and got to a disappointing wall-clock time of 25s on the test host for just one architecture. The old quinn-diff only needs half a second for the same job. But it bails down to the fact that iterating through a Packages file takes that long, the additional logic is barely noticeable. At least multiple architectures could be processed in parallel, assuming a multi-core/SMP system on the wanna-build host.

Proposal accepted

2009-04-22T00:07:00.006+02:00

So it turns out that my proposal has been accepted by Debian's GSoC admins and Google. \o/ (Ok, probably also due to the many Debian people judging the proposal.) Looks like I will be kept pretty busy for quite a while.

I already started my work in the last days and I'm progressing slowly. Currently I'm still investigating what to keep and what to drop from the current PostgreSQL schema proposal (some incomplete ramblings available here: part 1, part 2). It contains more data than necessary and it is designed to serve as a new DB backend for the existing wanna-build and not how I imagined it to work. I will iron out those problems with my mentor, Luk Claes, and Adeodató Simo in a meeting somewhere during the weekend.

Wanna-Build PostgreSQL schema (take 2)

2009-04-19T21:26:00.000+02:00

So there we go with the next part of the new database: the binaries and sources representation. I will start with a tiny introduction on how we currently fetch the data, though.

This data is currently fetched from the Packages and Sources files dak on ftp-master generates for us. For unstable it is additionally merged with what others call "incoming". Actually this is not quite true as "incoming" as exported on the web is dak's accepted queue, which contains sources and binaries for all suites (i.e. oldstable, stable, testing, unstable and experimental). The buildds only see a subset: the sources and binaries for unstable. This process, misleadingly named "autobuilt accepted" by its creator, is thus currently unable to provide timely experimental or stable autobuilding as the input files for wanna-build are only refreshed during dinstall. Our FTP masters plan to change dak to just dump Packages and Sources files from the database which then gathers all needed information to be able to do this. At that point it could also be possible for us to just import a dump of dak's data (or getting a realtime feed of it). Canonical unified package building and publishing into one large application (Soyuz), with some synergy effects having all available in one database. (On the other hand I do not have deep insight into Soyuz and my statements on that are mostly guesswork.)

Now to the binaries and sources in the database. Again you can find a ER diagram on the right. The current autobuilding implementation already knows which source versions are available for each suite, quinn-diff can tell what's missing and binary versions are used to clear Dep-Waits. Most of the information is only used during processing and not store. With the new schema all the information about the available sources and binaries, their (Build-)Depends and (Build-)Conflicts is available in the database. I think it is currently collecting more than needed, but that's probably due to the fact that the database layout is already used for transitions tracking. The "depends in a table" modelling is incomplete (and only introduced by transitions addons, to be honest), considering that there could be alternatives. Build-Depends, on the other hand, are text fields. We will need to evaluate which information we really need unless we really want to store the complete Sources and Packages files in the database. What I really want to get done is a sane way to Dep-Wait packages, also with the possibility to look up how many packages are blocked by which other packages, influencing the scheduling decision.

We also need to come up with a sane exclusion/inclusion scheme (based on sources and binaries). One use case is of course the denial of non-free autobuilding except for few whitelisted packages. The other would involve the porting of Packages-arch-specific into the database and enhance it with the possibility of exclusion from, say, non-Linux architectures. Also I really, really want the build slaves to have as few configuration as possible and move stuff like "(weak_)no_auto_build", specifying if a package should be built on a buildd (or only if nothing else is to be done, onto the central coordinator.

Another curiosity are the various architecture tables. One table (not depicted on the diagram) holds all architectures known to the wanna-build instance. Another table (packages_architectures) holds all architectures that are valid for package architectures. Yet another table (source_architectures) stores which architectures a source package should be built for, refering to the packages_architectures table as foreign key. I think we should have one global architectures table and then associate it with the suites and tell what should be autobuilt through that.

I got Dato's importer running on the test host today, mirroring and importing some data from the current official wanna-build host. Let's see if I can tweak the schema a bit in the coming days. I also need to get an idea of PostgreSQL's performance.

Wanna-Build PostgreSQL schema (take 1)

2009-04-17T22:15:00.002+02:00

Some of you might know that there were efforts at the beginning of this year to create a DB schema for an improved wanna-build using PostgreSQL as its backend. The current version uses Berkeley DBs with Perl's MLDBM module to store the data (basically in dumped Perl hashes). For one this is not portable to other programming languages and thus lead to ugly cronjobs dumping and re-importing the data for use with web interfaces like Jeroen's. On the other hand it also resulted in wanna-build scaling badly with more buildds due to locking and performance issues. The version currently in use even tries to lock the database for read-only accesses. History and audit trails were kept in rotated log files.

Roger Leigh, Adeodato Simó and Marc Brockschmidt then sketched something out that could accommodate the current data with more accessible history and also gathers more data about the packages and binaries. Dato subsequently used it to base his transition monitoring on it. I was not around when it got developed, though. So I had to dig into it.

Let's start with the saving of the state data (i.e. which packages need (re)building). On the right you can expand the ER diagram of the build_status part of the database (courtesy of the non-free but free as in beer DbVisualizer). Roger Leigh's initial proposal had a build_jobs table where new records were inserted for new jobs to process. In its current form the build handling is status-based, which means that you get one record per source, suite, arch triplet. Updated rows cause history to accumulate in a separate table build_status_history. This is nearer to the current wanna-build as it also operates only on the current state (although it preserves exactly one state step back in its database files).

I can see the merits of both approaches. Somehow the job-based approach makes more sense to me because I wanted to have the scheduling job-(or rather task-)based anyway. Partitioning should be able to relieve much of the pain involved with keeping history in the same table. Additionally history would not be separated from the current state and thus requiring special-casing when representing the changes trail. The status-based approach relies on a trigger creating history entries (with their new ctime) which is a clean approach that preserves all old entries but might fail if you want to provide more information about a change.

There is one part which did not make it into this ER diagram which is build_states_properties which is intended to store additional information about source, arch, suite tuples. It is currently used to state if a package is not yet compiled or out of date on an architecture, something quinn-diff gives us in the current system to penalize completely new packages. Theoretically we could deduce this from the tables now anyway, probably at a higher computing cost if done on-demand. I will evaluate that in the coming days.

Summer of Code students to be announced soon

2009-04-17T02:32:00.004+02:00

On April 20th we will know who will take part in Google's Summer of Code this year. Sadly student proposals are not made available for the general public, even after the submission deadline. Thus, if you want to know more about the project I proposed, please refer to the project proposal in my personal scratch space on the Google Summer of Code application.