RFC: build all versions of a package

[bgruening]

It would be nice to build all versions of a package given the version number *, at least for package managers that support it.

Moreover, if we do not consider the build string in conda, we need a way to trigger a rebuild.
What do you think?

[thriqon]

Wildcard Version

This poses many difficult challenges. The first is the algorithm that decides whether to build a given package. Currently, it takes information from the API database and the packages.tsv and (broadly speaking) diffs it. Any packages not in the database are built. A wildcard matching would require a complete list of versions for any package that appears in the packages.tsv with a wildcard.
This diffing is necessary since we can't get a complete list of tags and repositories in our Quay organization in O(1).

The second is the different behavior of package managers. Conda is already a special case in that it treats the revision number differently. I think this is not difficult to 'sell', since it's just a small feature that we can't provide with the others. How should Homebrew react if I give it an asterisk? Alpine?

I could imagine a tool that generates lines for a specific package name and pre-fills a PR. This is not that difficult to implement, and keeps the 'core' mulled easier to reason about.

Rebuilding

This is already supported, albeit a little bit hidden 😉 It directly follows from the diffing above: Just delete an image file from the API database and trigger a rebuild of master in Travis.

---

The question we have to ask is this: Why do we need to rebuild? Is it because the image is damaged, i.e. something went wrong, or because we want to replace a (correct and working) image with something different? Don't we lose reproducibility?

[thriqon]

Something in the style of: http://output.jsbin.com/gewaso/4

[bgruening]

This poses many difficult challenges. The first is the algorithm that decides whether to build a given package. Currently, it takes information from the API database and the packages.tsv and (broadly speaking) diffs it. Any packages not in the database are built. A wildcard matching would require a complete list of versions for any package that appears in the packages.tsv with a wildcard.
This diffing is necessary since we can't get a complete list of tags and repositories in our Quay organization in O(1).

Why are you diffing against the database and not just rebuild/build all lines that have changed in the last git-commit? Assuming that everything that was not touched is already build.

The second is the different behaviour of package managers. Conda is already a special case in that it treats the revision number differently. I think this is not difficult to 'sell', since it's just a small feature that we can't provide with the others. How should Homebrew react if I give it an asterisk? Alpine?

This would be packager-specific feature. Currently, just for conda.

I could imagine a tool that generates lines for a specific package name and pre-fills a PR. This is not that difficult to implement, and keeps the 'core' mulled easier to reason about.

But it will overload at some point the tsv file and makes updating also challenging. If I want to add new bedtools in 1 year I need to do the diff and see which ones are already included in the tsv file.

The interface looks very nice and usable no matter what we decide here :)

The question we have to ask is this: Why do we need to rebuild? Is it because the image is damaged, i.e. something went wrong, or because we want to replace a (correct and working) image with something different? Don't we lose reproducibility?

This was in response to the idea to drop the conda build-number. If we drop them we should build the latest release automatically. It was more about damaged or not fully working builds.
I just want to find a way to make it

• easier to insert a new release (+ every build number) in the tsv file but
• keep the tsv file short and readable
• have the latest version more or less automatically

Let's discuss the rest on Thursday.