In the past, we've discussed with @psabata that a module should be able to support globs of streams it should be built against.

I would still prefer regexes, being more flexible and all. On the other hand, fewer people understand them.

For example, the "allpurpose" stream of the "testmodule" module, may want to be built against base-runtime-master as well as base-runtime-f26. Today, you have to maintain two branches of testmodule: "allpurpose-master" and "allpurpose-f26" and the only different between them is that they declare build time deps on two different base-runtime streams.
Instead, we should support some kind of globbing in the buildsystem, so that you can declare:
dependencies:
buildrequires:
base-runtime: *

When you submit a build of this modulemd to MBS, it should spawn N different module builds, one for every combination of buildrequires.
Some problems arise: Each module-build for testmodule should be uniquely identifiable with its NSV, like "testmodule-master 20170524". Now, however, we are expecting to produce two or more module builds from that same commit. Those built modules will need some kind of distinguishing marker in their names. Perhaps using something like "testmodule-master 20170524.0" and "testmodule-master 20170524.1" may be sufficient?

I don't know. That might be okay for unique identifiers in PDC but:

• the user shouldn't probably see this, they should only get the right module built for their configuration and they shouldn't care there are fifty more variants of the same thing available
• the tooling will definitely need structured info to know what was built against what and what's the expected runtime compatibility

Each of these built modules will need to run time require different streams of base-runtime also, so the globbing will need to be expanded from the build section and copied into the run section. For instance, a modulemd that looks like this:
dependencies:
buildrequires:
base-runtime: *
requires:
base-runtime: *

would need to be expanded into two modulemd for two module builds. One that looks like:
dependencies:
buildrequires:
base-runtime: master
requires:
base-runtime: master

.. and another that looks like:
dependencies:
buildrequires:
base-runtime: f26
requires:
base-runtime: f26

This is not really true. Build time and runtime dependencies might and often do differ quite a bit. We'll need this for both, yes, but I'm not yet sure how to pair them. A difficult example would be a module providing some static data, such as fonts. Your module builds against platform (or base-runtime) in three variants but run-requires platform: *, which should be paired with whatever was used to build that particular variant, and fonts: *, which wasn't needed at build time and really means "I don't care what stream is present on the system, I just need some fonts". Maybe it doesn't make any sense, I don't know :) Will have to think about it some more.

Things of course get complicated if you want to depend on multiple modules with globs for streams. Do we kick off builds of every possible combination?

Yes, most likely.

I've got to stop writing here, but this deserves a document and some discussion about how it will work before we get into any implementation.

+1

Also before we get any further, we should really agree on what the values in dependencies mean. The specification says they are stream names but MBS treats them as refs, I think. Stream names are simpler but refs might be more useful for module build reproducibility and this new globbing feature. Treating them as refs from the start introduces a new set of problems too, however.

Perhaps using something like "testmodule-master 20170524.0" and "testmodule-master 20170524.1" may be sufficient?

I don't see how we can do this with the same NS and only change V. We'd need a DNF that knows to filter module versions based on their requires ... can't we limit this to stream rebuilds?

Say I want to build this module for streams "master" and "stable", and then those have maps for each module dep. Eg.

build-streeams: master, stable
dependencies:
    stable:
        buildrequires:
            base-runtime: f26
        requires:
            base-runtime: f26
    master:
        buildrequires:
            base-runtime: master
        requires:
            base-runtime: master

...but that moduleMD shouldn't leak out of MBS, so I don't think it's that interesting what it looks like.

The big problems I see with "globs" are:

1. What happens to any new streams created after, by the depending module, when testmodule is rebult?
2. What happens if I want "f26, f27" but not f28? And don't let psabat say regexps solve this.
3. What happens if we depend on two modules, which each have three streams, and we need two stream builds?

...in theory some of these could be "just give up and use two module files", but then that works for all cases now.

FYI - I'm reducing the scope of this ticket.

Let's use this ticket to plan the work to implement this. File 2-10 other tickets detailing the work that needs to be done to support this.

Module stream expansion as implemented today solves this issue.

If you disagree with this, please feel free to re-open this issue.

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/koji/mbs/issues/570

Please continue any further discussion there.