From e4d8259173ec39e47ed61d08661fb78be6255109 Mon Sep 17 00:00:00 2001 From: asamalik Date: May 29 2017 07:47:10 +0000 Subject: [PATCH 1/8] update ifrastructure --- diff --git a/source/architecture/infrastructure.rst b/source/architecture/infrastructure.rst deleted file mode 100644 index a8bbd87..0000000 --- a/source/architecture/infrastructure.rst +++ /dev/null @@ -1,25 +0,0 @@ -Infrastructure -============ - -One part of the so-called “Factory 2.0” - -The goal of this document is to describe a high level plan for what kinds of systems will need to be modified and introduced to Fedora Infrastructure to support building, maintaining, and shipping modular things. - - - -.. rubric:: Background - -We know from the originally nebulous discussions that led to the formation of a modularity initiative in Fedora that we were going to create a new framework for building and composing the distribution, one which could have negative side-effects on our existing efforts if we weren’t careful. “Modularity will be very powerful. It will give us enough power to shoot ourselves in the foot.” We’re talking about allowing the creation of combinations of components with independent lifecycles. There’s the possibility of a combinatorial explosion in there that we’ll need to contain. We’ll do that in part by vigorously limiting the number of supported modules with policy, to be taken up in another document, and by providing infrastructure automation to reduce the amount of manual work required. - -Towards accomplishing the second goal, we set out to study existing workflows and hypothesize what non-automated modularity workflows would entail. - -.. rubric:: TOC - -.. toctree:: - :maxdepth: 1 - - infrastructure/life-of-a-package-update - infrastructure/life-of-a-module-update - infrastructure/infra-services-proposal - infrastructure/two-approaches-to-the-orchestrator - infrastructure/open-questions-and-notes diff --git a/source/architecture/infrastructure/Modularity_Systems.png b/source/architecture/infrastructure/Modularity_Systems.png deleted file mode 100644 index d568179..0000000 Binary files a/source/architecture/infrastructure/Modularity_Systems.png and /dev/null differ diff --git a/source/architecture/infrastructure/infra-services-proposal.rst b/source/architecture/infrastructure/infra-services-proposal.rst deleted file mode 100644 index b68dfb1..0000000 --- a/source/architecture/infrastructure/infra-services-proposal.rst +++ /dev/null @@ -1,34 +0,0 @@ -Infrastructure Service Proposal -====== - -Here is an overview diagram of the set of systems and services that -would be involved in maintaining, building, and shipping a module. - -.. figure:: Modularity_Systems.png - :alt: Modularity_Systems.png - :width: 800px - - -Let’s go through each piece. - -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Dist-git** | We’ll be keeping the definitions of modules in dist-git, using the namespace approach that we already have implemented in pkgdb and dist-git in Fedora Infrastructure. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Branch history** | In the event that we decide on version-pinning approach #2, we’re going to need a way to remember what branch refs pointed to which git hashes and what point in time. This is so that, when a patch is applied to a supported branch for a component, we can determine what modules have been built against the old hash and then rebuild them. As noted in the diagram, it may be that we can extract this information from the git history itself - nonetheless, it will be useful to have a queryable network service that exposes that information to other services. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **pdc-updater** | In Fedora Infrastructure, we currently have a service running that listens to our message bus. When it notices that pungi has completed a new compose, it uploads the metadata about that compose to PDC. In our diagram here we have two depictions of pdc-updater where it needs to be modified to handle two new types of data to be included in PDC. On the left hand side, we want to listen for dist-git changes for modules which have added or removed a dependency. We want to push that dependency to the release-component-relationships endpoint in PDC, which we’ll then query later to find out “what depends on what”. There’s a `taiga card `__ for this requiring changes to `pdc-updater `__. Eventually we'll want to store and query other metadata, so we can e.g. query what components are part of a module, which modules contain a component, too. On the right hand side of the diagram, we want to continue to listen for new pungi composes as we do now, but we want to additionally import information about *built modules included in that compose*. These are modules that we are preparing to ship. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Orchestrator (ad hoc pungi)** | We’re reluctant to build a workflow engine directly into koji itself. It makes more sense to relegate koji to *building and storing metadata about artifacts* and to instead devise a separate service dedicated to scheduling and orchestrating those builds. It will make heavy use of PDC (and perhaps the branch-history service) to know **what** needs to be rebuilt. When a component changes, the orchestrator will be responsible for asking what depends on that component, and then scheduling rebuilds of those modules directly. Once those module rebuilds have completed and have been validated by CI, the orchestrator will be triggered again to schedule rebuilds of a subsequent tier of dependencies. This cycle will repeat until the tree of dependencies is fully rebuilt. In the event that a rebuild fails, or if CI validation fails, maintainers will be notified in the usual ways (the Fedora notification service). A module maintainer could then respond by manually fixing their module and scheduling another module build, at which point the trio of systems would pick up where they left off and would complete the rebuild of subsequent tiers (stacks). | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Taskotron (CI)** | Taskotron itself will likely need only minor patching, to be aware of modules as an entity that can be tested. There will be much more involved work required of the Modularity Working Group to propose and implement some default tests for all modules, as well as some guidelines for writing tests specific to individual modules. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Koji** | As mentioned earlier, Petr Sabata and Lubos Kocman are working on the details here, but here are some highlights: A module defines its own buildroot, which doesn’t inherit from other buildroots. Accordingly, rebuilding a module will entail building its components from source (or from srpm). We’ll be looking for optimizations, so that we can avoid rebuilding binary rpms if the buildroot of a pre-built rpm matches bit-for-bit. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Pungi** | Pungi currently works by (at the request of cron or a releng admin) scheduling a number of tasks in koji. Deps are resolved, repos are created, and images (live, install, vagrant, etc..) are created out of those repos. This takes a number of hours to complete and when done, the resulting artifacts are assembled in a directory called The Compose. That compose is then noted in PDC. Some CI and manual QA work is done to validate the compose for final releases, and it is rsynced to the mirrors for distribution. With the introduction of modules, we’ll have an explosion in the amount of time taken to build all of the repos for all of those combinations which is why we’re going to break out a good deal of that work into the orchestrator, which would would like to pre-build the parts the constitute a compose, **before we ask for them**. Pungi’s job then, primarily, is reduced to harvesting those pre-built artifacts. In the event that those artifacts are not available in koji, pungi will of course have to schedule new builds for them before proceeding. We have a (good) requirement to allow developers to run pungi in a local environment, disconnected from our infrastructure. This will be hard, but worth it. The gist will be to have pungi contain libs that know *how* to build the artifacts. In production, pungi will schedule a koji task, which in turn makes a builder *call* that koji lib to do the work. In an offline development environment, we’ll configure pungi to just call that lib itself, directly. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Comps-As-A-Service** | More information for you on how pungi currently builds a compose: pungi takes as its input a pungi config file which, while it defines many aspects of the compose, it primarily defines the set of outputs: the artifacts. It furthermore takes in a *variants.xml* file which defines the variants to be produced, in terms of comps groups. Those comps groups are then defined in another comps.xml file. They are just groups of packages -- *our variants are currently defined in terms of groups of packages from the flat Fedora package namespace*. At minimum, we’ll need to modify pungi to accept a definition of the variants in terms of modules, but additionally, we have problems with trying to maintain and copy the comps.xml file all around our infrastructure to build things. We’d like to replace that with *CaaS: Comps-as-a-service*, so we can query for this stuff over the network and manage it (hopefully) more sanely. The big work item here is defining the variants in terms of modules. We’ll still need to produce a comps.xml file to mash into the repo metadata, but we will *generate* that file on the fly from CaaS data. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Metadata Service** | This is an optional client-facing service which can provide cached pre-computed resolutions of dependencies. We don’t have anything like this currently for RPMs. It could be nice to have a generic system which can serve fast dep resolutions for all kinds of artifacts. It is optional, because we expect that we can build the client tools to work just fine with the metadata lists distributed over the mirrors (or CDN). If we find we have UX issues with long waits for dep resolution, we could invest work in a system like this to supplement. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| **Build Pipeline Overview** | Another optional client-facing service. It could be nice to be able to query and ask “I have module X installed. Do you have a fresh build of X underway? Is it complete, but not yet available on the mirrors?” This is moreso targeted for developers - it would be nice to be able to query and find the status of any kind of component, module, or image in the pipeline through a homogenous interface. | -+-----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ diff --git a/source/architecture/infrastructure/life-of-a-module-update.rst b/source/architecture/infrastructure/life-of-a-module-update.rst deleted file mode 100644 index 52538a8..0000000 --- a/source/architecture/infrastructure/life-of-a-module-update.rst +++ /dev/null @@ -1,61 +0,0 @@ -Life of a Module Update -====== - - -Here’s our depiction of some scenarios for module maintainers, without -any automation in place. Take a look at the four starting points, -organized into columns, at the top: - -.. figure:: life-of-a-package-update-future-no-automation.png - :alt: Life-of-a-package-update-future-no-automation.png - :width: 800px - - -**Submitting a new module**: This is straightforward. There’s no need to -invent a new submission framework here. We’ll reuse the existing -bugzilla->pkgdb->dist-git workflow we have for packages. It will provide -well-known gate point where we can deny new modules that are unfit for -Fedora (criteria to be determined) would create unsustainable burden -(criteria to be determined) or are problematic in some other way. - -**Updating an existing module**: The flow here is also similar to -existing packaging workflows. A maintain can update their module’s -metadata definition, commit and push it to dist-git, and then kick off a -“module build” in koji (the details for a koji “module build” here are -non-trivial, and are being lead by Petr Sabata and Lubos Kocman). - -**Updating a component**: This is what packagers do today when they -update their package to a new release in rawhide. We may carry modules -that include that package, but they *pin* their dependency to a -*specific version* of that component. Since this is a new version of -that component, we do not have to automatically worry about updating any -modules that depend on this component. - -**Backporting a security fix**: This gets tricky. There are currently -(at least) two different competing approaches for *how a module should -pin its dependencies*: - -#. Modules should specify their dependency versions by the *git hash of - a commit in the dist-git repo of that component*. -#. Modules should specify their dependency versions by a *git branch or - tag on the dist-git repo of that component*. - -Note that the author of this document favors approach #2 and that the -workflow diagram above depicts the steps required of module maintainers -for approach #1. - -There are problems with each of these and full discussion of those is -outside the scope of this document. Stephen Tweedie will be taking up -the question of versioning and releases in a different document. - -In either case, the backporting of a security patch to older versions is -a situation that requires lots of manual work. Patches need to be -applied to the component’s old (either) commits or branches. Dependant -modules either need to have their pinned git hashes updated in their own -dist-git repos (or have their pinned git branch names left unmodified). -Finally, rebuilds need to be scheduled for the entire tree of dependant -modules. - -Whichever approach we decide on for pinning component versions, the -exercise above in figuring out “what a module maintainer needs to do” -has been instructive. diff --git a/source/architecture/infrastructure/life-of-a-package-update-current.png b/source/architecture/infrastructure/life-of-a-package-update-current.png deleted file mode 100644 index 62b6b7c..0000000 Binary files a/source/architecture/infrastructure/life-of-a-package-update-current.png and /dev/null differ diff --git a/source/architecture/infrastructure/life-of-a-package-update-future-no-automation.png b/source/architecture/infrastructure/life-of-a-package-update-future-no-automation.png deleted file mode 100644 index 7d9583c..0000000 Binary files a/source/architecture/infrastructure/life-of-a-package-update-future-no-automation.png and /dev/null differ diff --git a/source/architecture/infrastructure/life-of-a-package-update.rst b/source/architecture/infrastructure/life-of-a-package-update.rst deleted file mode 100644 index 2800264..0000000 --- a/source/architecture/infrastructure/life-of-a-package-update.rst +++ /dev/null @@ -1,9 +0,0 @@ -Life of a Package Update -====== - -For reference, here’s our depiction of the life of a package update as -it travels through Fedora Infrastructure, today: - -.. figure:: life-of-a-package-update-current.png - :alt: Life-of-a-package-update-current.png - :width: 800px diff --git a/source/architecture/infrastructure/open-questions-and-notes.rst b/source/architecture/infrastructure/open-questions-and-notes.rst deleted file mode 100644 index e107934..0000000 --- a/source/architecture/infrastructure/open-questions-and-notes.rst +++ /dev/null @@ -1,76 +0,0 @@ -Open Questions and Notes -====== - -Open Questions --------------- - -As mentioned before, we have an open question about how to **uniquely -identify components** and modules. That’s outside the scope of this -document, but a decision there will have impact on what we need to -build. - -Not mentioned so far - we need a way to **store, query, and view SLA and -EOL information for modules**. Bear in mind that the “EOL” information -for packages currently is tied to the distro-release for that package. -So, whenever F17 goes EOL, that’s when the branch for that package goes -EOL. One of the principal reasons for getting into this whole modularity -mess was to have independent lifecycles between modules; tying it only -to a distro release just won’t do. - -One idea here is to keep the EOL information in the module’s yaml -metadata definition. If we go that route, we’re going to need to also -store a cache of that authoritative data somewhere so that it can be -queried by our other web services. We’ll need to build an interface to -the data for the compose side of the pipeline, so that people putting -together a release of the distro can be sure that none of the modules in -that release are going to go EOL before the distro release is supposed -to go EOL. We’ll need to build a separate interface to the data from the -packager side of the pipeline, so that people modifying modules can see -what the EOLs of things they depend on are, and what things what what -EOLs depend on their module. - -Note that **bodhi** is notoriously absent from the above diagram. That's -because we haven't yet decided how module lifecycles are going to -related to the distro lifecycle. Once we have that figured out, we'll -have to circle back here and figure out how bodhi fits in (since it is -primarily about shipping updates to previous releases of the distro). - -Lastly, we need to appreciate what kind of **load this could put on the -mirror network**. We’ll be creating many more repos. What kind of -overhead does that bring? Will we need to restructure the way mirrors -selectively pull content? Nothing fundamentally changes here. It is a -matter of quantity. - -Notes ------ - -It is useful to step back for a moment and think about some of the -really cool new systems we have, like Koschei and OSBS. - -**Koschei** is a kind of continuous integration service. It monitors new -builds in koji and in response, tries to also rebuild packages that -depend on that package (as scratch buidls). In doing so, it attempts to -find situations where packages inadvertently fail to rebuild from -source, which is really useful. It has to suss out and maintain a -dependency graph to accomplish this. For rpms only, it looks somewhat -like the orchestrator tool in our diagram above (except less committed. -It does scratch builds, not real builds.) - -**OSBS** is a build system that we use to build docker containers. We -run it as a kind of child of koji. It performs builds, and submits them -back for koji to store via koji’s Content Generator API. It is -particularly cool in that it automatically rebuilds containers that -depend on one another. For docker containers only, it looks somewhat -like the orchestrator tool in our diagram above (except less integrated -with our environment. It is buried behind koji.) - -You can see that we’re all heading towards some of the same patterns in -our approach, but we lack a unified approach to modifying the pipeline -as a whole. That’s understandable.. It’s big and hard to change! -Furthermore, there’s likely a social explanation at root, with respect -to `Conway’s Law `__. We -have an opportunity here to change that. - -If we can solve the dep-graph modelling, chain-rebuilding, and CI -problems generally in the pipeline, then we’ll be all the more situated -to easily adapt to the next wave of technological change. diff --git a/source/architecture/infrastructure/two-approaches-to-the-orchestrator.rst b/source/architecture/infrastructure/two-approaches-to-the-orchestrator.rst deleted file mode 100644 index 283b029..0000000 --- a/source/architecture/infrastructure/two-approaches-to-the-orchestrator.rst +++ /dev/null @@ -1,78 +0,0 @@ -Two Approaches to the Orchestrator -====== - - -In the Modularity Group, we have two approaches to take in how the -orchestrator and koji relate to each other. They differ in which system -owns the code for finishing the build of a module. - -Modules as first-class citizens in koji -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In this approach, koji owns the code for building a module from -front-to-back. - -A packager might execute the following: - -:: - - $ fedpkg module-build - -Which would talk to koji over XMLRPC and schedule a module-build. That -module build would entail rebuilding \*all\* of the constituent RPMs -from source as subtasks, and it would in the end create a repo for them -with the built module metadata baked in. - -When that build of the module completes, it would publish a message to -the bus which gets picked up by taskotron which would run any checks we -have defined for whole modules. - -When those checks finish, they publish a message which gets picked up by -the orchestrator. In this approach, the orchestrator is \*very\* simple. -Its pseudocode looks like this: - -:: - - def on_event_from_taskotron(event): - - if not event.is_about_a_module: - return - - if not event.check_passed_successfully: - email_relevant_people("rebuild is stuck") - return - - dependants = ask_PDC_for_dependants(events.module_that_just_got_built) - for other_module in dependants: - koji.schedule_rebuild_of(other_module) - -And that's it. It listens to the bus only. There is no other way to talk -to the orchestrator. Users initiate the process by asking koji - -Orchestrator as the owner of module-build -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In this approach, modules are not first-class citizens in koji, and the -orchestrator owns more of the logic about how to build them. - -A packager might execute the following: - -:: - - $ fedpkg module-build - -Which would talk to the orchestrator over a REST api (which means it has -to be a webapp now). The orchestrator would pull down the yaml file and -start scheduling builds for all of those components individually in a -tag/target specifically for that module. - -It has to keep track (in a database) of all the ones it has submitted so -that, as they finish and each get validated by taskotron, it can know if -they are now all done or not. - -Once they are all done, it then has to build the repo (either itself, or -by scheduling another task in koji that would have to be modified -anyways to insert the built module metadata yaml). - -Finally, it would have to publish a fedmsg message saying that it is -done. diff --git a/source/architecture/pdc-changes.rst b/source/architecture/pdc-changes.rst deleted file mode 100644 index a4d1cc5..0000000 --- a/source/architecture/pdc-changes.rst +++ /dev/null @@ -1,10 +0,0 @@ -Changes to PDC -========= - - -.. rubric:: TOC - -.. toctree:: - :maxdepth: 1 - - pdc-changes/database diff --git a/source/architecture/pdc-changes/Modularity-Architecture-PDC-Compose-Release-Trees-ER.png b/source/architecture/pdc-changes/Modularity-Architecture-PDC-Compose-Release-Trees-ER.png deleted file mode 100644 index 9cc2bb9..0000000 Binary files a/source/architecture/pdc-changes/Modularity-Architecture-PDC-Compose-Release-Trees-ER.png and /dev/null differ diff --git a/source/architecture/pdc-changes/database.rst b/source/architecture/pdc-changes/database.rst deleted file mode 100644 index 2101cdc..0000000 --- a/source/architecture/pdc-changes/database.rst +++ /dev/null @@ -1,38 +0,0 @@ -Database structures -================================== - -PDC already stores data about products, releases and related -information, but for modules we need to record additional things. - -This page is about these changes, and how they should fit together with -the existing structures. - -Models ------- - -The main new models are ``UnreleasedVariant`` which is where we store -modules (but other stuff may use this, too, in the future, so it's a -generic term) and ``Tree`` which carries each tree created for one such -module. This is so we can store e.g. RPM packages for a module which -isn't yet tied to a release, because we may not know that at the time we -build the module. There are peripheral models, e.g. for storing -inter-module dependencies, but they aren't fleshed out yet, so not -described here. - -Diagram -------- - -This abridged ER diagram outlines the changes we plan to do in terms of -what's stored in the database. Everything upwards from and including -``ReleaseVariant`` are existing tables, everything below is new for -Modularity. New relations are from the ``UnreleasedVariant`` to -``ReleaseVariant`` and from ``Tree`` to ``ComposeTree``. - -.. figure:: Modularity-Architecture-PDC-Compose-Release-Trees-ER.png - :alt: Modularity-Architecture-PDC-Compose-Release-Trees-ER.png - - -See Also --------- - -:doc:`../infrastructure` diff --git a/source/infrastructure/infrastructure.rst b/source/infrastructure/infrastructure.rst new file mode 100644 index 0000000..ab6b2a8 --- /dev/null +++ b/source/infrastructure/infrastructure.rst @@ -0,0 +1,18 @@ +Infrastructure +========= + +Modularity will be built using the `Factory 2.0, documented on the Fedora wiki `__. + + +Focus Documents +--------------- + +The Factory 2.0 team produces a confusing number of documents. The first round was about the Problem Statements we were trying to solve. Let’s retroactively call them Problem Documents. The `Focus Documents `__ focus on some system or some aspect of our solutions that cut across different problems. The content here doesn’t fit cleanly in one problem statement document, which is why we broke it out. + +**The most important ones related to Modularity include**: + +`Module Build Service (MBS) `__ - A service that orchestrates module builds in the Fedora infrastructure. + +`Arbitrary Branching `__ - Support of version branches in dist-git which allows us to maintain multiple versions of packages and modules. + +`Freshmaker `__ - A service that will automatically rebuild artifacts (like module repositories or container images) to make sure that everything is up to date. From b276b4e69ec87b21f5baad0e7c858873ac9f004b Mon Sep 17 00:00:00 2001 From: asamalik Date: May 29 2017 07:47:44 +0000 Subject: [PATCH 2/8] rename Architecture to Design --- diff --git a/source/architecture/building-naming.rst b/source/architecture/building-naming.rst deleted file mode 100644 index a3fe526..0000000 --- a/source/architecture/building-naming.rst +++ /dev/null @@ -1,49 +0,0 @@ -Building and naming modular things -========= - -We have some basic terminology confusion around modules. Is a container -image the same thing as a module? Is a software collection a single -module, or a group of modules? We can often get away with being vague, -but for technical planning we need to be able to distinguish between all -these concepts. - -We use these terms: - -- **Package**. Essentially, the same thing as an rpm. In the future it - might be non-rpm content but should fit the same role. -- **Module.** A set of packages tested and released together as a - distinct unit, complete with the metadata needed to manage it as a - unit. May depend on other modules. -- **Stack.** A complete tree of modules. A stack can be thought of as - a top-level module, with the understanding that we’re implicitly - including all of that module’s dependencies in the stack. HOwever, when - the stack is delivered to the end-user it appears as "just another module." -- **Artifact** or **image**. An actual set of bits built out of - modules, in a format intended to be distributed or deployed in some - way. - -Generally, these serve distinct purposes. A module is a building block; -a stack contains all the software for a complete solution; an artifact -is a concrete object containing a stack (or stacks) for distribution to -users. - -We will also distinguish between: - -- A **Build** of a package: a process which involves compiling source - code and creating a packaged output -- A **Compose** of a module: a process which assembles pre-compiled - packages into an organised module, but which includes no compilation - step itself. Sometimes we also use the term **Coallesce** for this scenario - to distinguish between the existing, Fedora term "Compose" which is very - similar but not quite the same. - -.. rubric:: TOC - -.. toctree:: - :maxdepth: 1 - - building-naming/package - building-naming/module - building-naming/stack - building-naming/artifact - building-naming/fit-together diff --git a/source/architecture/building-naming/artifact.rst b/source/architecture/building-naming/artifact.rst deleted file mode 100644 index 2deaed4..0000000 --- a/source/architecture/building-naming/artifact.rst +++ /dev/null @@ -1,19 +0,0 @@ -The Image or Artifact -===================== - -We build stacks into *images* or other *artifacts* such as ostree trees. -These are the formats in which we release a stack. - -For example, we could take a single stack such as a LAMP stack, and -build and release it both as SCL rpms in a yum repository, and as -container images: indeed, we do so today. It’s the same stack in each -artifact. We could add more such formats: virtual appliances, vagrant -boxes, anaconda isos or ostree trees, and it would still be the same -stack in each; only the artifact / format / image has changed. - -.. figure:: building-image5.png - :alt: building-image5.png - -The format of the content we deliver may be different in each case, but -it’s the same binaries from the same stack inside, and the same stack -metadata describing that content. diff --git a/source/architecture/building-naming/building-image1.png b/source/architecture/building-naming/building-image1.png deleted file mode 100644 index 6b955f6..0000000 Binary files a/source/architecture/building-naming/building-image1.png and /dev/null differ diff --git a/source/architecture/building-naming/building-image2.png b/source/architecture/building-naming/building-image2.png deleted file mode 100644 index a17a8f1..0000000 Binary files a/source/architecture/building-naming/building-image2.png and /dev/null differ diff --git a/source/architecture/building-naming/building-image3.png b/source/architecture/building-naming/building-image3.png deleted file mode 100644 index 59c2ccb..0000000 Binary files a/source/architecture/building-naming/building-image3.png and /dev/null differ diff --git a/source/architecture/building-naming/building-image4.png b/source/architecture/building-naming/building-image4.png deleted file mode 100644 index d180292..0000000 Binary files a/source/architecture/building-naming/building-image4.png and /dev/null differ diff --git a/source/architecture/building-naming/building-image5.png b/source/architecture/building-naming/building-image5.png deleted file mode 100644 index 91c53fd..0000000 Binary files a/source/architecture/building-naming/building-image5.png and /dev/null differ diff --git a/source/architecture/building-naming/building-image6.png b/source/architecture/building-naming/building-image6.png deleted file mode 100644 index 8456279..0000000 Binary files a/source/architecture/building-naming/building-image6.png and /dev/null differ diff --git a/source/architecture/building-naming/fit-together.rst b/source/architecture/building-naming/fit-together.rst deleted file mode 100644 index 9cc0d73..0000000 --- a/source/architecture/building-naming/fit-together.rst +++ /dev/null @@ -1,30 +0,0 @@ -How do these parts fit together? -================================ - -Let’s look at a couple of examples to make it (hopefully!) clear what -we’re talking about. - -Example 1: Our Atomic Host stack -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We already release content that is organised in much this way, just not -with any formal definition of modules or stacks. - -Atomic, for example, is composed from multiple sources. Some content -comes from traditional Fedora with its usual 6-month lifecycle; and some -comes from other repos with much shorter lifecycle and rapid rebasing to -pick up new features. - -And even within Fedora we have different levels of lifecycle. An obvious -example is Gnome; the Anaconda installer stack that we use for Atomic -installer ISO images depends on this faster-moving content, even when -the installed atomic host does not include it. - -But even though these distinct sets of components all have differing -lifecycles, they still belong within Fedora and we still produce -coherent output images or artifacts from them today, with planned -releases that require alignment between all the relevant parts. In the -language of this document, it might look like this: - -.. figure:: building-image6.png - :alt: building-image6.png diff --git a/source/architecture/building-naming/module.rst b/source/architecture/building-naming/module.rst deleted file mode 100644 index 753b052..0000000 --- a/source/architecture/building-naming/module.rst +++ /dev/null @@ -1,40 +0,0 @@ -The Module -========== - -We group these binary packages into *modules*. Note, we can pick just a -subset of the subpackages from a particular build into a given module, -eg. a hypothetical example base runtime module: - -.. figure:: building-image2.png - :alt: building-image2.png - - -These are internal build groupings: basically *repositories*, but with -additional metadata and semantics that allow them to play nicely with -other modules and module tooling. - -We attach metadata to these modules, on top of the normal metadata in a -repository. Metadata may include: support metadata (SLAs or EOL dates for the -module); metadata on how to use the module (eg. its dependencies, -default installed packages); identity (builder, vendor, version etc), or -many other things that we’re still just starting to imagine. - -We should be able to update the module as a unit: updating and testing -its component parts as necessary, but also testing the module as a whole -before releasing an update. - -We refer to the assembly of a module by a build tool as a *compose*. -This step includes no compilation; it is merely the creation of a -bundled set of packages and metadata comprising a single module. The -closest analogue in today’s build system is the “puddle build”, where a -custom repository is created for a particular purpose. - -When we talk about a **module compose**, we are *not* talking about “the -Compose” of a full distribution release. The two are similar, in that -they both assemble existing compiled packages into bundled output; but -the module compose is just one single targeted assembly, whereas the -full distribution Compose typically creates multiple repositories and -images as output in a single large job. Also note that we can, in theory, -perform the build (compilation) of the packages within a module, and the compose of -the module itself, as a single step; we might term this a **module -build.** diff --git a/source/architecture/building-naming/package.rst b/source/architecture/building-naming/package.rst deleted file mode 100644 index b66b446..0000000 --- a/source/architecture/building-naming/package.rst +++ /dev/null @@ -1,17 +0,0 @@ -The Package -=========== - -We can start with the familiar ''package ''and *subpackage*, *built* as -usual from a *component* (or *source package*): - -.. figure:: building-image1.png - :alt: building-image1.png - -We’re not changing anything at this level (yet… non-rpm packages are a -possible topic for the future!) - -It is important to distinguish the build of a package from the compose -of a module; the build here is the step which includes compilation and -creation of a reusable bundle of the compiled output. We might be -building specifically to compile for a module’s contents; but the build -step is still a distinct step. diff --git a/source/architecture/building-naming/stack.rst b/source/architecture/building-naming/stack.rst deleted file mode 100644 index adec375..0000000 --- a/source/architecture/building-naming/stack.rst +++ /dev/null @@ -1,34 +0,0 @@ -The Stack -========= - -We can then combine modules into *stacks*. - -A stack should represent something distinct that the user wants. It may -be a traditional developer stack (LAMP, ruby-on-rails, etc.); or it may -be an application (but extended to include all the dependencies that -the application needs to run); or it could be the set of modules needed -to deliver something like Atomic Host or Cockpit. - -.. figure:: building-image3.png - :alt: building-image3.png - -The stack represents this full set of software. It doesn’t presume how -we distribute it, we’re still just talking about the set of modules -making up the stack. - -**A stack is still just a module here.** It’s just a way of referring to -the module plus all its implied dependencies as a single unit, to -distinguish that from the individual modules within the stack; the stack -content and metadata may have exactly the same format as module metadata -(the metadata is the same colour here for a reason!) But it’s still -important (at least internally, probably not for end users) to make the -distinction between a single module, and a module plus all the external -dependencies it relies on. - -Importantly, we can take two modules with different lifecycles and -combine them in a single stack. The definition of the stack gives us the -way to plan and track the relationship or dependency between the -modules. - -.. figure:: building-image4.png - :alt: building-image4.png diff --git a/source/architecture/constructing.rst b/source/architecture/constructing.rst deleted file mode 100644 index 41e2593..0000000 --- a/source/architecture/constructing.rst +++ /dev/null @@ -1,49 +0,0 @@ -Constructing a modular distribution -========= - -The fundamental objective of the -Modularity effort is to -break up the monolithic concept of a “distribution release” or -traditional Compose into something more fine-grained. We should be able -to release applications or stacks such as LAMP or ruby-on-rails on a -lifecycle that suits the application, rather than being dictated by the -distribution release schedule; and we should be able to compose releases -more flexibly from the various components available. - -But modularity is unlikely to succeed if it requires both users and -maintainers to immediately and completely discard the existing concept -of Releases and Editions of Fedora. So in this document we look at how -to break the traditional distribution down into a modular construction, -while still preserving the ability to build the traditional releases. - -This document does not try to explore how we should organise a -distribution release in detail. Exact lifecycles for kernel vs. -applications, for containers vs. baremetal etc. are beyond the scope -here; indeed, our objective is explicitly ''not'' to assume what we need -in a release, but rather to add flexibility so we can change release -objectives later on. If we want Fedora Server to have a longer lifecycle -than Workstation; or for Atomic to rebase docker more rapidly than -Server; or for a new python to be released outside the normal -distribution cycle; all these things may be possible if we have a more -flexible underlying release structure. The flexibility is the subject -here; planning the actual releases is a different topic. - -But, ultimately we still need the well-defined concept of a **release**, -so that all the modules we are maintaining can still come together into -a well-tested, planned release on a known schedule. We also define and -justify a **static manifest** to assign packages to modules. - -Once we have parts of the distribution on different release cycles, we -also have the issue of how to maintain different versions branching on -different criteria and different schedules. Branching and versioning is -relevant here, but is a complex topic in its own right and is covered under -:doc:`/architecture/versioning`. - -.. rubric:: TOC - -.. toctree:: - :maxdepth: 1 - - constructing/breaking-down - constructing/compose-distribution - constructing/back-together diff --git a/source/architecture/constructing/back-together.rst b/source/architecture/constructing/back-together.rst deleted file mode 100644 index 31f48d7..0000000 --- a/source/architecture/constructing/back-together.rst +++ /dev/null @@ -1,70 +0,0 @@ -Putting it all back together -============================ - -We need the flexibility of a smaller, more modular distribution that can -be reassembled in various different fashions. But we also need to be -able to reproduce our current product lineup, or something very similar, -in order not to disrupt our existing install base: - -Example: Reproducing the traditional distribution versions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Having categorised the Fedora rpms into groups and composed those -package groups into distinct modules, creating what we have -traditionally called a distribution variant or edition can now be done simply by -creating a flat yum repository containing all the desired modules for -that offering. In the terminology we are using here, the output -repository is simply another output artifact. - -.. figure:: constructing-image6.png - :alt: constructing-image6.png - - -And of course we can repeat this for different Fedora editions. Having -separate GTK and Gnome modules would allow us to include those modules -when composing the Fedora Workstation repository, but omit those from -the Cloud Edition if we chose. It would also give us a natural way -to reflect that Gnome may rebase at a different cadence from the Base -Runtime. - -So today’s editions can simply become different ways of selecting from -the available modules. - -The installer iso can also be composed from the various modules here, as -a separate output artifact. The main difference from the traditional -monolithic Compose is that the repositories, images etc. that are -composed today are still composed, but each is created as a logically -separate build. - -Add-ons are not drawn above. An add-on is almost completely synonymous -with a module here, and the modules composed as above can naturally -reflect the composition of add-ons. - -Example: Creating atomic host/container images -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A key point of a broken-down, asynchronous build toolchain is that we -can create composes from different modules with different SLAs or -lifecycles, and can combine both modules and existing artifacts into yet -further artifacts. - -For example, we can define a tree of required artifact builds for Atomic -Host and our Docker container images (each purple box here represents a -distinct artifact created by a distinct compose task): - -.. figure:: constructing-image7.png - :alt: constructing-image7.png - - -The Fedora Server and COPR repos today already have distinct -SLAs and lifecycles. We could build atomic host trees out of both of -those sources, and this can be done easily as a compose step based on -the Base Runtime and Container Runtime modules in a modular build of -the same components. - -But we also need to support layered builds, such as additional container -images layered on top of the base image, or Anaconda installs of an -existing ostree tree. As long as we can track which compose steps are -used as input to each compose task, we can track the relationship -between these output artifacts even when we have decomposed the -monolithic Compose into many parts. diff --git a/source/architecture/constructing/breaking-down.rst b/source/architecture/constructing/breaking-down.rst deleted file mode 100644 index 72c0d37..0000000 --- a/source/architecture/constructing/breaking-down.rst +++ /dev/null @@ -1,68 +0,0 @@ -Breaking down the monolithic release -==================================== - -We have many steps in the release pipeline today: starting with bugzilla -and dist-git; building and tagging in brew; the errata tool and mirrors for -pushing the bits out. - -But currently it all comes together in The Compose. - -Remember, I’m not talking about branching and versioning in this -particular document. So let’s start with the assumption that we’ve -already built our individual packages, and that (for now) we have a flat -namespace of binary rpms already built from all our components. That -namespace could be living in a yum repository like the rawhide repo, or -a koji tag such as fedora-24.0-candidate. - -.. figure:: constructing-image1.png - :alt: constructing-image1.png - - -We build all of the images and repositories, for all of the -architectures and their addons, for all of the variants, all at once. -Fedora’s Editions have a similar structure. - -This served us well once upon a time. It enabled a single consistent -major or minor release, and it built everything necessary for that -release. - -But it now falls short on multiple fronts: - -- We have many more types of artifacts being built than are handled by - the compose. We have ostree trees and installer images; there are - cloud targets such as qcow images and AMIs; and docker base and - layered images. We have additional, often ad-hoc, - incompletely-automated builds to create this additional content. -- The modularity effort is explicitly trying to get away from the - concept of a single monolithic distribution, and to release modular - parts of the distribution on independent release cycles. -- Scaling: as we increase the number of modules, we do not want to - spend the effort of rebuilding the entire distribution when any small - part changes. For example, with more container images to build, we - should be trying to rebuild only those affected by any change. -- Self-service: to scale the modular decomposition of the distribution, - we will need the ability for individual engineers or groups who own a - module or image to build that themselves, not dependent on release - engineering. -- Continuous Integration. For automated testing, we want rebuilds to - happen automatically when a dependency changes, instead of having to - wait for a compose that happens on a predetermined schedule, or when - manually triggered by release engineering. - -So how can we address some of these concerns? We take the following -steps: - -- Break down the distribution compose into smaller parts: -- Ensure each part can be composed based on configuration and content - in SCM. **Everything must be recomposable based on static content**: - never, ever require manual configuration of a compose. That way - composes can be automated; -- Combine the smaller composes up in stages building towards a full - distribution; -- Record a compose ID for every stage of the compose, and record which - compose IDs are used as input to subsequent layered composes -- Record the most recent successful compose for each module, so that - failed composes (or composes that fail testing) do not impact layered - modules -- Optionally, we can automate the process of chain composes and add CI - to the mix to achieve a fully automatic build toolchain. diff --git a/source/architecture/constructing/compose-distribution.rst b/source/architecture/constructing/compose-distribution.rst deleted file mode 100644 index a106478..0000000 --- a/source/architecture/constructing/compose-distribution.rst +++ /dev/null @@ -1,262 +0,0 @@ -Incremental compose of the distribution -======================================= - -**1. Break the distribution into distinct modules** - -Let’s take this in stages. We can start by simply assigning individual -packages to modules (at the sub-package level: we may have a need to -have a library in a base module but have its -devel subpackage in a -developer-only module, for example.) This ignores the branching of -packages and modules, but as stated before, that’s a separate topic. - -Note: `The module boundaries here are just examples.` They are -illustrations to show how we can break the monolithic distribution up. -They are *not* proposals that we should draw the lines between modules -in any particular places: that is an entirely separate discussion. - -We can start with some obvious “core” modules such as hardware -enablement, system runtime and core application runtime. - -The installer needs GTK and Anaconda; let’s make sure we have homes for -those. - -This leaves a lot of content currently not assigned to a module. That’s -fine, we can figure out how to handle those; for example, we could -choose to define a module for “uncategorised” for now. - -All we have done so far is to draw boundaries around groups of existing -packages: - -.. figure:: constructing-image2.png - :alt: constructing-image2.png - - -We can add Apache, php and mariadb, and we could have the components of -a LAMP stack, etc; there are plenty more groups we can imagine here. - -Note: This picture shows a static allocation of packages to modules. So, -modules have a **static manifest**. This is deliberate: - -If, over time, a package in a module picks up a new dependency, then it -should be a human decision to determine which module that new dependency -should live in (or even whether we want that dependency at all.) This -allows us to combat dependency sprawl over time, and also to record -*why* a particular package lives in a particular module, at the time we -add a dependency. - -To enable an engineer to manually allocate new dependencies to modules, -it it necessary that module manifests should not just be automatically -generated. We consider that manifest to be one of the defining -properties of the module. - -**2. Automate the composing of modules from packages** - -We now want to compose a module. Bear in mind, “composing” the module -here involves assembling pre-existing binaries: it’s more a “createrepo” -than “rpmbuild” (and is similar to a “puddle” build in current RCM -terms.) And the automation we’re talking about here is the execution of -the module compose, even if the compose is actually triggered manually. - -A module compose is now simply the creation of a yum repository of the -required packages or subpackages, with the addition of a small amount of -module metadata as described in :doc:`../building-naming`. - -.. figure:: constructing-image3.png - :alt: constructing-image3.png - - -We plan the module definition metadata living in dist-git, similarly to -how we currently maintain component spec files, sources definitions and -patches. Regardless of where it lives, **the module compose must be -well-defined based on static content.** The essence of an automated -build system is that when a compose is required, there should be no -compose-time configuration added; everything to compose the module is -stored in SCM and should be completely repeatable from there. - -This workflow mirrors existing practice for building binaries, where a -developer expects to be able to build and test an rpm locally for a -single architecture, before submitting to the build system to be built -on multiple architectures. In the same way, we would expect the -build-system module compose tool to pull together the necessary packages -and to create arch-specific modules for each required architecture. - -We mentioned before that we expect modules to have a static manifest. -During module compose, however, we would still expect to perform -*repoclosure* on the module: ie. to verify that the module satisfies the -dependency requirements of all packages it includes. Failure of -repoclosure should constitute a failed compose of the module. The -tooling may still attempt to solve missing dependencies automatically, -but should only report to the user which packages could be included to -satisfy those dependencies; it should not include those in the output -without confirmation from the module owner. - -To automate module composes it will be vital to know exactly which -version of which packages should be used to compose any given module -(eg. for koji builds, this would be selected by koji tag.) The important -point is that all of this should be configured in the module metadata so -that we can compose the entire module automatically. - -**3. Automate the composing of stacks from modules** - -Recall that a stack is nothing more than a module that depends on other -modules; the stack simply refers to that whole tree of dependencies. - -So composing a stack requires nothing more than composing a module that -has dependencies. We define that top-level module via a metadata file in -dist-git just as for any other module. The required dependent modules -are listed in the module’s defining metadata. This is important: just as -for a compose of an independent module, the entire module compose is -well-defined based on static content. - -The only difference in the compose of such a higher-level module is in -the checking of internal rpm dependencies: both the repoclosure test, -and any hinting done to suggest missing packages, should be performed on -*all* packages in the entire stack, including all of the packages from -other modules that this module depends on. - -Of course, we may be composing a module multiple times. And to manage -consistency between modules when things are changing, we really need to -be able to determine exactly which version of a module has been composed -from exactly which versions of its dependencies. So "each module -compose must have a unique compose identifier" of some form; and -during a layered module compose, we must record the compose IDs of all -the modules used as input for this compose. - -So to compose a PHP-on-Apache stack we might simply compose the base -runtime module, then a layered Apache module, and finally a php module -on top, in that order: - -.. figure:: constructing-image4.png - :alt: constructing-image4.png - - -**4. Automate the composing of images and other artifacts** - -So, now we’ve got enough modules to represent entire stacks of content -suitable for installation to solve some user problem. But still, these -are no more than repositories of available content; we still need to -actually install them to use them. - -Sometimes the user will install them themselves; the module client can -automate the complete installation of an entire stack of dependent -modules seamlessly. - -But sometimes we’ll want to pre-install the content into some other -artifact: a container image, an installer iso, or an ostree tree, for -subsequent delivery to the user. That’s fine, we can drive whatever -tools we need to build these artifacts. The only principles we rely on -are the ones we’ve already mentioned: - -- We need to compose the images from static configuration/manifests: no - manual invocation of builds with magic command line parameters, the - tools should be able to find all the configuration/input they need - from SCM etc. -- Record compose IDs of everything we consume as input, and generate a - build ID from the output; -- Each compose produces just one artifact (or as few as possible); no - monolithic composes that try to build massive amounts of the distro - in a single step. We end up with the same traceability because the - composes can all use the same exact inputs guaranteeing the artifacts - are the same except for their distribution packaging. - -**5. What happens when a compose fails?** - -One problem with a monolithic compose is that a single problem can fail -the entire compose. - -How do we prevent a similar situation where a broken compose of a Base -Runtime module causes compose failures for all other modules depending -on it? - -Having a granular compose process actually improves this situation -significantly. We simply need to remember the most recent *successful* -compose for any given module, and use that compose for any subsequent -layered composes. - -.. figure:: constructing-image5.png - :alt: constructing-image5.png - - -So composing modules here from left to right still works fine; the -failed compose #3 of our base runtime module does not prevent a -subsequent compose #4 of the Apache module from succeeding; that compose -simply picks up the most recent successful compose of the base runtime. -And when a new compose of the base runtime does eventually succeed, then -it becomes available to future composes of the Apache module on top. - -**6. Chaining recomposes** - -So far we have talked about “composing” a module in terms of re-running -the creation of that module from its component parts (the packages it -contains, and the metadata belonging to the module) on demand. When a -module compose is requested, the build system must have enough -information to complete that compose on its own, but the build system -does not necessarily know itself *when* to compose a module. - -But of course, we can automate that part too. Just as a CI package -workflow might want to rebuild and retest an rpm when a dist-git event -occurs, we want to be able to recompose both modules and artifacts -whenever their content changes. - -The details of each individual task here are not important; indeed, it -is important to include many different types of tasks in this automation. -Recomposing a module may be triggered by an rpm rebuild; that recompose -may trigger other module recomposes, and additionally may trigger other -image rebuilds. We may add CI to the mix so that testing tasks are also -triggered when appropriate. - -What *is* important is that the chaining of tasks is automatic. When -something changes in the content tree, the build system **must** know -(or be able to tell us) what else needs to be rebuilt or recomposed as a -consequence; it **must** have the information needed to perform those -rebuilds without additional manual configuration; and it **should** have -the ability to trigger those rebuilds automatically subject to rel-eng -policy. - -Which leads us to: - -**7. The Release Concept** - -A CI / chain-build toolchain is fundamentally asynchronous. Content is -rebuilt, recomposed or re-tested when its dependencies change. This is -deliberate: a modular release is, by definition, attempting to break up -the monolithic, synchronous Compose. - -But ultimately, Fedora still wants to be able to produce a planned -release spanning many modules, and users still want to have controlled, -predictable releases, rather than having modules dribble into a release -in drips. We still have major release events such as Fedora major/minor -updates. - -So decomposing the release into loosely-coupled compose steps is all -well enough, but we need more than just chain rebuilds to bring it all -back together. We need to know what modules need to come together on -what date to make a given release. - -So we introduce the formal concept of a **release** as a concept that -defines requirements across multiple modules. The release would define, -for example: - -- Which modules are included in the release (and, if those modules have - multiple versions available, which versions come together for this - release); -- The date scheduled for the release; -- Minimum SLA and lifecycle expectations for the release, so that we - can verify during planning that all the required modules and all - their dependencies have the required support - -This information cannot be inferred from the individual modules -comprising a release. The reason is simple: it’s a primary assumption in -modularity that any one module may be included in multiple products -and/or multiple releases. - -Furthermore, there is a chicken-and-egg situation involving build-system -automation of modules. If module definitions live in dist-git, then how -does a build system know which dist-git repositories to look up to know -what builds need to be performed automatically? The release is a natural -place to define that. - -Note: Technically, a release could easily be encoded as yet another -dist-git module itself, one with many dependencies and no additional -content of its own. It could be similar to an rpm meta-package in this -regard. But it is still a distinct concept. diff --git a/source/architecture/constructing/constructing-image1.png b/source/architecture/constructing/constructing-image1.png deleted file mode 100644 index 29a3c69..0000000 Binary files a/source/architecture/constructing/constructing-image1.png and /dev/null differ diff --git a/source/architecture/constructing/constructing-image2.png b/source/architecture/constructing/constructing-image2.png deleted file mode 100644 index d58fb02..0000000 Binary files a/source/architecture/constructing/constructing-image2.png and /dev/null differ diff --git a/source/architecture/constructing/constructing-image3.png b/source/architecture/constructing/constructing-image3.png deleted file mode 100644 index 0f50346..0000000 Binary files a/source/architecture/constructing/constructing-image3.png and /dev/null differ diff --git a/source/architecture/constructing/constructing-image4.png b/source/architecture/constructing/constructing-image4.png deleted file mode 100644 index 4f6d44d..0000000 Binary files a/source/architecture/constructing/constructing-image4.png and /dev/null differ diff --git a/source/architecture/constructing/constructing-image5.png b/source/architecture/constructing/constructing-image5.png deleted file mode 100644 index e574799..0000000 Binary files a/source/architecture/constructing/constructing-image5.png and /dev/null differ diff --git a/source/architecture/constructing/constructing-image6.png b/source/architecture/constructing/constructing-image6.png deleted file mode 100644 index 7ca2e7c..0000000 Binary files a/source/architecture/constructing/constructing-image6.png and /dev/null differ diff --git a/source/architecture/constructing/constructing-image7.png b/source/architecture/constructing/constructing-image7.png deleted file mode 100644 index eabb626..0000000 Binary files a/source/architecture/constructing/constructing-image7.png and /dev/null differ diff --git a/source/architecture/versioning.rst b/source/architecture/versioning.rst deleted file mode 100644 index c7c5788..0000000 --- a/source/architecture/versioning.rst +++ /dev/null @@ -1,65 +0,0 @@ -Module versioning and branching -========= - -The primary core principle of Modularity is that our content should be -released not as large, monolithic distribution releases, but in units of -smaller modules designed to be assembled in different combinations. - -Closely related is a second core principle: we should be able to release -new major versions of these modules on their own schedules to serve -a component's requirements; they should not all be tied to the master cadence -of today’s major release cycles. - -So, at a deep level, Modularity requires us to branch and version -modules independently of each other. - -Yet we need to control this complexity: the engineering involved has to -be sustainable, and the combinations we offer to the user need to be -manageable. This is especially true as different tools represent -branches in different ways (eg. bugzilla represents branches as the -“version” field for a product, and also has version-specific flags; -koji has koji tags which act as branches, etc.) - -This leads to some complex constraints, which we will explore in this -document. - -Ultimately, we can identify many distinct variants on branching: -sometimes different parts of the release pipeline end up with multiple -different views of the underlying branches. Just consider Fedora fXX -branches: these either look like a continuous update stream to the end -user, or *some* of the branches end up having distinct lifespans. - -So while a single consistent end-to-end branching model for any module -is possible in simple cases, it is unlikely to satisfy all the product -demands for complex release structures. This suggests an approach to -branching involving: - -- A centralised representation of the current branching at any point in - time; -- Retain separate branch definitions in our various tools such as - bugzilla, koji, bodhi, etc, as we have today; -- A flexible scripting approach to automating creation of new branches - and key transitions on existing branches (dev to beta to release - etc), to keep branches on the different tools synchronised. - -This splits *mechanism* — the central branching repository and the branch -definitions in the different tools — from *policy* — the specific branches -and transitions actioned by the scripts. - -It is important to remember that the mechanism needs to be flexible -enough to represent any potential desired branching structure; but this -does not mean that all modules have to have complex branching we can -still (and we should) adopt branching policy that is as simple as -possible for any given module. - - -.. rubric:: TOC - -.. toctree:: - :maxdepth: 1 - - versioning/terminology - versioning/properties - versioning/coherency - versioning/managing - versioning/constraints diff --git a/source/architecture/versioning/coherency.rst b/source/architecture/versioning/coherency.rst deleted file mode 100644 index 9ccd4c0..0000000 --- a/source/architecture/versioning/coherency.rst +++ /dev/null @@ -1,71 +0,0 @@ -Coherency of branching -====================== - -There are many advantages to be had in a single, coherent view of the -branching structure for a given module. - -Maintainers and users alike have to deal with branches in multiple -places: - -- We plan, develop and build content in bugzilla, dist-git - and koji, then release it through bodhi. All of these - tools share a common understanding of the various product branches - (fXX etc.) and work flows naturally between them. -- Users consume products knowing which branch they are on; they - have the option to choose between branches (F25 after F26 has - been released), and they report bugs and - review CVEs corresponding to those branches - -It also seems highly desirable to automate creation of branches, -especially when we consider a future with many more modules than we have -today, each with their own branches. - -But the idea of a clean, consistent view of branching that is unified -end-to-end falls down in several places. Some of the difficulties -include: - -**Utility side-branches:** Some of our tools have slight variants on the -main branch naming to support specific workflow requirements. - -For example, in CI we can have staging branches alongside the production -release branches, and in koji we have scratch builds; these follow the -main branching but are intended for developer use cases, rather -than automatically being candidates for release. - -We have candidate tags, beta tags and release tags in koji, indicating -packages on various different stages of the lifecycle from development -to release. Beta branches in general represent a special case here. - -**Multiple views of branching:** There are several places where two -different parts of the release pipeline can treat branching differently -from each other. Two important examples here include **minor version -branching** and **per-edition views** of a component: - -**Minor-version branching**: TBD - -**Per-edition views** of branching: TBD - -**Branch fluidity:** TBD - -**Naming policy:** TBD - -**Consistency of release:** Finally, we need to consider the granularity -of branches. The purpose of modularity is to allow us to release modules -independently from a single master release cadence. But do we really -want all modules to be released without any synchronisation or common -branching at all? - -History suggests we do not. - -In the future we likely have many completely-decoupled modules for -additional content outside the base Fedora runtime platform. But we may -still eventually decide that we want to have synchronised releases of -new content across different modules. - -So while modules can have independent branches, we still need the -ability to drive a common branching structure across a set of modules -when that is needed for product release requirements. First-class -support for such a **consolidated release** is absolutely necessary; to -devolve the distribution into an unmanaged, completely-uncoordinated set -of independent modules is likely unsustainable for both engineers, maintainers, and -users alike. diff --git a/source/architecture/versioning/constraints.rst b/source/architecture/versioning/constraints.rst deleted file mode 100644 index 6d0045a..0000000 --- a/source/architecture/versioning/constraints.rst +++ /dev/null @@ -1,59 +0,0 @@ -Constraints on branching -======================== - -Everything so far seems to say “branching is hard, let’s not assume what it -looks like but just store a flexible representation that we can adapt as -we need.” - -That’s true to some extent… but there *are* concerns we can anticipate -that we need to handle in our branching structure. Having covered the -fundamental principle that branching policy needs to remain flexible, -let’s look at some of the issues we need to handle as we define that -policy. - -Splitting a binary package build over multiple modules ------------------------------------------------------- - -This is something that is surprisingly common. - -Examples might be when we want to include a library to support our own -package/application, but do not want to give it full support for end-users; we -might include the library itself, but not include the components -that provide the ability to develop against it (the include files, static libraries etc. that -typically land in a -devel binary rpm). - -Can we do this naturally in a modular build chain? Clearly it breaks any -assumption that a module can be both compiled and composed in complete -independence from any other: if a package build ends up in multiple -modules, then the compile phase of building those modules is now linked. -We need to determine how important it is to support this. - -But it is still quite possible to achieve, if the modules which are to -share binaries have matching branches. In that case, module composes can -always agree on which koji branches [tags] to consume packages -from. So this may be fairly easy for modules which are part of a single -consolidated release, as defined above; it would be fair to restrict -this possibility to that case. - -Building a module in multiple build roots ------------------------------------------ - -Does a single module source branch result in a single composed binary -branch? Or do we build that same source multiple times against different -base distribution build roots? - -Clearly, branching becomes enormously more complicated if we need to -support builds for multiple different build roots in a single branch. -The idea of a single coherent branching structure from git to release is -broken if we have multiple output branches from a single input branch. - -But the entire point of ABI forwards compatibility is to avoid the need -to do this: to run a module on a set of major runtimes, it should, in -theory, be necessary simply to build it on the oldest runtime in that -set. A module built on F25 should run on F26 or f27, as long as it -is using only dependencies with long-term stability guarantees. - -So before working through the complexities of commit-once, -compile-multiple-times, it will be important to determine to what extent -we can simply depend on ABI compatibility to ensure a module works -against multiple runtimes. diff --git a/source/architecture/versioning/managing.rst b/source/architecture/versioning/managing.rst deleted file mode 100644 index b9f2e3a..0000000 --- a/source/architecture/versioning/managing.rst +++ /dev/null @@ -1,133 +0,0 @@ -Managing this branching complexity -================================== - -Given that the exact branching policy for a module is currently -inconsistent, needs to remain flexible, and changes -over time, how do we manage this? The question is especially significant -given that we are looking at significant changes to the way we divide -and release the distribution in the future; our future branching model -is currently completely unknown. - -This suggests that we should not try to formalise a branching and naming -policy at all. But we **must** eventually have automation for the -creation of branches and for branch transitions, especially given that -release consistency may require us to coordinate new branches across -many modules simultaneously. And our tools still need consistent views -across this complex branching structure. - -Separation of policy from representation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This suggests that we need: - -- A canonical definition of our modules and their branches at any point - in time, including the way those branch names are represented in - different tools: -- Consistent use of that canonical branching structure within our - tools, but with -- Flexible, scripted events to drive changes in the branching. - -We can do this by separating the central representation of branching -(eg. in PDC) from the mechanism used to define and update that -branching. - -Changes in branching also need to be orchestrated: we should not define -a new branch and allow a developer to start building on that branch, -before the branch has been created in bugzilla, dist-git, koji etc. -There are many tools that could be used here: ansible is just one such -tool. The point here is to identify that as a separate concern. -Automation here is important if we want to be able to support -coordinated release branching across a set of modules. - -For now, we are dealing with a relatively simple branching structure, -building simple modules out of the latest Fedora. We don’t *need* -complex branching policy right now. But separating out representation -from policy allows us to start with a simple branching structure -initially, and still lets us define, and script, more complex, -product-specific branching requirements later, while having those -consistently represented in a central database that our tools can refer -to and agree on. - -Forking a new branch -~~~~~~~~~~~~~~~~~~~~ - -We have mentioned that creating a new branch for a module involves -branching multiple different tools: we need a branch for the module in -dist-git, new branches for its components, and corresponding branches in -bugzilla; we may need new tags in koji. - -This implies that the branching for a module is (usually) the same thing -as the branching for all the component packages of that module. - -But sometimes we will not want to branch *all* packages; we may want a -variant branch of a module which overrides just some of the packages, -and which otherwise inherits the content (''including new content) -''from its base branch. - -There are many examples which would suit such a **inheriting branch.** -The f-stream model which allows early access to new features prior -to an update, is an example. Another might be the specialized version of -the virtualisation stack, which contains a version of kvm-qemu with -newer features but which otherwise follows Fedora. The model also -works for scratch or staging branches, where we can build and test -updates to an existing branch as needed to suit internal developer -needs. - -This suggests that we want to include tooling support for inheriting such a - branch. Technically, this might involve creating new branches -for only a subset of the packages of a module; and recording the base -module from which we pull other packages during a module compose. - -Converging branches -~~~~~~~~~~~~~~~~~~~ - -Just as important as forking a new branch is converging existing -branches. In an f-stream model, a new feature scheduled for the next release -is made available in a prior release. The f-stream is the early-access branch; -the intent is that when the next release occurs, it introduces that -feature into the mainline stream, and the f-stream is no longer -needed: any component depending on that feature moves back off the -f-stream branch and onto the mainline. - -Extending this to a modular build, we can imagine a component -needing a new feature within any module in our stack. If that module -does not plan the feature to be released in time, we can fork a specific -version of the module to serve the needs of the one component needing the -new feature; but if and when that feature is released in some mainline -version of the module, we want the ability to move the component -off the forked feature branch and back onto mainline. - -There are likely to be many complexities here; the important point is to -imagine up-front that forking a new branch is only half the picture, it -will be useful to have tooling support for converging branches again -afterwards too. - -Managing unsynchronised stacks of branches -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Modules can depend in turn on other modules. We have defined a “stack” -as the entire tree of modules needed to satisfy dependencies for one -top-level module or application. But as we combine modules in this way, -not all those modules will have the same branches or lifecycle. - -So when we have multiple, different, unsynchronised branching models for -different modules within a stack, how do we know exactly which branches -of which modules we need to combine together? We can agree that we need -to constrain this complexity, and define specific subsets of modules -which we will test and support together. The issue is where, and how, to -define this. - -This is an issue we still need to solve. There two obvious places to -hold this structure: in the **release** that defines multiple modules -and their combined release schedules; or by defining specific branch -dependencies in each module’s own module metadata. - -Both have pros and cons. Defining specific branch dependencies in a -module’s metadata helps by keeping more of the module’s defining -structure in one place. However, the downside is that it becomes -impossible to use that same metadata in multiple places without changing -it: eg. building a single module from the same module source on multiple -buildroots is impossible if the module source itself defines its -buildroot dependency. - -So this is a topic for future consideration. diff --git a/source/architecture/versioning/properties.rst b/source/architecture/versioning/properties.rst deleted file mode 100644 index 1257658..0000000 --- a/source/architecture/versioning/properties.rst +++ /dev/null @@ -1,74 +0,0 @@ -Properties of branches and update streams -========================================= - -**ABI Compatibility:** Updates within a single update stream are likely -to maintain ABI backwards compatibility in most cases. Users should be -able to consume updates from an update stream without being concerned -about breaking applications that depend on that module. A major change -introducing an incompatible ABI would normally be expected to require a -new version branch. - -But this is ultimately a policy decision: there is nothing technical to -stop ABI breakage within a version stream. For example, the container-runtime -is rapidly changing so it has broken compatibility across versions. Our tools -should be able to detect incompatible ABI changes as far as possible, -but should not prevent them if we have an exceptional case where such a -change is desired. - -(ABI compatibility here includes anything that may have a compatibility -on user or application compatibility, including for example semantics of -configuration files, library ABIs, command line option handling and -error codes, and so on.) - -**Constraining the scope of ABI dependency:** As preserving ABI on -updates is a burden which imposes constraints on our maintenance of a -module within a single version branch, we would like the ability to -limit the parts of a module to which ABI stability applies. We currently -define which packages within a module form the **external ABI** of the -module: this is defined by the maintainer of a given module’s metadata. -Conversely, packages not declared as external are implicit internal -implementation details of the module. - -Defining the external ABI as a set of packages will allow us to: - -- Rebase internal packages without constraint from ABI guarantees, - removing overhead from the module maintenance burden over time; -- Verify that layered modules or applications depend only on packages - defined as external ABI, by checking rpm dependency chains - -Over time, we may find other ways to describe the external ABI of a module -but, at present, RPMs are the best stand in. - -**Lifecycle:** Given that we define no formal policy on ABI -lifecycle—rather leaving this up to policy—it follows that there is no -strong requirement that version numbers of packages within a single -update stream have to follow any particular pattern. We can easily -rebase a package within an update stream, even adding new features, as -long as any claimed backwards compatibility is preserved. - -We do need to be concerned about whether 3rd-party application -"certification" is expected to be preserved when such an application -depends on a module’s version branch containing rebased packages. This -is an important question, and we need to add tooling and policy around -it; but for now this is primarily a policy question, and beyond the -scope of this document. Different modules may have different appetite -for risk and rebases, and hence have different policy around -"certification." - -**Parallel Availability:** The update streams for different module -version branches must be able to coexist in our pipeline and released -content, without interfering with each other. If a given base system -install has both httpd-2.2 and httpd-2.4 available in different version -branches, then it is important that these remain independent. - -The update streams must not interfere with each other. If httpd-2.2 is -installed, then updating it via yum or dnf should update it to the most -recent version in the httpd-2.2 update stream, and must not -automatically update it to 2.4. Any dependencies brought in by either -must also prevent such interference. - -And yet if a certain package *does* support parallel installation of -different version branches at the same time -(eg. software collection-based packages), then the -separate installed versions at any time must each be updatable by their -own specific update stream. diff --git a/source/architecture/versioning/terminology.rst b/source/architecture/versioning/terminology.rst deleted file mode 100644 index 56b6734..0000000 --- a/source/architecture/versioning/terminology.rst +++ /dev/null @@ -1,40 +0,0 @@ -Basic branching terminology -=========================== - -First, though, note that the very word “version” is overloaded here. It -can refer either to a completely separate branch of some module, or to a -single specific instance or compose of a module. To keep terminology -straight, we need to distinguish between: - -- A **version branch**: or more simply just a **branch**: a completely - new, carefully planned version stream for a module. This might - correspond to a major Fedora release, or a new Fedora SCL release. - Creating a new branch should come with a new **release** target for - the branch (or branches if we’re releasing multiple modules - simultaneously.) - - A version branch **may** correspond to a new **major version** of - the module, but there may be exceptions: for example, when we add - specific features as a side-branch of an existing major version. The - new branch may differ by SLA (eg. a faster moving Python) or by - feature (eg. an “f-stream” branch giving early access to a - specific new feature planned to be released in a subsequent update). - - Users must opt into a new version branch. By definition here, - creating a new version branch **must** have no impact on users who - have already enabled an existing branch of a module. If a user - provisions a new environment and asks for the most recent version - branch, then they may get the new version branch automatically; but - no existing environments will transparently receive content for the - new version branch. - -- A **point-in-time version** or **instance** representing a single - compose of a module on a single version branch, built by and - identifiable by the **compose ID** of the task used to compose the - module within the build system. - - Such a point-in-time update may be just a scratch build, or may be - internal-only and not released to the user. But once it is released, - it forms a new **update** for that version branch of the module. - Multiple released point-in-time updates therefore form an **update - stream** over time for that branch. diff --git a/source/design/building-naming.rst b/source/design/building-naming.rst new file mode 100644 index 0000000..a3fe526 --- /dev/null +++ b/source/design/building-naming.rst @@ -0,0 +1,49 @@ +Building and naming modular things +========= + +We have some basic terminology confusion around modules. Is a container +image the same thing as a module? Is a software collection a single +module, or a group of modules? We can often get away with being vague, +but for technical planning we need to be able to distinguish between all +these concepts. + +We use these terms: + +- **Package**. Essentially, the same thing as an rpm. In the future it + might be non-rpm content but should fit the same role. +- **Module.** A set of packages tested and released together as a + distinct unit, complete with the metadata needed to manage it as a + unit. May depend on other modules. +- **Stack.** A complete tree of modules. A stack can be thought of as + a top-level module, with the understanding that we’re implicitly + including all of that module’s dependencies in the stack. HOwever, when + the stack is delivered to the end-user it appears as "just another module." +- **Artifact** or **image**. An actual set of bits built out of + modules, in a format intended to be distributed or deployed in some + way. + +Generally, these serve distinct purposes. A module is a building block; +a stack contains all the software for a complete solution; an artifact +is a concrete object containing a stack (or stacks) for distribution to +users. + +We will also distinguish between: + +- A **Build** of a package: a process which involves compiling source + code and creating a packaged output +- A **Compose** of a module: a process which assembles pre-compiled + packages into an organised module, but which includes no compilation + step itself. Sometimes we also use the term **Coallesce** for this scenario + to distinguish between the existing, Fedora term "Compose" which is very + similar but not quite the same. + +.. rubric:: TOC + +.. toctree:: + :maxdepth: 1 + + building-naming/package + building-naming/module + building-naming/stack + building-naming/artifact + building-naming/fit-together diff --git a/source/design/building-naming/artifact.rst b/source/design/building-naming/artifact.rst new file mode 100644 index 0000000..2deaed4 --- /dev/null +++ b/source/design/building-naming/artifact.rst @@ -0,0 +1,19 @@ +The Image or Artifact +===================== + +We build stacks into *images* or other *artifacts* such as ostree trees. +These are the formats in which we release a stack. + +For example, we could take a single stack such as a LAMP stack, and +build and release it both as SCL rpms in a yum repository, and as +container images: indeed, we do so today. It’s the same stack in each +artifact. We could add more such formats: virtual appliances, vagrant +boxes, anaconda isos or ostree trees, and it would still be the same +stack in each; only the artifact / format / image has changed. + +.. figure:: building-image5.png + :alt: building-image5.png + +The format of the content we deliver may be different in each case, but +it’s the same binaries from the same stack inside, and the same stack +metadata describing that content. diff --git a/source/design/building-naming/building-image1.png b/source/design/building-naming/building-image1.png new file mode 100644 index 0000000..6b955f6 Binary files /dev/null and b/source/design/building-naming/building-image1.png differ diff --git a/source/design/building-naming/building-image2.png b/source/design/building-naming/building-image2.png new file mode 100644 index 0000000..a17a8f1 Binary files /dev/null and b/source/design/building-naming/building-image2.png differ diff --git a/source/design/building-naming/building-image3.png b/source/design/building-naming/building-image3.png new file mode 100644 index 0000000..59c2ccb Binary files /dev/null and b/source/design/building-naming/building-image3.png differ diff --git a/source/design/building-naming/building-image4.png b/source/design/building-naming/building-image4.png new file mode 100644 index 0000000..d180292 Binary files /dev/null and b/source/design/building-naming/building-image4.png differ diff --git a/source/design/building-naming/building-image5.png b/source/design/building-naming/building-image5.png new file mode 100644 index 0000000..91c53fd Binary files /dev/null and b/source/design/building-naming/building-image5.png differ diff --git a/source/design/building-naming/building-image6.png b/source/design/building-naming/building-image6.png new file mode 100644 index 0000000..8456279 Binary files /dev/null and b/source/design/building-naming/building-image6.png differ diff --git a/source/design/building-naming/fit-together.rst b/source/design/building-naming/fit-together.rst new file mode 100644 index 0000000..9cc0d73 --- /dev/null +++ b/source/design/building-naming/fit-together.rst @@ -0,0 +1,30 @@ +How do these parts fit together? +================================ + +Let’s look at a couple of examples to make it (hopefully!) clear what +we’re talking about. + +Example 1: Our Atomic Host stack +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We already release content that is organised in much this way, just not +with any formal definition of modules or stacks. + +Atomic, for example, is composed from multiple sources. Some content +comes from traditional Fedora with its usual 6-month lifecycle; and some +comes from other repos with much shorter lifecycle and rapid rebasing to +pick up new features. + +And even within Fedora we have different levels of lifecycle. An obvious +example is Gnome; the Anaconda installer stack that we use for Atomic +installer ISO images depends on this faster-moving content, even when +the installed atomic host does not include it. + +But even though these distinct sets of components all have differing +lifecycles, they still belong within Fedora and we still produce +coherent output images or artifacts from them today, with planned +releases that require alignment between all the relevant parts. In the +language of this document, it might look like this: + +.. figure:: building-image6.png + :alt: building-image6.png diff --git a/source/design/building-naming/module.rst b/source/design/building-naming/module.rst new file mode 100644 index 0000000..753b052 --- /dev/null +++ b/source/design/building-naming/module.rst @@ -0,0 +1,40 @@ +The Module +========== + +We group these binary packages into *modules*. Note, we can pick just a +subset of the subpackages from a particular build into a given module, +eg. a hypothetical example base runtime module: + +.. figure:: building-image2.png + :alt: building-image2.png + + +These are internal build groupings: basically *repositories*, but with +additional metadata and semantics that allow them to play nicely with +other modules and module tooling. + +We attach metadata to these modules, on top of the normal metadata in a +repository. Metadata may include: support metadata (SLAs or EOL dates for the +module); metadata on how to use the module (eg. its dependencies, +default installed packages); identity (builder, vendor, version etc), or +many other things that we’re still just starting to imagine. + +We should be able to update the module as a unit: updating and testing +its component parts as necessary, but also testing the module as a whole +before releasing an update. + +We refer to the assembly of a module by a build tool as a *compose*. +This step includes no compilation; it is merely the creation of a +bundled set of packages and metadata comprising a single module. The +closest analogue in today’s build system is the “puddle build”, where a +custom repository is created for a particular purpose. + +When we talk about a **module compose**, we are *not* talking about “the +Compose” of a full distribution release. The two are similar, in that +they both assemble existing compiled packages into bundled output; but +the module compose is just one single targeted assembly, whereas the +full distribution Compose typically creates multiple repositories and +images as output in a single large job. Also note that we can, in theory, +perform the build (compilation) of the packages within a module, and the compose of +the module itself, as a single step; we might term this a **module +build.** diff --git a/source/design/building-naming/package.rst b/source/design/building-naming/package.rst new file mode 100644 index 0000000..b66b446 --- /dev/null +++ b/source/design/building-naming/package.rst @@ -0,0 +1,17 @@ +The Package +=========== + +We can start with the familiar ''package ''and *subpackage*, *built* as +usual from a *component* (or *source package*): + +.. figure:: building-image1.png + :alt: building-image1.png + +We’re not changing anything at this level (yet… non-rpm packages are a +possible topic for the future!) + +It is important to distinguish the build of a package from the compose +of a module; the build here is the step which includes compilation and +creation of a reusable bundle of the compiled output. We might be +building specifically to compile for a module’s contents; but the build +step is still a distinct step. diff --git a/source/design/building-naming/stack.rst b/source/design/building-naming/stack.rst new file mode 100644 index 0000000..adec375 --- /dev/null +++ b/source/design/building-naming/stack.rst @@ -0,0 +1,34 @@ +The Stack +========= + +We can then combine modules into *stacks*. + +A stack should represent something distinct that the user wants. It may +be a traditional developer stack (LAMP, ruby-on-rails, etc.); or it may +be an application (but extended to include all the dependencies that +the application needs to run); or it could be the set of modules needed +to deliver something like Atomic Host or Cockpit. + +.. figure:: building-image3.png + :alt: building-image3.png + +The stack represents this full set of software. It doesn’t presume how +we distribute it, we’re still just talking about the set of modules +making up the stack. + +**A stack is still just a module here.** It’s just a way of referring to +the module plus all its implied dependencies as a single unit, to +distinguish that from the individual modules within the stack; the stack +content and metadata may have exactly the same format as module metadata +(the metadata is the same colour here for a reason!) But it’s still +important (at least internally, probably not for end users) to make the +distinction between a single module, and a module plus all the external +dependencies it relies on. + +Importantly, we can take two modules with different lifecycles and +combine them in a single stack. The definition of the stack gives us the +way to plan and track the relationship or dependency between the +modules. + +.. figure:: building-image4.png + :alt: building-image4.png diff --git a/source/design/constructing.rst b/source/design/constructing.rst new file mode 100644 index 0000000..41e2593 --- /dev/null +++ b/source/design/constructing.rst @@ -0,0 +1,49 @@ +Constructing a modular distribution +========= + +The fundamental objective of the +Modularity effort is to +break up the monolithic concept of a “distribution release” or +traditional Compose into something more fine-grained. We should be able +to release applications or stacks such as LAMP or ruby-on-rails on a +lifecycle that suits the application, rather than being dictated by the +distribution release schedule; and we should be able to compose releases +more flexibly from the various components available. + +But modularity is unlikely to succeed if it requires both users and +maintainers to immediately and completely discard the existing concept +of Releases and Editions of Fedora. So in this document we look at how +to break the traditional distribution down into a modular construction, +while still preserving the ability to build the traditional releases. + +This document does not try to explore how we should organise a +distribution release in detail. Exact lifecycles for kernel vs. +applications, for containers vs. baremetal etc. are beyond the scope +here; indeed, our objective is explicitly ''not'' to assume what we need +in a release, but rather to add flexibility so we can change release +objectives later on. If we want Fedora Server to have a longer lifecycle +than Workstation; or for Atomic to rebase docker more rapidly than +Server; or for a new python to be released outside the normal +distribution cycle; all these things may be possible if we have a more +flexible underlying release structure. The flexibility is the subject +here; planning the actual releases is a different topic. + +But, ultimately we still need the well-defined concept of a **release**, +so that all the modules we are maintaining can still come together into +a well-tested, planned release on a known schedule. We also define and +justify a **static manifest** to assign packages to modules. + +Once we have parts of the distribution on different release cycles, we +also have the issue of how to maintain different versions branching on +different criteria and different schedules. Branching and versioning is +relevant here, but is a complex topic in its own right and is covered under +:doc:`/architecture/versioning`. + +.. rubric:: TOC + +.. toctree:: + :maxdepth: 1 + + constructing/breaking-down + constructing/compose-distribution + constructing/back-together diff --git a/source/design/constructing/back-together.rst b/source/design/constructing/back-together.rst new file mode 100644 index 0000000..31f48d7 --- /dev/null +++ b/source/design/constructing/back-together.rst @@ -0,0 +1,70 @@ +Putting it all back together +============================ + +We need the flexibility of a smaller, more modular distribution that can +be reassembled in various different fashions. But we also need to be +able to reproduce our current product lineup, or something very similar, +in order not to disrupt our existing install base: + +Example: Reproducing the traditional distribution versions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Having categorised the Fedora rpms into groups and composed those +package groups into distinct modules, creating what we have +traditionally called a distribution variant or edition can now be done simply by +creating a flat yum repository containing all the desired modules for +that offering. In the terminology we are using here, the output +repository is simply another output artifact. + +.. figure:: constructing-image6.png + :alt: constructing-image6.png + + +And of course we can repeat this for different Fedora editions. Having +separate GTK and Gnome modules would allow us to include those modules +when composing the Fedora Workstation repository, but omit those from +the Cloud Edition if we chose. It would also give us a natural way +to reflect that Gnome may rebase at a different cadence from the Base +Runtime. + +So today’s editions can simply become different ways of selecting from +the available modules. + +The installer iso can also be composed from the various modules here, as +a separate output artifact. The main difference from the traditional +monolithic Compose is that the repositories, images etc. that are +composed today are still composed, but each is created as a logically +separate build. + +Add-ons are not drawn above. An add-on is almost completely synonymous +with a module here, and the modules composed as above can naturally +reflect the composition of add-ons. + +Example: Creating atomic host/container images +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A key point of a broken-down, asynchronous build toolchain is that we +can create composes from different modules with different SLAs or +lifecycles, and can combine both modules and existing artifacts into yet +further artifacts. + +For example, we can define a tree of required artifact builds for Atomic +Host and our Docker container images (each purple box here represents a +distinct artifact created by a distinct compose task): + +.. figure:: constructing-image7.png + :alt: constructing-image7.png + + +The Fedora Server and COPR repos today already have distinct +SLAs and lifecycles. We could build atomic host trees out of both of +those sources, and this can be done easily as a compose step based on +the Base Runtime and Container Runtime modules in a modular build of +the same components. + +But we also need to support layered builds, such as additional container +images layered on top of the base image, or Anaconda installs of an +existing ostree tree. As long as we can track which compose steps are +used as input to each compose task, we can track the relationship +between these output artifacts even when we have decomposed the +monolithic Compose into many parts. diff --git a/source/design/constructing/breaking-down.rst b/source/design/constructing/breaking-down.rst new file mode 100644 index 0000000..72c0d37 --- /dev/null +++ b/source/design/constructing/breaking-down.rst @@ -0,0 +1,68 @@ +Breaking down the monolithic release +==================================== + +We have many steps in the release pipeline today: starting with bugzilla +and dist-git; building and tagging in brew; the errata tool and mirrors for +pushing the bits out. + +But currently it all comes together in The Compose. + +Remember, I’m not talking about branching and versioning in this +particular document. So let’s start with the assumption that we’ve +already built our individual packages, and that (for now) we have a flat +namespace of binary rpms already built from all our components. That +namespace could be living in a yum repository like the rawhide repo, or +a koji tag such as fedora-24.0-candidate. + +.. figure:: constructing-image1.png + :alt: constructing-image1.png + + +We build all of the images and repositories, for all of the +architectures and their addons, for all of the variants, all at once. +Fedora’s Editions have a similar structure. + +This served us well once upon a time. It enabled a single consistent +major or minor release, and it built everything necessary for that +release. + +But it now falls short on multiple fronts: + +- We have many more types of artifacts being built than are handled by + the compose. We have ostree trees and installer images; there are + cloud targets such as qcow images and AMIs; and docker base and + layered images. We have additional, often ad-hoc, + incompletely-automated builds to create this additional content. +- The modularity effort is explicitly trying to get away from the + concept of a single monolithic distribution, and to release modular + parts of the distribution on independent release cycles. +- Scaling: as we increase the number of modules, we do not want to + spend the effort of rebuilding the entire distribution when any small + part changes. For example, with more container images to build, we + should be trying to rebuild only those affected by any change. +- Self-service: to scale the modular decomposition of the distribution, + we will need the ability for individual engineers or groups who own a + module or image to build that themselves, not dependent on release + engineering. +- Continuous Integration. For automated testing, we want rebuilds to + happen automatically when a dependency changes, instead of having to + wait for a compose that happens on a predetermined schedule, or when + manually triggered by release engineering. + +So how can we address some of these concerns? We take the following +steps: + +- Break down the distribution compose into smaller parts: +- Ensure each part can be composed based on configuration and content + in SCM. **Everything must be recomposable based on static content**: + never, ever require manual configuration of a compose. That way + composes can be automated; +- Combine the smaller composes up in stages building towards a full + distribution; +- Record a compose ID for every stage of the compose, and record which + compose IDs are used as input to subsequent layered composes +- Record the most recent successful compose for each module, so that + failed composes (or composes that fail testing) do not impact layered + modules +- Optionally, we can automate the process of chain composes and add CI + to the mix to achieve a fully automatic build toolchain. diff --git a/source/design/constructing/compose-distribution.rst b/source/design/constructing/compose-distribution.rst new file mode 100644 index 0000000..a106478 --- /dev/null +++ b/source/design/constructing/compose-distribution.rst @@ -0,0 +1,262 @@ +Incremental compose of the distribution +======================================= + +**1. Break the distribution into distinct modules** + +Let’s take this in stages. We can start by simply assigning individual +packages to modules (at the sub-package level: we may have a need to +have a library in a base module but have its -devel subpackage in a +developer-only module, for example.) This ignores the branching of +packages and modules, but as stated before, that’s a separate topic. + +Note: `The module boundaries here are just examples.` They are +illustrations to show how we can break the monolithic distribution up. +They are *not* proposals that we should draw the lines between modules +in any particular places: that is an entirely separate discussion. + +We can start with some obvious “core” modules such as hardware +enablement, system runtime and core application runtime. + +The installer needs GTK and Anaconda; let’s make sure we have homes for +those. + +This leaves a lot of content currently not assigned to a module. That’s +fine, we can figure out how to handle those; for example, we could +choose to define a module for “uncategorised” for now. + +All we have done so far is to draw boundaries around groups of existing +packages: + +.. figure:: constructing-image2.png + :alt: constructing-image2.png + + +We can add Apache, php and mariadb, and we could have the components of +a LAMP stack, etc; there are plenty more groups we can imagine here. + +Note: This picture shows a static allocation of packages to modules. So, +modules have a **static manifest**. This is deliberate: + +If, over time, a package in a module picks up a new dependency, then it +should be a human decision to determine which module that new dependency +should live in (or even whether we want that dependency at all.) This +allows us to combat dependency sprawl over time, and also to record +*why* a particular package lives in a particular module, at the time we +add a dependency. + +To enable an engineer to manually allocate new dependencies to modules, +it it necessary that module manifests should not just be automatically +generated. We consider that manifest to be one of the defining +properties of the module. + +**2. Automate the composing of modules from packages** + +We now want to compose a module. Bear in mind, “composing” the module +here involves assembling pre-existing binaries: it’s more a “createrepo” +than “rpmbuild” (and is similar to a “puddle” build in current RCM +terms.) And the automation we’re talking about here is the execution of +the module compose, even if the compose is actually triggered manually. + +A module compose is now simply the creation of a yum repository of the +required packages or subpackages, with the addition of a small amount of +module metadata as described in :doc:`../building-naming`. + +.. figure:: constructing-image3.png + :alt: constructing-image3.png + + +We plan the module definition metadata living in dist-git, similarly to +how we currently maintain component spec files, sources definitions and +patches. Regardless of where it lives, **the module compose must be +well-defined based on static content.** The essence of an automated +build system is that when a compose is required, there should be no +compose-time configuration added; everything to compose the module is +stored in SCM and should be completely repeatable from there. + +This workflow mirrors existing practice for building binaries, where a +developer expects to be able to build and test an rpm locally for a +single architecture, before submitting to the build system to be built +on multiple architectures. In the same way, we would expect the +build-system module compose tool to pull together the necessary packages +and to create arch-specific modules for each required architecture. + +We mentioned before that we expect modules to have a static manifest. +During module compose, however, we would still expect to perform +*repoclosure* on the module: ie. to verify that the module satisfies the +dependency requirements of all packages it includes. Failure of +repoclosure should constitute a failed compose of the module. The +tooling may still attempt to solve missing dependencies automatically, +but should only report to the user which packages could be included to +satisfy those dependencies; it should not include those in the output +without confirmation from the module owner. + +To automate module composes it will be vital to know exactly which +version of which packages should be used to compose any given module +(eg. for koji builds, this would be selected by koji tag.) The important +point is that all of this should be configured in the module metadata so +that we can compose the entire module automatically. + +**3. Automate the composing of stacks from modules** + +Recall that a stack is nothing more than a module that depends on other +modules; the stack simply refers to that whole tree of dependencies. + +So composing a stack requires nothing more than composing a module that +has dependencies. We define that top-level module via a metadata file in +dist-git just as for any other module. The required dependent modules +are listed in the module’s defining metadata. This is important: just as +for a compose of an independent module, the entire module compose is +well-defined based on static content. + +The only difference in the compose of such a higher-level module is in +the checking of internal rpm dependencies: both the repoclosure test, +and any hinting done to suggest missing packages, should be performed on +*all* packages in the entire stack, including all of the packages from +other modules that this module depends on. + +Of course, we may be composing a module multiple times. And to manage +consistency between modules when things are changing, we really need to +be able to determine exactly which version of a module has been composed +from exactly which versions of its dependencies. So "each module +compose must have a unique compose identifier" of some form; and +during a layered module compose, we must record the compose IDs of all +the modules used as input for this compose. + +So to compose a PHP-on-Apache stack we might simply compose the base +runtime module, then a layered Apache module, and finally a php module +on top, in that order: + +.. figure:: constructing-image4.png + :alt: constructing-image4.png + + +**4. Automate the composing of images and other artifacts** + +So, now we’ve got enough modules to represent entire stacks of content +suitable for installation to solve some user problem. But still, these +are no more than repositories of available content; we still need to +actually install them to use them. + +Sometimes the user will install them themselves; the module client can +automate the complete installation of an entire stack of dependent +modules seamlessly. + +But sometimes we’ll want to pre-install the content into some other +artifact: a container image, an installer iso, or an ostree tree, for +subsequent delivery to the user. That’s fine, we can drive whatever +tools we need to build these artifacts. The only principles we rely on +are the ones we’ve already mentioned: + +- We need to compose the images from static configuration/manifests: no + manual invocation of builds with magic command line parameters, the + tools should be able to find all the configuration/input they need + from SCM etc. +- Record compose IDs of everything we consume as input, and generate a + build ID from the output; +- Each compose produces just one artifact (or as few as possible); no + monolithic composes that try to build massive amounts of the distro + in a single step. We end up with the same traceability because the + composes can all use the same exact inputs guaranteeing the artifacts + are the same except for their distribution packaging. + +**5. What happens when a compose fails?** + +One problem with a monolithic compose is that a single problem can fail +the entire compose. + +How do we prevent a similar situation where a broken compose of a Base +Runtime module causes compose failures for all other modules depending +on it? + +Having a granular compose process actually improves this situation +significantly. We simply need to remember the most recent *successful* +compose for any given module, and use that compose for any subsequent +layered composes. + +.. figure:: constructing-image5.png + :alt: constructing-image5.png + + +So composing modules here from left to right still works fine; the +failed compose #3 of our base runtime module does not prevent a +subsequent compose #4 of the Apache module from succeeding; that compose +simply picks up the most recent successful compose of the base runtime. +And when a new compose of the base runtime does eventually succeed, then +it becomes available to future composes of the Apache module on top. + +**6. Chaining recomposes** + +So far we have talked about “composing” a module in terms of re-running +the creation of that module from its component parts (the packages it +contains, and the metadata belonging to the module) on demand. When a +module compose is requested, the build system must have enough +information to complete that compose on its own, but the build system +does not necessarily know itself *when* to compose a module. + +But of course, we can automate that part too. Just as a CI package +workflow might want to rebuild and retest an rpm when a dist-git event +occurs, we want to be able to recompose both modules and artifacts +whenever their content changes. + +The details of each individual task here are not important; indeed, it +is important to include many different types of tasks in this automation. +Recomposing a module may be triggered by an rpm rebuild; that recompose +may trigger other module recomposes, and additionally may trigger other +image rebuilds. We may add CI to the mix so that testing tasks are also +triggered when appropriate. + +What *is* important is that the chaining of tasks is automatic. When +something changes in the content tree, the build system **must** know +(or be able to tell us) what else needs to be rebuilt or recomposed as a +consequence; it **must** have the information needed to perform those +rebuilds without additional manual configuration; and it **should** have +the ability to trigger those rebuilds automatically subject to rel-eng +policy. + +Which leads us to: + +**7. The Release Concept** + +A CI / chain-build toolchain is fundamentally asynchronous. Content is +rebuilt, recomposed or re-tested when its dependencies change. This is +deliberate: a modular release is, by definition, attempting to break up +the monolithic, synchronous Compose. + +But ultimately, Fedora still wants to be able to produce a planned +release spanning many modules, and users still want to have controlled, +predictable releases, rather than having modules dribble into a release +in drips. We still have major release events such as Fedora major/minor +updates. + +So decomposing the release into loosely-coupled compose steps is all +well enough, but we need more than just chain rebuilds to bring it all +back together. We need to know what modules need to come together on +what date to make a given release. + +So we introduce the formal concept of a **release** as a concept that +defines requirements across multiple modules. The release would define, +for example: + +- Which modules are included in the release (and, if those modules have + multiple versions available, which versions come together for this + release); +- The date scheduled for the release; +- Minimum SLA and lifecycle expectations for the release, so that we + can verify during planning that all the required modules and all + their dependencies have the required support + +This information cannot be inferred from the individual modules +comprising a release. The reason is simple: it’s a primary assumption in +modularity that any one module may be included in multiple products +and/or multiple releases. + +Furthermore, there is a chicken-and-egg situation involving build-system +automation of modules. If module definitions live in dist-git, then how +does a build system know which dist-git repositories to look up to know +what builds need to be performed automatically? The release is a natural +place to define that. + +Note: Technically, a release could easily be encoded as yet another +dist-git module itself, one with many dependencies and no additional +content of its own. It could be similar to an rpm meta-package in this +regard. But it is still a distinct concept. diff --git a/source/design/constructing/constructing-image1.png b/source/design/constructing/constructing-image1.png new file mode 100644 index 0000000..29a3c69 Binary files /dev/null and b/source/design/constructing/constructing-image1.png differ diff --git a/source/design/constructing/constructing-image2.png b/source/design/constructing/constructing-image2.png new file mode 100644 index 0000000..d58fb02 Binary files /dev/null and b/source/design/constructing/constructing-image2.png differ diff --git a/source/design/constructing/constructing-image3.png b/source/design/constructing/constructing-image3.png new file mode 100644 index 0000000..0f50346 Binary files /dev/null and b/source/design/constructing/constructing-image3.png differ diff --git a/source/design/constructing/constructing-image4.png b/source/design/constructing/constructing-image4.png new file mode 100644 index 0000000..4f6d44d Binary files /dev/null and b/source/design/constructing/constructing-image4.png differ diff --git a/source/design/constructing/constructing-image5.png b/source/design/constructing/constructing-image5.png new file mode 100644 index 0000000..e574799 Binary files /dev/null and b/source/design/constructing/constructing-image5.png differ diff --git a/source/design/constructing/constructing-image6.png b/source/design/constructing/constructing-image6.png new file mode 100644 index 0000000..7ca2e7c Binary files /dev/null and b/source/design/constructing/constructing-image6.png differ diff --git a/source/design/constructing/constructing-image7.png b/source/design/constructing/constructing-image7.png new file mode 100644 index 0000000..eabb626 Binary files /dev/null and b/source/design/constructing/constructing-image7.png differ diff --git a/source/design/versioning.rst b/source/design/versioning.rst new file mode 100644 index 0000000..c7c5788 --- /dev/null +++ b/source/design/versioning.rst @@ -0,0 +1,65 @@ +Module versioning and branching +========= + +The primary core principle of Modularity is that our content should be +released not as large, monolithic distribution releases, but in units of +smaller modules designed to be assembled in different combinations. + +Closely related is a second core principle: we should be able to release +new major versions of these modules on their own schedules to serve +a component's requirements; they should not all be tied to the master cadence +of today’s major release cycles. + +So, at a deep level, Modularity requires us to branch and version +modules independently of each other. + +Yet we need to control this complexity: the engineering involved has to +be sustainable, and the combinations we offer to the user need to be +manageable. This is especially true as different tools represent +branches in different ways (eg. bugzilla represents branches as the +“version” field for a product, and also has version-specific flags; +koji has koji tags which act as branches, etc.) + +This leads to some complex constraints, which we will explore in this +document. + +Ultimately, we can identify many distinct variants on branching: +sometimes different parts of the release pipeline end up with multiple +different views of the underlying branches. Just consider Fedora fXX +branches: these either look like a continuous update stream to the end +user, or *some* of the branches end up having distinct lifespans. + +So while a single consistent end-to-end branching model for any module +is possible in simple cases, it is unlikely to satisfy all the product +demands for complex release structures. This suggests an approach to +branching involving: + +- A centralised representation of the current branching at any point in + time; +- Retain separate branch definitions in our various tools such as + bugzilla, koji, bodhi, etc, as we have today; +- A flexible scripting approach to automating creation of new branches + and key transitions on existing branches (dev to beta to release + etc), to keep branches on the different tools synchronised. + +This splits *mechanism* — the central branching repository and the branch +definitions in the different tools — from *policy* — the specific branches +and transitions actioned by the scripts. + +It is important to remember that the mechanism needs to be flexible +enough to represent any potential desired branching structure; but this +does not mean that all modules have to have complex branching we can +still (and we should) adopt branching policy that is as simple as +possible for any given module. + + +.. rubric:: TOC + +.. toctree:: + :maxdepth: 1 + + versioning/terminology + versioning/properties + versioning/coherency + versioning/managing + versioning/constraints diff --git a/source/design/versioning/coherency.rst b/source/design/versioning/coherency.rst new file mode 100644 index 0000000..9ccd4c0 --- /dev/null +++ b/source/design/versioning/coherency.rst @@ -0,0 +1,71 @@ +Coherency of branching +====================== + +There are many advantages to be had in a single, coherent view of the +branching structure for a given module. + +Maintainers and users alike have to deal with branches in multiple +places: + +- We plan, develop and build content in bugzilla, dist-git + and koji, then release it through bodhi. All of these + tools share a common understanding of the various product branches + (fXX etc.) and work flows naturally between them. +- Users consume products knowing which branch they are on; they + have the option to choose between branches (F25 after F26 has + been released), and they report bugs and + review CVEs corresponding to those branches + +It also seems highly desirable to automate creation of branches, +especially when we consider a future with many more modules than we have +today, each with their own branches. + +But the idea of a clean, consistent view of branching that is unified +end-to-end falls down in several places. Some of the difficulties +include: + +**Utility side-branches:** Some of our tools have slight variants on the +main branch naming to support specific workflow requirements. + +For example, in CI we can have staging branches alongside the production +release branches, and in koji we have scratch builds; these follow the +main branching but are intended for developer use cases, rather +than automatically being candidates for release. + +We have candidate tags, beta tags and release tags in koji, indicating +packages on various different stages of the lifecycle from development +to release. Beta branches in general represent a special case here. + +**Multiple views of branching:** There are several places where two +different parts of the release pipeline can treat branching differently +from each other. Two important examples here include **minor version +branching** and **per-edition views** of a component: + +**Minor-version branching**: TBD + +**Per-edition views** of branching: TBD + +**Branch fluidity:** TBD + +**Naming policy:** TBD + +**Consistency of release:** Finally, we need to consider the granularity +of branches. The purpose of modularity is to allow us to release modules +independently from a single master release cadence. But do we really +want all modules to be released without any synchronisation or common +branching at all? + +History suggests we do not. + +In the future we likely have many completely-decoupled modules for +additional content outside the base Fedora runtime platform. But we may +still eventually decide that we want to have synchronised releases of +new content across different modules. + +So while modules can have independent branches, we still need the +ability to drive a common branching structure across a set of modules +when that is needed for product release requirements. First-class +support for such a **consolidated release** is absolutely necessary; to +devolve the distribution into an unmanaged, completely-uncoordinated set +of independent modules is likely unsustainable for both engineers, maintainers, and +users alike. diff --git a/source/design/versioning/constraints.rst b/source/design/versioning/constraints.rst new file mode 100644 index 0000000..6d0045a --- /dev/null +++ b/source/design/versioning/constraints.rst @@ -0,0 +1,59 @@ +Constraints on branching +======================== + +Everything so far seems to say “branching is hard, let’s not assume what it +looks like but just store a flexible representation that we can adapt as +we need.” + +That’s true to some extent… but there *are* concerns we can anticipate +that we need to handle in our branching structure. Having covered the +fundamental principle that branching policy needs to remain flexible, +let’s look at some of the issues we need to handle as we define that +policy. + +Splitting a binary package build over multiple modules +------------------------------------------------------ + +This is something that is surprisingly common. + +Examples might be when we want to include a library to support our own +package/application, but do not want to give it full support for end-users; we +might include the library itself, but not include the components +that provide the ability to develop against it (the include files, static libraries etc. that +typically land in a -devel binary rpm). + +Can we do this naturally in a modular build chain? Clearly it breaks any +assumption that a module can be both compiled and composed in complete +independence from any other: if a package build ends up in multiple +modules, then the compile phase of building those modules is now linked. +We need to determine how important it is to support this. + +But it is still quite possible to achieve, if the modules which are to +share binaries have matching branches. In that case, module composes can +always agree on which koji branches [tags] to consume packages +from. So this may be fairly easy for modules which are part of a single +consolidated release, as defined above; it would be fair to restrict +this possibility to that case. + +Building a module in multiple build roots +----------------------------------------- + +Does a single module source branch result in a single composed binary +branch? Or do we build that same source multiple times against different +base distribution build roots? + +Clearly, branching becomes enormously more complicated if we need to +support builds for multiple different build roots in a single branch. +The idea of a single coherent branching structure from git to release is +broken if we have multiple output branches from a single input branch. + +But the entire point of ABI forwards compatibility is to avoid the need +to do this: to run a module on a set of major runtimes, it should, in +theory, be necessary simply to build it on the oldest runtime in that +set. A module built on F25 should run on F26 or f27, as long as it +is using only dependencies with long-term stability guarantees. + +So before working through the complexities of commit-once, +compile-multiple-times, it will be important to determine to what extent +we can simply depend on ABI compatibility to ensure a module works +against multiple runtimes. diff --git a/source/design/versioning/managing.rst b/source/design/versioning/managing.rst new file mode 100644 index 0000000..b9f2e3a --- /dev/null +++ b/source/design/versioning/managing.rst @@ -0,0 +1,133 @@ +Managing this branching complexity +================================== + +Given that the exact branching policy for a module is currently +inconsistent, needs to remain flexible, and changes +over time, how do we manage this? The question is especially significant +given that we are looking at significant changes to the way we divide +and release the distribution in the future; our future branching model +is currently completely unknown. + +This suggests that we should not try to formalise a branching and naming +policy at all. But we **must** eventually have automation for the +creation of branches and for branch transitions, especially given that +release consistency may require us to coordinate new branches across +many modules simultaneously. And our tools still need consistent views +across this complex branching structure. + +Separation of policy from representation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This suggests that we need: + +- A canonical definition of our modules and their branches at any point + in time, including the way those branch names are represented in + different tools: +- Consistent use of that canonical branching structure within our + tools, but with +- Flexible, scripted events to drive changes in the branching. + +We can do this by separating the central representation of branching +(eg. in PDC) from the mechanism used to define and update that +branching. + +Changes in branching also need to be orchestrated: we should not define +a new branch and allow a developer to start building on that branch, +before the branch has been created in bugzilla, dist-git, koji etc. +There are many tools that could be used here: ansible is just one such +tool. The point here is to identify that as a separate concern. +Automation here is important if we want to be able to support +coordinated release branching across a set of modules. + +For now, we are dealing with a relatively simple branching structure, +building simple modules out of the latest Fedora. We don’t *need* +complex branching policy right now. But separating out representation +from policy allows us to start with a simple branching structure +initially, and still lets us define, and script, more complex, +product-specific branching requirements later, while having those +consistently represented in a central database that our tools can refer +to and agree on. + +Forking a new branch +~~~~~~~~~~~~~~~~~~~~ + +We have mentioned that creating a new branch for a module involves +branching multiple different tools: we need a branch for the module in +dist-git, new branches for its components, and corresponding branches in +bugzilla; we may need new tags in koji. + +This implies that the branching for a module is (usually) the same thing +as the branching for all the component packages of that module. + +But sometimes we will not want to branch *all* packages; we may want a +variant branch of a module which overrides just some of the packages, +and which otherwise inherits the content (''including new content) +''from its base branch. + +There are many examples which would suit such a **inheriting branch.** +The f-stream model which allows early access to new features prior +to an update, is an example. Another might be the specialized version of +the virtualisation stack, which contains a version of kvm-qemu with +newer features but which otherwise follows Fedora. The model also +works for scratch or staging branches, where we can build and test +updates to an existing branch as needed to suit internal developer +needs. + +This suggests that we want to include tooling support for inheriting such a + branch. Technically, this might involve creating new branches +for only a subset of the packages of a module; and recording the base +module from which we pull other packages during a module compose. + +Converging branches +~~~~~~~~~~~~~~~~~~~ + +Just as important as forking a new branch is converging existing +branches. In an f-stream model, a new feature scheduled for the next release +is made available in a prior release. The f-stream is the early-access branch; +the intent is that when the next release occurs, it introduces that +feature into the mainline stream, and the f-stream is no longer +needed: any component depending on that feature moves back off the +f-stream branch and onto the mainline. + +Extending this to a modular build, we can imagine a component +needing a new feature within any module in our stack. If that module +does not plan the feature to be released in time, we can fork a specific +version of the module to serve the needs of the one component needing the +new feature; but if and when that feature is released in some mainline +version of the module, we want the ability to move the component +off the forked feature branch and back onto mainline. + +There are likely to be many complexities here; the important point is to +imagine up-front that forking a new branch is only half the picture, it +will be useful to have tooling support for converging branches again +afterwards too. + +Managing unsynchronised stacks of branches +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Modules can depend in turn on other modules. We have defined a “stack” +as the entire tree of modules needed to satisfy dependencies for one +top-level module or application. But as we combine modules in this way, +not all those modules will have the same branches or lifecycle. + +So when we have multiple, different, unsynchronised branching models for +different modules within a stack, how do we know exactly which branches +of which modules we need to combine together? We can agree that we need +to constrain this complexity, and define specific subsets of modules +which we will test and support together. The issue is where, and how, to +define this. + +This is an issue we still need to solve. There two obvious places to +hold this structure: in the **release** that defines multiple modules +and their combined release schedules; or by defining specific branch +dependencies in each module’s own module metadata. + +Both have pros and cons. Defining specific branch dependencies in a +module’s metadata helps by keeping more of the module’s defining +structure in one place. However, the downside is that it becomes +impossible to use that same metadata in multiple places without changing +it: eg. building a single module from the same module source on multiple +buildroots is impossible if the module source itself defines its +buildroot dependency. + +So this is a topic for future consideration. diff --git a/source/design/versioning/properties.rst b/source/design/versioning/properties.rst new file mode 100644 index 0000000..1257658 --- /dev/null +++ b/source/design/versioning/properties.rst @@ -0,0 +1,74 @@ +Properties of branches and update streams +========================================= + +**ABI Compatibility:** Updates within a single update stream are likely +to maintain ABI backwards compatibility in most cases. Users should be +able to consume updates from an update stream without being concerned +about breaking applications that depend on that module. A major change +introducing an incompatible ABI would normally be expected to require a +new version branch. + +But this is ultimately a policy decision: there is nothing technical to +stop ABI breakage within a version stream. For example, the container-runtime +is rapidly changing so it has broken compatibility across versions. Our tools +should be able to detect incompatible ABI changes as far as possible, +but should not prevent them if we have an exceptional case where such a +change is desired. + +(ABI compatibility here includes anything that may have a compatibility +on user or application compatibility, including for example semantics of +configuration files, library ABIs, command line option handling and +error codes, and so on.) + +**Constraining the scope of ABI dependency:** As preserving ABI on +updates is a burden which imposes constraints on our maintenance of a +module within a single version branch, we would like the ability to +limit the parts of a module to which ABI stability applies. We currently +define which packages within a module form the **external ABI** of the +module: this is defined by the maintainer of a given module’s metadata. +Conversely, packages not declared as external are implicit internal +implementation details of the module. + +Defining the external ABI as a set of packages will allow us to: + +- Rebase internal packages without constraint from ABI guarantees, + removing overhead from the module maintenance burden over time; +- Verify that layered modules or applications depend only on packages + defined as external ABI, by checking rpm dependency chains + +Over time, we may find other ways to describe the external ABI of a module +but, at present, RPMs are the best stand in. + +**Lifecycle:** Given that we define no formal policy on ABI +lifecycle—rather leaving this up to policy—it follows that there is no +strong requirement that version numbers of packages within a single +update stream have to follow any particular pattern. We can easily +rebase a package within an update stream, even adding new features, as +long as any claimed backwards compatibility is preserved. + +We do need to be concerned about whether 3rd-party application +"certification" is expected to be preserved when such an application +depends on a module’s version branch containing rebased packages. This +is an important question, and we need to add tooling and policy around +it; but for now this is primarily a policy question, and beyond the +scope of this document. Different modules may have different appetite +for risk and rebases, and hence have different policy around +"certification." + +**Parallel Availability:** The update streams for different module +version branches must be able to coexist in our pipeline and released +content, without interfering with each other. If a given base system +install has both httpd-2.2 and httpd-2.4 available in different version +branches, then it is important that these remain independent. + +The update streams must not interfere with each other. If httpd-2.2 is +installed, then updating it via yum or dnf should update it to the most +recent version in the httpd-2.2 update stream, and must not +automatically update it to 2.4. Any dependencies brought in by either +must also prevent such interference. + +And yet if a certain package *does* support parallel installation of +different version branches at the same time +(eg. software collection-based packages), then the +separate installed versions at any time must each be updatable by their +own specific update stream. diff --git a/source/design/versioning/terminology.rst b/source/design/versioning/terminology.rst new file mode 100644 index 0000000..56b6734 --- /dev/null +++ b/source/design/versioning/terminology.rst @@ -0,0 +1,40 @@ +Basic branching terminology +=========================== + +First, though, note that the very word “version” is overloaded here. It +can refer either to a completely separate branch of some module, or to a +single specific instance or compose of a module. To keep terminology +straight, we need to distinguish between: + +- A **version branch**: or more simply just a **branch**: a completely + new, carefully planned version stream for a module. This might + correspond to a major Fedora release, or a new Fedora SCL release. + Creating a new branch should come with a new **release** target for + the branch (or branches if we’re releasing multiple modules + simultaneously.) + + A version branch **may** correspond to a new **major version** of + the module, but there may be exceptions: for example, when we add + specific features as a side-branch of an existing major version. The + new branch may differ by SLA (eg. a faster moving Python) or by + feature (eg. an “f-stream” branch giving early access to a + specific new feature planned to be released in a subsequent update). + + Users must opt into a new version branch. By definition here, + creating a new version branch **must** have no impact on users who + have already enabled an existing branch of a module. If a user + provisions a new environment and asks for the most recent version + branch, then they may get the new version branch automatically; but + no existing environments will transparently receive content for the + new version branch. + +- A **point-in-time version** or **instance** representing a single + compose of a module on a single version branch, built by and + identifiable by the **compose ID** of the task used to compose the + module within the build system. + + Such a point-in-time update may be just a scratch build, or may be + internal-only and not released to the user. But once it is released, + it forms a new **update** for that version branch of the module. + Multiple released point-in-time updates therefore form an **update + stream** over time for that branch. From 22c70b07c32aa0b4701080c7a45ed5089a68bd40 Mon Sep 17 00:00:00 2001 From: asamalik Date: May 29 2017 07:49:01 +0000 Subject: [PATCH 3/8] move get-involved one lvl up --- diff --git a/source/development/getting-started.rst b/source/development/getting-started.rst index 65ad4d5..cb46a9c 100644 --- a/source/development/getting-started.rst +++ b/source/development/getting-started.rst @@ -1,10 +1,104 @@ -Getting Started -=============== +Get Involved +============ -.. rubric:: TOC +Steps to get involved +--------------------- -.. toctree:: - :maxdepth: 1 +.. raw:: mediawiki + + {{Team contact|Modularity WG|devel|#fedora-modularity}} + +To get involved with the Modularity Working Group, please follow the +following steps for becoming a part of the team. + +#. Read this wiki page and understand the development methods and + technical details. +#. Get in touch with us on the `devel mailing + list `__ + and our IRC channel, + `#fedora-modularity `__. +#. Attend one of our `weekly + meetings `__ and + introduce yourself. +#. Become familiar with the Agile project management and create an + account on the `Fedora + Taiga `__. +#. Create an account on `Pagure `__ and file an + issue in the `Fedora @modularity + group `__ requesting membership. + +Get in touch +------------ + +There's no dedicated mailing list yet and everything regarding this +topic should be discussed on the general `Fedora Development +list `__. +Most of us also hang out on the #fedora-modularity channel on Freenode. + +Planning +-------- + +Formal meetings are held once a week by the Modularity Working Group. +See the +`Fedocal `__ to +find out when the group is meeting. + +Development Method +------------------ + +We will use `agile software +development `__ +methods for Modularity, more specifically: a hybrid of +`Scrum `__ +and `Kanban `__ +adapted to the constraints we have in Fedora. For instance, not all +contributors can commit to be involved like a regular, full-time +employee, meaning that rigid use of 2-week-long Scrum Sprints can be an +obstacle to participating for some people. + +If you're not familiar with agile development or the methods we use, +here are some links to get you started: + +- `Learn Scrum in 8 + minutes `__ +- `Kanban applied to + Scrum `__ + +Agile development methods often come with their own lingo that can be +confusing to the "uninitiated"—like +`"epic" `__, +`"sprint" `__ (or +`"iteration" `__), +`"spike" `__. Many of the terms used +are explained over at the `Agile +Dictionary `__. + +We manage the project using `Taiga `__ on Fedora +Infrastructure, `one +instance `__ for +individual sprint work items and +`another `__ +for higher-level stuff ("epics"). + +Technical details +----------------- + +Architecture +~~~~~~~~~~~~ + +Refer to :doc:`/architecture/infrastructure` + +Code repositories +~~~~~~~~~~~~~~~~~ + +We currently host all of our code at `Pagure `__ — +the infrastructure, the client-side tooling, metadata specification +drafts and even a couple of proof-of-concept modules. Repositories +typically start with the *fm-* prefix and are open to all members of the +`Pagure @modularity group `__. + +Services and tools +~~~~~~~~~~~~~~~~~~ + +Refer to :doc:`../prototype/developer-notes` - getting-started/get-involved - getting-started/faq diff --git a/source/development/getting-started/get-involved.rst b/source/development/getting-started/get-involved.rst deleted file mode 100644 index cb46a9c..0000000 --- a/source/development/getting-started/get-involved.rst +++ /dev/null @@ -1,104 +0,0 @@ -Get Involved -============ - -Steps to get involved ---------------------- - -.. raw:: mediawiki - - {{Team contact|Modularity WG|devel|#fedora-modularity}} - -To get involved with the Modularity Working Group, please follow the -following steps for becoming a part of the team. - -#. Read this wiki page and understand the development methods and - technical details. -#. Get in touch with us on the `devel mailing - list `__ - and our IRC channel, - `#fedora-modularity `__. -#. Attend one of our `weekly - meetings `__ and - introduce yourself. -#. Become familiar with the Agile project management and create an - account on the `Fedora - Taiga `__. -#. Create an account on `Pagure `__ and file an - issue in the `Fedora @modularity - group `__ requesting membership. - -Get in touch ------------- - -There's no dedicated mailing list yet and everything regarding this -topic should be discussed on the general `Fedora Development -list `__. -Most of us also hang out on the #fedora-modularity channel on Freenode. - -Planning --------- - -Formal meetings are held once a week by the Modularity Working Group. -See the -`Fedocal `__ to -find out when the group is meeting. - -Development Method ------------------- - -We will use `agile software -development `__ -methods for Modularity, more specifically: a hybrid of -`Scrum `__ -and `Kanban `__ -adapted to the constraints we have in Fedora. For instance, not all -contributors can commit to be involved like a regular, full-time -employee, meaning that rigid use of 2-week-long Scrum Sprints can be an -obstacle to participating for some people. - -If you're not familiar with agile development or the methods we use, -here are some links to get you started: - -- `Learn Scrum in 8 - minutes `__ -- `Kanban applied to - Scrum `__ - -Agile development methods often come with their own lingo that can be -confusing to the "uninitiated"—like -`"epic" `__, -`"sprint" `__ (or -`"iteration" `__), -`"spike" `__. Many of the terms used -are explained over at the `Agile -Dictionary `__. - -We manage the project using `Taiga `__ on Fedora -Infrastructure, `one -instance `__ for -individual sprint work items and -`another `__ -for higher-level stuff ("epics"). - -Technical details ------------------ - -Architecture -~~~~~~~~~~~~ - -Refer to :doc:`/architecture/infrastructure` - -Code repositories -~~~~~~~~~~~~~~~~~ - -We currently host all of our code at `Pagure `__ — -the infrastructure, the client-side tooling, metadata specification -drafts and even a couple of proof-of-concept modules. Repositories -typically start with the *fm-* prefix and are open to all members of the -`Pagure @modularity group `__. - -Services and tools -~~~~~~~~~~~~~~~~~~ - -Refer to :doc:`../prototype/developer-notes` - From 26a980fe71d0bc8c522ef5c7df275ec8fb323225 Mon Sep 17 00:00:00 2001 From: asamalik Date: May 29 2017 07:49:59 +0000 Subject: [PATCH 4/8] move and update resources doc --- diff --git a/source/development/integration/code-repositories.rst b/source/development/integration/code-repositories.rst deleted file mode 100644 index 46d5974..0000000 --- a/source/development/integration/code-repositories.rst +++ /dev/null @@ -1,126 +0,0 @@ -Code Repositories -================= - -This page contains list of code repositories of all Modularity related -projects with short description of each project. This page is updated -regularly, but to see the most actual list of projects, check Modularity -group on `Pagure `__ or on -`GitHub `__. Note that on these -pages, there can be also repositories for already abandoned projects. - -Common libraries ----------------- - -This is list of common libraries we use across various Modularity -projects: - -Modulemd -~~~~~~~~ - -Link: `https://pagure.io/fm-metadata `__ - -This repository contains the definition of metadata format (Modulemd) -used to describe every module. It also contains Python library used to -manipulate files in this format. - -Modulemd-resolver -~~~~~~~~~~~~~~~~~ - -Link -`https://pagure.io/fm-modulemd-resolver `__ - -This repository contains Python library which is used to resolve -dependencies between multiple Modulemd objects. It uses similar way as -yum/dnf uses to resolve dependencies between RPM packages. - -Server side projects --------------------- - -This is list of server side projects which runs on Modularity servers: - -Orchestrator (aka Rida) -~~~~~~~~~~~~~~~~~~~~~~~ - -Link: -`https://pagure.io/fm-orchestrator `__ - -Orchestrator coordinates module builds. It accepts new build requests -from client tools, schedules and coordinates build of module and tracks -the module build status. - -PDC fork -~~~~~~~~ - -Link: -`https://github.com/fedora-modularity/product-definition-center `__ - -This is Modularity aware fork of PDC (Product Definition Center). It is -used as server-side storage of built modules' metadata. Other server -side services uses it to get information about built modules. - -PDC updater -~~~~~~~~~~~ - -Link: -`https://github.com/fedora-modularity/pdc-updater `__ - -PDC updater updates the Product Definition Center based on the fedmsg -messages. - -Build Pipeline Overview (BPO) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Link: -`https://github.com/asamalik/BPO `__ - -This is a web service providing a single UI for accessing information -about build states of modules. - -Client side projects --------------------- - -This is list of client side projects which are executed on client system -to work with modules: - -Modularity DNF plugin -~~~~~~~~~~~~~~~~~~~~~ - -Link: -`https://pagure.io/fm-dnf-plugin `__ - -This repository contains "dnf module" plugin which is use to managing -modules on client systems. It can search for modules, enable them, -disable them, update them and so on. - -Pyrpkg fork -~~~~~~~~~~~ - -Link: -`https://pagure.io/fork/karsten/rpkg.git `__ - -This is Modularity aware fork of pyrpkg library. When installed, it -extends the fedpkg command with "module-build" subcommand. - -Modules -------- - -This is list of modules described in the Modulemd metadata format which -can be used as an examples of modules: - -Base-runtime -~~~~~~~~~~~~ - -Link: -`https://pagure.io/base-runtime `__ - -This repository contains module definitions for the so-called -base-runtime as well as other essential modules needed at both run- and -build-time. This set of modules should provide everything module -developers need to build their own. - -FM Modules -~~~~~~~~~~ - -Link: `https://pagure.io/fm-modules `__ - -Example of various modules like httpd or mariadb. diff --git a/source/development/resources.rst b/source/development/resources.rst new file mode 100644 index 0000000..0b8e7aa --- /dev/null +++ b/source/development/resources.rst @@ -0,0 +1,36 @@ +Resources +========= + +This page lists our Git repositories and other resources we use. + + +Git Repositories +---------------- + +Specifications +~~~~~~~~~~~~~~ + +`Modulemd `__ - +This repository contains the definition of metadata format (Modulemd) +used to describe every module. It also contains Python library used to +manipulate files in this format. + +Content +~~~~~~~ + +`Base Runtime `__ - +The Base Runtime is a small system used as a base for running modules. + +`Modules `__ - +All the modules we are working on are in the following GitHub organization. Modules are defined by modulemd. + +`Containers `__ - +All modules can be also delivered as containers. The following GitHub organization contains definitions of all our containers. + + +Scripts and tooling +~~~~~~~~~~~~~~~~~~~ + +`Modularity on GitHub `__ + +`Modularity on Pagure `__ From c738871c4794be7964d1fe9c1973d06834f38698 Mon Sep 17 00:00:00 2001 From: asamalik Date: May 29 2017 07:50:52 +0000 Subject: [PATCH 5/8] remove outdated/redundant docs from Development --- diff --git a/source/development/getting-started/faq.rst b/source/development/getting-started/faq.rst deleted file mode 100644 index 49c0235..0000000 --- a/source/development/getting-started/faq.rst +++ /dev/null @@ -1,159 +0,0 @@ -FAQ -=== - -How is this related to `Environment Modules `__? ------------------------------------------------------------------------------- - -Environment Modules are a concept unrelated to the Modularity -initiative. However, they are a different attempt to solve a similar issue. - -What is the concept of a module? --------------------------------- - -- A module can be thought of as a dotted line we draw around a set of - components that we declare a “thing.” It makes sense to think of a - module as unit of measure larger than a traditional RPM. -- A module is a unit of delivery, that is always tested together and - released together. That said, we would like to provide a - short-circuiting mechanism for a sys-admin/ops person whereby they - can knowingly "break" a module by applying a library update because - the module is not being released in a timely manner. -- A module as a whole has its own lifecycle that is independent of any - other module. Maintainers may decide to release multiple modules - together on a common release schedule, but it is always possible to - release modules independently when desired. -- A module may include different versions of components than other - modules. -- A module comes with associated metadata: this may include such things - as lifecycle information (when does the module go end of life), who - maintains it and to what support level, etc - -What is a module in a practical sense? --------------------------------------- - -We want to use this definition for the sake of prototyping and early -implementation. We may find this changes based on the prototypes. For -now, a module is: - -- A repository. Yeah, just a plain old RPM repository (for now). A - module definition declares what RPMs it includes (both hard requires - and optionally requires). All of these RPMs are included in the repo - that “is” the module. A module may also specify in its definition - that it depends on one or more other modules, but it may not specify - any of the RPMs in that "remote" module. -- A module has a unique name -- A module can have multiple distinct versions, likely corresponding to - distinct functionality or ABI versions; and multiple versions may be - available at the same time. -- Each version of a module has its own independent update stream - associated with it. We avoid changing ABI or intentionally breaking - forwards compatibility in any way within the update stream of a - single version. -- Has a well known set of non-runtime dependencies which are not - available in the same "repo" as the module itself. While it seems - like this could be easily supported in the "for now" case, having - this requirement makes sure we don't paint ourselves into a corner. -- A module has an API. In essence, the API is what "makes" the module. - For example, if we had a "Web server" module, its "api" might be - HTTP/2, we could provide that using httpd or nginx, and, next week, - swap it, because the api is king, not the binaries inside. However, - while we need to consider the API model, full support of this may not - be necessary for the MVP. -- Alluded to earlier, but, a module is \*not\* self-hosted. That - doesn't mean Fedora doesn't know how to build it, or that the - information and steps to build it aren't available, just that the - consumer has to take some extra steps to find this information. We - don't, necessarily, want to consider the build deps the same level of - quality as the module itself. In the future, an “edition” or a “spin” - would be composed of a set of modules (vs a set of RPMs). -- Not directly installable and/or may be installers themselves. In - other words, a module does not have to carry tooling to get itself to - the end user. - -OK, so what is modularity? --------------------------- - -Modularity is an ongoing initiative in Fedora to resolve the issue of -divergent, occasionally conflicting, lifecycles of different components -(modules). - -What is a module’s lifecycle? ------------------------------ - -A module as a whole has its own lifecycle independent of any other -module. Maintainers may decide to release multiple modules together on a -common release schedule, but it is always possible to release modules -independently when desired. - -What are a module’s standard properties? ----------------------------------------- - -- A module has a unique name. -- Each version of a module has its own independent update stream - associated with it. We avoid changing ABI or intentionally breaking - forwards compatibility in any way within the update stream of a - single version. -- A module has a well known set of non-runtime dependencies which are - not available in the same “repo” as the module itself. While it seems - like this could be easily supported in the “for now” case, having - this requirement makes sure we don't paint ourselves into a corner. -- A module has an API. In essence, the API is what makes the module. - For example, if we had a Web server module, its API might be HTTP/2. - We could provide that using httpd or nginx, and, next week, swap it, - because the API is king, not the binaries inside. However, while we - need to consider the API model, full support of this may not be - necessary for the - `MVP `__. -- A module comes with associated metadata such as lifecycle information - (when does the module go end of life), who maintains it and to what - support level, etc. -- A module may include different versions of components than other - modules. - -Note: We still don't know what kind of API we're going to define. So far -we've only considered marking some of the provided binary packages as -the module's "external API". That way we know what to test on updates -and module consumers know what binary packages they can rely on. I'm not -sure modules are ever going to provide abstract APIs. - -Why do we call it modularity? ------------------------------ - -We are trying to use an agnostic term so as not to indicate the specific -nature of a module or a component. Modules can be big or small, low or -high-level, brand new or very mature. As a result, please try not to -ascribe too much meaning to the word module or component aside from -being a slightly prettier and shorter version of “chunk of some stuff.” - -Why are we pursuing this goal? ------------------------------- - -Well, there a a lot of reasons but, I think, the simplest is to try to -disconnect the lifecycle of major components from each other so that -they can grow and change at the speed that is appropriate to the -component. Why does that matter? Well, that is a significantly more -complex conversation and somewhat beyond the scope of this document. - -I heard there were videos? --------------------------- - -Yes, videos are delivered at the completion of every sprint and posted -to a `Fedora Modularity -YouTube `__ -channel. - -Where can I find out more? --------------------------- - -The best place to start is where you already are: -`Modularity `__ - -As the wiki evolves, categories and new content will be added. Make sure -not to miss the blog: https://communityblog.fedoraproject.org/ - -- Blog posts tagged with “Modularity” - https://communityblog.fedoraproject.org/tag/modularity/ - -- :doc:`/architecture/infrastructure` - -- Refer to :doc:`../prototype/developer-notes` diff --git a/source/development/integration.rst b/source/development/integration.rst deleted file mode 100644 index 8667318..0000000 --- a/source/development/integration.rst +++ /dev/null @@ -1,13 +0,0 @@ -Integration of changes -====================== - -In order so that our code base is always clean and maintainable, we have to enforce certain rules on how code is written or formatted, how changes are broken up into commits and how pull requests are handled. - -.. rubric:: TOC - -.. toctree:: - :maxdepth: 1 - - integration/coding-style - integration/grooming - integration/code-repositories diff --git a/source/development/integration/coding-style.rst b/source/development/integration/coding-style.rst deleted file mode 100644 index 8915dc9..0000000 --- a/source/development/integration/coding-style.rst +++ /dev/null @@ -1,503 +0,0 @@ -Coding Style -============ - -Most of our code is written in Python, so this document will concentrate -on it. - -Upstream guidelines -------------------- - -Fortunately, with PEP 8 there's an extensive official `Style Guide for -Python Code `__. All new -Python code you submit should conform to it, unless you have good -reasons to deviate from it, `for instance -readability `__. - -Keep PEP 20, the `Zen of -Python `__, under your -pillow. - -Keep It Simple --------------- - -The code you write now probably needs to be touched by someone else down -the road, and that someone else might be less experienced than you, or -have a terrible headache and be under pressure of time. So while a -particular construct may be a clever way of doing something, a simple -way of doing the same thing can be and often is preferrable. If (when) -complexity can't be avoided, try to isolate it: put a difficult -operation into its own function, method or class, add comments. If -complexity can be hidden from upper layers of the code, do so. - -Comments and Docstrings ------------------------ - -Be generous when it comes to commenting your code, it's better to have a -superfluous comment than if one were necessary but is missing. However, -if there is a comment it should be correct and agree with the code, -otherwise people have to guess if the comment or the code needs to be -straightened out. - -Adding `docstrings `__ to -modules, classes, methods and functions is encouraged. If you use the -`Sphinx -format `__ -to describe parameters, return values, etc., even better! - -Python 2 and 3 --------------- - -Python comes in two major versions nowadays: - -- The legacy version 2, of which the `first release 2.0 came out in - October 2000 `__. The - Python project `will maintain its final minor release 2.7 until - 2020 `__. - -- The current version 3, its `first release 3.0 was published in - December 2008 `__. At - the time of writing, the current minor release is version 3.5, to be - superseded by 3.6 around the end of 2016. - -Version 3 is not backwards compatible to version 2. While we mainly -target "the future", there are some components we have to work with that -haven't yet been ported over the Python 3, most notably -`koji `__. Additionally, we may also -want to support the "user tools" we create on legacy systems, so we -can't write code that uses all the latest features. Fortunately, many of -the original Python 3 features have been back-ported to Python 2.7, so -we can and should write code that is very close to writing idiomatic -Python 3 but can still be run on version 2.7. Targeting older minor -releases (Python 2.6 and earlier) is much more of a balancing act, so we -won't aim for it. - -The following sections cover areas that require some attention. The -Python project itself has a great `Porting Python 2 Code to Python -3 `__ document which -goes into much detail about the differences and is worth a read, even -though it mainly addresses existing Python 2 code bases. - -Absolute and relative imports -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In Python 2, importing modules can be ambiguous when a module of that -name exists in the same package and elsewhere in the module search path -``sys.path``. To work around this ambiguity, programmers often resorted -to adding paths private to the project to the beginning of ``sys.path`` -to force loading modules from a project-internal location (which adds -unwanted noise and can make e.g. testing code that isn't installed -difficult). Python 3 introduces new syntax for import statements which -makes both cases distinct, this is available since version 2.5 from the -``__future__`` module: - -:: - - from __future__ import absolute_import - - # Import the sys module from the module search path - import sys - - # Import the foo module from the same directory - from . import foo - - # Import snafu from the bar module one directory above - from ..bar import snafu - -Print function -~~~~~~~~~~~~~~ - -Python 3 did away with ``print`` as a statement and introduced it as a -function. In order to use it the same way in Python 2.7, add the -following to the top of source code files where you use ``print``: - -:: - - from __future__ import print_function - -Numbers -~~~~~~~ - -Python 2 has two integer types, \`int\` which is whatever integer-type -is native to the system (which has certain maximal and minimal values -and can overflow) and \`long\` which can store arbitrary integer -numbers. Python 3 only the latter type, but it's called ``int``. - -Dividing integer numbers using ``/`` truncates the result to an integer -in Python 2 by default, but yields a floating point number in Python 3. -In order for code to do the same thing on either version, include the -following line at the top of your source files where you divide numbers, -and use ``/`` for normal divisions and ``//`` for divisions that should -truncate the result: - -:: - - from __future__ import division - -Strings -~~~~~~~ - -Some consider this the main difference between Python 2 and 3: Both -versions have a type for strings of bytes and strings of Unicode -character points. They are called ``str`` and ``unicode`` in version 2 -and ``bytes`` and ``str`` in version 3, respectively. - -String Literals -^^^^^^^^^^^^^^^ - -Python 2 and 3 use different ways of marking literals of the different -types by default. Byte strings can have no prefix or ``b`` in Python -2.7, but must be prefixed in Python 3, and text strings must have the -``u`` prefix in Python 2 which can be and usually is omitted in Python -3: - -:: - - # a byte string in Python 2 and 3 - string1 = b"abc" - - # a byte string in Python 2, but a text string in Python 3 - string2 = "def" - - # a text string in Python 2 and 3 - string3 = u"ghi" - -In order to ease writing code that is compatible between the versions, -you can switch Python 2 to treat unprefixed string literals as -``unicode``, the text string type, by adding this snippet to the top of -the relevant source code files: - -:: - - from __future__ import unicode_literals - -Explicit Encoding and Decoding -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In Python 2, the byte and text string types are exchangeable in many -places, taking the user's or system default locale into account (and -sometimes failing, when the locale didn't match up with encoded data). -Apart from the change in type names and how literals look like, Python 3 -requires you to explicitly encode ``str`` and decode ``bytes`` objects -if you need them cast into the respective other string type. It is good -practice to exclusively use text strings for strings that represent text -in a program and decode byte strings as early and encode text strings as -late as possible at interfaces that produce or consume encoded data. - -.. raw:: mediawiki - - {{admon/note|Implicit string type conversion in Python 2|Python 2 lets you attempt to replace a str substring in a unicode object (or vice versa) and would attempt to cast the one into the other by encoding or decoding on the fly as needed. This piece of code won't work in Python 3:}} - -:: - - from __future__ import print_function - text_string = u"Hello, world!" - print(text_string.replace("world", "gang")) - -.. raw:: mediawiki - - {{admon/tip|Explicit string type conversion in Python 2 and 3|Python 3 requires explicit encoding/decoding to cast between byte and text strings. This also works in Python 2 and is preferred of course.}} - -:: - - from __future__ import print_function, unicode_literals - text_string = "Hello, world!" - print(text_string.replace(b"world".decode('utf-8'), b"gang".decode('ascii'))) - -String formatting -^^^^^^^^^^^^^^^^^ - -With version 3.6 around the corner, there are four ways to format -strings in Python now: - -#. using the ``%`` operator -#. using ``string.Template`` of `PEP - 292 `__ -#. with the ``str.format()`` method -#. using `PEP 498 literal string - interpolation `__ - -The last method isn't available yet in a stable Python release and will -never be in Python 2, so it's not suitable for our purposes. The other -three variants work in all Python versions we're interested in, -formatting with ``string.Template`` is very rarely done however. The -remaining two ways, commonly called old-style (``%`` operator) and -new-style (``str.format()``), are both in wide-spread use, `here's a -site showcasing the differences between -them `__. New-style formatting is more powerful -and often easier to read, but on the other hand can be a little more to -type. From a technical point of view, this is a case of "use what works -for you", but for consistency sake the new-style ``str.format()`` way is -preferrable if you're comfortable with using it. If not, others can -convert old-style to new-style formatting for you during review or when -happening across it. At any rate, consistently use one way or the other -in what you submit. - -Old- and New-style Classes -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Python 2 and earlier knows two types of classes, old-style which have no -base class, and new-style which have ``object`` as the base class. -Because their behavior is slightly different in some places, and some -things can't be done with old-style classes, we want to stick to -new-style classes wherever possible. - -The syntactical difference is that new-style classes have to explicitly -be derived from ``object`` or another new-style class. - -:: - - # old-style classes - class OldFoo: - pass - - class OldBar(OldFoo): - pass - - # new-style classes - class NewFoo(object): - pass - - class NewBar(NewFoo): - pass - -Python 3 only knows new-style classes and the requirement to explicitly -derive from ``object`` was dropped. In projects that will only ever run -on Python 3, it's acceptable not to explicitly derive classes without -parents from ``object``, but if in doubt, do it just the same. - -Idiomatic code --------------- - -In Python, it's easy to inadvertently emulate idiomatic styles of other -languages like C/C++ or Java. In cases where there are constructs -"native" to the language, it's preferrable to use them. - -Literals and Comprehensions -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Python has special syntax for literals for a couple of built-in compound -data types: lists, tuples, dictionaries, strings, sets. It's customary -to use that syntax instead of the class constructor to create objects -for these data types unless you have good reason not to. Apart from how -it looks, the literal syntax is performing a little bit better (because -it doesn't have to look up the class name in the current scope). NB: Set -literals are peculiar in that you can't create empty ones­—they would -look the same as empty dicts. - -+-------------+-------------------------------------+----------------------------------------+ -| Data Type | Good | Bad | -+=============+=====================================+========================================+ -| ``str`` | | ``a_str = "abc"`` | ``empty_str = str()`` | -| | | ``empty_str = ""`` | | -+-------------+-------------------------------------+----------------------------------------+ -| ``list`` | | ``a_list = [1, 2]`` | | ``a_list = list((1, 2))`` | -| | | ``empty_str = []`` | | ``empty_list = list()`` | -+-------------+-------------------------------------+----------------------------------------+ -| ``tuple`` | | ``a_tuple = ('a', 'b', 3)`` | | ``a_tuple = tuple(['a', 'b', 3])`` | -| | | ``empty_tuple = ()`` | | ``empty_tuple = tuple()`` | -+-------------+-------------------------------------+----------------------------------------+ -| ``dict`` | | ``a_dict = {'a': 1}`` | | ``a_dict = dict(('a', 1))`` | -| | | ``empty_dict = {}`` | | ``empty_dict = dict()`` | -+-------------+-------------------------------------+----------------------------------------+ -| ``set`` | | ``a_set = {"banana", "apple"}`` | ``a_set = set(["banana", "apple"])`` | -| | | **``empty_set = set()``** | | -+-------------+-------------------------------------+----------------------------------------+ - -Table: Creating compound objects - -Often the initial contents of a compound object are only known when it's -created at runtime. For simple cases like mere type conversions, calling -the class constructors are the way to go: - -- Converting a tuple to a list or vice versa: - -| ``   a_tuple = (1, 2, 3)`` -| ``   ...`` -| ``   a_list = list(a_tuple)`` -| ``   ...`` -| ``   another_list = [4, 5, 6]`` -| ``   ...`` -| ``   another_tuple = tuple(another_list)`` - -- Convert a list to a set, e.g. to filter out duplicates: - -| ``   a_list = [1, 2, 3, 2]`` -| ``   ...`` -| ``   a_set = set(a_list)`` - -For more involved cases, say some values need to be filtered or a -specific attribute of the objects is wanted, Python has so-called -comprehensions to create compound objects in a syntactically "nice" way. -These largely supersede the old (ugly) way of using ``map()`` and -``filter()`` in conjunction with class constructors. - -+-------------------------------------------------------------------------------------------------------+-------------+------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Comprehension type | Data Type | Example | Remarks | -+=======================================================================================================+=============+================================================+=============================================================================================================================================================================+ -| `List Comprehension `__ | ``list`` | ``a_list = [x for x in range(20) if x % 2]`` | Put all odd numbers smaller than 20 into a list. | -+-------------------------------------------------------------------------------------------------------+-------------+------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| `Dict Comprehension `__ | ``dict`` | | ``a_dict = {k: getattr(an_obj, k)`` | Fill a dict with those attribute names and values of an object that aren't considered "protected" or "private" (names with one or two leading underscores, respectively). | -| | | | ``    for k in dir(an_obj)`` | | -| | | | ``    if not k.startswith("_")}`` | | -+-------------------------------------------------------------------------------------------------------+-------------+------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| `Set Comprehension `__ | ``set`` | ``a_set = {o.name for o in a_list}`` | Create a set containing the value of the attribute ``name`` of objects in a list. | -+-------------------------------------------------------------------------------------------------------+-------------+------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - -Table: Using comprehensions to create compound objects - -Looping -~~~~~~~ - -Languages like C normally use incremented indices to loop over arrays: - -:: - - float pixels[NUMBER_OF_PIXELS] = [...]; - - for (int i = 0; i < NUMBER_OF_PIXELS; i++) - { - do_something_with_a_pixel(pixels[i]); - } - -.. raw:: mediawiki - - {{admon/warning|Looping C-style in Python|Avoid looping over indices of sequences, rather than the sequences themselves in Python.}} - -Implementing the loop like this would give away that you've programmed -in C or a similar language before: - -:: - - pixels = [...] - - for i in range(len(pixels)): - do_something_with_a_pixel(pixels[i]) - -.. raw:: mediawiki - - {{admon/note|Looping over iterables in Python|In Python, you can simply iterate over many non-scalar data types.}} - -Here's the "native" way to implement the above loop: - -:: - - pixels = [...] - - for p in pixels: - do_something_with_a_pixel(p) - -.. raw:: mediawiki - - {{admon/tip|Using enumerate()|If you need to keep track of the current count of looped-over items, use the enumerate() built-in.}} - -It yields pairs of count (starting at 0 by default) and the current -value like this: - -:: - - pixels = [...] - - for p_no, p in enumerate(pixels, 1): - print("Working on pixel no. {}".format(p_no)) - do_something_with_a_pixel(p) - -Properties rather than explicit accessor methods -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In order to allow future changes in how object attributes (member -variables) are set, some languages encourage always using getter and/or -setter methods. This is unnecessary in Python, as you can intercept -access to an attribute by wrapping it into a -`property `__ -if and when this becomes necessary. Properties allow having accessor -methods without making the user of the class have to use them -explicitly. This way you can validate values when an attribute is set, -or translate back and forth between the interface used on the attribute -and an internal representation. - -Validating a value when setting an attribute -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To ensure that an ``Employee`` object only has positive values for its -``salary`` attribute, you'd put a property in its place which checks -values before storing them in an attribute called e.g. ``_salary``: - -:: - - class Employee(object): - - @property - def salary(self): - return self._salary - - @salary.setter - def salary(self, salary): - if salary <= 0: - raise ValueError("Salary must be positive.") - self._salary = salary - -.. raw:: mediawiki - - {{admon/caution|Avoid recursion|In order to avoid endless recursion, you must use a different attribute than the one using the property to store actual values.}} - -Translating between attribute interface and internal representation -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Take these classes of geometric primitives, ``Point`` and ``Circle``: - -:: - - class Point(object): - def __init__(self, x, y): - self.x = x - self.y = y - - class Circle(object): - def __init__(self, point, radius): - self.point = point - self.radius = radius - -If you wanted to add a ``diameter`` attribute to ``Circle``, you can do -so as a property which translates back and forth between it and the -existing ``radius`` attribute: - -:: - - ... - class Circle(object): - def __init__(self, point, radius=None, diameter=None): - self.point = point - if (radius is None) == (diameter is None): - raise ValueError("Exactly one of radius or diameter must be set") - if radius is not None: - self.radius = radius - else: - self.diameter = diameter - - @property - def diameter(self): - return self.radius * 2 - - @diameter.setter - def diameter(self, diameter): - self.radius = diameter / 2.0 - ... - -Even setting ``self.diameter`` in the constructor goes by way of the -property and therefore the setter method. - -External links --------------- - -- `Python Design and History - FAQ `__ -- `PEP8: Style Guide for Python - Code `__ -- `PEP20: The Zen of - Python `__ -- `PyFormat: Using ``%`` and ``.format()`` for great - good! `__ -- `Sphinx Info field - lists `__ - for docstrings - diff --git a/source/development/integration/grooming.rst b/source/development/integration/grooming.rst deleted file mode 100644 index 3f7994e..0000000 --- a/source/development/integration/grooming.rst +++ /dev/null @@ -1,174 +0,0 @@ -Grooming Your Changes -===================== - -Apart from :doc:`coding-style`, -there are some things that you should keep in mind regarding the changes -you submit. Normally you'd develop your changes in a private branch on -your fork of a repository and, when you're done, submit them as pull -requests ("PR") against a public branch of the repository. The following -guidelines concentrate on changes in this format, their goal is to -enable you to groom the commits forming your pull request so that -another person can review it without great effort, that the changes can -be integrated well with the existing code and can be easily debugged -later if necessary. - -Pull Requests -------------- - -Scope -~~~~~ - -One pull request should really be about implementing one feature or -solving one problem. For instance, when developing your changes you -might spot a bug in existing code and fix it. Mixing these changes with -your new feature make reviewing them more work because the person doing -it needs to assess if a chunk of your changes is related to the feature, -or the bug fix. Similarly, if the review of your feature drags out, the -bug fix might take that much longer before it's available to others. In -most cases you should therefore create separate pull requests for both -sets of changes. - -As an exception to that, merely janitorial changes to the parts of the -code your pull request touches anyway—say, fixing trailing whitespace or -indentation, superficial changes that make the code you worked on better -to read or understand—are acceptable as long as you put these changes in -a commit or commits of their own, ideally put before your "real" changes -in the commit order. This makes it easier to cope with other PRs that -might fix the same things. - -Describing your changes -~~~~~~~~~~~~~~~~~~~~~~~ - -The bigger the changes you submit are, the more important it is to give -the reviewer a high level summary of what it is they are reviewing. If a -pull request consists only of one commit, then its commit log should be -sufficient in most cases and the forges hosting our repositories (Pagure -and GitHub) use it as the default description text on submission. If it -is longer, you may need to condense the individual changes of your -commits, and maybe lose some comments about the problem you wanted to -solve and your approach. If you are unsure about parts of your changes, -this is also the place give the reviewer a heads-up. - -Linear History and Rebasing -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The changes you submit for reviewing should be a linear string of -commits, please don't have merges in there. Therefore, in order to track -upstream changes while you are still developing in a private branch, you -should rebase it on top of the upstream branch you track. You can of -course do that manually, but it's easier to tell git to automatically -attempt to rebase your changes on top of the branch from which you pull -(replace ``$branchname`` with the actual name of your local branch): - -``   git config branch.$branchname.rebase true`` - -You can also set this globally for any newly created branch (you'd have -to do the above for all existing branches, though): - -``   git config --global branch.autosetuprebase always`` - -Set up this way, pulling from upstream will attempt to apply your -private commits in order on top of the new upstream ones, one after -another. If that fails at some point, e.g. because of conflicts, it'll -interrupt the rebasing process, so that you can resolve the issue, and -continue with ``git rebase --cont``. Alternatively, you could also -restore the previous state by running ``git rebase --abort``, e.g. to -assess the differences between your (unrebased) branch and upstream -before giving it a go again. - -The Review -~~~~~~~~~~ - -When you've submitted your changes as a pull request, hopefully someone -will pick it up soon (if not, poke some people on IRC: -`#fedora-modularity on -Freenode `__) and give you -feedback in form of comments, questions or suggestions. The comment -section of a pull request isn't very suitable for longer discussions, so -you might switch to email, IRC or another medium to discuss a topic, and -then summarize in the PR. Consulting other contributors is encouraged, -if additional opinions are needed. The job of a reviewer is not just to -act as a gatekeeper for the project, but also to assist you in getting -your changes into an acceptable state. This can go as far as making -minor fixes on the fly rather than asking you to do it, or bringing the -stack of commits "into shape" before merging the pull request. - -Individual commits ------------------- - -Commit Scope and Size -~~~~~~~~~~~~~~~~~~~~~ - -Like a pull request itself, a commit should also be about just one -thing. For example, you should split the implementation of a new class -from where existing code is converted to use it, as well as removing the -legacy code it replaces. The reverse also holds true—one concern should -be dealt with in one commit: if you discover bugs in a newly introduced -piece of code while you're still developing it, the buggy commit -introducing it and the fix should be rolled into one. This keeps the -number of broken commits down which e.g. makes it easier to use -``git bisect`` at a later point. - -.. raw:: mediawiki - - {{admon/note|"Commit early, commit often."|It's much easier to merge smaller commits into larger ones if they belong together, rather than disassembling a commit that actually addresses more than one concern.}} - -Commit Log Messages -~~~~~~~~~~~~~~~~~~~ - -The purpose of a commit log message is to briefly summarize the changes -in the commit, but it's also where background information should be put, -e.g. why some approach was used and not another. - -Format -^^^^^^ - -A commit log should consist of a short summary line (<50 characters, -also called "title"), optionally followed by a blank line and a more -thorough description. The summary should tersely describe the objective -of the commit, while the description would go into detail about the -actual implementation. - -Building a Commit -~~~~~~~~~~~~~~~~~ - -Often you'll want to pick only parts of your uncommitted changes, in -order to follow these guidelines, or to leave out debugging statements -which you don't want to submit. You can select the parts in your changes -you want to commit by using ``git add --patch`` which presents the -differences as hunks in unified diff format and lets you choose which -ones to add to the staging area and which to skip. After committing -these staged changes, you can repeat the process until all changes you -want to submit are taken care of. There are ways to separate a large -commit into smaller ones, but this approach is often more difficult one -of the two. - -Tools ------ - -- Adding using patch mode: With ``git add --patch ...`` you can pick - which changes you want to commit. -- Interactive rebasing: Use ``git rebase -i ... @{u}`` to reorder your - commits, reword their commit messages, merge or amend them. It's - important to not do this to upstream commits, therefore ``@{u}`` - specifies the point where your branch split off from upstream. - -.. raw:: mediawiki - - {{admon/important|If all else fails:|GIT remembers the history of revisions you had checked out in your repository, refer to the output of git reflog to find a "known good" one.}} - -.. raw:: mediawiki - - {{admon/caution|Using git reset|You can use git reset [--hard] $some_sha1_commit to bring you back to a known good state. Be careful, though: using the --hard option will lose any changes made to files under the control of GIT.}} - -See also --------- - -- The `Pro Git book `__ - - - The `"Rewriting - History" `__ - chapter for more detailed information about amending, interactive - rebasing, and other advanced ways of screwing up your repository - ;) - diff --git a/source/development/prototype.rst b/source/development/prototype.rst deleted file mode 100644 index e94ce09..0000000 --- a/source/development/prototype.rst +++ /dev/null @@ -1,14 +0,0 @@ -The Prototype -================================== - -.. rubric:: TOC - -.. toctree:: - :maxdepth: 1 - - prototype/getting-fedora-pkgs-for-testing - prototype/getting-pungi - prototype/composing-modules - prototype/local-pdc - prototype/developer-notes - prototype/container-in-openshift diff --git a/source/development/prototype/composing-modules.rst b/source/development/prototype/composing-modules.rst deleted file mode 100644 index 95ed553..0000000 --- a/source/development/prototype/composing-modules.rst +++ /dev/null @@ -1,102 +0,0 @@ -Composing Modules with Pungi -============================ - -Composing Modules with Pungi ----------------------------- - -After you have downloaded `Pungi (Modularity -style) `__ and -`the necessary Fedora -packages/repositories `__, -you can compose a module. Unlike with the versions of pungi currently -used for composing Fedora, this process is split into separate steps -with our prototype, using several distinct scripts. - -This document assumes that the checked out repositories are located in -``$WORKSPACE``, the local package repositories are in ``$REPOS`` and the -compose and intermediate data will be output into ``$COMPOSES``. You -need to activate the environment for running the pungi prototype, as the -case may be by either sourcing ``$HOME/.modularity.sh`` or, if you set -up a Python virtualenv, by running ``workon modularity``. - -Gathering the component packages -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``pungi-gather-prototype`` script produces a manifest of the desired -packages and their dependencies, in our example for a core and shells. -For productive use, these packages would come e.g. from a specific koji -tag. You may want to add the local repository with the minimized -packages, but then you have to ensure that they are preferred over the -normal Fedora packages (see -`Modularity/Development/Getting\_Fedora\_Packages\_for\_Testing `__ -for details). - -:: - - "$WORKSPACE"/pungi/bin/pungi-gather-prototype \ - --arch x86_64 --target-dir "$COMPOSES" \ - --config "$WORKSPACE"/pungi-modularity/pungi-inputs/core.yaml \ - --source-repo-from-path "$REPOS"/fedora-24-beta-x86_64 \ - --source-repo-from-path "$REPOS"/fedora-24-beta-src - "$WORKSPACE"/pungi/bin/pungi-gather-prototype \ - --arch x86_64 --target-dir "$COMPOSES" \ - --config "$WORKSPACE"/pungi-modularity/pungi-inputs/shells.yaml \ - --source-repo-from-path "$REPOS"/fedora-24-shells-x86_64 \ - --source-repo-from-path "$REPOS"/fedora-24-shells-src \ - --source-repo-from-path "$REPOS"/fedora-24-beta-x86_64 \ - --source-repo-from-path "$REPOS"/fedora-24-beta-src - -The script will output the location where it wrote the manifest after it -is run. At this point the manifests should be in -``$COMPOSES/manifest-{core,shells}-$date-$hash.$serial`` where ``$date`` -is the current date, ``$hash`` a hash of the configuration input file -and ``$serial`` a serial number which will be incremented if you run the -same invocation several times. - -Creating repositories from the manifests -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``pungi-createrepo-prototype`` script creates an installable -repository from the previously gathered manifests. You need to supply -the manifest directory locations from the previous step. - -:: - - "$WORKSPACE"/pungi/bin/pungi-createrepo-prototype \ - --arch x86_64 --target-dir "$COMPOSES" \ - --source-repo-from-path "$REPOS"/fedora-24-beta-x86_64 \ - --source-repo-from-path "$REPOS"/fedora-24-beta-src \ - --static-content-manifest "$COMPOSES/manifest_core-x86_64-$date-$hash.$serial/" \ - --extra-file "$COMPOSES/manifest_core-x86_64-$date-$hash.$serial/fm-metadata.yaml" - "$WORKSPACE"/pungi/bin/pungi-createrepo-prototype \ - --arch x86_64 --target-dir "$COMPOSES" \ - --source-repo-from-path "$REPOS"/fedora-24-shells-x86_64 \ - --source-repo-from-path "$REPOS"/fedora-24-shells-src \ - --source-repo-from-path "$REPOS"/fedora-24-beta-x86_64 \ - --source-repo-from-path "$REPOS"/fedora-24-beta-src \ - --static-content-manifest "$COMPOSES/manifest_shells-x86_64-$date-$hash.$serial/" \ - --extra-file "$COMPOSES/manifest_shells-x86_64-$date-$hash.$serial/fm-metadata.yaml" - -This may seem a bit roundabout since we started out with repositories, -but it will collect only the packages defined in the ``core.yaml`` and -``shells.yaml`` configuration files and their dependencies. - -At this point, the repositories should be at -``$COMPOSES/repo_{core,shells}-x86_64-$date-$hash.$serial`` - -Composing the module -~~~~~~~~~~~~~~~~~~~~ - -The ``pungi-compose-prototype`` script composes the module metadata from -the variants file -``"$WORKSPACE"/pungi-modularity/pungi-inputs/variants-fm.xml`` and the -previously created repositories. - -:: - - "$WORKSPACE"/pungi/bin/pungi-compose-prototype \ - --release fedora-24 --arch x86_64 --target-dir "$COMPOSES" \ - --variants-file "$WORKSPACE"/pungi-modularity/pungi-inputs/variants-fm.xml - -At this point the compose metadata directory should be at -``$COMPOSES/compose_fedora-24-$date.$serial``. diff --git a/source/development/prototype/container-in-openshift.rst b/source/development/prototype/container-in-openshift.rst deleted file mode 100644 index c55d5e7..0000000 --- a/source/development/prototype/container-in-openshift.rst +++ /dev/null @@ -1,365 +0,0 @@ -How to deploy container into OpenShift with ease -================================================ - -This chapter describes how to easily generate working OpenShift template -and what are useful OpenShift commands. - -OpenShift deployment possibilities ----------------------------------- - -OpenShift uses an abstraction called deployment to deploy applications. -A deployment could be basically explained as a load balancer for pods. - -A pod is the smallest deployable unit in OpenShift which is composed of -one or more containers. These containers share an IP address and -volumes, are always deployed together on a single host, and are scaled -together as a single unit. - -Scenario one Pod and two containers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This scenario is useful once you would like to have two containers, -where one is opened to anyone and second one is used as “hidden” -database. Like Internal register with hidden database. - -OpenShift linter command ------------------------- - -Once you wrote an OpenShift template, you would like to check it, -whether all fields are written properly. In order to verify the -template, **oc\_linter** or **oc lint** command would be welcome. - -Really basic YAML checker is *yamllint *, but it does not check -OpenShift specific things. - -I have already filed a RFE issue on GitHub `OpenShift Pull -Request `__. - -How to generate working template for OpenShift ----------------------------------------------- - -We need the templates, in order to test our containers on OpenShift. We -should simplify a way, for template generation. I have already filed a -RFE on OpenShift GitHub here `GitHub -RFE `__ These set of -scripts, can help the users for testing their containers together with -OpenShift. I don’t know if it is proper way, but for testing proposes it -works. - -Creating template with oc command -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In order to create a working template with **oc** command only two steps -are needed. - -- Run command: - -:: - - oc new-app - -- Run command: - -:: - - oc export dc/service_name> - -Can be taken from previous command. It is identical. - -Creating template by our tool -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Prerequisites -^^^^^^^^^^^^^ - -- Clone GitHub repository: `Petr Hracek - modularity\_tools `__ - - - The tools will be moved soon into repository `Pagure - modularity-tools `__ - -- Switch into your container directory. The directory has to contain - **Dockerfile** or like **Dockerfile.RHEL** and - **`openshift.yml `__** - - - Both files are important for proper template generation. - - If **Dockerfile** contains *ENV*, *VOLUMES* or *EXPOSE* - directives, they are add into OpenShift template. - -- Build your container image with **docker build ...** command. Do - **NOT** use '\_' in the image name. - -How to feed the template into OpenShift -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- From - `modularity\_tools `__ - repository, run command: - - - **get\_oc\_registry** gets your OpenShift docker-repository IP - address and stores it to file: **~/.config/openshift\_ip.ini** - -- In order to build OpenShift template from your container directory, - run command: - :: - - build_oc_template.py - - - In case of different Dockerfile name like **Dockerfile.RHEL** add - the option **--dockerfile Dockerfile.RHEL** - - Template is stored in **/tmp//openshift-template.yml** - -- For tagging your built image into OpenShift internal docker registry, - run command: - :: - - tag_into_oc_registry - - - The command adds the image into OpenShift internal docker registry - -- For adding the template into OpenShift, run command: - :: - - oc create -f /tmp//openshift-template.yml - -- The last step for deploying the **template** names as *IMAGE\_NAME* - is over OpenShift UI. By default, - :: - - "My Project" -> "Add to project" -> Select your template names as "IMAGE_NAME" in "Browsed Catalog" -> deploy it. - -- For getting template from running pod/deploymentconfig/is, run - command: - :: - - oc export {pod/dc/is}/|dc_name|is_name> > output.yml - - - Names are taken by commands - :: - - oc get {pod|dc|is} - -How to run container as a root under OpenShift ----------------------------------------------- - -Nowadays, OpenShift team provides a command, how to run container under -OpenShift with root privileges. - -:: - - oadm policy add-scc-tu-user anyuid system:serviceaccount::default - -where namespace is project name. Default one is *myproject*. - -The script -`add\_anyuid\_to\_project.sh `__ -does it automatically. Required argument is project name, like in our -case **myproject**. - -General commands with examples for using OpenShift --------------------------------------------------- - -All commands, in this section, should start with **sudo**. - -- To check whether OpenShift is running, run command: - -:: - - $ oc status - - In project My Project (myproject) on server https://10.200.136.26:8443 - dc/postfix-tls deploys istag/postfix-tls:latest - deployment #1 deployed 42 minutes ago - 1 pod - 2 warnings identified, use 'oc status -v' to see details. - -- Command for displaying all resources - pod\|deploymentconfigs\|imagestreams, run command: - -:: - - $ oc get - - $ oc get pod - NAME READY STATUS RESTARTS AGE - postfix-tls-1-kf0ud 1/1 Running 0 42m - $ oc get dc - NAME REVISION DESIRED CURRENT TRIGGERED BY - postfix-tls 1 1 1 image(postfix-tls:latest) - -- For getting what services are available on OpenShift, run command: - -:: - - $ oc get svc - -- For showing details of a specific resource, PODs, services, etc., run - command: - -:: - - oc describe pod|dc|is|svc - - $ oc describe pod postfix-tls-1-kf0ud - Name: postfix-tls-1-kf0ud - Namespace: myproject - Security Policy: anyuid - Node: 10.200.136.26/10.200.136.26 - Start Time: Fri, 20 Jan 2017 12:55:41 +0100 - Labels: deployment=postfix-tls-1 - deploymentconfig=postfix-tls - name=postfix-tls - Status: Running - IP: 172.17.0.3 - Controllers: ReplicationController/postfix-tls-1 - Containers: - postfix-tls: - Container ID: docker://6664727b761de3498eb863457aa4554820645b21dbea7e5b9a8a4d0382b22e7f - Image: postfix-tls - [..snip..] - 43m 43m 1 {kubelet 10.200.136.26} spec.containers{postfix-tls} Normal Created Created container with docker id 6664727b761d - 43m 43m 1 {kubelet 10.200.136.26} spec.containers{postfix-tls} Normal Started Started container with docker id 6664727b761d - -- Command for restarting POD is: - -:: - - oc scale --replicas=0 dc/ - -- For deploying template, run command: - -:: - - oc deploy --latest -n # default is myproject - -- For creating new POD, run command: - -:: - - oc new-app - -- For switching into system:admin, run command: - -:: - - oc login -u system:admin - -- For switching to developer mode, run command (default password is - developer): - -:: - - oc login -u developer - -- For modifying Security Content Constraints, switch to system:admin - and run command: - -:: - - oc get scc | jq …. | oc replace -f - - -Once it is done switch back to developer mode. - -- For getting Security Content Constraints, run command: - -:: - - oc get scc - NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP PRIORITY READONLYROOTFS VOLUMES - anyuid false [] MustRunAs RunAsAny RunAsAny RunAsAny 10 false [configMap downwardAPI emptyDir persistentVolumeClaim secret] - [..snip..] - privileged true [] RunAsAny RunAsAny RunAsAny RunAsAny false [*] - restricted false [] MustRunAs MustRunAsRange MustRunAs RunAsAny false [configMap downwardAPI emptyDir persistentVolumeClaim secret] - -- How to get YAML file from specific ImageStream - -:: - - oc get -o yaml is/ - -- How to get YAML file from specific container - -:: - - oc get -o yaml dc/ # name is taken from oc get dc - -- For deleting deployment - -:: - - oc delete dc/ # name is taken from oc get dc - -- For using container as root, run command: - -:: - - oadm policy add-scc-tu-user anyuid system:serviceaccount::default - -The command has now granted access for that namespace (only) to run pods -as the root UID. It is less secured than restricted but recommended if -you must run as root. It still does not allow privileged containers or -host namespaces (network, pid, ipc). It will only drop the mknod and -sys\_chroot caps (and not kill, setuid, setgid like restricted) - -How to debug service from OpenShift point of view -------------------------------------------------- - -This URL shows, how you are able to `debug a -service `__. -Basically it is a POD readiness issue. Therefore os get pod command and -the others mentioned below can help. - -Running your service in OpenShift environment ---------------------------------------------- - -OpenShift brings some security restrictions which make it tough to “just -run” your containerized services. This means that your service may run -easily in a docker container, but it may not be trivial to deploy it in -an OpenShift environment. Here is a list of sample steps to start the -process of integration: - -- If your container expects some mounts and you would like to perform - the mounting directly from host, here’s how to do it (by default this - is forbidden): - - - Login as system:admin - :: - - $ oc login -u system:admin - - - `Change restricted security context to allow host - mounts. `__ - - Login back as developer - :: - - $oc login -u developer - -- Here is `a simple, minimal pod - spec `__ - which takes your container image and runs bash inside so you can - quickly iterate. -- Run it. - :: - - oc create -f ./pod.yml - -- Attach to shell within the container - :: - - $ oc attach -t -i caching-dns-server - - - And now you can directly run the service and see what’s happening - -- In case something goes wrong, here’s how to get more info: - -:: - - $ oc logs caching-dns-server - $ oc describe pod caching-dns-server - -Links ------ - -- `Main OpenShift - documentation `__ -- `Introduction userns in Docker - engine `__ diff --git a/source/development/prototype/developer-notes.rst b/source/development/prototype/developer-notes.rst deleted file mode 100644 index 3062b19..0000000 --- a/source/development/prototype/developer-notes.rst +++ /dev/null @@ -1,391 +0,0 @@ -Developer Notes -=============== - -Abstract --------- - -The purpose of this document is to describe systems and services used -for generic `Modularity `__ development, research and -possibly future infrastructure deployments, and is intended as a -reference guide for the involved engineers and the so-called -*doers-of-things*. - -If you're new to Modularity, start with the following instead: - -- `Modularity `__ -- `Modularity/Getting\_involved `__ -- `Modularity/Infra `__ - -Systems -------- - -We use a number of systems for Modularity work. Some of those are -dedicated installations for our cause, some are generic and shared with -other Fedora initiatives. This section focuses on the former. - -dev.fed-mod.org -~~~~~~~~~~~~~~~ - -The main and currently the only dedicated system we have. - -This is an OpenStack instance running Fedora. In case the -``dev.fed-mod.org`` domain name no longer works, the IPv4 address is -``209.132.184.168``. We use a shared user account named ``fedora``. If -you need access, contact any of the current engineers and provide them -with your public SSH key. - -Automatic COPR rebuilds -^^^^^^^^^^^^^^^^^^^^^^^ - -We run automatic `COPR -rebuilds `__ -of certain modularity projects (fm, modulemd and modulemd-resolver) with -a cron job every 15 minutes. Edit ``~fedora/rebuild_packages.sh`` to add -yours. - -Automatic documentation rebuilds -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -We also run automatic `readthedocs.org `__ -documentation builds every 15 minutes for fm and modulemd with simple -cron jobs. - -Metadata service -^^^^^^^^^^^^^^^^ - -The metadata-service is deployed on this system and handles all -``^/fm/(.+)`` HTTP requests. This API isn't really defined yet. Browse -the project's sources to see how it works. - -Status reports and agile tools -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Our `Taiga status reports `__ are -also hosted on this sytem and regenerated every 15 minutes, again, with -a cron job. - -Other agile tools such as the Modularity Bot or sprint-tools for -Trello/Taiga synchronization are also hosted here. See -``~fedora/fm-trello-taiga-sync``, ``~fedora/sprint_tools`` and the -various related cron jobs. - -Webhosting in general -^^^^^^^^^^^^^^^^^^^^^ - -The host is running a generic httpd webserver. The configuration is kept -in ``/etc/httpd/conf.d/fm.conf`` and we use the default web root for our -stuff, ``/var/www/html``. For example, the experimental `modules -repository `__ is hosted there. - -Feel free to use this for whatever you need. However, keep in mind the -available disk space on this machine is fairly limited. - -Services --------- - -The high-level purpose of the majority of the services listed below is -described in the `Modularity/Infra `__ document. - -The input -^^^^^^^^^ - -Module input data consists of two main parts — the `module definition -file in the modulemd -format `__ -and the components, such as RPMs (coincidentally the only format we -currently support but expect that to change at some point in the -future). In Fedora, both the modulemd files and the components' SPEC -files are stored in dist-git and the associated ACLs are stored in pkgdb -(Package Database). - -We use namespaces to distinguish between modules and RPMs in those two -systems, aptly named ``modules`` and ``rpms``. - -.. raw:: mediawiki - - {{admon/note|How To Build A Module In Staging|A more up to date workflow is [[Modularity/HowToBuildAModuleInStaging|available]]}} - -dist-git -'''''''' - -For development and testing purposes we use the `staging dist-git -instance `__ which supports the -above mentioned namespaces. The recommended way to interact with -dist-git is using the ``fedpkg`` tool and configuring it to interact -with this instance. - -Once you have installed fedpkg, edit ``/etc/rpkg/fedpkg.conf`` (or -wherever your ``fedpkg.conf`` is located) to change all occurrences of -the default ``pkgs.fedoraproject.org`` to -``pkgs.stg.fedoraproject.org``. - -Here is an example: - -Install pag via ``$ sudo dnf install -y pag``. Once you have done this, -download and set up the custom rpkg repo: - -:: - - $ cd /tmp - $ pag clone karsten/rpkg - $ cd rpkg/src - -Next, set fpkg to point to stg/rida. To do this, add -``fedpkg-stage.conf`` to ``/tmp/rpkg`` and input the following content: - -:: - - [fedpkg] - lookaside = http://pkgs.stg.fedoraproject.org/repo/pkgs - lookasidehash = md5 - lookaside_cgi = https://pkgs.stg.fedoraproject.org/repo/pkgs/upload.cgi - gitbaseurl = ssh://%(user)s@pkgs.stg.fedoraproject.org/%(module)s - anongiturl = git://pkgs.stg.fedoraproject.org/%(module)s - tracbaseurl = https://%(user)s:%(password)s@fedorahosted.org/rel-eng/login/xmlrpc - branchre = f\d$|f\d\d$|el\d$|olpc\d$|master$ - kojiconfig = /etc/koji-stage.conf - build_client = koji - clone_config = - bz.default-tracker bugzilla.redhat.com - bz.default-product Fedora - bz.default-version rawhide - bz.default-component %(module)s - sendemail.to %(module)s-owner@fedoraproject.org - distgit_namespaced = True - ridaurl = https://dev.fed-mod.org:5000/rida - -Next, get the sources: - -:: - - $ fedpkg --config /etc/rpkg/fedpkg-stage.conf co modules/testmodule --anonymous - -Here, ``--anonymous`` is used in case you are not a packager on stg -envt. - -Now build: - -:: - - $ cd testmodule - $ fedpkg --config /etc/rpkg/fedpkg-stage.conf module-build - -Next, review the `BPO overview `__, then check -`the logs `__ (in case -that nothing happens). All repositories dumped by -`pungi-signed-repo `__ - -pkgdb -''''' - -A `staging pkgdb -instance `__ is also -available, storing the modules' ACL entries. - -Contact people with the admin ACL privileges for the given module for -commit access. Contact User:Ralph if you need a new module pkgdb entry & -dist-git repository. - -The message bus -^^^^^^^^^^^^^^^ - -We expect to use `fedmsg `__ -extensively for inter-component communication in all stages of module -build, testing and distribution. - -However, since we don't do any of those things yet, there's not much to -say about this. Read the upstream documentation to see what Fedora -Messaging is about. - -The module builder -^^^^^^^^^^^^^^^^^^ - -The module builder consists of three main components: the build -orchestrator, the koji build system and the pungi compose tool. The -basic, overly simplified idea of building modules is: - -#. The client (e.g. the module packager) contacts the orchestrator and - requests a build. -#. The orchestrator does all the heavy lifting -- prepares the buildroot - as a koji target, clones and builds all the components in koji in the - correct order, run CI checks for components and modules, tracks the - build states and rebuilds all dependant modules, if required. -#. After every module build, the orchestrator notifies pungi which in - turn creates module deliverables. - -There are still many open questions regarding this process. We will -fine-tune the details on the go. - -orchestrator -'''''''''''' - -The orchestrator is a service with a publicly available interface that -the clients can interact with, for example by issuing -``fedpkg module-build`` as a module packager, that *orchestrates* the -complete build of modules as noted above. Note the orchestrator doesn't -yet exist and although we don't have any specific design in mind, we -expect to have *something* ready in the near future. - -The orchestrator will emit fedmsg messages to interact with other -infrastructure components. It will work with PDC to both store (via -pdc-updater) and retrieve module dependency graphs. It will also require -its own database to track module build states. The public interface will -*not* be an XMLRPC. - -And the service will most likely be hosted on dev.fed-mod.org. - -koji -'''' - -The module RPM content will be built in koji. The current idea is to use -koji tags and targets to represent modules and tag inheritance to define -buildroots for RPM components. The orchestrator needs to be able to -manage these. - -We expect to use the `staging koji -instance `__ once we have a -usable and somewhat stable orchestrator. Until then, and to allow more -flexibility when developing against koji, we also have our own Fedora -24-based koji virtual machines you can play with. They're too large to -be shared on dev.fed-mod.org. Contact User:Psabata if you're interested -in getting them. - -pungi -''''' - -Once all the components in the module are built, the orchestrator will -signal pungi to create deliverables, such as RPM repositories or -container images, from the respective koji tags. pungi will also store -compose information in PDC (again, via pdc-updater), push the -deliverables to mirrors (either directly or via an update system) and -interact with various other currently nonexistent RCM tools. - -We're considering putting all or most of this functionality directly -into koji. - -The module knower -^^^^^^^^^^^^^^^^^ - -We would like to store certain practical bits about modules in PDC -(Product Definition Center) so that it can be easily viewed and queried -by both humans and other infrastructure tools. Specifically, we're -interested in two kinds of information: - -#. Source inter-modular dependencies — useful for tracking what modules - need to be rebuilt -#. Compose information — for listing contents of deliverables - -The first use case requires a new data type to be created in PDC. - -PDC -''' - -There's a `staging PDC instance `__ -available we hope to use later in the development cycle. - -We might set up our own instance on dev.fed-mod.org, if necessary. - -pdc-updater -''''''''''' - -pdc-updater is a simple, stateless service that responds to fedmsg -events and populates the PDC database. It already exists but needs some -patching to support the new data type we require. - -It is unclear whether a staging pdc-updater is available. We might as -well deploy our own on dev.fed-mod.org. - -The compose magic -^^^^^^^^^^^^^^^^^ - -Once built, modules could be composed into products such as *Fedora -Workstation*, *Fedora Server* or maybe even anything the user defines -either for integration QA purposes or building custom-tailored system -images for and by the end users. See the -`Modularity/Infra `__ document for more information -about these concepts. - -CaaS -'''' - -CaaS is not yet properly designed or implemented. Therefore we haven't -thought about deployment either. - -Pixie dust -'''''''''' - -The same for the so-called *Pixie dust*. - -The update system and distribution -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Both modules and module composes will need to be held by some -unspecified service before they get pushed to the master mirror. This is -necessary for two reasons: - -#. Human testing — verifying the builds and composes work fine beyond - the capabilities of our CI -#. Pushing mass rebuild results as one update — simply to avoid broken - states on the end users' systems - -This might be implemented by patching the current Fedora update system -and/or introducing yet another service/layer. We could use `the staging -bodhi instance `__ for -development. Note we have no staging mirrors or systems capable of -serving this kind of content at the moment. - -Miscellanous -^^^^^^^^^^^^ - -We also use a number of other systems and services. - -pagure.io -''''''''' - -All of our own code is hosted at `pagure.io `__. -Many of the projects grant commit access to the `@modularity -group `__. If you'd like to a part -of it, just ask any of the current members to add you. - -COPR -'''' - -There's a `@modularity project -space `__ -on COPR where we build & share our tools such as fm, modulemd or -modulemd-resolver. All `modularity-wg FAS group -members `__ -should have permissions to create new projects there. Contact any -modularity-wg sponsor or administrator to join the group. - -Jenkins -''''''' - -We have `a number of Fedora Infra Jenkins -jobs `__ -set up for the CI of our tooling. Contact User:James if you'd like -something added there. - -GitHub -'''''' - -In order to collect our changes and submit them as pull requests, we -have our own forks for some projects which are hosted on GitHub. For -grooming prior to submitting things upstream (e.g. reviewing and merging -pull requests from individual contributors against our forks), -contributors are added to the -```committers`` `__ -team in the -```fedora-modularity`` `__ -organization. Please ping `Nils Philippsen `__ -(`github `__) to get yourself added. - -Our forks: - -- `Product Definition - Center `__ - (`upstream `__) -- `pdc-updater `__ - (`upstream `__) - -Category:Modularity Category:Modularization diff --git a/source/development/prototype/getting-fedora-pkgs-for-testing.rst b/source/development/prototype/getting-fedora-pkgs-for-testing.rst deleted file mode 100644 index 74d83c2..0000000 --- a/source/development/prototype/getting-fedora-pkgs-for-testing.rst +++ /dev/null @@ -1,87 +0,0 @@ -Getting Fedora Packages for Testing -=================================== - -Getting Fedora Packages for Testing ------------------------------------ - -In order to build modules, you need component packages from which to -build them, and it's better to have them locally available. This page -describes where to download a set of packages which we use to build -modules in tests. - -Fedora 24 Beta -~~~~~~~~~~~~~~ - -As a base set of packages, we use the Beta of Fedora 24 at the moment. -Previously, we used the Alpha, but it shouldn't make much of a -difference unless you want to use the minimized packages below, then you -need to use the Alpha instead of the Beta, otherwise the non-minimized -packages may be preferred over the minimized ones because of their newer -NEVRA. - -You can download whole trees recursively from: - -- x86\_64 binary packages: - http://dl.fedoraproject.org/pub/fedora/linux/releases/test/24_Beta/Server/x86_64/os/ -- source packages: - http://dl.fedoraproject.org/pub/fedora/linux/releases/test/24_Beta/Server/source/tree/ - -Using ``wget``, this would download them to the current directory in -appropriate subdirectories: - -:: - - wget -r --no-parent -P fedora-24-beta-x86_64 --no-host-directories \ - --cut-dirs 9 http://dl.fedoraproject.org/pub/fedora/linux/releases/test/24_Beta/Server/x86_64/os/ - wget -r --no-parent -P fedora-24-beta-src --no-host-directories \ - --cut-dirs 9 http://dl.fedoraproject.org/pub/fedora/linux/releases/test/24_Beta/Server/source/tree/ - -"Esoteric" Shells -~~~~~~~~~~~~~~~~~ - -As an additional set of packages, we use a couple of shells. They aren't -grouped together like that ordinarily, so we download the packages -directly from koji: - -:: - - for arch in x86_64 src; do - d="fedora-24-shells-$arch" - (mkdir -p "$d" && cd "$d" && - for shell in aesh bash dash fish ksh mksh mosh tcsh yash zsh; do - _archspec="--arch $arch" - if [ "$arch" != "src" ]; then - _archspec="$_archspec --arch noarch" - fi - koji download-build --latestfrom=f24 $_archspec "$shell" - done) - done - -Afterwards, we need create the repository metadata: - -:: - - for d in fedora-24-shells-*; do - createrepo "$d" - done - -Minimized packages repositories -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -`Yaakov Selkowitz `__ hosts repositories with minimized -versions of some base packages on `fedorapeople.org -space `__: - -- Fedora 24 (roughly Alpha): - https://yselkowitz.fedorapeople.org/f24-minimization/ -- Rawhide: https://yselkowitz.fedorapeople.org/rawhide-minimization/ - -You can mirror them locally, e.g. using either or both of these -commands: - -:: - - wget -r --no-parent -P fedora-24-minimization --no-host-directories \ - --cut-dirs 1 https://yselkowitz.fedorapeople.org/f24-minimization/ - wget -r --no-parent -P rawhide-minimization --no-host-directories \ - --cut-dirs 1 https://yselkowitz.fedorapeople.org/rawhide-minimization/ diff --git a/source/development/prototype/getting-pungi.rst b/source/development/prototype/getting-pungi.rst deleted file mode 100644 index fa20a65..0000000 --- a/source/development/prototype/getting-pungi.rst +++ /dev/null @@ -1,133 +0,0 @@ -Getting Pungi (Modularity Style) -================================ - -Getting Pungi (Modularity Style) --------------------------------- - -Because Modularity is a work-in-progress, we don't commit code specific -to it to the main branches of projects, but keep them in separate -branches. To work with the modularized version of Pungi, this affects -the ``pungi`` and ``productmd`` projects (``modularity`` and -``modulemd`` are modularity-specific per se). - -This document will describe two slightly different ways to get the code -from the right branches: - -#. Downloading and running a script which will clone the necessary - source repositories into a newly created subdirectory, and print out - a sourcable shell snippet that sets up the environment to work with - it. -#. Cloning the repositories manually, and making them available in a - Python virtualenv. - -The quick way: using a script -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -#. Download - ````https://pagure.io/pungi-modularity/blob/master/f/checkout-modularity-into-pwd.sh`` `__ `__. -#. From a directory of your choice, execute this script. It will create - a subdirectory ``modularity`` into which the 4 projects are cloned. - It will also print out a shell snippet setting up the environment to - work there, which should be saved as e.g. ``$HOME/.modularity.sh``. -#. Run ``source $HOME/.modularity.sh`` to set up the environment. - -The slightly longer way: using a Python virtualenv -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -#. Download the projects. Go to the place where the projects should be - in your filesystem, we'll call it ``$WORKSPACE`` from here on, and - execute this: - :: - - git clone --branch modularity-prototype https://pagure.io/forks/lkocman/pungi.git pungi - git clone --branch modularity https://github.com/lkocman/productmd.git productmd - git clone https://pagure.io/pungi-modularity.git pungi-modularity - git clone https://pagure.io/modulemd.git modulemd - - Alternatively, you can use SSH URLs, of course, if you have an - account set up on these sites: - - :: - - git clone --branch modularity-prototype ssh://git@pagure.io/forks/lkocman/pungi.git pungi - git clone --branch modularity ssh://git@github.com/lkocman/productmd.git productmd - git clone ssh://git@pagure.io/pungi-modularity.git pungi-modularity - git clone ssh://git@pagure.io/modulemd.git modulemd - - This is largely what the ``checkout-modularity-into-pwd.sh`` does, - except that it creates a ``modularity`` subdirectory to hold the - repositories. - -#. Create the Python virtualenv. We'll use ``pungi-modularity`` as its - name in this document: - :: - - mkvirtualenv modularity - - The virtualenv will be activated immediately, later on you can switch - it on and off using ``workon modularity`` and ``deactivate``. - -#. *Optional:* Upgrade ``pip`` so it doesn't complain about the old - version being installed all the time: - :: - - pip install --upgrade pip - -#. Install some Python packages which aren't available from - `http://pypi.python.org `__ into the system, - using ``dnf`` or ``yum``: - :: - - koji - kobo-rpmlib - python2-dnf - yum - rpm-python - python-librepo - python-libcomps - python-hawkey - -#. Symlink the Python packages installed in the previous step from the - system into the virtualenv: - :: - - cdvirtualenv - cd lib/python2.7/site-packages - ln -s /usr/lib/python2.7/site-packages/koji koji - ln -s /usr/lib/python2.7/site-packages/kobo kobo - ln -s /usr/lib/python2.7/site-packages/dnf dnf - ln -s /usr/lib/python2.7/site-packages/rpmUtils rpmUtils - ln -s /usr/lib64/python2.7/site-packages/rpm rpm - ln -s /usr/lib64/python2.7/site-packages/librepo librepo - ln -s /usr/lib64/python2.7/site-packages/libcomps libcomps - ln -s /usr/lib64/python2.7/site-packages/hawkey hawkey - - If you use a 32bit system, the last four lines must be changed to: - - :: - - ln -s /usr/lib/python2.7/site-packages/rpm rpm - ln -s /usr/lib/python2.7/site-packages/librepo librepo - ln -s /usr/lib/python2.7/site-packages/libcomps libcomps - ln -s /usr/lib/python2.7/site-packages/hawkey hawkey - -#. Install Python dependencies needed by ``dnf``, ``koji`` and ``pungi`` - into the virtualenv: - :: - - pip install iniparse lxml pygpgme pyliblzma pyOpenSSL python-krbV - -#. Make the virtualenv use the checked out source of the projects (and - install dependencies): - :: - - cd "$WORKSPACE" - for proj in modulemd productmd pungi; do - (cd "$proj" && python setup.py develop) - done - -At this point, everything should be in set up to run the -``pungi/bin/pungi-*-prototype`` scripts. You'll need to download some -source and binary packages for them to work on, we mainly use the Server -edition of Fedora (currently the alphas and betas of version 24). Review -the ``pungi-modularity/example-*.sh`` scripts for command syntax. diff --git a/source/development/prototype/local-pdc.rst b/source/development/prototype/local-pdc.rst deleted file mode 100644 index 4e9c27d..0000000 --- a/source/development/prototype/local-pdc.rst +++ /dev/null @@ -1,222 +0,0 @@ -Setting up PDC locally -====================== - -Setting up PDC locally for Modularity development -------------------------------------------------- - -To develop tooling for Modularity and test it, we want to have a local -instance of PDC installed. The `upstream -documentation `__ -describes a couple of ways of doing that, this document describes how it -can be done `using a Python -virtualenv `__, -with some notable differences: - -- The virtualenv won't have system-wide packages enabled. This is so - that OS updates or upgrades have a reduced impact on the development - environment, "reduced" because neither ``koji`` nor ``rpm`` Python - modules lend themselves well to be installed into a virtualenv, so - for the sake of simplicity we'll symlink the system-wide installed - modules. -- We'll use the ``modularity`` branch of ``productmd``, not the version - that's `available on - PyPI `__. - -Walkthrough -~~~~~~~~~~~ - -#. Install the necessary system-wide packages (if missing) using - ``dnf``, ``yum`` or similar: - :: - - koji - python-virtualenvwrapper - rpm-python - - If you just installed ``python-virtualenvwrapper``, you need to - restart the shell for it to take effect, i.e. install the - ``mkvirtualenv`` and other shell functions, or - ``source /etc/profile.d/virtualenvwrapper.sh`` manually. - - The patternfly1 package is required by PDC but isn't part of the - Fedora distribution. You'll need to add a copr repository with - - :: - - dnf copr enable patternfly/patternfly1 - - before you can install it with - - :: - - dnf install patternfly1 - - Some development packages of libraries are needed for the Python - module dependencies to be installed as well. Here's the list of - development packages needed on Fedora 23: - - :: - - cyrus-sasl-devel - glibc-devel - graphviz-devel - keyutils-libs-devel - krb5-devel - libcom_err-devel - libffi-devel - libgcrypt-devel - libgpg-error-devel - libselinux-devel - libxml2-devel - libxslt-devel - nspr-devel - nss-devel - nss-softokn-freebl-devel - nss-util-devel - openldap-devel - openssl-devel - pcre-devel - python-devel - xz-devel - zlib-devel - -#. Create a workspace directory where all the checked out code - repositories live, if you haven't done so already. We'll refer to - that as ``$WORKSPACE``. -#. Make a virtualenv ``modularity`` for PDC development. This is one - point where we deviate from the upstream docs: - :: - - mkvirtualenv modularity - - This activates the ``modularity`` virtualenv right away, leave it - using the ``deactivate`` command, and activate it again with - ``workon modularity`` later on. - -#. *Optional:* Upgrade ``pip`` so it doesn't complain about the old - version being installed all the time: - :: - - pip install --upgrade pip - -#. Make the ``modularity`` branch of ``productmd`` available in the - virtualenv. - - #. Clone the repository into your workspace: - :: - - cd "$WORKSPACE" - git clone --branch modularity https://github.com/lkocman/productmd.git - - #. Make the virtualenv use the checked out source: - :: - - cd productmd - python setup.py develop - -#. Symlink the ``koji`` and ``rpm`` Python packages from the system into - the virtualenv: - :: - - cdvirtualenv - cd lib/python2.7/site-packages - ln -s /usr/lib/python2.7/site-packages/koji koji - ln -s /usr/lib64/python2.7/site-packages/rpm rpm - - If you use a 32bit system, the last line must be changed to: - - :: - - ln -s /usr/lib/python2.7/site-packages/rpm rpm - -#. Install Python dependencies needed by ``koji`` into the virtualenv: - :: - - pip install pyOpenSSL python-krbV - -#. Get PDC and set it up. - - #. Clone the PDC repository into your workspace: - :: - - cd "$WORKSPACE" - git clone https://github.com/product-definition-center/product-definition-center.git - - #. Install the dependencies needed for PDC development: - :: - - cd product-definition-center - pip install -r requirements/devel.txt - - #. Create the database and schema inside, this command also would - migrate the schema to a new version if subsequent changes in PDC - make this necessary: - :: - - ./manage.py migrate - - #. Create a superuser for PDC: - :: - - ./manage.py createsuperuser - - This will ask you for a user name, email address and password. - Going forward, we'll assume the user name is ``superuser``. - - #. Set up local configuration so testing doesn't require - authentication etc. for what would be privileged operations in a - productive environment. - - #. Copy the local configuration file from the template: - :: - - cd pdc - cp settings_local.py.dist settings_local.py - - #. Make some changes in ``settings_local.py``: - - #. Enable debugging, change this line: - :: - - DEBUG = False - - to this one: - - :: - - DEBUG = True - - #. Don't restrict connecting to PDC (it listens on the loopback - device only, anyway), comment out the ``ALLOWED_HOSTS`` - line: - :: - - #ALLOWED_HOSTS = [...] - - #. Make unauthenticated access user the ``superuser`` account, - forego permissions checking. Add these lines to the end of - the file: - :: - - # mock login for debugging - DEBUG_USER = "superuser" - - DISABLE_RESOURCE_PERMISSION_CHECK = True - - del get_setting('REST_FRAMEWORK')['DEFAULT_PERMISSION_CLASSES'] - - #. Start up the local PDC instance: - :: - - cd "$WORKSPACE"/product-definition-center - ./manage.py runserver - - #. Manually create the ``module`` Variant Type in the PDC interface. - - #. Go to `http://127.0.0.1:8000 `__ with - your web browser, then go the administrative interface - ``👤 superuser`` → ``PDC Administration interface``. - #. Locate ``Release`` → ``Variant types``, click on ``+ Add``, - enter ``module`` as the name, click on ``Save``. - -At this point, PDC should be set up and ready for working with modules. From ad892e7769e0587597f86678f5d00dfaef47bc12 Mon Sep 17 00:00:00 2001 From: asamalik Date: May 29 2017 07:51:20 +0000 Subject: [PATCH 6/8] reflect the previous changes in the navigation --- diff --git a/source/docs.rst b/source/docs.rst index 9ba6bd9..5777d96 100644 --- a/source/docs.rst +++ b/source/docs.rst @@ -12,38 +12,32 @@ Contents: .. toctree:: - :maxdepth: 2 - :caption: Architecture + :caption: Design - architecture/building-naming - architecture/constructing - architecture/versioning - architecture/infrastructure - architecture/pdc-changes + design/building-naming + design/constructing + design/versioning .. toctree:: - :maxdepth: 2 :caption: Development development/getting-started + development/resources + development/resources2 development/building-modules - development/integration - development/prototype .. toctree:: - :maxdepth: 1 - :caption: Usability + :caption: Infrastructure - usability/user-stories + infrastructure/infrastructure .. toctree:: - :maxdepth: 1 - :caption: Misc + :caption: Usability - misc/faq + usability/user-stories From 0df484ea831ba9e50d3cd369f30855066ad34425 Mon Sep 17 00:00:00 2001 From: asamalik Date: May 29 2017 07:51:39 +0000 Subject: [PATCH 7/8] remove random files that are not needed --- diff --git a/source/first-one.rst b/source/first-one.rst deleted file mode 100644 index 228b16c..0000000 --- a/source/first-one.rst +++ /dev/null @@ -1,16 +0,0 @@ -First One -========= - -Lorem ipsum dolor sit amet, consectetur adipiscing elit. Fusce elementum orci erat, quis molestie lectus vestibulum eu. Aliquam luctus scelerisque urna, elementum vulputate massa rutrum id. Sed auctor, metus ut imperdiet congue, arcu velit commodo risus, nec aliquet ante nibh id tellus. Cras vehicula, orci ut auctor consectetur, nunc nibh porttitor lectus, vel iaculis dui lacus ultricies sem. Donec sed risus augue. Vivamus neque urna, finibus nec egestas non, feugiat at dolor. Duis hendrerit diam in nulla tempor, sed rutrum sem tincidunt. Etiam vel tristique dui. Nunc varius rutrum odio, sed porta purus. Quisque quis tortor tempus, tempus orci at, convallis sem. Donec pharetra augue eu nulla semper luctus. Aliquam quis purus sit amet augue commodo rutrum eget a massa. Maecenas rutrum aliquet gravida. - -.. code-block:: python - - def some_function(): - interesting = False - print 'This line is highlighted.' - print 'This one is not...' - print '...but this one is.' - -Another heading ---------------- -another text diff --git a/source/three-layer-arch.png b/source/three-layer-arch.png deleted file mode 100644 index 07da0cd..0000000 Binary files a/source/three-layer-arch.png and /dev/null differ From 9385e554f57fdd27c3179173061bb054455bc442 Mon Sep 17 00:00:00 2001 From: asamalik Date: May 29 2017 07:51:50 +0000 Subject: [PATCH 8/8] fix filename of get-involved and move personas --- diff --git a/source/development/get-involved.rst b/source/development/get-involved.rst new file mode 100644 index 0000000..cb46a9c --- /dev/null +++ b/source/development/get-involved.rst @@ -0,0 +1,104 @@ +Get Involved +============ + +Steps to get involved +--------------------- + +.. raw:: mediawiki + + {{Team contact|Modularity WG|devel|#fedora-modularity}} + +To get involved with the Modularity Working Group, please follow the +following steps for becoming a part of the team. + +#. Read this wiki page and understand the development methods and + technical details. +#. Get in touch with us on the `devel mailing + list `__ + and our IRC channel, + `#fedora-modularity `__. +#. Attend one of our `weekly + meetings `__ and + introduce yourself. +#. Become familiar with the Agile project management and create an + account on the `Fedora + Taiga `__. +#. Create an account on `Pagure `__ and file an + issue in the `Fedora @modularity + group `__ requesting membership. + +Get in touch +------------ + +There's no dedicated mailing list yet and everything regarding this +topic should be discussed on the general `Fedora Development +list `__. +Most of us also hang out on the #fedora-modularity channel on Freenode. + +Planning +-------- + +Formal meetings are held once a week by the Modularity Working Group. +See the +`Fedocal `__ to +find out when the group is meeting. + +Development Method +------------------ + +We will use `agile software +development `__ +methods for Modularity, more specifically: a hybrid of +`Scrum `__ +and `Kanban `__ +adapted to the constraints we have in Fedora. For instance, not all +contributors can commit to be involved like a regular, full-time +employee, meaning that rigid use of 2-week-long Scrum Sprints can be an +obstacle to participating for some people. + +If you're not familiar with agile development or the methods we use, +here are some links to get you started: + +- `Learn Scrum in 8 + minutes `__ +- `Kanban applied to + Scrum `__ + +Agile development methods often come with their own lingo that can be +confusing to the "uninitiated"—like +`"epic" `__, +`"sprint" `__ (or +`"iteration" `__), +`"spike" `__. Many of the terms used +are explained over at the `Agile +Dictionary `__. + +We manage the project using `Taiga `__ on Fedora +Infrastructure, `one +instance `__ for +individual sprint work items and +`another `__ +for higher-level stuff ("epics"). + +Technical details +----------------- + +Architecture +~~~~~~~~~~~~ + +Refer to :doc:`/architecture/infrastructure` + +Code repositories +~~~~~~~~~~~~~~~~~ + +We currently host all of our code at `Pagure `__ — +the infrastructure, the client-side tooling, metadata specification +drafts and even a couple of proof-of-concept modules. Repositories +typically start with the *fm-* prefix and are open to all members of the +`Pagure @modularity group `__. + +Services and tools +~~~~~~~~~~~~~~~~~~ + +Refer to :doc:`../prototype/developer-notes` + diff --git a/source/development/getting-started.rst b/source/development/getting-started.rst deleted file mode 100644 index cb46a9c..0000000 --- a/source/development/getting-started.rst +++ /dev/null @@ -1,104 +0,0 @@ -Get Involved -============ - -Steps to get involved ---------------------- - -.. raw:: mediawiki - - {{Team contact|Modularity WG|devel|#fedora-modularity}} - -To get involved with the Modularity Working Group, please follow the -following steps for becoming a part of the team. - -#. Read this wiki page and understand the development methods and - technical details. -#. Get in touch with us on the `devel mailing - list `__ - and our IRC channel, - `#fedora-modularity `__. -#. Attend one of our `weekly - meetings `__ and - introduce yourself. -#. Become familiar with the Agile project management and create an - account on the `Fedora - Taiga `__. -#. Create an account on `Pagure `__ and file an - issue in the `Fedora @modularity - group `__ requesting membership. - -Get in touch ------------- - -There's no dedicated mailing list yet and everything regarding this -topic should be discussed on the general `Fedora Development -list `__. -Most of us also hang out on the #fedora-modularity channel on Freenode. - -Planning --------- - -Formal meetings are held once a week by the Modularity Working Group. -See the -`Fedocal `__ to -find out when the group is meeting. - -Development Method ------------------- - -We will use `agile software -development `__ -methods for Modularity, more specifically: a hybrid of -`Scrum `__ -and `Kanban `__ -adapted to the constraints we have in Fedora. For instance, not all -contributors can commit to be involved like a regular, full-time -employee, meaning that rigid use of 2-week-long Scrum Sprints can be an -obstacle to participating for some people. - -If you're not familiar with agile development or the methods we use, -here are some links to get you started: - -- `Learn Scrum in 8 - minutes `__ -- `Kanban applied to - Scrum `__ - -Agile development methods often come with their own lingo that can be -confusing to the "uninitiated"—like -`"epic" `__, -`"sprint" `__ (or -`"iteration" `__), -`"spike" `__. Many of the terms used -are explained over at the `Agile -Dictionary `__. - -We manage the project using `Taiga `__ on Fedora -Infrastructure, `one -instance `__ for -individual sprint work items and -`another `__ -for higher-level stuff ("epics"). - -Technical details ------------------ - -Architecture -~~~~~~~~~~~~ - -Refer to :doc:`/architecture/infrastructure` - -Code repositories -~~~~~~~~~~~~~~~~~ - -We currently host all of our code at `Pagure `__ — -the infrastructure, the client-side tooling, metadata specification -drafts and even a couple of proof-of-concept modules. Repositories -typically start with the *fm-* prefix and are open to all members of the -`Pagure @modularity group `__. - -Services and tools -~~~~~~~~~~~~~~~~~~ - -Refer to :doc:`../prototype/developer-notes` - diff --git a/source/docs.rst b/source/docs.rst index 5777d96..8d314d5 100644 --- a/source/docs.rst +++ b/source/docs.rst @@ -22,9 +22,8 @@ Contents: .. toctree:: :caption: Development - development/getting-started + development/get-involved development/resources - development/resources2 development/building-modules @@ -37,7 +36,7 @@ Contents: .. toctree:: :caption: Usability - usability/user-stories + usability/personas diff --git a/source/usability/personas.rst b/source/usability/personas.rst new file mode 100644 index 0000000..31655b5 --- /dev/null +++ b/source/usability/personas.rst @@ -0,0 +1,636 @@ +Personas +======== + +The information below relates to personas developed for the use of the +Modularity working group. These personas will be continually updated as +more information is gathered from existing users that fit these roles. +Also, any information provided by community members or working group +team members will be incorporated as well. + +What is a Persona +----------------- + +Personas are fictional characters created to represent the different +user types that might interact with a product in different ways. They +are not market segments but should be thought of as user archetypes. + +Personas are useful in considering the goals, desires and limitations of +users in order to guide decisions about the product. They should be +based on user research and should by 1-2 page descriptions that include +behavior patterns, goals, skills, attitudes and environment details. + +There should be a primary persona defined along with secondary personas. +More than one can be primary, but no more than 3 in order to keep the +team focused in their design and implementation work. + +Benefits of Personas +-------------------- + +Common benefits include: + +- Help the team share a common understanding of their various users + groups and audiences +- Proposed designs can be guided by how well they meet the needs of + certain personas +- Features can be prioritized based on how well they address the needs + of one or more personas +- Provides a human “face” to the user that can help create empathy for + the people represented + +Personas Types +-------------- + +The following have been identified as the first set of personas: + +- Student +- Educator +- Researcher +- IT Executive +- IT Manager +- Business Analyst +- Enterprise System Admin +- Enterprise System Architect +- Enterprise Developer +- Enterprise DevOps and Systems of Differentiation Developer +- Enterprise DevOps Developer +- Enterprise and Independent Software Vendor Developer - Systems of + Innovation +- Traditional Independent Software Vendor +- Next Gen Independent Software Vendor +- Business Operations +- Product Manager + +.. raw:: mediawiki + + {{admon/note|Primary personas have not yet been identified| Once identified, they will be highlighted as primary.}} + +Detailed Information +-------------------- + +Student +~~~~~~~ + +**Background** + +- Engineering or Computer Science student +- Multiple developer environment experience +- Plays 3D games + +**Primary Goals** + +- Needs a personal system for software classwork and personal projects +- Software class work may require particular tool chain versions +- To try out new versions of open source applications when released +- Play computer games +- Listen to music +- Interact with friends and family through social media primarily + through cellphone + +Educator +~~~~~~~~ + +**Background** + +- B.A. in Education +- 15 years in academia with different roles related to teaching and + administration + +**Primary Goals** + +- Create lesson plans and make available to students, i.e., blackboard +- Keep up to date with entering grades into on-line systems for + transparency with students +- Keep up with responding to emails from colleagues and students +- Create meaningful and engaging presentations + +Researcher +~~~~~~~~~~ + +TBD + +IT Executive - David +~~~~~~~~~~~~~~~~~~~~ + +*Aligns IT initiatives with business goals* + +**Quote** + +“In this competitive market, we need to be fast, efficient, and use +technologies that attract developers to the company.” + +**Background** + +- BS in Computer Science +- 30 years experience +- 15 years of management experience +- Advanced analytical skills + +**Primary Goals** + +- Improve IT operations, system performance, and risk management +- Implements new systems/architecture +- Cultivates IT and business partnerships +- Sell IT solutions internally +- Selects vendors that can meet his/her challenges. Convinces c-suite + peers to adopt the chosen vendor. + +**Pain Points** + +- Provide security while focusing on efficiency, mobility, innovation, + and downtime +- Doing more with less budget, skills, staff, financial transparency + and time +- Reducing server footprint while expanding services + +IT Manager - Stan +~~~~~~~~~~~~~~~~~ + +*Manages team that oversees infrastructure components* + +**Background** + +- BS in Computer Science +- 15 years experience + +**Primary Goals** + +- Determine execution plans for specific IT objectives +- Select solutions for the objectives and budgets specified by IT + Executive +- Make decisions for smaller, tactical purchases and influence larger + purchases +- Ensures reliable, 24x7 operations of existing infrastructures +- Identifies paths to resolution when issues arise + +**Pain Points** + +- Doing more with less resources +- Getting the blame when things go wrong +- Handling staffing issues (not having people trained in the skills and + capabilities required) +- Meeting regulatory and security compliance + +**A Day in the Life** + +Stan starts his day before arriving in the office by checking email and +making note of any issues that have arisen over night. Anything that +needs his immediate attention can be handled through email and requests +to others to get started on solutions. After getting to the office, he +touch base with his manager to make sure we are in sync for the day. +Then he has a quick meeting with his team to make sure they know the +goals and issues to be addressed that have come up over night. Most of +the rest of his day is spent putting out fires, dealing with email and +sitting in project meetings. + +Business Analyst - Karen +~~~~~~~~~~~~~~~~~~~~~~~~ + +*Identifies the need for changing how the company works* + +**Quote** + +“In my job I need to be a master multi-tasker. I love that there is +constant change and that every day is different.” + +**Background** + +- BS in Business or Management +- 10 years experience +- Strong written and communication skills +- Strong analytical and conceptual skills +- Working knowledge of software development processes + +**Primary Goals** + +- Be a change agent for the company +- Successfully broker change between company stakeholders and IT +- Be on top of industry trends to know when change may be needed + +**Pain Points** + +- Managing urgencies, crises, and timelines +- Expectation that better outcomes can be achieved faster with new + analysis and technology +- Difficult to achieve organizational and cultural changes necessary to + implement some solutions +- An ever increasing pace of change in the marketplace + +**A Day in the Life** + +Karen starts her day by checking email and reading the latest articles +from her feeds. Ideally, she can spend a few hours in the morning going +through industry trend documents and identifying potential items that +can be leveraged. Most days however, get full of stakeholder meetings +and project meetings where she helps to keep things on track and making +sure the business objectives are still going to be achieved. + +In the afternoon, her goals are to write up proposal and plans. She also +prefers to spend at least one hour looking through the data to support +upcoming proposals and making sure that they still make sense for the +business. + +Enterprise System Administrator - Yvonne +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Ensure current IT infrastructure works seamlessly* + +**Quote** + +“What people don’t realize is that being an System Administrator is a +24/7 job. There is a constant stream of emails with requests and issues +from all fronts and you really have to know how to prioritize and triage +effectively to be successful at this job. It is definitely not for +everyone, but I enjoy the pace and the constant challenge.” + +**Background** + +- B.S. in Computer Science +- 15 years experience +- Shell script expert +- Good at putting all the parts together + +**Primary Goals** + +- Suggest useful, secure and flexible tools that can be adopted by the + organization to achieve day-to-day tasks +- Make sure upgrades and new deployments are smooth with little to no + downtime +- Be a reliable source of information for co-workers and users +- Unified management of server resources +- Ability to understand resource usage across server inventory to + identify underutilized resources + +**Pain Points** + +- Constant interruptions +- Having to be available 24/7 depending on the issue +- Stress due to picking the right technology +- Proliferation of various management console interfaces to have to + manage + +**A Day in the Life** + +Yvonne starts her day by checking email before she even gets out of bed +to make sure there were no failures overnight and there are no fires +that can’t wait until she gets to the office. Once satisfied that all is +well, she gets ready, takes care of her family and heads into the +office. Upon arriving at the office, she makes a pot of coffee and again +checks her email. Her priorities include correcting any issues that are +stopping others from working and escalating anything that she cannot +work on herself. Yvonne then checks in with her supervisor on the day's +activities, checks on any hardware deliveries and communicates with +co-workers on overall system status. + +Enterprise System Architect - Kevin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Links the business mission, strategy and processes to the IT strategy* + +**Quote** + +“I love my job...I am one of a few who get to work strategically with +Leadership and still maintain a technological connection to the product. +Exploring new technologies and innovations and figuring out how our +product can benefit is one of my favorite aspects of the job. It can be +very stressful and demanding, but that is how I know I am making a +difference to my team, product and company.” + +**Background** + +- B.S. in Computer Science +- 25 years experience +- Expert skills in software design and testing; Agile +- Knows Java, JavaScript, C, C++, HTML5 and CSS +- Uses Eclipse IDE + +**Primary Goals** + +- To set strategic goals through collaboration with senior leadership. +- Ensure the overall integrity of the application. +- Document and communicate industry trends and a roadmap for + implementing. +- High-level understanding of the entire system. +- Responsible for selecting systems and specifying deployment + lifecycles and system configuration. + +**Pain Points** + +- Constantly need to sell ideas that may result in deadline changes. +- Not enough resources to keep things as current as needed. +- No authority to make things happen; always negotiating between + timeline, budget and technology. +- Spend a lot of time documenting requirements, needs and trends in + order to communicate to all stakeholders. + +**A Day in the Life** + +Kevin starts his day by catching up on his feeds related to security, +serviceability and other industry trends to make sure he is aware of new +ideas. He then has his daily standups with the dev teams and meets one +on one with team members that have issues or that he needs to ask +questions of. + +Kevin spends the majority of his afternoon documenting the roadmap and +evaluating options to determine if they meet feature requests/needs and +how they fit into his strategies to meet enterprise compliance. Once a +week, he meets with Senior Leadership to report on progress as well as +identify any recommended changes to direction that he has. If leadership +has issues with his recommendations, he needs to wrap back around and +come up with other alternatives or bolster his arguments for proceeding +with his original recommendation. + +Enterprise Developer - Rajiv +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Design and implement code* + +**Background** + +- BS in Computer Science +- 8 years of experience +- Java, Javascript, CSS, MVC frameworks, github + +**Primary Goals** + +- To design and implement code +- To leverage the best tools for the job +- To complete tasks in as efficient and timely manner as possible + +**Pain Points** + +- Not having enough time to complete tasks required +- Being restricted by existing resources/tools that don’t enable the + ability to deliver new functionality +- Too much bureaucracy in the organization + +**A Day in the Life** + +When Rajiv gets to the office he spends a few minutes catching up on his +feeds and checking email. Then he participates in his team’s standup +meeting and sets up some time with a coworker who he needs help from to +complete his story. Until he can meet with him, Rajiv works on fixing +bugs that have been found. Late morning, Rajiv has two project meetings +to attend and then has his quick meeting to unblock his story. + +After lunch, Rajiv finishes his story and starts on the next one in his +backlog. Usually there are additional bugs that get filed throughout the +afternoon so he reserves the last hour and half to work on those before +heading home. + +Enterprise DevOps Developer - Devin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Continuous integration/Continuous delivery* + +**Quote** + +“Some folks think I’m a sys admin, others think I’m a coder. The truth +is, I’m both and a lot more!” + +**Background** + +- B.S. in Computer Science +- 10 years experience +- Skilled in IT operations and software testing +- Specializes in Java as well as Python and Perl +- Uses Jenkins and Ansible + +**Primary Goals** + +- Ensuring that all applications migrate smoothly throughout the entire + lifecycle, from the development process to production support. +- Ensuring that the automation process works efficiently among the + team: developers who implement features, system administrators who + manage access control, and operators who monitor the applications and + servers. +- Identifying the root cause of problems in test, staging and + production environments, as well as the infrastructure. + +**Pain Points** + +- Multiple tools, integration points, incompatibilities, and problems + with PaaS and IaaS create many obstacles. +- When a build cannot pass the tests, a lot of time is spent looking + for the team members who are responsible for the build or the + specific feature that is causing a problem. This is exacerbated when + he does not know the responsibilities of certain people on the team. + +**A Day in the Life** + +Devin starts his workday by checking on the status of the overnight +builds. One build failed miserably, so his first order of business is to +find out why. After he figures out which component appeared to cause the +problem, he opens a Severity 1 ticket on that component so the +responsible developer will be notified immediately. Devin gets an +automated out-of-office notification from that developer, so he has to +inspect and fix the code himself, and then kick-off the test of the +build again. + +Because of the unexpected issue, Devin arrives late to the weekly DevOps +meeting with his team of developers, quality engineers, and product +managers. The main topic is the decreasing availability and performance +of the deployment system, so they need to plan for a more scalable and +performant solution. After this, he completes his daily status report +about the CI/CD pipeline to the engineering management team. Once that +is sent, Devin, intermittently coordinates work among his team members +in between the moments he gets to modify the scripts that orchestrate +the complete deployment of development, QA, and production environments. + +Enterprise DevOps Developer and Systems of Differentiation Developer - Ryan +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Background** + +- B.S. in Computer Science +- 10 years experience +- Skilled in IT operations and software testing +- Specializes in Java as well as Python and Perl +- Uses Jenkins and Ansible + +**Primary Goals** + +- To design and implement code +- To leverage the best tools for the job +- To complete tasks in as efficient and timely manner as possible + +**Pain Points** + +- Not having enough time to complete tasks required +- Being restricted by existing resources/tools that don’t enable him to + deliver new functionality +- Too much bureaucracy in the organization + +Enterprise and ISV Developer - Systems of Innovation - Ben +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Design and Implement Code* + +**Primary Goals** + +- To design and implement code +- To leverage the best tools for the job +- To complete tasks in as efficient and timely manner as possible + +**Pain Points** + +- Not having enough time to complete tasks required +- Being restricted by existing resources/tools that don’t enable him to + deliver new functionality +- Too much bureaucracy in the organization + +Traditional ISV +~~~~~~~~~~~~~~~ + +**Primary Goals** + +- To build, develop and sell consumer or enterprise software +- Partner with platform providers to sell more software +- Ensure customers are satisfied and their needs are being met +- Work with multiple teams including partner, customer and internal to + address issues and concerns + +**Pain Points** + +- Different customers have different requirements that can be + conflicting +- There is always something that needs to be corrected +- Can’t ship on a quick enough schedule to satisfy the customer base + +Next-Gen ISV +~~~~~~~~~~~~ + +**Primary Goals** + +- To build, develop and sell consumer or enterprise software +- Partner with platform providers to sell more software +- Ensure customers are satisfied and their needs are being met +- Work with multiple teams including partner, customer and internal to + address issues and concerns +- To ship apps faster +- To leverage containers and other emerging technologies as soon as + possible +- Multi-modal development +- SaaS model first, then deploy in organization + +**Pain Points** + +- Different customers have different requirements that can be + conflicting +- There is always something that needs to be corrected + +Business Operations - Isaac +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Management of the Operations Team* + +**Background** + +- B.S. in Information Systems +- 8 years experience - Cloud Services (IaaS, PaaS, SaaS) +- Specializes in Java, Python; knows C, C++ +- Expert knowledge of UNIX, LINUX and Windows + +**Primary Goals** + +- Ensuring successful day-to-day operations of the technology + infrastructure and that all Service Level Agreements (SLA) with + customers are met, and creating plans and providing support for + instances when an SLA is compromised. +- Quickly receiving and reviewing real-time reports of critical SLA + criteria: availability, system response times, security/privacy, + disaster recovery time, call center resolution time, dispute + mediation, and change management. +- Scheduling, budgeting, and staffing system maintenance and upgrades + to ensure maximum operational performance. + +**Pain Points** + +- Technological and financial limitations make it very challenging to + migrate legacy applications to the cloud + +**A Day in the Life** + +The first order of business for Isaac is to review the reports of system +operations from the previous day. He’s especially interested in the +availability numbers because a big software upgrade was applied to a few +servers overnight. Fortunately, system availability was not affected, +and the upgrade should improve the numbers for system response time, so +it was a win-win for customers and Isaac, who constantly keeps his eye +on the SLA commitments. + +Afterwards, Isaac attends a meeting with his staff to discuss the exit +strategy plan for a customer who will be leaving their service at the +end of the month. A smooth transition is a must. + +Isaac then has a series of meetings with the technical leads on his +staff to discuss proposals for rolling out new server hardware during +the next quarter. He needs to balance capacity and performance with the +tight budget he’s been given for expansion, so he needs lots of details +so he can evaluate and select from the multiple options on the table. + +Product Manager - Pablo +~~~~~~~~~~~~~~~~~~~~~~~ + +*Customer and industry liaison to the product and the company* + +**Quote** + +“A lot of my time is spent gathering resources, coordinating various +teams to agree on decisions or work to achieve customer goals. I need to +spend more time distilling down ideas and driving agreement across +disparate teams. My role is about working with other to build the grand +plan. Improving communication, clarity on commitment and direction, +would result in more timely decision making.” + +**Background** + +- B.S. in Computer Science +- 5+ years experience, including project management + software + development environment +- Deliver multiple projects, cross-functionally +- Expert planner with business process definition +- Influencer in org up to SVP with more experience + +**Primary Goals** + +- Understand product’s business + UX needs +- Coordinate design, development, delivery, documentation, launch, + sales enablement + requests, service, integration, support. +- To work with stakeholders, UX + Dev to transform requirements into + experiences to deliver + assure all are met. +- Managing market requirements, trends, competitive landscapes, + constraints, cost, models, time, scope, risks and P+L to the + executive team. +- Manage customer partnerships + grow business opportunities +- Understanding company's strategic goals + reflecting those in the + product + +**Pain Points** + +- Stakeholder buy in +- Delivering on time +- Visibility into cross functional statuses +- Having enough time to spend with all the team and juggling priorities +- Driving problem resolution + +**A Day in the Life** + +Pablo begins his day by checking his calendar, sees what meetings he has +today. Next he reviews emails and follows up on any IMs that he may not +have responded to. If any fires have occurred overnight, Pablo will jump +on resolving issues or unblocking progress. He gets on a couple of +customer calls and works to either gain insights or resolve issues with +the customer. His day is full of meetings with co-workers, executives +and conversations with customers. He is always working to coordinate and +manage the products across teams and throughout the company, as well as +with the customer base. This also means that Pablo occasionally touches +base with some of the Sales and Customer Experience teams. He +occasionally goes on site to visit customers + partners. He typically +makes rounds with Development + Project Management to assure that +everything is on track. Pablo says that his role requires him to wear +many hats. He considers his role part visionary, traffic coordinator, +technologist and communicator. “I love my job, it energizes and exhausts +me, all at the same time! Have more data to provide insights and +guidance would give me more time to focus on gaining deeper insights +into customer + market needs, rather than focusing on if the work is +going to get done.” diff --git a/source/usability/user-stories.rst b/source/usability/user-stories.rst deleted file mode 100644 index 4aeca55..0000000 --- a/source/usability/user-stories.rst +++ /dev/null @@ -1,9 +0,0 @@ -User Stories -============ - -.. rubric:: TOC - -.. toctree:: - :maxdepth: 1 - - user-stories/personas diff --git a/source/usability/user-stories/personas.rst b/source/usability/user-stories/personas.rst deleted file mode 100644 index 31655b5..0000000 --- a/source/usability/user-stories/personas.rst +++ /dev/null @@ -1,636 +0,0 @@ -Personas -======== - -The information below relates to personas developed for the use of the -Modularity working group. These personas will be continually updated as -more information is gathered from existing users that fit these roles. -Also, any information provided by community members or working group -team members will be incorporated as well. - -What is a Persona ------------------ - -Personas are fictional characters created to represent the different -user types that might interact with a product in different ways. They -are not market segments but should be thought of as user archetypes. - -Personas are useful in considering the goals, desires and limitations of -users in order to guide decisions about the product. They should be -based on user research and should by 1-2 page descriptions that include -behavior patterns, goals, skills, attitudes and environment details. - -There should be a primary persona defined along with secondary personas. -More than one can be primary, but no more than 3 in order to keep the -team focused in their design and implementation work. - -Benefits of Personas --------------------- - -Common benefits include: - -- Help the team share a common understanding of their various users - groups and audiences -- Proposed designs can be guided by how well they meet the needs of - certain personas -- Features can be prioritized based on how well they address the needs - of one or more personas -- Provides a human “face” to the user that can help create empathy for - the people represented - -Personas Types --------------- - -The following have been identified as the first set of personas: - -- Student -- Educator -- Researcher -- IT Executive -- IT Manager -- Business Analyst -- Enterprise System Admin -- Enterprise System Architect -- Enterprise Developer -- Enterprise DevOps and Systems of Differentiation Developer -- Enterprise DevOps Developer -- Enterprise and Independent Software Vendor Developer - Systems of - Innovation -- Traditional Independent Software Vendor -- Next Gen Independent Software Vendor -- Business Operations -- Product Manager - -.. raw:: mediawiki - - {{admon/note|Primary personas have not yet been identified| Once identified, they will be highlighted as primary.}} - -Detailed Information --------------------- - -Student -~~~~~~~ - -**Background** - -- Engineering or Computer Science student -- Multiple developer environment experience -- Plays 3D games - -**Primary Goals** - -- Needs a personal system for software classwork and personal projects -- Software class work may require particular tool chain versions -- To try out new versions of open source applications when released -- Play computer games -- Listen to music -- Interact with friends and family through social media primarily - through cellphone - -Educator -~~~~~~~~ - -**Background** - -- B.A. in Education -- 15 years in academia with different roles related to teaching and - administration - -**Primary Goals** - -- Create lesson plans and make available to students, i.e., blackboard -- Keep up to date with entering grades into on-line systems for - transparency with students -- Keep up with responding to emails from colleagues and students -- Create meaningful and engaging presentations - -Researcher -~~~~~~~~~~ - -TBD - -IT Executive - David -~~~~~~~~~~~~~~~~~~~~ - -*Aligns IT initiatives with business goals* - -**Quote** - -“In this competitive market, we need to be fast, efficient, and use -technologies that attract developers to the company.” - -**Background** - -- BS in Computer Science -- 30 years experience -- 15 years of management experience -- Advanced analytical skills - -**Primary Goals** - -- Improve IT operations, system performance, and risk management -- Implements new systems/architecture -- Cultivates IT and business partnerships -- Sell IT solutions internally -- Selects vendors that can meet his/her challenges. Convinces c-suite - peers to adopt the chosen vendor. - -**Pain Points** - -- Provide security while focusing on efficiency, mobility, innovation, - and downtime -- Doing more with less budget, skills, staff, financial transparency - and time -- Reducing server footprint while expanding services - -IT Manager - Stan -~~~~~~~~~~~~~~~~~ - -*Manages team that oversees infrastructure components* - -**Background** - -- BS in Computer Science -- 15 years experience - -**Primary Goals** - -- Determine execution plans for specific IT objectives -- Select solutions for the objectives and budgets specified by IT - Executive -- Make decisions for smaller, tactical purchases and influence larger - purchases -- Ensures reliable, 24x7 operations of existing infrastructures -- Identifies paths to resolution when issues arise - -**Pain Points** - -- Doing more with less resources -- Getting the blame when things go wrong -- Handling staffing issues (not having people trained in the skills and - capabilities required) -- Meeting regulatory and security compliance - -**A Day in the Life** - -Stan starts his day before arriving in the office by checking email and -making note of any issues that have arisen over night. Anything that -needs his immediate attention can be handled through email and requests -to others to get started on solutions. After getting to the office, he -touch base with his manager to make sure we are in sync for the day. -Then he has a quick meeting with his team to make sure they know the -goals and issues to be addressed that have come up over night. Most of -the rest of his day is spent putting out fires, dealing with email and -sitting in project meetings. - -Business Analyst - Karen -~~~~~~~~~~~~~~~~~~~~~~~~ - -*Identifies the need for changing how the company works* - -**Quote** - -“In my job I need to be a master multi-tasker. I love that there is -constant change and that every day is different.” - -**Background** - -- BS in Business or Management -- 10 years experience -- Strong written and communication skills -- Strong analytical and conceptual skills -- Working knowledge of software development processes - -**Primary Goals** - -- Be a change agent for the company -- Successfully broker change between company stakeholders and IT -- Be on top of industry trends to know when change may be needed - -**Pain Points** - -- Managing urgencies, crises, and timelines -- Expectation that better outcomes can be achieved faster with new - analysis and technology -- Difficult to achieve organizational and cultural changes necessary to - implement some solutions -- An ever increasing pace of change in the marketplace - -**A Day in the Life** - -Karen starts her day by checking email and reading the latest articles -from her feeds. Ideally, she can spend a few hours in the morning going -through industry trend documents and identifying potential items that -can be leveraged. Most days however, get full of stakeholder meetings -and project meetings where she helps to keep things on track and making -sure the business objectives are still going to be achieved. - -In the afternoon, her goals are to write up proposal and plans. She also -prefers to spend at least one hour looking through the data to support -upcoming proposals and making sure that they still make sense for the -business. - -Enterprise System Administrator - Yvonne -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Ensure current IT infrastructure works seamlessly* - -**Quote** - -“What people don’t realize is that being an System Administrator is a -24/7 job. There is a constant stream of emails with requests and issues -from all fronts and you really have to know how to prioritize and triage -effectively to be successful at this job. It is definitely not for -everyone, but I enjoy the pace and the constant challenge.” - -**Background** - -- B.S. in Computer Science -- 15 years experience -- Shell script expert -- Good at putting all the parts together - -**Primary Goals** - -- Suggest useful, secure and flexible tools that can be adopted by the - organization to achieve day-to-day tasks -- Make sure upgrades and new deployments are smooth with little to no - downtime -- Be a reliable source of information for co-workers and users -- Unified management of server resources -- Ability to understand resource usage across server inventory to - identify underutilized resources - -**Pain Points** - -- Constant interruptions -- Having to be available 24/7 depending on the issue -- Stress due to picking the right technology -- Proliferation of various management console interfaces to have to - manage - -**A Day in the Life** - -Yvonne starts her day by checking email before she even gets out of bed -to make sure there were no failures overnight and there are no fires -that can’t wait until she gets to the office. Once satisfied that all is -well, she gets ready, takes care of her family and heads into the -office. Upon arriving at the office, she makes a pot of coffee and again -checks her email. Her priorities include correcting any issues that are -stopping others from working and escalating anything that she cannot -work on herself. Yvonne then checks in with her supervisor on the day's -activities, checks on any hardware deliveries and communicates with -co-workers on overall system status. - -Enterprise System Architect - Kevin -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Links the business mission, strategy and processes to the IT strategy* - -**Quote** - -“I love my job...I am one of a few who get to work strategically with -Leadership and still maintain a technological connection to the product. -Exploring new technologies and innovations and figuring out how our -product can benefit is one of my favorite aspects of the job. It can be -very stressful and demanding, but that is how I know I am making a -difference to my team, product and company.” - -**Background** - -- B.S. in Computer Science -- 25 years experience -- Expert skills in software design and testing; Agile -- Knows Java, JavaScript, C, C++, HTML5 and CSS -- Uses Eclipse IDE - -**Primary Goals** - -- To set strategic goals through collaboration with senior leadership. -- Ensure the overall integrity of the application. -- Document and communicate industry trends and a roadmap for - implementing. -- High-level understanding of the entire system. -- Responsible for selecting systems and specifying deployment - lifecycles and system configuration. - -**Pain Points** - -- Constantly need to sell ideas that may result in deadline changes. -- Not enough resources to keep things as current as needed. -- No authority to make things happen; always negotiating between - timeline, budget and technology. -- Spend a lot of time documenting requirements, needs and trends in - order to communicate to all stakeholders. - -**A Day in the Life** - -Kevin starts his day by catching up on his feeds related to security, -serviceability and other industry trends to make sure he is aware of new -ideas. He then has his daily standups with the dev teams and meets one -on one with team members that have issues or that he needs to ask -questions of. - -Kevin spends the majority of his afternoon documenting the roadmap and -evaluating options to determine if they meet feature requests/needs and -how they fit into his strategies to meet enterprise compliance. Once a -week, he meets with Senior Leadership to report on progress as well as -identify any recommended changes to direction that he has. If leadership -has issues with his recommendations, he needs to wrap back around and -come up with other alternatives or bolster his arguments for proceeding -with his original recommendation. - -Enterprise Developer - Rajiv -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Design and implement code* - -**Background** - -- BS in Computer Science -- 8 years of experience -- Java, Javascript, CSS, MVC frameworks, github - -**Primary Goals** - -- To design and implement code -- To leverage the best tools for the job -- To complete tasks in as efficient and timely manner as possible - -**Pain Points** - -- Not having enough time to complete tasks required -- Being restricted by existing resources/tools that don’t enable the - ability to deliver new functionality -- Too much bureaucracy in the organization - -**A Day in the Life** - -When Rajiv gets to the office he spends a few minutes catching up on his -feeds and checking email. Then he participates in his team’s standup -meeting and sets up some time with a coworker who he needs help from to -complete his story. Until he can meet with him, Rajiv works on fixing -bugs that have been found. Late morning, Rajiv has two project meetings -to attend and then has his quick meeting to unblock his story. - -After lunch, Rajiv finishes his story and starts on the next one in his -backlog. Usually there are additional bugs that get filed throughout the -afternoon so he reserves the last hour and half to work on those before -heading home. - -Enterprise DevOps Developer - Devin -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Continuous integration/Continuous delivery* - -**Quote** - -“Some folks think I’m a sys admin, others think I’m a coder. The truth -is, I’m both and a lot more!” - -**Background** - -- B.S. in Computer Science -- 10 years experience -- Skilled in IT operations and software testing -- Specializes in Java as well as Python and Perl -- Uses Jenkins and Ansible - -**Primary Goals** - -- Ensuring that all applications migrate smoothly throughout the entire - lifecycle, from the development process to production support. -- Ensuring that the automation process works efficiently among the - team: developers who implement features, system administrators who - manage access control, and operators who monitor the applications and - servers. -- Identifying the root cause of problems in test, staging and - production environments, as well as the infrastructure. - -**Pain Points** - -- Multiple tools, integration points, incompatibilities, and problems - with PaaS and IaaS create many obstacles. -- When a build cannot pass the tests, a lot of time is spent looking - for the team members who are responsible for the build or the - specific feature that is causing a problem. This is exacerbated when - he does not know the responsibilities of certain people on the team. - -**A Day in the Life** - -Devin starts his workday by checking on the status of the overnight -builds. One build failed miserably, so his first order of business is to -find out why. After he figures out which component appeared to cause the -problem, he opens a Severity 1 ticket on that component so the -responsible developer will be notified immediately. Devin gets an -automated out-of-office notification from that developer, so he has to -inspect and fix the code himself, and then kick-off the test of the -build again. - -Because of the unexpected issue, Devin arrives late to the weekly DevOps -meeting with his team of developers, quality engineers, and product -managers. The main topic is the decreasing availability and performance -of the deployment system, so they need to plan for a more scalable and -performant solution. After this, he completes his daily status report -about the CI/CD pipeline to the engineering management team. Once that -is sent, Devin, intermittently coordinates work among his team members -in between the moments he gets to modify the scripts that orchestrate -the complete deployment of development, QA, and production environments. - -Enterprise DevOps Developer and Systems of Differentiation Developer - Ryan -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -**Background** - -- B.S. in Computer Science -- 10 years experience -- Skilled in IT operations and software testing -- Specializes in Java as well as Python and Perl -- Uses Jenkins and Ansible - -**Primary Goals** - -- To design and implement code -- To leverage the best tools for the job -- To complete tasks in as efficient and timely manner as possible - -**Pain Points** - -- Not having enough time to complete tasks required -- Being restricted by existing resources/tools that don’t enable him to - deliver new functionality -- Too much bureaucracy in the organization - -Enterprise and ISV Developer - Systems of Innovation - Ben -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Design and Implement Code* - -**Primary Goals** - -- To design and implement code -- To leverage the best tools for the job -- To complete tasks in as efficient and timely manner as possible - -**Pain Points** - -- Not having enough time to complete tasks required -- Being restricted by existing resources/tools that don’t enable him to - deliver new functionality -- Too much bureaucracy in the organization - -Traditional ISV -~~~~~~~~~~~~~~~ - -**Primary Goals** - -- To build, develop and sell consumer or enterprise software -- Partner with platform providers to sell more software -- Ensure customers are satisfied and their needs are being met -- Work with multiple teams including partner, customer and internal to - address issues and concerns - -**Pain Points** - -- Different customers have different requirements that can be - conflicting -- There is always something that needs to be corrected -- Can’t ship on a quick enough schedule to satisfy the customer base - -Next-Gen ISV -~~~~~~~~~~~~ - -**Primary Goals** - -- To build, develop and sell consumer or enterprise software -- Partner with platform providers to sell more software -- Ensure customers are satisfied and their needs are being met -- Work with multiple teams including partner, customer and internal to - address issues and concerns -- To ship apps faster -- To leverage containers and other emerging technologies as soon as - possible -- Multi-modal development -- SaaS model first, then deploy in organization - -**Pain Points** - -- Different customers have different requirements that can be - conflicting -- There is always something that needs to be corrected - -Business Operations - Isaac -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Management of the Operations Team* - -**Background** - -- B.S. in Information Systems -- 8 years experience - Cloud Services (IaaS, PaaS, SaaS) -- Specializes in Java, Python; knows C, C++ -- Expert knowledge of UNIX, LINUX and Windows - -**Primary Goals** - -- Ensuring successful day-to-day operations of the technology - infrastructure and that all Service Level Agreements (SLA) with - customers are met, and creating plans and providing support for - instances when an SLA is compromised. -- Quickly receiving and reviewing real-time reports of critical SLA - criteria: availability, system response times, security/privacy, - disaster recovery time, call center resolution time, dispute - mediation, and change management. -- Scheduling, budgeting, and staffing system maintenance and upgrades - to ensure maximum operational performance. - -**Pain Points** - -- Technological and financial limitations make it very challenging to - migrate legacy applications to the cloud - -**A Day in the Life** - -The first order of business for Isaac is to review the reports of system -operations from the previous day. He’s especially interested in the -availability numbers because a big software upgrade was applied to a few -servers overnight. Fortunately, system availability was not affected, -and the upgrade should improve the numbers for system response time, so -it was a win-win for customers and Isaac, who constantly keeps his eye -on the SLA commitments. - -Afterwards, Isaac attends a meeting with his staff to discuss the exit -strategy plan for a customer who will be leaving their service at the -end of the month. A smooth transition is a must. - -Isaac then has a series of meetings with the technical leads on his -staff to discuss proposals for rolling out new server hardware during -the next quarter. He needs to balance capacity and performance with the -tight budget he’s been given for expansion, so he needs lots of details -so he can evaluate and select from the multiple options on the table. - -Product Manager - Pablo -~~~~~~~~~~~~~~~~~~~~~~~ - -*Customer and industry liaison to the product and the company* - -**Quote** - -“A lot of my time is spent gathering resources, coordinating various -teams to agree on decisions or work to achieve customer goals. I need to -spend more time distilling down ideas and driving agreement across -disparate teams. My role is about working with other to build the grand -plan. Improving communication, clarity on commitment and direction, -would result in more timely decision making.” - -**Background** - -- B.S. in Computer Science -- 5+ years experience, including project management + software - development environment -- Deliver multiple projects, cross-functionally -- Expert planner with business process definition -- Influencer in org up to SVP with more experience - -**Primary Goals** - -- Understand product’s business + UX needs -- Coordinate design, development, delivery, documentation, launch, - sales enablement + requests, service, integration, support. -- To work with stakeholders, UX + Dev to transform requirements into - experiences to deliver + assure all are met. -- Managing market requirements, trends, competitive landscapes, - constraints, cost, models, time, scope, risks and P+L to the - executive team. -- Manage customer partnerships + grow business opportunities -- Understanding company's strategic goals + reflecting those in the - product - -**Pain Points** - -- Stakeholder buy in -- Delivering on time -- Visibility into cross functional statuses -- Having enough time to spend with all the team and juggling priorities -- Driving problem resolution - -**A Day in the Life** - -Pablo begins his day by checking his calendar, sees what meetings he has -today. Next he reviews emails and follows up on any IMs that he may not -have responded to. If any fires have occurred overnight, Pablo will jump -on resolving issues or unblocking progress. He gets on a couple of -customer calls and works to either gain insights or resolve issues with -the customer. His day is full of meetings with co-workers, executives -and conversations with customers. He is always working to coordinate and -manage the products across teams and throughout the company, as well as -with the customer base. This also means that Pablo occasionally touches -base with some of the Sales and Customer Experience teams. He -occasionally goes on site to visit customers + partners. He typically -makes rounds with Development + Project Management to assure that -everything is on track. Pablo says that his role requires him to wear -many hats. He considers his role part visionary, traffic coordinator, -technologist and communicator. “I love my job, it energizes and exhausts -me, all at the same time! Have more data to provide insights and -guidance would give me more time to focus on gaining deeper insights -into customer + market needs, rather than focusing on if the work is -going to get done.”