From c9c776ff3a7e687a3c765fca2f49e5e2f26a1d58 Mon Sep 17 00:00:00 2001 From: Adam Samalik Date: Nov 10 2017 13:11:54 +0000 Subject: [PATCH 1/2] update defining modules --- diff --git a/source/development/building-modules/developing.rst b/source/development/building-modules/developing.rst index 8e9fc70..b7bad0c 100644 --- a/source/development/building-modules/developing.rst +++ b/source/development/building-modules/developing.rst @@ -1,151 +1,349 @@ -Defining modules in modulemd -============================ +Defining modules (using modulemd) +================================= -Guidelines ----------- +To have your module build, you need to write a `modulemd +`__ file which is the definition of your module +including the components, dependencies, API, and more. This page describes all +the steps a module developer needs to do to define a module. -The Fedora Packaging Guidelines for Modules live in the -`Fedora Wiki `_. +Examples +-------- -Example -------- +To have an idea about the final result, please have a look at some existing +modules in the Fedora dist-git: -To have your module build, you need to start with writing a `modulemd -`__ file which is a definition of your module -including the components, dependencies, API, and more. The information -necessary to build your module is: build dependencies, source and build order -for the packages. Let’s have a look at an example Vim module: +* `PostgreSQL 9.6 `__ +* `Node.js 8 `__ +* `Node.js 6 `__ +* `autotools `__ + +The final module definition will be similar to the following example. Please +note that the example might not represent an actual buildable module +- dependencies or package names have changed - but it shows the overall +structure with all the fields this guide will be going through. + +Prerequisites - tooling +----------------------- + +`Fedmod `__ is a tool providing basic operation that significantly simplify +the process of creating a new module. Please install it before we start: + +:: + + $ sudo dnf copr enable @modularity/fedmod + $ sudo dnf install fedmod + $ fedmod fetch-metadata + + +Defining the module +------------------- + +Let's start with the following template and fill it in as we go. :: document: modulemd version: 1 data: - summary: The best text editor and IDE - description: The classic, extensible text editor, the legend. + summary: ... + description: ... license: - module: [ MIT ] + module: + - license-name dependencies: buildrequires: - base-runtime: f26 + module-name: stream + ... requires: - base-runtime: f26 + module-name: stream + ... references: - community: http://www.vim.org/ - documentation: http://www.vim.org/docs.php - tracker: https://github.com/vim/vim/issues + community: http://example.com + documentation: http://example.com + tracker: http://example.com profiles: - default: + example: rpms: - - vim-enhanced - - vim-common - - vim-filesystem - minimal: - rpms: - - vim-minimal + - package-name + - ... + ... api: rpms: - - vim-common + - package-name + - ... components: rpms: - vim: - rationale: Provides API for this module - ref: f26 - buildorder: 10 - generic-release: - rationale: build dependency - ref: f26 - perl-Carp: - rationale: build dependency - ref: f26 - gpm: - rationale: build dependency - ref: f26 - perl-Exporter: - rationale: build dependency - ref: f26 + package-name: + rationale: ... + ref: ... + buildorder: ... + ... -Notice that there is no information about the name or version of the module. -That’s because the build system takes this information from the git -repository, from which the module is build: -* Git repository name == module name -* Git repository branch == module stream -* Commit timestamp == module version +Step 1: Deciding what the module is +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -All dependencies of vim need to be listed under components/rpms, except -those that are already included in other modules, such as `Base Runtime -`__. -To see what modules are available, open `src.fedoraproject.org -`__ and type :code:`modules/` into the search -field. +The first step is to identify a package (or a set of packages) that +form the core of the module. For example, to create an nginx module, +an `nginx` package will be the core. -At the moment there is no service which tracks which packages are present in -which modules. In the meantime, `we've developed a tool -`__ which is able to construct a -preliminary modulemd file for a selected set of rpms you provide. It queries -information about previously built modules to fulfill build and runtime package -dependencies. +Based on this, fill in the `summary`, `description`, `license`, and `references` fields. +Also, add the main package(s) to `components`. Please note that +the `components.rpms` filed expects SRPM package names. :: - $ mod-tools rpm2module nodejs npm - $ cat *.yaml - data: - api: - rpms: [nodejs, npm] - components: - rpms: - http-parser: {rationale: Build and runtime dependency.} - nodejs: {buildorder: 10, rationale: Runtime dependency.} - dependencies: - buildrequires: {base-runtime: f26, shared-userspace: f26} - requires: {base-runtime: f26, shared-userspace: f26} - description: '' - license: - module: [MIT] - summary: '' document: modulemd version: 1 + data: + summary: A high performance web server and reverse proxy server + description: >- + Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 + and IMAP protocols, with a strong focus on high concurrency, + performance and low memory usage. + license: + module: + - MIT + references: + community: http://example.com + documentation: http://example.com + tracker: http://example.com + components: + rpms: + nginx: + rationale: The main package of this module. + ref: f27 + +Step 2: Dependencies +~~~~~~~~~~~~~~~~~~~~ + +Every module needs to specify its runtime and build dependencies. +These are modular dependencies (usually at least the `platform` module), +and RPM dependencies (everything else that is not provided by the modular dependencies). + +This is where the Fedmod tool becomes useful. To list all dependencies of +a given package (`nginx` in this case), run: + +:: + + $ fedmod resolve-deps nginx + + +This will return a very long list of packages. Many of these are included in +the `platform` module. Let's get all the missing dependencies that are not +in `platform` by running: + +:: + + $ fedmod resolve-deps -m platform nginx + + +This looks much better, right? At the time of writing this document, fedmod +returned the following four packages: + +:: + + gperftools-libs + nginx-mimetypes + nginx + nginx-filesystem + + +Fedmod can also tell if a certain package has been already added to an existing module. +To find whether the `gperftools-libs` is already somewhere, run: + +:: + + $ fedmod where-is-package gperftools-libs + + +At the time of writing, the `gperftools-libs` package is already in one module, +the `389-ds` module which is an LDAP server. In this case, it doesn't make sense +to use the `389-ds` module as a dependency of `nginx`. There are two ways how to resolve +this problem: + +1. Work with a maintainer of the `389-ds` module and get the `gperftools-libs` package separated into another, shared module. Maybe it could be added to the `platform` module. +2. Bundle the package in your module. Please note that this might make the `nginx` module conflict with the `389-ds` module. But this is the way we go in this guide. + +As mentioned before, the modulemd file expect components to be listed as SRPM packages. +Fedmod can give you an SRPM package name for a given RPM. Let's try it for the +`nginx-mimetypes` package by running: + +:: -You can also use `depchase `__ -to identify precise build and runtime dependencies of your packages. Please -consult the documentation of the tool for more information. + $ fedmod srpm-of-rpm nginx-mimetypes -Bootstrapping -------------- +By doing this for all the packages, the result was: -It may happen that some build requirements are missing for your package to -build in a module. There are two solutions: +:: + + gperftools-libs -> gperftools + nginx-mimetypes -> mailcap + nginx -> nginx + nginx-filesystem -> nginx + + +The `nginx` package is already present in the `components.rpm` field. +We just need to add `mailcap` and `gperftools`. We need to add `platform` +as our runtime dependency. + +We have now resolved all the runtime dependency. The next step would be resolving +all the build dependencies for this module. However, not all build dependencies +might have been modularized. As a workaround, we use a `bootstrap` module as the only +build dependency right now. + +After doing these changes, our modulemd will look as follows: + +:: + + document: modulemd + version: 1 + data: + summary: A high performance web server and reverse proxy server + description: >- + Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 + and IMAP protocols, with a strong focus on high concurrency, + performance and low memory usage. + license: + module: + - MIT + dependencies: + buildrequires: + bootstrap: f27 + requires: + platform: f27 + references: + community: http://example.com + documentation: http://example.com + tracker: http://example.com + components: + rpms: + nginx: + rationale: The main package of this module. + ref: f27 + mailcap: + rationale: Provides a dependency: nginx-mimetypes. + ref: f27 + gperftools: + rationale: Provides a dependency: gperftools-libs. + ref: f27 + + + +What do you support +~~~~~~~~~~~~~~~~~~~ + +As modules contain the main packages, along with their dependencies, it is useful +to distinguish between these two. The module maintainer will probably offer +an API/ABI stability on the main packages, but the dependencies might be considered +an implementation detail. To communicate this clearly, add all the main packages to +the `api` field in your modulemd. In this case, it could be the `nginx`, +`nginx-filesystem`, and the `nginx-mimetypes` packages. + +:: + + document: modulemd + version: 1 + data: + summary: A high performance web server and reverse proxy server + description: >- + Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 + and IMAP protocols, with a strong focus on high concurrency, + performance and low memory usage. + license: + module: + - MIT + dependencies: + buildrequires: + bootstrap: f27 + requires: + platform: f27 + references: + community: http://example.com + documentation: http://example.com + tracker: http://example.com + api: + rpms: + - nginx + - nginx-filesystem + - nginx-mimetypes + components: + rpms: + nginx: + rationale: The main package of this module. + ref: f27 + mailcap: + rationale: Provides a dependency: nginx-mimetypes. + ref: f27 + gperftools: + rationale: Provides a dependency: gperftools-libs. + ref: f27 -1. Create a new bootstrap module which helps bootstrapping your target module. - The bootstrap module will probably refer to a commit in dist-git rpm where - the package enables bootstrapping (cutting build dependencies down, - disabling features). For more info, see `this module - `__. -2. Tag existing packages from koji into the module build tag so they are - present during build. This solution isn't preferred because it makes it at - least very hard if not impossible to reproduce how a module was built at a - later point. +Pre-defined installation profiles +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +To help users with installing this module, a set of installation profiles +might be define for the module. -Module API ----------- +Profiles are very nicely described in the `modulemd +`__ specification. In short, they are lists of +packages to be installed on a system. -There are two important concepts you need to have in your module: +For example, database module might define two installation profiles, +`client` and `server`, to help user install the right package for each use case. -1. **API** — list of binary packages which are provided by the module and supported -2. **filter** — list of binary packages which should not be provided by the - module (should be filtered out during the compose phase). These are usually - packages which are used to build the target packages and the maintainer - doesn't want to support them. They can also be subpackages of target - packages which can't be installed due to missing runtime dependencies. +Some profile names, such as `default`, are reserved. The `default` profile +is installed by default when user doesn't specify any other profile. In this case, +our module will only contain this `default` profile. +:: -Check the syntax ----------------- + document: modulemd + version: 1 + data: + summary: A high performance web server and reverse proxy server + description: >- + Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 + and IMAP protocols, with a strong focus on high concurrency, + performance and low memory usage. + license: + module: + - MIT + dependencies: + buildrequires: + bootstrap: f27 + requires: + platform: f27 + references: + community: http://example.com + documentation: http://example.com + tracker: http://example.com + profiles: + default: + rpms: + - nginx + api: + rpms: + - nginx + - nginx-filesystem + - nginx-mimetypes + components: + rpms: + nginx: + rationale: The main package of this module. + ref: f27 + mailcap: + rationale: Provides a dependency: nginx-mimetypes. + ref: f27 + gperftools: + rationale: Provides a dependency: gperftools-libs. + ref: f27 + + +Validating the modulemd syntax +------------------- Once the modulemd file is finished, it is a good idea to check if there any errors in the yaml syntax. The From 576dc2d93636fd7839ac09b518c5fea7e2b361ed Mon Sep 17 00:00:00 2001 From: Adam Samalik Date: Nov 13 2017 11:46:38 +0000 Subject: [PATCH 2/2] fix typos and language errors --- diff --git a/source/development/building-modules/developing.rst b/source/development/building-modules/developing.rst index b7bad0c..2706b71 100644 --- a/source/development/building-modules/developing.rst +++ b/source/development/building-modules/developing.rst @@ -52,10 +52,10 @@ Let's start with the following template and fill it in as we go. - license-name dependencies: buildrequires: - module-name: stream + module-name: stream-name ... requires: - module-name: stream + module-name: stream-name ... references: community: http://example.com @@ -84,12 +84,12 @@ Step 1: Deciding what the module is ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The first step is to identify a package (or a set of packages) that -form the core of the module. For example, to create an nginx module, -an `nginx` package will be the core. +form the core of the module. For example, to create an `nginx` module +for the NGINX web server, the `nginx` package will be the core. -Based on this, fill in the `summary`, `description`, `license`, and `references` fields. -Also, add the main package(s) to `components`. Please note that -the `components.rpms` filed expects SRPM package names. +Based on this fill in the `summary`, `description`, `license`, and `references` fields. +Also add the main package(s) to `components`. Please note that +the `components.rpms` field expects SRPM package names. :: @@ -121,7 +121,7 @@ Every module needs to specify its runtime and build dependencies. These are modular dependencies (usually at least the `platform` module), and RPM dependencies (everything else that is not provided by the modular dependencies). -This is where the Fedmod tool becomes useful. To list all dependencies of +This is where the Fedmod tool becomes handy. To list all dependencies of a given package (`nginx` in this case), run: :: @@ -185,10 +185,10 @@ By doing this for all the packages, the result was: The `nginx` package is already present in the `components.rpm` field. -We just need to add `mailcap` and `gperftools`. We need to add `platform` -as our runtime dependency. +We just need to add the mailcap and gperftools packages in the +components.rpm field and the platform module as a runtime dependency.. -We have now resolved all the runtime dependency. The next step would be resolving +With this, we have resolved the runtime dependencies. The next step would be resolving all the build dependencies for this module. However, not all build dependencies might have been modularized. As a workaround, we use a `bootstrap` module as the only build dependency right now. @@ -284,18 +284,18 @@ the `api` field in your modulemd. In this case, it could be the `nginx`, Pre-defined installation profiles ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To help users with installing this module, a set of installation profiles -might be define for the module. +To help users with installing this module, a set of installation +profiles can and at least one should be defined for the module. Profiles are very nicely described in the `modulemd `__ specification. In short, they are lists of packages to be installed on a system. -For example, database module might define two installation profiles, -`client` and `server`, to help user install the right package for each use case. +For example, a database module might define two installation profiles, +`client` and `server`, to help the user install the right package for each use case. Some profile names, such as `default`, are reserved. The `default` profile -is installed by default when user doesn't specify any other profile. In this case, +is installed by default if user doesn't specify any other profile. In this case, our module will only contain this `default` profile. ::