PR#329: Fetch just container.yaml and content_sets.yml instead of cloning whole dist-git repo. -

freshmaker

#329 Fetch just container.yaml and content_sets.yml instead of cloning whole dist-git repo.

Merged Nov 30, 2018 by lucarval. Opened Nov 22, 2018 by jkaluza.

jkaluza/freshmaker fast-git into master

jkaluza commented Nov 22, 2018

Currently, Freshmaker is cloning the git repository of every image it is going
to rebuild to find out the content of container.yaml and content_sets.yml to
decide whether it needs to generate Pulp composes itself or if the OSBS can
do it iself.

Cloning whole repository is slow and downloads unnecessary files.

In this commit, new get_distgit_files method is introduce which uses
"git archive" instead of "git clone" to get only particular files which
are needed by Freshmaker from the dist-git repository.

This is much faster and also does not consume any extra resources.

rebased onto 51cc2ec85b880ca3bb8cb3be29429e089557b4f0

Nov 22, 2018

lucarval commented Nov 26, 2018

I can't seem to get this to work with https only. It seems to require ssh.

UPDATE: That's because https is not supported. But that's ok, our configuration specifies the git base URL with the git:// protocol which is just fine.

Edited Nov 26, 2018 by lucarval

lucarval commented Nov 26, 2018

You could fetch all the files at once. git archive accepts multiple path values.

lucarval commented Nov 26, 2018

:+1: Minor nitpick comment, but nothing that should block it from being merged.

@cqi, could you take a look?

lholecek commented Nov 27, 2018

Is this using PyYAML? I cannot find it in requirements.txt.

More importantly: yaml.safe_load() should be used instead if you don't trust source or don't need to load any fancy Python objects. [1]

[1] https://pyyaml.org/wiki/PyYAMLDocumentation

lucarval commented Nov 30, 2018

@lholecek, I'll follow up with a different PR for that, thanks!