releng

#9581 Container images builds fail constantly for couple of days
Closed: Duplicate by mohanboddu. Opened by hhorak.

Closed: Duplicate
Reopen Issue
  • Describe the issue
    Container images builds fail constantly for couple of days. An example of this failure: https://koji.fedoraproject.org/koji/taskinfo?taskID=46736082

I'm trying since Friday July 3 and still seeing the same issue.

  • When do you need this? (2020/07/10)
    Quite soon, this blocks all the containers rebuilds.

  • When is this no longer needed or useful? (YYYY/MM/DD)
    N/A

  • If we cannot complete your request, what is the impact?
    No new containers in Fedora, existing containers outdated and full of CVEs.

A relevant error message is probably this:

2020-07-07 08:40:55,776 platform:- - atomic_reactor.util - DEBUG - query_registry: querying /v2/f33/s2i-core/blobs/sha256:133d203e5aed6ff6504662994ea0d9eb77e438fd062bc32a2b1d78e05d6be6a7, headers: {'Accept': 'application/vnd.docker.distribution.manifest.v2+json'}
2020-07-07 08:41:39,863 platform:- - atomic_reactor.plugin - DEBUG - Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 334, in connect
    conn = self._new_conn()
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a1df675d0>: Failed to establish a new connection: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
    **response_kw
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cdn.registry.fedoraproject.org', port=443): Max retries exceeded with url: /v2/f33/s2i-core/blobs/sha256:133d203e5aed6ff6504662994ea0d9eb77e438fd062bc32a2b1d78e05d6be6a7 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a1df675d0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/atomic_reactor/plugin.py", line 268, in run
    plugin_response = plugin_instance.run()
  File "/usr/lib/python3.7/site-packages/atomic_reactor/plugins/pre_bump_release.py", line 237, in run
    parser = df_parser(self.workflow.builder.df_path, workflow=self.workflow)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/util.py", line 1211, in df_parser
    parent_config = workflow.builder.base_image_inspect[INSPECT_CONFIG]
  File "/usr/lib/python3.7/site-packages/atomic_reactor/build.py", line 312, in base_image_inspect
    self.base_image_dockercfg_path)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/util.py", line 1086, in get_inspect_for_image
    dockercfg_path=dockercfg_path)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/util.py", line 1153, in get_config_and_id_from_registry
    registry_session, image, digest=config_digest, version=version, is_blob=True)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/util.py", line 862, in query_registry
    response = registry_session.get(url, headers=headers)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/util.py", line 762, in get
    return self._do(self.session.get, relative_url, **kwargs)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/util.py", line 759, in _do
    return f(self._base + relative_url, *args, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3.7/site-packages/atomic_reactor/util.py", line 1339, in request
    return super(SessionWithTimeout, self).request(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 668, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 247, in resolve_redirects
    **adapter_kwargs
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='cdn.registry.fedoraproject.org', port=443): Max retries exceeded with url: /v2/f33/s2i-core/blobs/sha256:133d203e5aed6ff6504662994ea0d9eb77e438fd062bc32a2b1d78e05d6be6a7 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a1df675d0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

It seems like a registry networking issue, closing it in favor of https://pagure.io/fedora-infrastructure/issue/9116

Metadata Update from @mohanboddu:
- Issue close_status updated to: Duplicate
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.