releng

#7738 fedpkg build - Connection aborted.
Closed: Fixed by syeghiay. Opened by vondruch.

Closed: Fixed
Reopen Issue
  • Describe the issue
    The fedpkg connection with server fails during build almost every time:
$ fedpkg scratch-build --srpm 
/usr/lib/python3.7/site-packages/fedora/client/bodhi.py:48: DeprecationWarning: fedora.client.bodhi has been deprecated. Please use bodhi.client.bindings instead.
  DeprecationWarning)
/usr/lib/python3.7/site-packages/fedpkg/__init__.py:235: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5
  runtime_os, runtime_version, _ = platform.linux_distribution()
Zapsáno: /home/vondruch/fedora-scm/own/ruby/ruby-2.5.1-99.fc30.src.rpm
/usr/lib/python3.7/site-packages/koji/__init__.py:1704: DeprecationWarning: This method will be removed in future versions.  Use 'parser.read_file()' instead.
  config.readfp(f)
[====================================] 100% 00:00:03  10.90 MiB   3.16 MiB/sec
Building ruby-2.5.1-99.fc30.src.rpm for rawhide
Created task: 29377062
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=29377062
Watching tasks (this may be safely interrupted)...
Could not execute scratch_build: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

This happens already for some time and it is quite annoying.

$ rpm -q fedpkg
fedpkg-1.35-1.fc30.noarch

$ koji watch-task 29377062
Watching tasks (this may be safely interrupted)...
29377062 build (rawhide, ruby-2.5.1-99.fc30.src.rpm): open (buildvm-12.phx2.fedoraproject.org)
  29377065 buildArch (ruby-2.5.1-99.fc30.src.rpm, aarch64): open (buildvm-aarch64-08.arm.fedoraproject.org)
  29377067 buildArch (ruby-2.5.1-99.fc30.src.rpm, s390x): open (buildvm-s390x-06.s390.fedoraproject.org)
  29377064 buildArch (ruby-2.5.1-99.fc30.src.rpm, ppc64le): open (buildvm-ppc64le-15.ppc.fedoraproject.org)
  29377063 buildArch (ruby-2.5.1-99.fc30.src.rpm, x86_64): open (buildvm-09.phx2.fedoraproject.org)
  29377068 buildArch (ruby-2.5.1-99.fc30.src.rpm, armv7hl): open (buildvm-armv7-13.arm.fedoraproject.org)
  29377066 buildArch (ruby-2.5.1-99.fc30.src.rpm, i686): open (buildvm-06.phx2.fedoraproject.org)
  29377067 buildArch (ruby-2.5.1-99.fc30.src.rpm, s390x): open (buildvm-s390x-06.s390.fedoraproject.org) -> closed
  0 free  6 open  1 done  0 failed
  29377063 buildArch (ruby-2.5.1-99.fc30.src.rpm, x86_64): open (buildvm-09.phx2.fedoraproject.org) -> closed
  0 free  5 open  2 done  0 failed
  29377066 buildArch (ruby-2.5.1-99.fc30.src.rpm, i686): open (buildvm-06.phx2.fedoraproject.org) -> closed
  0 free  4 open  3 done  0 failed
2018-08-30 13:24:12,023 [ERROR] koji: ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

@lsedlar Any ideas here? Its not happening for me with the same version of fedpkg.

Not sure if its a connection issue or fedpkg issue.

This is purely a network connectivity issue, I see it in some locations not others, I'm not sure it's anything we can really do here, and it's certainly not a releng issue. the koji client/fedpkg might like to add some retry code but this isn't the place for that.

I don't think this is purely a network connectivity issue. While it might be a coincidence, I remember it started after some system update. My guess is that it could start with rpkg-1.53, i.e. first Python 3 version of rpkg.

I don't think this is purely a network connectivity issue. While it might be a coincidence, I remember it started after some system update. My guess is that it could start with rpkg-1.53, i.e. first Python 3 version of rpkg.

Well there's been an infra ticket about connectivity of late to eastern Europe, and I see if on and off, for example more from Aus than UK for example. Even if it was a issue with rpkg it should be reported there not here.

So, this might be a koji issue or fedora infrastructure config of our deployment... not sure.

@mikem any thoughts?

I'll note I captured one with -d: 

✗ koji -d build --scratch f30 /home/kevin/git/pkgs/ansible/master/ansible-2.6.4-1.fc30.src.rpm                        
2018-09-07 10:32:24,682 [DEBUG] koji: Opening new requests session                                                    
2018-09-07 10:32:24,683 [DEBUG] koji: Opening new requests session                                                    
successfully connected to hub
Uploading srpm: /home/kevin/git/pkgs/ansible/master/ansible-2.6.4-1.fc30.src.rpm                                      
2018-09-07 10:32:25,755 [DEBUG] koji: Fast upload: /home/kevin/git/pkgs/ansible/master/ansible-2.6.4-1.fc30.src.rpm to
cli-build/1536341545.7556603.CkobZAvz/ansible-2.6.4-1.fc30.src.rpm                                                    
2018-09-07 10:32:38,902 [DEBUG] koji: Fast upload: /home/kevin/git/pkgs/ansible/master/ansible-2.6.4-1.fc30.src.rpm com
plete. 10679827 bytes in 13.1 seconds
Created task: 29539961                                                                                                
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=29539961                                               
Watching tasks (this may be safely interrupted)...                                                                    
29539961 build (f30, ansible-2.6.4-1.fc30.src.rpm): free
2018-09-07 10:32:39,547 [DEBUG] koji: Opening new requests session                                                    
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen                             
    chunked=chunked)
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request                       
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from                                                                              
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request                       
    httplib_response = conn.getresponse()                                                                             
  File "/usr/lib64/python3.7/http/client.py", line 1321, in getresponse                                               
    response.begin()                                                                                                  
  File "/usr/lib64/python3.7/http/client.py", line 296, in begin                                                      
    version, status, reason = self._read_status()                                                                     
  File "/usr/lib64/python3.7/http/client.py", line 265, in _read_status                                               
    raise RemoteDisconnected("Remote end closed connection without"                                                   
http.client.RemoteDisconnected: Remote end closed connection without response                                         
During handling of the above exception, another exception occurred:                                                   
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.7/site-packages/urllib3/packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/usr/lib64/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.7/http/client.py", line 265, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without resp
onse'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/bin/koji", line 310, in <module>
    rv = locals()[command].__call__(options, session, args)
  File "/usr/lib/python3.7/site-packages/koji_cli/commands.py", line 500, in handle_build
    poll_interval=options.poll_interval)
  File "/usr/lib/python3.7/site-packages/koji_cli/lib.py", line 302, in watch_tasks
    for child in session.getTaskChildren(task_id):
  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2014, in __call__
    return self.__func(self.__name, args, opts)
  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2473, in _callMethod
    return self._sendCall(handler, headers, request)
  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2395, in _sendCall
    return self._sendOneCall(handler, headers, request)
  File "/usr/lib/python3.7/site-packages/koji/__init__.py", line 2434, in _sendOneCall
    r = self.rsession.post(handler, **callopts)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 559, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 495, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without r
esponse'))

I confirm I am also getting this Connection aborted error, due to that unable to understand if build is successful or failed. Need to use koji web for that.

I confirm I am also getting this Connection aborted error, due to that unable to understand if build is successful or failed. Need to use koji web for that.

You can use koji watch-task 7987978, but it repeatedly fails as well, so the web is a more reliable ATM.

This is probably related:

https://bugzilla.redhat.com/show_bug.cgi?id=1609298

I have yet to encounter this issue when running the Python 2 version of the client packages. I suspect it's a client-side issue, caused by changes in the HTTP implementation (perhaps different exceptions thrown?).

Similiar connection timeouts as described above are actually happening here, too.

I reported this upstream at https://pagure.io/koji/issue/1192

I believe https://pagure.io/koji/pull-request/1203 should fix this.

This issue should be fixed now. Please reopen ticket if not. Closing as fixed.

Metadata Update from @syeghiay:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Not reproducible any more, with Fedora 30. Several builds done so far without any connection got accidently lost.
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.