releng

#10540 Build of GPRbuild is stuck on four arches.
Closed: upstream by humaton. Opened by rombobeorn.

Closed: upstream
Reopen Issue

I hope this is the right place for this request.

Somebody please unblock whatever it is this build is waiting on:
https://koji.fedoraproject.org/koji/taskinfo?taskID=81360084

I cancelled the previous failing build after it was stuck for many hours:
https://koji.fedoraproject.org/koji/taskinfo?taskID=81321405

I also cancelled the two scratch builds that were also stuck:
https://koji.fedoraproject.org/koji/taskinfo?taskID=81325442
https://koji.fedoraproject.org/koji/taskinfo?taskID=81325674

This time I need the build to complete on all arches, or else fail with a useful error message.

I do not see anything "stuck". It's building just slowly.

for example on x86_64: 

  |-kojid,3039447 /usr/sbin/kojid --fg --force-lock --verbose
  |   `-kojid,2398644 /usr/sbin/kojid --fg --force-lock --verbose
  |       `-mock,2399029 -tt /usr/libexec/mock/mock -r koji/f36-build-32436670-4394507 --old-chroot --no-clean --target x86_64 ...
  |           `-rpmbuild,2399170 -bb --target x86_64 --nodeps /builddir/build/SPECS/gprbuild.spec
  |               `-sh,2399211 -e /var/tmp/rpm-tmp.YkJx6X
  |                   `-make,2399244...
  |                       `-gprbuild,2399266 -v -cargs:Ada -O2 -flto=auto -ffat-lto-objects -fexceptions -g ...
  |                           |-{gprbuild},2399267
  |                           |-gcc,2399303 -c -x ada -gnatA -gnat12 -gnaty -gnatQ -gnata -gnateE -g -gnatVa -gnatwaCJI -gnatwe ...
  |                           |   |-gnat1,2399304 -quiet -O2 -Wall -Werror=format-security -dumpbase gprbind.adb -dumpbase-ext ...
  |                           |   `-as,2399305 --gdwarf-5 --64 -o gprbind.o
  |                           `-gcc,2399379 -c -x ada -gnatA -gnat12 -gnaty -gnatQ -gnata -g -gnata -gnatVa -gnatwaCJI -gnatwe ...
  |                               |-gnat1,2399380 -quiet -O2 -Wall -Werror=format-security -dumpbase gpr-knowledge.adb ...
  |                               `-as,2399381 --gdwarf-5 --64 -o gpr-knowledge.o

I see that it's using only 2 cpus... perhaps smp_mflags would help ?

If it used to build faster, perhaps it's something with gcc12?

Kevin Fenzi wrote:

I see that it's using only 2 cpus... perhaps smp_mflags would help ?

We use _smp_build_ncpus if it exists. You should see a -j parameter
near the end of the gprbuild command line.

If it used to build faster, perhaps it's something with gcc12?

It's definitely not normal that i686 finishes in seven minutes and
x86-64 is still running after 16 hours. armv7hl is usually slowest but
that one finished in 21 minutes and 27 seconds.

Thanks for the process tree. It looks like the two assembler processes
might be hung or something. I guess I'll have to try to reproduce this
outside of Koji so I can see what's going on.

 
kojibui+ 2399244  0.0  0.0  10368  3596 ?        S    13:43   0:00 make BUILDER=gprbuild -v -cargs:Ada -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS  -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -gnatn -gnat-p -gnatVd -gnatwn -gnatyN -cargs:C -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS  -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -cargs:C++ -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS  -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -cargs:Fortran -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS  -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -largs -Wl,-z,relro -Wl,--as-needed   -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -gargs -j6 -R -p -vl -XHARDWARE_PLATFORM=x86_64 all BUILD=debug

kojibui+ 2399266  0.0  0.0  83704 13704 ?        Sl   13:43   0:00 gprbuild -v -cargs:Ada -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -gnatn -gnat-p -gnatVd -gnatwn -gnatyN -cargs:C -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -cargs:C++ -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -cargs:Fortran -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules -largs -Wl,-z,relro -Wl,--as-needed -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -gargs -j6 -R -p -vl -XHARDWARE_PLATFORM=x86_64 gprbuild.gpr -XLIBRARY_TYPE=static -XXMLADA_BUILD=static

So, yeah, there's a -j6 there, but doesn't seem to be working.

:(

Anyhow, if you want us to look at anything more let us know

Yep, it looks like a regression in GCC. After I managed to get GCC 12
installed on rawhide-test.fedorainfracloud.org, I find that the
compiler gets stuck in an infinite loop in add_sibling_attributes in
dwarf2out.c. With -j4 I get four compilations in parallel until all the
others are done and only the two looping processes remain.

On x86-64 it seems to be gprbind.adb and gpr-knowledge.adb that trigger
the problem every time. Could you check whether it's stuck on the same
two source files on the other three arches, or whether the problem
manifests differently on different arches?

GCC bug: https://bugzilla.redhat.com/show_bug.cgi?id=2041667
Edited by humaton

Metadata Update from @humaton:
- Issue close_status updated to: upstream
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.