Cl-update unable to complete as python hangs

Discussions & Opinions
1

Hi Calculate Brains Trust
cll-update is stalling on update.
Using “top” from cli shows that python3.11 is running
.python3.11 never finishes and cl-update is stuck on the this package.

Thanks in advance for your help with this,

emerge --info:

Portage 3.0.63 (python 3.11.9-final-0, default/amd64/20/desktop, gcc-13, glibc-2.39-r6, 6.1.12-x86_64 x86_64)
=================================================================
System uname: Linux-6.1.12-x86_64-x86_64-Intel-R-_Xeon-R-_CPU_W3680_@_3.33GHz-with-glibc2.39
KiB Mem:    24603188 total,   7550908 free
KiB Swap:    4194300 total,   4194300 free
Timestamp of repository amano-kenji: Thu, 06 Jun 2024 20:49:19 +0000
Head commit of repository amano-kenji: 948cb7bef6e2ad31afe1e9d96788425d34c826de

Timestamp of repository gig: Thu, 06 Jun 2024 20:49:11 +0000
Head commit of repository gig: 3698ab97600ebad3bb2a70063b5bf11116c8e7c9

Timestamp of repository guru: Sat, 08 Jun 2024 22:48:55 +0000
Head commit of repository guru: 975045a574a42361ed226bb00cab13e2ab456427

Timestamp of repository metahax: Sat, 08 Jun 2024 16:03:48 +0000
Head commit of repository metahax: eace20c23c6ce14ba95bb2f27b87fc6f4d90b97c

Timestamp of repository pentoo: Sun, 09 Jun 2024 03:03:49 +0000
Head commit of repository pentoo: 28330b1a8e76017d3853321eddcf14cc22fb3948

Timestamp of repository r7l: Tue, 04 Jun 2024 08:20:29 +0000
Head commit of repository r7l: a1cc354926e528cc0ddb3d471e3759cc92bf6548

Timestamp of repository stefantalpalaru: Sat, 08 Jun 2024 09:34:46 +0000
Head commit of repository stefantalpalaru: 34a5c8ff94cff74dbb6f50150e4053d148bcae40

Timestamp of repository torbrowser: Thu, 06 Jun 2024 20:49:07 +0000
Head commit of repository torbrowser: ac0274cfdbe6fea24fb3291f09ebc2ea82ed05d7

Timestamp of repository gentoo: Fri, 07 Jun 2024 05:30:01 +0000
sh bash 5.1_p16-r6
ld GNU ld (Gentoo 2.42 p3) 2.42.0
distcc[11428] (dcc_mkdir) ERROR: mkdir '/var/tmp/portage/.distcc/state' failed: No such file or directory [disabled]
ccache version 4.9.1 [enabled]
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.1_p16-r6::gentoo
dev-build/autoconf:        2.71-r7::gentoo
dev-build/automake:        1.16.5-r2::gentoo
dev-build/cmake:           3.28.5::gentoo
dev-build/libtool:         2.4.7_rc73::gentoo
dev-build/make:            4.4.1-r1::gentoo
dev-build/meson:           1.4.0-r1::gentoo
dev-java/java-config:      2.3.3-r1::gentoo
dev-lang/perl:             5.38.2-r3::gentoo
dev-lang/python:           3.10.14_p1::gentoo, 3.11.9::gentoo, 3.12.3::gentoo
dev-lang/rust:             1.77.1::gentoo
dev-util/ccache:           4.9.1-r1::gentoo
sys-apps/baselayout:       2.15::gentoo
sys-apps/openrc:           0.54::gentoo
sys-apps/sandbox:          2.38::gentoo
sys-devel/autoconf:        2.71-r5::gentoo
sys-devel/automake:        1.16.5::gentoo
sys-devel/binutils:        2.42-r1::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           17.0.6::gentoo
sys-devel/gcc:             13.2.1_p20240210::gentoo
sys-devel/gcc-config:      2.11::gentoo
sys-devel/llvm:            17.0.6::gentoo
sys-kernel/linux-headers:  6.6-r1::gentoo (virtual/os-headers)
sys-libs/glibc:            2.39-r6::gentoo
Repositories:

amano-kenji
    location: /var/db/repos/amano-kenji
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/amano-kenji.git
    masters: gentoo
    volatile: False

custom
    location: /var/calculate/custom-overlay
    masters: calculate gentoo
    volatile: True

gig
    location: /var/db/repos/gig
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gig.git
    masters: gentoo
    volatile: False

guru
    location: /var/db/repos/guru
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/guru.git
    masters: gentoo
    volatile: False

metahax
    location: /var/db/repos/metahax
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/metahax.git
    masters: gentoo
    volatile: False

pentoo
    location: /var/db/repos/pentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/pentoo.git
    masters: gentoo
    volatile: False

r7l
    location: /var/db/repos/r7l
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/r7l.git
    masters: gentoo
    volatile: False

stefantalpalaru
    location: /var/db/repos/stefantalpalaru
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/stefantalpalaru.git
    masters: gentoo guru
    volatile: False

torbrowser
    location: /var/db/repos/torbrowser
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/torbrowser.git
    masters: gentoo
    volatile: False

calculate
    location: /var/db/repos/calculate
    sync-type: calculate
    sync-uri: https://git.calculate-linux.org/calculate/calculate-overlay
    masters: gentoo
    priority: 50
    volatile: False

distros
    location: /var/db/repos/distros
    sync-type: calculate
    sync-uri: https://git.calculate-linux.org/calculate/distros-overlay
    masters: calculate gentoo
    priority: 50
    volatile: False

gentoo
    location: /var/db/repos/gentoo
    sync-type: calculate
    sync-uri: https://git.calculate-linux.org/calculate/gentoo-overlay
    priority: 50
    volatile: False

Installed sets: @custom
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php8.1/ext-active/ /etc/php/apache2-php8.2/ext-active/ /etc/php/cgi-php8.1/ext-active/ /etc/php/cgi-php8.2/ext-active/ /etc/php/cli-php8.1/ext-active/ /etc/php/cli-php8.2/ext-active/ /etc/php/fpm-php8.1/ext-active/ /etc/php/fpm-php8.2/ext-active/ /etc/php/phpdbg-php8.1/ext-active/ /etc/php/phpdbg-php8.2/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe"
DISTDIR="/var/calculate/distfiles"
EMERGE_DEFAULT_OPTS=" --autounmask --binpkg-changed-deps=n --binpkg-respect-use=y --jobs=4 --quiet-build=n --usepkg-exclude='x11-drivers/nvidia-drivers sys-fs/zfs-kmod'  --autounmask-write y --ask --binpkg-respect-use=n"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg buildpkg-live ccache clean-logs config-protect-if-modified distlocks ebuild-locks fixlafiles getbinpkg ipc-sandbox merge-sync merge-wait multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="https://distfiles.gentoo.org https://mirror.calculate-linux.org/distfiles/level0"
INSTALL_MASK="/etc/systemd /lib*/systemd/*/*.service /usr/lib*/systemd/*/*.service /etc/shorewall*/* -/etc/shorewall*/shorewall*.conf"
LANG="en_AU.UTF-8"
LC_ALL=""
LDFLAGS="-Wl,--as-needed"
LEX="flex"
MAKEOPTS="-j12 -l12"
PKGDIR="/var/calculate/packages"
PORTAGE_BINHOST="https://mirror-hk.koddos.net/calculate-linux/grp/x86_64"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="3dnow 3dnowext X a52 aac acl acpi aes alsa amd64 amr audiofile avi avx bittorrent bluetooth box bzip2 cairo caps cdda cddb cdparanoia cet cgroups chm cpudetection cracklib crypt css cue cups dav1d dbus dftk djvu dri dv dvd dvdr dvdread ebook elogind encode exif faac fexif ffmpeg flac flash fma3 fma4 foomaticdb ftp gadu gd gdbm gif gphoto2 gtk heif hunspell iconv id3tag idn ieee1394 imagemagick imap introspection ios ipod ipv6 irc jabber jpeg jpeg2k lame libnotify libtirpc libwww lm-sensors logrotate lz4 lzo mad matroska mikmod mimap mmx mmxext mp3 mpeg mplayer multilib musepack ncurses networkmanager nfs nls nsplugin ogg oggvorbis openal opencore-amr openexr opengl openmp opus oscar padlock pam pcre pdf pgo pipewire png policykit popcnt postscript ppp pulseaudio quicktime raw rdesktop readline real samba sasl scanner screencast sdl seccomp semantic-desktop smp smtp speex spell split-usr srt sse sse2 sse3 sse4_1 sse4_2 sse4a ssl ssse3 svg taglib test-rust tga theora threads tiff truetype udev udisks unicode upnp upower usb userlocales v4l v4l2 vaapi vcd vdpau vhosts vnc vorbis vpx vulkan wavpack wayland webkit webp wimax win32codecs wmf wxwindows x264 x265 xanim xattr xcomposite xinerama xmp xop xv xvid xvmc zlib zstd" ABI_X86="64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" CAMERAS="*" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="3dnow 3dnowext mmx sse sse2 * mmxext" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" GRUB_PLATFORMS="efi-64 pc" INPUT_DEVICES="libinput evdev mutouch synaptics" KERNEL="linux" L10N="en be bg bs cs da de de-1901 el es es-AR es-CL es-ES es-MX et fi fr fr-CA hr hu it kk lt lv nl pl pt pt-BR pt-PT ro ru sk sl sr sq sv uk" LCD_DEVICES="* bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PERL_FEATURES="ithreads" PHP_TARGETS="php8-2" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31 ruby32" THEMES="CLDX" VIDEO_CARDS="nouveau nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
2

Hi! Try run cl-update --sync-only ; emerge -uDNa world.

3

Thanks Lautre for your suggestion!

4

The commands were unable to sync or emerge world.

cl-update has message saying it can’t complete due a utf-8 error
Locale was checked and regenerated with locale-gen.
utf-8 was still present.
cl-update error:

System configuration
 * Update server https://mirror.cnet.kz
 * Fixing the settings |
    cl_templ.applyTemplates()
   File "/usr/lib/python3.11/site-packages/calculate/lib/cl_template.py", line 75, in wrapper
    return f(*args, **kw)
           ^^^^^^^^^^^^^^
   File "/usr/lib/python3.11/site-packages/calculate/lib/cl_template.py", line 89, in wrapper
    return f(self, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
   File "/usr/lib/python3.11/site-packages/calculate/lib/cl_template.py", line 4644, in applyTemplates
    self.cltObj.applyTemplatesClt()
   File "/usr/lib/python3.11/site-packages/calculate/lib/cl_template.py", line 6886, in applyTemplatesClt
    if self.scanningTemplates(dirTemplate, self._chrootDir) is False:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/lib/python3.11/site-packages/calculate/lib/cl_template.py", line 6716, in scanningTemplates
    if not self.scanningTemplates(absPath, prefix, True):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/lib/python3.11/site-packages/calculate/lib/cl_template.py", line 6710, in scanningTemplates
    self.hasBelong(absPath)):
    ^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/lib/python3.11/site-packages/calculate/lib/cl_template.py", line 6679, in hasBelong
    s = f.readline()
        ^^^^^^^^^^^^
   File "<frozen codecs>", line 322, in decode
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8e in position 35: invalid start byte

emerge error:
using 'top" emerge hangs on this:

/usr/lib/python-exec/python3.11/python3 /usr/sbin/cl-core --method core_setup --no-progress --pkg-version 2.15 --pkg-slot 0 --pkg-category sys-apps --pkg-path / --pkg-name baselayout

As emerge can’t complete the file upgrade, next step will be manually extract the files from the .xpak to bypass the hang in python
sys-apps/portage, calculate-utils and calculate-toolkit
then system packages.
After this, I am out of ideas.

You thoughts on this are very much appreciated!
Thanks in advance!

5

Have you tried ‘emaint -a sync’?

6

Hi! Is the problem still actual?

7

Hi akrasnyh
yes, problem is still there.
I stopped repairing and moved to another machine.
I resume fixing it later this month.
and will post a report.
cheers!

8

Maybe something with the locale?
$ cat /etc/env.d/02locale
$ locale -a

Update from another mirror?

9

good
Thanks Lautre for your suggestion!