Dream IT Enterprises

Tag: kubernets

  • k8s version upgrade nuisance – apt’s extreme configurability to the rescue

    Ever since last year’s migration of kubernetes’ APT repo to community repository, every kubernetes upgrade has become a minor nuisance for people like me who have proxy repositories sitting in between the k8s apt repo and actual target kubernetes nodes.

    Before migration to community apt repository, the old apt-repository – apt.kubernetes.io – was a single repository for all kubernetes versions toghether. This was simple to get proxied as we can just add apt.kubernetes.io repo in nexus / artifactory and use that proxy repo path as our apt repositories for machines.

    Now, in new world, each k8s version has it’s own repo e.g. https://pkgs.k8s.io/core:/stable:/v1.31/deb/. I cannot fathom the reason for such decision!

    So each time you are going to upgrade k8s version, you need to update the nexus apt proxy repository configuration to update the repository upstream path to have correct upstream version. This also means, we cannot properly have multiple kubernetes versions supported via proxy properly.

    While I have grown accustomed to this nuisance in last few months, recently, I stared observing another blocker issue – espl when I started upgrading OS for our kubernetes nodes to Ubuntu 24.04.

    apt version in Ubuntu 24.04 OS is more strict about where is it fetching packages from, and how is upstream behaving etc. So I started getting new errors when trying to use apt with nexus upstream.

    Error 1 – Enforcement of signature verification

    W: GPG error: https://nexus.xyz.com/repository/k8s-package-proxy jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
W: GPG error: https://nexus.xyz.com/repository/k8s-package-new-proxy  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436

    You can find the new requirements of apt failing with error rather than warning documented in this thread.

    As unfortunate as it is, nexus apt plugin does not proxy the public key from upstream.. not does it provide it’s own. So if you are using nexus as apt proxy, you must work around above error. See below how it can be done.

    Error 2 – Complaining that I have changed my proxy repo to repoint to new k8s version

    As noted above, with every kubernetes version, I had to update location in nexus apt proxy configuration. But with Ubuntu 24.04, this change started breaking with below errors on node OS upgrade.

    E: Repository 'https://nexus.xyz.com/repository/k8s-package-new-proxy  InRelease' changed its 'Origin' value from 'obs://build.opensuse.org/isv:kubernetes:core:stable:v1.29/deb' to 'obs://build.opensuse.org/isv:kubernetes:core:stable:v1.30/deb'
E: Repository 'https://nexus.xyz.com/repository/k8s-package-new-proxy  InRelease' changed its 'Label' value from 'isv:kubernetes:core:stable:v1.29' to 'isv:kubernetes:core:stable:v1.30'

    After running around searching for solution, I finally got solution to work around both the issues. I was thankful that I was dealing with a software written to address all such special needs via configuration while keeping sensible defaults and also one with great amount of documentation and huge community of users and mainteners!

    Solution(s)

    apt allowed me to create a new apt-conf file. e.g. /etc/apt/apt.conf.d/99-allow-nexus with below content:

    # NOTE: you should use below settings after careful consideration.
# If used incorrectly, you can trust a malicious online apt repository to install packages in your environment! You have been warned!!
# Fix issue 1 - ignore the absense of signature for apt update
Acquire::AllowInsecureRepositories "true";
# Fix issue 1 - ignore the absense of signature for apt install
APT::Get::AllowUnauthenticated "true";
# Fix issue 2 - Ignore the fact that upstream has changes various release information like Label, release, etc.
Acquire::AllowReleaseInfoChange "true";

    Once we provisioned above config file to our machines, all our machines got upgraded to Ubuntu 24.04 (and our k8s packages installed on them) without a hitch!

    … so configurability in apt package saved the day!

    Hope you enjoyed reading this quick bite!

    Now – I need to run to k8s infra mainteners to ask them why we cannot have a single root apt repository for all the versions! Until then – bye!