Dream IT Enterprises

Category: developer

  • k8s version upgrade nuisance – apt’s extreme configurability to the rescue

    Ever since last year’s migration of kubernetes’ APT repo to community repository, every kubernetes upgrade has become a minor nuisance for people like me who have proxy repositories sitting in between the k8s apt repo and actual target kubernetes nodes.

    Before migration to community apt repository, the old apt-repository – apt.kubernetes.io – was a single repository for all kubernetes versions toghether. This was simple to get proxied as we can just add apt.kubernetes.io repo in nexus / artifactory and use that proxy repo path as our apt repositories for machines.

    Now, in new world, each k8s version has it’s own repo e.g. https://pkgs.k8s.io/core:/stable:/v1.31/deb/. I cannot fathom the reason for such decision!

    So each time you are going to upgrade k8s version, you need to update the nexus apt proxy repository configuration to update the repository upstream path to have correct upstream version. This also means, we cannot properly have multiple kubernetes versions supported via proxy properly.

    While I have grown accustomed to this nuisance in last few months, recently, I stared observing another blocker issue – espl when I started upgrading OS for our kubernetes nodes to Ubuntu 24.04.

    apt version in Ubuntu 24.04 OS is more strict about where is it fetching packages from, and how is upstream behaving etc. So I started getting new errors when trying to use apt with nexus upstream.

    Error 1 – Enforcement of signature verification

    W: GPG error: https://nexus.xyz.com/repository/k8s-package-proxy jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
W: GPG error: https://nexus.xyz.com/repository/k8s-package-new-proxy  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436

    You can find the new requirements of apt failing with error rather than warning documented in this thread.

    As unfortunate as it is, nexus apt plugin does not proxy the public key from upstream.. not does it provide it’s own. So if you are using nexus as apt proxy, you must work around above error. See below how it can be done.

    Error 2 – Complaining that I have changed my proxy repo to repoint to new k8s version

    As noted above, with every kubernetes version, I had to update location in nexus apt proxy configuration. But with Ubuntu 24.04, this change started breaking with below errors on node OS upgrade.

    E: Repository 'https://nexus.xyz.com/repository/k8s-package-new-proxy  InRelease' changed its 'Origin' value from 'obs://build.opensuse.org/isv:kubernetes:core:stable:v1.29/deb' to 'obs://build.opensuse.org/isv:kubernetes:core:stable:v1.30/deb'
E: Repository 'https://nexus.xyz.com/repository/k8s-package-new-proxy  InRelease' changed its 'Label' value from 'isv:kubernetes:core:stable:v1.29' to 'isv:kubernetes:core:stable:v1.30'

    After running around searching for solution, I finally got solution to work around both the issues. I was thankful that I was dealing with a software written to address all such special needs via configuration while keeping sensible defaults and also one with great amount of documentation and huge community of users and mainteners!

    Solution(s)

    apt allowed me to create a new apt-conf file. e.g. /etc/apt/apt.conf.d/99-allow-nexus with below content:

    # NOTE: you should use below settings after careful consideration.
# If used incorrectly, you can trust a malicious online apt repository to install packages in your environment! You have been warned!!
# Fix issue 1 - ignore the absense of signature for apt update
Acquire::AllowInsecureRepositories "true";
# Fix issue 1 - ignore the absense of signature for apt install
APT::Get::AllowUnauthenticated "true";
# Fix issue 2 - Ignore the fact that upstream has changes various release information like Label, release, etc.
Acquire::AllowReleaseInfoChange "true";

    Once we provisioned above config file to our machines, all our machines got upgraded to Ubuntu 24.04 (and our k8s packages installed on them) without a hitch!

    … so configurability in apt package saved the day!

    Hope you enjoyed reading this quick bite!

    Now – I need to run to k8s infra mainteners to ask them why we cannot have a single root apt repository for all the versions! Until then – bye!

  • Byobu Tips

    I use Byobu as my default terminal multiplexer for many years now. Best part is Byobu comes pre-installed in ubuntu. My experience is largely with Ubuntu so I am not sure if CentOS/RHEL and other Linux OS provide this package OOTB or not.

    Internally Byobu uses tmux as the multiplexer. So many of the features are also inherited from tmux but basically, byobu makes it so much easier to exploit the tmux features in those cases.

    Some of the not-so-obvious features that I like about Byobu are:
    – save scrollback buffer to file! You can use printscreen facility in Byobu. Hit Ctrl + F7 to open current window history in your editor. Then you can save that file somewhere for your records. Very handy when you are making some changes and want to keep record of what happened.
    – rename the tabs. You can create multiple tabs / windows using F2 key and navigate between them using F3 and F4 keys. But those tabs usually get just number index. But you can F8 to rename those tabs to have more meaningful names like dev and prod etc.
    – Resize the panes in the given tab – In a given tab of byobu can create further split screens using Ctrl+F2 and Shift+F2 keys. Those will split current screen area into vertical or horizontal half. But then sometimes, you want to resize these panes to have better viewing area for some tasks at hand. You can use Shift + Alt +arrow keys to resize those panes.
    – Re-arrange the panes – Sometimes, you are not happy that you created panes by splitting screen horizontally, then you use Shift+F8 (sometimes twice) to cycle through various different layout changes offered by Byobu to suit a new layout for yourself.

    For now, that’s it. Do you have your favourite trick for Byobu? leave a message as I would like to learn it as well!

  • Add `allowVolumeExpansion: true` to your storage class

    StorageClass

    If you have a Kubernetes cluster running for couple of years, you would probably have legacy StorageClass defined which does not contain allowVolumeExpansion: true bit set. If this bit is not set on your storageClass, then you cannot expand your PVCs which are using this storageClass.

    But, if you try to edit storage class to add this attribute to the specs, kubernetes does not allow you to do so saying that this field is immutable and cannot be changed after definition.

    So now, how do we expand these PVCs which are created using legacy storage class without allowVolumeExpansion bit set?

    It turns out – its quite easy. You can delete your old storage class without affecting your PVCs. And recreate it again, with same name, but this time with allowVolumeExpansion: true in the sepcs.

    Now the old PVCs which were creating using the said StorageClass can now be expanded, automatically!!

    That’s it for today!

  • push / pull images without Docker

    Increasingly, docker is being replaced with containerd in kubernetes clusters. These days, in my daily work, I am dealing with clusters almost exclusively with containerd as CRI. In such cases, sometimes there is a need to pull or push some images to say, internal / private registries. without docker cli installed and docker daemon running, how do we achieve this? Read on.

    Option 1 – use ctr – the containerd CLI

    ContainerD gets installed with its own CLI – ctr – which is lot more minimal than docker cli. But fortunately, push and pull commands are available. I am yet to find a way to see the logs and exec into a containers ( at least in v1.4)

    # pull image using ctr 
sudo ctr i pull docker.io/utkuozdemir/pv-migrate-sshd:1.0.0

# tag the image for your internal container registry
sudo ctr i tag docker.io/utkuozdemir/pv-migrate-sshd:1.0.0 my-internal-nexus:5002/pv-migrate-sshd:1.0.0

# Push to internal container registry server
sudo ctr i push --platform linux/amd64 my-internal-nexus:5002/pv-migrate-sshd:1.0.0

    One thing to remember about ctr is that it expects that you have image for all the platforms to push. If you have downloaded the image, it would only be for one platform say.. linux-amd64. So in push command, we must specify --platform linux/amd64. If we don’t do this, ctr push command fails with very cryptic error about not having image content.

    Option 2 – use crictl

    kubernetes CRI team has created crictl which you can download and install and it provides docker like CLI to deal with containers.

    crictl is certainly much easier to use than ctr but downloading arbitary things on your servers is not an option in many places, rightfully, due to security concerns. In such cases, ctr is the only option you are left with.

    That’s it for now.