Today I Learned That ... on Josia's Blog

ArgoCD is like the Kubernetes Dashboard but for GitOps

Thu, 25 Aug 2022 15:00:09 +0200

FluxCD was my first love in the GitOps space. I worked with it when it was still v1, had no chance to use, and since about a year I returned to using it in its glorious, rewritten v2.

I haven’t given ArgoCD a try so far until this week somebody suggested trying out ArgoCD Autopilot. The process was super smooth and in less than 10 minutes I had a running ArgoCD dashboard in my local minikube cluster.

And holy moly I have to say: ArgoCD is awesome! In the less than one hour I have seen it close up now, I already see so many possibilities that are hard to do when you try to scale Flux:

Having a UI at all
Managing applications in multiple clusters from 1 Git repo with explicit assignment to these clusters
The UI is basically the better Default Kubernetes Dashboard but for GitOps:
- The Kubernetes Dashboard gives you a very “relational” view of your resources, much like k9s does (albeit with fewer resource types).
- I tried Kubevious in some projects because it allows you to view related resources hierarchically together.
  - But it is pretty buggy (especially the mysql DB deployed with it) and has not fulfilled its promise for me.
  - Additionally, it’s read-only and does not show you a Events or container logs.
- ArgoCD however gives us the best of both worlds:
  - Related resources are shown hierarchically (e.g. Deployments create ReplicaSets create Pods).
  - You can even group them into Apps, ApplicationSets, and Projects.
  - If you open a Resource (e.g. Pod), you can see its Events and also its logs!
Preview Environments seem relatively straightforward.
I have heard many rumors that Progressive Delivery is easier with ArgoCD (or at least involves less manually scripted yaml generation pipelines).

As a Cloud Automation Engineer, one of my main missions is to deliver stable apps. And this highly involves stable processes for my team. ArgoCD feels like the perfect fit: I can deploy my apps with ArgoCD and everybody with access to it can debug resources, see which versions are currently deployed, and maybe even promote apps to the next environment!

There also seem to be use cases (probably especially in a setup with separate platform and app dev teams) where deploying FluxCD and ArgoCD together makes sense. But I am definitively going to give ArgoCD a shot in some important environment soon.

You can Gitopsify your Ansible

Tue, 01 Mar 2022 08:34:20 +0100

I have a dotfiles repository for setting up my MacOS machine easily and reproducibly. For this I am mostly using Ansible, just slightly wrapping it with Make and seasoning it with a pinch of Shell.

The core piece of this repository is a Brewfile and a collection of Shell aliases and functions. I don’t regularly run the Ansible playbook itself because I created it for initially setting up a machine. The only thing I run very frequently is the upgrade command which includes persisting new Brew formulae to the Brewfile.

At the same time I have come to love applying GitOps principles to everything I work with. Just the other day I was moving some commands (in the vein of “global Git pull”) out of said upgrade command into a cronjob.

And then it hit me: If I create a cronjob for running my Ansible playbook through the playbook itself, I have GitOps-like reconciliation!

Let’s do it in just a few lines of yaml:

- name: Ensure dotfiles are applied
 cron:
 name: Ensure dotfiles are applied
 minute: "0"
 hour: "9"
 weekday: "1-5" # on workdays
 job: make -f "{{ ansible_env.PWD }}"/Makefile build

kubectl can explain all resource types in detail

Tue, 01 Feb 2022 14:53:18 +0100

$ kubectl explain statefulset
KIND: StatefulSet
VERSION: apps/v1

DESCRIPTION:
 StatefulSet represents a set of pods with consistent identities. Identities
 are defined as:
 - Network: A single stable DNS and hostname.
 - Storage: As many VolumeClaims as requested. The StatefulSet guarantees
 that a given network identity will always map to the same storage identity.

FIELDS:
 apiVersion	<string>
 APIVersion defines the versioned schema of this representation of an
 object. Servers should convert recognized schemas to the latest internal
 value, and may reject unrecognized values. More info:
 https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

 kind	<string>
 Kind is a string value representing the REST resource this object
 represents. Servers may infer this from the endpoint the client submits
 requests to. Cannot be updated. In CamelCase. More info:
 https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

 metadata	<Object>

 spec	<Object>
 Spec defines the desired identities of pods in this set.

 status	<Object>
 Status is the current status of Pods in this StatefulSet. This data may be
 out of date by some window of time.

I guess this will reduce a lot of my googling in future.

And it even works on nested fields!

$ kubectl explain deployment.spec.template.spec.containers
KIND: Deployment
VERSION: apps/v1

RESOURCE: containers <[]Object>

DESCRIPTION:
 List of containers belonging to the pod. Containers cannot currently be
 added or removed. There must be at least one container in a Pod. Cannot be
 updated.

 A single application container that you want to run within a pod.

FIELDS:
 args	<[]string>
 Arguments to the entrypoint. The docker image's CMD is used if this is not
 provided. Variable references $(VAR_NAME) are expanded using the
 container's environment. If a variable cannot be resolved, the reference in
 the input string will be unchanged. The $(VAR_NAME) syntax can be escaped
 with a double $$, ie: $$(VAR_NAME). Escaped references will never be
 expanded, regardless of whether the variable exists or not. Cannot be
 updated. More info:
 https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell

 command	<[]string>
 Entrypoint array. Not executed within a shell. The docker image's
 ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME)
 are expanded using the container's environment. If a variable cannot be
 resolved, the reference in the input string will be unchanged. The
 $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME).
 Escaped references will never be expanded, regardless of whether the
 variable exists or not. Cannot be updated. More info:
 https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell

 env	<[]Object>
 List of environment variables to set in the container. Cannot be updated.

 ...

Trying to simplify k8s labels can be dangerous for your routing

Fri, 03 Dec 2021 10:43:01 +0100

Today I was refactoring a bigger configuration setup that is built with kustomize. I see kustomize as a light-weight way of packaging multiple Kubernetes manifests, together with a little bit of logic.

The base kustomization of the application config repo I was refactoring looked something like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - manifests/cronjob-backup.yaml
 - manifests/deployment.yaml
 - manifests/ingress.yaml
 - manifests/pod-disruption-budget.yaml
 - manifests/service.yaml
commonAnnotations:
 source: java
 tenant: acme
configMapGenerator:
 - name: acme-app-config
 files:
 - configs/10-local.properties
 - configs/30-local.properties
 - configs/40-local.properties

I have an almost pathologic tendency to simplify and DRY everything up that I find, especially in configuration code. And as I saw a lot of repeating Labels in the manifests, I thought “Well, let’s just unify them.”

So I went ahead, removed the repetitive Labels from the manifests and added the following block to the kustomization:

commonLabels:
 app: acme-app

I knew a bit about Kubernetes Services and that they use Labels to find the Pods to which they should direct their traffic. That’s why I thoroughly verified that after my change the Service and the Deployment would still have the same selector labels. 🤗 Nothing seemed off …

I committed and pushed my changes, and after the Deployment had finished restarting, I clicked through the application. 👀 Oddly enough some of the requests succeeded as expected, but some kept failing with a 💥 502 Bad Gateway error! At first I tried troubleshooting quickly, but soon I opted for just reverting my changes and pushing the revert commits to undo my changes.

For debugging I compared the output of kustomize build before and after my changes. After some scrolling I came across the CronJobs I defined for backup¹.

I should probably not have been surprised that the CronJob also had the same Labels I gave it via the kustomization. But now a suspicion started sneaking in:

What if the completed Pods of the CronJob received traffic from the Service because they had the same Labels?

Following this idea, I refactored my configs a bit. Soon, I was able to ensure that requests to the Service would only point to my target Deployment (and not to any other Pods):

diff --git a/base/kustomization.yaml b/base/kustomization.yaml
index 6c1c6f7..b77b0fc 100644
--- a/base/kustomization.yaml
+++ b/base/kustomization.yaml
@@ -15,7 +15,6 @@ commonAnnotations:
 source: java
 tenant: acme
-commonLabels:
- app: acme-app
 configMapGenerator:
 - name: acme-app-config
diff --git a/base/manifests/deployment.yaml b/base/manifests/deployment.yaml
index 1e4aa3f..a60a798 100644
--- a/base/manifests/deployment.yaml
+++ b/base/manifests/deployment.yaml
@@ -16,6 +16,9 @@ metadata:
 spec:
 replicas: 1
+ selector:
+ matchLabels:
+ app: acme-app
 template:
 metadata:
 labels:
+ app: acme-app
 spec:
 containers:
diff --git a/base/manifests/service.yaml b/base/manifests/service.yaml
index b05d898..89af021 100644
--- a/base/manifests/service.yaml
+++ b/base/manifests/service.yaml
@@ -10,3 +10,5 @@ spec:
 - name: api
 port: 8080
+ selector:
+ app: acme-app

(Notice that the Deployment needs the Label both in .spec.selector.matchLabels and .spec.template.metadata.labels!)

And surely enough, after deploying this fix, the request to the application worked flawlessly 😊✅.

It does not matter for which purpose I created this CronJob, it could have been any Kubernetes Resource that creates Pods. ↩︎

Kubernetes is still willing to show you Dangling Resources in your Terminating Namespace

Fri, 29 Oct 2021 12:11:47 +0200

kubectl api-resources --verbs=list --namespaced -o name \
 | xargs -n 1 kubectl get --show-kind --ignore-not-found -n "$NAMESPACE"

Props to RedHat.

HTML limits you to a signed 32-bit int in ordered lists

Thu, 02 Sep 2021 08:55:22 +0200

HTML allows you to specify a starting number from which an ordered list (<ol>) should start. I thought “Is there an upper bound?”.

Turns out: Yes, there is. It’s 2147483647 (i.e. 2³¹-1 ). Looks like a signed 32-bit integer to me.

<ol start=2147483645>
 <li>I am still in order</li>
 <li>As am I</li>
 <li>Me too</li>
 <li>💥 Limit reached</li>
 <li>💥 Limit reached</li>
 <li>💥 Limit reached</li>
</ol>

The above snippet will render as the following:

I am still in order
As am I
Me too
💥 Limit reached
💥 Limit reached
💥 Limit reached

You can create TOTP tokens via CLI without a smartphone

Fri, 27 Aug 2021 08:20:41 +0200

All you need is the secret of your TOTP.

The QR code is just a representation of a oath:// URL That URL contains the secret as a query parameter.

# Install oathtool
brew install oathtool
# Use your secret, e.g. as a base32-encoded string
oathtool --totp --base32 "MFRGCZDTMVRXEZLUBI======"

Docker does not really help you a lot trying to get permissions right

Fri, 23 Jul 2021 14:42:07 +0200

I really love Docker, and I also come to like security more and more. One advice I have been hearing a lot (e.g. in this Container Security Cheat Sheet from Snyk) is that you should not run your container as a root user.

“Easy thing,” I thought to myself, “I am just going to put something like USER {app}” at the top of my Dockerfile." Well, think again, because:

FROM node:lts

USER node
# I would have thought that after this point, every action will happen in the name of this user
# and also that every created directory and file will belong to this user ... 😕 But:

# ⚠️ This directory is created by root:root!
WORKDIR /app

# ⚠️ These files will be copied over to be owned by root:root!
COPY package*.json ./

# 💥 This step fails in some (not all!) environments with errors like "Not enough permissions on /app"
RUN npm install && \
 npm run verify

# ⚠️ If you manage to get to this point, these files, too, will be copied over to be owned by root:root!
COPY . .

ENTRYPOINT ["npm"]

I ended up fixing it by creating the directory and then chown-ing it. Equally I executed the COPY instructions with the --chown flag. In the end I refactored it a bit using some ENVs, too:

FROM node:lts

# Ensure that target WORKDIR exists and is owned by target (non-root) user
ENV USERNAME=node
ENV USERID=$USERNAME:$USERNAME
ENV TARGETDIR=/app
RUN mkdir -p $TARGETDIR && chown -R $USERID $TARGETDIR
WORKDIR $TARGETDIR

USER $USERNAME

COPY --chown=$USERID package*.json ./

RUN npm install && \
 npm run verify

COPY --chown=$USERID . .

ENTRYPOINT ["npm"]

You can :wq in vim with :x

Fri, 16 Jul 2021 00:00:00 +0000

I have developed quite some muscle memory in terminating vim via :wqa. Now I can save one more character by retraining myself to use :xa! 😄

I also like how :x sort of symbolizes the close button ❌ of many window managers and their default behavior of saving when you close a file.

For more documentation open vim and type :h :x.

Wait, now that you mention it …

Well, turns out the documentation mentions an even easier way: Just press ZZ (without the colon) 😴

Now if only I could combine this to perform :wqa …

Happy vim-ing! 👋