Using AKS kubelogin for ArgoCD external clusters

In this brief article, we look at the process of adding Azure Kubernetes Service (AKS) to ArgoCD as an external cluster. When AKS is integrated with Azure Active Directory, it requires the use of kubelogin for non-interactive scenarios. We’ll cover the use of both “argocd cluster add” CLI and the “declarative setup” methods.

As of July 2023, ArgoCD version 2.7.7 did not yet support Kubelogin through the built-in argocd-k8s-auth command plugin. Nevertheless, Pull Request #10700, which targets this functionally, has been proposed for merging in the forthcoming release 2.8-rc1.

In the meantime, before this PR is incorporated, a workaround is available. This involves using the execProviderConfig to invoke the ‘kubelogin get-token’ command. This command fetches client-go credential (exec) plugin token from Azure AD.

ArgoCD Custom Tooling for Kubelogin

In order to be able to execute the kubelogin command, it needs to be included as a custom tool in both “argocd-server” deployment and “argocd-application-controller” stateful set. One way to download ArgoCD custom tools is using initContainers and volume mounts.

This process includes downloading the YAML of both ‘argocd-server’ and ‘argocd-application-controller’. These YAML files are then modified to incorporate the initContainer, volume, and volumeMounts. Following these modifications, the updates are applied to the running ArgoCD

kubectl get statefulset argocd-application-controller -n argocd -o yaml > argocd-application-controller.yaml
kubectl get deployment argocd-server -n argocd -o yaml > argocd-server.yaml

spec:
    ...
    ...
    volumes:
    - name: custom-tools
      emptyDir: {}
    initContainers:
      - name: download-tools
        image: alpine:3.8
        command: [sh, -c]
        args:
        - >-
            wget -qO- https://github.com/Azure/kubelogin/releases/latest/download/kubelogin-linux-amd64.zip |
            unzip -x bin/linux_amd64/kubelogin -p - > /custom-tools/kubelogin &&
            chmod a+x /custom-tools/kubelogin
        volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    ...
    ...
    containers:
      - name: argocd-server OR argocd-application-controller
        volumeMounts:
        - mountPath: /usr/local/bin/kubelogin
          name: custom-tools
          subPath: kubelogin

kubectl apply -f argocd-application-controller.yaml
kubectl apply -f argocd-server.yaml

After kubelogin command is available within the ArgoCD container, we can use it for the execCommand of the external clusters.

ArgoCD CLI

One method involves using the imperative ‘argocd cluster add’ CLI tool. This tool adds the external cluster with the relevant ‘exec-command’ configuration. The values for client-id and secret correspond to the Service Principal that we created earlier. We had previously assigned proper roles to this Service Principal for AKS using Azure RBAC.

argocd login $ARGO_IP

argocd cluster list

# The exec-command-args must be specified in a specific way below in order to work
argocd cluster add NAME-OF-KUBECONFIG-CLUSTER --name aks-kubelogin --cluster-endpoint "kubeconfig" \
    --exec-command-api-version "client.authentication.k8s.io/v1beta1"  \
    --exec-command "/usr/local/bin/kubelogin" \
    --exec-command-args "get-token" \
    --exec-command-args "--login=spn" \
    --exec-command-args "--server-id=6dae42f8-4368-4678-94ff-3960e28e3630" \
    --exec-command-args "--client-id=00000000-0000-0000-0000-000000000000" \
    --exec-command-args "--tenant-id=00000000-0000-0000-0000-000000000000" \
    --exec-command-env "AAD_SERVICE_PRINCIPAL_CLIENT_SECRET=SECRET_VALUE" \
    --exec-command-install-hint "kubelogin missing" \

# If command above fails, review the logs of the argocd-server pod to understand what happened
kubectl logs argocd-server-666b6cfbb7-7jv9k -n argocd

ArgoCD Declarative Setup

An alternative method involves using a declarative setup to add the external cluster. This can be achieved by creating a properly annotated and structured Secret in ArgoCD.”

cluster-kubelogin.yaml

apiVersion: v1
kind: Secret
metadata:
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: cluster
  name: aks-kubelogin
  namespace: argocd
type: Opaque
stringData:
  name: aks-kubelogin
  server: https://my-aks-cluster-url
  config: |
    {
      "execProviderConfig": {
        "apiVersion": "client.authentication.k8s.io/v1beta1",
        "command": "/usr/local/bin/kubelogin",
        "args": [
          "get-token",
          "--login",
          "spn",
          "--server-id",
          "6dae42f8-4368-4678-94ff-3960e28e3630",
          "--client-id",
          "00000000-0000-0000-0000-000000000000",
          "--tenant-id",
          "00000000-0000-0000-0000-000000000000",
          "--environment",
          "AzurePublicCloud"
        ],
        "env": {
          "AAD_SERVICE_PRINCIPAL_CLIENT_SECRET": "SECRET_VALUE"
        },
        "installHint": "kubelogin missing"
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": "cluster.certificate-authority-data from kubeconfig"
      }
    }

# Add external cluster using declarative approach
kubectl apply -f cluster-kubelogin.yaml -n argocd

ArgoCD UI

After adding the external cluster using either the CLI or declarative methods, it should be visible in the ArgoCD UI, as illustrated below.

The external cluster can now be used to deploy the applications.

Thank you!

Please leave feedback and questions below or on Twitter @ArsenVlad.