今年小弟處理不少從 Kubernetes 各種遷移到 AWS 的案例，有的是整座移轉、有的是先遷移部分，不外乎都是希望透過 Managed service 來減少維運成本，其中最大宗的是自建 Harbor 最後受不了想轉到 AWS ECR 而過程中有一些技巧還是記錄在這裡方便查詢。

原本 Kubernetes 就支援自訂義 docker-registry 所以即使是 AWS ECR 或 Docker Hub 都不是問題，以 AWS private ECR repositroy 為例；建立 AWS ECR repository 時會拿到一個 image repository：<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/<image-name>:<tag> 但是要存取 private repository 必須要有 AWS 權限才能訪問，所以必須先設定 secret docker-registry：

$ kubectl create secret docker-registry regcred \
       --docker-server=<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com \
       --docker-username=AWS \
       --docker-password=$(aws ecr get-login-password) \
       --namespace=default

然後在 deployment 時引用 imagePullSecrets 就可以用對應的 registry secret 來拉 AWS ECR private repository。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-2048
  namespace: default
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
      - image: <aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/<image-name>:<tag>
        imagePullPolicy: Always
        name: app-2048
        ports:
        - containerPort: 80
      imagePullSecrets:
          - name: regcred

這時眼尖的你可能發現到 secret docker-registry 裡的 docker-password 是一組臨時的 AWS ECR Token 時效最多只有 12 小時，如果沒有定期 rotate 那麼當 Token 失效就會發現 Pod ImagePullBackOff 因為權限不足而拉不了 AWS private ECR repo。

所以必須要有個定期運作的機制來更新 AWS ECR Token，做法很多提供幾種給各位參考：

在 Worker nodes 運行 cronjob 腳本定期汰換 regcred secret

#!/bin/bash
NAMESPACE=($(kubectl get secret --all-namespaces | grep regcred | awk '{print $1}'))

for i in "${NAMESPACE[@]}"
do
  :
  kubectl delete secret regcred --namespace $i
  kubectl create secret docker-registry regcred \
  --docker-server=${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password) \
  --namespace=$i
done

在 Kubernetes 運行 CronJob 定期汰換 regcred secret

apiVersion: v1
kind: Secret
metadata:
  name: ecr-registry-helper-secrets
  namespace: default
stringData:
  AWS_SECRET_ACCESS_KEY: "xxxx"
  AWS_ACCESS_KEY_ID: "xxx"
  AWS_ACCOUNT: "xxx"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ecr-registry-helper-cm
  namespace: default
data:
  AWS_REGION: "us-east-1"
  DOCKER_SECRET_NAME: regcred
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: ecr-registry-helper
  namespace: default
spec:
  schedule: "0 */10 * * *"
  successfulJobsHistoryLimit: 3
  suspend: false
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: private-ecr-sa-default
          containers:
          - name: ecr-registry-helper
            image: odaniait/aws-kubectl:latest
            imagePullPolicy: IfNotPresent
            envFrom:
              - secretRef:
                  name: ecr-registry-helper-secrets
              - configMapRef:
                  name: ecr-registry-helper-cm
            command:
              - /bin/sh
              - -c
              - |-
                ECR_TOKEN=`aws ecr get-login-password --region ${AWS_REGION}`
                NAMESPACE_NAME=default
                kubectl delete secret --ignore-not-found $DOCKER_SECRET_NAME -n $NAMESPACE_NAME
                kubectl create secret docker-registry $DOCKER_SECRET_NAME \
                --docker-server=https://${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com \
                --docker-username=AWS \
                --docker-password="${ECR_TOKEN}" \
                --namespace=$NAMESPACE_NAME
                echo "Secret was successfully updated at $(date)"
          restartPolicy: Never
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: private-ecr-sa-default
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: role-full-access-to-secrets
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["regcred"]
  verbs: ["delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: private-ecr-sa-default
  namespace: default
subjects:
- kind: ServiceAccount
  name: private-ecr-sa-default
  namespace: default
  apiGroup: ""
roleRef:
  kind: Role
  name: role-full-access-to-secrets
  apiGroup: ""
---

Kubernetes Secret 上述是使用固定的 AWS AKSK 或是 IRSA 以取得臨時 ASKS 是更保險的方法。

References

• Pull the Docker Image from AWS ECR in Kubernetes
• Kubernetes – pull an image from private ECR registry. Auto refresh ECR token.

Amazon ECR AWS Kubernetes