Merge pull request #1206 from dod-ccpo/azure-key-vault

Initial Setup for Consuming FlexVol in AKS
2019-11-25 12:00:11 -05:00 · 2019-11-25 12:00:11 -05:00 · 4d72d8dece
commit 4d72d8dece
parent 018229dcd8 bc9e4fd142
5 changed files with 381 additions and 2 deletions
--- a/deploy/README.md
+++ b/deploy/README.md
@ -12,13 +12,15 @@ The production configuration (azure.atat.code.mil, currently) is reflected in th
 - PORT_PREFIX: "8" for production, "9" for staging
 - MAIN_DOMAIN: the host domain for the environment
 - AUTH_DOMAIN: the host domain for the authentication endpoint for the environment
+- KV_MI_ID: the fully qualified id (path) of the managed identity for the key vault (instructions on retrieving this are down in section on [Setting up FlexVol](#configuring-the-identity))
+- KV_MI_CLIENT_ID: the client id of the managed identity for the key vault

 We use envsubst to substitute values for these variables.

 To apply config to the main environment, you should first do a diff to determine whether your new config introduces unexpected changes:

 ```
-kubectl kustomize deploy/azure | CONTAINER_IMAGE=myregistry.io/atat-some-commit-sha PORT_PREFIX=8 MAIN_DOMAIN=azure.atat.code.mil AUTH_DOMAIN=auth-azure.atat.code.mil envsubst '$CONTAINER_IMAGE $PORT_PREFIX $MAIN_DOMAIN $AUTH_DOMAIN' | kubectl diff -f -
+kubectl kustomize deploy/azure | CONTAINER_IMAGE=myregistry.io/atat-some-commit-sha PORT_PREFIX=8 MAIN_DOMAIN=azure.atat.code.mil AUTH_DOMAIN=auth-azure.atat.code.mil KV_MI_ID=/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/RESOURCE_GROUP_NAME/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MANAGED_IDENTITY_NAME KV_MI_CLIENT_ID=00000000-0000-0000-0000-000000000000 envsubst '$CONTAINER_IMAGE $PORT_PREFIX $MAIN_DOMAIN $AUTH_DOMAIN $KV_MI_ID $KV_MI_CLIENT_ID' | kubectl diff -f -
 ```

 Here, `kubectl kustomize` assembles the config and streams it to STDOUT. We specify environment variables for envsubst to use and pass the names of those env vars as a string argument to envsubst. This is important, because envsubst will override NGINX variables in the NGINX config if you don't limit its scope. Finally, we pipe the result from envsubst to `kubectl diff`, which reports a list of differences. Note that some values tracked by K8s internally might have changed, such as [`generation`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#objectmeta-v1-meta). This is fine and expected.
@ -26,7 +28,7 @@ Here, `kubectl kustomize` assembles the config and streams it to STDOUT. We spec
 If you are satisfied with the output from the diff, you can apply the new config the same way:

 ```
-kubectl kustomize deploy/azure | CONTAINER_IMAGE=myregistry.io/atat-some-commit-sha PORT_PREFIX=8 MAIN_DOMAIN=azure.atat.code.mil AUTH_DOMAIN=auth-azure.atat.code.mil envsubst '$CONTAINER_IMAGE $PORT_PREFIX $MAIN_DOMAIN $AUTH_DOMAIN' | kubectl apply -f -
+kubectl kustomize deploy/azure | CONTAINER_IMAGE=myregistry.io/atat-some-commit-sha PORT_PREFIX=8 MAIN_DOMAIN=azure.atat.code.mil AUTH_DOMAIN=auth-azure.atat.code.mil KV_MI_ID=/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/RESOURCE_GROUP_NAME/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MANAGED_IDENTITY_NAME KV_MI_CLIENT_ID=00000000-0000-0000-0000-000000000000 envsubst '$CONTAINER_IMAGE $PORT_PREFIX $MAIN_DOMAIN $AUTH_DOMAIN $KV_MI_ID $KV_MI_CLIENT_ID' | kubectl apply -f -
 ```

 **Note:** Depending on how your `kubectl` config is set up, these commands may need to be adjusted. If you have configuration for multiple clusters, you may need to specify the `kubectl` context for each command with the `--context` flag (something like `kubectl --context=my-cluster [etc.]` or `kubectl --context=azure [etc.]`).
@ -165,3 +167,52 @@ Then:
 ```
 kubectl -n atat create secret tls azure-atat-code-mil-tls --key="[path to the private key]" --cert="[path to the full chain]"
 ```
+
+---
+
+# Setting Up FlexVol for Secrets
+
+## Preparing Azure Environment
+
+A Key Vault will need to be created. Save it's full id (the full path) for use later.
+
+## Preparing Cluster
+
+The 2 following k8s configs will need to be applied to the cluster. They do not need to be namespaced, the latter will create a `kv` namespace for itself.
+```
+kubectl apply -f deploy/azure/keyvault/deployment-rbac.yaml
+kubectl apply -f deploy/azure/keyvault/kv-flex-vol-installer.yaml
+```
+
+## Configuring The Identity
+
+1. Creat the identity in a resource group that is able to manage the cluster (`RESOURCE_GROUP_NAME`). You'll also determine a name for the identity (`MANAGED_IDENTITY_NAME`) that you'll use to refer to the identity later:
+
+    `az identity create -g <RESOURCE_GROUP_NAME> -n <MANAGED_IDENTITY_NAME> -o json`
+
+2. From the resulting JSON, we'll need to use: `id`, `clientId` and `principalId` for subsequent commands and configuration.
+Example values:
+    ```
+    id: "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/RESOURCE_GROUP_NAME/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MANAGED_IDENTITY_NAME"
+    clientId: 00000000-0000-0000-0000-000000000000
+    principalId: 00000000-0000-0000-0000-000000000000
+    ```
+
+    > You can recover these values later by running the following command. Verify you are looking at the correct identity by making sure the end of the first line (id) is the same as the name you provided above. If you want the full details of the identity, leave off the last section.
+    >
+    >`az identity list -g <RESOURCE_GROUP_NAME> | jq '.[] | select(.type == "Microsoft.ManagedIdentity/userAssignedIdentities") | .id, .clientId, principalId'`
+
+3. Assign the new identity two roles: Managed Identity Operator (for itself) and Reader (of the vault). The `VAULT_ID` can be found in the azure portal.
+
+    ```
+    az role assignment create --role "Managed Identity Operator" --assignee <principalId> --scope <id>
+    az role assignment create --role Reader --assignee <principalId> --scope <VAULT_ID>
+    ```
+
+4. Grant the identity get permissions for each of the types of values the vault can store, Keys, Secrets, and Certificates:
+    ```
+    az keyvault set-policy -n <VAULT_NAME> --spn <clientId> --key-permissions get --secret-permissions get --certificate-permissions get
+    ```
+
+5. The file `deploy/azure/aadpodidentity.yml` is templated via Kustomize, so you'll need to include clientId (as `KV_MI_CLIENT_ID`) and id (as `KV_MI_ID`) of the managed identity as part of the call to Kustomize.
+
--- a/deploy/azure/aadpodidentity.yml
+++ b/deploy/azure/aadpodidentity.yml
@ -0,0 +1,19 @@
+---
+apiVersion: "aadpodidentity.k8s.io/v1"
+kind: AzureIdentity
+metadata:
+  name: atat-kv-identity
+spec:
+  type: 0
+  ResourceID: $KV_MI_ID
+  ClientID: $KV_MI_CLIENT_ID
+---
+apiVersion: "aadpodidentity.k8s.io/v1"
+kind: AzureIdentityBinding
+metadata:
+  name: atat-key-vault-identity-binding
+spec:
+  AzureIdentity: atat-kv-identity
+  Selector: atat-kv-id-binding
+---
+
--- a/deploy/azure/keyvault/deployment-rbac.yml
+++ b/deploy/azure/keyvault/deployment-rbac.yml
@ -0,0 +1,259 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: aad-pod-id-nmi-service-account
+  namespace: default
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: azureassignedidentities.aadpodidentity.k8s.io
+spec:
+  group: aadpodidentity.k8s.io
+  version: v1
+  names:
+    kind: AzureAssignedIdentity
+    plural: azureassignedidentities
+  scope: Namespaced
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: azureidentitybindings.aadpodidentity.k8s.io
+spec:
+  group: aadpodidentity.k8s.io
+  version: v1
+  names:
+    kind: AzureIdentityBinding
+    plural: azureidentitybindings
+  scope: Namespaced
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: azureidentities.aadpodidentity.k8s.io
+spec:
+  group: aadpodidentity.k8s.io
+  version: v1
+  names:
+    kind: AzureIdentity
+    singular: azureidentity
+    plural: azureidentities
+  scope: Namespaced
+---
+apiVersion: apiextensions.k8s.io/v1beta1
+kind: CustomResourceDefinition
+metadata:
+  name: azurepodidentityexceptions.aadpodidentity.k8s.io
+spec:
+  group: aadpodidentity.k8s.io
+  version: v1
+  names:
+    kind: AzurePodIdentityException
+    singular: azurepodidentityexception
+    plural: azurepodidentityexceptions
+  scope: Namespaced
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: aad-pod-id-nmi-role
+rules:
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["pods"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get"]
+  - apiGroups: ["aadpodidentity.k8s.io"]
+    resources:
+      ["azureidentitybindings", "azureidentities", "azurepodidentityexceptions"]
+    verbs: ["get", "list", "watch"]
+  - apiGroups: ["aadpodidentity.k8s.io"]
+    resources: ["azureassignedidentities"]
+    verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: aad-pod-id-nmi-binding
+  labels:
+    k8s-app: aad-pod-id-nmi-binding
+subjects:
+  - kind: ServiceAccount
+    name: aad-pod-id-nmi-service-account
+    namespace: default
+roleRef:
+  kind: ClusterRole
+  name: aad-pod-id-nmi-role
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    kubernetes.io/cluster-service: "true"
+    component: nmi
+    tier: node
+    k8s-app: aad-pod-id
+  name: nmi
+  namespace: default
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  selector:
+    matchLabels:
+      component: nmi
+      tier: node
+  template:
+    metadata:
+      labels:
+        component: nmi
+        tier: node
+    spec:
+      serviceAccountName: aad-pod-id-nmi-service-account
+      hostNetwork: true
+      volumes:
+        - hostPath:
+            path: /run/xtables.lock
+            type: FileOrCreate
+          name: iptableslock
+      containers:
+        - name: nmi
+          image: "mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.5.3"
+          imagePullPolicy: Always
+          args:
+            - "--host-ip=$(HOST_IP)"
+            - "--node=$(NODE_NAME)"
+          env:
+            - name: HOST_IP
+              valueFrom:
+                fieldRef:
+                  fieldPath: status.podIP
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+          resources:
+            limits:
+              cpu: 200m
+              memory: 512Mi
+            requests:
+              cpu: 100m
+              memory: 256Mi
+          securityContext:
+            privileged: true
+            capabilities:
+              add:
+                - NET_ADMIN
+          volumeMounts:
+            - mountPath: /run/xtables.lock
+              name: iptableslock
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8080
+            initialDelaySeconds: 10
+            periodSeconds: 5
+      nodeSelector:
+        beta.kubernetes.io/os: linux
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: aad-pod-id-mic-service-account
+  namespace: default
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: aad-pod-id-mic-role
+rules:
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["*"]
+  - apiGroups: [""]
+    resources: ["pods", "nodes"]
+    verbs: ["list", "watch"]
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["create", "patch"]
+  - apiGroups: [""]
+    resources: ["endpoints"]
+    verbs: ["create", "get", "update"]
+  - apiGroups: ["aadpodidentity.k8s.io"]
+    resources: ["azureidentitybindings", "azureidentities"]
+    verbs: ["get", "list", "watch", "post"]
+  - apiGroups: ["aadpodidentity.k8s.io"]
+    resources: ["azureassignedidentities"]
+    verbs: ["*"]
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: aad-pod-id-mic-binding
+  labels:
+    k8s-app: aad-pod-id-mic-binding
+subjects:
+  - kind: ServiceAccount
+    name: aad-pod-id-mic-service-account
+    namespace: default
+roleRef:
+  kind: ClusterRole
+  name: aad-pod-id-mic-role
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    component: mic
+    k8s-app: aad-pod-id
+  name: mic
+  namespace: default
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      component: mic
+      app: mic
+  template:
+    metadata:
+      labels:
+        component: mic
+        app: mic
+    spec:
+      serviceAccountName: aad-pod-id-mic-service-account
+      containers:
+        - name: mic
+          image: "mcr.microsoft.com/k8s/aad-pod-identity/mic:1.5.3"
+          imagePullPolicy: Always
+          args:
+            - "--cloudconfig=/etc/kubernetes/azure.json"
+            - "--logtostderr"
+          resources:
+            limits:
+              cpu: 200m
+              memory: 1024Mi
+            requests:
+              cpu: 100m
+              memory: 256Mi
+          volumeMounts:
+            - name: k8s-azure-file
+              mountPath: /etc/kubernetes/azure.json
+              readOnly: true
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8080
+            initialDelaySeconds: 10
+            periodSeconds: 5
+      volumes:
+        - name: k8s-azure-file
+          hostPath:
+            path: /etc/kubernetes/azure.json
+      nodeSelector:
+        beta.kubernetes.io/os: linux
--- a/deploy/azure/keyvault/kv-flex-vol-installer.yml
+++ b/deploy/azure/keyvault/kv-flex-vol-installer.yml
@ -0,0 +1,49 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: kv
+---
+apiVersion: extensions/v1beta1
+kind: DaemonSet
+metadata:
+  labels:
+    app: keyvault-flexvolume
+  name: keyvault-flexvolume
+  namespace: kv
+spec:
+  updateStrategy:
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        app: keyvault-flexvolume
+    spec:
+      tolerations:
+      containers:
+        - name: flexvol-driver-installer
+          image: "mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.15"
+          imagePullPolicy: Always
+          resources:
+            requests:
+              cpu: 50m
+              memory: 100Mi
+            limits:
+              cpu: 50m
+              memory: 100Mi
+          env:
+            # if you have used flex before on your cluster, use same directory
+            # set TARGET_DIR env var and mount the same directory to to the container
+            - name: TARGET_DIR
+              value: "/etc/kubernetes/volumeplugins"
+          volumeMounts:
+            - mountPath: "/etc/kubernetes/volumeplugins"
+              name: volplugins
+      volumes:
+        - hostPath:
+            # Modify this directory if your nodes are using a different one
+            # default is "/usr/libexec/kubernetes/kubelet-plugins/volume/exec"
+            # below is Azure default
+            path: "/etc/kubernetes/volumeplugins"
+          name: volplugins
+      nodeSelector:
+        beta.kubernetes.io/os: linux
--- a/deploy/azure/kustomization.yaml
+++ b/deploy/azure/kustomization.yaml
@ -10,3 +10,4 @@ resources:
  - volume-claim.yml
  - nginx-client-ca-bundle.yml
  - acme-challenges.yml
+  - aadpodidentity.yml