How to configure Tanzu Kubernetes cluster nodes to securely access the Registry Service in vSphere 7 with Kubernetes

  Purpose

This article provides instructions for configuring Tanzu Kubernetes cluster nodes to securely access the Registry Service in vSphere 7 with Kubernetes.

Resolution

Prerequisites:

• You have a Tanzu Kubernetes cluster deployed
• You have enabled the Registry Service on the cluster via the Configuration page on the vCenter instance
• You can SSH into the VC appliance as root

Obtain the credentials for the supervisor control plane

1.  (Step 1)SSH into the VC instance as the root user.
2.  (Step 2)Run the /usr/lib/vmware-wcp/decryptK8Pwd.py command to obtain the root password for the control plane VMs.    

Note: You will see output similar to the following. Make a note of the IP address and password:

Read key from file
Connected to PSQL
Cluster: domain-c9:c9ad9cee-30ca-404e-b675-50fd6e242e96
IP: 10.x.x.x
PWD: 49JhdFBEMJOpPW7bsZ1M1Q6YYqb0L3EV/0R+ooxkEXrRGXhjjKBhlz4ZqFcSMZVgtFzbPF7ia6YnWv5DxwwKqrJFXWvsTvg5FOIxDlIiQFKXgYhlwM0ukZJRr6Fk9rnugNExz2JPJotKXDyNmN/gPjioIUU5FchJN3CaII5cGBg=

3. Download the Registry Service Root CA in the VC UI. Configure > Namespaces dropdown > Image Registry > Root certificate > Download SSL Root Certificate.
4. Use a file transfer utility to copy this root CA file to the supervisor control plane using the IP address/password obtained in Step 2 .

Install Registry Service Root CA on the Tanzu Kubernetes cluster Nodes

1. Download the tkg-registry-cert.tgz file attached to this article.
2. Use a file transfer utility to copy the downloaded file to the supervisor cluster control plane node IP address noted in Step 2.
3. SSH to the supervisor cluster control plane node IP address noted in Step 2.
4. Issue a command similar to the following to extract the contents of the downloaded file:

tar -zxvf <path>/tkg-registry-cert-tgz

Note: A single file named tkg-registry-cert.sh should be extracted.

5. Open the tkg-registry-cert.sh file in a text editor and  make the following changes:

• Change the value for rootCA to the path/file name of the Registry Service root certificate.
• Change the value for gcname to the name of the Tanzu Kubernetes Cluster.
• Change the value for gcnamespace to the namespace in the supervisor cluster where the Tanzu Kubernetes Cluster is deployed.
• Save and close the file.

6. Issue a command similar to the following to make the tkg-registry-cert.sh script executable:

chmod +x <path>/tkg-registry-cert.sh

7. Run <path>/tkg-registry-cert.sh to execute the script.

Set up the Registry Service Secret in the Tanzu Kubernetes cluster

Each supervisor cluster namespace will have a push/pull secret for the Registry Service instance. This section provides instructions on creating a similar secret in the Tanzu Kubernetes cluster.

Note: Make sure you're using the supervisor kubeconfig if you pointed it to the Tanzu Kubernetes cluster kubeconfig at any previous point.

1. Issue a command similar to the following to collect the image pull secret for the namespace and store it in a file:

kubectl get secret -n <namespace> <namespace>-default-image-pull-secret -o yaml > <path>/image-pull-secret.yaml

Note: Replace <namespace> with the namespsace in the supervisor cluster where the Tanzu Kubernetes cluster is deployed.

2. Open the <path>/image-pull-secret.yaml file with a text editor and make the following changes:

• Change the value for name to something meaningful, like "registry-secret".
• Change the value for namespace to the appropriate namespace in the Tanzu Kubernetes Cluster.
• Save and close the file.

3. Issue a command similar to the following to create a kubeconfig file that can be used to access the Tanzu Kubernetes cluster:

kubectl get secret -n <namespace> <clustername>-kubeconfig -o jsonpath='{.data.value}' | base64 -d > <path>/cluster-kubeconfig

Note:  Replace <namespace> with the name of the namespace in the supervisor cluster where the Tanzu Kubernetes cluster is deployed and replace <clustername> with the name of the Tanzu Kubernetes cluster.

4. Issue a command similar to the following to create the Registry Service secret in the Tanzu Kubernetes cluster:

kubectl --kubeconfig=<path>/cluster-kubeconfig apply -f <path>/image-pull-secret.yaml

Pushing Images to the Registry Service as a User

Notes:

• These steps are for testing purposes only.
• These steps can be performed on a node that has network access to the Registry Service.
• Make sure your user has 'edit' permissions on the namespace, granted from the VC UI. 

1. Issue a command similar to the following to log in to the Registry Service:

docker login https://192.168.123.3

Note: replace 192.168.123.3 with the IP of the Registry Service in your cluster.

2. Assuming you have a Dockerfile accessible, build an image and tag it with the name:

docker build -t <Registry Service IP Address>/<namespace>/<image name>:<version> <path>/<Dockerfile>

Note: Replace <Registry Service IP Address> with the IP Address for the Registry Service instance running in your cluster, replace <namespace> with the supervisor cluster namespace where the Tanzu Kubernetes cluster is deployed, replace <image name> with an image name of your choice, replace <version> with the appropriate version of the image, replace <path>/<Dockerfile> with the location of the Docker file you'll be using to build the image.

3. Issue a command similar to the following to push the image to the Registry Service instance

docker push <Registry Service IP Address>/<namespace>/<image name>:<version>

Note: Replace <Registry Service IP Address> with the IP Address for the Registry Service instance running in your cluster, replace <namespace> with the supervisor cluster namespace where the Tanzu Kubernetes cluster is deployed, replace <image name> with an image name of your choice, replace <version> with the appropriate version of the image.

Pulling Images From the Registry Service in a Tanzu Kubernetes cluster

Notes:

• These steps are for testing purposes only.
• These steps can be performed on a node that has network access to the Registry Service.
• Make sure your user has 'edit' permissions on the namespace, granted from the VC UI. 
• Reference the image pull secret you created previously in a deployment/pod spec that needs to pull an image from the Registry Service.

1. Create a pod specification that will use an image stored in the Registry Service and utilize the image pull secret previously configured. See https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ for additional details.

Note: The following is a sample that can be modified to work in your environment:
 

apiVersion: v1
kind: Pod
metadata:
  name: <name>
  namespace: <gcnamespace>
spec:
  containers:
  - name: private-reg-container
    image: <Registry Service IP Address>/<namespace>/<image name>:<version>
  imagePullSecrets:
  - name: <registry-secret name>

 

Note: Replace <name> with the name of the pod, replace <gcnamespace> with the namespace in the Tanzu Kubernetes cluster where the pod will be created (must be the same namespace where the Registry Service image pull secret is stored in the Tanzu Kubernetes cluster), replace <Registry Service IP Address> with the IP Address for the Registry Service instance running in your cluster, replace <namespace> with the supervisor cluster namespace where the Tanzu Kubernetes cluster is deployed, replace <image name> with an image name of your choice, replace <version> with the appropriate version of the image, replace <registry-secret name> with the name of the Registry Service image pull secret created previously.

2. Issue a command similar to the following to create a pod in the Tanzu Kubernetes cluster based on the specification created in Step 1 :

kubectl --kubeconfig=<path>/cluster-kubeconfig apply -f <pod.yaml>

 

** confidential **

The following steps can be used to manually install the Registry Serivce root CA to the Tanzu Kubernetes cluster nodes (in lieu of running the script attached to this article).

1. First, fetch the secret to SSH into the TKG nodes. This can be found in the namespace where the TKG cluster is deployed, as '<cluster name>-ssh'

For instance, if your TKG cluster is named 'app1' under the 'prod' namespace, the SSH key would be in the 'app1-ssh' Secret, in the 'prod' namespace. The base64 encoded key is present in a field called 'ssh-privatekey' under the 'data' of the Secret, and you can obtain the SSH key by running the following

kubectl get secret -n prod app1-ssh -o jsonpath='{.data.ssh-privatekey}' | base64 -d > cluster-ssh

2. Fetch the kubeconfig file for the TKG cluster, '<cluster name>-kubeconfig':

For instance, if your TKG cluster is named 'app1' under the 'prod' namespace, the kubeconfig would be in the 'app1-kubeconfig' Secret, in the 'prod' namespace. The base64 encoded kubeconfig is present in a field called 'value' under the 'data' of the Secret.

kubectl get secret -n prod app1-kubeconfig -o jsonpath='{.data.value}' | base64 -d > cluster-kubeconfig

3. Switch to the guest cluster kubeconfig

export KUBECONFIG=cluster-kubeconfig

4. For each of the TKG nodes, do the following:

Find the IP address of the node using:

kubectl get node <node name>  -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'

Copy the Registry root CA bundle to the TKG node and install it into the system trust bundle by appending it to '/etc/pki/tls/certs/ca-bundle.crt':

scp -i cluster-ssh <path to Registry Service CA bundle>  vmware-system-user@<node IP>:/home/vmware-system-user/registry_ca.crt

ssh -i cluster-ssh vware-system-user@10.244.1.4 'sudo bash -c "cat /home/vware-system-user/registry_ca.crt >> /etc/pki/tls/certs/ca-bundle.crt"'
ssh -i cluster-ssh vmware-system-user@<node IP> 'sudo systemctl restart docker.service'