How to update the Private Image Registry SSL certificate with user-provided CA and SSL certificates

 

Details

An administrator may want to replace the SSL certificate with their own certificate for the Private Image Registry in a vSphere 7 kubernetes cluster. The Registry Service in vSphere 7 with Kubernetes does not currently support SSL certificate replacements for private image registries. However, this can be done manually if you have the CA certificate and signed SSL certificates.

Solution

1. Make a note of the IP of the Private Image Registry, which is available at cluster Configuration → Namespaces → Image Registry.
2. Create a file named openssl.conf with content similar to the following:

[ req ]
default_bits       = 2048
distinguished_name = req_distinguished_name
x509_extensions    = SAN
req_extensions     = SAN
[ req_distinguished_name ]
[ SAN ]
subjectAltName = @alt_names
[alt_names]
IP.1=<registry IP>

Note: Replace <registry IP> with the IP address of your Private Image Registry noted in Step 1.

3. Generate a CSR for the new Private Image Registry SSL certificate based on the IP noted in Step 1.

Example: openssl req -nodes -new -keyout registry.key -out registry.csr -subj "/C=US/ST=CA/L=Palo Alto/O=VMware/OU=REGISTRY/CN=<registry IP>" -config openssl.conf

Note: Replace <regisgtry IP> with the IP address of your Private Image Registry noted in Step 1. You can replace the  other subj values if you prefer. Files named registry.key and registry.csr will be created in the current directory.

4. Generate the new Private Image Registry SSL certificate using the registry.csr CSR file created in Step 3. This will require submitting the CSR to a certificate signing authority (either internal or public).
5. Obtain the CA certificate from the issuing authority.
6. Encode the content of the registry.key file, the new Private Image Registry certificate and the CA certificate by issuing commands similar to the following:

cat registry.key | base64 | tr -d '\n'
cat registry.crt | base64 | tr -d '\n'
cat ca.crt | base64 | tr -d '\n'

7. On the vCenter Server Appliance, issue the /usr/lib/vmware-wcp/decryptK8Pwd.py command to get the IP address and root password for one of the control plane nodes.
8. SSH as the root user to the IP address obtained in Step 7, using the password also obtained in Step 7.
9. Issue the following command to obtain the name of the Private Image Registry namespace:

kubectl get ns |grep vmware-system-registry-

Note: You will see output similar to the following:

vmware-system-registry-990781218            Active   43h

10. Issue a command similar to the following to get the name of the Private Image Registry secret that needs to be edited:

kubectl -n <registry ns> get secrets |grep ssl

Note: Replace <registry ns> with the Private Image Registry namespace noted in Step 9. You will see output similar to the following:

harbor-990781218-ssl                     Opaque                                3      43h

11. Issue a command similar to the following to edit the Private Image Registry secret:

kubectl -n <registry ns> edit secret <registry secret>

Note: Replace <registry ns> with the Private Image Registry namespace noted in Step 9 and replace <registry secret> with the Private Image Registry secret noted in Step 10.

Note: The following values need to be replaced:

• ca.crt value should be replaced with the encoded content of the CA certificate noted in Step 6.
• tls.crt value should be replaced with the encoded content of the Private Image Registry certificate noted in Step 6.
• tls.key value should be replaced with the encoded content of the registry.key file noted in Step 6.

Note: Type :wq when you are finished editing the Private Image Registry secret.

12. Issue a command similar to the following to identify and restart the Private Image Registry nginx pod:

kubectl get pod -n <registry ns> | grep nginx

Note: Replace <registry ns> with the Private Image Registry namespace noted in Step 9. You will see output similar to the following:

harbor-990781218-harbor-nginx-85657d5f98-dlvcv       1/1     Running   1          43h

kubectl -n <registry ns> delete pod <nginx pod>

Note: Replace <nginx pod> with the name of the nginx pod returned by the previous command.

13. Issue a command similar to the following to identify the IP address of the node on which the Private Image Registry registry agent pod is running:

kubectl -n vmware-system-registry get pod -o wide

Note: You will see output similar to the following:

NAME                                                READY   STATUS    RESTARTS   AGE   IP            NODE                               NOMINATED NODE   READINESS GATES
vmware-registry-controller-manager-8fd79b8b-mxcsn   2/2     Running   26         43d   10.244.0.66   420ea13e894835e4dc382f7e7733b09a   <none>           <none>

Note: In the previous output, the IP address of the node where the Private Image Registry registry agent pod is running is 10.244.0.66.

14. From the SSH session to the control plane node that was initiated in Step 8, SSH to the IP address noted in Step 13 as the root user using the password noted in Step 7.
15. Issue the following command to identify the container running the Private Image Registry registry agent:

docker ps | grep admin-agent

Note: You will see output similar to the following:

482346669ea3        937c13fe9a44          "/registry-agent -te…"   19 hours ago        Up 19 hours                             k8s_admin-agent_vmware-registry-controller-manager-8fd79b8b-mxcsn_vmware-system-registry_332b2b4d-8b1e-422b-8dac-bb4a3e8227bb_13

Note: In the previous output, 482346669ea3 is the container ID for the container running the Private Image Registry registry agent.

16. Issue a command similar to the following to remove the existing Private Image Registry certificate from the Private Image Registry registry agent container:

docker exec -it <registry-agent-container-id> rm /etc/registry-agent/certs/harbor-ca.crt

Note: Replace <registry-agent-container-id> with the container ID value noted in Step 15

Note: The new certificate should now be updated to the Private Image Registry in vSphere 7 kubernetes cluster and will be for any new requests.