云原生之容器编排实践-在K8S集群中使用Registry2搭建私有镜像仓库
背景
基于前面搭建的3节点 Kubernetes 集群，今天我们使用 Registry2 搭建私有镜像仓库，这在镜像安全性以及离线环境下运维等方面具有重要意义。
Note: 由于是测试环境，以下创建了一个 local-storage 的 StorageClass ，并使用本地磁盘的方式创建使用 PV ，实际建议使用 NFS 。
虚机资源
共用到了三台虚机，1台作为 Master 节点，2台 Worker 节点。
| 主机名 | IP | 说明 |
|---|---|---|
| k8s-master | 172.16.201.25 | 主节点 |
| k8s-node1 | 172.16.201.26 | 工作节点 |
| k8s-node2 | 172.16.201.27 | 工作节点 |
[root@k8s-master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready control-plane,master 37h v1.20.9
k8s-node1 Ready <none> 35h v1.20.9
k8s-node2 Ready <none> 35h v1.20.9
系统环境
[root@k8s-master ~]# uname -a
Linux k8s-master 3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@k8s-master ~]# cat /proc/version
Linux version 3.10.0-1160.71.1.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Jun 28 15:37:28 UTC 2022
[root@k8s-master ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
使用Registry2搭建私有镜像仓库
采用 YAML 文件的方式，搭建私有 Docker Registry 的整体流程：
- 创建一个Docker Registry的命名空间
- 创建一个持久卷来存储Docker Registry的数据
- 创建一个Secret用于存储Docker Registry的凭证
- 创建一个Deployment来部署Docker Registry
- 创建一个Service来暴露Docker Registry
- 登录并推送镜像到私有镜像仓库
创建一个Docker Registry的命名空间
apiVersion: v1
kind: Namespace
metadata:
name: docker-registry
创建一个持久卷来存储Docker Registry的数据
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
namespace: docker-registry
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: docker-registry-pv
labels:
pv: docker-registry-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /data/docker
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- k8s-node1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: docker-registry-pvc
namespace: docker-registry
spec:
resources:
requests:
storage: 5Gi
accessModes:
- ReadWriteMany
storageClassName: local-storage
selector:
matchLabels:
pv: docker-registry-pv
Note：
- 没有必要将namespace字段添加到kind: persistantVolume因为PersistentVolumes绑定(bind)是排他的；
- 出于安全考虑Pod不会被调度到Master Node上，也就是说默认情况下Master节点（taint：污点）不参与工作负载。如果没有配置nodeAffinity，在创建PV时会报错：
0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn’t tolerate, 2 node(s) didn’t find available persistent volumes to bind.
解决方法：添加 nodeAffinity ，这里将 PV 创建到 k8s-node1 节点；记得先在 Worker 节点上创建 PV 目录：/data/docker
创建一个Secret用于存储Docker Registry的凭证
mkdir auth
docker run --entrypoint htpasswd registry:2 -Bbn username password > auth/htpasswd
报错：unable to start container process: exec: “htpasswd”: executable file not found in $PATH: unknown.
原因： registry 镜像在 2.x 相关版本中删除了 /usr/bin/htpasswd 文件，导致创建容器时找不到可执行文件，参考：https://github.com/docker/distribution-library-image/issues/106
使用旧版本的 registry 或者在服务器本地安装 httpd ，再执行生成密码的命令。
yum install httpd
htpasswd -Bbn username password > auth/htpasswd
kubectl create secret generic docker-registry-secret --from-file=auth/htpasswd -n docker-registry
# 创建了一个Opaque类型的Secret
[root@k8s-master ~]# kubectl get secret -n docker-registry
NAME TYPE DATA AGE
default-token-prfsz kubernetes.io/service-account-token 3 11m
docker-registry-secret Opaque 1 20s
创建 Docker 私有镜像仓库的 secret ，这里是在 docker-registry 命名空间，如果新增了 namespace ，那么这个 namespace 就需要单独添加一次 secret ，而且这个 namespace 下拉取镜像时也需要添加 imagePullSecrets 。
kubectl create secret docker-registry ruoyi-registry-secret --docker-server=10.96.198.223:5000 --docker-username=username --docker-password=password -n docker-registry
[root@k8s-master ~]# kubectl get secret -n docker-registry
NAME TYPE DATA AGE
default-token-prfsz kubernetes.io/service-account-token 3 2d23h
docker-registry-secret Opaque 1 2d23h
ruoyi-registry-secret kubernetes.io/dockerconfigjson 1 3s
Note:
- docker-registry-secret: 用于配置Registry的账号与域名，并在控制台通过username与password登录、推送镜像到私有镜像仓库；
- ruoyi-registry-secret: 用于通过在不同的Worker节点上拉取私有镜像。
将上述创建 NameSpace , StorageClass , PV 以及 PVC 的 YAML 合并为一个 docker-registry-ns-sc-pv-pvc.yaml 并执行。
kubectl apply -f docker-registry-ns-sc-pv-pvc.yaml
# 查看sc,pv,pvc状态
[root@k8s-master registry]# kubectl get sc,pv,pvc -n docker-registry
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storageclass.storage.k8s.io/local-storage kubernetes.io/no-provisioner Retain WaitForFirstConsumer false 4m45s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/docker-registry-pv 5Gi RWX Retain Available local-storage 4m45s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/docker-registry-pvc Pending local-storage 4m45s
看到 PVC 一直是 Pending 状态。
# 查看PVC详细信息
[root@k8s-master registry]# kubectl describe pvc -n docker-registry
Name: docker-registry-pvc
Namespace: docker-registry
StorageClass: local-storage
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 9s (x26 over 6m13s) persistentvolume-controller waiting for first consumer to be created before binding
提示需要有个消费者进行关联，后面应用 Deployment 后， PVC 状态会变为 Binding 。
创建一个Deployment来部署Docker Registry
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: docker-registry
name: docker-registry
namespace: docker-registry
spec:
replicas: 1
revisionHistoryLimit: 5
selector:
matchLabels:
app: docker-registry
template:
metadata:
labels:
app: docker-registry
spec:
securityContext:
runAsUser: 0
containers:
- name: docker-registry
image: registry:2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5000
name: web
protocol: TCP
resources:
requests:
memory: 200Mi
cpu: "0.1"
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
env:
- name: REGISTRY_AUTH
value: htpasswd
- name: REGISTRY_AUTH_HTPASSWD_REALM
value: Registry Realm
- name: REGISTRY_AUTH_HTPASSWD_PATH
value: /auth/htpasswd
volumeMounts:
- name: docker-registry-data
mountPath: /var/lib/registry/
- name: docker-registry-auth
mountPath: /auth
readOnly: true
volumes:
- name: docker-registry-data
persistentVolumeClaim:
claimName: docker-registry-pvc
- name: docker-registry-auth
secret:
secretName: docker-registry-secret
创建一个Service来暴露Docker Registry
apiVersion: v1
kind: Service
metadata:
name: docker-registry-service
namespace: docker-registry
spec:
ports:
- name: port-name
port: 5000
protocol: TCP
targetPort: 5000
selector:
app: docker-registry
type: NodePort
部署 Deployment 与 Service 。
kubectl apply -f docker-registry-deploy-svc.yaml
[root@k8s-master registry]# kubectl get deploy -n docker-registry
NAME READY UP-TO-DATE AVAILABLE AGE
docker-registry 1/1 1 1 21s
[root@k8s-master registry]# kubectl get svc -n docker-registry
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
docker-registry-service NodePort 10.96.198.223 <none> 5000:30858/TCP 15s
登录私有镜像仓库
# 直接查看下镜像仓库的内容，提示未认证
[root@k8s-master ~]# curl http://10.96.198.223:5000/v2/_catalog
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}
# 使用用户名、密码登录，报错：http: server gave HTTP response to HTTPS client
[root@k8s-master ~]# docker login -uusername -ppassword 10.96.198.223:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://10.96.198.223:5000/v2/: http: server gave HTTP response to HTTPS client
# 修改配置文件，添加insecure-registries
[root@k8s-master ~]# vi /etc/docker/daemon.json
"insecure-registries": ["10.96.198.223:5000"]
# 重启Docker服务
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart docker
# 再次使用用户名、密码登录，成功
[root@k8s-master registry]# docker login -uusername -ppassword 10.96.198.223:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
# 会自动生成~/.docker/config.json文件
[root@k8s-master ~]# cat ~/.docker/config.json
{
"auths": {
"10.96.198.223:5000": {
"auth": "dXNlcm5hbWU6cGFzc3dvcmQ="
}
}
}
# 附带用户名、密码，获取仓库内镜像，为空
[root@k8s-master ~]# curl -u username:password http://10.96.198.223:5000/v2/_catalog
{"repositories":[]}
推送镜像到私有仓库
# 以hello-world为例测试推送镜像到私有仓库功能
[root@k8s-master ~]# docker run hello-world
[root@k8s-master ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
calico/node v3.20.6 daeec7e26e1f 17 months ago 156MB
calico/pod2daemon-flexvol v3.20.6 39b166f3f936 17 months ago 18.6MB
calico/cni v3.20.6 13b6f63a50d6 17 months ago 138MB
calico/kube-controllers v3.20.6 4dc6e7685020 17 months ago 60.2MB
registry 2 b8604a3fe854 2 years ago 26.2MB
hello-world latest feb5d9fea6a5 2 years ago 13.3kB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-proxy v1.20.9 8dbf9a6aa186 2 years ago 99.7MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-scheduler v1.20.9 295014c114b3 2 years ago 47.3MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-controller-manager v1.20.9 eb07fd4ad3b4 2 years ago 116MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-apiserver v1.20.9 0d0d57e4f64c 2 years ago 122MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/etcd 3.4.13-0 0369cf4303ff 3 years ago 253MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/coredns 1.7.0 bfe3a36ebd25 3 years ago 45.2MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause 3.2 80d28bedfe5d 3 years ago 683kB
# 对hello-world镜像重新打标签
[root@k8s-master ~]# docker tag hello-world 10.96.198.223:5000/hello-world:1
[root@k8s-master ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
calico/node v3.20.6 daeec7e26e1f 17 months ago 156MB
calico/pod2daemon-flexvol v3.20.6 39b166f3f936 17 months ago 18.6MB
calico/cni v3.20.6 13b6f63a50d6 17 months ago 138MB
calico/kube-controllers v3.20.6 4dc6e7685020 17 months ago 60.2MB
registry 2 b8604a3fe854 2 years ago 26.2MB
10.96.198.223:5000/hello-world 1 feb5d9fea6a5 2 years ago 13.3kB
hello-world latest feb5d9fea6a5 2 years ago 13.3kB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-proxy v1.20.9 8dbf9a6aa186 2 years ago 99.7MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-scheduler v1.20.9 295014c114b3 2 years ago 47.3MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-controller-manager v1.20.9 eb07fd4ad3b4 2 years ago 116MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-apiserver v1.20.9 0d0d57e4f64c 2 years ago 122MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/etcd 3.4.13-0 0369cf4303ff 3 years ago 253MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/coredns 1.7.0 bfe3a36ebd25 3 years ago 45.2MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause 3.2 80d28bedfe5d 3 years ago 683kB
# 推送到私有镜像仓库，成功
[root@k8s-master ~]# docker push 10.96.198.223:5000/hello-world:1
The push refers to repository [10.96.198.223:5000/hello-world]
e07ee1baac5f: Pushed
1: digest: sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4 size: 525
# 附带用户名、密码，获取仓库内镜像
[root@k8s-master ~]# curl -u username:password http://10.96.198.223:5000/v2/_catalog
{"repositories":["hello-world"]}
# 退出操作
[root@k8s-master ~]# docker logout 10.96.198.223:5000
安装Registry前端
直接通过 Docker 安装 Registry 前端，通过用户名与密码登录后，可以网页展示已推送的私有镜像。
docker run -d -e ENV_DOCKER_REGISTRY_HOST=10.96.198.223 -e ENV_DOCKER_REGISTRY_PORT=5000 -p 5001:80 konradkleine/docker-registry-frontend:v2
- 认证
- 私有镜像列表
小总结
作为测试环境，本次使用 Registry2 搭建了私有镜像仓库，实际生产环境建议使用 Harbor 。 Registry2 和 Harbor 都是用于 Docker 镜像存储和分发的解决方案，但它们各自具有不同的特点：
Registry2
- 基本功能：Registry2（通常称为Docker Registry）是Docker官方的开源镜像仓库，提供了存储和分发Docker镜像的基本功能。
- 简单轻量：它比较轻量级，适合需要快速部署的场景。
- 自主部署：可以自主部署在私有环境中，满足私有化需求。
- 缺少高级功能：相比于Harbor，Registry2缺少一些高级功能，如图形用户界面、角色基础的访问控制、镜像扫描和签名等。
Harbor
- 企业级特性：Harbor是一个开源的企业级Docker Registry解决方案，提供了Registry2的所有基本功能，并增加了许多企业级特性。
- 图形用户界面：Harbor提供了用户友好的图形界面，便于管理和浏览镜像。
- 访问控制：支持基于角色的访问控制，可以细粒度地管理用户权限。
- 安全性：提供了镜像扫描和签名功能，增强了镜像的安全性。
- 高可用性：支持高可用部署，适合企业级应用。
总结来说，如果你需要一个简单的、轻量级的 Docker 镜像仓库， Registry2 可能是一个不错的选择。而如果你需要更多的企业级特性，如图形界面、细粒度的访问控制、镜像安全扫描等， Harbor 会是更好的选择。
If you have any questions or any bugs are found, please feel free to contact me.
Your comments and suggestions are welcome!