harbor镜像仓库一、镜像仓库1、类型2、构建私有仓库的方案二、部署harbor仓库(单机版)1、安装docker(略)2、安装docker-compose工具3、安装harbor4、生成harbor需要的证书、密钥(V3版本证书)4.1 创建CA4.2 创建harbor仓库需要的证书5、编辑harbor配置文件6、启动harbor三、harbor仓库的使用1、创建项目2、创建用户、授权3、上传镜像3.1 登录仓库3.2 为镜像打标记3.3 上传镜像3.4 退出仓库四、harbor核心组件五、harbor高可用设计方案1、配置nfs作harbor的数据存储目录2、安装redis作为harbor共享缓存3、安装postgreSQL 作为harbor的共享数据库4、两台harbor仓库挂载nfs存储作持久卷5、编辑harbor配置文件禁用自带数据库连接外部数据库6、启动harbor7、测试两台harbor仓库可正常访问8、配置haproxy做harbor仓库的负载均衡一、镜像仓库1、类型公有仓库 DockerHub私有仓库 企业级的应用2、构建私有仓库的方案registry镜像早期harbor软件vmware开源的提供webUI日志审计权限分配统一认证二、部署harbor仓库(单机版)1、安装docker(略)2、安装docker-compose工具[rootlocalhost ~]# mv docker-compose /usr/local/bin/[rootlocalhost ~]# chmod ax /usr/local/bin/docker-compose[rootlocalhost ~]# docker-compose versiondocker-composeversion1.29.0, build 07737305 docker-py version:5.0.0 CPython version:3.7.10 OpenSSL version: OpenSSL1.1.0l10Sep20193、安装harbor[rootlocalhost ~]# mkdir /work[rootlocalhost ~]# tar xf harbor-offline-installer-v2.2.2.tgz -C /work/[rootlocalhost ~]# cp /work/harbor/harbor.yml.tmpl /work/harbor/harbor.yml4、生成harbor需要的证书、密钥(V3版本证书)4.1 创建CA[rootlocalhost ~]# mkdir /opt/ssl[rootlocalhost ~]# cd /opt/ssl[rootlocalhost ssl]# openssl genrsa -out ca.key 4096Generating RSA private key,4096bit long modulus................................................................................................................................................................................................. e is65537(0x10001)[rootlocalhost ssl]#[rootlocalhost ssl]# openssl req -x509 -new -nodes -sha512 -days 3650 -subj /CNharbor.linux.com -key ca.key -out ca.crt[rootlocalhost ssl]# lsca.crt ca.key4.2 创建harbor仓库需要的证书[rootlocalhost ssl]# openssl genrsa -out server.key 4096Generating RSA private key,4096bit long modulus............................................................................................... e is65537(0x10001)[rootlocalhost ssl]# openssl req -new -sha512 -subj /CNharbor.linux.com -key server.key -out server.csr[rootlocalhost ssl]# cat v3.extauthorityKeyIdentifierkeyid,issuerbasicConstraintsCA:FALSE keyUsagedigitalSignature, nonRepudiation, keyEncipherment, dataEncipherment extendedKeyUsageserverAuth subjectAltNamealt_names[alt_names]DNS.1harbor.linux.com[rootlocalhost ssl]# openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in server.csr -out server.crtSignature oksubject/CNharbor.linux.com Getting CA Private Key[rootlocalhost ssl]#5、编辑harbor配置文件[rootlocalhost ~]# vim /work/harbor/harbor.ymlhostname: harbor.linux.com# http related confighttp:# port for http, default is 80. If https enabled, this port will redirect to https portport:80# https related confighttps:# https port for harbor, default is 443port:443# The path of cert and key files for nginxcertificate: /opt/ssl/server.crt private_key: /opt/ssl/server.key6、启动harbor[rootlocalhost harbor]# ./prepareprepare basedirissetto /work/harbor Generated configuration file: /config/portal/nginx.conf Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /config/registryctl/config.yml Generated configuration file: /config/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml Generated and saved secret to file: /data/secret/keys/secretkey Successfully called func: create_root_cert Generated configuration file: /compose_location/docker-compose.yml Clean up the inputdir[rootlocalhost harbor]# ./install.sh[Step0]: checkingifdockeris installed... Note:dockerversion:26.1.4[Step1]: checkingdocker-composeis installed... Note:docker-composeversion:1.29.0 Creating networkharbor_harborwith the default driver Creating harbor-log...doneCreating registry...doneCreating registryctl...doneCreating redis...doneCreating harbor-portal...doneCreating harbor-db...doneCreating harbor-core...doneCreating nginx...doneCreating harbor-jobservice...done✔ ----Harbor has been installed and started successfully.----[rootlocalhost harbor]# docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2f9f9ae9ded5 goharbor/harbor-jobservice:v2.2.2/harbor/entrypoint.…About a minute ago Up About a minute(healthy)harbor-jobservice 7c9156960ee5 goharbor/nginx-photon:v2.2.2nginx -g daemon of…About a minute ago Up About a minute(healthy)0.0.0.0:80-8080/tcp, :::80-8080/tcp,0.0.0.0:443-8443/tcp, :::443-8443/tcp nginx 4d42f7530a21 goharbor/harbor-core:v2.2.2/harbor/entrypoint.…About a minute ago Up About a minute(healthy)harbor-core 6b20c3728d4b goharbor/harbor-db:v2.2.2/docker-entrypoint.…About a minute ago Up About a minute(healthy)harbor-db 7e5d825d7161 goharbor/harbor-registryctl:v2.2.2/home/harbor/start.…About a minute ago Up About a minute(healthy)registryctl 79c07047e9e3 goharbor/redis-photon:v2.2.2redis-server /etc/r…About a minute ago Up About a minute(healthy)redis 2b6f7e8ce7b0 goharbor/harbor-portal:v2.2.2nginx -g daemon of…About a minute ago Up About a minute(healthy)harbor-portal ca7e332c47e9 goharbor/registry-photon:v2.2.2/home/harbor/entryp…About a minute ago Up About a minute(healthy)registry c2ddc9c858a5 goharbor/harbor-log:v2.2.2/bin/sh -c /usr/loc…About a minute ago Up About a minute(healthy)127.0.0.1:1514-10514/tcp三、harbor仓库的使用1、创建项目2、创建用户、授权3、上传镜像3.1 登录仓库[rootmartin-host ~]# mkdir /etc/docker/certs.d/harbor.linux.com -p[rootmartin-host ~]#[rootmartin-host ~]# scp root192.168.140.11:/opt/ssl/server.crt /etc/docker/certs.d/harbor.linux.com/root192.168.140.11s password: server.crt100%18462.4MB/s 00:00[rootmartin-host ~]#[rootmartin-host ~]# ls /etc/docker/certs.d/harbor.linux.com/server.crt[rootmartin-host ~]#[rootmartin-host ~]# docker login harbor.linux.comUsername: martin Password: WARNING!Your password will be stored unencryptedin/root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded3.2 为镜像打标记[rootmartin-host ~]# docker tag websocket-server:1.0 harbor.linux.com/miaosha/websocket-server:1.03.3 上传镜像[rootmartin-host ~]# docker push harbor.linux.com/miaosha/websocket-server:1.0The push refers to repository[harbor.linux.com/miaosha/websocket-server]00f2a652d376: Pushed 1b9504956550: Pushed 20f340df0413: Pushed 6c3e7df31590: Pushed1.0: digest: sha256:9a8a24800c9eef5700585a8f461169735b14519760750c4c946857926c737a7e size:1166[rootmartin-host ~]#3.4 退出仓库[rootmartin-host ~]# docker logout harbor.linux.comRemoving login credentialsforharbor.linux.com[rootmartin-host ~]#四、harbor核心组件core service提供web UI、令牌token、与registry交互获取镜像的元数据信息在webUI上展示registry负责镜像的上传、下载Log collector采集日志Job service负责在多个harbor仓库间同步数据Proxy使用nginx对后端所有组件进行反向代理Databaseredis存储前端用户产生的令牌harbor-db关系型数据库存放harbor仓库上数据默认是postgreSQL/pgSQL五、harbor高可用设计方案核心思想禁用其自带的数据库配置连接第三方的库 保证多个harbor间的数据同步192.168.140.11 harbor仓库192.168.140.12 harbor仓库192.168.140.10 后端数据库、存储1、配置nfs作harbor的数据存储目录[rootlocalhost ~]# cat /etc/exports/harbor_data192.168.140.11(rw,no_root_squash)192.168.140.12(rw,no_root_squash)[rootlocalhost ~]#[rootlocalhost ~]# systemctl enable --now nfs-serverCreated symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.[rootlocalhost ~]#[rootlocalhost ~]#2、安装redis作为harbor共享缓存[rootmartin-host ~]# docker run -tid --nameharbor_redis --nethost --restartalways redis:latestb61a5a1b15ef6c87f1ff9a37673bbf789945ace577b7aeaea069f4734a601b16[rootmartin-host ~]# docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b61a5a1b15ef redis:latestdocker-entrypoint.s…3seconds ago Up2seconds harbor_redis3、安装postgreSQL 作为harbor的共享数据库[rootmartin-host ~]# mkdir -p /pgsql/data[rootmartin-host ~]# docker run -tid --nameharbor_pgsql -e POSTGRES_PASSWORDredhat -e PGDATA/var/lib/postgresql/data/pgdata -v /pgsql/data:/var/lib/postgresql/data/pgdata --nethost --restartalways postgres:12.287474ae582f7850b75b3adb3b73dba88645c7ce9c14c68b83d34657ca2eaeca6[rootmartin-host ~]# docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 87474ae582f7 postgres:12.2docker-entrypoint.s…6seconds ago Up4seconds harbor_pgsql[rootmartin-host ~]# docker exec -ti harbor_pgsql bashrootmartin-host:/#rootmartin-host:/# psql -h 127.0.0.1 -p 5432 -U postgrespsql(12.2(Debian12.2-2.pgdg1001))Typehelpforhelp.postgres# create user harbor with password redhat;CREATE ROLEpostgres# create database harbor;CREATE DATABASEpostgres# create database harbor_clair;CREATE DATABASEpostgres# create database harbor_notary_server;CREATE DATABASEpostgres# create database harbor_notary_signer;CREATE DATABASEpostgres# grant all on database harbor to harbor;GRANTpostgres# grant all on database harbor_clair to harbor;GRANTpostgres# grant all on database harbor_notary_server to harbor;GRANTpostgres# grant all on database harbor_notary_signer to harbor;GRANTpostgres# exit修改pgSQL的配置文件允许远程主机(harbor仓库)连接rootmartin-host:/var/lib/postgresql/data/pgdata# echo host all all all trust pg_hba.conf4、两台harbor仓库挂载nfs存储作持久卷[rootlocalhost harbor]# tail -n 1 /etc/fstab192.168.140.13:/harbor_data /data nfs defaults00[rootlocalhost harbor]# mount -a[rootlocalhost harbor]# df -hT | grep nfs192.168.140.13:/harbor_data nfs4 20G 33M 20G1% /data5、编辑harbor配置文件禁用自带数据库连接外部数据库# Harbor DB configuration# database:# The password for the root user of Harbor DB. Change this before any production use.# password: root123# The maximum number of connections in the idle connection pool. If it 0, no idle connections are retained.# max_idle_conns: 50# The maximum number of open connections to the database. If it 0, then there is no limit on the number of open connections.# Note: the default number of connections is 1024 for postgres of harbor.# max_open_conns: 1000// 配置harbor连接外部的pgsql external_database: harbor: host:192.168.140.10 port:5432db_name: harbor username: harbor password: redhat ssl_mode: disable max_idle_conns:2max_open_conns:0notary_signer: host:192.168.140.10 port:5432db_name: harbor_notary_signer username: harbor password: redhat ssl_mode: disable notary_server: host:192.168.140.10 port:5432db_name: harbor_notary_server username: harbor password: redhat ssl_mode: disable //配置harbor连接外部redis external_redis:# # support redis, redissentinel# # host for redis: host_redis:port_redis# # host for redissentinel:# # host_sentinel1:port_sentinel1,host_sentinel2:port_sentinel2,host_sentinel3:port_sentinel3host:192.168.140.10:6379# password:# # sentinel_master_set must be set to support redissentinel# #sentinel_master_set:# # db_index 0 is for core, its unchangeableregistry_db_index:1jobservice_db_index:2chartmuseum_db_index:3trivy_db_index:5idle_timeout_seconds:306、启动harbor[rootharbor_01 harbor]# ./prepare[rootharbor_01 harbor]# ./install.sh// 注意查看harbor启动的容器数量缺少自带的数据库;redis/harbor-db[rootlocalhost harbor]# docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 814ac469d518 goharbor/nginx-photon:v2.2.2nginx -g daemon of…32seconds ago Up31seconds(healthy)0.0.0.0:80-8080/tcp, :::80-8080/tcp,0.0.0.0:443-8443/tcp, :::443-8443/tcp nginx ad3918f2fefc goharbor/harbor-jobservice:v2.2.2/harbor/entrypoint.…32seconds ago Up31seconds(healthy)harbor-jobservice 91312576c578 goharbor/harbor-core:v2.2.2/harbor/entrypoint.…32seconds ago Up31seconds(healthy)harbor-core 782d76121ad3 goharbor/registry-photon:v2.2.2/home/harbor/entryp…33seconds ago Up32seconds(healthy)registry 11e5acde4206 goharbor/harbor-portal:v2.2.2nginx -g daemon of…33seconds ago Up32seconds(healthy)harbor-portal 02eb98775c46 goharbor/harbor-registryctl:v2.2.2/home/harbor/start.…33seconds ago Up32seconds(healthy)registryctl e61628856f6f goharbor/harbor-log:v2.2.2/bin/sh -c /usr/loc…33seconds ago Up32seconds(healthy)127.0.0.1:1514-10514/tcp另外一台harbor仓库配置参考上述7、测试两台harbor仓库可正常访问8、配置haproxy做harbor仓库的负载均衡[rootmartin-host ~]# cat /opt/work/haproxy.cfgfrontend harborbind0.0.0.0:9443 mode tcp use_backend harbor_server backend harbor_server mode tcp balance roundrobin server harbor01192.168.140.11:443 server harbor02192.168.140.12:443[rootmartin-host ~]#[rootmartin-host ~]#[rootmartin-host ~]# docker run -tid --nameharbor_haproxy -p 443:9443 -v /opt/work/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg --restartalways haproxy:latestf1e660162065993f5de24ac24ad9357418cde58de11bcbe06e8d07402d39d1fe[rootmartin-host ~]#[rootmartin-host ~]# docker ps -aCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f1e660162065 haproxy:latestdocker-entrypoint.s…4seconds ago Up3seconds0.0.0.0:443-9443/tcp, :::443-9443/tcp harbor_haproxy // 客户端测试通过haproxy访问仓库