作者：liugp
来源：https://u.kubeinfo.cn/9Da2ym
一、前言
二、基础环境部署
- 1）前期准备（所有节点）
- 2）安装容器 docker（所有节点）
- 3）配置 k8s yum 源（所有节点）
- 4）将 sandbox_image 镜像源设置为阿里云 google_containers 镜像源（所有节点）
- 5）配置 containerd cgroup 驱动程序 systemd（所有节点）
- 6）开始安装 kubeadm，kubelet 和 kubectl（master 节点）
- 7）使用 kubeadm 初始化集群（master 节点）
- 8）安装 Pod 网络插件（CNI：Container Network Interface）(master)
- 9）node 节点加入 k8s 集群
- 10）配置 IPVS
- 11）集群高可用配置
- 12）部署 Nginx+Keepalived 高可用负载均衡器
三、k8s 管理平台 dashboard 环境部署
- 1）dashboard 部署
- 2）创建登录用户
- 3）配置 hosts 登录 dashboard web
四、k8s 镜像仓库 harbor 环境部署
- 1）安装 helm
- 2）配置 hosts
- 3）创建 stl 证书
- 4）安装 ingress
- 5）安装 nfs
- 6）创建 nfs provisioner 和持久化存储 SC
- 7）部署 Harbor（Https 方式）
1）前期准备（所有节点）
1、修改主机名和配置 hosts
先部署 1master 和 2node 节点，后面再加一个 master 节点
# 在192.168.0.113执行
hostnamectl set-hostname k8s-master-168-0-113
# 在192.168.0.114执行
hostnamectl set-hostname k8s-node1-168-0-114
# 在192.168.0.115执行
hostnamectl set-hostname k8s-node2-168-0-115
讯享网
配置 hosts
讯享网cat >> /etc/hosts<<EOF
192.168.0.113 k8s-master-168-0-113
192.168.0.114 k8s-node1-168-0-114
192.168.0.115 k8s-node2-168-0-115
EOF
2、配置 ssh 互信
# 直接一直回车就行
ssh-keygen
ssh-copy-id -i /.ssh/id_rsa.pub root@k8s-master-168-0-113
ssh-copy-id -i /.ssh/id_rsa.pub root@k8s-node1-168-0-114
ssh-copy-id -i /.ssh/id_rsa.pub root@k8s-node2-168-0-115
3、时间同步
讯享网yum install chrony -y
systemctl start chronyd
systemctl enable chronyd
chronyc sources
4、关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
5、关闭 swap
讯享网# 临时关闭；关闭swap主要是为了性能考虑
swapoff -a
# 可以通过这个命令查看swap是否关闭了
free
# 永久关闭
sed -ri ’s/.swap./#&/’ /etc/fstab
6、禁用 SELinux
# 临时关闭
setenforce 0
# 永久禁用
sed -i ’s/^SELINUX=enforcing$/SELINUX=disabled/’ /etc/selinux/config
7、允许 iptables 检查桥接流量（可选，所有节点）
若要显式加载此模块，请运行 sudo modprobe br_netfilter，通过运行 lsmod | grep br_netfilter 来验证 br_netfilter 模块是否已加载。
讯享网sudo modprobe br_netfilter
lsmod | grep br_netfilter
为了让 Linux 节点的 iptables 能够正确查看桥接流量，请确认 sysctl 配置中的 net.bridge.bridge-nf-call-iptables 设置为 1。例如：
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# 设置所需的 sysctl 参数，参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# 应用 sysctl 参数而不重新启动
sudo sysctl –system
2）安装容器 docker（所有节点）
提示：v1.24 之前的 Kubernetes 版本包括与 Docker Engine 的直接集成，使用名为 dockershim 的组件。这种特殊的直接整合不再是 Kubernetes 的一部分 （这次删除被作为 v1.20 发行版本的一部分宣布）。你可以阅读检查 Dockershim 弃用是否会影响你 以了解此删除可能会如何影响你。要了解如何使用 dockershim 进行迁移，请参阅从 dockershim 迁移。
讯享网# 配置yum源
cd /etc/yum.repos.d ; mkdir bak; mv CentOS-Linux-* bak/
# centos7
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# centos8
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo
# 安装yum-config-manager配置工具
yum -y install yum-utils
# 设置yum源
yum-config-manager –add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安装docker-ce版本
yum install -y docker-ce
# 启动
systemctl start docker
# 开机自启
systemctl enable docker
# 查看版本号
docker –version
# 查看版本具体信息
docker version
# Docker镜像源设置
# 修改文件 /etc/docker/daemon.json，没有这个文件就创建
# 添加以下内容后，重启docker服务：
cat >/etc/docker/daemon.json<<EOF
{
“registry-mirrors”: [“http://hub-mirror.c.163.com”]
}
EOF
# 加载
systemctl reload docker
# 查看
systemctl status docker containerd
【温馨提示】dockerd 实际真实调用的还是 containerd 的 api 接口，containerd 是 dockerd 和 runC 之间的一个中间交流组件。所以启动 docker 服务的时候，也会启动 containerd 服务的。
3）配置 k8s yum 源（所有节点）
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[k8s]
name=k8s
enabled=1
gpgcheck=0
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
EOF
4）将 sandbox_image 镜像源设置为阿里云 google_containers 镜像源（所有节点）
讯享网# 导出默认配置，config.toml这个文件默认是不存在的
containerd config default > /etc/containerd/config.toml
grep sandbox_image /etc/containerd/config.toml
sed -i “s#k8s.gcr.io/pause#registry.aliyuncs.com/google_containers/pause#g” /etc/containerd/config.toml
grep sandbox_image /etc/containerd/config.toml
5）配置 containerd cgroup 驱动程序 systemd（所有节点）
kubernets 自ｖ 1.24.0 后，就不再使用 docker.shim，替换采用 containerd 作为容器运行时端点。因此需要安装 containerd（在 docker 的基础下安装），上面安装 docker 的时候就自动安装了 containerd 了。这里的 docker 只是作为客户端而已。容器引擎还是 containerd。
sed -i ’s#SystemdCgroup = false#SystemdCgroup = true#g’ /etc/containerd/config.toml
# 应用所有更改后,重新启动containerd
systemctl restart containerd
6）开始安装 kubeadm，kubelet 和 kubectl（master 节点）
讯享网# 不指定版本就是最新版本，当前最新版就是1.24.1
yum install -y kubelet-1.24.1 kubeadm-1.24.1 kubectl-1.24.1 –disableexcludes=kubernetes
# disableexcludes=kubernetes：禁掉除了这个kubernetes之外的别的仓库
# 设置为开机自启并现在立刻启动服务 –now：立刻启动服务
systemctl enable –now kubelet
# 查看状态，这里需要等待一段时间再查看服务状态，启动会有点慢
systemctl status kubelet
查看日志，发现有报错，报错如下：
kubelet.service: Main process exited, code=exited, status=1/FAILURE kubelet.service: Failed with result ‘exit-code’.
【解释】重新安装（或第一次安装）k8s，未经过 kubeadm init 或者 kubeadm join 后，kubelet 会不断重启，这个是正常现象……，执行 init 或 join 后问题会自动解决，对此官网有如下描述，也就是此时不用理会 kubelet.service。
查看版本
kubectl version
yum info kubeadm
7）使用 kubeadm 初始化集群（master 节点）
最好提前把镜像下载好，这样安装快
讯享网docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.1
docker pull registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.1
docker pull registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.1
docker pull registry.aliyuncs.com/google_containers/kube-proxy:v1.24.1
docker pull registry.aliyuncs.com/google_containers/pause:3.7
docker pull registry.aliyuncs.com/google_containers/etcd:3.5.3-0
docker pull registry.aliyuncs.com/google_containers/coredns:v1.8.6
集群初始化
kubeadm init <br /> –apiserver-advertise-address=192.168.0.113 </span>
–image-repository registry.aliyuncs.com/google_containers </span>
–control-plane-endpoint=cluster-endpoint </span>
–kubernetes-version v1.24.1 </span>
–service-cidr=10.1.0.0/16 </span>
–pod-network-cidr=10.244.0.0/16 </span>
–v=5
# –image-repository string： 这个用于指定从什么位置来拉取镜像（1.13版本才有的），默认值是k8s.gcr.io，我们将其指定为国内镜像地址：registry.aliyuncs.com/google_containers
# –kubernetes-version string： 指定kubenets版本号，默认值是stable-1，会导致从https://dl.k8s.io/release/stable-1.txt下载最新的版本号，我们可以将其指定为固定版本（v1.22.1）来跳过网络请求。
# –apiserver-advertise-address 指明用 Master 的哪个 interface 与 Cluster 的其他节点通信。如果 Master 有多个 interface，建议明确指定，如果不指定，kubeadm 会自动选择有默认网关的 interface。这里的ip为master节点ip，记得更换。
# –pod-network-cidr 指定 Pod 网络的范围。Kubernetes 支持多种网络方案，而且不同网络方案对 –pod-network-cidr有自己的要求，这里设置为10.244.0.0/16 是因为我们将使用 flannel 网络方案，必须设置成这个 CIDR。
# –control-plane-endpoint cluster-endpoint 是映射到该 IP 的自定义 DNS 名称，这里配置hosts映射：192.168.0.113 cluster-endpoint。 这将允许你将 –control-plane-endpoint=cluster-endpoint 传递给 kubeadm init，并将相同的 DNS 名称传递给 kubeadm join。 稍后你可以修改 cluster-endpoint 以指向高可用性方案中的负载均衡器的地址。
【温馨提示】kubeadm 不支持将没有 –control-plane-endpoint 参数的单个控制平面集群转换为高可用性集群。
重置再初始化
讯享网kubeadm reset
rm -fr /.kube/ /etc/kubernetes/ var/lib/etcd/
kubeadm init <br /> –apiserver-advertise-address=192.168.0.113 </span>
–image-repository registry.aliyuncs.com/google_containers </span>
–control-plane-endpoint=cluster-endpoint </span>
–kubernetes-version v1.24.1 </span>
–service-cidr=10.1.0.0/16 </span>
–pod-network-cidr=10.244.0.0/16 </span>
–v=5
# –image-repository string： 这个用于指定从什么位置来拉取镜像（1.13版本才有的），默认值是k8s.gcr.io，我们将其指定为国内镜像地址：registry.aliyuncs.com/google_containers
# –kubernetes-version string： 指定kubenets版本号，默认值是stable-1，会导致从https://dl.k8s.io/release/stable-1.txt下载最新的版本号，我们可以将其指定为固定版本（v1.22.1）来跳过网络请求。
# –apiserver-advertise-address 指明用 Master 的哪个 interface 与 Cluster 的其他节点通信。如果 Master 有多个 interface，建议明确指定，如果不指定，kubeadm 会自动选择有默认网关的 interface。这里的ip为master节点ip，记得更换。
# –pod-network-cidr 指定 Pod 网络的范围。Kubernetes 支持多种网络方案，而且不同网络方案对 –pod-network-cidr有自己的要求，这里设置为10.244.0.0/16 是因为我们将使用 flannel 网络方案，必须设置成这个 CIDR。
# –control-plane-endpoint cluster-endpoint 是映射到该 IP 的自定义 DNS 名称，这里配置hosts映射：192.168.0.113 cluster-endpoint。 这将允许你将 –control-plane-endpoint=cluster-endpoint 传递给 kubeadm init，并将相同的 DNS 名称传递给 kubeadm join。 稍后你可以修改 cluster-endpoint 以指向高可用性方案中的负载均衡器的地址。
配置环境变量
mkdir -p \(HOME</span>/.kube<br />sudo cp -i /etc/kubernetes/admin.conf <span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)HOME/.kube/config
sudo chown \((id -u):\)(id -g) \(HOME</span>/.kube/config<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 临时生效（退出当前窗口重连环境变量失效）</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">export</span> KUBECONFIG=/etc/kubernetes/admin.conf<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 永久生效（推荐）</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">echo</span> <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"export KUBECONFIG=/etc/kubernetes/admin.conf"</span> >> ~/.bash_profile<br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">source</span> ~/.bash_profile</code></pre></section><p><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIgdDwUgQEw7kbqejV8WNiaic4tEa8w9ZHRDMDialNQtPxejiatNwrg65MRA/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0.07022" data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIgdDwUgQEw7kbqejV8WNiaic4tEa8w9ZHRDMDialNQtPxejiatNwrg65MRA/640?wx_fmt=png" data-type="png" data-w="1239" style="margin-right: auto;margin-bottom: 15px;margin-left: auto;outline: 0px;color: rgb(58, 58, 58);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, 'PingFang SC', Cambria, Cochin, Georgia, Times, 'Times New Roman', serif;font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);width: 656.992px;border-radius: 5px;display: block;box-sizing: border-box !important;visibility: visible !important;" /></p><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">发现节点还是有问题，查看日志 /var/log/messages</p><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">"Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"</p></blockquote></section><p><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIjdeqMOvbnwt8ZPnG8l5icpWCkVjbO3OktLuI6ul7XhSFel1fW41Dqng/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0.54578" data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIjdeqMOvbnwt8ZPnG8l5icpWCkVjbO3OktLuI6ul7XhSFel1fW41Dqng/640?wx_fmt=png" data-type="png" data-w="1671" style="margin-right: auto;margin-bottom: 15px;margin-left: auto;outline: 0px;color: rgb(58, 58, 58);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, 'PingFang SC', Cambria, Cochin, Georgia, Times, 'Times New Roman', serif;font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);width: 656.992px;border-radius: 5px;display: block;box-sizing: border-box !important;visibility: visible !important;" /></p><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">接下来就是安装 Pod 网络插件</p><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">8）安装 Pod 网络插件（CNI：Container Network Interface）(master)</span></h4><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">你必须部署一个基于 Pod 网络插件的 容器网络接口 (CNI)，以便你的 Pod 可以相互通信。</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 最好提前下载镜像（所有节点）</span><br />docker pull quay.io/coreos/<span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">flannel:</span>v<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">0</span>.<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">14.0</span><br />kubectl apply -f <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">https:</span>/<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">/raw.githubusercontent.com/coreos</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">/flannel/master</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">/Documentation/kube</span>-flannel.yml<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">如果上面安装失败，则下载我百度里的，离线安装</p><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">链接：https://pan.baidu.com/s/1HB9xuO3bssAW7v5HzpXkeQ<br />提取码：8888</p></blockquote><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">再查看 node 节点，就已经正常了</p></section><p><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIwVoAccOK71cSBicePavwV0l9vDzickCvSvYEEzicOUMpNNKLiaIq3iaebQA/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0." data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIwVoAccOK71cSBicePavwV0l9vDzickCvSvYEEzicOUMpNNKLiaIq3iaebQA/640?wx_fmt=png" data-type="png" data-w="1080" /></p><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">9）node 节点加入 k8s 集群</span></h4><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">先安装 kubelet</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">yum <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">install</span> -y kubelet kubeadm kubectl <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--disableexcludes=kubernetes</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 设置为开机自启并现在立刻启动服务 --now：立刻启动服务</span><br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">enable</span> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--now kubelet</span><br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">status</span> kubelet<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">如果没有令牌，可以通过在控制平面节点上运行以下命令来获取令牌：</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">kubeadm token <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">list</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">默认情况下，令牌会在24小时后过期。如果要在当前令牌过期后将节点加入集群， 则可以通过在控制平面节点上运行以下命令来创建新令牌：</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">kubeadm token <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">create</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 再查看</span><br />kubeadm token <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">list</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">如果你没有 –discovery-token-ca-cert-hash 的值，则可以通过在控制平面节点上执行以下命令链来获取它：</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">openssl x509 -pubkey -<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">2</span>><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">/dev/</span><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">null</span> | openssl dgst -sha256 -hex | sed <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">'s/^.* //'</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">如果执行 kubeadm init 时没有记录下加入集群的命令，可以通过以下命令重新创建（推荐）一般不用上面的分别获取 token 和 ca-cert-hash 方式，执行以下命令一气呵成：</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">kubeadm token create --<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">print</span>-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">join</span>-command<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">这里需要等待一段时间，再查看节点节点状态，因为需要安装 kube-proxy 和 flannel。</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">kubectl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">get</span> pods -A<br />kubectl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">get</span> nodes<br /></code></pre></section><p><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIWdNgPccicUv0VH9P2iat4LicQkgQzJd5aaGvCZcALmPLnh5HTZkGdZRNA/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0.037035" data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIWdNgPccicUv0VH9P2iat4LicQkgQzJd5aaGvCZcALmPLnh5HTZkGdZRNA/640?wx_fmt=png" data-type="png" data-w="1080"></p><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">10）配置 IPVS</span></h4><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【问题】集群内无法 ping 通 ClusterIP（或 ServiceName）</p><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">1、加载 ip_vs 相关内核模块</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">modprobe <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">-- ip_vs</span><br />modprobe <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">-- ip_vs_sh</span><br />modprobe <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">-- ip_vs_rr</span><br />modprobe <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">-- ip_vs_wrr</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">所有节点验证开启了 ipvs：</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">lsmod</span> |grep ip_vs<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">2、安装 ipvsadm 工具</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">yum</span> install ipset ipvsadm -y<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">3、编辑 kube-proxy 配置文件，mode 修改成 ipvs</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">kubectl</span> edit configmap -n kube-system kube-proxy</code></pre></section><p><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIcmZjHYP05sgZRyed3rcdvkyYN7IyTanKguq0GOdxSAGKbXHDe5THLA/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0.93877" data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIcmZjHYP05sgZRyed3rcdvkyYN7IyTanKguq0GOdxSAGKbXHDe5THLA/640?wx_fmt=png" data-type="png" data-w="784" style="margin-right: auto;margin-bottom: 15px;margin-left: auto;outline: 0px;color: rgb(0, 0, 0);font-family: Optima-Regular, Optima, PingFangSC-light, PingFangTC-light, 'PingFang SC', Cambria, Cochin, Georgia, Times, 'Times New Roman', serif;font-size: 16px;text-align: left;white-space: normal;background-color: rgb(255, 255, 255);width: 656.992px;border-radius: 5px;display: block;box-sizing: border-box !important;visibility: visible !important;" /></p><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">4、重启 kube-proxy</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 先查看</span><br />kubectl get pod -n kube-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">system</span> | <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">grep</span> kube-proxy<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 再delete让它自拉起</span><br />kubectl get pod -n kube-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">system</span> | <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">grep</span> kube-proxy |awk <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">'{system("kubectl delete pod "\)1” -n kube-system”)}’
# 再查看
kubectl get pod -n kube-system | grep kube-proxy
5、查看 ipvs 转发规则
讯享网ipvsadm -Ln
11）集群高可用配置
配置高可用（HA）Kubernetes 集群实现的两种方案：
使用堆叠（stacked）控制平面节点，其中 etcd 节点与控制平面节点共存（本章使用），架构图如下：
使用外部 etcd 节点，其中 etcd 在与控制平面不同的节点上运行，架构图如下：
这里新增一台机器作为另外一个 master 节点：192.168.0.116 配置跟上面 master 节点一样。只是不需要最后一步初始化了。
1、修改主机名和配置 hosts
所有节点都统一如下配置：
# 在192.168.0.113执行
hostnamectl set-hostname k8s-master-168-0-113
# 在192.168.0.114执行
hostnamectl set-hostname k8s-node1-168-0-114
# 在192.168.0.115执行
hostnamectl set-hostname k8s-node2-168-0-115
# 在192.168.0.116执行
hostnamectl set-hostname k8s-master2-168-0-116
配置 hosts
讯享网cat >> /etc/hosts<<EOF
192.168.0.113 k8s-master-168-0-113 cluster-endpoint
192.168.0.114 k8s-node1-168-0-114
192.168.0.115 k8s-node2-168-0-115
192.168.0.116 k8s-master2-168-0-116
EOF
2、配置 ssh 互信
# 直接一直回车就行
ssh-keygen
ssh-copy-id -i /.ssh/id_rsa.pub root@k8s-master-168-0-113
ssh-copy-id -i /.ssh/id_rsa.pub root@k8s-node1-168-0-114
ssh-copy-id -i /.ssh/id_rsa.pub root@k8s-node2-168-0-115
ssh-copy-id -i /.ssh/id_rsa.pub root@k8s-master2-168-0-116
3、时间同步
讯享网yum install chrony -y
systemctl start chronyd
systemctl enable chronyd
chronyc sources
关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
4、关闭 swap
讯享网# 临时关闭；关闭swap主要是为了性能考虑
swapoff -a
# 可以通过这个命令查看swap是否关闭了
free
# 永久关闭
sed -ri ’s/.swap./#&/’ /etc/fstab
5、禁用 SELinux
# 临时关闭
setenforce 0
# 永久禁用
sed -i ’s/^SELINUX=enforcing\(/SELINUX=disabled/'</span> /etc/selinux/config<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">6、允许 iptables 检查桥接流量（可选，所有节点）</p><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">若要显式加载此模块，请运行 sudo modprobe br_netfilter，通过运行 lsmod | grep br_netfilter 来验证 br_netfilter 模块是否已加载，</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">sudo</span> modprobe br_netfilter<br />lsmod | grep br_netfilter<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">为了让 Linux 节点的 iptables 能够正确查看桥接流量，请确认 sysctl 配置中的 net.bridge.bridge-nf-call-iptables 设置为 1。例如：</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf<br />overlay<br />br_netfilter<br />EOF<br /><br />sudo modprobe overlay<br />sudo modprobe br_netfilter<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 设置所需的 sysctl 参数，参数在重新启动后保持不变</span><br />cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf<br />net.bridge.bridge-nf-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">call</span>-iptables = <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">1</span><br />net.bridge.bridge-nf-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">call</span>-ip6tables = <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">1</span><br />net.ipv4.ip_forward = <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">1</span><br />EOF<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 应用 sysctl 参数而不重新启动</span><br />sudo sysctl --system<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">7、安装容器 docker（所有节点）</p><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">提示：v1.24 之前的 Kubernetes 版本包括与 Docker Engine 的直接集成，使用名为 dockershim 的组件。这种特殊的直接整合不再是 Kubernetes 的一部分 （这次删除被作为 v1.20 发行版本的一部分宣布）。你可以阅读检查 Dockershim 弃用是否会影响你 以了解此删除可能会如何影响你。要了解如何使用 dockershim 进行迁移，请参阅从 dockershim 迁移。</p></blockquote><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 配置yum源</span><br />cd /etc/yum.repos.d ; mkdir bak; mv CentOS-Linux-* bak/<br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># centos7</span><br />wget -O /etc/yum.repos.d/CentOS-Base.repo http:<span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">//mirrors.aliyun.com/repo/Centos-7.repo</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># centos8</span><br />wget -O /etc/yum.repos.d/CentOS-Base.repo http:<span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">//mirrors.aliyun.com/repo/Centos-8.repo</span><br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 安装yum-config-manager配置工具</span><br />yum -y install yum-utils<br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 设置yum源</span><br />yum-config-manager --<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">add</span>-repo http:<span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">//mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 安装docker-ce版本</span><br />yum install -y docker-ce<br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 启动</span><br />systemctl start docker<br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 开机自启</span><br />systemctl enable docker<br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 查看版本号</span><br />docker --version<br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 查看版本具体信息</span><br />docker version<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># Docker镜像源设置</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 修改文件 /etc/docker/daemon.json，没有这个文件就创建</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 添加以下内容后，重启docker服务：</span><br />cat >/etc/docker/daemon.json<<EOF<br />{<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"registry-mirrors"</span>: [<span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"http://hub-mirror.c.163.com"</span>]<br />}<br />EOF<br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 加载</span><br />systemctl reload docker<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;"># 查看</span><br />systemctl status docker containerd<br /></code></pre><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">【温馨提示】dockerd 实际真实调用的还是 containerd 的 api 接口，containerd 是 dockerd 和 runC 之间的一个中间交流组件。所以启动 docker 服务的时候，也会启动 containerd 服务的。</p></blockquote><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">8、配置 k8s yum 源（所有节点）</span></h4><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">cat > /etc/yum.repos.d/kubernetes.repo << EOF<br />[k8s]<br />name=k8s<br />enabled=1<br />gpgcheck=0<br />baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/<br />EOF<br /></code></pre><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">9、将 sandbox_image 镜像源设置为阿里云 google_containers 镜像源（所有节点）</span></h4><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"># 导出默认配置，<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">config</span>.toml这个文件默认是不存在的<br />containerd <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">config</span> default > /etc/containerd/<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">config</span>.toml<br />grep sandbox_image /etc/containerd/<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">config</span>.toml<br />sed -i <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"s#k8s.gcr.io/pause#registry.aliyuncs.com/google_containers/pause#g"</span> /etc/containerd/<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">config</span>.toml<br />grep sandbox_image /etc/containerd/<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">config</span>.toml<br /></code></pre></section><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;"><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIEfR6pMVkqBCwNebbX1ic3ScxYnxNiapkhIIniaUnedvShicQ2HEj7LybOg/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0." data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUIIEfR6pMVkqBCwNebbX1ic3ScxYnxNiapkhIIniaUnedvShicQ2HEj7LybOg/640?wx_fmt=png" data-type="png" data-w="1080" /></p></section><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">10、配置 containerd cgroup 驱动程序 systemd</span></h4><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">kubernets 自ｖ 1.24.0 后，就不再使用 docker.shim，替换采用 containerd 作为容器运行时端点。因此需要安装 containerd（在 docker 的基础下安装），上面安装 docker 的时候就自动安装了 containerd 了。这里的 docker 只是作为客户端而已。容器引擎还是 containerd。</p></blockquote><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">sed</span> -i <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">'s#SystemdCgroup = false#SystemdCgroup = true#g'</span> /etc/containerd/config.toml<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 应用所有更改后,重新启动containerd</span><br />systemctl restart containerd<br /></code></pre><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">11、开始安装 kubeadm，kubelet 和 kubectl（master 节点）</span></h4><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 不指定版本就是最新版本，当前最新版就是1.24.1</span><br />yum <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">install</span> -y kubelet<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">-1.24</span><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">.1</span> kubeadm<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">-1.24</span><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">.1</span> kubectl<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">-1.24</span><span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">.1</span> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--disableexcludes=kubernetes</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># disableexcludes=kubernetes：禁掉除了这个kubernetes之外的别的仓库</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 设置为开机自启并现在立刻启动服务 --now：立刻启动服务</span><br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">enable</span> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--now kubelet</span><br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 查看状态，这里需要等待一段时间再查看服务状态，启动会有点慢</span><br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">status</span> kubelet<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 查看版本</span><br /><br />kubectl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">version</span><br />yum info kubeadm<br /></code></pre><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">12、加入 k8s 集群</span></h4><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 证如果过期了，可以使用下面命令生成新证书上传，这里会打印出certificate key，后面会用到</span><br />kubeadm init phase upload-certs <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--upload-certs</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 你还可以在 【init】期间指定自定义的 --certificate-key，以后可以由 join 使用。 要生成这样的密钥，可以使用以下命令（这里不执行，就用上面那个自命令就可以了）：</span><br />kubeadm certs certificate-key<br /><br />kubeadm token <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">create</span> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--print-join-command</span><br /><br />kubeadm <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">join</span> cluster-endpoint:<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">6443</span> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--token wswrfw.fc81au4yvy6ovmhh --discovery-token-ca-cert-hash sha256:43a3924c25104d39f6a02b8ceef9f9c30eed8e0abc0f --control-plane --certificate-key 8db74e35d05a420bd2c19fd8c11914eb45f2ff22937b245bed5b68</span><br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># --control-plane 标志通知 kubeadm join 创建一个新的控制平面。加入master必须加这个标记</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># --certificate-key ... 将导致从集群中的 kubeadm-certs Secret 下载控制平面证书并使用给定的密钥进行解密。这里的值</span><br /></code></pre></section><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;"><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUII9nib9icxESZ7GzrDPHL3BLB1CKTMia0LkzIBvBjU9R11Rpw480BAJ2r3A/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0." data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUII9nib9icxESZ7GzrDPHL3BLB1CKTMia0LkzIBvBjU9R11Rpw480BAJ2r3A/640?wx_fmt=png" data-type="png" data-w="1080" /></p></section><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">根据提示执行如下命令：</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">mkdir</span> -p \)HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf \(HOME/.kube/config<br />sudo <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">chown</span> \)(id -u):\((id -g) \)HOME/.kube/config
查看
讯享网kubectl get nodes
kubectl get pods -A -owide
虽然现在已经有两个 master 了，但是对外还是只能有一个入口的，所以还得要一个负载均衡器，如果一个 master 挂了，会自动切到另外一个 master 节点。
12）部署 Nginx+Keepalived 高可用负载均衡器
1、安装 Nginx 和 Keepalived
# 在两个master节点上执行
yum install nginx keepalived -y
2、Nginx 配置
在两个 master 节点配置
讯享网cat > /etc/nginx/nginx.conf << “EOF”
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/.conf;
events {
worker_connections 1024;
}
# 四层负载均衡，为两台Master apiserver组件提供负载均衡
stream {
log_format main ‘\(remote_addr</span> <span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)upstream_addr - [\(time_local</span>] <span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)status \(upstream_bytes_sent</span>'</span>;<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">access_log</span> /var/log/nginx/k8s-access.log main;<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">upstream</span> k8s-apiserver {<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># Master APISERVER IP:PORT</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">server</span> <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">192.168.0.113:6443</span>;<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># Master2 APISERVER IP:PORT</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">server</span> <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">192.168.0.116:6443</span>;<br /> }<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(129, 162, 190);overflow-wrap: inherit !important;word-break: inherit !important;">server</span> {<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">listen</span> <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">16443</span>;<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">proxy_pass</span> k8s-apiserver;<br /> }<br />}<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(129, 162, 190);overflow-wrap: inherit !important;word-break: inherit !important;">http</span> {<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">log_format</span> main <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">'<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)remote_addr - \(remote_user</span> [<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)time_local] “\(request</span>" '</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">'<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)status \(body_bytes_sent</span> "<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)http_referer” ’
’”\(http_user_agent</span>" "<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)http_x_forwarded_for”’;
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
server {
listen 80 default_server;
servername ;
location / {
}
}
}
EOF
【温馨提示】如果只保证高可用，不配置 k8s-apiserver 负载均衡的话，可以不装 nginx，但是最好还是配置一下 k8s-apiserver 负载均衡。
3、Keepalived 配置（master）
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
}
notification_email_from
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}
vrrp_script check_nginx {
script ”/etc/keepalived/check_nginx.sh”
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
# 虚拟IP
virtual_ipaddress {
192.168.0.120/24
}
track_script {
check_nginx
}
}
EOF
vrrp_script：指定检查 nginx 工作状态脚本（根据 nginx 状态判断是否故障转移）
virtual_ipaddress：虚拟 IP（VIP）
检查 nginx 状态脚本：
讯享网cat > /etc/keepalived/check_nginx.sh << “EOF”
#!/bin/bash
count=\((ps -ef |grep nginx |egrep -cv <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"grep|\)\("</span>)<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">if</span> [ <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)count“ -eq 0 ];then
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh
4、Keepalived 配置（backup）
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from fage@.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}
vrrp_script check_nginx {
script ”/etc/keepalived/check_nginx.sh”
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.120/24
}
track_script {
check_nginx
}
}
EOF
检查 nginx 状态脚本：
讯享网cat > /etc/keepalived/check_nginx.sh << “EOF”
#!/bin/bash
count=\((ps -ef |grep nginx |egrep -cv <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"grep|\)\("</span>)<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">if</span> [ <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">\)count“ -eq 0 ];then
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_nginx.sh
5、启动并设置开机启动
systemctl daemon-reload
systemctl restart nginx && systemctl enable nginx && systemctl status nginx
systemctl restart keepalived && systemctl enable keepalived && systemctl status keepalived
查看 VIP
讯享网ip a
6、修改 hosts（所有节点）
将 cluster-endpoint 之前执行的 ip 修改执行现在的 VIP
192.168.0.113 k8s-master-168-0-113
192.168.0.114 k8s-node1-168-0-114
192.168.0.115 k8s-node2-168-0-115
192.168.0.116 k8s-master2-168-0-116
192.168.0.120 cluster-endpoint
7、测试验证
查看版本（负载均衡测试验证）
讯享网curl -k https://cluster-endpoint:16443/version
高可用测试验证，将 k8s-master-168-0-113 节点关机
shutdown -h now
curl -k https://cluster-endpoint:16443/version
kubectl get nodes -A
kubectl get pods -A
【温馨提示】堆叠集群存在耦合失败的风险。如果一个节点发生故障，则 etcd 成员和控制平面实例都将丢失， 并且冗余会受到影响。你可以通过添加更多控制平面节点来降低此风险。
1）dashboard 部署
GitHub 地址：https://github.com/kubernetes/dashboard
讯享网kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.0/aio/deploy/recommended.yaml
kubectl get pods -n kubernetes-dashboard
但是这个只能内部访问，所以要外部访问，要么部署 ingress，要么就是设置 service NodePort 类型。这里选择 service 暴露端口。
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.0/aio/deploy/recommended.yaml
修改后的内容如下：
讯享网# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the “License”);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Namespace
metadata:
name: kubernetes-dashboard
—
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
—
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 31443
selector:
k8s-app: kubernetes-dashboard
—
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
namespace: kubernetes-dashboard
type: Opaque
—
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
namespace: kubernetes-dashboard
type: Opaque
data:
csrf: ””
—
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
namespace: kubernetes-dashboard
type: Opaque
—
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
namespace: kubernetes-dashboard
—
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [””]
resources: [“secrets”]
resourceNames: [“kubernetes-dashboard-key-holder”, “kubernetes-dashboard-certs”, “kubernetes-dashboard-csrf”]
verbs: [“get”, “update”, “delete”]
# Allow Dashboard to get and update ‘kubernetes-dashboard-settings’ config map.
- apiGroups: [””]
resources: [“configmaps”]
resourceNames: [“kubernetes-dashboard-settings”]
verbs: [“get”, “update”]
# Allow Dashboard to get metrics.
- apiGroups: [””]
resources: [“services”]
resourceNames: [“heapster”, “dashboard-metrics-scraper”]
verbs: [“proxy”]
- apiGroups: [””]
resources: [“services/proxy”]
resourceNames: [“heapster”, “http:heapster:”, “https:heapster:”, “dashboard-metrics-scraper”, “http:dashboard-metrics-scraper”]
verbs: [“get”]
—
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: [“metrics.k8s.io”]
resources: [“pods”, “nodes”]
verbs: [“get”, “list”, “watch”]
—
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
—
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kubernetes-dashboard
—
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: kubernetes-dashboard
image: kubernetesui/dashboard:v2.6.0
imagePullPolicy: Always
ports:
- containerPort: 8443
protocol: TCP
args:
- –auto-generate-certificates
- –namespace=kubernetes-dashboard
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - –apiserver-host=http://my-address:port
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: kubernetes-dashboard-certs
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
nodeSelector:
“kubernetes.io/os”: linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
—
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: dashboard-metrics-scraper
—
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: dashboard-metrics-scraper
name: dashboard-metrics-scraper
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: dashboard-metrics-scraper
template:
metadata:
labels:
k8s-app: dashboard-metrics-scraper
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: dashboard-metrics-scraper
image: kubernetesui/metrics-scraper:v1.0.8
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- mountPath: /tmp
name: tmp-volume
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
serviceAccountName: kubernetes-dashboard
nodeSelector:
“kubernetes.io/os”: linux
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
volumes:
- name: tmp-volume
emptyDir: {}
重新部署
kubectl delete -f recommended.yaml
kubectl apply -f recommended.yaml
kubectl get svc,pods -n kubernetes-dashboard
2）创建登录用户
讯享网cat >ServiceAccount.yaml<<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
—
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOF
kubectl apply -f ServiceAccount.yaml
创建并获取登录 token
kubectl -n kubernetes-dashboard create token admin-user
3）配置 hosts 登录 dashboard web
讯享网192.168.0.120 cluster-endpoint
登录：https://cluster-endpoint:31443
输入上面创建的 token 登录
GitHub 地址：https://github.com/helm/helm/releases
这使用 helm 安装，所以得先安装 helm
1）安装 helm
mkdir -p /opt/k8s/helm && cd /opt/k8s/helm
wget https://get.helm.sh/helm-v3.9.0-rc.1-linux-amd64.tar.gz
tar -xf helm-v3.9.0-rc.1-linux-amd64.tar.gz
ln -s /opt/k8s/helm/linux-amd64/helm /usr/bin/helm
helm version
helm help
2）配置 hosts
讯享网192.168.0.120 myharbor.com
3）创建 stl 证书
mkdir /opt/k8s/helm/stl && cd /opt/k8s/helm/stl
# 生成 CA 证书私钥
openssl genrsa -out ca.key 4096
# 生成 CA 证书
openssl req -x509 -new -nodes -sha512 -days 3650 <br /> -subj ”/C=CN/ST=Guangdong/L=Shenzhen/O=harbor/OU=harbor/CN=myharbor.com” <br /> -key ca.key <br /> -out ca.crt
# 创建域名证书，生成私钥
openssl genrsa -out myharbor.com.key 4096
# 生成证书签名请求 CSR
openssl req -sha512 -new <br /> -subj ”/C=CN/ST=Guangdong/L=Shenzhen/O=harbor/OU=harbor/CN=myharbor.com” <br /> -key myharbor.com.key <br /> -out myharbor.com.csr
# 生成 x509 v3 扩展
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=myharbor.com
DNS.2=.myharbor.com
DNS.3=hostname
EOF
#创建 Harbor 访问证书
openssl x509 -req -sha512 -days 3650 <br /> -extfile v3.ext <br /> -CA ca.crt -CAkey ca.key -CAcreateserial <br /> -in myharbor.com.csr <br /> -out myharbor.com.crt
ingress 官方网站：https://kubernetes.github.io/ingress-nginx/
ingress 仓库地址：https://github.com/kubernetes/ingress-nginx
部署文档：https://kubernetes.github.io/ingress-nginx/deploy/
1、通过 helm 部署
讯享网helm upgrade –install ingress-nginx ingress-nginx <br /> –repo https://kubernetes.github.io/ingress-nginx </span>
–namespace ingress-nginx –create-namespace
2、通过 YAML 文件安装（本章使用这个方式安装 ingress）
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.0/deploy/static/provider/cloud/deploy.yaml
如果下载镜像失败，可以用以下方式修改镜像地址再安装
讯享网# 可以先把镜像下载，再安装
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.2.0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.0/deploy/static/provider/cloud/deploy.yaml
# 修改镜像地址
sed -i ’s@k8s.gcr.io/ingress-nginx/controller:v1.2.0(.)@registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.2.0@’ deploy.yaml
sed -i ’s@k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1(.)\(<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">@registry</span>.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1@'</span> deploy.yaml<br /><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">还需要修改两地方</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#1、kind: 类型修改成DaemonSet，replicas: 注销掉，因为DaemonSet模式会每个节点运行一个pod</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#2、在添加一条：hostnetwork：true</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#3、把LoadBalancer修改成NodePort</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#4、在--validating-webhook-key下面添加- --watch-ingress-without-class=true</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#5、设置master节点可调度</span><br />kubectl taint nodes k8s-master-<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">168</span>-<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">0</span>-<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">113</span> node-role.kubernetes.io/control-plane:NoSchedule-<br />kubectl taint nodes k8s-master2-<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">168</span>-<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">0</span>-<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">116</span> node-role.kubernetes.io/control-plane:NoSchedule-<br /><br />kubectl apply -f deploy.yaml<br /></code></pre></section><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;"><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUII2wm3sNm8uibdHiale81lVLkTsQ4xARicxZu30Im5zWmmONaJHCB8AzX9A/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0.85185" data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUII2wm3sNm8uibdHiale81lVLkTsQ4xARicxZu30Im5zWmmONaJHCB8AzX9A/640?wx_fmt=png" data-type="png" data-w="1080" /></p></section><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">5）安装 nfs</span></h4><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">1、所有节点安装 nfs</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">yum</span> -y install nfs-utils rpcbind<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">2、在 master 节点创建共享目录并授权</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">mkdir</span> /opt/nfsdata<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 授权共享目录</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">chmod</span> <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">666</span> /opt/nfsdata<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">3、配置 exports 文件</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">cat > <span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">/etc/exports</span><span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;"><<EOF<br />/opt/nfsdata *(rw,no_root_squash,no_all_squash,sync)<br />EOF</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 配置生效</span><br />exportfs -r<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">exportfs 命令</p><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">常用选项<br />-a 全部挂载或者全部卸载<br />-r 重新挂载<br />-u 卸载某一个目录<br />-v 显示共享目录 以下操作在服务端上</p></blockquote><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">4、启动 rpc 和 nfs（客户端只需要启动 rpc 服务）（注意顺序）</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">start</span> rpcbind<br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">start</span> nfs-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">server</span><br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">enable</span> rpcbind<br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">enable</span> nfs-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">server</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">查看</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">showmount</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">-e</span><br /># <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">VIP</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">showmount</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">-e</span> 192<span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.168</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.0</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.120</span><br /></code></pre><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">-e 显示 NFS 服务器的共享列表<br />-a 显示本机挂载的文件资源的情况 NFS 资源的情况<br />-v 显示版本号</p></blockquote><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">5、客户端</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 安装</span><br />yum -y <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">install</span> nfs-utils rpcbind<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 启动rpc服务</span><br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">start</span> rpcbind<br />systemctl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">enable</span> rpcbind<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 创建挂载目录</span><br />mkdir /mnt/nfsdata<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 挂载</span><br />echo <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"192.168.0.120:/opt/nfsdata /mnt/nfsdata nfs defaults 0 1"</span>>> /etc/fstab<br /><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">mount</span> -a<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">6、rsync 数据同步</p><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【1】rsync 安装</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 两端都得安装</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">yum</span> -y install rsync<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【2】配置<br />在/etc/rsyncd.conf 中添加</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">cat >/etc/rsyncd.conf<<EOF<br />uid = root<br />gid = root<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#禁锢在源目录</span><br />use chroot = yes<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#监听地址</span><br />address = 192.168.0.113<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#监听地址tcp/udp 873，可通过cat /etc/services | grep rsync查看</span><br />port 873<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#日志文件位置</span><br />log file = /var/log/rsyncd.log<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#存放进程 ID 的文件位置</span><br />pid file = /var/run/rsyncd.pid<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#允许访问的客户机地址</span><br />hosts allow = 192.168.0.0/16<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#共享模块名称</span><br />[nfsdata]<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#源目录的实际路径</span><br />path = /opt/nfsdata<br />comment = Document Root of www.kgc.com<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#指定客户端是否可以上传文件，默认对所有模块为 true</span><br />read only = yes<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#同步时不再压缩的文件类型</span><br />dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#授权账户，多个账号以空格分隔，不加则为匿名，不依赖系统账号</span><br />auth users = backuper<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#存放账户信息的数据文件</span><br />secrets file = /etc/rsyncd_users.db<br />EOF<br />配置 rsyncd_users.db<br /><br />cat >/etc/rsyncd_users.db<<EOF<br /><span style="font-size: inherit;line-height: inherit;color: rgb(129, 162, 190);overflow-wrap: inherit !important;word-break: inherit !important;">backuper:</span><br />EOF<br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#官方要求，最好只是赋权600！</span><br />chmod 600 /etc/rsyncd_users.db<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【3】rsyncd.conf 常用参数详解</p><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">rsyncd.conf 参数</p><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><table><thead style="font-size: inherit;color: inherit;line-height: inherit;"><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><th style="color: inherit;line-height: inherit;font-size: 1em;border-top-width: 1px;border-color: rgb(204, 204, 204);padding: 0.5em 1em;background-color: rgb(240, 240, 240);text-align: left;" width="144">rsyncd.conf 参数</th><th style="color: inherit;line-height: inherit;font-size: 1em;border-top-width: 1px;border-color: rgb(204, 204, 204);padding: 0.5em 1em;background-color: rgb(240, 240, 240);text-align: left;" width="405">参数说明</th></tr></thead><tbody style="font-size: inherit;color: inherit;line-height: inherit;border-width: 0px;border-style: initial;border-color: initial;"><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="145">uid=root</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="353">rsync 使用的用户。</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">gid=root</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="365">rsync 使用的用户组（用户所在的组）</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">use chroot=no</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="377">如果为 true，daemon 会在客户端传输文件前“chroot to the path”。这是一种安全配置，因为我们大多数都在内网，所以不配也没关系</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">max connections=200</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="389">设置最大连接数，默认 0，意思无限制，负值为关闭这个模块</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">timeout=400</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">默认为 0，表示 no timeout，建议 300-600（5-10 分钟）</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">pid file</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">rsync daemon 启动后将其进程 pid 写入此文件。如果这个文件存在，rsync 不会覆盖该文件，而是会终止</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">lock file</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">指定 lock 文件用来支持“max connections”参数，使得总连接数不会超过限制</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">log file</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">不设或者设置错误，rsync 会使用 rsyslog 输出相关日志信息</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">ignore errors</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">忽略 I/O 错误</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">read only=false</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">指定客户端是否可以上传文件，默认对所有模块为 true</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">list=false</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">是否允许客户端可以查看可用模块列表，默认为可以</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">hosts allow</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">指定可以联系的客户端主机名或和 ip 地址或地址段，默认情况没有此参数，即都可以连接</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">hosts deny</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">指定不可以联系的客户端主机名或 ip 地址或地址段，默认情况没有此参数，即都可以连接</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">auth users</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">指定以空格或逗号分隔的用户可以使用哪些模块，用户不需要在本地系统中存在。默认为所有用户无密码访问</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">secrets file</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">指定用户名和密码存放的文件，格式；用户名；密码，密码不超过 8 位</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">[backup]</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">这里就是模块名称，需用中括号扩起来，起名称没有特殊要求，但最好是有意义的名称，便于以后维护</td></tr><tr style="font-size: inherit;color: inherit;line-height: inherit;border-width: 1px 0px 0px;border-right-style: initial;border-bottom-style: initial;border-left-style: initial;border-right-color: initial;border-bottom-color: initial;border-left-color: initial;border-top-style: solid;border-top-color: rgb(204, 204, 204);background-color: white;"><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="146">path</td><td style="color: inherit;line-height: inherit;font-size: 1em;border-color: rgb(204, 204, 204);padding: 0.5em 1em;" width="393">这个模块中，daemon 使用的文件系统或目录，目录的权限要注意和配置文件中的权限一致，否则会遇到读写的问题</td></tr></tbody></table><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【4】rsync 常用命令参数详解</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">rsync --help<br /><br />rsync [选项] 原始位置 目标位置<br /><br />常用选项 说明<br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-r 递归模式，包含目录及子目录中的所有文件</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-l 对于符号链接文件仍然复制为符号链接文件</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-v 显示同步过程的详细信息</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-z 在传输文件时进行压缩goD</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-p 保留文件的权限标记</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-a 归档模式，递归并保留对象属性，等同于-rlpt</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-t 保留文件的时间标记</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-g 保留文件的属组标记（仅超级用户使用）</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-o 保留文件的属主标记（仅超级用户使用）</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-H 保留硬链接文件</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-A 保留ACL属性信息</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">-D 保留设备文件及其他特殊文件</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">--delete 删除目标位置有而原始位置没有的文件</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">--checksum 根据对象的校验和来决定是否跳过文件</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【5】启动服务（数据源机器）</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#rsync监听端口：873</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">#rsync运行模式：C/S</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">rsync</span> --daemon --config=/etc/rsyncd.conf<br />netstat -tnlp|grep :<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">873</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【6】执行命令同步数据</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 在目的机器上执行</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># rsync -avz 用户名@源主机地址/源目录 目的目录</span><br />rsync -avz root@192.<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">168.0</span>.<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">113</span><span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">:/opt/nfsdata/*</span> /opt/nfsdata/<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【7】crontab 定时同步</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(129, 162, 190);overflow-wrap: inherit !important;word-break: inherit !important;"># 配置crontab， 每五分钟同步一次，这种方式不好</span><br /><span style="font-size: inherit;color: inherit;line-height: inherit;font-style: italic;overflow-wrap: inherit !important;word-break: inherit !important;">*/5 *</span> <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">* *</span> <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">* rsync -avz root@192.168.0.113:/opt/nfsdata/*</span> /opt/nfsdata/<br /></code></pre><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">【温馨提示】crontab 定时同步数据不太好，可以使用rsync+inotify做数据实时同步，这里篇幅有点长了，先不讲，如果后面有时间会出一篇单独文章来讲。</p></blockquote><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">【温馨提示】这里跟我之前的文章有点不同，之前的方式也不适用新版本。</p></blockquote><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">GitHub 地址：https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner</p><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">helm 部署 nfs-subdir-external-provisioner</p><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">1、添加 helm 仓库</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">helm repo <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">add</span> nfs-subdir-external-provisioner https:<span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">//kubernetes-sigs.github.io/nfs-subdir-external-provisioner/</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">2、helm 安装 nfs provisioner</p><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">【温馨提示】默认镜像是无法访问的，这里使用 dockerhub 搜索到的镜像willdockerhub/nfs-subdir-external-provisioner:v4.0.2，还有就是 StorageClass 不分命名空间，所有在所有命名空间下都可以使用。</p></blockquote><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">helm <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">install</span> nfs-subdir-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">external</span>-provisioner nfs-subdir-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">external</span>-provisioner/nfs-subdir-<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">external</span>-provisioner \<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--namespace=nfs-provisioner \</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--create-namespace \</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--set image.repository=willdockerhub/nfs-subdir-external-provisioner \</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--set image.tag=v4.0.2 \</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--set replicaCount=2 \</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--set storageClass.name=nfs-client \</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--set storageClass.defaultClass=true \</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--set nfs.server=192.168.0.120 \</span><br /> <span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">--set nfs.path=/opt/nfsdata</span><br /></code></pre><blockquote style="line-height: inherit;padding: 15px 15px 15px 1rem;font-size: 0.9em;color: rgb(129, 145, 152);border-left-width: 6px;border-left-color: rgb(220, 230, 240);background: rgb(242, 247, 251);overflow: auto;overflow-wrap: normal;word-break: normal;"><p style="font-size: inherit;color: inherit;line-height: inherit;">【温馨提示】上面 nfs.server 设置为 VIP，可实现高可用。</p></blockquote><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">3、查看</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">kubectl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">get</span> pods,deploy,sc -n nfs-provisioner<br /></code></pre></section><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;"><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUII2QSbb1SefPToGciaJCDSUzqRh2OMuvWmHok4czX0bgibvXhq2Prnguyg/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0.74074" data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUII2QSbb1SefPToGciaJCDSUzqRh2OMuvWmHok4czX0bgibvXhq2Prnguyg/640?wx_fmt=png" data-type="png" data-w="1080"></p></section><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><h4 style="color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;font-weight: bold;font-size: 1.2em;"><span style="font-size: inherit;color: inherit;line-height: inherit;">7）部署 Harbor（Https 方式）</span></h4><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">1、创建 Namespace</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(240, 198, 116);overflow-wrap: inherit !important;word-break: inherit !important;">kubectl</span> create ns harbor<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">2、创建证书秘钥</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">kubectl</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">create</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">secret</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">tls</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">myharbor</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.com</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">--key</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">myharbor</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.com</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.key</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">--cert</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">myharbor</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.com</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.crt</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">-n</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">harbor</span><br /><span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">kubectl</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">get</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">secret</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">myharbor</span><span style="font-size: inherit;line-height: inherit;color: rgb(204, 102, 102);overflow-wrap: inherit !important;word-break: inherit !important;">.com</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">-n</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">harbor</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">3、添加 Chart 库</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">helm repo <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">add</span> harbor https:<span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">//helm.goharbor.io</span><br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">4、通过 helm 安装 harbor</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">helm install myharbor --<span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">namespace</span> harbor harbor/harbor \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> expose.ingress.hosts.core=myharbor.com \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> expose.ingress.hosts.notary=notary.myharbor.com \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span>-<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">string</span> expose.ingress.annotations<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">.'</span>nginx\.org/client-max-body-size'=<span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">"1024m"</span> \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> expose.tls.secretName=myharbor.com \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> persistence.persistentVolumeClaim.registry.storageClass=nfs-client \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> persistence.persistentVolumeClaim.jobservice.storageClass=nfs-client \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> persistence.persistentVolumeClaim.database.storageClass=nfs-client \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> persistence.persistentVolumeClaim.redis.storageClass=nfs-client \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> persistence.persistentVolumeClaim.trivy.storageClass=nfs-client \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> persistence.persistentVolumeClaim.chartmuseum.storageClass=nfs-client \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> persistence.enabled=<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">true</span> \<br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> externalURL=https:<span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;">//myharbor.com \</span><br /> --<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">set</span> harborAdminPassword=Harbor12345<br /></code></pre><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">这里稍等一段时间在查看资源状态</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">kubectl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">get</span> ingress,svc,pods,pvc -n harbor<br /></code></pre></section><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;"><img src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUII2peH4vy1awPnwZKNEkQ74icc6T84xJNNQFvIHJBqzMicF5xklGdGTeWA/640?wx_fmt=png" class="rich_pages wxw-img" data-ratio="0.475" data-src="https://mmbiz.qpic.cn/mmbiz_png/fEsWkVrSk548VEHHslAONpOdL84WRUII2peH4vy1awPnwZKNEkQ74icc6T84xJNNQFvIHJBqzMicF5xklGdGTeWA/640?wx_fmt=png" data-type="png" data-w="1080"></p><section style="font-size: 16px;color: rgb(62, 62, 62);line-height: 1.6;letter-spacing: 0px;font-family: 'Helvetica Neue', Helvetica, 'Hiragino Sans GB', 'Microsoft YaHei', Arial, sans-serif;"><p style="font-size: inherit;color: inherit;line-height: inherit;margin-top: 1.5em;margin-bottom: 1.5em;">【分析】，发现"error: endpoints “default-http-backend” not found"</p><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;">cat << EOF > <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend.yaml<br />---<br /><br />apiVersion: apps/v1<br />kind: Deployment<br />metadata:<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">name</span>: <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend<br /> labels:<br /> app: <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend<br /> namespace: harbor<br />spec:<br /> replicas: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">1</span><br /> selector:<br /> matchLabels:<br /> app: <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend<br /> template:<br /> metadata:<br /> labels:<br /> app: <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend<br /> spec:<br /> terminationGracePeriodSeconds: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">60</span><br /> containers:<br /> - <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">name</span>: <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend<br /> # Any image <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">is</span> permissible <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">as</span> long <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">as</span>:<br /> # <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">1</span>. It serves a <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">404</span> page at /<br /> # <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">2</span>. It serves <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">200</span> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">on</span> a /healthz endpoint<br /> image: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend:<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">1.4</span><br /># image: gcr.io/google_containers/defaultbackend:<span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">1.4</span><br /> livenessProbe:<br /> httpGet:<br /> path: /healthz<br /> port: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">8080</span><br /> scheme: HTTP<br /> initialDelaySeconds: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">30</span><br /> timeoutSeconds: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">5</span><br /> ports:<br /> - containerPort: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">8080</span><br /> resources:<br /> limits:<br /> cpu: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">10</span>m<br /> memory: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">20</span>Mi<br /> requests:<br /> cpu: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">10</span>m<br /> memory: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">20</span>Mi<br />---<br /><br />apiVersion: v1<br />kind: Service<br />metadata:<br /> <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">name</span>: <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend<br /> namespace: harbor<br /> labels:<br /> app: <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend<br />spec:<br /> ports:<br /> - port: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">80</span><br /> targetPort: <span style="font-size: inherit;line-height: inherit;color: rgb(222, 147, 95);overflow-wrap: inherit !important;word-break: inherit !important;">8080</span><br /> selector:<br /> app: <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend<br />EOF<br />kubectl apply -f <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">default</span>-http-backend.yaml<br /></code></pre><pre style="font-size: inherit;color: inherit;line-height: inherit;"><code style="margin-right: 2px;margin-left: 2px;line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;background: rgb(29, 31, 33);color: rgb(197, 200, 198);padding: 0.5em;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: inherit;line-height: inherit;color: rgb(150, 152, 150);overflow-wrap: inherit !important;word-break: inherit !important;"># 卸载</span><br />helm <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">uninstall</span> myharbor -n harbor<br />kubectl <span style="font-size: inherit;line-height: inherit;color: rgb(178, 148, 187);overflow-wrap: inherit !important;word-break: inherit !important;">get</span> pvc -n harbor| awk <span style="font-size: inherit;line-height: inherit;color: rgb(181, 189, 104);overflow-wrap: inherit !important;word-break: inherit !important;">'NR!=1{print \)1}’ | xargs kubectl delete pvc -n harbor
# 部署
helm install myharbor –namespace harbor harbor/harbor </span>
–set expose.ingress.hosts.core=myharbor.com </span>
–set expose.ingress.hosts.notary=notary.myharbor.com </span>
–set-string expose.ingress.annotations.‘nginx.org/client-max-body-size’=“1024m” </span>
–set expose.tls.secretName=myharbor.com </span>
–set persistence.persistentVolumeClaim.registry.storageClass=nfs-client </span>
–set persistence.persistentVolumeClaim.jobservice.storageClass=nfs-client </span>
–set persistence.persistentVolumeClaim.database.storageClass=nfs-client </span>
–set persistence.persistentVolumeClaim.redis.storageClass=nfs-client </span>
–set persistence.persistentVolumeClaim.trivy.storageClass=nfs-client </span>
–set persistence.persistentVolumeClaim.chartmuseum.storageClass=nfs-client </span>
–set persistence.enabled=true </span>
–set externalURL=https://myharbor.com
–set harborAdminPassword=Harbor12345
5、访问 harbor
“insecure-registries”:[“https://myharbor.com”]
重启 docker
讯享网systemctl restart docker
【3】服务器上登录 harbor
docker login https://myharbor.com
#账号/密码：admin/Harbor12345
【4】打标签并把镜像上传到 harbor
讯享网docker tag rancher/pause:3.6 myharbor.com/bigdata/pause:3.6
docker push myharbor.com/bigdata/pause:3.6
7、修改 containerd 配置
以前使用 docker-engine 的时候，只需要修改/etc/docker/daemon.json 就行，但是新版的 k8s 已经使用 containerd 了，所以这里需要做相关配置，要不然 containerd 会失败。证书（ca.crt）可以在页面上下载：
创建域名目录
mkdir /etc/containerd/myharbor.com
cp ca.crt /etc/containerd/myharbor.com/
配置文件：/etc/containerd/config.toml
讯享网[plugins.“io.containerd.grpc.v1.cri”.registry]
config_path = ””
[plugins.“io.containerd.grpc.v1.cri”.registry.auths]
[plugins.“io.containerd.grpc.v1.cri”.registry.configs]
[plugins.“io.containerd.grpc.v1.cri”.registry.configs.“myharbor.com”.tls]
ca_file = ”/etc/containerd/myharbor.com/ca.crt”
[plugins.“io.containerd.grpc.v1.cri”.registry.configs.“myharbor.com”.auth]
username = “admin”
password = “Harbor12345”
[plugins.“io.containerd.grpc.v1.cri”.registry.headers]
[plugins.“io.containerd.grpc.v1.cri”.registry.mirrors]
[plugins.“io.containerd.grpc.v1.cri”.registry.mirrors.“myharbor.com”]
endpoint = [“https://myharbor.com”]
重启 containerd
#重新加载配置
systemctl daemon-reload
#重启containerd
systemctl restart containerd
简单使用
讯享网# 把docker换成crictl 就行，命令都差不多
crictl pull myharbor.com/bigdata/mysql:5.7.38
执行 crictl 报如下错误的解决办法
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] unable to determine image API version: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory”
这个报错是 docker 的报错，这里没使用，所以这个错误不影响使用，但是还是解决好点，解决方法如下：
讯享网cat <<EOF> /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
再次拉取镜像
crictl pull myharbor.com/bigdata/mysql:5.7.38
Kubernetes（k8s）最新版最完整版基础环境部署+master 高可用实现详细步骤就到这里了
版权声明：本文内容由互联网用户自发贡献，该文观点仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容，请联系我们，一经查实，本站将立刻删除。
如需转载请保留出处：https://51itzy.com/kjqy/155261.html