一.集群规划

环境规划:

  • pod网段:10.244.0.0/16
  • service网段:10.10.0.0/16
  • 注意: pod和service网段不可冲突,如果冲突会导致K8S集群安装失败。

需要三台虚拟机,搭载组件如下

节点名部署组件ip配置
master-1etcd, kube-apiserver, kube-controller-manager, kube-scheduler192.168.10.1472C2G
node-1etcd, kubelet, kube-proxy, docker, flannel192.168.10.1482C2G
node-2etcd, kubelet, kube-proxy, docker,flannel192.168.10.1492C2G

二.系统初始化准备(所有节点同步操作)

1、关闭防火墙

systemctl disable firewalld --now
setenforce 0
sed  -i -r 's/SELINUX=[ep].*/SELINUX=disabled/g' /etc/selinux/config

2、配置域名解析

cat  >> /etc/hosts << EOF
192.168.10.147 master-1
192.168.10.148 node-1
192.168.10.149 node-2
EOF

在指定主机上面修改主机名

hostnamectl set-hostname master-1 && bash
hostnamectl set-hostname node-1 && bash
hostnamectl set-hostname node-2 && bash

3、配置服务器时间保持一致

yum -y install ntpdate
ntpdate ntp1.aliyun.com

添加定时同步 每天凌晨1点自动同步时间

echo "0 1 * * * ntpdate ntp1.aliyun.com" >> /var/spool/cron/root
crontab -l

4、禁用swap交换分区(kubernetes强制要求禁用)

swapoff --all

禁止开机自启动swap交换分区

sed -i -r '/swap/ s/^/#/' /etc/fstab

5、修改Linux内核参数,添加网桥过滤器和地址转发功能

cat >> /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
modprobe  br_netfilter 
sysctl -p /etc/sysctl.d/kubernetes.conf

加载网桥过滤器模块

lsmod | grep br_netfilter # 验证是否生效
[root@master-1 ~]# lsmod | grep br_netfilter
br_netfilter           22256  0 
bridge                155432  1 br_netfilter

6、配置ipvs功能

在kubernetes中Service有两种代理模型,一种是基于iptables的,一种是基于ipvs,两者对比ipvs的性能要高,如果想要使用ipvs模型,需要手动载入ipvs模块

yum -y install ipset ipvsadm

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4  
EOF

chmod +x /etc/sysconfig/modules/ipvs.modules 
# 执行脚本
/etc/sysconfig/modules/ipvs.modules

# 验证ipvs模块
lsmod | grep -e ip_vs -e nf_conntrack_ipv4
[root@master-1 ~]# lsmod | grep -e ip_vs -e nf_conntrack_ipv4
nf_conntrack_ipv4      19149  0 
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
ip_vs_sh               12688  0 
ip_vs_wrr              12697  0 
ip_vs_rr               12600  0 
ip_vs                 145458  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          143411  2 ip_vs,nf_conntrack_ipv4
libcrc32c              12644  3 xfs,ip_vs,nf_conntrack

7、安装Docker容器组件

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum makecache

# yum-utils软件用于提供yum-config-manager程序

yum install -y yum-utils

# 使用yum-config-manager创建docker阿里存储库

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum install docker-ce-20.10.6 docker-ce-cli-20.10.6 -y

Docker配置加速源:

mkdir /etc/docker
cat <<EOF > /etc/docker/daemon.json
{
  "registry-mirrors": ["https://aoewjvel.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

# 启动docker并设置开机自启

systemctl enable docker --now
systemctl status docker

8、重启服务器 可略过

reboot

三、安装并配置cri-dockerd插件

三台服务器同时操作

1、安装cri-dockerd插件

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd-0.3.1-3.el7.x86_64.rpm
rpm -ivh cri-dockerd-0.3.1-3.el7.x86_64.rpm

2、备份并更新cri-docker.service文件

mv /usr/lib/systemd/system/cri-docker.service{,.default}
vim /usr/lib/systemd/system/cri-docker.service 
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target

3、启动cir-dockerd

systemctl daemon-reload
systemctl start cri-docker.service 
systemctl enable cri-docker.service 

四、安装kubeadm(所有节点同步操作)

1、配置国内yum源,一键安装 kubeadm、kubelet、kubectl

cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 EOF yum install -y kubelet-1.27.0 kubeadm-1.27.0 kubectl-1.27.0

2、kubeadm将使用kubelet服务以容器方式部署kubernetes的主要服务,所以需要先启动kubelet服务

systemctl enable kubelet.service --now

五、初始化集群

在master-1主机上进行操作

1、生成初始化默认配置文件

kubeadm config print init-defaults > kubeadm.yaml

我们根据自己需求进行修改默认配置文件,我主要更改了一下配置如下:

  • advertiseAddress:更改为master的IP地址
  • criSocket:指定容器运行时
  • imageRepository:配置国内加速源地址
  • podSubnet:pod网段地址
  • serviceSubnet:services网段地址
  • 末尾添加了指定使用ipvs,开启systemd
  • nodeRegistration.name:改为当前主机名称

最终初始化配置文件如下:

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.10.147
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/cri-dockerd.sock
  imagePullPolicy: IfNotPresent
  name: master-1
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.27.0
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

2、进行初始化

kubeadm init --config=kubeadm.yaml --ignore-preflight-errors=SystemVerificatio

初始化成功后输出如下内容:

[init] Using Kubernetes version: v1.27.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0308 16:31:16.246138    9236 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master-1] and IPs [10.96.0.1 192.168.10.147]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master-1] and IPs [192.168.10.147 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master-1] and IPs [192.168.10.147 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
W0308 16:32:46.127811    9236 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0)
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 16.003367 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master-1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master-1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.147:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:d576b0a9486cfe2b4b538b6e7e0ff82747f1e374127a41c4524323a50926d8d4 

3、配置kubectl的配置文件config,相当于对kubectl进行授权,这样kubectl命令可以使用这个证书对k8s集群进行管理

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

验证使用可以使用 kubectl 命令

kubectl get nodes
[root@master-1 ~]# kubectl get nodes
NAME       STATUS     ROLES           AGE   VERSION
master-1   NotReady   control-plane   2m    v1.27.0

六、Node节点添加到集群

在两台node节点进行操

1、创建并查看token

使用以下命令创建并查看token(在master主机执行)

kubeadm token create --print-join-command

2、节点机添加到master主机

在两台node节点执行,注意添加--cri-socket=指定cri-dockerd.sock。

#注意每个人生成的token不一样,以自己生成的为准,不要直接粘贴复制
kubeadm join 192.168.10.147:6443 --token 36vw7l.w4spujb7xnbtwvge --discovery-token-ca-cert-hash sha256:d576b0a9486cfe2b4b538b6e7e0ff82747f1e374127a41c4524323a50926d8d4 --cri-socket=unix:///var/run/cri-dockerd.sock

如果出现以下报错

[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR CRI]: container runtime is not running: output: E0308 16:44:06.189189    9882 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/cri-dockerd.sock: connect: no such file or directory\""
time="2024-03-08T16:44:06+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/cri-dockerd.sock: connect: no such file or directory\""
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

尝试启动

systemctl enable cri-docker

成功加入到集群如下:

[root@node-1 ~]# kubeadm join 192.168.10.147:6443 --token 36vw7l.w4spujb7xnbtwvge --discovery-token-ca-cert-hash sha256:d576b0a9486cfe2b4b538b6e7e0ff82747f1e374127a41c4524323a50926d8d4 --cri-socket=unix:///var/run/cri-dockerd.sock
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

3、给两台node节点打上标签

master-1主机上执行

kubectl label nodes node-1 node-role.kubernetes.io/work=work
kubectl label nodes node-2 node-role.kubernetes.io/work=work

4、查看集群节点

[root@master-1 ~]# kubectl get nodes
NAME       STATUS     ROLES           AGE   VERSION
master-1   NotReady   control-plane   22m   v1.27.0
node-1     NotReady   work            16m   v1.27.0
node-2     NotReady   work            10s   v1.27.0

七、安装网络组件Calico

Calico在线文档地址:

1、上传calico.yaml文件到服务器中,下面提供calico.yaml文件内容:

Calico.yaml下载地址1(度盘方式):

Calico.yaml下载地址2(博客云盘方式):

PS:这里因为使用的是Kubeadm 1.27.0版本,对应的calico为3.25,注意版本对应关系,否则无法正常安装插件

在master主机执行

kubectl apply -f  calico.yaml

2、查看集群状态 && 查看自带Pod状态

kubectl get [root@master-1 ~]# kubectl get nodes
NAME       STATUS     ROLES           AGE   VERSION
master-1   NotReady   control-plane   38m   v1.27.0
node-1     NotReady   work            32m   v1.27.0
node-2     NotReady   work            16m   v1.27.0

3、查看组件状态 是否为 Running状态 如下

[root@master-1 ~]# kubectl get pods -n kube-system -o wide
NAME                                       READY   STATUS    RESTARTS   AGE     IP               NODE       NOMINATED NODE   READINESS GATES
calico-kube-controllers-75c9ddd877-5zgzt   1/1     Running   0          2m35s   10.244.247.3     node-2     <none>           <none>
calico-node-2zj4p                          1/1     Running   0          2m35s   192.168.10.149   node-2     <none>           <none>
calico-node-ghpw7                          1/1     Running   0          2m35s   192.168.10.147   master-1   <none>           <none>
calico-node-rbvng                          1/1     Running   0          2m35s   192.168.10.148   node-1     <none>           <none>
coredns-65dcc469f7-5z95c                   1/1     Running   0          26m     10.244.247.1     node-2     <none>           <none>
coredns-65dcc469f7-zdh9x                   1/1     Running   0          26m     10.244.247.2     node-2     <none>           <none>
etcd-master-1                              1/1     Running   0          26m     192.168.10.147   master-1   <none>           <none>
kube-apiserver-master-1                    1/1     Running   0          26m     192.168.10.147   master-1   <none>           <none>
kube-controller-manager-master-1           1/1     Running   0          26m     192.168.10.147   master-1   <none>           <none>
kube-proxy-59rmr                           1/1     Running   0          23m     192.168.10.148   node-1     <none>           <none>
kube-proxy-b62dm                           1/1     Running   0          21m     192.168.10.149   node-2     <none>           <none>
kube-proxy-ngv24                           1/1     Running   0          26m     192.168.10.147   master-1   <none>           <none>
kube-scheduler-master-1                    1/1     Running   0          26m     192.168.10.147   master-1   <none>           <none>

八、测试CoreDNS解析可用性

1、下载busybox:1.28镜像

docker pull busybox:1.28

2、测试coredns

kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh

If you don't see a command prompt, try pressing enter.

/ # nslookup kubernetes.default.svc.cluster.local
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default.svc.cluster.local
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

参考资料

【Kubernetes部署篇】Kubeadm方式搭建K8s集群 1.27.0版本

System requirements | Calico Documentation

关于k8s安装calico “calico.yaml“: no matches for kind “PodDisruptionBudget“ in version “policy/v1“

是一名喜欢每天折腾的咸鱼! 也是一名半退役的算竞摸鱼选手,参与过icpc,天梯赛,蓝桥等比赛.Datawhale 优秀队长 --------------------------------------------------- 认证类: 华为 Harmony OS应用开发者高级认证, NISP 一级认证, H3C NE-RS网络工程师认证 --------------------------------------------------- 荣获奖项荣誉: 第十八届“挑战杯”全国大学生课外学术科技作品竞赛 “揭榜挂帅”专项赛全国特等奖、 “美亚杯”第八届中国电子取证大赛 三等奖、 中国高校计算机大赛-团体程序天梯赛 省高校一等奖、 “蓝桥杯”省二等奖、 H3C新华三杯 省三等奖、 中国移动“梧桐杯”大数据创新大赛 省三等奖、 百度 飞桨领航团 金牌团长 https://pan.imgs.run/s/GQFr
最后更新于 2024-03-09