在CentOS 7中使用kubeadm搭建kubernetes集群

在CentOS 7上使用kubeadm安装kubernetes cluster的方法。

安装公共组件

1
sudo yum install -y wget net-tools vim

安装Docker

CentOS 7中安装docker参见之前文章Docker基础

安装Kubernetes

添加kubenetes的yum源

在Master和Node中同时添加阿里云的yum源。

1
2
3
4
5
6
7
8
9
sudo cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装kubelet, kubeadm和kubectl

在Master和Node中安装kubelet, kubeadm和kubectl。

1
2
3
sudo yum install -y kubelet kubeadm kubectl
sudo systemctl enable kubelet
sudo systemctl start kubelet

设置主机名

设置主机名, 根据预先的规划，设置主机名字。

1
2
3
sudo hostnamectl set-hostname k8smaster01
sudo hostnamectl set-hostname k8snode01
sudo hostnamectl set-hostname k8snode02

在master上修改主机记录, 这边IP地址修改为自己的IP地址。

1
2
3
4
5
sudo cat >> /etc/hosts << EOF
192.168.187.51 k8smaster01 
192.168.187.55 k8snode01
192.168.187.56 k8snode02
EOF

配置防火墙

k8s默认使用一些固定的端口来进行通讯, 参见Check required ports

master节点需要的端口

Protocol Direction Port Range Purpose Used By
TCP Inbound 6443* Kubernetes API server All
TCP Inbound 2379-2380 etcd server client API kube-apiserver, etcd
TCP Inbound 10250 Kubelet API Self, Control plane
TCP Inbound 10251 kube-scheduler Self
TCP Inbound 10252 kube-controller-manager Self

Node节点需要打开的端口

Protocol Direction Port Range Purpose Used By
TCP Inbound 10250 Kubelet API Self, Control plane
TCP Inbound 30000-32767 NodePort Services† All

所以，在Master节点上，执行如下命令开启对应的端口

1
2
3
4
5
6
7
sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=2379-2380/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=10251/tcp
sudo firewall-cmd --permanent --add-port=10252/tcp
sudo firewall-cmd --permanent --add-port=10255/tcp
sudo firewall-cmd --reload

在Node节点上，执行如下命令开启对应的端口

1
2
3
4
sudo firewall-cmd --permanent --add-port=10251/tcp
sudo firewall-cmd --permanent --add-port=10255/tcp
sudo firewall-cmd --permanent --add-port=30000-32767/tcp
firewall-cmd --reload

更新iptable设置

在Master和Node节点上，更新iptable设置，确保桥接的 IPv4 流量传递到 iptables 的链

1
2
3
4
5
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

设置SELinux为permissive mode

在Master和Node节点上，设置SELinux为permissive模式

1
2
3
# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

禁用swap分区

在Master和Node节点上，禁用swap分区。

1
2
sudo sed -i '/swap/d' /etc/fstab
sudo swapoff -a

时钟同步

在Master和Node节点上，安装ntp来同步时间。

1
2
sudo yum install ntpdate -y
sudo ntpdate time.windows.com

安装k8s cluster

Master节点中kubeadm init

在Master节点上，使用kubeadm init来初始化kubeadm。

  • 参数--image-repository来设置image来源为阿里云
  • 参数--pod-network-cidr来设置Pod的网路地址快，需要和后续使用的Pod网络插件中设置的网络地址块相同。
    1
    2
    3
    4
    kubeadm init \
    --image-repository registry.aliyuncs.com/google_containers \
    --service-cidr=10.96.0.0/12 \
    --pod-network-cidr=10.244.0.0/16

完成后，会有kubeadm join命令的输出，使用该命令来将worker node添加到k8s集群中。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.187.51:6443 --token igzzla.bqinr52oatzfcej0 \
    --discovery-token-ca-cert-hash sha256:df3922b35e6fe58aca068d85ad832566f7d3aeff0cafac69e9860944ef3e7e54

设置用户使用kubelet

在Master节点上的用户目录下，执行如下命令，赋予当前用户访问k8s资源的权限。

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

设置Pod网络

k8s自身没有实现Pod网络组件功能，Pod网络是由第三方插件来实现的。具体参见Installing a Pod network add-on
此处采用flannel。

1
sudo kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

使用kubectl get nodes检查cluster中node节点状态。应该可以看到状态为Ready。

将worker node加入cluster

在worker节点上执行之前master节点kubeadm init输出的kubeadm join命令。来将worker node加入到cluster中。

1
2
kubeadm join 192.168.187.51:6443 --token igzzla.bqinr52oatzfcej0 \
    --discovery-token-ca-cert-hash sha256:df3922b35e6fe58aca068d85ad832566f7d3aeff0cafac69e9860944ef3e7e54

加入完毕后，在master节点中kubectl get nodes，就可以看到worker node已经加入cluster了。

遇到的问题

kubeadm join失败

在worker node中执行kubeadm join加入cluster时，报错如下: error execution phase kubelet-start: a Node with name "k8smaster01" and status "Ready" already exists in the cluster. You must delete the existing Node or change the name of this new joining Node

1
2
3
4
5
6
7
8
9
[root@localhost kubernetes]# kubeadm join 192.168.187.51:6443 --token igzzla.bqinr52oatzfcej0 --discovery-token-ca-cert-hash sha256:df3922b35e6fe58aca068d85ad832566f7d3aeff0cafac69e9860944ef3e7e54
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING Hostname]: hostname "k8smaster01" could not be reached
	[WARNING Hostname]: hostname "k8smaster01": lookup k8smaster01 on 192.168.187.2:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
error execution phase kubelet-start: a Node with name "k8smaster01" and status "Ready" already exists in the cluster. You must delete the existing Node or change the name of this new joining Node
To see the stack trace of this error execute with --v=5 or higher

原因是误将worker node的hostname设置为和master节点一样的k8smaster01，导致在worker node上输入kubeadm join时，报了如上错误。修正hostname后，问题解决。

CentOS下kubelet命令自动补全

执行如下命令来启动kubelet启动命令自动补全功能。

1
2
3
4
5
k8s 命令自动补全 
yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

留言

▲