k8s 集群从0到1

后端
字数统计: 2.9k   |   阅读时长约 14 分钟

前言

   本文只讲 docker + kubernetes 集群的 kubeadm 部署方式，不讲 k8s 各组件工作原理。个人认为在学习 k8s 的过程中，先动手搭建起来一个测试集群，比一味的看书学原理要好得多，一边操作一边学习是比较有效率的。

  使用 yum 安装 docker、 kubelet、 kubeadm 包的方式，会自动部署 service 文件及环境，相比二进制安装更简单一些。二进制安装更复杂但是也更灵活一些，学习二进制安装有助于用户学习各部件工作原理，以及排错能力。下面先介绍比较简单的 yum 安装的方式。

Yum 安装包部署k8s 集群流程

一、准备工作 （所有节点）

  1. 配置所有节点互通 ssh 免密（ssh-copy-id）

  2. 配置所有节点 /etc/hosts 文件统一记录所有机器 hostname 和ip。hosts 文件需要在新增节点时更新，在大型集群中最好通过 git 来同步所有机器。

    1
    2
    3
    4
    5
    6
    7
    [root@k8s-master-1 data]# cat /etc/hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    192.168.63.131 k8s-master-1
    192.168.63.141 k8s-node-1
    192.168.63.142 k8s-node-2
    192.168.63.143 k8s-node-3

  3. ntp 时间同步，k8s 很多组件工作需要保证所有节点 时间一致。

    1
    2
    3
    4
    5
    6
    7
    # yum install -y ntpdate

    # 使用公网 ntp
    # ntpdate us.pool.ntp.org

    时间不对的话确认系统时区
    # ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

  4. 关闭 防火墙，selinux ，swap

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    测试环境
    # systemctl stop firewalld
    # systemctl disable firewalld
    正式环境（不允许关闭防火墙，保证默认规则为 ACCEPT ）
    # iptables -L | grep Chain
    Chain INPUT (policy ACCEPT)
    Chain FORWARD (policy ACCEPT)
    Chain OUTPUT (policy ACCEPT)


        # setenforce 0
    # vim /etc/selinux/config
    SELINUX=disabled

    # swapoff -a
    ## 编辑/etc/fstab，注释掉包含swap的那一行即可，重启后可永久关闭

  5. 修改必要的内核参数

    1
    2
    3
    4
    5
    6
    7
    8
    9
    二层的网桥在转发包时也会被iptables的FORWARD规则所过滤
    linux主机有多个网卡时一个网卡收到的信息是否能够传递给其他的网卡

    # modprobe br_netfilter
    # vim /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.ipv4.ip_forward = 1
    # sysctl -p /etc/sysctl.d/k8s.conf

  6. 配置 yum 源装包。

    • 使用 docker-ce 官方 yum 源安装 docker-ce 软件包
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    # 第一种：官方提供的脚本
    # curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

    # 第二种：手动添加源安装
    $ sudo yum remove docker \
                     docker-client \
                     docker-client-latest \
                     docker-common \
                     docker-latest \
                     docker-latest-logrotate \
                    docker-logrotate \
                docker-engine
    $ sudo yum install -y yum-utils \
     device-mapper-persistent-data \
     lvm2

    $ sudo yum-config-manager \
      --add-repo \
      http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

    $ sudo yum install docker-ce docker-ce-cli containerd.io
    指定版本
    $ sudo yum install docker-ce-18.09.1 docker-ce-cli-18.09.1 containerd.io

    • 使用 阿里云的 yum 源安装 kubelet kubeadm kubectl 软件包
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    [root@k8s-master-1 data]# cat /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=0
    repo_gpgcheck=0
    gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
            http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

    [root@k8s-master-1 data]# yum install -y kubelet-1.15.6 kubeadm-1.15.6 kubectl-1.15.6

  7. 开启docker 服务

    1
    2
    # systemctl start docker ; systemctl enable docker
    # docker run hello-world      测试docker 是否可用

  8. 开启kubelet 服务

    1
    2
    # systemctl enable kubelet ; systemctl start kubelet
    # 发现kubelet 启动报错，这是正常的，因为缺少证书，kubeadm 部署后，即可正常启动。

  9. 确保 kubelet 和 docker 的 cgroup drive 一致，使用 systemd 作为initd 的发行版本，推荐使用systemd ，cgroup 管理更加稳定

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    # docker info | grep -i cgroup
    # cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

    ## 修改 docker
    # cat /etc/docker/daemon.json
    {
     "exec-opts":["native.cgroupdriver=systemd"]
    }

    ## 修改 kubelet
    # sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
    # systemctl daemon-reload

  10. 提前准备镜像，kubeadm 工具部署集群时需要用到一些镜像，默认去墙外的网站拉取，如果你的机器不能访问墙外的世界，那就需要提前准备，并且把准备的镜像打上默认需要的版本的 tag， 学习的路上总是有一些障碍的，还好阿里的源为我们准备了这些镜像

    • 查看kubeadm 需要的所有镜像
    1
    2
    3
    4
    5
    6
    7
    8
    [root@k8s-master-1 data]# kubeadm config images list --kubernetes-version=v1.16.2
    k8s.gcr.io/kube-apiserver:v1.16.2
    k8s.gcr.io/kube-controller-manager:v1.16.2
    k8s.gcr.io/kube-scheduler:v1.16.2
    k8s.gcr.io/kube-proxy:v1.16.2
    k8s.gcr.io/pause:3.1
    k8s.gcr.io/etcd:3.3.15-0
    k8s.gcr.io/coredns:1.6.2
    • 上面 k8s.gcr.io 的地址在墙外，我们换成阿里云地址就可以拉下来。registry.cn-hangzhou.aliyuncs.com/google_containers/kube-xxx:v1.16.2
    1
    2
    3
    4
    5
    6
    kubeadm config images list |sed -e 's/^/docker pull /g' -e 's#k8s.gcr.io#docker.io/registry.cn-hangzhou.aliyuncs.com/google_containers#g' |sh -x
    docker images |grep registry.cn-hangzhou.aliyuncs.com/google_containers |awk '{print "docker tag ",$1":"$2,$1":"$2}' |sed -e 's#docker.io/mirrorgooglecontainers#k8s.gcr.io#2' |sh -x
    docker images |grep registry.cn-hangzhou.aliyuncs.com/google_containers |awk '{print "docker rmi ", $1":"$2}' |sh -x
    docker pull coredns/coredns:1.2.2
    docker tag coredns/coredns:1.2.2 k8s.gcr.io/coredns:1.2.2
    docker rmi coredns/coredns:1.2.2

    • 如果我们的 k8s 版本在 1.13 以上 可以直接用 –image-repository 修改 kubeadm 拉取镜像的地址。

      kubeadm init –image-repository registry.aliyuncs.com/google_containers

二、部署 master 节点（master节点）

  1. 使用 kubeadm 初始化 k8s 集群 master 节点，可以看到 kubeadm 都做了那些工作。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    [root@k8s-master-1 ~]# kubeadm init --kubernetes-version=v1.16.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12；
    [init] Using Kubernetes version: v1.16.2
    [preflight] Running pre-flight checks
    	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09
    	[WARNING Hostname]: hostname "k8s-master-1" could not be reached
    	[WARNING Hostname]: hostname "k8s-master-1": lookup k8s-master-1 on 192.168.63.2:53: server misbehaving
    	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
    [preflight] Pulling images required for setting up a Kubernetes cluster
    [preflight] This might take a minute or two, depending on the speed of your internet connection
    [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Activating the kubelet service
    [certs] Using certificateDir folder "/etc/kubernetes/pki"
    [certs] Generating "ca" certificate and key
    [certs] Generating "apiserver" certificate and key
    [certs] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.63.131]
    [certs] Generating "apiserver-kubelet-client" certificate and key
    [certs] Generating "front-proxy-ca" certificate and key
    [certs] Generating "front-proxy-client" certificate and key
    [certs] Generating "etcd/ca" certificate and key
    [certs] Generating "etcd/server" certificate and key
    [certs] etcd/server serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.63.131 127.0.0.1 ::1]
    [certs] Generating "etcd/peer" certificate and key
    [certs] etcd/peer serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.63.131 127.0.0.1 ::1]
    [certs] Generating "etcd/healthcheck-client" certificate and key
    [certs] Generating "apiserver-etcd-client" certificate and key
    [certs] Generating "sa" key and public key
    [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
    [kubeconfig] Writing "admin.conf" kubeconfig file
    [kubeconfig] Writing "kubelet.conf" kubeconfig file
    [kubeconfig] Writing "controller-manager.conf" kubeconfig file
    [kubeconfig] Writing "scheduler.conf" kubeconfig file
    [control-plane] Using manifest folder "/etc/kubernetes/manifests"
    [control-plane] Creating static Pod manifest for "kube-apiserver"
    [control-plane] Creating static Pod manifest for "kube-controller-manager"
    [control-plane] Creating static Pod manifest for "kube-scheduler"
    [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
    [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
    [apiclient] All control plane components are healthy after 22.511803 seconds
    [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
    [kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
    [upload-certs] Skipping phase. Please see --upload-certs
    [mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
    [mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
    [bootstrap-token] Using token: tacwl8.r9rybowvi486j94t
    [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
    [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
    [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
    [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
    [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
    [addons] Applied essential addon: CoreDNS
    [addons] Applied essential addon: kube-proxy

    Your Kubernetes control-plane has initialized successfully!

    To start using your cluster, you need to run the following as a regular user:

      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/

    Then you can join any number of worker nodes by running the following on each as root:

    kubeadm join 192.168.63.131:6443 --token tacwl8.r9rybowvi486j94t \
        --discovery-token-ca-cert-hash sha256:83d9e155464739407fe5a782b41eac35c42fcba2a1e8c252066cd7e00eaf21ff

  2. 初始化后根据 提示 完成 config 文件的配置，并且记住 kubeadm join 命令，以便于后续添加node 节点。

    1
    2
    3
    # mkdir -p $HOME/.kube
    # sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    # sudo chown $(id -u):$(id -g) $HOME/.kube/config

  3. 部署网络插件,使用 flannel 或者 calico 需要自己选择。calico 需要配置rbac ，新版本已经集成到 calico.yaml文件中。

    1
    2
    3
    4
    5
    6
    7
    8
    # kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    或
    # kubectl apply -f https://docs.projectcalico.org/v3.9/manifests/calico-etcd.yaml

    如果没有关闭防火墙需要为calico 打开端口
    # firewall-cmd --permanent --add-port=5543/tcp --zone=public
    # firewall-cmd --permanent --add-port=179/tcp --zone=public
    # firewall-cmd --reload

  4. 部署 dashboard web ui 管理界面。

    • 事先准备镜像

      1
      2
      3
      # docker pull registry.cn-hangzhou.aliyuncs.com/rsqlh/kubernetes-dashboard:v1.10.1
      # docker tag registry.cn-hangzhou.aliyuncs.com/rsqlh/kubernetes-dashboard:v1.10.1 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
      # docker rmi registry.cn-hangzhou.aliyuncs.com/rsqlh/kubernetes-dashboard:v1.10.1

    • 应用 yaml 文件

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      [root@k8s-master-1 data]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
      secret/kubernetes-dashboard-certs created
      serviceaccount/kubernetes-dashboard created
      role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
      rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
      deployment.apps/kubernetes-dashboard created
      service/kubernetes-dashboard created

      查看pod
      # kubectl get pods -n kube-system
      kubernetes-dashboard-57df4db6b-p9sm8      1/1     Running   0          15s

    • 开启集群外部访问

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      [root@k8s-master-1 data]# kubectl get svc -n kube-system
      NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
      kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   8d
      kubernetes-dashboard   ClusterIP   10.110.144.227   <none>        443/TCP                  40s
      [root@k8s-master-1 data]# kubectl patch svc kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}' -n kube-system
      service/kubernetes-dashboard patched
      [root@k8s-master-1 data]# kubectl get svc -n kube-system
      NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
      kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   8d
      kubernetes-dashboard   NodePort    10.110.144.227   <none>        443:30541/TCP            51s

    • 通过 ip 和端口 访问会看到需要 token 验证或者 kubeconfig 文件验证。具体的token 获取方法需要创建 serivceaccount 然后 拿到secret 的token 即可，这里不详细阐述。

三、部署 node 节点加入集群（node节点）

  1. 使用上面记住的命令 初始化 node 节点。

    1
    2
    3
    4
    5
    # kubeadm join 192.168.63.131:6443 --token tacwl8.r9rybowvi486j94t --discovery-token-ca-cert-hash sha256:83d9e155464739407fe5a782b41eac35c42fcba2a1e8c252066cd7e00eaf21ff

    ## 如果没有关闭防火墙
    # firewall-cmd --permanent --add-port=6443/tcp
    # firewall-cmd --reload

  2. 如果在node 节点需要使用kubectl 命令 ，需要为对应用户配置 config 文件。

    1
    2
    3
    # mkdir -p $HOME/.kube
    # scp -i master:/etc/kubernetes/admin.conf $HOME/.kube/config
    # sudo chown $(id -u):$(id -g) $HOME/.kube/config

  3. kubectl get pod –all-namespaces 检查 pod 运行情况

参考文档

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

https://github.com/kubernetes/dashboard



结束福利

开源实战利用 k8s 作微服务的架构设计代码:

https://gitee.com/damon_one/spring-cloud-k8s

欢迎大家 star，多多指教。


关于作者

  笔名：Damon，技术爱好者，长期从事 Java 开发、Spring Cloud 的微服务架构设计，以及结合 Docker、K8s 做微服务容器化，自动化部署等一站式项目部署、落地。目前主要从事基于 K8s 云原生架构研发的工作。Golang 语言开发，长期研究边缘计算框架 KubeEdge、调度框架 Volcano 等。公众号 交个朋友之猿天地 发起人。个人微信 DamonStatham，星球：《交个朋友之猿田地》，个人网站：交个朋友之猿天地 | 微服务 | 容器化 | 自动化，欢迎來撩。


欢迎关注：InfoQ

欢迎关注：腾讯自媒体专栏


欢迎关注

公号：交个朋友之猿天地

公号：damon8

公号：天山六路折梅手


打赏
  • 版权声明： 本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！
  • Copyrights © 2020-2023 交个朋友之猿天地
  • Powered By Hexo | Title - Nothing
  • 访问人数: | 浏览次数:

请我喝杯咖啡吧~

支付宝
微信