약 1개월 전 (2026년 1월 21일)

k8s upgrade by kubeadm

15 views

블로그 불러오는 중...

이 블로그는 NextJS 14, Strapi를 이용해 만들었습니다

문의사항은 admin@devchanki.com 으로 주시면 감사 하겠습니다.

Blog

LinkedIn GitHub

Kubeadm 을 이용한 kubernetes 버전 업그레이드

쿠버네티스 업그레이드(skew) 정책

쿠버네티스 클러스터는 여러가지 컴포넌트로 나눠져 있으며, 여러개로 나눠져 있기 때문에 업그레이드 시 여러 다른 버전이 공존하게 된다.

그렇기 때문에 버전차이 규칙이 있다.

컨트롤 플레인 노드

kube-apiserver
- 모든 요청을 받아 유효성 검사를 합니다. 해당 컴포넌트는 가장 높은 버전이여야 합니다.
kube-controller-manager, kube-scheduler
- Controller Manager: 상태를 유지(예: 파드 개수 유지)하는 컨트롤러들을 관리.
- Scheduler: 파드를 어느 노드에 배치할지 결정.

워커 노드 (Node) 컴포넌트

kubelet
- 각 노드에서 파드(컨테이너)의 실행을 관리하고 API 서버에 상태를 보고하는 에이전트.
- API 서버보다 최대 3단계 낮은 Minor 버전까지 허용 이유: 컨트롤 플레인을 먼저 업그레이드한 후, 수많은 노드들을 천천히 순차적으로 업그레이드할 수 있게 하기 위함 API 서버가 1.30이면, Kubelet은 1.30, 1.29, 1.28, 1.27까지 가능.
kube-proxy
- 노드의 네트워크 규칙을 관리(Service 통신 처리).
- API 서버보다 최대 3단계 낮은 Minor 버전까지 허용됩니다. (kubelet과 동일)

실습을 위한 환경 셋팅

bash

# 실습용 디렉터리 생성
mkdir k8s-upgrade
cd k8s-upgrade
 
# 파일 다운로드
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-upgrade/Vagrantfile
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-upgrade/init_cfg.sh
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-upgrade/k8s-ctr.sh
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-upgrade/k8s-w.sh
 
# 실습 환경 배포
vagrant up
vagrant status

현재 설치된 서비스는 다음과 같다.

항목	버전	버전 호환성
Rocky Linux	10.0-1.6	version (kernel 6.12 - k8s_kernel) - kubespary
containerd	v2.1.5	CRI Version(v1), k8s 1.32~1.35 지원 - Link
runc	v1.3.3	정보 조사 필요 https://github.com/opencontainers/runc
kubelet, kubeadm, kubectl	v1.32.11	k8s 버전 정책 문서 참고 - Docs
helm	v3.18.6	k8s 1.30.x ~ 1.33.x 지원 - Docs
flannel cni	v0.27.3	k8s 1.28~ 이후 - Release

최종 업그레이드 목표는 다음과 같다.

항목	버전	버전 호환성
Rocky Linux	10.1-1.4	version (kernel 6.12 - k8s_kernel) - kubespary
containerd	v.2.1.5	CRI Version(v1), k8s 1.32~1.35 지원 - Link
runc	v1.x.y	정보 조사 필요 https://github.com/opencontainers/runc
kubelet, kubeadm, kubectl	v1.34.x	k8s 버전 정책 문서 참고 - Docs
helm	v3.19.x	k8s 1.31.x ~ 1.34.x 지원 - Docs
flannel cni	v0.28.0	k8s 1.28~ 이후 - Release , Upgrade

이전에 나는 경험했듯이, 재부팅 시에 swap 메모리와 네트워크 문제가 좀 있었다. swap 메모리를 제거하는 코드를 각 노드별로 작성한다. 해당 실습의 경우 리눅스 커널을 업그레이드 하고 재부팅이 필요하므로 미리 작업한다.

나는 모든 노드에 아래 코드의 방식(2) 를 이용해 작업했다.

bash

kubens default
 
# swap 사용 확인
free -h
lsblk
├─sda2   8:2    0  3.8G  0 part # swap 사용이었으내, 스크립트 명령으로 swapoff 상태
 
# 이미 swap 삭제되어 있음
cat /etc/fstab
 
# 기동 중 컨테이너 확인
crictl ps
 
# 재기동
reboot
 
# 다시 접속
vagrant ssh k8s-w2
 
 
# 파드 기동 실패 반복
crictl ps
crictl ps -a
 
# 원인 확인
systemctl status kubelet.service --no-pager
journalctl -u kubelet.service -f
 
free -h
swapon --show
NAME      TYPE      SIZE USED PRIO
/dev/sda2 partition 3.8G   0B   -2
 
# swap 파티션 즉시 비활성화
swapoff /dev/sda2
swapon --show
 
# swap 파티션 완전 삭제 : (방식1) fdisk 사용
fdisk /dev/sda
--------------------
p        # 파티션 확인
d        # delete
2        # sda2 선택
p        # 다시 확인 (sda2 없어야 함)
w        # 저장 후 종료
--------------------
 
# swap 파티션 완전 삭제 : (방식2) sfdisk 사용
sfdisk -l
sfdisk --delete /dev/sda 2
lsblk
 
# 커널에 파티션 테이블 반영
lsblk               # 아직도 sda2 남아 있음
partprobe /dev/sda
 
# 확인
lsblk               # sda2 삭제 처리 확인
 
# reboot 해서 한번 더 확인
reboot
 
 
# 다시 접속
vagrant ssh k8s-w2
 
# 확인
cricrl ps
free -h
swapon --show
lsblk
 
# (옵션) sda2를 xfs type / 파티션에서 확장 사용 설정 해보기

reboot 시 외부통신이 안되었다. 아래에 명령어를 사용한다. 한번에 복사 붙여넣기 할 시, 적용이 잘 안되었다. 한줄씩 입력하면서 변경을 확인한 후 적용해야 했다.

bash

echo "[TASK 6] Delete default routing - enp0s9 NIC"
nmcli connection modify enp0s9 ipv4.never-default yes
nmcli connection up enp0s9
 
# 확인
cat /etc/NetworkManager/system-connections/enp0s9.nmconnection
# gateway=192.168.10.1 줄 제거
never-default=true # 추가

업그레이드 하면서, 모니터링을 위해 prometheus stack 을 설치한다.

bash

# repo 추가
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
 
# 파라미터 파일 생성
cat <<EOT > monitor-values.yaml
prometheus:
  prometheusSpec:
    scrapeInterval: "20s"
    evaluationInterval: "20s"
    externalLabels:
      cluster: "myk8s-cluster"
  service:
    type: NodePort
    nodePort: 30001
 
grafana:
  defaultDashboardsTimezone: Asia/Seoul
  adminPassword: prom-operator
  service:
    type: NodePort
    nodePort: 30002
 
alertmanager:
  enabled: true
defaultRules:
  create: true
 
kubeProxy:
  enabled: false
prometheus-windows-exporter:
  prometheus:
    monitor:
      enabled: false
EOT
cat monitor-values.yaml
 
# 배포
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 80.13.3 \
-f monitor-values.yaml --create-namespace --namespace monitoring
 
# 확인
helm list -n monitoring
kubectl get pod,svc,ingress,pvc -n monitoring
kubectl get prometheus,servicemonitors,alertmanagers -n monitoring
kubectl get crd | grep monitoring
 
# 각각 웹 접속 실행 : NodePort 접속
open http://192.168.10.100:30001 # prometheus
open http://192.168.10.100:30002 # grafana : 접속 계정 admin / prom-operator
 
# 프로메테우스 버전 확인
kubectl exec -it sts/prometheus-kube-prometheus-stack-prometheus -n monitoring -c prometheus -- prometheus --version
prometheus, version 3.9.1
 
# 그라파나 버전 확인
kubectl exec -it -n monitoring deploy/kube-prometheus-stack-grafana -- grafana --version
grafana version 12.3.1

dashboard import 를 해 준다. dashboards -> new -> import 로 들어가 15661 을 입력해 이미 만들어진 템플릿을 이용해 다운 받았다.

샘플 애플리케이션을 배포한다.

bash

# 샘플 애플리케이션 배포
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
EOF

테스트를 위해 curl 을 할 수 있는 pod 를 control plain 노드에 설치한다.

bash

# k8s-ctr 노드에 curl-pod 파드 배포
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  nodeName: k8s-ctr
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF

kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'

명령어로 반복 호출 해 본다.

여러 쉘을 띄워서 모니터링 해본다.

bash

# 터미널1
SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
while true; do curl -s $SVCIP | grep Hostname; sleep 1; done
혹은
kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'
 
# 터미널2
watch -d kubectl get node
 
# 터미널3
watch -d kubectl get pod -A -owide
 
# 터미널4
watch -d kubectl top node
 
# 터미널5
[k8s-w1] watch -d crictl ps
 
# 터미널6
[k8s-w2] watch -d crictl ps

etcd 백업을 진행해 본다

bash

# etcd 컨테이너 버전 확인
crictl images | grep etcd
registry.k8s.io/etcd                      3.5.24-0            1211402d28f58       21.9MB
 
kubectl exec -n kube-system etcd-k8s-ctr -- etcdctl version
etcdctl version: 3.5.24
API version: 3.5
 
# 버전 변수 지정
ETCD_VER=3.5.24
 
# cpu arch 변수 지정
ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then ARCH=arm64; fi
echo $ARCH
 
# etcd 바이너리 다운로드
curl -L https://github.com/etcd-io/etcd/releases/download/v${ETCD_VER}/etcd-v${ETCD_VER}-linux-${ARCH}.tar.gz -o /tmp/etcd-v${ETCD_VER}.tar.gz
ls /tmp
 
# 압축 해제
mkdir -p /tmp/etcd-download
tar xzvf /tmp/etcd-v${ETCD_VER}.tar.gz -C /tmp/etcd-download --strip-components=1
 
# 실행 파일 이동 (etcdctl, etcdutl)
mv /tmp/etcd-download/etcdctl /usr/local/bin/
mv /tmp/etcd-download/etcdutl /usr/local/bin/
chown root:root /usr/local/bin/etcdctl
chown root:root /usr/local/bin/etcdutl
 
# 설치 확인
etcdctl version
etcdctl version: 3.5.24
API version: 3.5
 
# 인증서 위치 등 설정 
kubectl describe pod -n kube-system etcd-k8s-ctr | grep isten-client-urls
      --listen-client-urls=https://127.0.0.1:2379,https://192.168.10.100:2379
      
# etcdctl 환경변수 설정
export ETCDCTL_API=3  # etcd v3 API를 사용
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379  # listen-client-urls
 
mkdir /backup
etcdctl snapshot save /backup/etcd-snapshot-$(date +%F).db
 
# 백업 파일 확인
tree /backup/
(⎈|HomeLab:N/A) root@k8s-ctr:~# tree /backup/
/backup/
└── etcd-snapshot-2026-01-24.db
 
1 directory, 1 file

다음과 같이 etcd 를 백업했다. 매우 빠른 속도로 백업되며 진행이 안된다 싶으면 환경변수 등 설정이 잘못된 것으로 다시 한번 시도하면 된다.

Flannel CNI 업그레이드 : v0.27.3 → v0.27.4

플라넬 설치 시간이 있으니, 미리 설치를 해둔다. 플라넬은 사실 금방 설치되었으나, 해당 방식으로 미리 받아놓는게 다운타임을 줄일 수 있을 것이다.

bash

# 
crictl images | grep flannel
ghcr.io/flannel-io/flannel-cni-plugin     v1.7.1-flannel1     127562bd9047f       5.14MB
ghcr.io/flannel-io/flannel                v0.27.3             d84558c0144bc       33.1MB
 
# cni-pligin 버전 정보도 확인 후 pull 해두자!
crictl pull ghcr.io/flannel-io/flannel:v0.27.4
crictl pull ghcr.io/flannel-io/flannel:v1.8.0-flannel1
crictl images | grep flannel
ghcr.io/flannel-io/flannel-cni-plugin     v1.7.1-flannel1     127562bd9047f       5.14MB
ghcr.io/flannel-io/flannel                v0.27.3             d84558c0144bc       33.1MB
ghcr.io/flannel-io/flannel                v0.27.4             24d577aa4188d       33.2MB
 
# [w1/w2]
crictl pull ghcr.io/flannel-io/flannel:v0.27.4
crictl pull ghcr.io/flannel-io/flannel:v1.8.0-flannel1
crictl images | grep flannel

helm 이용해 설치한다

bash

# 모니터링
## 터미널1 : 아래 호출 중단 발생 확인!
SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
while true; do curl -s $SVCIP | grep Hostname; sleep 1; done
혹은
kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'
 
## 터미널3 : kube-flannel 파드 재기동 후 정상 기동까지 시간 확인
watch -d kubectl get pod -n kube-flannel
 
 
# 작업 전 정보 확인
helm list -n kube-flannel
helm get values -n kube-flannel flannel
kubectl get pod -n kube-flannel -o yaml | grep -i image: | sort | uniq
kubectl get ds -n kube-flannel -owide
NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE    CONTAINERS     IMAGES                               SELECTOR
kube-flannel-ds   3         3         3       3            3           <none>          126m   kube-flannel   ghcr.io/flannel-io/flannel:v0.27.3   app=flannel
 
 
# 신규 버전 명시 values 파일 작성
cat << EOF > flannel.yaml
podCidr: "10.244.0.0/16"
flannel:
  cniBinDir: "/opt/cni/bin"
  cniConfDir: "/etc/cni/net.d"
  args:
  - "--ip-masq"
  - "--kube-subnet-mgr"
  - "--iface=enp0s9"  
  backend: "vxlan"
image:
  tag: v0.27.4
EOF
 
# helm 업그레이드 수행 : Flannel은 DaemonSet 이므로 노드별 순차 업데이트
helm upgrade flannel flannel/flannel -n kube-flannel -f flannel.yaml --version 0.27.4
kubectl -n kube-flannel rollout status ds/kube-flannel-ds
 
# 확인
helm list -n kube-flannel
helm get values -n kube-flannel flannel
 
# 확인
crictl ps
kubectl get ds -n kube-flannel -owide
NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE    CONTAINERS     IMAGES                               SELECTOR
kube-flannel-ds   3         3         3       3            3           <none>          127m   kube-flannel   ghcr.io/flannel-io/flannel:v0.27.3   app=flannel
 
kubectl get pod -n kube-flannel -o yaml | grep -i image: | sort | uniq
      image: ghcr.io/flannel-io/flannel-cni-plugin:v1.8.0-flannel1
      image: ghcr.io/flannel-io/flannel:v0.27.4

다음과 같이 업그레이드 된 것을 확인할 수 있다.

rocky os 업그레이드

리눅스 업그레이드 시 reboot 해야 할 수 있으므로 문제가 될만한 사항들을 파악해 본다.

bash

kubectl get pod -A -owide |grep k8s-ctr
 
k scale deployment -n kube-system coredns --replicas 1
kubectl get pod -A -owide |grep k8s-ctr
kubectl scale deployment -n kube-system coredns --replicas 2
kubectl get pod -n kube-system -owide | grep coredns

coredns 의 경우 현재 업그레이드 할 컨트롤 플레인 노드에 있는 것으로 확인하였다. 해당 노드가 reboot 될 수 있으므로 replica 2 로 다른 노드에도 pod 이 기동될 수 있게 설정한다.

리눅스 업그레이드를 진행한다. 중간에 containerd 버전을 고정하는 게 있는데, 가시다님께서 실습 진행을 하면서 리눅스 버전을 업그레이드 하면 containerd 가 같이 업그레이드 되어 해당 버전을 고정시키는 부분이 들어 있다.

bash

# 모니터링
## 터미널1 : 아래 호출 중단 발생 확인!
SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
[k8s-w1] SVCIP=<IP 직접 입력>
[k8s-w1] while true; do curl -s $SVCIP | grep Hostname; sleep 1; done # 이유는 k8s-ctr reboot 시에도 app 워크로드 통신 확인을 위함
혹은
kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'
 
 
# Rocky Linux 버전 확인
rpm -aq | grep release # rocky-release-10.0-1.6.el10.noarch
uname -r # 6.12.0-55.39.1.el10_0.aarch64
 
-------------------------------------------------------------
# (옵션) containerd.io 버전 업그레이드 되지 않게, 현재 설치 버전 고정 설정
rpm -q containerd.io  # 현재 설치 버전 정보 확인
containerd.io-2.1.5-1.el10.aarch64
 
## 버전 잠금 플러그인 설치
dnf install -y 'dnf-command(versionlock)'
 
## containerd.io 버전 잠금
dnf versionlock add containerd.io
Adding versionlock on: containerd.io-0:2.1.5-1.el10.*
 
## 버전 잠금 목록 확인
dnf versionlock list
containerd.io-0:2.1.5-1.el10.*
-------------------------------------------------------------
 
# update 수행 : 해당 과정에서 containerd 가 업그레이드 된것으로 보임. 정확한 버전 관리가 필요 시, containerd/runc 예외 설정 할 것
dnf -y update # 6.12.0-124.21.1.el10_1.aarch64
## 커널 업데이트 완료 : 새로운 리눅스 커널 버전인 6.12.0-124.27.1.el10_1.aarch64이 시스템에 성공적으로 설치됨.
## 부트로더 갱신 : 다음 재부팅 시 시스템은 가장 최신 버전인 6.12.0-124 커널로 부팅됨.
  Running scriptlet: kernel-modules-core-6.12.0-124.27.1.el10_1.aarch64                                           584/584 
  Running scriptlet: kernel-core-6.12.0-124.27.1.el10_1.aarch64                                                   584/584 
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done
kdump: For kernel=/boot/vmlinuz-6.12.0-124.27.1.el10_1.aarch64, crashkernel=2G-4G:256M,4G-64G:320M,64G-:576M now. 
Please reboot the system for the change to take effect.
Note if you don't want kdump-utils to manage the crashkernel kernel parameter, please set auto_reset_crashkernel=no in /etc/kdump.conf.
 
  Running scriptlet: kernel-modules-6.12.0-124.27.1.el10_1.aarch64                                                584/584 
  Running scriptlet: kdump-utils-1.0.54-7.el10.aarch64                                                            584/584 
kdump: For kernel=/boot/vmlinuz-6.12.0-55.39.1.el10_0.aarch64, crashkernel=2G-4G:256M,4G-64G:320M,64G-:576M now. 
Please reboot the system for the change to take effect. 
Note if you don't want kdump-utils to manage the crashkernel kernel parameter, please set auto_reset_crashkernel=no in /etc/kdump.conf.
 
 
# 재기동 : 단일 Controlplane node 환경이므로, reboot 후 부팅 정상화 완료 기간에는 k8s api 통신 불가!
reboot
ping 192.168.10.100
64 bytes from 192.168.10.100: icmp_seq=47 ttl=64 time=0.452 ms
64 bytes from 192.168.10.100: icmp_seq=48 ttl=64 time=0.406 ms
Request timeout for icmp_seq 49                                 # 재부팅 시점에 ping 통신 불가
Request timeout for icmp_seq 50
Request timeout for icmp_seq 51
Request timeout for icmp_seq 52
Request timeout for icmp_seq 53
Request timeout for icmp_seq 54
Request timeout for icmp_seq 55
Request timeout for icmp_seq 56
Request timeout for icmp_seq 57
Request timeout for icmp_seq 58
Request timeout for icmp_seq 59
Request timeout for icmp_seq 60
Request timeout for icmp_seq 61
Request timeout for icmp_seq 62
Request timeout for icmp_seq 63
Request timeout for icmp_seq 64
Request timeout for icmp_seq 65
Request timeout for icmp_seq 66                   
64 bytes from 192.168.10.100: icmp_seq=67 ttl=64 time=0.364 ms  # 부팅 과정에서 nic 처리 정상 시점~
64 bytes from 192.168.10.100: icmp_seq=68 ttl=64 time=0.291 ms
...
 
# vagrant ssh k8s-ctr
rpm -aq | grep release # rocky-release-10.1-1.4.el10.noarch
uname -r               # 6.12.0-124.27.1.el10_1.aarch64
 
# 파드 기동 확인
kubectl get pod -A -owide |grep k8s-ctr
default        curl-pod                                                    1/1     Running   1 (2m25s ago)   61m    10.244.0.3       k8s-ctr   <none>           <none>
kube-flannel   kube-flannel-ds-f2572                                       1/1     Running   1 (2m25s ago)   15m    192.168.10.100   k8s-ctr   <none>           <none>
kube-system    coredns-668d6bf9bc-ctkmb                                    1/1     Running   1 (2m25s ago)   142m   10.244.0.2       k8s-ctr   <none>           <none>
kube-system    etcd-k8s-ctr                                                1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
kube-system    kube-apiserver-k8s-ctr                                      1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
kube-system    kube-controller-manager-k8s-ctr                             1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
kube-system    kube-proxy-wwd9c                                            1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
kube-system    kube-scheduler-k8s-ctr                                      1/1     Running   1 (2m25s ago)   142m   192.168.10.100   k8s-ctr   <none>           <none>
monitoring     kube-prometheus-stack-prometheus-node-exporter-6k5rk        1/1     Running   1 (2m25s ago)   36m    192.168.10.100   k8s-ctr   <none>           <none>

아래 사진처럼 수집이 일부 중단된 것을 확인할 수 있다.

[k8s-ctr] Kubeadm , Kubelet, Kubectl Upgrade : v1.32.11 → v1.33.11

bash

# repo 수정 : 기본 1.32 -> 1.33
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
dnf makecache
 
# 전체 설치 가능 버전 확인 : --disableexcludes=... kubernetes repo에 설정된 exclude 규칙을 이번 설치에서만 무시(1회성 옵션 처럼 사용)
dnf list --showduplicates kubeadm --disableexcludes=kubernetes
Installed Packages
kubeadm.aarch64                                       1.32.11-150500.1.1                                       @kubernetes
Available Packages
..
kubeadm.aarch64                                       1.33.7-150500.1.1                                        kubernetes 
kubeadm.ppc64le                                       1.33.7-150500.1.1                                        kubernetes 
kubeadm.s390x                                         1.33.7-150500.1.1                                        kubernetes 
kubeadm.src                                           1.33.7-150500.1.1                                        kubernetes 
kubeadm.x86_64                                        1.33.7-150500.1.1                                        kubernetes 
 
# 설치
dnf install -y --disableexcludes=kubernetes kubeadm-1.33.7-150500.1.1
Upgrading:
 kubeadm                  aarch64                  1.33.7-150500.1.1                    kubernetes                   11 M
 
# 설치 파일들 확인
which kubeadm && kubeadm version -o yaml
 
# 업그레이드 계획 확인 : 현재 버전 , 대상 버전 , 컴포넌트 변경 사항
kubeadm upgrade plan
[upgrade/versions] Target version: v1.33.7
[upgrade/versions] Latest version in the v1.32 series: v1.32.11
 
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT    TARGET
kubelet     k8s-ctr   v1.32.11   v1.33.7
kubelet     k8s-w1    v1.32.11   v1.33.7
kubelet     k8s-w2    v1.32.11   v1.33.7
 
Upgrade to the latest stable version:
 
COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            k8s-ctr   v1.32.11   v1.33.7
kube-controller-manager   k8s-ctr   v1.32.11   v1.33.7
kube-scheduler            k8s-ctr   v1.32.11   v1.33.7
kube-proxy                          1.32.11    v1.33.7   # kube-proxy 데몬셋으로 모든 노드에 영향
CoreDNS                             v1.11.3    v1.12.0   # coredns 도메인 처리로 영향
etcd                      k8s-ctr   3.5.24-0   3.5.24-0  # 기본 버전 사용으로 영향 없음
 
You can now apply the upgrade by executing the following command:
 
        kubeadm upgrade apply v1.33.7
 
_____________________________________________________________________
 
 
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
 
API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

kubeadm 버전을 올리고, 그다음에 업그레이드 계획을 확인한다.

업그레이드를 진행한다

bash

# 모니터링
## 터미널1 : 아래 호출 중단 발생 확인!
SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
[k8s-w1] SVCIP=<IP 직접 입력>
[k8s-w1] while true; do curl -s $SVCIP | grep Hostname; sleep 1; done
혹은
kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'
 
# 터미널2
watch -d kubectl get node
 
## 터미널3
watch -d kubectl get pod -n kube-system
 
## 터미널4
watch -d etcdctl member list -w table
 
# 컨테이너 이미지 미리 다운로드 : 특히 업그레이드 작업 시, 작업 시간 단축을 위해서 수행할 것
kubeadm config images pull
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.33.7
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.33.7
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.33.7
[config/images] Pulled registry.k8s.io/kube-proxy:v1.33.7
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.12.0
[config/images] Pulled registry.k8s.io/pause:3.10
[config/images] Pulled registry.k8s.io/etcd:3.5.24-0
 
crictl images
...
 
# [k8s-w1/w2] kube-proxy 데몬셋과 coredns 신규 버전은 all node 에 미리 다운로드 해두자!
crictl pull registry.k8s.io/kube-proxy:v1.33.7
crictl pull registry.k8s.io/coredns/coredns:v1.12.0
 
 
# 실행 후 y 입력
kubeadm upgrade apply v1.33.7
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.33.7"
[upgrade/versions] Cluster version: v1.32.11
[upgrade/versions] kubeadm version: v1.33.7
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.7".
[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.

업그레이드 된 것들을 확인해 본다

bash

# control component(apiserver, kcm 등)의 컨테이너 이미지는 1.33.7 업그레이드 되었고,
# kubelet 은 아직 업그레이드 되지 않은 상태. node 출력에 버전은 kubelet 버전을 기준 출력으로 보임.
kubectl get node -owide
NAME      STATUS   ROLES           AGE   VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                   CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   68m   v1.32.11   192.168.10.100   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-124.27.1.el10_1.aarch64   containerd://2.1.5
k8s-w1    Ready    <none>          63m   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
k8s-w2    Ready    <none>          62m   v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
 
kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.32.11
 
kubectl get pod -A
...
 
# 이미지 업데이트 확인 : etcd, pause 기존 버전만 local 존재 확인.
crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
registry.k8s.io/coredns/coredns           v1.11.3             2f6c962e7b831       16.9MB
registry.k8s.io/coredns/coredns           v1.12.0             f72407be9e08c       19.1MB
registry.k8s.io/etcd                      3.5.24-0            1211402d28f58       21.9MB
registry.k8s.io/kube-apiserver            v1.32.11            58951ea1a0b5d       26.4MB
registry.k8s.io/kube-apiserver            v1.33.7             6d7bc8e445519       27.4MB
registry.k8s.io/kube-controller-manager   v1.32.11            82766e5f2d560       24.2MB
registry.k8s.io/kube-controller-manager   v1.33.7             a94595d0240bc       25.1MB
registry.k8s.io/kube-proxy                v1.32.11            dcdb790dc2bfe       27.6MB
registry.k8s.io/kube-proxy                v1.33.7             78ccb937011a5       28.3MB
registry.k8s.io/kube-scheduler            v1.32.11            cfa17ff3d6634       19.2MB
registry.k8s.io/kube-scheduler            v1.33.7             94005b6be50f0       19.9MB
registry.k8s.io/pause                     3.10                afb61768ce381       268kB
 
# static pod yaml 파일 내용 업데이트 확인 : yaml 파일 내에 images 부분 업데이트
ls -l /etc/kubernetes/manifests/
cat /etc/kubernetes/manifests/*.yaml | grep -i image:
    image: registry.k8s.io/etcd:3.5.24-0
    image: registry.k8s.io/kube-apiserver:v1.33.7
    image: registry.k8s.io/kube-controller-manager:v1.33.7
    image: registry.k8s.io/kube-scheduler:v1.33.7
 
# kube-system 네임스페이스 내에 동작 중인 파드 내에 컨테이너 이미지 정보 출력
kubectl get pods -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}  - {.name}: {.image}{"\n"}{end}{"\n"}{end}'
coredns-674b8bbfcf-8qhd9
  - coredns: registry.k8s.io/coredns/coredns:v1.12.0
 
etcd-k8s-ctr
  - etcd: registry.k8s.io/etcd:3.5.24-0  # 기존 버전 동일(유지)
 
kube-apiserver-k8s-ctr
  - kube-apiserver: registry.k8s.io/kube-apiserver:v1.33.7
 
kube-controller-manager-k8s-ctr
  - kube-controller-manager: registry.k8s.io/kube-controller-manager:v1.33.7
 
kube-proxy-2c6d6
  - kube-proxy: registry.k8s.io/kube-proxy:v1.33.7
 
kube-scheduler-k8s-ctr
  - kube-scheduler: registry.k8s.io/kube-scheduler:v1.33.7

kubelet/kubectl 업그레이드를 진행한다.

bash

# 전체 설치 가능 버전 확인 
dnf list --showduplicates kubelet --disableexcludes=kubernetes
dnf list --showduplicates kubectl --disableexcludes=kubernetes
 
# Upgrade 설치
dnf install -y --disableexcludes=kubernetes kubelet-1.33.7-150500.1.1 kubectl-1.33.7-150500.1.1
Upgrading:
 kubectl                  aarch64                  1.33.7-150500.1.1                    kubernetes                  9.7 M
 kubelet                  aarch64                  1.33.7-150500.1.1                    kubernetes                   13 M
 
# 설치 파일들 확인
which kubectl && kubectl version --client=true
which kubelet && kubelet --version
 
# 재시작
systemctl daemon-reload
systemctl restart kubelet # k8s api 10초 정도 단절 
 
# 확인
kubectl get nodes -o wide
NAME      STATUS   ROLES           AGE   VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                   CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   75m   v1.33.7    192.168.10.100   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-124.27.1.el10_1.aarch64   containerd://2.1.5
k8s-w1    Ready    <none>          70m   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
k8s-w2    Ready    <none>          69m   v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
 
kc describe node k8s-ctr
kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.33.7

재시작 하면서, 약간 단절이 발생한다.

프로메테우스에서는 버전이 다르다는 alert 가 나와 있는 것을 알 수 있다.

[k8s-ctr] Kubeadm , Kubelet, Kubectl Upgrade : v1.33.11 → v1.34.3

한단계 더 업그레이드 한다.

bash

# repo 수정 : 기본 1.33 -> 1.34
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
dnf makecache
 
# 전체 설치 가능 버전 확인 : --disableexcludes=... kubernetes repo에 설정된 exclude 규칙을 이번 설치에서만 무시(1회성 옵션 처럼 사용)
dnf list --showduplicates kubeadm --disableexcludes=kubernetes
 
# 설치
dnf install -y --disableexcludes=kubernetes kubeadm-1.34.3-150500.1.1
 
# 설치 파일들 확인
which kubeadm && kubeadm version -o yaml
 
# 업그레이드 계획 확인 : 현재 버전 , 대상 버전 , 컴포넌트 변경 사항
kubeadm upgrade plan
[upgrade/versions] Target version: v1.34.3
[upgrade/versions] Latest version in the v1.33 series: v1.33.7
 
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT    TARGET
kubelet     k8s-w1    v1.32.11   v1.34.3
kubelet     k8s-w2    v1.32.11   v1.34.3
kubelet     k8s-ctr   v1.33.7    v1.34.3
 
Upgrade to the latest stable version:
 
COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            k8s-ctr   v1.33.7    v1.34.3
kube-controller-manager   k8s-ctr   v1.33.7    v1.34.3
kube-scheduler            k8s-ctr   v1.33.7    v1.34.3
kube-proxy                          1.33.7     v1.34.3
CoreDNS                             v1.12.0    v1.12.1
etcd                      k8s-ctr   3.5.24-0   3.6.5-0  # 1.32->1.33 과 다르게, 1.33->1.34시 etcd 업그레이드 필요
 
You can now apply the upgrade by executing the following command:
 
        kubeadm upgrade apply v1.34.3
 
_____________________________________________________________________
 
 
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
 
API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

아까와 같이, kubeadm 을 이용해 1단계 더 버전을 올린다.

bash

# 모니터링
## 터미널1 : 아래 호출 중단 발생 확인!
SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
[k8s-w1] SVCIP=<IP 직접 입력>
[k8s-w1] while true; do curl -s $SVCIP | grep Hostname; sleep 1; done
혹은
kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'
 
# 터미널2
watch -d kubectl get node
 
## 터미널3
watch -d kubectl get pod -n kube-system
 
## 터미널4
watch -d etcdctl member list -w table
 
 
# 컨테이너 이미지 미리 다운로드 : 특히 업그레이드 작업 시, 작업 시간 단축을 위해서 수행할 것
kubeadm config images pull
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.34.3
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.34.3
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.34.3
[config/images] Pulled registry.k8s.io/kube-proxy:v1.34.3
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.12.1
[config/images] Pulled registry.k8s.io/pause:3.10.1
[config/images] Pulled registry.k8s.io/etcd:3.6.5-0
 
crictl images
...
 
# [k8s-w1/w2] kube-proxy 데몬셋과 coredns 신규 버전, pause 은 all node 에 미리 다운로드 해두자!
crictl pull registry.k8s.io/kube-proxy:v1.34.3
crictl pull registry.k8s.io/coredns/coredns:v1.12.1
crictl pull registry.k8s.io/pause:3.10.1
crictl images
 
 
# 실행 : etcd 신규 버전 기동으로 이전 1.32 → 1.33 때보다 약간 소요 시간 추가
kubeadm upgrade apply v1.34.3 --yes
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.34.3"
[upgrade/versions] Cluster version: v1.33.7
[upgrade/versions] kubeadm version: v1.34.3
[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection
[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.34.3" (timeout: 5m0s)...
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests4082174448"
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-18-17-06-48/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-18-17-06-48/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-18-17-06-48/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-18-17-06-48/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/control-plane] The control plane instance for this node was successfully upgraded!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded!
W0118 17:09:20.581163   36197 postupgrade.go:116] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config2787812587 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config2787812587/config.yaml
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/bootstrap-token] Configuring bootstrap token and cluster-info RBAC rules
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
 
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.34.3".
 
[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.

쿠버네티스 정보를 확인해 본다.

bash

# control component(apiserver, kcm 등)의 컨테이너 이미지는 1.34.3 업그레이드 되었고,
# kubelet 은 아직 업그레이드 되지 않은 상태. node 출력에 버전은 kubelet 버전을 기준 출력으로 보임.
kubectl get node -owide
NAME      STATUS   ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                   CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   125m   v1.33.7    192.168.10.100   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-124.27.1.el10_1.aarch64   containerd://2.1.5
k8s-w1    Ready    <none>          120m   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
k8s-w2    Ready    <none>          119m   v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
 
kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.33.7
 
kubectl get pod -A
...
 
# 이미지 업데이트 확인 : etcd, pause 신규 버전 pull 확인.
crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
registry.k8s.io/coredns/coredns           v1.11.3             2f6c962e7b831       16.9MB
registry.k8s.io/coredns/coredns           v1.12.0             f72407be9e08c       19.1MB
registry.k8s.io/coredns/coredns           v1.12.1             138784d87c9c5       20.4MB
registry.k8s.io/etcd                      3.5.24-0            1211402d28f58       21.9MB
registry.k8s.io/etcd                      3.6.5-0             2c5f0dedd21c2       21.1MB
registry.k8s.io/kube-apiserver            v1.32.11            58951ea1a0b5d       26.4MB
registry.k8s.io/kube-apiserver            v1.33.7             6d7bc8e445519       27.4MB
registry.k8s.io/kube-apiserver            v1.34.3             cf65ae6c8f700       24.6MB
registry.k8s.io/kube-controller-manager   v1.32.11            82766e5f2d560       24.2MB
registry.k8s.io/kube-controller-manager   v1.33.7             a94595d0240bc       25.1MB
registry.k8s.io/kube-controller-manager   v1.34.3             7ada8ff13e54b       20.7MB
registry.k8s.io/kube-proxy                v1.32.11            dcdb790dc2bfe       27.6MB
registry.k8s.io/kube-proxy                v1.33.7             78ccb937011a5       28.3MB
registry.k8s.io/kube-proxy                v1.34.3             4461daf6b6af8       22.8MB
registry.k8s.io/kube-scheduler            v1.32.11            cfa17ff3d6634       19.2MB
registry.k8s.io/kube-scheduler            v1.33.7             94005b6be50f0       19.9MB
registry.k8s.io/kube-scheduler            v1.34.3             2f2aa21d34d2d       15.8MB
registry.k8s.io/pause                     3.10                afb61768ce381       268kB
registry.k8s.io/pause                     3.10.1              d7b100cd9a77b       268kB
 
 
# static pod yaml 파일 내용 업데이트 확인 : yaml 파일 내에 images 부분 업데이트
ls -l /etc/kubernetes/manifests/
cat /etc/kubernetes/manifests/*.yaml | grep -i image:
    image: registry.k8s.io/etcd:3.6.5-0
    image: registry.k8s.io/kube-apiserver:v1.34.3
    image: registry.k8s.io/kube-controller-manager:v1.34.3
    image: registry.k8s.io/kube-scheduler:v1.34.3
 
kubectl get pods -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}  - {.name}: {.image}{"\n"}{end}{"\n"}{end}'
coredns-66bc5c9577-v99kw
  - coredns: registry.k8s.io/coredns/coredns:v1.12.1
 
etcd-k8s-ctr
  - etcd: registry.k8s.io/etcd:3.6.5-0
 
kube-apiserver-k8s-ctr
  - kube-apiserver: registry.k8s.io/kube-apiserver:v1.34.3
 
kube-controller-manager-k8s-ctr
  - kube-controller-manager: registry.k8s.io/kube-controller-manager:v1.34.3
 
kube-proxy-9p996
  - kube-proxy: registry.k8s.io/kube-proxy:v1.34.3
 
kube-scheduler-k8s-ctr
  - kube-scheduler: registry.k8s.io/kube-scheduler:v1.34.3

kubelet, kubectl 을 업그레이드 한다.

bash

# control component(apiserver, kcm 등)의 컨테이너 이미지는 1.34.3 업그레이드 되었고,
# kubelet 은 아직 업그레이드 되지 않은 상태. node 출력에 버전은 kubelet 버전을 기준 출력으로 보임.
kubectl get node -owide
NAME      STATUS   ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                   CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   125m   v1.33.7    192.168.10.100   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-124.27.1.el10_1.aarch64   containerd://2.1.5
k8s-w1    Ready    <none>          120m   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
k8s-w2    Ready    <none>          119m   v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
 
kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.33.7
 
kubectl get pod -A
...
 
# 이미지 업데이트 확인 : etcd, pause 신규 버전 pull 확인.
crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
registry.k8s.io/coredns/coredns           v1.11.3             2f6c962e7b831       16.9MB
registry.k8s.io/coredns/coredns           v1.12.0             f72407be9e08c       19.1MB
registry.k8s.io/coredns/coredns           v1.12.1             138784d87c9c5       20.4MB
registry.k8s.io/etcd                      3.5.24-0            1211402d28f58       21.9MB
registry.k8s.io/etcd                      3.6.5-0             2c5f0dedd21c2       21.1MB
registry.k8s.io/kube-apiserver            v1.32.11            58951ea1a0b5d       26.4MB
registry.k8s.io/kube-apiserver            v1.33.7             6d7bc8e445519       27.4MB
registry.k8s.io/kube-apiserver            v1.34.3             cf65ae6c8f700       24.6MB
registry.k8s.io/kube-controller-manager   v1.32.11            82766e5f2d560       24.2MB
registry.k8s.io/kube-controller-manager   v1.33.7             a94595d0240bc       25.1MB
registry.k8s.io/kube-controller-manager   v1.34.3             7ada8ff13e54b       20.7MB
registry.k8s.io/kube-proxy                v1.32.11            dcdb790dc2bfe       27.6MB
registry.k8s.io/kube-proxy                v1.33.7             78ccb937011a5       28.3MB
registry.k8s.io/kube-proxy                v1.34.3             4461daf6b6af8       22.8MB
registry.k8s.io/kube-scheduler            v1.32.11            cfa17ff3d6634       19.2MB
registry.k8s.io/kube-scheduler            v1.33.7             94005b6be50f0       19.9MB
registry.k8s.io/kube-scheduler            v1.34.3             2f2aa21d34d2d       15.8MB
registry.k8s.io/pause                     3.10                afb61768ce381       268kB
registry.k8s.io/pause                     3.10.1              d7b100cd9a77b       268kB
 
 
# static pod yaml 파일 내용 업데이트 확인 : yaml 파일 내에 images 부분 업데이트
ls -l /etc/kubernetes/manifests/
cat /etc/kubernetes/manifests/*.yaml | grep -i image:
    image: registry.k8s.io/etcd:3.6.5-0
    image: registry.k8s.io/kube-apiserver:v1.34.3
    image: registry.k8s.io/kube-controller-manager:v1.34.3
    image: registry.k8s.io/kube-scheduler:v1.34.3
 
kubectl get pods -n kube-system -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.containers[*]}  - {.name}: {.image}{"\n"}{end}{"\n"}{end}'
coredns-66bc5c9577-v99kw
  - coredns: registry.k8s.io/coredns/coredns:v1.12.1
 
etcd-k8s-ctr
  - etcd: registry.k8s.io/etcd:3.6.5-0
 
kube-apiserver-k8s-ctr
  - kube-apiserver: registry.k8s.io/kube-apiserver:v1.34.3
 
kube-controller-manager-k8s-ctr
  - kube-controller-manager: registry.k8s.io/kube-controller-manager:v1.34.3
 
kube-proxy-9p996
  - kube-proxy: registry.k8s.io/kube-proxy:v1.34.3
 
kube-scheduler-k8s-ctr
  - kube-scheduler: registry.k8s.io/kube-scheduler:v1.34.3

kubelet, kubectl 설치

bash

# 전체 설치 가능 버전 확인 
dnf list --showduplicates kubelet --disableexcludes=kubernetes
dnf list --showduplicates kubectl --disableexcludes=kubernetes
 
# Upgrade 설치
dnf install -y --disableexcludes=kubernetes kubelet-1.34.3-150500.1.1 kubectl-1.34.3-150500.1.1
Upgrading:
 kubectl                  aarch64                  1.33.7-150500.1.1                    kubernetes                  9.7 M
 kubelet                  aarch64                  1.33.7-150500.1.1                    kubernetes                   13 M
 
# 설치 파일들 확인
which kubectl && kubectl version --client=true
which kubelet && kubelet --version
 
# 재시작
systemctl daemon-reload   # systemctl daemon-reexec 와 차이점은?
systemctl restart kubelet # k8s api 10초 정도 단절  
 
# 확인
kubectl get nodes -o wide
NAME      STATUS   ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                   CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   128m   v1.34.3    192.168.10.100   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-124.27.1.el10_1.aarch64   containerd://2.1.5
k8s-w1    Ready    <none>          123m   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
k8s-w2    Ready    <none>          122m   v1.32.11   192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
 
kc describe node k8s-ctr
kubectl describe node k8s-ctr | grep 'Kubelet Version:'
  Kubelet Version:            v1.34.3

kubeconfig 업그레이드 해준다

bash

#
ls -l ~/.kube/config
-rw-------. 1 root root 5618 Jan 18 13:58 /root/.kube/config
 
#
ls -l /etc/kubernetes/admin.conf 
-rw-------. 1 root root 5642 Jan 18 17:09 /etc/kubernetes/admin.conf
 
# admin 용 kubeconfig 업데이트
yes | cp  /etc/kubernetes/admin.conf ~/.kube/config ; echo
chown $(id -u):$(id -g) ~/.kube/config
kubectl config rename-context "kubernetes-admin@kubernetes" "HomeLab"
kubens default

워커노드 업그레이드

워커노드에는 Application 들이 있으므로 여러 고려사항들을 선제적으로 파악하고 작업한다.

worker node 작업 : v1.32.11 → v1.33.11 ⇒ v1.34.3

사전 정보를 확인하고 pdb 를 걸어 테스트를 진행한다.

pdb 는 처음 들어 봤는데, pdb 라는 기능은 간단하게 보면 노드에 적절히 분산되게 끔 하기 위한 기능이다.

bash

# 워커 1번에 '프로메테우스, 그라파나' 파드 기동 확인 -> 8번에서 워커 노드 작업 시 워커 2번 부터 하자.
kubectl get pod -A -owide | grep k8s-w1
default        webpod-697b545f57-kh44p                                     1/1     Running   1 (30m ago)    143m    10.244.1.5       k8s-w1    <none>           <none>
kube-flannel   kube-flannel-ds-4f2s2                                       1/1     Running   1 (30m ago)    96m     192.168.10.101   k8s-w1    <none>           <none>
kube-system    kube-proxy-spq7f                                            1/1     Running   1 (30m ago)    32m     192.168.10.101   k8s-w1    <none>           <none>
kube-system    metrics-server-5dd7b49d79-fvx7p                             1/1     Running   1 (30m ago)    3h43m   10.244.1.4       k8s-w1    <none>           <none>
monitoring     kube-prometheus-stack-grafana-5cb7c586f9-jfnhp              3/3     Running   3 (30m ago)    117m    10.244.1.2       k8s-w1    <none>           <none>
monitoring     kube-prometheus-stack-prometheus-node-exporter-hxmk7        1/1     Running   1 (30m ago)    117m    192.168.10.101   k8s-w1    <none>           <none>
monitoring     prometheus-kube-prometheus-stack-prometheus-0               2/2     Running   2 (30m ago)    116m    10.244.1.3       k8s-w1    <none>           <none>
 
kubectl get pod -A -owide | grep k8s-w2
default        webpod-697b545f57-cmwsk                                     1/1     Running   1 (28m ago)    143m    10.244.2.6       k8s-w2    <none>           <none>
kube-flannel   kube-flannel-ds-88r76                                       1/1     Running   1 (28m ago)    96m     192.168.10.102   k8s-w2    <none>           <none>
kube-system    coredns-66bc5c9577-v99kw                                    1/1     Running   1 (28m ago)    32m     10.244.2.4       k8s-w2    <none>           <none>
kube-system    kube-proxy-9p996                                            1/1     Running   1 (28m ago)    32m     192.168.10.102   k8s-w2    <none>           <none>
monitoring     alertmanager-kube-prometheus-stack-alertmanager-0           2/2     Running   2 (28m ago)    117m    10.244.2.3       k8s-w2    <none>           <none>
monitoring     kube-prometheus-stack-kube-state-metrics-7846957b5b-hz9vf   1/1     Running   18 (28m ago)   117m    10.244.2.5       k8s-w2    <none>           <none>
monitoring     kube-prometheus-stack-operator-584f446c98-p5nms             1/1     Running   1 (28m ago)    117m    10.244.2.2       k8s-w2    <none>           <none>
monitoring     kube-prometheus-stack-prometheus-node-exporter-gkcfw        1/1     Running   1 (28m ago)    117m    192.168.10.102   k8s-w2    <none>           <none>
 
# sts 확인
kubectl get sts -A
NAMESPACE    NAME                                              READY   AGE
monitoring   alertmanager-kube-prometheus-stack-alertmanager   1/1     118m
monitoring   prometheus-kube-prometheus-stack-prometheus       1/1     118m
 
# pv,pvc 확인
kubectl get pv,pvc -A
No resources found
 
# webpod deployment 에 pdb 설정 : 해당 정책은 항상 최소 2개의 Pod가 Ready 상태여야 함 , drain / eviction 시 단 하나의 Pod도 축출 불가
cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: webpod
  namespace: default
spec:
  maxUnavailable: 0
  selector:
    matchLabels:
      app: webpod
EOF
 
# 확인
kubectl get pdb
NAME     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
webpod   N/A             0                 0                     6s

가시다님께서, pdb 로 예시를 만들었다. pdb 가 있으면 evicted 가 되지 않는 예시를 확인 해 볼 수 있다.

bash

# 모니터링
## 터미널1 : 아래 drain 시 호출 상태 확인
SVCIP=$(kubectl get svc webpod -o jsonpath='{.spec.clusterIP}')
while true; do curl -s $SVCIP | grep Hostname; sleep 1; done
혹은
kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'
 
## 터미널2
watch -d kubectl get node
 
## 터미널3
watch -d kubectl get pod
 
 
# 워커 1번에 '프로메테우스, 그라파나' 파드 기동 확인 -> 워커 노드 작업 시 워커 2번 부터 진행
 
# 작업 대상 워커 노드 1대 drain : cordened 만 됨.
kubectl drain k8s-w2
node/k8s-w2 cordoned
error: unable to drain node "k8s-w2" due to error: [cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-88r76, kube-system/kube-proxy-9p996, monitoring/kube-prometheus-stack-prometheus-node-exporter-gkcfw, cannot delete Pods with local storage (use --delete-emptydir-data to override): monitoring/alertmanager-kube-prometheus-stack-alertmanager-0], continuing command...
There are pending nodes to be drained:
 k8s-w2
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-flannel/kube-flannel-ds-88r76, kube-system/kube-proxy-9p996, monitoring/kube-prometheus-stack-prometheus-node-exporter-gkcfw
cannot delete Pods with local storage (use --delete-emptydir-data to override): monitoring/alertmanager-kube-prometheus-stack-alertmanager-0
 
# 노드 및 파드 상태 확인
kubectl get node
NAME      STATUS                     ROLES           AGE     VERSION
k8s-ctr   Ready                      control-plane   3h51m   v1.34.3
k8s-w1    Ready                      <none>          3h50m   v1.32.11
k8s-w2    Ready,SchedulingDisabled   <none>          3h49m   v1.32.11
 
kubectl get pod -A -owide |grep k8s-w2
default        webpod-697b545f57-cmwsk                                     1/1     Running   1 (37m ago)    152m    10.244.2.6       k8s-w2    <none>           <none>
kube-flannel   kube-flannel-ds-88r76                                       1/1     Running   1 (37m ago)    105m    192.168.10.102   k8s-w2    <none>           <none>
kube-system    coredns-66bc5c9577-v99kw                                    1/1     Running   1 (37m ago)    41m     10.244.2.4       k8s-w2    <none>           <none>
kube-system    kube-proxy-9p996                                            1/1     Running   1 (37m ago)    41m     192.168.10.102   k8s-w2    <none>           <none>
monitoring     alertmanager-kube-prometheus-stack-alertmanager-0           2/2     Running   2 (37m ago)    126m    10.244.2.3       k8s-w2    <none>           <none>
monitoring     kube-prometheus-stack-kube-state-metrics-7846957b5b-hz9vf   1/1     Running   18 (37m ago)   126m    10.244.2.5       k8s-w2    <none>           <none>
monitoring     kube-prometheus-stack-operator-584f446c98-p5nms             1/1     Running   1 (37m ago)    126m    10.244.2.2       k8s-w2    <none>           <none>
monitoring     kube-prometheus-stack-prometheus-node-exporter-gkcfw        1/1     Running   1 (37m ago)    126m    192.168.10.102   k8s-w2    <none>           <none>
 
# 다시 시도 -> 취소
kubectl drain k8s-w2 --ignore-daemonsets --delete-emptydir-data
node/k8s-w2 already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-88r76, kube-system/kube-proxy-9p996, monitoring/kube-prometheus-stack-prometheus-node-exporter-gkcfw
evicting pod monitoring/kube-prometheus-stack-operator-584f446c98-p5nms
evicting pod kube-system/coredns-66bc5c9577-v99kw
evicting pod default/webpod-697b545f57-cmwsk
evicting pod monitoring/alertmanager-kube-prometheus-stack-alertmanager-0
evicting pod monitoring/kube-prometheus-stack-kube-state-metrics-7846957b5b-hz9vf
error when evicting pods/"webpod-697b545f57-cmwsk" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
pod/alertmanager-kube-prometheus-stack-alertmanager-0 evicted
pod/kube-prometheus-stack-operator-584f446c98-p5nms evicted
pod/kube-prometheus-stack-kube-state-metrics-7846957b5b-hz9vf evicted
evicting pod default/webpod-697b545f57-cmwsk
CTRL+C로 취소
 
# 노드 및 파드 상태 확인
kubectl get node
NAME      STATUS                     ROLES           AGE     VERSION
k8s-ctr   Ready                      control-plane   3h54m   v1.34.3
k8s-w1    Ready                      <none>          3h53m   v1.32.11
k8s-w2    Ready,SchedulingDisabled   <none>          3h52m   v1.32.11
 
kubectl get pod -A -owide |grep k8s-w2
default        webpod-697b545f57-cmwsk                                     1/1     Running           1 (39m ago)   154m    10.244.2.6       k8s-w2    <none>           <none>
kube-flannel   kube-flannel-ds-88r76                                       1/1     Running           1 (39m ago)   107m    192.168.10.102   k8s-w2    <none>           <none>
kube-system    kube-proxy-9p996                                            1/1     Running           1 (39m ago)   43m     192.168.10.102   k8s-w2    <none>           <none>
monitoring     kube-prometheus-stack-prometheus-node-exporter-gkcfw        1/1     Running           1 (39m ago)   128m    192.168.10.102   k8s-w2    <none>           <none>
 
# pdb 제거 후 다시 drain 시도
kubectl delete pdb webpod
kubectl drain k8s-w2 --ignore-daemonsets --delete-emptydir-data
node/k8s-w2 already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-88r76, kube-system/kube-proxy-9p996, monitoring/kube-prometheus-stack-prometheus-node-exporter-gkcfw
evicting pod default/webpod-697b545f57-cmwsk
pod/webpod-697b545f57-cmwsk evicted
node/k8s-w2 drained
 
# 노드 상태 확인
kubectl get node
NAME      STATUS                     ROLES           AGE     VERSION
k8s-ctr   Ready                      control-plane   3h55m   v1.34.3
k8s-w1    Ready                      <none>          3h54m   v1.32.11
k8s-w2    Ready,SchedulingDisabled   <none>          3h53m   v1.32.11
 
kc describe node k8s-w2
...
Taints:             node.kubernetes.io/unschedulable:NoSchedule

node 업그레이드 진행한다. 이번엔 워커노드에서 실행한다.

bash

# repo 수정 : 기본 1.32 -> 1.33
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
dnf makecache
 
# 설치
dnf install -y --disableexcludes=kubernetes kubeadm-1.33.7-150500.1.1
 
# 설치 파일들 확인
which kubeadm && kubeadm version -o yaml
 
# 노드 업그레이드 수행 : Upgrade commands for a node in the cluster
kubeadm upgrade node
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade/preflight] Running pre-flight checks
[upgrade/preflight] Skipping prepull. Not a control plane node.
[upgrade/control-plane] Skipping phase. Not a control plane node.
[upgrade/kubeconfig] Skipping phase. Not a control plane node.
W0118 18:03:11.809998   20192 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config1774036046 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1774036046/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/addon] Skipping the addon/coredns phase. Not a control plane node.
[upgrade/addon] Skipping the addon/kube-proxy phase. Not a control plane node.

kubelet 업그레이드 한다

bash

# 전체 설치 가능 버전 확인 
dnf list --showduplicates kubelet --disableexcludes=kubernetes
dnf list --showduplicates kubectl --disableexcludes=kubernetes
 
# Upgrade 설치
dnf install -y --disableexcludes=kubernetes kubelet-1.33.7-150500.1.1 kubectl-1.33.7-150500.1.1
 
# 설치 파일들 확인
which kubectl && kubectl version --client=true
which kubelet && kubelet --version
 
# 재시작
systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet --no-pager
 
# 데몬셋 파드 정상 기동 상태 확인
crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                                                    NAMESPACE
26dcd175dbdce       d84558c0144bc       58 minutes ago      Running             kube-flannel        1                   7f590b37d6d89       kube-flannel-ds-88r76                                  kube-flannel
381e37256545e       4461daf6b6af8       58 minutes ago      Running             kube-proxy          1                   01f149693ad57       kube-proxy-9p996                                       kube-system
fffdbdfe2dc47       6b5bc413b280c       58 minutes ago      Running             node-exporter       1                   7850bc5648f48       kube-prometheus-stack-prometheus-node-exporter-gkcfw   monitoring
 
# 확인 : 아직 k8s-k2 는 SchedulingDisabled 상태
kubectl get nodes -o wide
NAME	  STATUS                     ROLES           AGE   VERSION
k8s-ctr   Ready                      control-plane   57m   v1.34.3
k8s-w1    Ready                      <none>          56m   v1.32.11
k8s-w2    Ready,SchedulingDisabled   <none>          55m   v1.33.7

다시 업그레이드 진행해서, 1.34 업그레이드 한다.

bash

# repo 수정 : 기본 1.33 -> 1.34
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.34/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
dnf makecache
 
# 설치
dnf install -y --disableexcludes=kubernetes kubeadm-1.34.3-150500.1.1
 
# 설치 파일들 확인
which kubeadm && kubeadm version -o yaml
 
# 노드 업그레이드 수행 : Upgrade commands for a node in the cluster
kubeadm upgrade node
 
# Upgrade 설치
dnf install -y --disableexcludes=kubernetes kubelet-1.34.3-150500.1.1 kubectl-1.34.3-150500.1.1
 
# 설치 파일들 확인
which kubectl && kubectl version --client=true
which kubelet && kubelet --version
 
# 재시작
systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet --no-pager
 
# 데몬셋 파드 정상 기동 상태 확인
crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                                                    NAMESPACE
26dcd175dbdce       d84558c0144bc       58 minutes ago      Running             kube-flannel        1                   7f590b37d6d89       kube-flannel-ds-88r76                                  kube-flannel
381e37256545e       4461daf6b6af8       58 minutes ago      Running             kube-proxy          1                   01f149693ad57       kube-proxy-9p996                                       kube-system
fffdbdfe2dc47       6b5bc413b280c       58 minutes ago      Running             node-exporter       1                   7850bc5648f48       kube-prometheus-stack-prometheus-node-exporter-gkcfw   monitoring
 
# 확인
kubectl get nodes -o wide
NAME      STATUS                     ROLES           AGE   VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                   CONTAINER-RUNTIME
k8s-ctr   Ready                      control-plane   59m   v1.34.3    192.168.10.100   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-124.27.1.el10_1.aarch64   containerd://2.2.1
k8s-w1    Ready                      <none>          58m   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
k8s-w2    Ready,SchedulingDisabled   <none>          57m   v1.34.3    192.168.10.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5

업그레이드 이후 uncordon 해 준다.

bash

# 워커 노드 업그레이드 완료 후 정상화 상태면, uncordon 으로 정상화
kubectl uncordon k8s-w2
node/k8s-w2 uncordoned
 
kubectl get node -owide
NAME      STATUS   ROLES           AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                   CONTAINER-RUNTIME
k8s-ctr   Ready    control-plane   175m   v1.34.3    192.168.10.100   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-124.27.1.el10_1.aarch64   containerd://2.1.5
k8s-w1    Ready    <none>          171m   v1.32.11   192.168.10.101   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-55.39.1.el10_0.aarch64    containerd://2.1.5
k8s-w2    Ready    <none>          170m   v1.34.3    192.168.10.102   <none>        Rocky Linux 10.1 (Red Quartz)   6.12.0-124.27.1.el10_1.aarch64   containerd://2.1.5
 
# 파드 배치 확인
kubectl scale deployment webpod --replicas 1
kubectl get pod -owide
kubectl scale deployment webpod --replicas 2
kubectl get pod -owide
NAME                      READY   STATUS    RESTARTS      AGE    IP            NODE      NOMINATED NODE   READINESS GATES
curl-pod                  1/1     Running   0             59m    10.244.0.10   k8s-ctr   <none>           <none>
webpod-697b545f57-9lfzd   1/1     Running   0             36s    10.244.2.7    k8s-w2    <none>           <none>
webpod-697b545f57-kh44p   1/1     Running   1 (71m ago)   3h4m   10.244.1.5    k8s-w1    <none>           <none>

위와 같은 형태로 w1 노드도 실행한다.

전부 버전 업그레이드가 된 것을 알 수 있다.

워커 노드 제거

bash

# node drain
kubectl drain k8s-w1 --ignore-daemonsets --delete-emptydir-data
 
# node delete : API 서버에서 노드 객체 제거
kubectl delete node k8s-w1
 
# 노드 상태 확인
kubectl get node
NAME      STATUS   ROLES           AGE     VERSION
k8s-ctr   Ready    control-plane   4h48m   v1.34.3
k8s-w2    Ready    <none>          4h46m   v1.34.3

해당 형태로 제거 해도, 전체 제거가 되지 않는다. 폴더라던지, iptables rule 같은 경우는 직접 제거한다.

bash

# kubeadm reset 실행
kubeadm reset -f
[preflight] Running pre-flight checks
W0118 18:47:05.774616   33957 removeetcdmember.go:105] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Deleted contents of the etcd data directory: /var/lib/etcd
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
 
The reset process does not perform cleanup of CNI plugin configuration,
network filtering rules and kubeconfig files.
 
For information on how to perform this cleanup manually, please see:
    https://k8s.io/docs/reference/setup-tools/kubeadm/kubeadm-reset/
 
# 관련 디렉터리 확인
tree /etc/kubernetes/
/etc/kubernetes/
├── manifests
├── pki
└── tmp
    ├── kubeadm-kubelet-config2965303515
    │   └── config.yaml
    └── kubeadm-kubelet-config350067924
        └── config.yaml
 
tree /var/lib/kubelet/
/var/lib/kubelet/
 
tree /etc/cni
/etc/cni
└── net.d
    └── 10-flannel.conflist
 
# 디렉터리/파일 삭제
rm -rf /etc/cni/net.d
rm -rf /etc/kubernetes/
rm -rf /var/lib/kubelet
rm -rf /var/lib/etcd    # 컨트롤 플레인 노드일 경우만
 
# iptables 정리
iptables -t nat -S
iptables -t filter -S
iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -X
 
 
# containerd 는 삭제하지 않음 (재사용 가능)
systemctl status containerd --no-pager 
systemctl status kubelet --no-pager
 
# kubelet 서비스 중지
systemctl stop kubelet && systemctl disable kubelet
 
# (옵션) contrainrd 서비스 중지
systemctl stop containerd && systemctl disable containerd
 
# reboot
reboot

느낀점

오늘도 실습을 통해 업그레이드 과정을 진행해 보았다. 사실 회사에서 k8s 를 사용하지 않고 docker 만 사용하고 있어, 실질적으로 텍스트로만 공부 해 보는 것보단 실전 느낌이 있어 어려움도 있고 잘 모르는 컴포넌트 들도 있지만, 좋은 기회를 가시다님께서 제공하여 여러가지 시도를 해보았다.