Kubespray 분석 - 2 (분석편)
11 views
블로그 불러오는 중...
1편에서는 kubespray를 이용해서 노드에 쿠버네티스를 설치해 보았다.
이번에는 어떤 식으로 쿠버네티스가 설치되는지 확인해 볼 예정이다.
앤서블을 3주차에 학습했지만 다시 보려니 기억이 나지 않아 다시한번 기본 개념을 정리만 하고 넘어 간다.
앤서블에서 관리할 대상 서버들의 목록을 정의하는 파일입니다. Kubespray에서는 주로 inventory/mycluster/hosts.yaml 형식을 사용합니다.
kube_control_plane: 마스터 노드 그룹kube_node: 워커 노드 그룹etcd: 데이터 저장소 노드 그룹hosts: kube_node 등)을 타겟으로 하는지 확인해야 합니다.롤은 앤서블의 '기능별 꾸러미' 입니다. Kubespray의 코드를 보면 roles/ 디렉토리 아래 수많은 폴더가 있는데, 이것이 각각의 롤입니다.
tasks/main.yml: 실제 실행될 **작업(Task)**들이 정의됩니다. (가장 먼저 봐야 할 곳)defaults/main.yml: 해당 롤에서 사용하는 기본 변수값입니다. (우선순위가 낮음)vars/main.yml: 해당 롤 내에서 고정적으로 사용되는 변수입니다.templates/: 설정 파일(.conf, .yaml)의 템플릿입니다. (Jinja2 문법 사용)handlers/: 서비스 재시작 등 특정 조건에서만 실행되는 작업입니다.roles/container-engine/docker 폴더의 tasks/main.yml을 열어보면 됩니다.앤서블은 변수(Variable)를 통해 설정을 유연하게 변경합니다. Kubespray는 엄청나게 많은 변수를 제공하여 사용자가 클러스터를 커스텀하게 만듭니다.
inventory/mycluster/group_vars/all/all.yml: 네트워크 대역, 프록시 설정 등 전체 설정.inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml: 쿠버네티스 버전, 네트워크 플러그인(Calico 등) 설정.kube_version 항목을 찾으면 됩니다.앤서블은 작업을 시작하기 전, 대상 서버의 정보를 수집합니다. 이를 Gathering Facts라고 합니다.
ansible_os_family == "Debian" 이면 apt로 패키지를 설치하고,ansible_os_family == "RedHat" 이면 yum으로 설치하도록 조건문(when) 처리를 할 때 사용됩니다.when: ansible_distribution == "Ubuntu" 같은 내용이 있다면, 이는 팩트 정보를 기반으로 OS별 분기 처리를 하는 것입니다.인벤토리와 롤을 연결하여 전체 실행 순서를 정의한 설계도입니다.
cluster.yml입니다.cluster.yml을 열어보면 import_playbook이라는 명령어로 여러 개의 작은 플레이북을 불러오는 것을 볼 수 있습니다. (예: preinstall.yml -> etcd.yml -> kubernetes.yml) 이 순서가 전체 설치 순서입니다.graph TD
%% 사전 준비 단계
subgraph Preparation ["1. 사전 준비 및 검증"]
A["Check Ansible version"] --> B["Inventory setup and validation"]
B --> C["Install bastion ssh config"]
C --> D["Bootstrap hosts for Ansible"]
D --> E["Gather facts"]
end
%% etcd 설치 단계
subgraph ETCD ["2. 데이터베이스(etcd) 구축"]
E --> F["Prepare for etcd install"]
F --> G["Add worker nodes to etcd play"]
G --> H["Install etcd"]
end
%% K8s 코어 설치 단계
subgraph Core ["3. 쿠버네티스 코어 설치"]
H --> I["Install Kubernetes nodes"]
I --> J["Install the control plane"]
J --> K["Invoke kubeadm and install a CNI"]
end
%% 마무리 및 앱 설치
subgraph Finalization ["4. 부가 서비스 및 최적화"]
K --> L["Install Calico Route Reflector"]
L --> M["Patch Kubernetes for Windows"]
M --> N["Install Kubernetes apps"]
N --> O["Apply resolv.conf changes"]
end
%% 결과 리캡
O --> P{PLAY RECAP}
%% 스타일링
style Preparation fill:#f9f,stroke:#333,stroke-width:2px
style ETCD fill:#bbf,stroke:#333,stroke-width:2px
style Core fill:#bfb,stroke:#333,stroke-width:2px
style Finalization fill:#fdb,stroke:#333,stroke-width:2px
style P fill:#fff,stroke:#f00,stroke-width:4px이런 형식으로 진행이 되는 것을 알 수 있다.
Task 를 563개나 된다. 해당 내용은 아래 가시다님의 내용을 공유한다
cat kubespray_install.log | grep -E 'TASK' | wc -l
559
cat kubespray_install.log | grep -E 'TASK'
TASK [Check 2.17.3 <= Ansible version < 2.18.0] ********************************
TASK [Check that python netaddr is installed] **********************************
TASK [Check that jinja is not too old (install via pip)] ***********************
TASK [dynamic_groups : Match needed groups by their old names or definition] ***
TASK [validate_inventory : Stop if removed tags are used] **********************
TASK [validate_inventory : Stop if kube_control_plane group is empty] **********
TASK [validate_inventory : Stop if etcd group is empty in external etcd mode] ***
TASK [validate_inventory : Stop if unsupported version of Kubernetes] **********
TASK [validate_inventory : Stop if known booleans are set as strings (Use JSON format on CLI: -e "{'key': true }")] ***
TASK [validate_inventory : Stop if even number of etcd hosts] ******************
TASK [validate_inventory : Guarantee that enough network address space is available for all pods] ***
TASK [validate_inventory : Check that kube_service_addresses is a network range] ***
TASK [validate_inventory : Check that kube_pods_subnet is a network range] *****
TASK [validate_inventory : Check that kube_pods_subnet does not collide with kube_service_addresses] ***
TASK [validate_inventory : Check that ipv4 IP range is enough for the nodes] ***
TASK [validate_inventory : Stop if unsupported options selected] ***************
TASK [validate_inventory : Ensure minimum containerd version] ******************
TASK [bootstrap_os : Fetch /etc/os-release] ************************************
TASK [bootstrap_os : Include tasks] ********************************************
TASK [bootstrap_os : Gather host facts to get ansible_distribution_version ansible_distribution_major_version] ***
TASK [bootstrap_os : Add proxy to yum.conf or dnf.conf if http_proxy is defined] ***
TASK [bootstrap_os : Check presence of fastestmirror.conf] *********************
TASK [system_packages : Gather OS information] *********************************
TASK [system_packages : Remove legacy docker repo file] ************************
TASK [system_packages : Manage packages] ***************************************
TASK [bootstrap_os : Create remote_tmp for it is used by another module] *******
TASK [bootstrap_os : Gather facts] *********************************************
TASK [bootstrap_os : Assign inventory name to unconfigured hostnames (non-CoreOS, non-Flatcar, Suse and ClearLinux, non-Fedora)] ***
TASK [bootstrap_os : Ensure bash_completion.d folder exists] *******************
TASK [network_facts : Gather ansible_default_ipv4] *****************************
TASK [network_facts : Set fallback_ip] *****************************************
TASK [network_facts : Gather ansible_default_ipv6] *****************************
TASK [network_facts : Set fallback_ip6] ****************************************
TASK [network_facts : Set main access ip(access_ip based on ipv4_stack/ipv6_stack options).] ***
TASK [network_facts : Set main ip(ip based on ipv4_stack/ipv6_stack options).] ***
TASK [network_facts : Set main access ips(mixed ips for dualstack).] ***********
TASK [network_facts : Set main ips(mixed ips for dualstack).] ******************
TASK [Gather minimal facts] ****************************************************
TASK [Gather necessary facts (network)] ****************************************
TASK [Gather necessary facts (hardware)] ***************************************
TASK [adduser : User | Create User Group] **************************************
TASK [adduser : User | Create User] ********************************************
TASK [kubernetes/preinstall : Check if /etc/fstab exists] **********************
TASK [kubernetes/preinstall : Remove swapfile from /etc/fstab] *****************
TASK [kubernetes/preinstall : Mask swap.target (persist swapoff)] **************
TASK [kubernetes/preinstall : Disable swap] ************************************
TASK [kubernetes/preinstall : Check resolvconf] ********************************
TASK [kubernetes/preinstall : Check existence of /etc/resolvconf/resolv.conf.d] ***
TASK [kubernetes/preinstall : Check status of /etc/resolv.conf] ****************
TASK [kubernetes/preinstall : Fetch resolv.conf] *******************************
TASK [kubernetes/preinstall : NetworkManager | Check if host has NetworkManager] ***
TASK [kubernetes/preinstall : Check systemd-resolved] **************************
TASK [kubernetes/preinstall : Set default dns if remove_default_searchdomains is false] ***
TASK [kubernetes/preinstall : Set dns facts] ***********************************
TASK [kubernetes/preinstall : Check if kubelet is configured] ******************
TASK [kubernetes/preinstall : Check if early DNS configuration stage] **********
TASK [kubernetes/preinstall : Target resolv.conf files] ************************
TASK [kubernetes/preinstall : Check if /etc/dhclient.conf exists] **************
TASK [kubernetes/preinstall : Check if /etc/dhcp/dhclient.conf exists] *********
TASK [kubernetes/preinstall : Target dhclient hook file for Red Hat family] ****
TASK [kubernetes/preinstall : Check /usr readonly] *****************************
TASK [kubernetes/preinstall : Stop if non systemd OS type] *********************
TASK [kubernetes/preinstall : Stop if the os does not support] *****************
TASK [kubernetes/preinstall : Stop if memory is too small for control plane nodes] ***
TASK [kubernetes/preinstall : Stop if memory is too small for nodes] ***********
TASK [kubernetes/preinstall : Stop if cgroups are not enabled on nodes] ********
TASK [kubernetes/preinstall : Stop if ip var does not match local ips] *********
TASK [kubernetes/preinstall : Stop if access_ip is not pingable] ***************
TASK [kubernetes/preinstall : Stop if bad hostname] ****************************
TASK [kubernetes/preinstall : Stop if /etc/resolv.conf has no configured nameservers] ***
TASK [kubernetes/preinstall : Create kubernetes directories] *******************
TASK [kubernetes/preinstall : Create other directories of root owner] **********
TASK [kubernetes/preinstall : Check if kubernetes kubeadm compat cert dir exists] ***
TASK [kubernetes/preinstall : Create kubernetes kubeadm compat cert dir (kubernetes/kubeadm issue 1498)] ***
TASK [kubernetes/preinstall : Create cni directories] **************************
TASK [kubernetes/preinstall : NetworkManager | Ensure NetworkManager conf.d dir] ***
TASK [kubernetes/preinstall : NetworkManager | Prevent NetworkManager from managing K8S interfaces (kube-ipvs0/nodelocaldns)] ***
TASK [kubernetes/preinstall : NetworkManager | Add nameservers to NM configuration] ***
TASK [kubernetes/preinstall : Set default dns if remove_default_searchdomains is false] ***
TASK [kubernetes/preinstall : NetworkManager | Add DNS search to NM configuration] ***
TASK [kubernetes/preinstall : NetworkManager | Add DNS options to NM configuration] ***
TASK [kubernetes/preinstall : Confirm selinux deployed] ************************
TASK [kubernetes/preinstall : Set selinux policy] ******************************
TASK [kubernetes/preinstall : Clean previously used sysctl file locations] *****
TASK [kubernetes/preinstall : Stat sysctl file configuration] ******************
TASK [kubernetes/preinstall : Change sysctl file path to link source if linked] ***
TASK [kubernetes/preinstall : Make sure sysctl file path folder exists] ********
TASK [kubernetes/preinstall : Enable ip forwarding] ****************************
TASK [kubernetes/preinstall : Check if we need to set fs.may_detach_mounts] ****
TASK [kubernetes/preinstall : Ensure kubelet expected parameters are set] ******
TASK [kubernetes/preinstall : Disable fapolicyd service] ***********************
TASK [kubernetes/preinstall : Check if we are running inside a Azure VM] *******
TASK [container-engine/validate-container-engine : Validate-container-engine | check if fedora coreos] ***
TASK [container-engine/validate-container-engine : Validate-container-engine | set is_ostree] ***
TASK [container-engine/validate-container-engine : Ensure kubelet systemd unit exists] ***
TASK [container-engine/validate-container-engine : Populate service facts] *****
TASK [container-engine/validate-container-engine : Check if containerd is installed] ***
TASK [container-engine/validate-container-engine : Check if docker is installed] ***
TASK [container-engine/validate-container-engine : Check if crio is installed] ***
TASK [container-engine/containerd-common : Containerd-common | check if fedora coreos] ***
TASK [container-engine/containerd-common : Containerd-common | set is_ostree] ***
TASK [container-engine/runc : Runc | check if fedora coreos] *******************
TASK [container-engine/runc : Runc | set is_ostree] ****************************
TASK [container-engine/runc : Runc | Uninstall runc package managed by package manager] ***
TASK [container-engine/runc : Runc | Download runc binary] *********************
TASK [container-engine/runc : Prep_download | Set a few facts] *****************
TASK [container-engine/runc : Download_file | Set pathname of cached file] *****
TASK [container-engine/runc : Download_file | Create dest directory on node] ***
TASK [container-engine/runc : Download_file | Download item] *******************
TASK [container-engine/runc : Download_file | Extract file archives] ***********
TASK [container-engine/runc : Copy runc binary from download dir] **************
TASK [container-engine/runc : Runc | Remove orphaned binary] *******************
TASK [container-engine/crictl : Install crictl] ********************************
TASK [container-engine/crictl : Crictl | Download crictl] **********************
TASK [container-engine/crictl : Prep_download | Set a few facts] ***************
TASK [container-engine/crictl : Download_file | Set pathname of cached file] ***
TASK [container-engine/crictl : Download_file | Create dest directory on node] ***
TASK [container-engine/crictl : Download_file | Download item] *****************
TASK [container-engine/crictl : Download_file | Extract file archives] *********
TASK [container-engine/crictl : Extract_file | Unpacking archive] **************
TASK [container-engine/crictl : Install crictl config] *************************
TASK [container-engine/crictl : Copy crictl binary from download dir] **********
TASK [container-engine/nerdctl : Nerdctl | Download nerdctl] *******************
TASK [container-engine/nerdctl : Prep_download | Set a few facts] **************
TASK [container-engine/nerdctl : Download_file | Set pathname of cached file] ***
TASK [container-engine/nerdctl : Download_file | Create dest directory on node] ***
TASK [container-engine/nerdctl : Download_file | Download item] ****************
TASK [container-engine/nerdctl : Download_file | Extract file archives] ********
TASK [container-engine/nerdctl : Extract_file | Unpacking archive] *************
TASK [container-engine/nerdctl : Nerdctl | Copy nerdctl binary from download dir] ***
TASK [container-engine/nerdctl : Nerdctl | Create configuration dir] ***********
TASK [container-engine/nerdctl : Nerdctl | Install nerdctl configuration] ******
TASK [container-engine/containerd : Containerd | Download containerd] **********
TASK [container-engine/containerd : Prep_download | Set a few facts] ***********
TASK [container-engine/containerd : Download_file | Set pathname of cached file] ***
TASK [container-engine/containerd : Download_file | Create dest directory on node] ***
TASK [container-engine/containerd : Download_file | Download item] *************
TASK [container-engine/containerd : Download_file | Extract file archives] *****
TASK [container-engine/containerd : Containerd | Unpack containerd archive] ****
TASK [container-engine/containerd : Containerd | Generate systemd service for containerd] ***
TASK [container-engine/containerd : Containerd | Ensure containerd directories exist] ***
TASK [container-engine/containerd : Containerd | Generate default base_runtime_spec] ***
TASK [container-engine/containerd : Containerd | Store generated default base_runtime_spec] ***
TASK [container-engine/containerd : Containerd | Write base_runtime_specs] *****
TASK [container-engine/containerd : Containerd | Copy containerd config file] ***
TASK [container-engine/containerd : Containerd | Create registry directories] ***
TASK [container-engine/containerd : Containerd | Write hosts.toml file] ********
TASK [container-engine/containerd : Containerd | Ensure containerd is started and enabled] ***
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Prep_download | Register docker images info] ******************
TASK [download : Prep_download | Create staging directory on remote node] ******
TASK [download : Download | Get kubeadm binary and list of required images] ****
TASK [download : Prep_kubeadm_images | Download kubeadm binary] ****************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Prep_kubeadm_images | Copy kubeadm binary from download dir to system path] ***
TASK [download : Prep_kubeadm_images | Create kubeadm config] ******************
TASK [download : Prep_kubeadm_images | Generate list of required images] *******
TASK [download : Prep_kubeadm_images | Parse list of images] *******************
TASK [download : Prep_kubeadm_images | Convert list of images to dict for later use] ***
TASK [download : Download | Download files / images] ***************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Extract_file | Unpacking archive] *****************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Extract_file | Unpacking archive] *****************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Extract_file | Unpacking archive] *****************************
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Prep_download | Set a few facts] ******************************
TASK [download : Download_file | Set pathname of cached file] ******************
TASK [download : Download_file | Create dest directory on node] ****************
TASK [download : Download_file | Download item] ********************************
TASK [download : Download_file | Extract file archives] ************************
TASK [download : Extract_file | Unpacking archive] *****************************
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [download : Set default values for flag variables] ************************
TASK [download : Set_container_facts | Display the name of the image being processed] ***
TASK [download : Set_container_facts | Set if containers should be pulled by digest] ***
TASK [download : Set_container_facts | Define by what name to pull the image] ***
TASK [download : Set_container_facts | Define file name of image] **************
TASK [download : Set_container_facts | Define path of image] *******************
TASK [download : Set image save/load command for containerd] *******************
TASK [download : Set image save/load command for containerd on localhost] ******
TASK [download : Download_container | Prepare container download] **************
TASK [download : Check_pull_required | Generate a list of information about the images on a node] ***
TASK [download : Check_pull_required | Set pull_required if the desired image is not yet loaded] ***
TASK [download : debug] ********************************************************
TASK [download : Download_container | Download image if required] **************
TASK [download : Download_container | Remove container image from cache] *******
TASK [Gathering Facts] *********************************************************
TASK [Check if nodes needs etcd client certs (depends on network_plugin)] ******
TASK [adduser : User | Create User Group] **************************************
TASK [adduser : User | Create User] ********************************************
TASK [adduser : User | Create User Group] **************************************
TASK [adduser : User | Create User] ********************************************
TASK [etcd : Check etcd certs] *************************************************
TASK [etcd : Check_certs | Register certs that have already been generated on first etcd node] ***
TASK [etcd : Check_certs | Set default value for 'sync_certs', 'gen_certs' and 'etcd_secret_changed' to false] ***
TASK [etcd : Check certs | Register ca and etcd admin/member certs on etcd hosts] ***
TASK [etcd : Check certs | Register ca and etcd node certs on kubernetes hosts] ***
TASK [etcd : Check_certs | Set 'gen_certs' to true if expected certificates are not on the first etcd node(1/2)] ***
TASK [etcd : Check_certs | Set 'gen_certs' to true if expected certificates are not on the first etcd node(2/2)] ***
TASK [etcd : Check_certs | Set 'gen_*_certs' groups to track which nodes needs to have certs generated on first etcd node] ***
TASK [etcd : Check_certs | Set 'etcd_member_requires_sync' to true if ca or member/admin cert and key don't exist on etcd member or checksum doesn't match] ***
TASK [etcd : Check_certs | Set 'sync_certs' to true] ***************************
TASK [etcd : Generate etcd certs] **********************************************
TASK [etcd : Gen_certs | create etcd cert dir] *********************************
TASK [etcd : Gen_certs | create etcd script dir (on k8s-ctr1)] *****************
TASK [etcd : Gen_certs | write openssl config] *********************************
TASK [etcd : Gen_certs | copy certs generation script] *************************
TASK [etcd : Gen_certs | run cert generation script for etcd and kube control plane nodes] ***
TASK [etcd : Gen_certs | run cert generation script for all clients] ***********
TASK [etcd : Gen_certs | check certificate permissions] ************************
TASK [etcd : Trust etcd CA] ****************************************************
TASK [etcd : Gen_certs | target ca-certificate store file] *********************
TASK [etcd : Gen_certs | add CA to trusted CA dir] *****************************
TASK [etcd : Gen_certs | update ca-certificates (RedHat)] **********************
TASK [etcd : Trust etcd CA on nodes if needed] *********************************
TASK [etcd : Gen_certs | target ca-certificate store file] *********************
TASK [etcd : Gen_certs | add CA to trusted CA dir] *****************************
TASK [etcd : Gen_certs | Get etcd certificate serials] *************************
TASK [etcd : Set etcd_client_cert_serial] **************************************
TASK [etcdctl_etcdutl : Download etcd binary] **********************************
TASK [etcdctl_etcdutl : Prep_download | Set a few facts] ***********************
TASK [etcdctl_etcdutl : Download_file | Set pathname of cached file] ***********
TASK [etcdctl_etcdutl : Download_file | Create dest directory on node] *********
TASK [etcdctl_etcdutl : Download_file | Download item] *************************
TASK [etcdctl_etcdutl : Download_file | Extract file archives] *****************
TASK [etcdctl_etcdutl : Extract_file | Unpacking archive] **********************
TASK [etcdctl_etcdutl : Copy etcd binary] **************************************
TASK [etcdctl_etcdutl : Copy etcdctl and etcdutl binary from download dir] *****
TASK [etcdctl_etcdutl : Create etcdctl wrapper script] *************************
TASK [etcd : Install etcd] *****************************************************
TASK [etcd : Get currently-deployed etcd version] ******************************
TASK [etcd : Restart etcd if necessary] ****************************************
TASK [etcd : Install | Copy etcd binary from download dir] *********************
TASK [etcd : Configure etcd] ***************************************************
TASK [etcd : Configure | Check if etcd cluster is healthy] *********************
TASK [etcd : Configure | Refresh etcd config] **********************************
TASK [etcd : Refresh config | Create etcd config file] *************************
TASK [etcd : Configure | Copy etcd.service systemd file] ***********************
TASK [etcd : Configure | reload systemd] ***************************************
TASK [etcd : Configure | Ensure etcd is running] *******************************
TASK [etcd : Configure | Wait for etcd cluster to be healthy] ******************
TASK [etcd : Configure | Check if member is in etcd cluster] *******************
TASK [etcd : Refresh etcd config] **********************************************
TASK [etcd : Refresh config | Create etcd config file] *************************
TASK [etcd : Refresh etcd config again for idempotency] ************************
TASK [etcd : Refresh config | Create etcd config file] *************************
TASK [kubernetes/node : Set kubelet_cgroup_driver_detected fact for containerd] ***
TASK [kubernetes/node : Set kubelet_cgroup_driver] *****************************
TASK [kubernetes/node : Ensure /var/lib/cni exists] ****************************
TASK [kubernetes/node : Install | Copy kubelet binary from download dir] *******
TASK [kubernetes/node : Ensure nodePort range is reserved] *********************
TASK [kubernetes/node : Verify if br_netfilter module exists] ******************
TASK [kubernetes/node : Verify br_netfilter module path exists] ****************
TASK [kubernetes/node : Enable br_netfilter module] ****************************
TASK [kubernetes/node : Persist br_netfilter module] ***************************
TASK [kubernetes/node : Check if bridge-nf-call-iptables key exists] ***********
TASK [kubernetes/node : Enable bridge-nf-call tables] **************************
TASK [kubernetes/node : Set kubelet api version to v1beta1] ********************
TASK [kubernetes/node : Write kubelet environment config file (kubeadm)] *******
TASK [kubernetes/node : Write kubelet config file] *****************************
TASK [kubernetes/node : Write kubelet systemd init file] ***********************
TASK [kubernetes/node : Enable kubelet] ****************************************
TASK [kubernetes/control-plane : Pre-upgrade | Delete control plane manifests if etcd secrets changed] ***
TASK [kubernetes/control-plane : Create kube-scheduler config] *****************
TASK [kubernetes/control-plane : Install | Copy kubectl binary from download dir] ***
TASK [kubernetes/control-plane : Install kubectl bash completion] **************
TASK [kubernetes/control-plane : Set kubectl bash completion file permissions] ***
TASK [kubernetes/control-plane : Check which kube-control nodes are already members of the cluster] ***
TASK [kubernetes/control-plane : Set fact first_kube_control_plane] ************
TASK [kubernetes/control-plane : Kubeadm | Check if kubeadm has already run] ***
TASK [kubernetes/control-plane : Kubeadm | aggregate all SANs] *****************
TASK [kubernetes/control-plane : Kubeadm | Create kubeadm config] **************
TASK [kubernetes/control-plane : Kubeadm | Initialize first control plane node (1st try)] ***
TASK [kubernetes/control-plane : Create kubeadm token for joining nodes with 24h expiration (default)] ***
TASK [kubernetes/control-plane : Set kubeadm_token] ****************************
TASK [kubernetes/control-plane : Kubeadm | Join other control plane nodes] *****
TASK [kubernetes/control-plane : Set kubeadm_discovery_address] ****************
TASK [kubernetes/control-plane : Upload certificates so they are fresh and not expired] ***
TASK [kubernetes/control-plane : Parse certificate key if not set] *************
TASK [kubernetes/control-plane : Wait for k8s apiserver] ***********************
TASK [kubernetes/control-plane : Check already run] ****************************
TASK [kubernetes/control-plane : Kubeadm | Remove taint for control plane node with node role] ***
TASK [kubernetes/control-plane : Include kubeadm secondary server apiserver fixes] ***
TASK [kubernetes/control-plane : Update server field in component kubeconfigs] ***
TASK [kubernetes/control-plane : Include kubelet client cert rotation fixes] ***
TASK [kubernetes/control-plane : Fixup kubelet client cert rotation 1/2] *******
TASK [kubernetes/control-plane : Fixup kubelet client cert rotation 2/2] *******
TASK [kubernetes/control-plane : Install script to renew K8S control plane certificates] ***
TASK [kubernetes/control-plane : Renew K8S control plane certificates monthly 1/2] ***
TASK [kubernetes/control-plane : Renew K8S control plane certificates monthly 2/2] ***
TASK [kubernetes/client : Set external kube-apiserver endpoint] ****************
TASK [kubernetes/client : Create kube config dir for current/ansible become user] ***
TASK [kubernetes/client : Copy admin kubeconfig to current/ansible become user home] ***
TASK [kubernetes/client : Wait for k8s apiserver] ******************************
TASK [kubernetes-apps/cluster_roles : Kubernetes Apps | Wait for kube-apiserver] ***
TASK [kubernetes-apps/cluster_roles : Kubernetes Apps | Add ClusterRoleBinding to admit nodes] ***
TASK [kubernetes-apps/cluster_roles : Apply workaround to allow all nodes with cert O=system:nodes to register] ***
TASK [kubernetes-apps/cluster_roles : Kubernetes Apps | Remove old webhook ClusterRole] ***
TASK [kubernetes-apps/cluster_roles : Kubernetes Apps | Remove old webhook ClusterRoleBinding] ***
TASK [kubernetes-apps/cluster_roles : PriorityClass | Copy k8s-cluster-critical-pc.yml file] ***
TASK [kubernetes-apps/cluster_roles : PriorityClass | Create k8s-cluster-critical] ***
TASK [kubernetes/kubeadm : Set kubeadm_discovery_address] **********************
TASK [kubernetes/kubeadm : Check if kubelet.conf exists] ***********************
TASK [kubernetes/kubeadm : Check if kubeadm CA cert is accessible] *************
TASK [kubernetes/kubeadm : Fetch CA certificate from control plane node] *******
TASK [kubernetes/kubeadm : Check if discovery kubeconfig exists] ***************
TASK [kubernetes/kubeadm : Get current resourceVersion of kube-proxy configmap] ***
TASK [kubernetes/kubeadm : Update server field in kube-proxy kubeconfig] *******
TASK [kubernetes/kubeadm : Get new resourceVersion of kube-proxy configmap] ****
TASK [kubernetes/kubeadm : Set ca.crt file permission] *************************
TASK [kubernetes/kubeadm : Restart all kube-proxy pods to ensure that they load the new configmap] ***
TASK [kubernetes/node-label : Kubernetes Apps | Wait for kube-apiserver] *******
TASK [kubernetes/node-label : Set role node label to empty list] ***************
TASK [kubernetes/node-label : Set inventory node label to empty list] **********
TASK [kubernetes/node-label : debug] *******************************************
TASK [kubernetes/node-label : debug] *******************************************
TASK [kubernetes/node-taint : Set role and inventory node taint to empty list] ***
TASK [kubernetes/node-taint : debug] *******************************************
TASK [kubernetes/node-taint : debug] *******************************************
TASK [network_plugin/cni : CNI | make sure /opt/cni/bin exists] ****************
TASK [network_plugin/cni : CNI | Copy cni plugins] *****************************
TASK [network_plugin/cni : CNI | make sure /opt/cni/bin exists] ****************
TASK [network_plugin/cni : CNI | Copy cni plugins] *****************************
TASK [network_plugin/flannel : Flannel | Create Flannel manifests] *************
TASK [network_plugin/flannel : Flannel | Start Resources] **********************
TASK [network_plugin/flannel : Flannel | Wait for flannel subnet.env file presence] ***
TASK [win_nodes/kubernetes_patch : Ensure that user manifests directory exists] ***
TASK [win_nodes/kubernetes_patch : Check current nodeselector for kube-proxy daemonset] ***
TASK [win_nodes/kubernetes_patch : Apply nodeselector patch for kube-proxy daemonset] ***
TASK [win_nodes/kubernetes_patch : debug] **************************************
TASK [win_nodes/kubernetes_patch : debug] **************************************
TASK [kubernetes-apps/ansible : Kubernetes Apps | Wait for kube-apiserver] *****
TASK [kubernetes-apps/ansible : Kubernetes Apps | CoreDNS] *********************
TASK [kubernetes-apps/helm : Helm | Gather os specific variables] **************
TASK [kubernetes-apps/helm : Helm | Install PyYaml] ****************************
TASK [kubernetes-apps/helm : Helm | Download helm] *****************************
TASK [kubernetes-apps/helm : Prep_download | Set a few facts] ******************
TASK [kubernetes-apps/helm : Download_file | Set pathname of cached file] ******
TASK [kubernetes-apps/helm : Download_file | Create dest directory on node] ****
TASK [kubernetes-apps/helm : Download_file | Download item] ********************
TASK [kubernetes-apps/helm : Download_file | Extract file archives] ************
TASK [kubernetes-apps/helm : Extract_file | Unpacking archive] *****************
TASK [kubernetes-apps/helm : Helm | Copy helm binary from download dir] ********
TASK [kubernetes-apps/helm : Helm | Get helm completion] ***********************
TASK [kubernetes-apps/helm : Helm | Install helm completion] *******************
TASK [kubernetes-apps/metrics_server : Metrics Server | Delete addon dir] ******
TASK [kubernetes-apps/metrics_server : Metrics Server | Create addon dir] ******
TASK [kubernetes-apps/metrics_server : Metrics Server | Templates list] ********
TASK [kubernetes-apps/metrics_server : Metrics Server | Create manifests] ******
TASK [kubernetes-apps/metrics_server : Metrics Server | Apply manifests] *******
TASK [adduser : User | Create User Group] **************************************
TASK [adduser : User | Create User] ********************************************
TASK [kubernetes/preinstall : Check resolvconf] ********************************
TASK [kubernetes/preinstall : Check existence of /etc/resolvconf/resolv.conf.d] ***
TASK [kubernetes/preinstall : Check status of /etc/resolv.conf] ****************
TASK [kubernetes/preinstall : Fetch resolv.conf] *******************************
TASK [kubernetes/preinstall : NetworkManager | Check if host has NetworkManager] ***
TASK [kubernetes/preinstall : Check systemd-resolved] **************************
TASK [kubernetes/preinstall : Set default dns if remove_default_searchdomains is false] ***
TASK [kubernetes/preinstall : Set dns facts] ***********************************
TASK [kubernetes/preinstall : Check if kubelet is configured] ******************
TASK [kubernetes/preinstall : Check if early DNS configuration stage] **********
TASK [kubernetes/preinstall : Target resolv.conf files] ************************
TASK [kubernetes/preinstall : Check if /etc/dhclient.conf exists] **************
TASK [kubernetes/preinstall : Check if /etc/dhcp/dhclient.conf exists] *********
TASK [kubernetes/preinstall : Target dhclient hook file for Red Hat family] ****
TASK [kubernetes/preinstall : Check /usr readonly] *****************************
TASK [kubernetes/preinstall : NetworkManager | Ensure NetworkManager conf.d dir] ***
TASK [kubernetes/preinstall : NetworkManager | Prevent NetworkManager from managing K8S interfaces (kube-ipvs0/nodelocaldns)] ***
TASK [kubernetes/preinstall : NetworkManager | Add nameservers to NM configuration] ***
TASK [kubernetes/preinstall : Set default dns if remove_default_searchdomains is false] ***
TASK [kubernetes/preinstall : NetworkManager | Add DNS search to NM configuration] ***
TASK [kubernetes/preinstall : NetworkManager | Add DNS options to NM configuration] ***playbooks/cluster.yml 를 한번 확인해 본다.
과정은 다음과 같다.
A. 초기화 및 정보 수집 (Boilerplate & Facts)
B. 인프라 및 엔진 준비 (Prepare for etcd & container-engine)
C. 데이터 저장소 및 노드 구성 (Etcd & K8s Nodes)
D. 컨트롤 플레인 및 네트워크 (Control Plane & CNI)
kubeadm init 또는 join을 통해 클러스터를 하나로 묶습니다.E. 부가 서비스 설치 (Apps & DNS)
resolv.conf)가 클러스터 내부 DNS를 바라보도록 수정합니다.위의 과정을 거치게 된다.
변수들은 kubespray_defaults 디렉토리에 있다.
kubespray-defaults 도 있는데, 해당 부분은 kubespray_defaults 로 리다이렉트 된다고 한다.
폴더 하위에 download.yml 을 보면 kubespray 가 다운로드 받는 위치들을 설정할 수 있다.
간단하게 살펴보았는데, 다음 편은 실제 kubespray 설치 이후에 서버 내에서 확인 해 본다.