Ceph Octopus (15.2.9) stable on CentOS8 virtal machines

Draft

Install Ceph cluster on CentOS8 virtual machines. The following steps will install a ceph deployer node , a ceph monitor node and three ceph osd nodes

Requirements

Networks
- Management / Provisioning (192.168.126.0/24)
- Public / NAT enabled (192.168.122.0/24)
ceph-adm
- CPU: 1
- Memory: 1GB
- Nics (virtio)
  - Nic1: Management / Provisioning (192.168.126.248)
  - Nic2: Public / NAT (192.168.122.3)
- Disk: 15GB
ceph-mon
- CPU: 2
- Memory: 2GB
- Nics (virtio)
  - Nic1: Management / Provisioning (192.168.126.249)
  - Nic2: Public / NAT (192.168.122.4)
- Disk: 15GB
ceph-osd1/2/3
- CPU: 2
- Memory: 4GB
- Nics (virtio)
  - Nic1: Management / Provisioning (192.168.126.250-252)
  - Nic2: Public / NAT (192.168.122.5-7)
- Disks
  - Disk 1 (OS): 15GB
  - Disk 2 (Pool) : 30GB
  - Disk 3 (Pool): 30GB

Setup

ceph-adm
- CentOS 8 default install
- Set the hostname
  - hostnamectl set-hostname ceph-adm.localdomain
- Network setup
  - NIC1 / ifcfg-ens3
    TYPE=Ethernet NAME=ens3 DEVICE=ens3 ONBOOT=yes IPADDR=192.168.126.248 NETMASK=255.255.255.0
  - NIC2 / ifcfg-ens4
    TYPE=Ethernet NAME=ens4 DEVICE=ens4 ONBOOT=yes IPADDR=192.168.122.3 NETMASK=255.255.255.0 GATEWAY=192.168.122.1 DNS1=192.168.122.1
  - Restart network
    - systemctl restart NetworkManager
  - Edit /etc/hosts to add
    192.168.126.248 ceph-adm 192.168.126.249 ceph-mon 192.168.126.250 ceph-osd1 192.168.126.251 ceph-osd2 192.168.126.252 ceph-osd3
- Edit /root/.ssh/config to add
  Host ceph-mon Hostname ceph-mon User cephadm Host ceph-osd1 Hostname ceph-osd1 User cephadm Host ceph-osd2 Hostname ceph-osd2 User cephadm Host ceph-osd3 Hostname ceph-osd3 User cephadm
- Install ceph repo
  - rpm -ivh http://download.ceph.com/rpm-octopus/el8/noarch/ceph-release-1-1.el8.noarch.rpm
- Set timezone
  - timedatectl set-timezone America/Chicago
  - timedatectl set-ntp true
  - systemctl restart chronyd
  - systemctl enable chronyd
- Add user cephadm
  - useradd cephadm && echo "CephAdm@123#" | passwd --stdin cephadm
  - echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm
  - chmod 0440 /etc/sudoers.d/cephadm
- Disable SELINUX
  - sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
  - setenforce 0
- After all ceph nodes are installed (refer bottom sections) , as root
  ssh-copy-id cephadm@ceph-mon
  ssh-copy-id cephadm@ceph-osd1
  ssh-copy-id cephadm@ceph-osd2
  ssh-copy-id cephadm@ceph-osd3
- EPEL install
  - rpm -ivh http://download.ceph.com/rpm-octopus/el8/noarch/ceph-release-1-1.el8.noarch.rpm
- Update
  - dnf update -y
  - reboot
- Change to user ceph-adm
  - su - cephadm
- Install required packages
  - dnf install vim bash-completion tmux python3-pip
- Clone ceph-ansible and configure
  - git clone https://github.com/ceph/ceph-ansible.git
  - cd ceph-ansible
  - git checkout stable-5.0
  - sudo pip3 install --upgrade pip3
  - sudo pip3 install -r requirements.txt
  - echo "PATH=$PATH:/usr/local/bin" >>~/.bashrc
  - source ~/.bashrc
  - ansible --version (2.9.18) . Ansible is installed as part of requirements.txt
- Prepare ceph config yml files
  - cd ceph-ansible
  - cp group_vars/all.yml.sample group_vars/all.yml and edit
  dummy: ceph_release_num: 15 cluster: ceph mon_group_name: mons osd_group_name: osds rgw_group_name: rgws mds_group_name: mdss nfs_group_name: nfss rbdmirror_group_name: rbdmirrors client_group_name: clients iscsi_gw_group_name: iscsigws mgr_group_name: mgrs rgwloadbalancer_group_name: rgwloadbalancers monitoring_group_name: monitoring grafana_server_group_name: grafana-server configure_firewall: True ntp_service_enabled: true ntp_daemon_type: chronyd ceph_origin: repository ceph_repository: community ceph_stable_release: octopus ceph_repository_type: cdn monitor_interface: ens3 radosgw_interface: ens3 dashboard_enabled: True dashboard_protocol: http dashboard_admin_user: admin dashboard_admin_password: 123456 grafana_admin_user: admin grafana_admin_password: 123456
- cp group_vars/osds.yml.sample group_vars/osds.yml and edit
  dummy: devices:
  - /dev/vdb
  - /dev/vdc
- cp site.yml.sample site.yml
- Edit hosts
  # Ceph admin user for SSH and Sudo
  [all:vars]
  ansible_ssh_user=cephadm
  ansible_become=true
  ansible_become_method=sudo
  ansible_become_user=root
  
  # Ceph Monitor Nodes
  [mons]
  ceph-mon
  
  # MDS Nodes
  [mdss]
  ceph-mon
  
  # RGW
  [rgws]
  ceph-mon
  
  # Manager Daemon Nodes
  [mgrs]
  ceph-mon
  
  # set OSD (Object Storage Daemon) Node
  [osds]
  ceph-osd1
  ceph-osd2
  ceph-osd3
  
  # Grafana server
  [grafana-server]
  ceph-mon
- Deploy the cluster
  - sudo ansible-playbook -i hosts site.yml
  - Status should look like
    INSTALLER STATUS *************************************************************************
    Install Ceph Monitor : Complete (0:00:30)
    Install Ceph Manager : Complete (0:00:34)
    Install Ceph OSD : Complete (0:01:03)
    Install Ceph MDS : Complete (0:00:37)
    Install Ceph RGW : Complete (0:00:27)
- Verify the cluster
  - On ceph-mon
    - sudo ceph -s
- Ceph dashboard
  - on ceph-mon
    - ceph dashboard ac-user-create cent password administrator
    - ceph dashboard ac-user-show cent
  - access URL
    - http://ceph-mon:8443/#/login
    - user: cent , password : password
Ceph-mon
- CentOS8 default install
- Hostname
  - hostnamectl set-hostname ceph-mon.localdomain
- Time
  - timedatectl set-timezone America/Chicago
  - timedatectl set-ntp true
  - systemctl restart chronyd
  - systemctl enable chronyd
- User
  - useradd cephadm && echo "CephAdm@123#" | passwd --stdin cephadm
  - echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm
  - chmod 0440 /etc/sudoers.d/cephadm
  - sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
- Update
  - OS update
    - dnf update -y
    - reboot
- Networks
  - ifcfg-ens3
    TYPE=Ethernet NAME=ens3 DEVICE=ens3 ONBOOT=yes IPADDR=192.168.126.249 NETMASK=255.255.255.0
  - Ifcfg-ens4
    TYPE=Ethernet NAME=ens4 DEVICE=ens4 ONBOOT=yes IPADDR=192.168.122.4 NETMASK=255.255.255.0 GATEWAY=192.168.122.1 DNS1=192.168.122.1
  - Restart network
    - systemctl restart NetworkManager
  - Edit /etc/hosts to add
    192.168.126.248 ceph-adm 192.168.126.249 ceph-mon 192.168.126.250 ceph-osd1 192.168.126.251 ceph-osd2 192.168.126.252 ceph-osd3
Ceph-ods1
- CentOS 8 default install
- Hostname
  - hostnamectl set-hostname ceph-osd1.localdomain
- Time
  - timedatectl set-timezone America/Chicago
  - timedatectl set-ntp true
  - systemctl restart chronyd
  - systemctl enable chronyd
- User
  - useradd cephadm && echo "CephAdm@123#" | passwd --stdin cephadm
  - echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm
  - chmod 0440 /etc/sudoers.d/cephadm
  - sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
- Update
  - OS update
    - dnf update -y
    - reboot
- Networks
  - ifcfg-ens3
    TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens3 DEVICE=ens3 ONBOOT=yes IPADDR=192.168.126.250 NETMASK=255.255.255.0
  - ifcfg-ens
    TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens4 DEVICE=ens4 ONBOOT=yes IPADDR=192.168.122.5 NETMASK=255.255.255.0 GATEWAY=192.168.122.1 DNS=192.168.122.1
  - Restart network
    - systemctl restart NetworkManager
  - Edit /etc/hosts to add
    192.168.126.248 ceph-adm 192.168.126.249 ceph-mon 192.168.126.250 ceph-osd1 192.168.126.251 ceph-osd2 192.168.126.252 ceph-osd3
Ceph-ods2
- CentOS 8 default install
- Hostname
  - hostnamectl set-hostname ceph-osd2.localdomain
- Time
  - timedatectl set-timezone America/Chicago
  - timedatectl set-ntp true
  - systemctl restart chronyd
  - systemctl enable chronyd
- User
  - useradd cephadm && echo "CephAdm@123#" | passwd --stdin cephadm
  - echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm
  - chmod 0440 /etc/sudoers.d/cephadm
  - sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
- Update
  - OS update
    - dnf update -y
    - reboot
- Networks
  - ifcfg-ens3
    TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens3 DEVICE=ens3 ONBOOT=yes IPADDR=192.168.126.251 NETMASK=255.255.255.0
  - Ifcfg-ens4
    TYPE=Ethernet BOOTPROTO=none NAME=ens4 DEVICE=ens4 ONBOOT=yes IPADDR=192.168.122.6 NETMASK=255.255.255.0 GATEWAY=192.168.122.1 DNS1=192.168.122.1
  - Restart network
    - systemctl restart NetworkManager
  - Edit /etc/hosts to add
    192.168.126.248 ceph-adm 192.168.126.249 ceph-mon 192.168.126.250 ceph-osd1 192.168.126.251 ceph-osd2 192.168.126.252 ceph-osd3
Ceph-ods3
- CentOS 8 default install
- Hostname
  - hostnamectl set-hostname ceph-osd3.localdomain
- Time
  - timedatectl set-timezone America/Chicago
  - timedatectl set-ntp true
  - systemctl restart chronyd
  - systemctl enable chronyd
- User
  - useradd cephadm && echo "CephAdm@123#" | passwd --stdin cephadm
  - echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm
  - chmod 0440 /etc/sudoers.d/cephadm
  - sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
- Update
  - OS update
    - dnf update -y
    - reboot
- Networks
  - ifcfg-ens3
    TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens3 DEVICE=ens3 ONBOOT=yes IPADDR=192.168.126.252 NETMASK=255.255.255.0
  - Ifcfg-ens4
    TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none NAME=ens4 DEVICE=ens4 ONBOOT=yes IPADDR=192.168.122.7 NETMASK=255.255.255.0 GATEWAY=192.168.122.1 DNS1=192.168.122.1
  - Restart network
    - systemctl restart NetworkManager
  - Edit /etc/hosts to add
    192.168.126.248 ceph-adm 192.168.126.249 ceph-mon 192.168.126.250 ceph-osd1 192.168.126.251 ceph-osd2 192.168.126.252 ceph-osd3