Ansible playbooks to deploy and configure Kubernetes/Openshift clusters on IBM Cloud (#5163)

scripts/ansible/kubernetes-cluster/README.md

@@ -0,0 +1,147 @@
+# Ansible Playbooks for odo testing
+
+## IBM Cloud Kubernetes Cluster
+
+This ansible playbook deploys a VPC Kubernetes/OpenShift cluster on IBM Cloud and an NFS server on the same VPC (to be used for dynamic storage provisioning - deploying the NFS provisioner is required, see below).
+It uses the [IBM Cloud Ansible Collections](https://github.com/IBM-Cloud/ansible-collection-ibm/).
+
+### VPC Resources
+
+The following VPC infrastructure resources will be created (Ansible modules in
+parentheses):
+
+* Resource group (ibm_resource_group)
+* VPC (ibm_is_vpc)
+* Security Group (ibm_is_security_group_rule)
+* Public gateway (ibm_is_public_gateway)
+* VPC Subnet (ibm_is_subnet)
+* SSH Key (ibm_is_ssh_key)
+* Virtual Server Instance (ibm_is_instance)
+* Floating IP (ibm_is_floating_ip)
+* Cloud Object Storage (ibm_resource_instance)
+* VPC Kubernetes Cluster (ibm_container_vpc_cluster)
+
+All created resources (expect resource group and SSH Key) will be inside the created Resource Group.
+
+Note that:
+- ibm_is_security_group_rule is not idempotent: each time the playbook is ran, an entry in the Inbound Rules of the Security Group allowing port 22 will be added. You should remove the duplicates from the UI and keep only one entry.
+- I (feloy) didn't find a way to uninstall an addon from a cluster using the IBM Cloud ansible collection (https://github.com/IBM-Cloud/ansible-collection-ibm/issues/70). You will need to remove the "Block Storage for VPC" default add-on if you install an NFS provisioner for this cluster.
+
+
+### Configuration Parameters
+
+The following parameters can be set by the user, either by editing the `vars.yaml` or by usning the `-e` flag from the `ansible-galaxy` command line:
+
+* `name_prefix`: Prefix used to name created resources
+* `cluster_zone`: Zone on which will be deployed the resources
+* `total_ipv4_address_count`: Number of IPv4 addresses available in the VPC subnet
+* `ssh_public_key`: SSH Public key deployed on the NFS server
+* `nfs_image`: The name of the image used to deploy the NFS server
+* `kube_version`: Kubernetes or OpenShift version. The list of versions can be obtained with `ibmcloud ks versions`
+* `node_flavor`: Flavor of workers of the cluster. The list of flavors can be obtained with `ibmcloud ks flavors --zone ${CLUSTER_ZONE} --provider vpc-gen2`
+* `workers`: Number of workers on the cluster
+* `cluster_id_file`: File on which the cluster ID will be saved
+* `nfs_ip_file`: File on which the private IP of the NFS server will be saved
+
+### Running
+
+#### Set API Key and Region
+
+1. [Obtain an IBM Cloud API key](https://cloud.ibm.com/docs/account?topic=account-userapikey&interface=ui).
+
+2. Export your API key to the `IC_API_KEY` environment variable:
+
+    ```
+    export IC_API_KEY=<YOUR_API_KEY_HERE>
+    ```
+
+3. Export desired IBM Cloud region to the 'IC_REGION' environment variable:
+
+    ```
+    export IC_REGION=<REGION_NAME_HERE>
+    ```
+
+You can get available regions supporting Kubernetes clusters on the page https://cloud.ibm.com/docs/containers?topic=containers-regions-and-zones.
+
+#### Install Ansible collections
+
+To install the required Ansible collections, run:
+
+```
+ansible-galaxy collection install -r requirements.yml
+```
+
+#### Create
+
+To create all resources, run the 'create' playbook:
+
+For example:
+
+```
+$ ansible-playbook create.yml \
+    -e name_prefix=odo-tests-openshift \
+    -e kube_version=4.7_openshift \
+    -e cluster_id_file=/tmp/openshift_id \
+    -e nfs_ip_file=/tmp/nfs_openshift_ip \
+    --key-file <path_to_private_key> # For an OpenShift cluster v4.7
+
+$ ansible-playbook create.yml \
+    -e name_prefix=odo-tests-kubernetes \
+    -e kube_version=1.20 \
+    -e cluster_id_file=/tmp/kubernetes_id \
+    -e nfs_ip_file=/tmp/nfs_kubernetes_ip \
+    --key-file <path_to_private_key> # For a Kubernetes cluster v1.20
+```
+
+The `path_to_private_key` file contains ths SSH private key associated with the SSH public key set in the `ssh_public_key` configuration parameter.
+
+#### Destroy
+
+To destroy all resources run the 'destroy' playbook:
+
+```
+ansible-playbook destroy.yml -e name_prefix=...
+```
+
+## Kubernetes Operators
+
+This ansible playbook deploys operators on a Kubernetes cluster. The cluster should be running the Operator Lifecycle Manager ([OLM](https://olm.operatorframework.io/)), either natively for an OpenShift cluster, or by installing it on a Kubernetes cluster.
+
+
+### Running
+
+1. Install necessary Python modules:
+```
+pip3 install ansible openshift
+```
+
+2. Install Ansible collections
+
+To install the required Ansible collections, run:
+
+```
+ansible-galaxy collection install -r requirements.yml
+```
+
+3. Connect to the cluster and make sure your `kubeconfig` points to the cluster.
+
+4. Install the operators:
+```
+ansible-playbook operators.yml
+```
+
+## NFS provisioner
+
+You can run the following commands upon a cluster to deploy the NFS provisioner to this cluster (either Kubernetes or OpenShift). You will need to uninstall the "Block Storage for VPC" add-on installed by default, to make the NFS provisioner work correctly.
+
+```
+$ helm repo add nfs-subdir-external-provisioner \
+    https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
+
+$ helm install nfs-subdir-external-provisioner \
+    nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
+    --set nfs.server=$(</tmp/nfs_ip) \
+    --set nfs.path=/mnt/nfs \
+    --set storageClass.defaultClass=true \
+    --set storageClass.onDelete=delete
+```

scripts/ansible/kubernetes-cluster/create.yml

@@ -0,0 +1,218 @@
+---
+- name: Create OpenShift Cluster on IBM Cloud
+  hosts: localhost
+  collections:
+   - ibm.cloudcollection
+  tasks:
+    - name: Fetch the variables from var file
+      include_vars:
+        file: vars.yml
+  
+    - name: Configure Resource Group
+      ibm_resource_group:
+        name: "{{ name_prefix }}-group"
+        state: available
+      register: rg_create_output
+
+    - name: Save Resource Group as fact
+      set_fact:
+        cacheable: True
+        rg: "{{ rg_create_output.resource }}"
+      when: rg_create_output.rc==0
+
+    - name: Configure VPC
+      ibm_is_vpc:
+        name: "{{ name_prefix }}-vpc"
+        resource_group: "{{ rg.id }}"
+        state: available
+      register: vpc_create_output
+
+    - name: Save VPC as fact
+      set_fact:
+        cacheable: True
+        vpc: "{{ vpc_create_output.resource }}"
+      when: vpc_create_output.rc==0
+
+    # ibm_is_security_group_rule is not idempotent - comment this block if you want to run several times
+    - name: Configure Security Group Rule to open SSH on the NFS server
+      ibm_is_security_group_rule:
+        group: "{{ vpc.default_security_group }}"
+        direction: inbound
+        remote: 0.0.0.0/0
+        tcp:
+          - port_max: 22
+            port_min: 22
+        state: available
+
+    - name: Configure Public Gateway
+      ibm_is_public_gateway:
+        name: "{{ name_prefix }}-gw"
+        resource_group: "{{ rg.id }}"
+        zone: "{{ cluster_zone }}"
+        vpc: "{{ vpc.id }}"
+        state: available
+      register: gw_create_output
+
+    - name: Save Public Gateway as fact
+      set_fact:
+        cacheable: True
+        gw: "{{ gw_create_output.resource }}"
+      when: gw_create_output.rc==0
+
+    - name: Configure VPC Subnet
+      ibm_is_subnet:
+        name: "{{ name_prefix }}-subnet"
+        resource_group: "{{ rg.id }}"
+        vpc: "{{ vpc.id }}"
+        zone: "{{ cluster_zone }}"
+        total_ipv4_address_count: "{{ total_ipv4_address_count }}"
+        public_gateway: "{{ gw.id }}"
+        state: available
+      register: subnet_create_output
+
+    - name: Save VPC Subnet as fact
+      set_fact:
+        cacheable: True
+        subnet: "{{ subnet_create_output.resource }}"
+      when: subnet_create_output.rc==0
+
+    - name: Retrieve image list
+      ibm_is_images_info:
+      register: images_list
+
+    - name: Set VM image name/id dictionary fact
+      set_fact:
+        cacheable: True
+        image_dict: "{{ images_list.resource.images |
+                        items2dict(key_name='name', value_name='id') }}"
+
+    - name: Configure SSH Key
+      ibm_is_ssh_key:
+        name: "ansible-ssh-key"
+        public_key: "{{ ssh_public_key }}"
+      register: ssh_key_create_output
+
+    - name: Save SSH Key as fact
+      set_fact:
+        cacheable: True
+        ssh_key: "{{ ssh_key_create_output.resource }}"
+
+    - name: Configure VSI for NFS server
+      ibm_is_instance:
+        name: "{{ name_prefix }}-nfs"
+        resource_group: "{{ rg.id }}"
+        vpc: "{{ vpc.id }}"
+        profile: "bx2-2x8"
+        image: "{{ image_dict[nfs_image] }}"
+        keys:
+            - "{{ ssh_key.id }}"
+        primary_network_interface:
+          - subnet: "{{ subnet.id }}"
+        zone: "{{ cluster_zone }}"
+        state: available
+      register: nfs_create_output
+
+    - name: Save VSI as fact
+      set_fact:
+        cacheable: True
+        nfs: "{{ nfs_create_output.resource }}"
+
+    - name: Configure Floating IP Address
+      ibm_is_floating_ip:
+        name: "{{ name_prefix }}-nfs-ip"
+        target: "{{ nfs.primary_network_interface[0]['id'] }}"
+        state: available
+      register: nfsip_create_output
+
+    - name: Save Floating IP as fact
+      set_fact:
+        cacheable: True
+        nfsip: "{{ nfsip_create_output.resource }}"
+
+    - name: Add NFS to Ansible inventory
+      add_host:
+        name: "{{ nfsip.address }}"
+        ansible_user: root
+        groups: new_vsi
+        ansible_ssh_extra_args: -o StrictHostKeyChecking=no
+
+    - name: Configure Cloud Object Storage
+      ibm_resource_instance:
+        name: "{{ name_prefix }}-cos"
+        resource_group_id: "{{ rg.id }}"
+        service: "cloud-object-storage"
+        plan: "standard"
+        location: "global"
+        state: available
+      register: cos_create_output
+
+    - name: Save Cloud ObjectStorage Subnet as fact
+      set_fact:
+        cacheable: True
+        cos: "{{ cos_create_output.resource }}"
+      when: cos_create_output.rc==0
+
+    - name: Configure cluster
+      ibm_container_vpc_cluster:
+        name: "{{ name_prefix }}-cluster"
+        resource_group_id: "{{ rg.id }}"
+        kube_version: "{{ kube_version }}"
+        flavor: "{{ node_flavor }}"
+        worker_count: "{{ workers }}"
+        vpc_id: "{{ vpc.id }}"
+        cos_instance_crn: "{{ cos.crn }}"
+        zones: 
+        - {
+          subnet_id: "{{ subnet.id }}",
+          name: "{{ cluster_zone }}"
+        }
+        state: available
+      register: cluster_create_output
+
+    - name: Save Cluster as fact
+      set_fact:
+        cacheable: True
+        cluster: "{{ cluster_create_output.resource }}"
+      when: cluster_create_output.rc==0
+
+    - local_action:
+        module: copy 
+        content: "{{ cluster.id }}"
+        dest: "{{ cluster_id_file }}"
+
+    - local_action:
+        module: copy 
+        content: "{{ nfs.primary_network_interface[0].primary_ipv4_address }}"
+        dest: "{{ nfs_ip_file }}"
+
+
+- name: Check Ansible connection to new NFS server
+  hosts: new_vsi
+  gather_facts: False
+  tasks:
+    - name: Wait for VSI to become reachable over SSH
+      wait_for_connection:
+
+- name: Configure VSI as NFS server
+  hosts: new_vsi
+  tasks:
+  - name: Install required packages
+    ansible.builtin.package:
+      name: nfs-kernel-server
+      state: present
+  
+  - name: Create shared directory
+    ansible.builtin.file:
+      path: /mnt/nfs
+      state: directory
+      mode: '0777'
+
+  - name: Create NFS configuration
+    copy:
+      content: "/mnt/nfs *(rw,all_squash)"
+      dest: "/etc/exports"
+
+  - name: Restart service nfs-kernel-server
+    ansible.builtin.service:
+      name: nfs-kernel-server
+      state: restarted

scripts/ansible/kubernetes-cluster/destroy.yml

@@ -0,0 +1,209 @@
+---
+- name: Destroy OpenShift Cluster on IBM Cloud
+  hosts: localhost
+  collections:
+   - ibm.cloudcollection
+  tasks:
+
+    - name: Fetch the variables from var file
+      include_vars:
+        file: vars.yml
+
+    - name: Get the NFS IP details
+      ibm_is_floating_ip_info:
+        name: "{{ name_prefix }}-nfs-ip"
+      failed_when:
+         - nfsip_output.rc != 0
+         - '"No floatingIP found" not in nfsip_output.stderr'
+      register: nfsip_output
+
+    - name: set nfsip in fact
+      set_fact:
+        cacheable: True
+        nfsip: "{{ nfsip_output.resource }}"
+      when: nfsip_output.resource.id is defined
+
+    - name: Remove NFS IP
+      ibm_is_floating_ip:
+        id: "{{ nfsip.id }}"
+        state: absent
+      when:
+      - nfsip is defined
+
+
+    - name: Get the NFS server details
+      ibm_is_instance_info:
+        name: "{{ name_prefix }}-nfs"
+      failed_when:
+         - nfs_output.rc != 0
+         - '"No Instance found" not in nfs_output.stderr'
+      register: nfs_output
+
+    - name: set nfs in fact
+      set_fact:
+        cacheable: True
+        nfs: "{{ nfs_output.resource }}"
+      when: nfs_output.resource.id is defined
+
+    - name: Remove NFS server
+      ibm_is_instance:
+        id: "{{ nfs.id }}"
+        image: "{{ nfs.image }}"
+        resource_group: "{{ nfs.resource_group }}"
+        vpc: "{{ nfs.vpc }}"
+        profile: "{{ nfs.profile }}"
+        keys: "{{ nfs.keys }}"
+        primary_network_interface:
+        - subnet: "{{ nfs.primary_network_interface[0].subnet }}"
+        zone: "{{ nfs.zone }}"
+        state: absent
+      when:
+      - nfs is defined
+
+
+    - name: Get the vpc details
+      ibm_is_vpc_info:
+        name: "{{ name_prefix }}-vpc"
+      failed_when:
+         - vpc_output.rc != 0
+         - '"No VPC found" not in vpc_output.stderr'     
+      register: vpc_output
+
+    - name: set vpc in fact
+      set_fact:
+        cacheable: True
+        vpc: "{{ vpc_output.resource }}"
+      when: vpc_output.resource.id is defined
+
+
+    - name: Get the subnet details
+      ibm_is_subnet_info:
+        name: "{{ name_prefix }}-subnet"
+      failed_when:
+         - subnet_output.rc != 0
+         - '"No subnet found" not in subnet_output.stderr'
+      register: subnet_output
+
+    - name: set subnet in fact
+      set_fact:
+        cacheable: True
+        subnet: "{{ subnet_output.resource }}"
+      when: subnet_output.resource.id is defined
+
+
+    - name: Get the cluster details
+      ibm_container_vpc_cluster_info:
+        name: "{{ name_prefix }}-cluster"
+      failed_when:
+         - cluster_output.rc != 0
+         - '"cluster could not be found" not in cluster_output.stderr'
+      register: cluster_output
+
+    - name: set cluster in fact
+      set_fact:
+        cacheable: True
+        cluster: "{{ cluster_output.resource }}"
+      when: cluster_output.resource.id is defined
+
+    - name: Remove Cluster
+      ibm_container_vpc_cluster:
+        id: "{{ cluster.id }}"
+        state: absent
+        name: "{{ name_prefix }}-cluster"
+        flavor: "{{ cluster.worker_pools.0.flavor }}"
+        vpc_id: "{{ vpc.id }}"
+        zones: 
+        - {
+          subnet_id: "{{ subnet.id }}",
+          name: "{{ cluster.worker_pools.0.zones.0.zone }}"
+        }
+      when:
+      - vpc is defined
+      - subnet is defined
+      - cluster is defined
+
+
+    - name: Get the Resource group details
+      ibm_resource_group_info:
+        name: "{{ name_prefix }}-group"
+      failed_when:
+         - rg_output.rc != 0
+         - '"Given Resource Group is not found" not in rg_output.stderr'     
+      register: rg_output
+
+    - name: set Resource group in fact
+      set_fact:
+        cacheable: True
+        rg: "{{ rg_output.resource }}"
+      when: rg_output.resource.id is defined
+
+
+    - name: Get the Cloud Object Storage details
+      ibm_resource_instance_info:
+        name: "{{ name_prefix }}-cos"
+        resource_group_id: "{{ rg.id }}"
+      failed_when:
+         - cos_output.rc != 0
+         - '"No resource instance found" not in cos_output.stderr'
+      when: rg is defined
+      register: cos_output
+
+    - name: set Cloud Object Storage in fact
+      set_fact:
+        cacheable: True
+        cos: "{{ cos_output.resource }}"
+      when: cos_output.resource.id is defined
+
+
+    - name: Remove Cloud Object Storage
+      ibm_resource_instance:
+        id: "{{ cos.id }}"
+        name: "{{ name_prefix }}-cos"
+        service: "cloud-object-storage"
+        plan: "standard"
+        location: "global"
+        state: absent
+      when: cos is defined
+
+
+    - name: Remove VPC Subnet
+      ibm_is_subnet:
+        state: absent
+        id: "{{ subnet.id }}"
+      when: subnet is defined
+
+
+    - name: Get the Public Gateway details
+      ibm_is_public_gateway_info:
+        name: "{{ name_prefix }}-gw"
+      failed_when:
+         - gw_output.rc != 0
+         - '"No Public gateway found" not in gw_output.stderr'
+      register: gw_output
+
+    - name: set Public Gateway in fact
+      set_fact:
+        cacheable: True
+        gw: "{{ gw_output.resource }}"
+      when: gw_output.resource.id is defined
+
+
+    - name: Remove Public Gateway
+      ibm_is_public_gateway:
+        id: "{{ gw.id }}"
+        state: absent
+      when: gw is defined
+
+
+    - name: Remove VPC
+      ibm_is_vpc:
+        state: absent
+        id: "{{ vpc.id }}"
+      when: vpc is defined
+
+
+    - name: Remove Resource Group
+      ibm_resource_group:
+        state: absent
+        id: "{{ rg.id }}"
+      when: rg is defined

scripts/ansible/kubernetes-cluster/operators.yml

@@ -0,0 +1,21 @@
+---
+- name: Install Operators on Kubernetes Cluster
+  hosts: localhost
+  collections:
+   - community.kubernetes
+  tasks:
+
+  - name: Create a Subscription for Redis Operator
+    k8s:
+      state: present
+      definition:
+        apiVersion: operators.coreos.com/v1alpha1
+        kind: Subscription
+        metadata:
+          name: my-redis-operator
+          namespace: openshift-operators
+        spec:
+          channel: stable
+          name: redis-operator
+          source: community-operators
+          sourceNamespace: openshift-marketplace

scripts/ansible/kubernetes-cluster/requirements.yml

@@ -0,0 +1,5 @@
+---
+collections:
+- ibm.cloudcollection
+- name: community.kubernetes
+  version: 2.0.0

scripts/ansible/kubernetes-cluster/vars.yml

@@ -0,0 +1,9 @@
+---
+name_prefix: odo-test-openshift
+total_ipv4_address_count: 256
+cluster_zone: eu-de-2
+kube_version: 4.7_openshift
+node_flavor: bx2.4x16
+workers: 3
+nfs_image: ibm-ubuntu-20-04-2-minimal-amd64-1
+ssh_public_key: AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFJaWm3EMxPLf/JMfJamXeVrsOQBgKe4F7thRZ5RYYmA0a9owJ9pievDPB459K7EWBNYZH1UtuG9ipCR25Y8baDZ5XWnK4e3z4nNhEbMxUnS2or52JCuix6LBbYEbsgLPPiof8kXUEKUGvzfJ0vASFcrF9XKMnQ789A9ee8BMd0SZMAs2Jp2wQ3gjzyg4ZpQvfmQ0ua4EVDu3RItSba8F5cfgGLVQPj+4K+WHfQf0jG7ew5F7LolfdjgAj041RbpYZx8SNDxdsoi06rttn92aEUgutGDHaa/P4JhuFZg4IRFwqDrEEyLuEhP89DniWmTyewyetEScmu2gVmFSFfHzPx3L9pkOP/zDTRzuuOYXNmqNUH3nt/kOxxzzuubTiSpGCP3H9NgYzOQk2E4m94znFVb3fifXETL9kOsZQ90yMnRrYRSPSmAcIiFXSWAFUIQ4Y6aTMI2D1ZoS2gMniKRN3SldkH3UeuKq2jf7TZpDMSQCcPbwZN0Qe+uLt/fiQKM=