Development Guide

Prerequisites

The OpenLab CI is built based on NodePool and Zuul tools of the OpenStack infrastructure. The CI jobs definition uses Ansible playbooks. Learn more about the tools first:

  • Zuul documentation is hosted on Zuul

  • NodePool documentation is hosted on NodePool

  • Ansible documentation is hosted on Ansible Docs

Testing workflow

Working on test request with OpenLab CI

The accepted test requests can be found in TODO list, developers can get start with following steps:

  1. Pick one test request you wish to work on, move it to IN PROGRESS list, edit it to add the development plan into comments

  2. The target project of the request should install the OpenLab-CI github APP, so that OpenLab CI system and target project can communicate with each other. See the details in connecting guideline

  3. Fork your own copy of the CI job repository openlab-zuul-jobs to your account, develop the test job and submit a PR(Pull Request) to openlab-zuul-jobs. In order to manage the test request well, please add change details and link the test request number in PR's commit message, the format like:

    Related-Bug: theopenlab/openlab#number

  4. OpenLab development team @theopenlab/dev will review the PR and give some suggestion, development core team @theopenlab/dev-core have permission to merge it if there is no issue, at the meanwhile, you should contact @theopenlab/dev-core to add the target project into main.yaml in zuul node if it is not in project list of zuul

  5. When your PR is merged into master of openlab-zuul-jobs, trigger to test the integration job by trigger comments, see the build status, click result FAILURE enter the build web page and then click the log url to get the log details if the job fail. Maybe you can find some problems of job, feel free to submit new pull request to openlab-zuul-jobs to fix job issue, link the issue number of related test request in PR's commit message as we do on step3

  6. Close the In Progress test request by yourself when the work completely, notify @theopenlab/governance and @theopenlab/docs to update website and documents in RP comments.

Working on test request with OpenLab physical machine

  1. If your test request need some physical machines, please contact @theopenlab/governance members in IRC channel or send them an email, they will help you to get OpenLab dedicated resources to launch testing

  2. When you can access the physical machines, move your test request to IN PROGRESS list, edit it to add the development plan into comments

  3. When the work is completely, close the In Progress test request by yourself, please commit the test result into comments of test request related issue. A blog also is a good choice to show more details of testing. Notify @theopenlab/governance and @theopenlab/docs to update website and documents in RP comments.

Working on feature and bug

  1. Features and bugs for OpenLab are tracked in issue list

  2. Perhaps the merged CI job will fail due to some reasons after it has been running for some time, or if you find any other enhancement points or problems of OpenLab, welcome to new an issue

  3. The development core team @theopenlab/dev-core will confirm the feature or bug, add them into TODO list

  4. You can assign one feature or bug from TODO list, move it to IN PROGRESS list. Feel free to pick up the issues with label need volunteer, these issues are not claimed by anyone yet

  5. Submit a PR(Pull Request) to openlab-zuul-jobs to implement feature or fix bug, please add change details and link the issue number in PR's commit message:

    Closes: theopenlab/openlab#{number}

    Related-Bug: theopenlab/openlab#{number}

  6. OpenLab development team @theopenlab/dev will review the PR and give some suggestion, development core team @theopenlab/dev-core have permission to merge it if there is no problem. When the PR is merged into openlab-zuul-jobs, you can test jobs by trigger comments

  7. Close the In Progress feature or bug by yourself when feature is implemented or bug is fixed.

Integrate with OpenLab CI

There are two ways to integrate development activities of target project with OpenLab CI, you can apply them individually in different scenarios, and applying both two ways is OK to OpenLab.

  1. Pull request of target project triggered

    • Pros: Verifying against with pull request commit to avoid merging code with issue into master

    • Cons: Need to install OpenLab github APP into target project for receiving github events, refer to Connect to OpenLab

  2. Periodic pipeline of OpenLab triggered

    • Pros: Don't need to install OpenLab github APP into target project

    • Cons: Verifying in a fixed period, issues maybe exist in master of target project, issue detection and resolution are delayed.

Branch range of target and backend project validated by OpenLab

OpenLab focus on verifying cross community application integration scenario, we assume base platform (backend) is stable and available to use, like: Kubernetes, OpenStack and so on, and assume target project should be improved to work together well with base platform (backend), so the following table show the scope OpenLab should covered.

Target project

Backend

OpenLab scope

master

master

yes

master

released

yes

released

released

yes

released

master

no

The last case is just to verify backend project, should be covered in backend project CI.

Job naming notations

When we implement an integration test request, usually we need to add new job into openlab-zuul-jobs. To unify the job name format, we have the following naming notations:

{target project}-{target project version}-{test type}-{backend}-{backend version}-{arch}
  • The target project usually is the name of the project repository which contains tests to run.

  • The version is the version of target project, optionally, default is master.

  • The test type is the type of test to run, e.g. acceptance test, integration test, unit test or build.

  • The backend is the test environment provider, include: deploying tools, public cloud, hardware platform, e.g. devstack, kubeadm, minikube, VEXXHOST, optionally, default is devstack.

  • The backend version is the specific backend version which this job will run against, optionally, default is master.

  • The arch is the platform architecture to run testing, e.g. x86_64, arm64, optionally, default is x86_64.

For an example, the job definition about running master acceptance tests of terraform-provider-openstack against with master OpenStack that is deployed by devstack can be named as:

terraform-provider-openstack-acceptance-test

Running Spark v2.4.3 integration test against Kubernetes 1.13.0 that is deployed by minikube:

spark-v2.4.3-integration-test-minikube-k8s-1.13.0

Running Hadoop master branch build test on arm64 platform:

hadoop-build-arm64

Job inherit

OpenLab's job definition should support to put multiple versions of target project together with multiple backends to verify integration stacks as more as possible, the version matrix looks like:

Job name

Target project

Backend

JobA

Spark v2.4.3

Kubernetes 1.14.0

JobB

Spark v2.4.3

Kubernetes 1.13.0

JobC

...

...

JobX

Spark v2.3.3

Kubernetes 1.14.0

JobY

Spark v2.3.3

Kubernetes 1.13.0

JobZ

...

...

Considering test resource limits of OpenLab, we will cover master and latest released of target project and non-EOL backends at the begin, but if you have special needs to specific testing versions, please commit a test request in OpenLab, let us knowing that, just like we have support multiple OpenStack EOL releases in OpenLab integration tests.

Supporting so many integration tests is not a easy thing for OpenLab CI, should design job inherit structure carefully and try to reuse Ansible playbook to support multiple different testing cases, please follow the rule to add your code:

  • Define a base OpenLab zuul job and playbook for master of target project and master of backend. Base, son and grandson jobs run against same playbook run.yaml .

Base job
- job:
name: spark-master-integration-test-kubeadm-k8s-master
parent: init-test
run: playbooks/spark-integration-test-kubeadm-k8s/run.yaml
Son job
- job:
name: spark-master-integration-test-kubeadm-k8s-1.14.0
parent: spark-master-integration-test-kubeadm-k8s-master
vars:
kubernetes_version: 1.14.0
  • The grandson job should be pinned to a specific target project version to testing and inherit from son job, we can use Zuul built-in attribute override-checkout to checkout target project code to specific branch or tag.

Grandson job
- job:
name: spark-v2.4.3-integration-test-kubeadm-k8s-1.14.0
parent: spark-master-integration-test-kubeadm-k8s-1.14.0
override-checkout: v2.4.3

Periodic job and pipeline schedule

OpenLab manage lots of periodic jobs for many projects, these jobs will be triggered by OpenLab CI to execute testing and collect result in everyday. For decreasing OpenLab resource usage, we should split all of periodic jobs into each pipeline as evenly as possible, and avoiding concurrent jobs with same test target affect each other. So there are three basic principles to help you to decide which pipeline when you want to add new periodic job:

  1. Split all of periodic jobs into each pipeline as evenly as possible.

  2. Job backend is local deployed OpenStack, Kubernetes and other, that is OK in same pipeline.

  3. Choose the trigger time you like.

Current pipeline and job mapping:

Pipeline Name

Trigger Time

(UTC-0)

Projects

Job count

periodic-0/12

0:00/12:00

ansible/ansible

apache/spark

11

periodic-2/14

2:00/14:00

terraform-providers/terraform-provider-openstack

containerd/containerd

helm/charts

10

periodic-4/16

4:00/16:00

cloudfoundry/bosh-openstack-cpi-release kubernetes/cloud-provider-openstack

envoyproxy/envoy

11

periodic-6/18

6:00/18:00

hashicorp/packer

kubernetes-sigs/kind

10

periodic-8/20

8:00/20:00

docker/machine

gophercloud/gophercloud

15

periodic-10/22

10:00/22:00

manageiq/manageiq-providers-openstack kubernetes-sigs/cluster-api-provider-openstack

dtantsur/rust-openstack

12