An alternate approach to CI/CD for WSO2 using Bitbucket and Bitbucket Branch Source Plugin

7 min readJan 31, 2021

Background

Continuous integration and continuous deployments have become a popular DevOps approach for increasing the efficiency of the software development lifecycle. Using Maven for builds, releases, and deployments is the most predominant CI/CD approach for WSO2 integration artifacts. Having said that, certain reasons such as aligning your organization’s existing CI/CD process to align with WSO2 releases can prompt the DevOps to take alternate approaches to CI/CD.

In this article, we will see how a non-maven based approach can be implemented to support CI/CD using Bitbucket Server and Jenkins. The enablement of this approach is done by the Bitbucket Branch Source Plugin.

Note: It is possible to use different source code management systems such as Bitbucket Cloud or Github and Github Branch Source Plugin to implement the same process.

Approach

Two environments Dev and Prod are used for demonstration purposes. It can be extended to additional environments and pipelines. The methodology can be depicted at a high level using the following diagrams.

At a high level, there are two main pipelines that can be orchestrated using a single orchestration pipeline. Developers implement the code and push the code into the Bitbucket repository and depending on the content of the push to the repository, a decision can be taken to execute one of the two pipelines.

Dev Pipeline

After the code is implemented and is locally tested by running unit tests and integration tests, it is pushed to the Bitbucket repository. A webhook attached to the repository triggers a build-job in Jenkins which will discover the branches with a valid Jenkinsfile and start building the codebase. The built artifact is deployed to the Dev environment for further testing for functionality and performance. This step can even be extended to deploy the same artifact into potential higher environments such as Test or Pre Prod as well.

Prod Pipeline

When the developer testing is complete and the developers and the testers are confident of the functionality and the performance, the integrations are ready to be released to the Prod environment. This process is initiated by creating a tag in the source control system and push it. The CI/CD process is capable of identifying a tag and only execute steps related to the production pipeline or even execute a separate different pipeline for production. This process should also be capable of reverting the artifacts in case of a failure as well.

Prerequisites

Installing and configuring Java 8 on your machine.
Installing and configuring Maven on your machine.
Two WSO2 Enterprise Integrator 6.X.X instances available to simulate the Dev and Prod environments. Tip you can run multiple WSO2 EI servers in the same machine by configuring the port offset in the <EI_HOME>/conf/carbon.xml file

Implementation

Configure the Bitbucket Server

Download the zip archive from the bitbucket server download page and unzip it. Let’s call the unzipped folder as <bitbucket_home>.
Go into <bitbucket_home>/bin and start it by executing ./start-bitbucket.sh
Create a project in Bitbucket called wso2.

Create a new repository called “cicdtest” and provide the default branch as master.

Commit this sample code to the bitbucket respository.
Now that the codebase is ready, a webhook needs to be configured to invoke the Jenkins job in the availability of a new push. This can be achieved by traversing to Repository settings → Webhooks → Create webhook from the left menu. Provide the name as Jenkins and the URL as http://localhost:8080/bitbucket-scmsource-hook/notify?server_url=http%3A%2F%2Flocalhost%3A7990 given that the Jenkins server is running on port 8080 and the Bitbucket server is running on port 7990. Select “Push” under the Events section and make sure the webhook is active. Click create to provision the webhook.

Create a new Bitbucket repo under the same project called “wso2-jenkins-shared-library”. Commit and push the shared pipeline library code from here. Make sure to change the DEPLOYMENT_PATH variable in the DevBuild.groovy and ProdBuild.groovy

Configure Jenkins

Note: The provided pipelines use the cp command to locally copy, delete files. In a real world scenario this can be replaced with scp commands to execute the file operations remotely

Download and Jenkins from the download page and configure it.
Initially it is required to install the required plugins in Jenkins. Navigate to Jenkins Dashboard → Manage Jenkins → Manage Plugins page. From the available section install the Bitbucket Branch Source plugin, Jenkins Git plugin and Basic Branch Build Strategies plugin.
Configure the Bitbucket Endpoint in Jenkins by navigating to Jenkins Dashboard → Manage Jenkins → Configure System and scrolling to Bitbucket Endpoints section.

Navigate to Jenkins Dashboard → Manage Jenkins → Configure System page. Scroll down to the Global Pipeline Libraries section. Configure the shared wso2 pipeline as below.

A Jenkins job is provisioned for each integration project repository under the WSO2 space in Jenkins.

Bitbucket branch source plugin is used in the process of configuring the multi branch pipelines in Jenkins. The plugin and the documentation around it can be found under https://plugins.jenkins.io/cloudbees-bitbucket-branch-source/ .

The Bitbucket branch source plugin is capable of discovering branches and tags in a given bitbucket repository with a Jenkinsfile which describes the various stages of the CI/CD process. If it finds such branches and tags, it starts executing the pipelines associated with a configured Jenkinsfile or a pipeline library automatically.

The configuration of a single job is illustrated below along with how the development pipelines and production pipelines are triggered.

Configuring new jobs in Jenkins

Inside the WSO2 space in Jenkins, select ‘New Item’ from the left menu pane.

Provide the name of the build job (repository name can be used) and select ‘Multibranch Pipeline’ and then click ‘OK’.

Afterward, on the configuration page, the Branch Source section needs to be configured. The following sequence of screenshots depicts the configuration. The following parameters need to be configured as explained.

Server: Select the Bitbucket cloud

Credentials: Select the credentials that allows Jenkins to access the Bitbucket cloud. Select Add → Jenkins and provide the credentials to be able to clone the Bitbucket repository. Provide the name bitbucket-integration for the credential pair.

Provision Bitbucket Credentials in Jenkins

Owner: Provide the project name in Bitbucket

Repository Name: Select the repository name from the repository list

Configure the rest of the configurations as shown in the diagrams and click save at the bottom.

Running Scan Multibranch Pipeline Now triggers Jenkins to look for branches and tags with Jenkinsfile in the cicd repository in Bitbucket.

Testing

Developer machine to Dev environment

A developer can simply commit the code into the master branch and push it to the Bitbucket repository to trigger a development CI/CD flow.

E.g If a pom.xml file has been updated in the repository:

git add pom.xmlgit commit -m “updating pom file”git push origin master

This triggers the Dev pipeline and deploys the artifact to the Dev environment.

Car file copied to file system

Releasing to Prod environment

A developer can simply create a tag and push it to the Bitbucket repository to trigger the production CI/CD pipeline.

git tag -a v1.0.0 -m “tagging v1.0.0”git push origin v1.0.0

This will trigger a production build pipeline. It is designed to seek approval of a privileged user before completing the deployment to production as shown in the image below. Choosing Yes will continue the deployment process and Abort will cease the deployment.