Autoscaling WSO2 API Gateway and WSO2 Choreo Connect on Kubernetes — Part 3
Autoscaling GWs with Horizontal Pod Autoscaler (HPA)
This is a three-part article series on how to deploy WSO2 API manager and Choreo Connect in the same Kubernetes environment and connect up with a single control plane. Please start following the article series from Part 1.
Autoscaling is scaling an application or a component of an application in response to a change in certain metrics such as CPU, Memory or Transactions Per Second. The Gateway component of the WSO2 API Manager and the Router and the Enforcer components of Choreo Connect is architected in such a way that those components can be scaled independently per each gateway environment.
Kubernetes supports both vertical and horizontal scaling. For the scope of this article, we will consider horizontal scaling. Kubernetes supports horizontal scaling through the Horizontal Pod AutoScalars.
More information about implementing autoscaling with Kubernetes can found in the following page.
Prerequisites for autoscaling
- Running K8S cluster — Covered in Part1
2. WSO2 API Manager deployed in a distributed manner with Control Plane and Gateway in different pods — Covered in Part1
3. Choreo Connect Gateway environment deployed (Can be connected to the same CP or kept standalone) — Covered in Part2
4. An API is created and deployed in each gateway environment along with a sample backend.
We will deploy a netty-based backend to demonstrate this scenario. The following yaml definition can be used to deploy the netty backend pod and the service.
Create, publish and deploy an API in the API Manager by logging into the publisher app via https://am.wso2.com/publisher
Provide the Required Values as below.
- API Name: EchoAPI
- Context: /echo
- Version: 1.0.0
- Production Endpoint: http://netty-backend-service:8688
Click ‘Save and Deploy’ and select both gateway environments (Default and cc1) to deploy the API to both gateway environments.
Now we have an API that we can use to generate some load and subsequently increase CPU consumption.
5. K8S Metrics Server installed
Horizontal Pod Autoscalars in Kubernetes rely on the Kubernetes metrics server to expose the necessary APIs it requires to read the metrics values which is the trigger for autoscaling. It can be installed on EKS via the following command.
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yamlAfter installing verify that it is running by checking the existence and the status of the pod.
kubectl get deployment metrics-server -n kube-systemIt should show an output similar to the following. Notice at the bottom of the list, the metrics server is running and ready.
6. Sample script to generate a load (e.g. shell script or jmeter)
A sample script is needed to simulate a load so that we can observe a CPU spike which will cause the API Gateway Deployment to autoscale. The following Jmeter sample script can be used to execute this. I have deployed Jmeter on an EC2 instance and the script is copied to the same instance.
After the prerequisites are implemented, the Horizontal Pod Autoscalars (hpa) can be implemented and tested. For demonstration purposes, 5% CPU metric will be used as the benchmark for autoscaling.
Autoscaling policies can be deployed for each of the standard API Gateway deployments as well as the Choreo Connect Deployment.
Autoscaling the Choreo Connect Deployment
Connect to the EC2 instance via EC2 Instance Connect and run the following command. The Jmeter script is parameterized to accept different values for host, resource path and a JWT token. Replace the <JWT_TOKEN> with the JWT token obtained above.
jmeter -n -t /home/ec2-user/K8sPerfTest.jmx -Jtoken=<JWT_TOKEN> -Jgateway_server=gw.cc.wso2.com -Japi_path=echo/1.0.0/abc -Jhttp_method=POST
Watch the HPA for any changes using the command below.
kubectl get hpa -n wso2 -w
List the pods to notice that 8 pods of the choreo-connect-deployment are running now as a result of autoscaling being triggered.
As the gateways are increased, it can be observed that the throughput is getting increased as well.
Autoscaling the Standard Gateway
Deploy the autoscaling policy for the Standard API Gateway deployment using the follwing command.
kubectl autoscale deployment wso2am-pattern-3-am-gateway-deployment — cpu-percent=5 — min=1 — max=8 -n wso2In order to access the API a valid JWT token is required. A valid token can be generated by using a pair of clientId and clientSecret which can be obtained by logging into the devportal via https://am.wso2.com/devportal and subscribing to the Echo API. More information on how to generate a new token can be found in the following page.
Connect to the EC2 instance via EC2 Instance Connect and run the following command. The Jmeter script is parameterized to accept different values for host, resource path, and a JWT token. Replace the <JWT_TOKEN> with the JWT token obtained above.
jmeter -n -t /home/ec2-user/K8sPerfTest.jmx -Jtoken=<JWT_TOKEN> -Jgateway_server=gateway.am.wso2.com -Japi_path=echo/1.0.0/abc -Jhttp_method=POSTThe following Jmeter output can be observed.
Watch the HPA for any changes using the command below.
kubectl get hpa -n wso2 -w
List the pods to notice that 8 pods of the wso2am-pattern-3-am-gateway-deployment are running now as a result of autoscaling being triggered.
As the gateways are increased, it can be observed that the throughput is getting increased as well.
We have successfully demonstrated how to autoscale the scale the standard API Gateway deployment and a Choreo Connect deployment on top of kubernetes !!!!!
Please let me know if you have any questions. Happy autoscaling.
