Web Analytics Made
Easy - StatCounter

Menu

About us Contact us

Elastic Beanstalk Worker Tier Auto Scaling based on SQS message depth

Tech 2019 / 07 / 16

Elastic Beanstalk Worker Tier Auto Scaling based on SQS message depth



AWS Elastic Beanstalk Worker tier is an amazing tool, especially for managing scheduled tasks. AWS Elastic Beanstalk manages it using Amazon SQS queue and running a daemon process that reads messages from the SQS and performs the desired task.

Why do we need SQS message depth based auto scaling?

One problem for scheduled task is, if the tasks are long running or if it’s difficult to calculate the processing time required for each task, then we cannot accurately predict and allocate instances to the Beanstalk Application. Also, after deployment, at any time, the messages in SQS may start stacking up if the allocated instances are unable to process all of them in their capacity. To tackle this problem, we need to trigger auto scaling based on the SQS message depth.

Key points to consider

This solution is based on the following key points.

• It will be triggered by the application code. For this, we will use “. ebextensions” to create a configuration i.e. “.ebextensions/01_sqs_based_autoscaling_policy.config” file.

• After we deploy this, the configuration will add a scaling policy to the auto scaling group associated with the EC2 instances of that Beanstalk application.

• For this configuration to take effect, we need to define the maximum and the minimum number of instances of Beanstalk application. So that, after scaling up or down it never goes beyond the maximum or the minimum number of instances respectively.

• If any change is required in the configuration like the evaluation period or threshold, we just need to update the file and deploy. After deployment, the auto scaling group policy will be updated accordingly.

Alarm creation based on ApproximateNumberOfMessagesVisible metric

Firstly, we need to create an Alarm to detect the number of messages being stacked up in the SQS.

For that, we need the SQS queue name first. If we are using the auto generated SQS queue defined by AWS, then we can get it as follows.

{ "Fn::GetAtt": ["AWSEBWorkerQueue", "QueueName"] }

If the SQS queue was manually defined, then we can get the name in the following way,

"Fn::Select":
- 4
- "Fn::Split":
- '/'
- "Fn::GetOptionSetting":
Namespace: "aws:elasticbeanstalk:sqsd"
OptionName: "WorkerQueueURL"
Then, we will define the threshold for scaling up or down. For this, we will use the default “ApproximateNumberOfMessagesVisible” metric provided by AWS. Then, we need to define the evaluation period and threshold. For example, we would create Alarm if the “ApproximateNumberOfMessagesVisible” metric is greater than 20 for 1 datapoint within 5 minutes. So, the complete configuration will be as follows.

Resources:
ApproximateNumberOfMessagesVisibleAutoScalingAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: "Alarm if queue depth grows beyond threshold"
Namespace: "AWS/SQS"
MetricName: ApproximateNumberOfMessagesVisible
Dimensions:
— Name: QueueName
Value:
"Fn::Select":
— 4
— "Fn::Split":
— '/'
— "Fn::GetOptionSetting":
Namespace: "aws:elasticbeanstalk:sqsd"
OptionName: "WorkerQueueURL"
Statistic: Average
Period: 300
EvaluationPeriods: 1
Threshold: 20
ComparisonOperator: GreaterThanThreshold
AlarmActions:
— Ref: ScaleUpPolicy
OKActions:
— Ref: ScaleDownPolicy

 

Alarm Actions

Now, we will create 2 policies namely, “ScaleUpPolicy” and “ScaleDownPolicy” to increase and decrease instance size by 1 respectively. So, when Alarm is raised, “ScaleUpPolicy” will be triggered and when Alarm is in OK state, “ScaleDownPolicy” will be triggered.

Please note that Cloudwatch Alarms associated with auto scaling is different from other Cloudwatch Alarm. A Cloudwatch Alarm keeps triggering Auto Scaling actions when that Alarm is in a specified state, even if there are no state changes and the Alarm remains in that state. So, it will keep on adding instances if the number of SQS messages go beyond our desired threshold up to the maximum number of instances defined. Conversely, it will keep on removing instances if the number of SQS messages go below our desired threshold up to the minimum number of instances defined.

https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-alarms-trigger-actions/

So, now it’s time to define the policies.

ScaleUpPolicy:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AdjustmentType: ChangeInCapacity
AutoScalingGroupName: {Ref: AWSEBAutoScalingGroup}
ScalingAdjustment: 1

ScaleDownPolicy:
Type: AWS::AutoScaling::ScalingPolicy
Properties:
AdjustmentType: ChangeInCapacity
AutoScalingGroupName: {Ref: AWSEBAutoScalingGroup}
ScalingAdjustment: -1

 

The complete configuration is given here for your convenience.

https://gist.github.com

If you are facing this kind of problem, hopefully, this will make your life a little bit easier. Thanks for your patience. Feel free to .

You have ideas, We have solutions.

CONTACT US!