Scheduled Tasks with AWS Fargate

Concept

  1. Continue to run my API as-is as a Fargate-based task in ECS.
  2. Create AWS Event Bridge rules that are cron-based and spin up short-lived Fargate tasks running the same container as my API, but with a different command sent to the container.
  3. Code to be run in the scheduled tasks follows a convention to make dynamic command/entry point easy.
  4. Task schedule config is handled by a JSON file in the app.

Convention

/app
|__tasks
|__ data_ingestion_task.py
|__ spam_task.py.py
python -c "from app.tasks import spam_task; spam_task.run()"
# I use pipenv. You do you.
if [ ! -z "$1" ]; then
pipenv run python -c "from app.tasks import $1; $1.run()"
else
pipenv run uvicorn app.main:app --host 0.0.0.0
fi

Configuration

{
"tasks": [
{
"name": "IngestData",
"task": "data_ingestion_task",
"cron": "0 6 * * ? *"
}
]
}

Deployment

  1. Merge new code in Github.
  2. Have the build server run tests, build the new version of the container, push the container to AWS ECR, and then deploy with the following steps…
  3. Delete all the current scheduled tasks.
  4. Loop over the tasks in scheduled_tasks.json and create new Event Bridge rules and targets that point to new revisions of our ECS task definition using the latest container.
  5. Create a final new revision of our ESC task definition to run our API and update the ECS service that runs it to point to that revision.
# function used to strip quotes in parsing
function dequote() {
sed -e 's/^"//' -e 's/"$//' <<< $1
}
set -e
# define some variables
ECS_CLUSTER=your-cluster
SERVICE_NAME=your-service
HASH="$(git rev-parse HEAD)" # we use this for versioning
TASK_FAMILY=your-task
ECR_IMAGE="123456.dkr.ecr.us-east-1.amazonaws.com/foobar:$HASH"
# grab the json task definition of what's currently running
TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition "$TASK_FAMILY" --region us-east-1)
# list all current tasks
RULES=$(aws events list-rules --name-prefix "your__prefix" | jq '.Rules[].Name')
# delete all scheduled tasks
IFS=$'\n'
SPLIT_RULES=($RULES)
for rule in "${SPLIT_RULES[@]}"; do
rule=$(dequote $rule)

# get the targets
targets=$(aws events list-targets-by-rule --rule $rule | jq '.Targets[].Id')
split_targets=($targets)

#remove the targets
for target in "$split_targets"; do
target=$(dequote $target)
echo "Removing target $target for rule $rule..."
aws events remove-targets --rule $rule --ids $target
done
echo "Deleting rule $rule..."
aws events delete-rule --name $rule
done
i=0
task=$(jq --argjson I $i '.tasks[$I]' ./app/scheduled_tasks.json)
# for each task in config:
while [ "$task" != null ]
do

# extract config
name=$(jq --argjson I $i '.tasks[$I].name' ./app/scheduled_tasks.json)
name=$(dequote $name)
file=$(jq --argjson I $i '.tasks[$I].task' ./app/scheduled_tasks.json)
file=$(dequote $file)
cron=$(jq --argjson I $i '.tasks[$I].cron' ./app/scheduled_tasks.json)
cron=$(dequote $cron)
# create task revision with appropriate command
NEW_TASK_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$ECR_IMAGE" --arg CMD "$file" '.taskDefinition | .containerDefinitions[0].image = $IMAGE | .containerDefinitions[0].command = ["./bin/start", $CMD] | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities) | del(.registeredAt) | del(.registeredBy)')

NEW_TASK_INFO=$(aws ecs register-task-definition --region us-east-1 --cli-input-json "$NEW_TASK_DEFINTIION")

# grab the revision number for the target
NEW_REVISION=$(echo $NEW_TASK_INFO | jq '.taskDefinition.revision')
# create event bridge rule pointing to that task revision
hush=$(aws events put-rule --schedule-expression "cron($cron)" --name "your__prefix__$name")
# create the JSON describing the scheduled event's target
# this json template file is below
target_template=$(jq '.' ./bin/scheduled_task_definition.json)
cluster=$(aws ecs describe-clusters --cluster your-cluster)
cluster_arn=$(echo $cluster | jq '.clusters[0].clusterArn')
cluster_arn=$(dequote $cluster_arn)
task_arn=$(echo $NEW_TASK_INFO | jq '.taskDefinition.taskDefinitionArn')

task_arn=$(dequote $task_arn)

target=$(echo $target_template | jq --arg name "your__prefix__rule_target__$name" --arg CLUSTER_ARN "$cluster_arn" --arg TASK_ARN "$task_arn" '.[0].Id = $name | .[0].Arn = $CLUSTER_ARN | .[0].EcsParameters.TaskDefinitionArn = $TASK_ARN')
hush=$(aws events put-targets --rule "staging__discovery__$name" --targets "$target")
echo "Created rule and target for staging__discovery__$name."
#iterate
i=$((i+1))
task=$(jq --argjson I $i '.tasks[$I]' ./app/scheduled_tasks.json)
done
.containerDefinitions[0].command = [“./bin/start”, $CMD]
[{
"Id": "[arbitrary id]",
"Arn": "[cluster arn]",
"RoleArn": "arn:aws:iam::123456:role/ecsEventsRole",
"EcsParameters": {
"TaskDefinitionArn": "[passed back from task creation]",
"TaskCount": 1,
"LaunchType": "FARGATE",
"NetworkConfiguration": {
"awsvpcConfiguration": {
"Subnets": [
"subnet-123",
"subnet-4556"
],
"SecurityGroups": ["sg-987"],
"AssignPublicIp": "ENABLED"
}
},
"PlatformVersion": "LATEST"
}
}]
echo "Deploying API..."
NEW_TASK_DEFINTIION=$(echo $TASK_DEFINITION | jq --arg IMAGE "$ECR_IMAGE" '.taskDefinition | .containerDefinitions[0].image = $IMAGE | .containerDefinitions[0].portMappings[0].containerPort = 8000 | .containerDefinitions[0].portMappings[0].hostPort = 8000 | .containerDefinitions[0].portMappings[0].protocol = "tcp" | .containerDefinitions[0].command = ["./bin/start"] | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities) | del(.registeredAt) | del(.registeredBy)')
NEW_TASK_INFO=$(aws ecs register-task-definition --region us-east-1 --cli-input-json "$NEW_TASK_DEFINTIION")NEW_REVISION=$(echo $NEW_TASK_INFO | jq '.taskDefinition.revision')aws ecs update-service --cluster ${ECS_CLUSTER} --service ${SERVICE_NAME} --task-definition ${TASK_FAMILY}:${NEW_REVISION} --region us-east-1

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store