Some jobs are failing because EC2 is terminating Spot instances from being
outbid (BidEvictedEvents in CloudTrail).
* Increase bid percentage to 100%
* Remove highly interrupted instance types, leaving only instance types with
<5% frequency of interruption. https://aws.amazon.com/ec2/spot/instance-advisor/
* Limit InstanceTypes to specific sizes (xlarge or 2xlarge) so only 1-2 jobs
can run at a time, instead of potentially 10+. Reduce the number of Docker
images being pulled at one time to mitigate the 4m Docker image pull
timeout failures from ECS.
* Restore the r5/r5a instance families for consistency with other serverless
projects.
The main purpose of this change is to trigger an infrastructure update for
the Batch compute environment, since changing the LaunchTemplateData doesn't
trigger that.
The instance types change is arbitrary, but memory optimized instances
aren't really necessary either.
Many jobs were failing to start from an error:
DockerTimeoutError: Could not transition to created; timed out after waiting 4m0s
Try to mitigate this by increasing the Docker timeout from the default 4m to 10m.
ECS_CONTAINER_CREATE_TIMEOUT is not documented in the AWS developer guide, but
it does seem to be a real setting from the ECS agent repo README:
a5f8d25f66/README.md
Changes to the launch template (e.g., ECS volume type/size) were not being
applied because the launch template was set to the default version 1 rather
than the latest changed version.
