devxy/r-builds
Watch
2
0
Fork
You've already forked r-builds
 0
Code Issues 1 Pull requests Releases Activity
371 commits 2 branches 0 tags 982 KiB
Commit graph r-builds/serverless-resources.yml
Author SHA1 Message Date
Joshua C. Forest
042612f906 match python-builds with new allocation strategy 2023-09-07 18:53:59 -04:00
Joshua C. Forest
21aad5acc5 support staging and production 2023-09-07 17:33:02 -04:00
Joshua C. Forest
1472fe13fc security groups is a list 2023-09-07 12:07:49 -04:00
Joshua C. Forest
e619f3bd40 Tag list is different here 2023-09-06 16:09:47 -04:00
Joshua C. Forest
d88e1c13ca Switch to using vpcId and creating the security group directly from within serverless 2023-09-06 15:35:21 -04:00
Joshua C. Forest
f0582d6fcb Need to write to the filesystem 2023-09-06 10:41:48 -04:00
Joshua C. Forest
c4fcfe533f capitalization 2023-09-06 10:13:27 -04:00
Joshua C. Forest
dc89efcd1c It turns out we can't fix ECS.1 as far as I can tell, we cannot change the networkmode at all unless we're using eks. 
https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html

Revert "explicitly set privileged=False to attempt to quiet security hub finding ECS.1"

This reverts commit dab250f9cb.
 2023-09-01 11:00:45 -04:00
Joshua C. Forest
dab250f9cb explicitly set privileged=False to attempt to quiet security hub finding ECS.1 2023-09-01 10:40:06 -04:00
Joshua C. Forest
a641169eea solve the ECS.5 security hub error, root filesystem should be read only 2023-09-01 10:36:00 -04:00
Joshua C. Forest
5f5418baf1 Encrypt the ebs and use IMDSv2, with the hop limit of 2 since ECS 2023-08-31 10:57:41 -04:00
Colin Rundel
6c9247cfea Attempt at adding builds for Fedora 37 and 38 2023-07-25 17:36:23 -04:00
Greg Lin
fc59a69afd Discontinue openSUSE 15.3 and Ubuntu 18.04 builds for EOL 2023-07-10 15:09:17 -05:00
Gábor Csárdi
6a56d0aa9f Add support for Debian 12 2023-06-17 09:47:41 +02:00
Steve Nolen
2b312547a0 add sns topic publishing of successful jobs 2023-02-27 09:28:19 -05:00
Greg Lin
2da405d505 Mitigate EC2 Spot instance interruptions 
Some jobs are failing because EC2 is terminating Spot instances from being
outbid (BidEvictedEvents in CloudTrail).

* Increase bid percentage to 100%
* Remove highly interrupted instance types, leaving only instance types with
  <5% frequency of interruption. https://aws.amazon.com/ec2/spot/instance-advisor/
 2022-09-07 14:31:07 -05:00
Greg Lin
b2c157f047 Limit instance sizes to mitigate Docker timeout failures 
* Limit InstanceTypes to specific sizes (xlarge or 2xlarge) so only 1-2 jobs
  can run at a time, instead of potentially 10+. Reduce the number of Docker
  images being pulled at one time to mitigate the 4m Docker image pull
  timeout failures from ECS.
* Restore the r5/r5a instance families for consistency with other serverless
  projects.
 2022-09-01 15:42:42 -05:00
Joshua C. Forest
0b6c741323 remove for testing purposes 2022-09-01 13:57:17 -04:00
Greg Lin
4cb2d64968 Update instance types, removing memory optimized instances 
The main purpose of this change is to trigger an infrastructure update for
the Batch compute environment, since changing the LaunchTemplateData doesn't
trigger that.

The instance types change is arbitrary, but memory optimized instances
aren't really necessary either.
 2022-08-31 13:34:57 -05:00
Greg Lin
f6447d8408 Increase ECS_CONTAINER_CREATE_TIMEOUT to 10m to mitigate Docker timeouts 
Many jobs were failing to start from an error:

DockerTimeoutError: Could not transition to created; timed out after waiting 4m0s

Try to mitigate this by increasing the Docker timeout from the default 4m to 10m.
ECS_CONTAINER_CREATE_TIMEOUT is not documented in the AWS developer guide, but
it does seem to be a real setting from the ECS agent repo README:
a5f8d25f66/README.md
 2022-08-29 15:53:29 -05:00
Greg Lin
27472d6bc3 Update block device name for Amazon Linux 2 
Amazon Linux 1 AMIs had a separate /dev/xvdcz volume used for Docker storage.
Amazon Linux 2 AMIs now only have a single root volume at /dev/xvda.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-ami-storage-config.html
 2022-08-26 14:41:10 -05:00
Greg Lin
4c91f1e9d6 Use latest version of the launch template 
Changes to the launch template (e.g., ECS volume type/size) were not being
applied because the launch template was set to the default version 1 rather
than the latest changed version.
 2022-08-26 12:52:00 -05:00
Greg Lin
1e34ec86b5 Increase ECS volume size to 96 GiB 2022-08-25 11:48:19 -05:00
Greg Lin
f8392bdea8 Add R builds for openSUSE 15.4/SLES 15 SP4 2022-08-18 13:15:45 -05:00
Greg Lin
768316ab6f Add R builds for RHEL 9 2022-08-17 15:30:54 -05:00
Greg Lin
b3a54b1583 Drop Debian 9 builds for EOL 
* Update default R version in README
 2022-08-09 16:38:33 -05:00
Joshua C. Forest
c3ee528dc0 update to gp3 volumes as well 2022-07-26 10:29:40 -04:00
Joshua C. Forest
f9030c707f also update bid percentage to 85% like in python-builds 2022-07-26 10:20:16 -04:00
Joshua C. Forest
5128deda83 update to the instance types we use in python-builds 2022-07-26 10:18:17 -04:00
Greg Lin
37d8c527c1 Drop SLES 12 support for EOL 2022-05-20 15:38:39 -05:00
Gábor Csárdi
04a2ff4fc3 Build the next version of R as well 2022-03-30 15:26:12 +02:00
Greg Lin
c7b30828b7 Drop openSUSE 15.2 builds for EOL 2022-03-23 17:47:23 -05:00
Greg Lin
3a1b8a2a10 Add builds for Debian 11 2022-03-21 16:21:25 -05:00
Greg Lin
c87a7a623e Update supported platforms; discontinue Ubuntu 16 and openSUSE 15 2022-02-09 17:52:45 -06:00
Joshua C. Forest
cb84021aad add jammy 2022-02-04 12:06:45 -05:00
Toni Noble
4ef223954c build on opensuse15.3 2021-10-21 10:46:56 -04:00
Greg Lin
c0ee373540 Discontinue CentOS 6 builds for EOL 
This only stops the builds. Existing binaries will continue to be
hosted, and the quick install script continues to support CentOS 6.
 2020-12-02 16:37:15 -06:00
Greg Lin
a4ccbeb3f3 Add builds for openSUSE 15.2 
openSUSE 15.2 moved from ICU 60 to 65, breaking compatibility with
openSUSE 15.0 and 15.1.
 2020-08-31 18:12:46 -05:00
Greg Lin
acf40429a8 Add CloudWatch Events rule to build R-devel every day 
Build at 4 AM UTC. R-devel sources are uploaded at around 12 AM UTC daily.
 2020-07-20 17:49:18 -05:00
Dan Buch
d4e773ca48
 		Merge branch 'master' into mbh-add-debian-10 2020-05-05 12:30:43 -04:00
Greg Lin
034b3712d7 Add timeout to Batch jobs 
R builds can hang forever; for example, the Ubuntu 20.04 builds
with OpenBLAS present.
 2020-05-04 13:55:59 -05:00
Dan Buch
87c9ed44b0
 		Add support for Debian 10 (buster) 
Closes #58
 2020-04-30 15:10:14 -04:00
Jonathan Curran
833b44f315 Add Ubuntu 20 support 2020-04-23 08:40:24 -06:00
Greg Lin
501a66c96c Add R builds for CentOS 8 2019-10-02 17:03:39 -05:00
Steve Nolen
099ca72df5 create a staging s3 bucket 2019-08-16 08:55:25 -04:00
Steve Nolen
e3fc28cb82 larger disks for ecs instances 2019-08-16 07:38:37 -04:00
Steve Nolen
314a3f41a8 add staging env attributes 2019-08-16 07:37:58 -04:00
Steve Nolen
bd3505e426 make repo public 2019-03-25 12:59:14 -04:00