With AWS Batch, you simply package the code for your batch jobs, specify their dependencies, and submit your batch job using the AWS Management Console, CLIs, or SDKs. AWS Batch allows you to specify execution parameters and job dependencies, and facilitates integration with a broad range of popular batch computing workflow engines and languages (e.g., Pegasus WMS, Luigi, and AWS Step Functions). AWS Batch efficiently and dynamically provisions and scales Amazon EC2 and Spot Instances based on the requirements of your jobs. AWS Batch provides default job queues and compute environment definitions that enable you to get started quickly.
Start Using AWS Batch Today
Get Started
AWS Batch allows you to specify resource requirements, such as vCPU and memory, AWS Identity and Access Management (IAM) roles, volume mount points, container properties, and environment variables, to define how jobs are to be run. AWS Batch executes your jobs as containerized applications running on Amazon ECS.
AWS Batch enables you to define dependencies between different jobs. For example, your batch job can be composed of three different stages of processing with differing resource needs. With dependencies, you can create three jobs with different resource requirements where each successive job depends on the previous job.
AWS Batch can be integrated with commercial and open-source workflow engines and languages such as Pegasus WMS and Luigi, enabling you to use familiar workflow languages to model your batch computing pipelines.
AWS Batch provides Managed Compute Environments that dynamically provision and scale compute resources based the volume and resource requirements of your submitted jobs. You can configure your AWS Batch Managed Compute Environments with requirements such as type of EC2 instances, VPC subnet configurations, the min/max/desired vCPUs across all instances, and Spot bid price.
Alternatively, you can provision and manage your own compute resources within AWS Batch Unmanaged Compute Environments if you need to use different configurations (e.g., larger EBS volumes or a different operating system) for your EC2 instances than what is provided by AWS Batch Managed Compute Environments. You just need to provision EC2 instances that include the Amazon ECS agent and run supported versions of Linux and Docker. AWS Batch will then run batch jobs on the EC2 instances that you provision.
AWS Batch enables you to set up multiple queues with different priority levels. Batch jobs are stored in the queues until compute resources are available to execute the job. The AWS Batch scheduler evaluates when, where, and how to run jobs that have been submitted to a queue based on the resource requirements of each job. The scheduler evaluates the priority of each queue and runs jobs in priority order on optimal compute resources (e.g., memory vs CPU optimized), as long as those jobs have no outstanding dependencies.
AWS Batch makes it easy for you to use Amazon EC2 Spot to further reduce the cost of running your batch jobs. AWS Batch dynamically bids for Spot Instances on your behalf, based on the quantity and resource requirements needed for the jobs, as well as the configuration parameters set for your compute resources.
AWS Batch displays key operational metrics for your batch jobs in the AWS Management Console. You can view metrics related to compute capacity, as well as running, pending, and completed jobs. Logs for your jobs (e.g., STDERR and STDOUT) are available in the AWS Management Console and are also written to Amazon CloudWatch Logs.
AWS Batch uses IAM to control and monitor the AWS resources that your jobs can access, such as Amazon DynamoDB tables. Through IAM, you can also define policies for different users in your organization. For example, admins can be granted full access permissions to any AWS Batch API operation, developers can have limited permissions related to configuring compute environments and registering jobs, and end users can be restricted to the permissions needed to submit and delete jobs.
