Category: Java

I’m glad to announce that the AWS Lambda plugin in the AWS Toolkit for Eclipse now supports serverless application development for Java. The serverless application (also called a Lambda-based application) is composed of functions triggered by events. In this blog, I provide two examples to show you how to leverage the Eclipse IDE to create and deploy a serverless application quickly.

Install the AWS Toolkit for Eclipse

To install the latest AWS Toolkit for Eclipse, go to this page and follow the instructions at the top right of the page. You should install the AWS Toolkit for Eclipse Core, AWS CloudFormation Tool, and AWS Lambda Plugin to use this feature. The following figure shows where you can choose these three components in the installation wizard. To complete the installation, you need to review and accept the license and restart Eclipse.

Create a Serverless Project

To create a serverless project, click the Toolbar AWS and choose New AWS Serverless Project…, The following wizard opens. You can also create a new serverless project using the AWS Toolkit for Eclipse in the usual way: Choose File, New, Other, AWS and then choose AWS Serverless Java Project. As you can see in the following figure, the Toolkit provides two blueprints for you to start with: article and hello-world.

• article – This is a simple serverless application that helps manage articles. It consists of two Lambda functions triggered by API events. The two functions are GetArticle and PutArticle, which manage storing articles to the backend service and retrieving articles to the front end. This blueprint also leverages an Amazon S3 bucket for storing article content and an Amazon DynamoDB table for storing article metadata.
• hello-world – This blueprint project only includes a simple stand alone Lambda function, HelloWorld, which is not triggered by any event and not bound to any resource. It simply takes in a String and outputs it with the prefix “Hello”. If an empty String is provided, it outputs “Hello World”.

You can also use a serverless template to create a serverless project by choosing Select a Serverless template file and then importing the template file. This template file is a simplified version of the SAM (AWS Serverless Application Model) file that is used in a serverless application to define the application resources stack. The following snippet is from the blueprint articles template for defining the Lambda function GetArticle. Different from the real SAM file, you don’t need to provide the CodeUri and Runtime properties, and you only need to provide the class name for the Handler property instead of the Fully Qualified Class Name. By importing a template file, the AWS Toolkit for Eclipse will generate all the Lambda function hooks and the Lambda Proxy Integration models used as the API event Input and Output for the Lambda functions.

{
  "Type": "AWS::Serverless::Function",
  "Properties": {
    "Handler": "com.serverless.demo.GetArticle",
    "Runtime": "Java",
    "CodeUri": "s3://serverless-bucket/get-article.zip",
    "Policies": [
      "AmazonDynamoDBReadOnlyAccess",
      "AmazonS3ReadOnlyAccess"
    ],
    ...
}

The following figure shows the startup view after you create the article blueprint project. As you can see from the project structure, the AWS Toolkit for Eclipse puts all the Lambda functions defined in the template into a function package, and all the required models into a model package. You can check the serverless.template file for a closer look at this project. As we mentioned earlier, this is a simplified version of a SAM file, which is derived from AWS CloudFormation template. See the README.html page for the next.

Deploy a Serverless Project

If the serverless project is created from a blueprint, you can deploy it directly to AWS. Notice that the article blueprint will create an S3 bucket and a DynamoDB table for use of the Lambda functions. You can open the serverless.template file and customize the resource names in the Parameters property section, as shown in the following snippet.

"Parameters" : {
    "ArticleBucketName" : {
        "Type" : "String",
        "Default" : "serverless-blueprint-article-bucket",
        "Description" : "Name of S3 bucket used to store the article content.",
        "MinLength" : "0"
    },
    "ArticleTableName" : {
        "Type" : "String",
        "Default" : "serverless-blueprint-article-table",
        "Description" : "Name of DynamoDB table used to store the article metadata.",
        "MinLength" : "0"
      },
      ...
}

To deploy this project to AWS, click the project name in the explorer view, choose Amazon Web Services, and then choose Deploy Serverless Project. Or right click the workspace of any Lambda function file, choose AWS Lambda, and then choose Deploy Serverless Project. You will see the following wizard. Choose the S3 bucket, type the CloudFormation stack name, and then choose Finish. The AWS Toolkit for Eclipse generates the fat JAR file for the underlying Lambda functions, and uploads it to the S3 bucket you chose. It’ll also update the serverless.template file in memory to be a real SAM file and upload it to the S3 bucket. AWS CloudFormation reads this file to create the stack.

While the AWS CloudFormation stack is being created, a Stack Editor view is shown to indicate the current status of the stack. This page is automatically refreshed every five seconds, but you can also manually refresh it by clicking the refresh icon at the top right of the view. Upon CREATE_COMPLETE, you will see a link to the right  of the Output label in the top section. This link is the Prod stage endpoint of the API Gateway API created by this serverless project.

Test a Serverless Project

After successfully deploying the article project, you can test the two APIs by hitting the API Prod endpoint through browser tools or command line tools.

• Using the Curl command line tool.

  $ curl --data "This is an article!" https://s5cvlouqwe.execute-api.us-west-2.amazonaws.com/Prod?id=1
  Successfully inserted article 1
  $ curl -X GET https://s5cvlouqwe.execute-api.us-west-2.amazonaws.com/Prod?id=1
  This is an article!

• Using the Simple rest client plugin in Chrome. You can also use this plugin to send a POST request to the endpoint.

We’d like to know what you think of the workflow for developing serverless applications with the AWS Toolkit for Eclipse. Please let us know if there are other features you want to see in this toolkit. We appreciate your comments.

We are excited to announce support for generating a Java SDK for services fronted by Amazon API Gateway. The generated Java SDKs are compatible with Java 8 and later. Generated SDKs have first-class support for API keys, custom or AWS Identity and Access Management (IAM) authentication, automatic and configurable retries, exception handling, and more. In this blog post, we’ll walk through how to create a sample API, and generate a Java SDK from that API, and explore various features of the generated SDK. This post assumes you have some familiarity with API Gateway concepts.

Create an Example API

To start, let’s create a sample API by using the API Gateway console. Navigate to the API Gateway console and select your preferred region. Choose Create API, and then choose the Example API option. Choose Import to create the example API.

The example API is pretty simple. It consists of four operations.

1. A GET on the API root resource that returns HTML describing the API.
2. A GET on the /pets resource that returns a list of Pets.
3. A POST on the /pets resource that creates a new Pet
4. A GET on the /pets/{petId} resource that returns a specific Pet by ID.

Deploy the API

Next, you’ll deploy the API to a stage.

Under Actions, choose Deploy API, name the stage test, and then choose Deploy.

After you deploy the API, on the SDK Generation tab, choose Java as the platform. For Service Name, type PetStore. For Java Package Name, type com.petstore.client. Leave the other fields empty. Choose Generate SDK, and then download and unzip the SDK package.

There are several configuration options available for the Java platform. Before proceeding, let’s go over them.

Service Name – Used to name the Java Interface you’ll use to make calls to your API.

Java Package Name – The name of the package your generated SDK code will be placed under. This name is typically named based on your organization.

The following optional parameters are used when publishing the SDK to a remote repository, like Maven Central.

Java Build System – The build system to configure for the generated SDK, either maven or gradle. The default is maven.

Java Group ID – Typically identifies your organization. Defaults to Java Package Name if not provided.

Java Artifact ID – Identifies the library or product. Defaults to Service Name if not provided.

Java Artifact Version – Version identifier for the published SDK. Defaults to 1.0-SNAPSHOT if not provided.

Compile Client

Navigate to the location where you unzipped the SDK package. If you’ve been following the example, the package will be setup as a Maven project. Ensure Maven and a JDK have been installed correctly, and run the following command to install the client package into your local Maven repository This makes it available for other local projects to use.

mvn install

Set Up an Application

Next, you’ll set up an application that depends on the client package you previously installed. Because the client requires Java 8 or later, any application that depends on the client must also be built with Java 8. Here, you’ll use a simple Maven Archetype to generate an empty Java 8 project.

mvn archetype:generate -B -DarchetypeGroupId=pl.org.miki -DarchetypeArtifactId=java8-quickstart-archetype -DarchetypeVersion=1.0.0 \
    -DgroupId=com.petstore.app \
    -DartifactId=petstore-app \
    -Dversion=1.0 \
    -Dpackage=com.petstore.app

Navigate to the newly created project and open the pom.xml file. Add the following snippet to the <dependencies>…</dependencies section of the XML file. If you changed any of SDK export parameters in the console, use those values instead.

<dependency>
    <groupId>com.petstore.client</groupId>
    <artifactId>PetStore</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>

Create a file src/main/java/com/petstore/app/AppMain.java with the following contents.

package com.petstore.app;

import com.petstore.client.*;
import com.petstore.client.model.*;
import com.amazonaws.opensdk.*;
import com.amazonaws.opensdk.config.*;

public class AppMain {

    public static void main(String[] args) {

    }
}

Build the application to ensure everything is configured correctly.

mvn install

To run the application, you can use the following Maven command. (As you make changes, be sure to rerun mvn install before running the application.)

mvn exec:java -Dexec.mainClass="com.petstore.app.AppMain"

Exploring the SDK

Creating the Client

The first thing you need to do is construct an instance of the client. You can use the client builder obtained from a static factory method on the client interface. All configuration methods on the builder are optional (except for authorization related configuration).In the following code, you obtain an instance of the builder, override some of the configuration, and construct a client. The following settings are for demonstration only, and are not necessarily the recommended settings for creating service clients.

PetStore client = PetStore.builder()
        .timeoutConfiguration(new TimeoutConfiguration()
                                      .httpRequestTimeout(20_000)
                                      .totalExecutionTimeout(30_000))
		.connectionConfiguration(new ConnectionConfiguration()
                                      .maxConnections(100)
                                      .connectionMaxIdleMillis(120))
        .build();

The builder exposes a ton of useful configuration methods for timeouts, connection management, proxy settings, custom endpoints, and authorization. Consult the Javadocs for full details on what is configurable.

Making API Calls

Once you’ve built a client, you’re ready to make an API call.

Call the GET /pets API to list the current pets. The following code prints out each pet to STDOUT. For each API in the service, a method is generated on the client interface. That method’s name will be based on a combination of the HTTP method and resource path, although this can be overridden (more on that later in this post).

client.getPets(new GetPetsRequest())
        .getPets()
        .forEach(p -> System.out.printf("Pet: %s\n", p));

The GET /pets operation exposes a query parameter named type that can be used to filter the pets that are returned. You can set modeled query parameters and headers on the request object.

client.getPets(new GetPetsRequest().type("dog"))
        .getPets()
        .forEach(p -> System.out.printf("Dog: %s\n", p));

Let’s try creating a Pet and inspecting the result from the service. Here you call the POST /pets operation, supplying information about the new Pet. The CreatePetResult contains the unmarshalled service response (as modeled in the Method Response) and additional HTTP-level metadata that’s available via the sdkResponseMetadata() method.

final CreatePetResult result = client.createPet(
        new CreatePetRequest().newPet(new NewPet()
                                              .type(PetType.Bird)
                                              .price(123.45)));
System.out.printf("Response message: %s \n", result.getNewPetResponse().getMessage());
System.out.println(result.sdkResponseMetadata().header("Content-Type"));
System.out.println(result.sdkResponseMetadata().requestId());
System.out.println(result.sdkResponseMetadata().httpStatusCode());

The GET /pets/{petId} operation uses a path placeholder to get a specific Pet, identified by its ID. When making a call with the SDK, all you need to do is supply the ID. The SDK handles the rest.

GetPetResult pet = client.getPet(new GetPetRequest().petId("1"));
System.out.printf("Pet by ID: %s\n", pet);

Overriding Configuration at the Request Level

In addition to the client-level configuration you supply when creating the client (by using the client builder), you can also override certain configurations at the request level. This “request config” is scoped only to calls made with that request object, and takes precedence over any configuration in the client.

client.getPets(new GetPetsRequest()
                       .sdkRequestConfig(SdkRequestConfig.builder()
                                                 .httpRequestTimeout(1000).build()))
        .getPets()
        .forEach(p -> System.out.printf("Pet: %s\n", p));

You can also set custom headers or query parameters via the request config. This is useful for adding headers or query parameters that are not modeled by your API. The parameters are scoped to calls made with that request object.

client.getPets(new GetPetsRequest()
                       .sdkRequestConfig(SdkRequestConfig.builder()
                                                 .customHeader("x-my-custom-header", "foo")
                                                 .customQueryParam("MyCustomQueryParam", "bar")
                                                 .build()))
        .getPets()
        .forEach(p -> System.out.printf("Pet: %s\n", p));

It’s possible to override the default names given to operations through the API Gateway console or during an import from a Swagger file. Let’s rename the GetPet operation (GET /pets/{petId}) to GetPetById by using the console. First, navigate to the GET method on the /pets/{petId} resource.

Choose Method Request, and then expand the SDK Settings section.

Edit the Operation Name field and enter GetPetById. Save the change and deploy the API to the stage you created previously. Regenerate a Java SDK, and it should have the updated naming for that operation.

GetPetByIdResult pet = client.getPetById(new GetPetByIdRequest().petId("1"));
System.out.printf("Pet by ID: %s\n", pet);

If you are importing an API from a Swagger file, you can customize the operation name by using the operationId field. The following snippet is from the example API, and shows how the operationId field is used.

...
    "/pets/{petId}": {
      "get": {
        "tags": [
          "pets"
        ],
        "summary": "Info for a specific pet",
        "operationId": "GetPet",
        "produces": [
          "application/json"
        ],
...

Final Thoughts

This post highlights how to generate the Java SDK of an API in API Gateway, and how to call the API using the SDK in an application. For more information about how to build the SDK package, initiate a client with other configuration properties, make raw requests, configure authorization, handle exceptions, and configure retry behavior, see the README.html file in the uncompressed SDK project folder.

I’m glad to announce that the AWS Elastic Beanstalk plugin in the AWS Toolkit for Eclipse now supports Configuring VPC with Elastic Beanstalk. If you’re new to AWS Toolkit for Eclipse, see the User Guide for a basic introduction and setup guidance. If you’re new to AWS Elastic Beanstalk plugin, see AWS Elastic Beanstalk and Eclipse Integration to learn how to manage your application and environment within Eclipse. If you’re not familiar with VPC configurations for Elastic Beanstalk environments, see Using Elastic Beanstalk with Amazon VPC.

The following screenshots show the plugin New Server wizard pages for VPC configuration. If you’ve used the Elastic Beanstalk console, these user interfaces should be familiar. On the Configure Application and Environment page, you can choose Select a VPC to use when creating your environment to open the VPC Configuration page. Otherwise, when you click Next, the Permissions page opens directly.

Configure Application and Environment Page

On the VPC Configuration page, you can set up the VPC configurations, such as subnets, security group, etc. You must select at least one subnet for a valid configuration. ELB visibility is disabled unless you chose Load Balanced Web Server Environment for the Environment type on the previous Configure Application and Environment page. For more information about the options and how to configure them for your VPC needs, see Using Elastic Beanstalk with Amazon VPC.

VPC Configuration Page

As you can see, it’s easy to configure a VPC for your Elastic Beanstalk environment using the AWS Toolkit for Eclipse plugin. Please let us know if there are other features you want to see in this toolkit. We appreciate your comments.

We’re pleased to announce the addition of the waiters feature in the AWS SDK for Java (take a look at the release notes). Waiters make it easier to wait for a resource to transition into a desired state, which is a very common task when you’re working with services that are eventually consistent (such as Amazon DynamoDB) or have a lead time for creating resources (such as Amazon EC2). Before waiters, it was difficult to come up with the polling logic to determine whether a particular resource had transitioned into a desired state. Now with waiters, you can more simply and easily abstract out the polling logic into a simple API call.

Polling without Waiters

For example, let’s say you wanted to create a DynamoDB table and access it soon after it’s created to add an item into it. There’s a chance that if the table isn’t created already, a ResourceNotFoundException error will be thrown. In this scenario, you have to poll until the table becomes active and ready for use.

//Create an AmazonDynamoDb client 
AmazonDynamoDB client = AmazonDynamoDBClientBuilder
                	.standard()
                	.withRegion(Regions.US_WEST_2)
                	.build();

//Create a table
 client.createTable(new CreateTableRequest().withTableName(tableName)
            .withKeySchema(new KeySchemaElement().withKeyType(KeyType.HASH)
                                                 .withAttributeName("hashKey"))
            .withAttributeDefinitions(new AttributeDefinition()
                                                 .withAttributeType(ScalarAttributeType.S)
                                                 .withAttributeName("hashKey"))
            .withProvisionedThroughput(new ProvisionedThroughput(5L, 5L)));

Without waiters, polling would look like this.

//Polling 5 times for table to become active 
int attempts = 0;
while(attempts < 5){
     try{
           DescribeTableRequest request = new DescribeTableRequest(tableName);
           DescribeTableResult result = client.describeTable(request);
           String status = res.getTable().getTableStatus();
           if(status.equals(“ACTIVE”)){
               break;
	   }
	   Thread.sleep(5000);
	   attempts++;
     }
     catch(ResourceNotFoundException e){
     }
}

Polling with Waiters

Waiters make it easier to abstract out the polling logic into a simple API call. Let’s take a look at how you can create and use waiters to more easily determine whether a DynamoDB table is successfully created and ready to use for further transactions.

//Create waiter to wait on successful creation of table.
Waiter waiter = client.waiters().tableExists();
     try{
          waiter.run(new WaiterParameters<>(new DescribeTableRequest(tableName)); 
     }
     catch(WaiterUnrecoverableException e){
          //Explicit short circuit when the resource transitions into 
          //an undesired state. 
     }
     catch(WaiterTimedOutException e){
          //Failed to transition into desired state even after polling
     }
     catch(DynamoDBException e){
          //Unexpected service exception
     }

For more details, see AmazonDynamoDBWaiters.

Async Waiters

We also offer an async variant of waiters that returns a Future object that promises to hold the result of the computation after it’s done. An async waiter requires a callback interface that is invoked after the Future object is fulfilled. Callback provides an interface to carry out other tasks, depending on whether the resource entered a desired state (onWaitSuccess) or not (onWaitFailure).

To use an async waiter, you must call an async variant of run.

Future future = client.waiters()
                   .tableExists()
                   .runAsync(new WaiterParameters()
                      .withRequest(new DescribeTableRequest(tableName)),
                      new WaiterHandler() {                
                      @Override
                      public void onWaitSuccess(DescribeTableRequest request) {
                          System.out.println("Table creation success!!!!!");
                      }

                      @Override
                      public void onWaitFailure(Exception e) {
	                      e.printStackTrace();
                      }
                 });
				
future.get(5, TimeUnit.MINUTES);

To learn more, see Waiters.

We are excited about this new addition to the SDK! Let us know what you think in the comments section below.

Back in March (1.10.59), the AWS SDK for Java introduced throttled retries, an opt-in feature that could be enabled in the SDK ClientConfiguration to retry failed service requests. Typically, client-side retries are used to avoid unnecessarily surfacing exceptions caused by transient network or service issues. However, when there are longer-running issues (for example, a network or service outage) these retries are less useful. With throttled retries enabled, service calls can fail fast rather than retrying pointlessly.

After testing this code for the past five months, we’ve turned on throttled retries by default.

If you use Amazon CloudWatch to collect AWS SDK metrics (see this post for details), you’ll be pleased to know that there’s a new metric that tracks when retries are throttled. Look for the ThrottledRetryCount metric in the CloudWatch console.

Of course, this remains an option you can configure. If you don’t want to use throttled retries, you can disable the feature through the ClientConfiguration option like so:

ClientConfiguration config = new ClientConfiguration().withThrottledRetries(false);

Feel free to leave questions or feedback in the comments.

This is part III of a blog post series in which we do a deep dive on automated security testing for AWS applications. In part I, we discussed how AWS Java developers can create security unit tests to verify the correctness of their AWS applications by testing individual units of code in isolation. In part II we went one step further and showed how developers can create integration tests that, unlike unit tests, interact with real software components and AWS resources. In this last post in the series, we’ll walk you through how to incorporate the provided security tests into a CI/CD pipeline (created in AWS CodePipeline) to automate security verification when new changes are pushed into the code repository

In part I and part II of this post, we created a suite of unit and integration tests for a simple S3 wrapper Java class. Unit tests focused on testing the class in isolation by using mock objects instead of real Amazon S3 objects and resources. In addition, integration tests were created to complement unit tests and provide an additional layer of verification that uses real objects and resources like S3 buckets, objects, and versions. In this last post in the series (part III), we’ll show how the unit and integration security tests can be incorporated into a CI/CD pipeline to automatically verify the security behavior of code being pushed through the pipeline.

Setting Up

Git and CodeCommit 

Follow the steps in the Integrating AWS CodeCommit with Jenkins blog post to install Git and create an AWS CodeCommit repo. Download the source code and push it to the AWS CodeCommit repo you created.

Jenkins and plugins on EC2

Follow the steps in the Building Continuous Deployment on AWS with AWS CodePipeline, Jenkins and AWS Elastic Beanstalk blog post to install and configure Jenkins on Amazon EC2. Make sure you install the AWS CodePipeline Jenkins plugin to enable AWS CodePipeline and Jenkins integration. In addition, create three Jenkins Maven jobs following the steps described in section “Create a Jenkins Build Job” in that blog post. However, for the Job parameters described below use the values indicated in the table instead.

Make sure you pick an AWS region where AWS CodePipeline is available.

Here’s an example of the configuration options in the Jenkins UI for project SecTestsOnAWS:

Setting up Jenkins to build S3ArtifactManager using Maven

AWS CodePipeline

In the AWS CodePipeline console, create a pipeline with three stages, as shown here.

AWS CodePipeline CI/CD pipeline with security unit/integration tests actions

Stage #1: Source

• Choose AWS CodeCommit as your source provider and enter your repo and branch names where indicated.

Stage #2: Build

Create a build action with the following parameters:

• Action category: Build
• Action name: Build
• Build provider: SecTestsBuildProvider (must match the corresponding Jenkins entry in project SecTestsOnAWS)
• Project name: SecTestsOnAWS
• Input Artifact #1: MyApp
• Output Artifact #1: MyAppBuild

Stage #3: Security-Tests

Create two pipeline actions as follows:

We are not adding a pipeline stage/action for application deployment because we have built a software component (S3ArtifactManager), not a full-fledged application. However, we encourage the reader to create a simple web or standalone application that uses the S3ArtifactManager class and then add a deployment action to the pipeline targeting an AWS Elastic Beanstalk environment, as described in this blog post.

Triggering the Pipeline

After the pipeline has been created, choose the Release Change button and watch the pipeline build the S3ArtifactManager component.

If you are looking for a more hands-on experience in writing security tests, we suggest that you extend the S3ArtifactManager API to allow clients to retrieve versioned objects from an S3 bucket (for example, getObject(String bucketName, String key, String versionId)) and write security tests for the new API. 

Final Remarks

In this last post of the series, we showed how to automate the building and testing of our S3ArtifactManager component by creating a pipeline using AWS CodePipeline, AWS CodeCommit, and Jenkins. As a result, any code changes pushed to the repo are now automatically verified by the pipeline and rejected if security tests fail.

We hope you found this series helpful. Feel free to leave your feedback in the comments.

Happy security testing!

This is part II of a blog post series in which we do a deep dive on automated security testing for AWS applications. In part I, we discussed how AWS Java developers can create security unit tests to verify the correctness of their AWS applications by testing individual units of code in isolation. In this post, we go one step further and show how developers can create integration tests that, unlike unit tests, interact with real software components and AWS resources. In part III of this series, we’ll walk through how to incorporate the provided security tests into a CI/CD pipeline (created in AWS CodePipeline) to enforce security verification when new code is pushed into the code repository. Even though we focus on security, the tests provided can be easily generalized to other domains.

In part I of this post, we introduced a simple S3 wrapper component built to illustrate the security tests discussed in this post series. The wrapper, represented by a Java class named S3ArtifactManager (full source code can be accessed here), uses AWS SDK for Java APIs to provide a more secure way to store objects in Amazon S3.

Here we illustrate the method upload() implemented as part of class S3ArtifactManager that can be used to securely upload objects to an S3 bucket. For details about this method, see part I of this series. In the next section, we’ll write integration tests for method upload().

public String upload(String s3Bucket, String s3Key, File file) 
   throws AmazonServiceException, AmazonClientException {
   if (!s3.doesBucketExist(s3Bucket)) {
      s3.createBucket(s3Bucket);
   }

   // enable bucket versioning
   SetBucketVersioningConfigurationRequest configRequest = 
      new SetBucketVersioningConfigurationRequest(s3Bucket, 
         new BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED));
   s3.setBucketVersioningConfiguration(configRequest);

   // enable server-side encryption (SSE-S3)
   PutObjectRequest request = new PutObjectRequest(s3Bucket, s3Key, file);
   ObjectMetadata objectMetadata = new ObjectMetadata();
   objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
   request.setMetadata(objectMetadata);

   // upload object to S3
   PutObjectResult putObjectResult = s3.putObject(request);

   return putObjectResult.getVersionId();
}

@Test
public void testUploadWillEnableVersioningOnExistingS3Bucket() 
   throws Exception {   
   
   // bucket must exist prior to uploading object for this test
   s3Client.createBucket(s3Bucket);
   
   // call object under test
   String versionId = s3ArtifactManager.upload(s3Bucket, s3Key, 
      createSampleFile(s3Key));
   
   // assert object's version exists in the bucket
   S3Object s3Object = s3Client.getObject(
      new GetObjectRequest(s3Bucket, s3Key));
   assertEquals("Uploaded S3 object's versionId does not 
      match expected value", versionId, 
         s3Object.getObjectMetadata().getVersionId());
 
   // assert bucket versioning is enabled
   BucketVersioningConfiguration bucketConfig = s3Client
      .getBucketVersioningConfiguration(s3Bucket);
   assertEquals(BucketVersioningConfiguration.ENABLED, 
      bucketConfig.getStatus());
}

@Test
public void testUploadAddsSSE_S3EncryptedObjectToBucket() 
   throws Exception {
   
   // call object under test
   s3ArtifactManager.upload(s3Bucket, s3Key, createSampleFile(s3Key));
   
   // verify uploaded object is encrypted (SSE-S3)
   ObjectMetadata s3ObjectMetadata = s3Client.getObjectMetadata(
      new GetObjectMetadataRequest(s3Bucket, s3Key));
   assertEquals("Object has not been encrypted using SSE-S3 
      (AES256 encryption algorithm)", AES256, 
         s3ObjectMetadata.getSSEAlgorithm());      
}

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

Concurrency config is parallel='none', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false

Running com.amazonaws.samples.s3.artifactmanager.integrationtests.S3ArtifactManagerIntegrationTest

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.802 sec

The adoption of DevOps practices allows organizations to be agile while deploying high-quality software to customers on a regular basis. The CI/CD pipeline is an important component of the DevOps model. It automates critical verification tasks, making fully automated software deployments possible. Security tests are critical to the CI/CD pipeline. These tests verify whether the code conforms to the security specifications. For example, a security unit test could be used to enforce that a given software component must use server-side encryption to upload objects to an Amazon S3 bucket. Similarly, a security integration test could be applied to verify that the same software component always enables S3 bucket versioning.

In this three-part blog post series, we’ll do a deep dive on automated security testing for AWS applications. In this first post, we’ll discuss how AWS Java developers can create security unit tests to verify the correctness of their AWS applications by testing individual units of code in isolation. In part II, we’ll go one step further and show how developers can create integration tests that, unlike unit tests, interact with real software components and AWS resources. Finally, in part III, we’ll walk through how the provided security tests can be incorporated into a CI/CD pipeline (created through AWS CodePipeline) to enforce security verification whenever new code changes are pushed into the code repository. Even though we focus on security, the tests provided can be easily generalized to other domains.

We start by introducing a simple S3 wrapper component built to illustrate the security tests discussed in this series. The wrapper, represented by a Java class named S3ArtifactManager (full source code can be accessed here), uses AWS SDK for Java APIs to provide a more secure way to store objects in S3.

Here we show an excerpt of class S3ArtifactManager that describes a method called upload() that can be used to securely upload objects to an S3 bucket. The method uses S3 bucket versioning to make sure  each new upload of the same object will preserve all previous versions of that object. A versionId is returned to clients each time an object (or a new version of it) is stored in the bucket so that specific versions can be retrieved later. Versioning is enabled by using a SetBucketVersioningConfigurationRequest object that takes a BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED) instance as parameter and by calling s3.setBucketVersioningConfiguration() passing the request object (lines 8-11). 

In addition, method upload() uses the server-side encryption with Amazon S3-managed encryption keys (SSE-S3) feature to enforce that objects stored in the bucket are encrypted. We simply create a metadata object setting – objectMetadata.setSSEAlgorithm() – as the encryption algorithm and attach the metadata object to the PutObjectRequest instance used to store the S3 object (lines 14-17). In line 20, the object is uploaded to S3 and its versionId is returned to client in line 22. 

public String upload(String s3Bucket, String s3Key, File file) 
   throws AmazonServiceException, AmazonClientException {
   if (!s3.doesBucketExist(s3Bucket)) {
      s3.createBucket(s3Bucket);
   }

   // enable bucket versioning
   SetBucketVersioningConfigurationRequest configRequest = 
      new SetBucketVersioningConfigurationRequest(s3Bucket, 
         new BucketVersioningConfiguration(BucketVersioningConfiguration.ENABLED));
   s3.setBucketVersioningConfiguration(configRequest);

   // enable server-side encryption (SSE-S3)
   PutObjectRequest request = new PutObjectRequest(s3Bucket, s3Key, file);
   ObjectMetadata objectMetadata = new ObjectMetadata();
   objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
   request.setMetadata(objectMetadata);

   // upload object to S3
   PutObjectResult putObjectResult = s3.putObject(request);

   return putObjectResult.getVersionId();
}

Because security is key in the cloud, security components like our S3ArtifactManager might interest individuals and organizations responsible for meeting security compliance requirements (for example, PCI). In this context, developers and other users of such components must be confident that the security functionality provided behaves as expected. A bug in the component (for example, an object that is stored unencrypted or overwrites a previous version) can be disastrous. In addition, users must remain confident as new versions of the component are released. How can confidence be achieved continuously?

It turns out that DevOps practices improve confidence. In a traditional software development approach, coding the logic of the method upload() and running a few manuals test might be enough, but in a DevOps setting, this  is not acceptable. DevOps practices require that mechanisms that automatically verify code behavior are in place. In fact, these mechanisms are just as important as the code’s main logic. Which mechanisms are we talking about? Unit and integration tests!

In the next section, we’ll discuss how unit tests can be leveraged to verify the security behavior of our S3ArtifactManager wrapper. In parts II and III of this series, we’ll dive deep into integration tests and CI/CD automation, respectively.

Next, we’ll create a suite of security unit tests to verify the behavior of our upload() method. We’ll leverage two popular test frameworks in Java named JUnit and Mockito to code the unit tests.

The primary purpose of unit tests is to test a unit of code in isolation. Here we define unit as the Java class under test (in our case, the S3ArtifactManager class). In order to isolate the class under test, we mock all other objects used in the class, such as the S3 client object. Mocking means that our unit tests will not interact with a real S3 resource and will not upload objects into a S3 bucket. Instead, we’re using a mock object with predefined behavior.

Verifying bucket versioning enablement on S3 buckets

The first security unit test is named testUploadWillEnableVersioningOnExistingS3Bucket. It verifies whether method upload() enables bucket versioning on an existing bucket upon uploading an object to that bucket. Note that we are using a mock object in Mockito instead of a real object to represent an S3 client instance. For this reason, we need to specify the behavior of the mock object for the functionality used by method upload(). In line 5, we use Mockito’s when statement to return true when s3Client.doesBucketExist() is called because this is the condition we want to test. In line 8, method upload() is called using test values for S3 bucket, key, and file parameters. 

@Test
public void testUploadWillEnableVersioningOnExistingS3Bucket() {
   
   // set Mock behavior
   when(s3Client.doesBucketExist(s3Bucket)).thenReturn(true); 
   
   // call object under test
   String versionId = s3ArtifactManager.upload(s3Bucket, s3Key, file);
   
   // assert versionID is the expected value
   assertEquals("VersionId returned is incorrect", 
      VERSION_ID, versionId);
   
   // assert that a new bucket has NOT been created
   verify(s3Client, never()).createBucket(s3Bucket);
   
   // capture BucketVersioningConfigurationReques object 
   ArgumentCaptor<SetBucketVersioningConfigurationRequest> 
      bucketVerConfigRequestCaptor = ArgumentCaptor.forClass(
         SetBucketVersioningConfigurationRequest.class);
   verify(s3Client).setBucketVersioningConfiguration(
      bucketVerConfigRequestCaptor.capture());
   
   // assert versioning is set on the bucket
   SetBucketVersioningConfigurationRequest bucketVerConfigRequest = 
      bucketVerConfigRequestCaptor.getValue();
   assertEquals("Versioning of S3 bucket could not be 
      verified", BucketVersioningConfiguration.ENABLED, 
         bucketVerConfigRequest.getVersioningConfiguration()
            .getStatus());
}

The first verification we perform is shown in lines 11-12. It verifies that the versionId value returned matches the constant value expected in the test. In line 15, we verify that a call to s3Client.createBucket() has never been made because the bucket already exists (as mocked using when in line 5). These are standard verifications, not related to security.

In line 18, we start verifying security behavior. We use Mockito’s argument captor feature to capture the parameter passed to setBucketVersioningConfiguration, which is a real object (lines 18-22). Later, in lines 25-30, we check whether bucket versioning is enabled in that object by comparing the value captured with constant BucketVersioningConfiguration.ENABLED. If this security verification fails, it means that versioning was not correctly configured. In this scenario, because a critical security assertion could not be verified, the CI/CD pipeline should be blocked until the code is fixed.

We also created a security unit test to verify bucket versioning enablement for newly created buckets. We’ve omitted the code for brevity, but you can download full source here. This test is similar to the one we just discussed. The main differences are in line 5 (which now returns false) and line 15 which verifies that the createBucket API was called once (verify(s3Client, times(1)).createBucket(s3Bucket)).

Verifying server-side-encryption of uploaded S3 objects

The second security unit test verifies that uploaded S3 objects use server-side-encryption with Amazon S3 encryption keys (SSE-S3). In lines 8-12, the security unit test verification starts by once again using Mockito’s argument captor to capture the request object passed to s3Client.putObject(). This object is used later in two ways: first, in lines 16-18, to verify that no customer keys were provided (because upload() is expected to use SSE-S3) and then in lines 22-26 to assert that the object’s metadata is not null and returns AES256 as the encryption algorithm, the value expected for SSE-S3 encryption. Once again, if this security verification fails, the CI/CD pipeline should be blocked until SSE-S3 code implementation is fixed and verified.

@Test
public void TestUploadAddsSSE_S3EncryptedObjectToBucket() {
   
   // call object under test
   s3ArtifactManager.upload(s3Bucket, s3Key, file);
   
   // capture putObjectRequest object
   ArgumentCaptor<PutObjectRequest> putObjectRequestCaptor = 
      ArgumentCaptor.forClass(PutObjectRequest.class);
   verify(s3Client).putObject(putObjectRequestCaptor.capture());
   PutObjectRequest putObjectRequest = 
      putObjectRequestCaptor.getValue();
   
   // assert that there's no customer key provided as 
   // we're expecting SSE-S3
   assertNull("A customer key was incorrectly used (SSE-C). 
      SSE-S3 encryption expected instead.", 
         putObjectRequest.getSSECustomerKey());
   
   // assert that the SSE-S3 'AES256' algorithm was set as part of 
   // the request's metadata 
   assertNotNull("PutObjectRequest's metadata object must be non-null 
      and enforce SSE-S3 encryption", putObjectRequest.getMetadata());
   assertEquals("Object has not been encrypted using SSE-S3 (AES256 
      encryption algorithm)", AES256, putObjectRequest.getMetadata()
         .getSSEAlgorithm());
}

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

Running com.amazonaws.samples.s3.artifactmanager.unittests.S3ArtifactManagerUnitTest

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.155 sec

Exception handling in the AWS SDK for Java just got a little easier! We just introduced base exception classes for all AWS services in the SDK. Now all modeled exceptions extend from this service-specific base exception and all unmodeled exceptions (unknown exceptions thrown by the service) will be unmarshalled into the service-specific base exception, and not into the generic AmazonServiceException. The new service base exceptions still extend AmazonServiceException, so existing exception handling code you’ve written won’t be affected. This change gives you greater control to handle exceptions differently, depending on the originating service.

Writing robust applications is challenging, and distributed systems occasionally fail with transient (often retriable) errors. Although the SDK retries certain errors by default, sometimes these exceptions bubble up to the caller’s code and must be handled appropriately. Previously, when you used the SDK, you only had the option to catch a service-specific modeled exception (i.e., ResourceInUseException for Amazon DynamoDB) or a generic AmazonServiceException, which could be thrown by any service. When you wrote code that involved multiple service dependencies, you had to write error handling code for each service independently.

Consider the following example where you read messages from an Amazon SQS queue and write each message to a DynamoDB table. The contrived requirements are to treat SQS errors as transient and retry them after an appropriate backoff time, and to treat all DynamoDB errors (except for ProvisionedThroughputExceededException) as fatal and persist the messages to local disk for future processing. ProvisionedThroughputExceededException should be treated as retriable and the program should back off before attempting to process another message. Finally, if any other non-service exceptions are thrown, the program should terminate. For simplicity, we’ll assume message processing is idempotent and it’s acceptable to process a message multiple times.

Previously, this was difficult to do in the SDK. Your options included catching service specific modeled exceptions (like ProvisionedThroughputExceededException) or catching the generic AmazonServiceException which applies to all AWS services. This makes handling errors differently per service challenging. You might have written code like this to meet the example’s requirements.

while (true) {
    ReceiveMessageResult currentResult = null;
    try {
        // Receive messages from the queue
        currentResult = sqs.receiveMessage(QUEUE_URL);
        for (Message message : currentResult.getMessages()) {
            // Save message to DynamoDB
            ddb.putItem(new PutItemRequest().withTableName(TABLE_NAME).withItem(
                 ImmutableMapParameter
                   .of("messageId", new AttributeValue(message.getMessageId()),
                       "messageBody", new AttributeValue(message.getBody()))));
            // Delete message from queue to indicate it's been processed
            sqs.deleteMessage(new DeleteMessageRequest()
                                    .withQueueUrl(QUEUE_URL)
                                    .withReceiptHandle(message.getReceiptHandle()));
        }
    } catch (ProvisionedThroughputExceededException e) {
        LOG.warn("Table capacity exceeding. Backing off and retrying.");
        Thread.sleep(1000);
    } catch (AmazonServiceException e) {
        switch (e.getServiceName()) {
            // Have to use magic service name constants that aren't publicly exposed
            case "AmazonSQS":
                LOG.warn("SQS temporarily unavailable. Backing off and retrying.");
                Thread.sleep(1000);
                break;
            case "AmazonDynamoDBv2":
                LOG.fatal("Could not save messages to DynamoDB. Saving to disk.");
                saveToFile(currentResult.getMessages());
                break;
        }
    } catch (AmazonClientException e) {
        // Caught unexpected error
        System.exit(1);
    }
}

You can rewrite the example above much more cleanly by using the new exception base classes in the SDK, as shown here.
 

while (true) {
    ReceiveMessageResult currentResult = null;
    try {
        // Receive messages from the queue
        currentResult = sqs.receiveMessage(QUEUE_URL);
        for (Message message : currentResult.getMessages()) {
            // Save message to DynamoDB
            ddb.putItem(new PutItemRequest().withTableName(TABLE_NAME).withItem(
                 ImmutableMapParameter
                   .of("messageId", new AttributeValue(message.getMessageId()),
                       "messageBody", new AttributeValue(message.getBody()))));
            // Delete message from queue to indicate it's been processed
            sqs.deleteMessage(new DeleteMessageRequest()
                                    .withQueueUrl(QUEUE_URL)
                                    .withReceiptHandle(message.getReceiptHandle()));
        }
    } catch (AmazonSQSException e) {
        LOG.warn("SQS temporarily unavailable. Backing off and retrying.");
        Thread.sleep(1000);
    } catch (ProvisionedThroughputExceededException e) {
        // You can catch modeled exceptions
        LOG.warn("Table capacity exceeding. Backing off and retrying.");
        Thread.sleep(1000);
    } catch (AmazonDynamoDBException e) {
        // Or you can catch the service base exception to handle any exception that
        // can be thrown by a particular service
        LOG.fatal("Could not save messages to DynamoDB. Saving to file.");
        saveToFile(currentResult.getMessages());
    } catch (AmazonClientException e) {
        // Caught unexpected error
        System.exit(1);
    }
}

We hope this addition to the SDK makes writing robust applications even easier! Let us know what you think in the comments section below.

We are pleased to announce a better, more intuitive way to construct and configure service clients in the AWS SDK for Java. Previously, the only way to construct a service client was through one of the many overloaded constructors in the client class. Finding the right constructor was difficult and sometimes required duplicating the default configuration to supply a custom dependency. In addition, the constructors of the client don’t expose the full set of options that can be configured through the client. You must provide a region and custom request handlers through a setter on the client after construction. This approach isn’t intuitive, and it promotes unsafe practices when using a client in a multithreaded environment.

New Client Builders

To address the shortcomings of the current approach, we’ve introduced a new builder style to create a client in a fluent and easy-to-read way. The builder approach makes the set of available configurations more discoverable. You can use fluent setters for each option instead of relying on positional parameters in an overloaded constructor. The fluent setters allow for more readable code by using method chaining. The builder pattern also allows for overriding only the options you care about. If you just want a custom RequestMetricCollector, then that’s all you have to set and you can still benefit from the defaults of all the other dependencies. After a client is created with the builder, it is immutable to enforce thread safety when using the client in a multithreaded environment.

Let’s compare the current approach and the new builder approach.

Current Approach

/**
 * Just to provide a custom metric collector I have to supply the defaults
 * of AWSCredentialsProvider and ClientConfiguration. Also note that 
 * I've gotten the default ClientConfiguration for Dynamo wrong, 
 * it's actually got a service specific default config.
**/
AmazonDynamoDBClient client = new AmazonDynamoDBClient(
                                    new DefaultAWSCredentialsProviderChain(), 
                                    new ClientConfiguration(),
                                    new MyCustomRequestMetricsCollector());
/**
 * I have to set the region after which could cause issues in a multi 
 * threaded application if you aren't safely publishing the reference.
**/
client.configureRegion(Regions.US_EAST_1);

Builder Approach

/**
 * With the new builder I only have to supply what I want to customize
 * and all options are set before construction of the client
**/
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
                          .withMetricsCollector(new MyCustomRequestMetricsCollector())
                          .withRegion(Regions.US_EAST_1)
                          .build();

Region Configuration Improvements

In the AWS SDK for Java, we strongly recommend setting a region for every client created. If you create a client through the constructor, it initializes the client with a default endpoint (typically, us-east-1 or a global endpoint). For this reason, many customers don’t bother setting a region because it just works with the default endpoint. When these customers attempt to deploy their application to another region, a lot of code has to be changed to configure the client with the correct region based on the environment. With the builder, we enforce setting a region to promote best practices to make it easier to write applications that can be deployed to multiple regions. When you create clients through the builder, you can provide a region explicitly with the fluent setters or implicitly through the new region provider chain introduced.

The region provider chain is similar to the credentials provider chain. If no region is explicitly provided, the SDK consults the region provider chain to find a region to use from the environment. The SDK will first check if the AWS_REGION environment variable is set. If not, it will look in the AWS Shared Config file (usually located at ~/.aws/config) and use the region in the default profile (unless the AWS_PROFILE environment variable is set, in which case the SDK will look at that profile). Finally, if the SDK still hasn’t found a region, it will attempt to find one from the EC2 metadata service for applications that run on AWS infrastructure. If a region is not explicitly provided or if the chain cannot determine the region, able to be determined by the chain then the builder will not allow the client to be created.

// Creates a client with an explicit region that's in the Regions enum
AmazonDynamoDBClientBuilder.standard()
            .withRegion(Regions.US_WEST_2)
            .build();
/**
 * Creates a client with an explicit region string. Useful when a new
 * region isn't present in the enum and upgrading to a newer 
 * version of the SDK isn't feasible.
**/
AmazonDynamoDBClientBuilder.standard()
            .withRegion("ap-south-1")
            .build();
/** 
 * Creates a client with the default credentials provider chain and a 
 * region found from the new region provider chain.
**/
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.defaultClient();

Async Client Builder

The SDK offers async variants of most clients. Although the async client duplicates many of the constructors of the sync client, they’ve often gotten out of sync. For example, the async clients do not expose a constructor that takes a RequestMetricsCollector. The new builder remedies that. Going forward, it will allow us to keep the customizable options for both variants of the client in sync. Because the async client has additional dependencies, it is a separate builder class.

/**
 * For ExecutorService we've introduced a factory to supply new 
 * executors for each async client created as it's completely 
 * possible to use a builder instance to create multiple clients and
 * you'll rarely want to share a common executor between them. 
 * The factory interface is a functional interface so it integrates 
 * fully with Java8's lambda expressions and method references.
 */
final AmazonDynamoDBAsync client = AmazonDynamoDBAsyncClientBuilder
         .standard()
         .withExecutorFactory(() -> Executors.newFixedThreadPool(10))
         .withCredentials(new ProfileCredentialsProvider("my-profile"))
         .build();

S3 and TransferManager

S3 is a little different from most clients in the SDK. It has special configuration options (previously in S3ClientOptions) that apply only to S3. It also doesn’t have a typical async client. Instead, it provides the TransferManager utility for asynchronous uploads and downloads. Both the S3 client and TransferManager have their own builders. Note that the options in S3ClientOptions have been collapsed into the builder interface.

// Create a client with accelerate mode enabled
final AmazonS3 client = AmazonS3ClientBuilder.standard()
        .enableAccelerateMode()
        .withRegion("us-east-1").build();

The TransferManager is similar to other async client builders. Options that were in the TransferManagerConfiguration class have been collapsed into the builder interface.

/** 
 * The TransferManager requires the low level client so it must be
 * provided via AmazonS3ClientBuilder
**/
TransferManager tf = TransferManagerBuilder.standard()
        .withExecutorFactory(() -> Executors.newFixedThreadPool(100))
        .withMinimumUploadPartSize(UPLOAD_PART_SIZE)
        .withS3Client(AmazonS3ClientBuilder.defaultClient())
        .build();

// The following factory method is provided for convenience
TransferManagerBuilder.defaultTransferManager();
// The above is equivalent to
TransferManagerBuilder.standard().withS3Client(AmazonS3ClientBuilder.defaultClient()).build();

Fast Facts

• Each service client has two dedicated builder classes for its sync and async variants
• Builder class names are based on the interface they create (for example, the builders for the AmazonDynamoDB and AmazonDynamoDBAsync interfaces are AmazonDynamoDBClientBuilder and AmazonDynamoDBAsyncClientBuilder, respectively).
• Builder instances can be obtained through the static factory method `standard()`.
• A convenience factory method, defaultClient, initializes the client with the default credential and region provider chains.
• Builders can be used to create multiple clients.
• The async builder has all of the same options as the sync client builder plus an additional executor factory option.
• Region must be provided either explicitly or implicitly (through the region provider chain) before creating a client.
• Clients built through the builder are immutable.
• S3 and TransferManager have custom builders with additional configuration options.

We are excited about these new additions to the SDK! Feel free to leave your feedback in the comments. To use the new builders, declare a dependency on the 1.11.18 version of the SDK.