AWS SDK for Java Tips and Tricks
Using the SDK, you can build solutions for Amazon Simple Storage Service (Amazon S3), Amazon Elastic Compute Cloud (Amazon EC2), Amazon SimpleDB, and more. The AWS SDK for Java includes the AWS Java library, code samples, and reference documentation, but what about information on how to use the Java library? This article addresses some important tips for developing software with the AWS SDK for Java. The primary areas we'll address are:
- Client configuration
- Logging control
- Exception handling
See our Java Development forum if you have additional questions or comments about developing applications on the AWS SDK for Java.
Client Configuration
The AWS SDK for Java allows you to change the default client configuration, which is helpful when you want to:
- Change the region endpoint
- Connect to the Internet through proxy
- Tweak HTTP transport settings, such as connection timeout and request retries.
- Specify TCP socket buffer size hints
AWS Region Selection
Regions allow you to access AWS services that physically reside in a specific geographic region. This can be useful both for redundancy and to keep your data and applications running close to where you and your users will access them.
Each client can be pointed to a specific endpoint through the setEndpoint(String endpointUrl) method.
For example, to point the AWS EC2 Java client at the EU West region, use the following code:
AmazonEC2 ec2 = new AmazonEC2(myCredentials);
ec2.setEndpoint("https://eu-west-1.ec2.amazonaws.com");
Be aware that regions are logically isolated from each other, so for example, you won't be able to access US East resources when communicating with the EU West endpoint.
See Available Region Endpoints for the AWS SDKs for a list of valid endpoints.
Proxy Configuration
When constructing a client object, you can pass in an optional com.amazonaws.ClientConfiguration object to customize the client's configuration.
If you're connecting to the Internet through a proxy server, you'll need to configure your proxy server settings (proxy host, port and username/password) through the ClientConfiguration object.
HTTP Transport Configuration
Several HTTP transport options can be configured through the com.amazonaws.ClientConfiguration object. Default values will suffice for the majority of users, but users who want more control can configure:
- Socket timeout
- Connection timeout
- Maximum retry attempts for retry-able errors
- Maximum open HTTP connections
TCP Socket Buffer Size Hints
Advanced users who want to tune low-level TCP parameters can additionally set TCP buffer size hints through the ClientConfiguration object. The majority of users will never need to tweak these values, but they are provided for advanced users.
Optimal TCP buffer sizes for an application are highly dependent on network and OS configuration and capabilities. For example, most modern operating systems provide auto-tuning logic for TCP buffer sizes, which can have a big impact on performance for TCP connections that are held open long enough for the auto-tuning to optimize buffer sizes.
Large buffer sizes (ex: 2MB) allow the OS to buffer more data in memory without requiring the remote server to acknowledge receipt of that information, so can be particularly useful when the network has high latency.
This is only a hint, and the OS may choose not to honor it. When using this option, users should always check the operating system's configured limits and defaults. Most OS's have a maximum TCP buffer size limit configured, and won't let you go beyond that limit unless you explicitly raise the max TCP buffer size limit.
Many resources available to help with configuring TCP buffer sizes and operating system specific TCP settings, including:
Logging Control
The AWS SDK for Java uses Apache Commons Logging to log diagnostic messages. Apache Commons Logging provides a generic interface to logging libraries so that the user can plug in any compatible logging library. AWS recommends using Apache log4j as the logging library underneath Commons Logging and the configuration examples below are all log4j.properties files.
Service-Specific Errors and Warnings
AWS recommends that you always leave the "com.amazonaws" logger hierarchy set to "WARN" in order to catch any important messages from the client libraries. For example, if the Amazon S3 client detects that your application hasn't properly closed an InputStream and could be leaking resources, it will report it through a warning message to the logs. This will also ensure that messages are logged if the client has any problems handling requests or responses.
The following log4j.properties file sets the rootLogger to WARN, which will cause warning and error messages from all loggers in the "com.amazonaws" hierarchy to be included. Alternatively, you can explicitly set the com.amazonaws logger to WARN.
log4j.rootLogger=WARN, A1 log4j.appender.A1=org.apache.log4j.ConsoleAppender log4j.appender.A1.layout=org.apache.log4j.PatternLayout log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n # Or you can explicitly enable WARN and ERROR messages for the AWS Java clients log4j.logger.com.amazonaws=WARN
Request/Response Summary Logging
Every request to an AWS service generates a unique AWS request ID that is useful if you run into an issue with how an AWS service is handling a request. AWS request IDs are accessible programmatically through Exception objects in the SDK for any failed service call, and can also be reported through the INFO log level in the "com.amazonaws.request" logger.
The following log4j.properties file enables a summary of requests and responses, including AWS request IDs.
log4j.rootLogger=WARN, A1 log4j.appender.A1=org.apache.log4j.ConsoleAppender log4j.appender.A1.layout=org.apache.log4j.PatternLayout log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n # Turn on INFO logging in com.amazonaws.request to log # a summary of requests/responses with AWS request IDs log4j.logger.com.amazonaws.request=INFO
Here is an example of the log output:
2009-12-17 09:53:04,269 [main] INFO com.amazonaws.request - Sending Request: POST https://rds.amazonaws.com / Parameters: (MaxRecords: 20, Action: DescribeEngineDefaultParameters, SignatureMethod: HmacSHA256, AWSAccessKeyId: AKIAJUHAG74EVMS3UVZA, Version: 2009-10-16, SignatureVersion: 2, Engine: mysql5.1, Timestamp: 2009-12-17T17:53:04.267Z, Signature: 4ydexGGkC77PovHhbfzAMA1H0nDnqIQxG9q+Yq3uw5s=, ) 2009-12-17 09:53:04,464 [main] INFO com.amazonaws.request - Received successful response: 200, AWS Request ID: 06c12a39-eb35-11de-ae07-adb69edbb1e4 2009-12-17 09:53:04,469 [main] INFO com.amazonaws.request - Sending Request: POST https://rds.amazonaws.com / Parameters: (ResetAllParameters: true, Action: ResetDBParameterGroup, SignatureMethod: HmacSHA256, DBParameterGroupName: java-integ-test-param-group-1261072381023, AWSAccessKeyId: AKIAJUHAG74EVMS3UVZA, Version: 2009-10-16, SignatureVersion: 2, Timestamp: 2009-12-17T17:53:04.467Z, Signature: 9WcgfPwTobvLVcpyhbrdN7P7l3uH0oviYQ4yZ+TQjsQ=, ) 2009-12-17 09:53:04,646 [main] INFO com.amazonaws.request - Received successful response: 200, AWS Request ID: 06e071cb-eb35-11de-81f7-01604e1b25ff
Verbose Wire Logging
In some cases, it may be useful to see the exact requests and responses being sent and received by the AWS SDK for Java. This logging should not be enabled in production systems since writing out large requests (ex: a file being uploaded to Amazon S3) or responses can significantly slow down an application. If you really need access to this information, you can temporarily enable it through Apache HttpClient's logger. Enabling the DEBUG level on the "httpclient.wire" logger enables logging for all request and response data.
The following log4j.properties file turns on full wire logging in Apache HttpClient and should only be turned on temporarily since it can have a significant performance impact on your application.
log4j.rootLogger=WARN, A1 log4j.appender.A1=org.apache.log4j.ConsoleAppender log4j.appender.A1.layout=org.apache.log4j.PatternLayout log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n # Log all HTTP content (headers, parameters, content, etc) for # all requests and responses. Use caution with this since it can # be very expensive to log such verbose data! log4j.logger.httpclient.wire=DEBUG
Exception Handling
Understanding how and when the AWS SDK for Java throws exceptions is important in order to build high quality applications using the SDK. The following sections describe the different cases of exceptions that are thrown by the SDK and how to handle them appropriately.
Why Unchecked Exceptions?
The AWS Java SDK uses runtime (or unchecked) exceptions instead of checked exceptions for a few reasons:
- To allow developers fine grained control over the errors they want to handle without forcing them to handle exceptional cases they aren't concerned about (and making their code overly verbose)
- To prevent scalability issues inherent with checked exceptions in large applications
In general, checked exceptions work well on small scales, but can become troublesome as applications grow and become more complex.
For more information on the ongoing discussion in the Java community around the use of checked and unchecked exceptions, see the following references:
- http://www.artima.com/intv/handcuffs2.html
- http://www.mindview.net/Etc/Discussions/CheckedExceptions
- http://radio.weblogs.com/0122027/stories/2003/04/01/JavasCheckedExceptionsWereAMistake.html
AmazonServiceException (and Subclasses)
This is the main type of exception that you'll need to deal with when using the AWS SDK for Java. This exception represents an error response from an AWS service. For example, if you request to terminate an Amazon EC2 instance that doesn't exist, EC2 will return an error response and all the details of that error response will be included in the thrown AmazonServiceException. For some cases, a subclass of AmazonServiceException will be thrown to allow developers fine grained control over handling error cases through catch blocks.
When you encounter an AmazonServiceException, you know that your request was successfully sent to the AWS service, but could not be successfully processed either because of errors in the request's parameters or because of issues on the service side.
AmazonServiceException has several useful fields in it, including:
- The returned HTTP status code
- The returned AWS error code
- The detailed error message from the service
- The AWS request ID for the failed request
AmazonServiceException also includes a field that describes whether the failed request was the caller's fault (i.e. a request with illegal values) or the AWS service's fault (i.e. an internal service error).
AmazonClientException
This exception indicates that a problem occurred inside the Java client code, either while trying to send a request to AWS or while trying to parse a response from AWS. AmazonClientExceptions are more severe than AmazonServiceExceptions and indicate a major problem that is preventing the client from being able to make service calls to AWS services. For example, the AWS Java SDK will throw an AmazonClientException if no network connection is available when you try to call an operation on one of the clients.
IllegalArgumentException
When calling operations if you pass an illegal argument, the AWS SDK for Java will throw an IllegalArgumentException. For example, if you call an operation and pass a null value in for one of the required parameters, the SDK will throw an IllegalArgumentException describing the illegal argument.
Using Access Control Policies
AWS access control policies allow you to specify fine grained access controls on your AWS resources. You can allow or deny access to your AWS resources based on:
- what '''resource''' is being accessed
- who is accessing the resource (i.e. the principal)
- what action is being taken on the resource
- a variety of conditions including date restrictions, IP address restrictions, etc.
Access control policies are a collection of statements. Each statement takes the form: "A has permission to do B to C where D applies".
- A is the principal - the AWS account that is making a request to access or modify one of your AWS resources.
- B is the action - the way in which your AWS resource is being accessed or modified, such as sending a message to an Amazon SQS queue, or storing an object in an Amazon S3 bucket.
- C is the resource - your AWS entity that the principal wants to access, such as an Amazon SQS queue, or an object stored in Amazon S3.
- D is the set of conditions - optional constraints that specify when to allow or deny access for the principal to access your resource. Many expressive conditions are available, some specific to each service. For example you can use date conditions to allow access to your resources only after or before a specific time.
Amazon S3 Example
The following example demonstrates a policy that allows anyone access to read all the objects in a bucket, but restricts access to uploading objects to that bucket to two specific AWS accounts (in addition to the bucket owner's account).
Statement allowPublicReadStatement = new Statement(Effect.Allow)
.withPrincipals(Principal.AllUsers)
.withActions(S3Actions.GetObject)
.withResources(new S3ObjectResource(myBucketName, "*"));
Statement allowRestrictedWriteStatement = new Statement(Effect.Allow)
.withPrincipals(new Principal("123456789"), new Principal("876543210"))
.withActions(S3Actions.PutObject)
.withResources(new S3ObjectResource(myBucketName, "*"));
Policy policy = new Policy()
.withStatements(allowPublicReadStatement, allowRestrictedWriteStatement);
AmazonS3 s3 = new AmazonS3Client(myAwsCredentials);
s3.setBucketPolicy(myBucketName, policy.toJson());
Amazon SQS Example
One common use of policies is to authorize an Amazon SQS queue to receive messages from an Amazon SNS topic.
/*
* This policy allows an SNS topic to send messages to an SQS queue.
* You can find your SNS topic's ARN through the SNS getTopicAttributes operation.
*/
Policy policy = new Policy().withStatements(
new Statement(Effect.Allow)
.withPrincipals(Principal.AllUsers)
.withActions(SQSActions.SendMessage)
.withConditions(ConditionFactory.newSourceArnCondition(myTopicArn)));
Map queueAttributes = new HashMap();
queueAttributes.put(QueueAttributeName.Policy.toString(), policy.toJson());
AmazonSQS sqs = new AmazonSQSClient(myAwsCredentials);
sqs.setQueueAttributes(new SetQueueAttributesRequest(myQueueUrl, queueAttributes));
Amazon SNS Example
Some services offer additional conditions that can be used in policies. Amazon SNS provides conditions for allowing or denying subscriptions to SNS topics based on the protocol ''(ex: email, HTTP, HTTPS, SQS)'' and endpoint ''(ex: email address, URL, SQS ARN)'' of the request to subscribe to a topic.
/*
* This SNS condition allows you to restrict subscriptions to an Amazon SNS topic
* based on the requested endpoint (email address, SQS queue ARN, etc) used when
* someone tries to subscribe to your SNS topic.
*/
Condition endpointCondition =
SNSConditionFactory.newEndpointCondition("*@mycompany.com");
Policy policy = new Policy().withStatements(
new Statement(Effect.Allow)
.withPrincipals(Principal.AllUsers)
.withActions(SNSActions.Subscribe)
.withConditions(endpointCondition));
AmazonSNS sns = new AmazonSNSClient(myAwsCredentials);
sns.setTopicAttributes(
new SetTopicAttributesRequest(myTopicArn, "Policy", policy.toJson()));
Additional Resources
Some additional resources to help you build your application include:
