Category: Ruby

Recently we announced the modularization of the AWS SDK for Ruby. This blog post will focus on how to upgrade your application to use the new service specific gems.

This blog post is divided up into sections based on how you currently depend on the AWS SDK for Ruby today. Find the section below that describes how you load the SDK today, and it will guide you in upgrading. Since version 3 is backwards compatible with version 2, you should not need to make additional changes beyond those described below.

Bunder: gem ‘aws-sdk’, ‘~>2’

Congratulations! You are following recommended best practices for how to depend on the SDK today. The simplest path to upgrade is to change the version from 2 to 3.

#gem 'aws-sdk', '~> 2'
gem 'aws-sdk', '~> 3'

See the section about using service specific gems below.

Bundler: gem ‘aws-sdk’ (without version)

It is not recommended to depend on the SDK without a major version constraint. Fortunately, version 3 is backwards compatible with version 2. Bundle updating your dependencies will work, but consider yourself lucky! You should add the version constraint to protect from future major version changes:

#gem 'aws-sdk'
gem 'aws-sdk', '~> 3'

See the section about using service specific gems below.

Bundler: gem ‘aws-sdk-core’, ‘~> 2’

The aws-sdk-core gem changes signification from version 2 to 3. In version 3, the core gem no longer defines any services. It will only contain shared utilities, such as credential providers, serializers, etc. To upgrade you must make two changes:

• Change your bundler dependency
• Change your ruby require statements

In bundler, replace aws-sdk-core, with aws-sdk:

#gem 'aws-sdk-core', '~> 2'
gem 'aws-sdk', '~> 3'

In code, replace any require statements on aws-sdk-core with aws-sdk.

#require 'aws-sdk-core'
require 'aws-sdk'

See the section about using service specific gems below.

Bundler: gem ‘aws-sdk-core’ (without version)

If you happen to bundle update before changing your Gemfile, your application will be broken. Version 3 of the aws-sdk-core gem no longer defines service modules and clients. It is a shared dependency of the 75+ service gems. To upgrade you must make two changes:

• Change your bundler dependency
• Change your ruby require statements

In bundler, replace aws-sdk-core, with aws-sdk:

#gem 'aws-sdk-core'
gem 'aws-sdk', '~> 3'

In code, replace any require statements on aws-sdk-core with aws-sdk.

#require 'aws-sdk-core'
require 'aws-sdk'

See the section about using service specific gems below.

Bundler: gem ‘aws-sdk-resource’ (with or without version)

In version 3, the aws-sdk-resources gem has been removed. This gem will not receive any further updates. Each service gem contains both the client interface, and the resource interfaces. To upgrade you must make two changes:

• Change your bundler dependency
• Change your ruby require statements

#gem 'aws-sdk-resources', '~> 2'
gem 'aws-sdk', '~> 3'

In code, replace any require statements on aws-sdk-resources with aws-sdk.

#require 'aws-sdk-core'
require 'aws-sdk'

See the section about using service specific gems below.

Using the Service Specific Gems

Each of the instructions above suggested using version 3 of the aws-sdk gem. This will work and is the shortest path to upgrading. It will however install 75+ service specific gems. You may choose to replace your dependency on the aws-sdk gem with service specific gems.

If my application depends on Amazon DynamoDB and Amazon S3, I could make the following changes:

In my Gemfile:

#gem 'aws-sdk', '~> 3'
gem 'aws-sdk-dynamodb', '~> 1'
gem 'aws-sdk-s3', '~> 1'

In my code:

#require 'aws-sdk'
require 'aws-sdk-s3'
require 'aws-sdk-dynamodb'

If you are a library maintainer, and you depend on the AWS SDK for Ruby, you should use service specific gems. Do no force your users to install every AWS service gem if you only depend on one.

Conclusion

Upgrading should be very simple. If you encounter any backwards incompatible changes, please open a GitHub issue. The modularized SDK will be in preview for a short period to hopefully catch these issues before going GA. You can also catch us in the gitter channel.

Version 3 of the AWS SDK for Ruby is available now as a preview release. This version modularizes the monolithic SDK into service specific gems. Aside from gem packaging differences, version 3 interfaces are backwards compatible with version 2.

You can install individual gems like so:

$ gem install aws-sdk-s3 --version 1.0.0.rc1

You can install everything using the aws-sdk gem:

$ gem install aws-sdk --version 3.0.0.rc1

To see a complete list of gems, checkout the project README for a list of supported services and gems.

Motivation

Modularization allows us to make some long requested changes to the SDK. Some of these changes were not reasonable when we shipped a single monolithic gem with 75+ services. Some of the primary motivating factors for breaking up the monolith include:

• To provide better versioning information. When 75+ services share a single gem, it is difficult to communicate when a change is meaningful to a user. We can now use strict semantic versioning for each gem.
• To improve our ability to deliver AWS API updates continuously. The number of new services and updates has been significantly increasing the frequency with which we update. We want to avoid situations where one update is blocked by an unrelated service. We can now continuously deliver updates per service gem.
• Remove the usage of Ruby `autoload` statements. When you require a service gem, such as aws-sdk-ec2, all of the code is loaded and ready to use. This should eliminate a large number of thread safety issues that users encounter due to the use of autoload.
• A large amount of the dynamic runtime has been replaced with code generation. This allows users to reason better about what code is doing, receive better stack traces, improve performance, etc.

What Has Changed?

Our intent for the modularization is to keep SDK interfaces backwards compatible. You may need to modify your gem dependency on the AWS SDK for Ruby. The aws-sdk and aws-sdk-core gems have been bumped to version 3.0 to protect users from package level changes.

* Every service has a gem, such as aws-sdk-s3.
* The aws-sdk-core gem now only contains shared utilities.
* The aws-sdk-resources is obsolete. Service gems contain both client and resource interfaces.
* The aws-sdk gem now has a dependency on 75+ service gems.

Here is a diagram showing the dependencies of the aws-sdk gem across its major versions.

Why Bump to Version 3?

The version 2 aws-sdk-core gem includes code that defines 75+ service modules and shared utilities. It is important to prevent a service specific gem, such as aws-sdk-s3 and the core gem from both defining the same interfaces.

While we have worked hard to ensure full backwards compatibility in the service interfaces, a small number of private internal interfaces have been removed or changed. For users that have relied on these un-documented interfaces, this will prevent unexpected breaks with a gem update. Some of these changes include:

• Removed the internal runtime methods Aws.add_service and Aws.service_added. These methods were used by the runtime to detect when a service was autoloaded.
• Removed the internal Aws::Signers module and the various signature classes therein. These classes were marked with @api private. They are now available as separate gems:
  • aws-sigv4
  • aws-sigv2

Migrating Code From Version 2 to Version 3

Migrating should be very simple. If you depend on aws-sdk, then you do not need to change anything. If you depend on aws-sdk-resources or aws-sdk-core, replace these with a dependency on one of the following:

* aws-sdk ~> 3.0
* Service specific gems, such as aws-sdk-s3 ~> 1.0

You will also need to replace your require statements. You should no longer call require "aws-sdk-resources" or require "aws-sdk-core". A follow-up blog post provides detailed instructions on upgrading.

Questions?

Join us in our Gitter channel with your questions and feedback. The modularized released is currently published as a preview gem (rc1). We would love for you to try things out and to share feedback for these are GA.

Today, we’re pleased to announce the GA release of version 1.0.0 of the aws-record gem.

What Is aws-record?

In version 1 of the AWS SDK for Ruby, the AWS::Record class provided a data mapping abstraction over Amazon DynamoDB operations. Earlier this year, we released the aws-record developer preview as a separately packaged library to provide a similar data mapping abstraction for DynamoDB, built on top of the AWS SDK for Ruby version 2. After customer feedback and some more development work, we’re pleased to move the library out of developer preview to general availability.

How to Include the aws-record Gem in Your Project

The aws-record gem is available now from RubyGems:

gem install aws-record

You can also include it in your project’s Gemfile:

# Gemfile
gem 'aws-record', '~> 1.0'

This automatically includes a dependency on the aws-sdk-resources gem, major version 2. Be sure to include the aws-sdk or aws-sdk-resources gem in your Gemfile if you need to lock to a specific version, like so:

 # Gemfile
gem 'aws-record', '~> 1.0'
gem 'aws-sdk-resources', '~> 2.5'

Working with DynamoDB Tables Using the aws-record Gem

Defining an Aws::Record Model

The aws-record gem provides the Aws::Record module, which you can include in a class definition. This decorates your class with a variety of helper methods that can simplify interactions with Amazon DynamoDB. For example, the following model uses a variety of preset attribute definition helper methods and attribute options:

require 'aws-record'

class Forum
  include Aws::Record  

  string_attr     :forum_uuid, hash_key: true
  integer_attr    :post_id,    range_key: true
  string_attr     :author_username
  string_attr     :post_title
  string_attr     :post_body
  string_set_attr :tags,       default_value: Set.new 
  datetime_attr   :created_at, database_attribute_name: "PostCreatedAtTime"
  boolean_attr    :moderation, default_value: false
end

Using Validation Libraries with an Aws::Record Model

The aws-record gem does not come with a built-in validation process. Rather, it is designed to be a persistence layer, and to allow you to bring your own validation library. For example, the following model includes the popular ActiveModel::Validations module, and has defined a set of validations that will be run when we attempt to save an item:

require 'aws-record'
require 'active_model'

class Forum
  include Aws::Record
  include ActiveModel::Validations

  string_attr     :forum_uuid, hash_key: true
  integer_attr    :post_id,    range_key: true
  string_attr     :author_username
  string_attr     :post_title
  string_attr     :post_body
  string_set_attr :tags,       default_value: Set.new 
  datetime_attr   :created_at, database_attribute_name: "PostCreatedAtTime"
  boolean_attr    :moderation, default_value: false 


  validates_presence_of :forum_uuid, :post_id, :author_username
  validates_length_of :post_title, within: 4..30
  validates_length_of :post_body,  within: 2..5000
end

Creating a DynamoDB Table for a Model with Aws::Record::TableMigration

The aws-record gem provides a helper class for table operations, such as migrations. If we wanted to create a table for our Forum model in DynamoDB, we would run the following migration:

migration = Aws::Record::TableMigration.new(Forum)
migration.create!(
  provisioned_throughput: {
    read_capacity_units: 5,
    write_capacity_units: 2
  }
)
migration.wait_until_available

You can write these migrations in your Rakefile or as standalone helper scripts for your application. Because you don’t need to update your table definition for additions of non-key attributes, you may find that you’re not running migrations as often for your Aws::Record models.

Working with DynamoDB Items Using the aws-record Gem

Creating and Persisting a New Item

Using the example model above, once it has been created in the DynamoDB remote end using Aws::Record::TableMigration (or if it already existed in the remote end), it is simple to create and save a new item:

post = Forum.new(
  forum_uuid: FORUM_UUID,
  post_id: 1,
  author_username: "Author One",
  post_title: "Hello!",
  post_body: "Hello, world!"
)
post.created_at = Time.now
post.save # Performs a put_item call.

You can set attributes when you initialize a new item and with setter methods that are defined for you automatically.

Finding and Modifying an Item

A class-level method #find is provided to look up items from DynamoDB using your model’s key attributes. After setting a few new attribute values, calling #save will make an update call to DynamoDB, reflecting only the item changes you’ve made. This is important for users who are fetching items with projections (which may not include all attributes), or using single-table inheritance patterns (who may not have modeled all attributes present in a remote item), to avoid clobbering unmodeled or non-included attribute values.

post = Forum.find(forum_uuid: FORUM_UUID, post_id: 1)
post.post_title = "(Removed)"
post.post_body = "(Removed)"
post.moderation = true
post.save # Performs an update_item call on dirty attributes only.

There is also a class-level method to directly build and make an update call to DynamoDB, using key attributes to identify the item and non-key attributes to form the update expression:

Forum.update(
  forum_uuid: FORUM_UUID,
  post_id: 1,
  post_title: "(Removed)",
  post_body: "(Removed)",
  moderation: true
)

The preceding two code examples are functionally equivalent. You’ll have the same database state after running either snippet.

A Note on Dirty Tracking

In our last example, we talked about how item updates only reflect changes to modified attributes. Users of ActiveRecord or similar libraries will be familiar with the concept of tracking dirty attribute values, but aws-record is a bit different. That is because DynamoDB supports collection attribute types, and in Ruby, collection types are often modified through object mutation. To properly track changes to an item when objects can be changed through mutable state, Aws::Record items will, by default, keep deep copies of your attribute values when loading from DynamoDB. Attribute changes through mutation, like this example, will work the way you expect:

post = Forum.find(forum_uuid: FORUM_UUID, post_id: 1)
post.tags.add("First")
post.dirty? # => true
post.save # Will call update_item with the new tags collection.

Tracking deep copies of attribute values has implications for performance and memory. You can turn off mutation tracking at the model level. If you do so, dirty tracking will still work for new object references, but will not work for mutated objects:

class NoMTModel
  include Aws::Record
  disable_mutation_tracking
  string_attr :key, hash_key: true
  string_attr :body
  map_attr    :map
end

item = NoMTModel.new(key: "key", body: "body", map: {})
item.save # Will call put_item
item.map[:key] = "value"
item.dirty? # => false, because we won't track mutations to objects
item.body = "New Body"
item.dirty? # => true, because we will still notice reassignment
# Will call update_item, but only update :body unless we mark map as dirty explicitly.
item.save

Try the aws-record Gem Today!

We’re excited to hear about what you’re building with aws-record. Feel free to leave your feedback in the comments, or open an issue in our GitHub repo. Read through the documentation and get started!

We are happy to announce that the aws-record gem is now in Developer Preview and available for you to try.

What Is Aws::Record?

In version 1 of the AWS SDK for Ruby, the AWS::Record class provided a data mapping abstraction over Amazon DynamoDB operations. As version 2 of the AWS SDK for Ruby was being developed, many of you asked for an updated version of the library.

The aws-record gem provides a data mapping abstraction for DynamoDB built on top of the AWS SDK for Ruby version 2.

Using Aws::Record

You can download the aws-record gem from RubyGems by including the --pre flag in a gem installation:

gem install 'aws-record' --pre

You can also include it in your Gemfile. Do not include a version lock yet, so that bundler can find the pre-release version:

# Gemfile
gem 'aws-record'

Defining a Model

To create an aws-record model, include the Aws::Record module in your class definition:

require 'aws-record'

class Forum
  include Aws::Record
end

This will decorate your class with helper methods you can use to create a model compatible with DynamoDB’s table schemas. You might define keys for your table:

require 'aws-record'

class Forum
  include Aws::Record
  string_attr  :forum_uuid, hash_key: true
  integer_attr :post_id,    range_key: true
end

When you use these helper methods, you do not need to worry about how to define these attributes and types in DynamoDB. The helper methods and marshaler classes are able to define your table and item operations for you. The aws-record gem comes with predefined attribute types that cover a variety of potential use cases:

require 'aws-record'

class Forum
  include Aws::Record
  string_attr   :forum_uuid, hash_key: true
  integer_attr  :post_id,    range_key: true
  string_attr   :author_username
  string_attr   :post_title
  string_attr   :post_body
  datetime_attr :created_at
  map_attr      :post_metadata
end

Creating a DynamoDB Table

The aws-record gem provides a helper class for table operations, such as migrations. If we wanted to create a table for our Forum model in DynamoDB, we would run the following migration:

require 'forum' # Depending on where you defined the class above.

migration = Aws::Record::TableMigration.new(Forum)

migration.create!(
  provisioned_throughput: {
    read_capacity_units: 10,
    write_capacity_units: 4
  }
)

migration.wait_until_available # Blocks until table creation is complete.

Operations with DynamoDB Items

With a model and table defined, we can perform operations that relate to items in our table. Let’s create a post:

require 'forum'
require 'securerandom'

uuid = SecureRandom.uuid

post = Forum.new
post.forum_uuid = uuid
post.post_id = 1
post.author_username = "User1"
post.post_title = "Hello!"
post.post_body = "Hello Aws::Record"
post.created_at = Time.now
post.post_metadata = {
  this_is_a: "Post",
  types_supported_include: ["String", "Integer", "DateTime"],
  how_many_times_ive_done_this: 1
}

post.save # Writes to the database.

This example shows us some of the types that are supported and serialized for you. Using the key we’ve defined, we can also find this object in our table:

my_post = Forum.find(forum_uuid: uuid, post_id: 1)
my_post.post_title # => "Hello!"
my_post.created_at # => #<DateTime: 2016-02-09T14:39:07-08:00 ((2457428j,81547s,0n),-28800s,2299161j)>

You can use the same approach to save changes or, as shown here, you can delete the item from the table:

my_post.delete! # => true

At this point, we know how to use Aws::Record to perform key-value store operations powered by DynamoDB and have an introduction to the types available for use in our tables.

Querying, Scanning, and Collections

Because it is likely that you’re probably doing Query and Scan operations in addition to key-value operations, aws-record provides support for integrating them with your model class.

When you include the Aws::Record module, your model class is decorated with #query and #scan methods, which correspond to the AWS SDK for Ruby client operations. The response is wrapped in a collection enumerable for you. Consider the following basic scan operation:

Forum.scan # => #<Aws::Record::ItemCollection:0x007ffc293ec790 @search_method=:scan, @search_params={:table_name=>"Forum"}, @model=Forum, @client=#<Aws::DynamoDB::Client>>

No client call has been made yet: ItemCollection instances are lazy, and only make client calls only when needed. Because they provide an enumerable interface, you can use any of Ruby’s enumerable methods on your collection, and your result page is saved:

resp = Forum.scan
resp.take(1) # Makes a call to the underlying client. Returns a 'Forum' object.
resp.take(1) # Same result, but does not repeat the client call.

Because the Aws::Record::ItemCollection uses version 2 ofthe AWS SDK for Ruby, pagination support is built-in. So, if your operation requires multiple DynamoDB client calls due to response truncation, ItemCollection will handle the calls required in your enumeration:

def author_posts
  Forum.scan.inject({}) do |acc, post|
    author = post.author_username
    if acc[author]
      acc[author] += 1
    else
      acc[author] = 1
    end
    acc
  end
end

The same applies for queries. Your query result will also be provided as an enumerable ItemCollection:

def posts_by_forum(uuid)
  Forum.query(
    key_condition_expression: "#A = :a",
    expression_attribute_names: {
      "#A" => "forum_uuid"
    },
    expression_attribute_values: {
      ":a" => uuid
    }
  )
end

Given this functionality, you have the flexibility to mix and match Ruby’s enumerable functionality with DynamoDB filter expressions, for example, to curate your results. These two functions return the same set of responses:

def posts_by_author_in_forum(uuid, author)
  posts_by_forum(uuid).select do |post|
    post.author_username == author
  end
end

def posts_by_author_in_forum_with_filter(uuid, author)
  Forum.query(
    key_condition_expression: "#A = :a",
    filter_expression: "#B = :b",
    expression_attribute_names: {
      "#A" => "forum_uuid",
      "#B" => "author_username"
    },
    expression_attribute_values: {
      ":a" => uuid,
      ":b" => author
    }
  )
end

Support for Secondary Indexes

Aws::Record also supports both local and global secondary indexes. Consider this modified version of our Forum table:

require 'aws-record'

class IndexedForum
  include Aws::Record

  string_attr   :forum_uuid, hash_key: true
  integer_attr  :post_id,    range_key: true
  string_attr   :author_username
  string_attr   :post_title
  string_attr   :post_body
  datetime_attr :created_at
  map_attr      :post_metadata

  global_secondary_index(:author,
    hash_key: :author_username,
    projection: {
      projection_type: "INCLUDE",
      non_key_attributes: ["post_title"]
    }
  )

  local_secondary_index(:by_date,
    range_key: :created_at,
    projection: {
      projection_type: "ALL"
    }
  )
end

You can see the table’s attributes are the same, but we’ve included a couple potentially useful indexes.

• :author: This uses the author name as a partition, which provides a way to search across forums by author user name without having to scan and filter. Take note of the projection, because your global secondary index results will only return the :forum_uuid, :post_id, :author_username, and :post_title. Other attributes will be missing from this projection, and you would have to hydrate your item by calling #reload! on the item instance.
• :by_date: This provides a way to sort and search within a forum by post creation date.

To create this table with secondary indexes, you create a migration like we did before:

require 'indexed_forum'

migration = Aws::Record::TableMigration.new(IndexedForum)

migration.create!(
  provisioned_throughput: {
    read_capacity_units: 10,
    write_capacity_units: 4
  },
  global_secondary_index_throughput: {
    author: {
      read_capacity_units: 5,
      write_capacity_units: 3
    }
  }
)

migration.wait_until_available

You can use either of these indexes with the query interface:

require 'indexed_forum'

def search_by_author(author)
  IndexedForum.query(
    index_name: "author",
    key_condition_expression: "#A = :a",
    expression_attribute_names: {
      "#A" => "author_username"
    },
    expression_attribute_values: {
      ":a" => author
    }
  )
)

Secondary indexes can be a powerful performance tool, and aws-record can simplify the process of managing them.

Get Involved!

Please download the gem, give it a try, and let us know what you think. This project is a work in progress, so we welcome feature requests, bug reports, and information about the kinds of problems you’d like to solve by using this gem. And, as with other SDKs and tools we produce, we’d also be happy to look at contributions.

You can find the project on GitHub at https://github.com/awslabs/aws-sdk-ruby-record

Please reach out and let us know what you think!

The AWS SDKs and Tools team invites you to the first-ever online office hour hosted by the maintainers of the AWS SDK for Ruby. It will be held via Google Hangouts at 11:00am-12:00pm PDT (UTC -7:00) on Tuesday 6/30. If you don’t have one already, you will need to create an account with Google to join the video chat.

This first office hour will be driven by customer questions. We expect to focus on questions about the SDK, but any questions related to Ruby development on AWS are welcome. We’re excited to meet you and help you be successful in developing Ruby applications on AWS!

Please register for the event, add it to your calendar, and join the office hour next Monday.

Recently, Trevor, Loren, and myself from the AWS SDK team attended RailsConf in Atlanta. We had a great time at the conference and enjoyed connecting with many of you there.

Our Rails on Amazon Web Services Workshop

At RailsConf, we ran a workshop called Deploy and Manage Ruby on Rails Apps on AWS. It was an amazing experience for us, with attendees of all experience levels gettings hands-on experience not only deploying to AWS, but learning about the tools we’ve made to help make integrations easier.

For those of you who could not make it, you can still give this workshop a try!

• Detailed step-by-step instructions, the same as we provided to attendees, are available here.
• You can also follow along with the presentation recording on YouTube.
• Code for the sample app is available on GitHub.
• If you’d like to try using Amazon Relational Database Service instead of using AWS OpsWorks managed MySQL, you can reference our blog post on that topic as well.

Continuing the Conversation

We hope to see more of you on the conference trail again soon! Apropos of this, it is worth mentioning that AWS re:Invent registration is open at the time of writing. We will be there, and we hope to see you there!

I blogged previously about downloading objects from Amazon S3 using the version 2 AWS SDK for Ruby. It was requested that I write about uploading objects as well.

Managed File Uploads

The simplest and most common task is upload a file from disk to a bucket in Amazon S3. This is very straightforward when using the resource interface for Amazon S3:

s3 = Aws::S3::Resource.new

s3.bucket('bucket-name').object('key').upload_file('/source/file/path')

You can pass additional options to the Resource constructor and to #upload_file. This expanded example demonstrates configuring the resource client, uploading a public object and then generating a URL that can be used to download the object from a browser.

s3 = Aws::S3::Resource.new(
  credentials: Aws::Credentials.new('akid', 'secret'),
  region: 'us-west-1'
)

obj = s3.bucket('bucket-name').object('key')
obj.upload_file('/source/file/path', acl:'public-read')
obj.public_url
#=> "https://bucket-name.s3-us-west-1.amazonaws.com/key"

This is the recommended method of using the SDK to upload files to a bucket. Using this approach has the following benefits:

• Manages multipart uploads for objects larger than 15MB.
• Correctly opens files in binary mode to avoid encoding issues.
• Uses multiple threads for uploading parts of large objects in parallel.

Other Methods

In addition to Aws::S3::Object#upload_file, you can upload an object using #put or using the multipart upload APIs.

PUT Object

For smaller objects, you may choose to use #put instead. The #put method accepts an optional body, which can be a string or any IO object.

obj = s3.bucket('bucket-name').object('key')

# from a string
obj.put(body:'Hello World!')

# from an IO object
File.open('/source/file', 'rb') do |file|
  obj.put(body:file)
end

Multipart APIs

I recommend you use #upload_file whenever possible. If you need to manage large object copies, then you will need to use the multipart interfaces. There are restrictions on the minimum file, and part sizes you should be aware of. Typically these are reserved for advanced use cases.

Feedback

I’d love to hear feedback. If you find the AWS SDK for Ruby lacks a utility for working with Amazon S3, I’d love to hear about it. Please feel free to open a GitHub issue or drop into our Gitter channel.

You can now use version 2 of the AWS SDK for Ruby to verify the signatures of Amazon SNS messages. To help prevent spoofing attacks, you should verify messages are sent by Amazon SNS.

The new verifier follows the documented best practices for verification, including:

• Always use HTTPS when getting the certificate from Amazon SNS.
• Validate the authenticity of the certificate.
• Verify the certificate was received from Amazon SNS.

Basic Usage

Usage is straightforward. Construct a message verifier and call one of two methods. The given message body should be the JSON document string of the message.

verifier = Aws::SNS::MessageVerifier.new

verifier.authentic?(message_body)
#=> returns true or false

verifier.authenticate!(message_body)
#=> returns true or raises a VerificationError

You can use one instance of Aws::SNS::MessageVerifier.new to verify multiple messages.

Feedback

As always, we love to hear your feedback. It helps us prioritize our development efforts. In fact, this feature was added by customer request. Feel free to join our Gitter channel or open a GitHub issue.

We’ve recently added a utility class to the AWS SDK for Ruby that makes it easy to poll an Amazon SQS queue for messages.

poller = Aws::SQS::QueuePoller.new(queue_url)

poller.poll do |msg|
  puts msg.body
end

Messages are automatically deleted from the queue at the end of the block. This tool supports receiving and deleting messages in batches, long-polling, client-side tracking of stats, and more.

Long Polling

By default, messages are received using long polling. This method will force a default :wait_time_seconds of 20 seconds. If you prefer to use the queue default wait time, then pass a nil value for :wait_time_seconds.

# disables 20 second default, use queue ReceiveMessageWaitTimeSeconds attribute
poller.poll(wait_time_seconds:nil) do |msg|
  # ...
end

When disabling :wait_time_seconds by passing nil, you must ensure the queue ReceiveMessageWaitTimeSeconds attribute is set to a non zero value, or you will be short polling. This will trigger significantly more API calls.

Batch Receiving Messages

You can specify a maximum number of messages to receive with each polling attempt via :max_number_of_messages. When this is set to a positive value, greater than 1, the block will receive an array of messages, instead of a single message.

# receives and yields up to 10 messages at a time
poller.poll(max_number_of_messages:10) do |messages|
  messages.each do |msg|
    # ...
  end
end

The maximum value for :max_number_of_messages is enforced by Amazon SQS.

Visibility Timeouts

When receiving messages, you have a fixed amount of time to process and delete each message before it is added back into the queue. This is the visibility timeout. By default, the queue’s VisibilityTimeout attribute is used. You can provide an alternative visibility timeout when polling.

# override queue visibility timeout
poller.poll(visibility_timeout:10) do |msg|
  # do work ...
end

You can reset the visibility timeout of a single message by calling #change_message_visibility. This is useful when you need more time to finish processing the message.

poller.poll do |msg|

  # do work ...

  # need more time for processing
  poller.change_message_visibility(msg, 60)

  # finish work ...

end

If you change the visibility timeout of a message to zero, it will return to the queue immediately.

Deleting Messages

Messages are deleted from the queue when the block returns normally.

poller.poll do |msg|
  # do work
end # messages deleted here

You can skip message deletion by passing skip_delete: true. This allows you to manually delete the messages using {#deletemessage}, or {#deletemessages}.

# single message
poller.poll(skip_delete: true) do |msg|
  poller.delete_message(msg) # if successful
end

# message batch
poller.poll(skip_delete: true, max_number_of_messages:10) do |messages|
  poller.delete_messages(messages)
end

Another way to manage message deletion is to throw :skip_delete from the poll block. You can use this to choose when a message, or message batch is deleted on an individual basis:

poller.poll do |msg|
  begin
    # do work
  rescue
    # unexpected error occurred while processing messages,
    # log it, and skip delete so it can be re-processed later
    throw :skip_delete
  end
end

Terminating the Polling Loop

By default, polling will continue indefinitely. You can stop the poller by providing an idle timeout or by throwing :stop_polling from the {#before_request} callback.

:idle_timeout

This is a configurable, maximum number of seconds to wait for a new message before the polling loop exists. By default, there is no idle timeout.

# stops polling after a minute of no received messages
poller.poll(idle_timeout: 60) do |msg|
  # ...
end

:stop_polling

If you want more fine-grained control, you can configure a before request callback to trigger before each long poll. Throwing :stop_polling from this callback will cause the poller to exit normally without making the next request.

# stop after processing 100 messages
poller.before_request do |stats|
  throw :stop_polling if stats.receive_message_count >= 100
end

poller.poll do |msg|
  # do work ...
end

Tracking Progress

The poller will automatically track a few statistics client-side in a PollerStats object. You can access the poller stats three ways:

• The first block argument of {#before_request}
• The second block argument of {#poll}.
• The return value from {#poll}.

Here are examples of accessing the statistics.

• Configure a {#before_request} callback.

  poller.before_reqeust do |stats|
    logger.info("requests: #{stats.request_count}")
    logger.info("messages: #{stats.received_message_count}")
    logger.info("last-timestamp: #{stats.last_message_received_at}")
  end

• Accept a second argument in the poll block, for example:

  poller.poll do |msg, stats|
    logger.info("requests: #{stats.request_count}")
    logger.info("messages: #{stats.received_message_count}")
    logger.info("last-timestamp: #{stats.last_message_received_at}")
  end

• Return value:

  stats = poller.poll(idle_timeout:10) do |msg|
    # do work ...
  end
  logger.info("requests: #{stats.request_count}")
  logger.info("messages: #{stats.received_message_count}")
  logger.info("last-timestamp: #{stats.last_message_received_at}")

Feedback

Let us know what you think about the new queue poller. Join the conversation in our Gitter channel or open a GitHub issue.

With the release of V2 of the AWS SDK for Ruby, we’ve received customer feedback asking for support for the Ruby on Rails integration features provided by V1 of the SDK.

Today, we’re excited to announce the release of the aws-sdk-rails gem, available now via RubyGems and, of course, on GitHub.

To get started, add the aws-sdk-rails gem to your Gemfile:

gem 'aws-sdk-rails', '~> 1.0'

ActionMailer and Amazon Simple Email Service (SES)

The gem will automatically configure Rails to include an :aws_sdk delivery method for ActionMailer, that uses Amazon SES as a backend. It is simple to configure Rails to use this delivery method:

# config/application.rb
config.action_mailer.delivery_method = :aws_sdk

The aws-sdk-rails gem will use the AWS SDK for Ruby V2’s SES client automatically for any mail delivery event.

Logging

The gem will automatically wire the AWS SDK for Ruby’s logger to use Rails.logger by default.

You can customize the SDK log level and an optional log formatter in a config initializer:

# config/initializers/aws-sdk.rb
# log level defaults to :info
Aws.config[:log_level] = :debug

It is important to understand that all SDK log messages are logged at the same log level. Why is this important? When you set the Rails log level, you’re muting all log messages below that log level. So, if you want to, for example, only see SDK log messages in development, you might set the SDK log level to :debug as shown above, and set the Rails logger to show debug in development.

Credentials

The AWS SDK for Ruby will attempt locate credentials by searching the following locations:

• ENV['AWS_ACCESS_KEY_ID'] and ENV['AWS_SECRET_ACCESS_KEY']
• The shared credentials ini file at ~/.aws/credentials
• From an instance profile when running on Amazon EC2

If you need to manually configure credentials, you should add them to your initializer:

# config/initializers/aws-sdk.rb
Aws.config[:credentials] = Aws::Credentials.new(access_key, secret)

Learn more about credentials in the AWS SDK for Ruby V2.

Never commit your credentials to source control. Besides being a security risk, it makes it very difficult to rotate your credentials.

Available Now

The aws-sdk-rails gem is available now.

As always, we’d love to hear your feedback, and welcome any Issues or Pull Requests at the aws-sdk-rails GitHub repo.