Category: Ruby

Today, we’re pleased to announce the GA release of version 1.0.0 of the aws-record gem.

What Is aws-record?

In version 1 of the AWS SDK for Ruby, the AWS::Record class provided a data mapping abstraction over Amazon DynamoDB operations. Earlier this year, we released the aws-record developer preview as a separately packaged library to provide a similar data mapping abstraction for DynamoDB, built on top of the AWS SDK for Ruby version 2. After customer feedback and some more development work, we’re pleased to move the library out of developer preview to general availability.

How to Include the aws-record Gem in Your Project

The aws-record gem is available now from RubyGems:

gem install aws-record

You can also include it in your project’s Gemfile:

# Gemfile
gem 'aws-record', '~> 1.0'

This automatically includes a dependency on the aws-sdk-resources gem, major version 2. Be sure to include the aws-sdk or aws-sdk-resources gem in your Gemfile if you need to lock to a specific version, like so:

 # Gemfile
gem 'aws-record', '~> 1.0'
gem 'aws-sdk-resources', '~> 2.5'

Working with DynamoDB Tables Using the aws-record Gem

Defining an Aws::Record Model

The aws-record gem provides the Aws::Record module, which you can include in a class definition. This decorates your class with a variety of helper methods that can simplify interactions with Amazon DynamoDB. For example, the following model uses a variety of preset attribute definition helper methods and attribute options:

require 'aws-record'

class Forum
  include Aws::Record  

  string_attr     :forum_uuid, hash_key: true
  integer_attr    :post_id,    range_key: true
  string_attr     :author_username
  string_attr     :post_title
  string_attr     :post_body
  string_set_attr :tags,       default_value: Set.new 
  datetime_attr   :created_at, database_attribute_name: "PostCreatedAtTime"
  boolean_attr    :moderation, default_value: false
end

Using Validation Libraries with an Aws::Record Model

The aws-record gem does not come with a built-in validation process. Rather, it is designed to be a persistence layer, and to allow you to bring your own validation library. For example, the following model includes the popular ActiveModel::Validations module, and has defined a set of validations that will be run when we attempt to save an item:

require 'aws-record'
require 'active_model'

class Forum
  include Aws::Record
  include ActiveModel::Validations

  string_attr     :forum_uuid, hash_key: true
  integer_attr    :post_id,    range_key: true
  string_attr     :author_username
  string_attr     :post_title
  string_attr     :post_body
  string_set_attr :tags,       default_value: Set.new 
  datetime_attr   :created_at, database_attribute_name: "PostCreatedAtTime"
  boolean_attr    :moderation, default_value: false 


  validates_presence_of :forum_uuid, :post_id, :author_username
  validates_length_of :post_title, within: 4..30
  validates_length_of :post_body,  within: 2..5000
end

Creating a DynamoDB Table for a Model with Aws::Record::TableMigration

The aws-record gem provides a helper class for table operations, such as migrations. If we wanted to create a table for our Forum model in DynamoDB, we would run the following migration:

migration = Aws::Record::TableMigration.new(Forum)
migration.create!(
  provisioned_throughput: {
    read_capacity_units: 5,
    write_capacity_units: 2
  }
)
migration.wait_until_available

You can write these migrations in your Rakefile or as standalone helper scripts for your application. Because you don’t need to update your table definition for additions of non-key attributes, you may find that you’re not running migrations as often for your Aws::Record models.

Working with DynamoDB Items Using the aws-record Gem

Creating and Persisting a New Item

Using the example model above, once it has been created in the DynamoDB remote end using Aws::Record::TableMigration (or if it already existed in the remote end), it is simple to create and save a new item:

post = Forum.new(
  forum_uuid: FORUM_UUID,
  post_id: 1,
  author_username: "Author One",
  post_title: "Hello!",
  post_body: "Hello, world!"
)
post.created_at = Time.now
post.save # Performs a put_item call.

You can set attributes when you initialize a new item and with setter methods that are defined for you automatically.

Finding and Modifying an Item

A class-level method #find is provided to look up items from DynamoDB using your model’s key attributes. After setting a few new attribute values, calling #save will make an update call to DynamoDB, reflecting only the item changes you’ve made. This is important for users who are fetching items with projections (which may not include all attributes), or using single-table inheritance patterns (who may not have modeled all attributes present in a remote item), to avoid clobbering unmodeled or non-included attribute values.

post = Forum.find(forum_uuid: FORUM_UUID, post_id: 1)
post.post_title = "(Removed)"
post.post_body = "(Removed)"
post.moderation = true
post.save # Performs an update_item call on dirty attributes only.

There is also a class-level method to directly build and make an update call to DynamoDB, using key attributes to identify the item and non-key attributes to form the update expression:

Forum.update(
  forum_uuid: FORUM_UUID,
  post_id: 1,
  post_title: "(Removed)",
  post_body: "(Removed)",
  moderation: true
)

The preceding two code examples are functionally equivalent. You’ll have the same database state after running either snippet.

A Note on Dirty Tracking

In our last example, we talked about how item updates only reflect changes to modified attributes. Users of ActiveRecord or similar libraries will be familiar with the concept of tracking dirty attribute values, but aws-record is a bit different. That is because DynamoDB supports collection attribute types, and in Ruby, collection types are often modified through object mutation. To properly track changes to an item when objects can be changed through mutable state, Aws::Record items will, by default, keep deep copies of your attribute values when loading from DynamoDB. Attribute changes through mutation, like this example, will work the way you expect:

post = Forum.find(forum_uuid: FORUM_UUID, post_id: 1)
post.tags.add("First")
post.dirty? # => true
post.save # Will call update_item with the new tags collection.

Tracking deep copies of attribute values has implications for performance and memory. You can turn off mutation tracking at the model level. If you do so, dirty tracking will still work for new object references, but will not work for mutated objects:

class NoMTModel
  include Aws::Record
  disable_mutation_tracking
  string_attr :key, hash_key: true
  string_attr :body
  map_attr    :map
end

item = NoMTModel.new(key: "key", body: "body", map: {})
item.save # Will call put_item
item.map[:key] = "value"
item.dirty? # => false, because we won't track mutations to objects
item.body = "New Body"
item.dirty? # => true, because we will still notice reassignment
# Will call update_item, but only update :body unless we mark map as dirty explicitly.
item.save

Try the aws-record Gem Today!

We’re excited to hear about what you’re building with aws-record. Feel free to leave your feedback in the comments, or open an issue in our GitHub repo. Read through the documentation and get started!

We are happy to announce that the aws-record gem is now in Developer Preview and available for you to try.

What Is Aws::Record?

In version 1 of the AWS SDK for Ruby, the AWS::Record class provided a data mapping abstraction over Amazon DynamoDB operations. As version 2 of the AWS SDK for Ruby was being developed, many of you asked for an updated version of the library.

The aws-record gem provides a data mapping abstraction for DynamoDB built on top of the AWS SDK for Ruby version 2.

Using Aws::Record

You can download the aws-record gem from RubyGems by including the --pre flag in a gem installation:

gem install 'aws-record' --pre

You can also include it in your Gemfile. Do not include a version lock yet, so that bundler can find the pre-release version:

# Gemfile
gem 'aws-record'

Defining a Model

To create an aws-record model, include the Aws::Record module in your class definition:

require 'aws-record'

class Forum
  include Aws::Record
end

This will decorate your class with helper methods you can use to create a model compatible with DynamoDB’s table schemas. You might define keys for your table:

require 'aws-record'

class Forum
  include Aws::Record
  string_attr  :forum_uuid, hash_key: true
  integer_attr :post_id,    range_key: true
end

When you use these helper methods, you do not need to worry about how to define these attributes and types in DynamoDB. The helper methods and marshaler classes are able to define your table and item operations for you. The aws-record gem comes with predefined attribute types that cover a variety of potential use cases:

require 'aws-record'

class Forum
  include Aws::Record
  string_attr   :forum_uuid, hash_key: true
  integer_attr  :post_id,    range_key: true
  string_attr   :author_username
  string_attr   :post_title
  string_attr   :post_body
  datetime_attr :created_at
  map_attr      :post_metadata
end

Creating a DynamoDB Table

The aws-record gem provides a helper class for table operations, such as migrations. If we wanted to create a table for our Forum model in DynamoDB, we would run the following migration:

require 'forum' # Depending on where you defined the class above.

migration = Aws::Record::TableMigration.new(Forum)

migration.create!(
  provisioned_throughput: {
    read_capacity_units: 10,
    write_capacity_units: 4
  }
)

migration.wait_until_available # Blocks until table creation is complete.

Operations with DynamoDB Items

With a model and table defined, we can perform operations that relate to items in our table. Let’s create a post:

require 'forum'
require 'securerandom'

uuid = SecureRandom.uuid

post = Forum.new
post.forum_uuid = uuid
post.post_id = 1
post.author_username = "User1"
post.post_title = "Hello!"
post.post_body = "Hello Aws::Record"
post.created_at = Time.now
post.post_metadata = {
  this_is_a: "Post",
  types_supported_include: ["String", "Integer", "DateTime"],
  how_many_times_ive_done_this: 1
}

post.save # Writes to the database.

This example shows us some of the types that are supported and serialized for you. Using the key we’ve defined, we can also find this object in our table:

my_post = Forum.find(forum_uuid: uuid, post_id: 1)
my_post.post_title # => "Hello!"
my_post.created_at # => #<DateTime: 2016-02-09T14:39:07-08:00 ((2457428j,81547s,0n),-28800s,2299161j)>

You can use the same approach to save changes or, as shown here, you can delete the item from the table:

my_post.delete! # => true

At this point, we know how to use Aws::Record to perform key-value store operations powered by DynamoDB and have an introduction to the types available for use in our tables.

Querying, Scanning, and Collections

Because it is likely that you’re probably doing Query and Scan operations in addition to key-value operations, aws-record provides support for integrating them with your model class.

When you include the Aws::Record module, your model class is decorated with #query and #scan methods, which correspond to the AWS SDK for Ruby client operations. The response is wrapped in a collection enumerable for you. Consider the following basic scan operation:

Forum.scan # => #<Aws::Record::ItemCollection:0x007ffc293ec790 @search_method=:scan, @search_params={:table_name=>"Forum"}, @model=Forum, @client=#<Aws::DynamoDB::Client>>

No client call has been made yet: ItemCollection instances are lazy, and only make client calls only when needed. Because they provide an enumerable interface, you can use any of Ruby’s enumerable methods on your collection, and your result page is saved:

resp = Forum.scan
resp.take(1) # Makes a call to the underlying client. Returns a 'Forum' object.
resp.take(1) # Same result, but does not repeat the client call.

Because the Aws::Record::ItemCollection uses version 2 ofthe AWS SDK for Ruby, pagination support is built-in. So, if your operation requires multiple DynamoDB client calls due to response truncation, ItemCollection will handle the calls required in your enumeration:

def author_posts
  Forum.scan.inject({}) do |acc, post|
    author = post.author_username
    if acc[author]
      acc[author] += 1
    else
      acc[author] = 1
    end
    acc
  end
end

The same applies for queries. Your query result will also be provided as an enumerable ItemCollection:

def posts_by_forum(uuid)
  Forum.query(
    key_condition_expression: "#A = :a",
    expression_attribute_names: {
      "#A" => "forum_uuid"
    },
    expression_attribute_values: {
      ":a" => uuid
    }
  )
end

Given this functionality, you have the flexibility to mix and match Ruby’s enumerable functionality with DynamoDB filter expressions, for example, to curate your results. These two functions return the same set of responses:

def posts_by_author_in_forum(uuid, author)
  posts_by_forum(uuid).select do |post|
    post.author_username == author
  end
end

def posts_by_author_in_forum_with_filter(uuid, author)
  Forum.query(
    key_condition_expression: "#A = :a",
    filter_expression: "#B = :b",
    expression_attribute_names: {
      "#A" => "forum_uuid",
      "#B" => "author_username"
    },
    expression_attribute_values: {
      ":a" => uuid,
      ":b" => author
    }
  )
end

Support for Secondary Indexes

Aws::Record also supports both local and global secondary indexes. Consider this modified version of our Forum table:

require 'aws-record'

class IndexedForum
  include Aws::Record

  string_attr   :forum_uuid, hash_key: true
  integer_attr  :post_id,    range_key: true
  string_attr   :author_username
  string_attr   :post_title
  string_attr   :post_body
  datetime_attr :created_at
  map_attr      :post_metadata

  global_secondary_index(:author,
    hash_key: :author_username,
    projection: {
      projection_type: "INCLUDE",
      non_key_attributes: ["post_title"]
    }
  )

  local_secondary_index(:by_date,
    range_key: :created_at,
    projection: {
      projection_type: "ALL"
    }
  )
end

You can see the table’s attributes are the same, but we’ve included a couple potentially useful indexes.

• :author: This uses the author name as a partition, which provides a way to search across forums by author user name without having to scan and filter. Take note of the projection, because your global secondary index results will only return the :forum_uuid, :post_id, :author_username, and :post_title. Other attributes will be missing from this projection, and you would have to hydrate your item by calling #reload! on the item instance.
• :by_date: This provides a way to sort and search within a forum by post creation date.

To create this table with secondary indexes, you create a migration like we did before:

require 'indexed_forum'

migration = Aws::Record::TableMigration.new(IndexedForum)

migration.create!(
  provisioned_throughput: {
    read_capacity_units: 10,
    write_capacity_units: 4
  },
  global_secondary_index_throughput: {
    author: {
      read_capacity_units: 5,
      write_capacity_units: 3
    }
  }
)

migration.wait_until_available

You can use either of these indexes with the query interface:

require 'indexed_forum'

def search_by_author(author)
  IndexedForum.query(
    index_name: "author",
    key_condition_expression: "#A = :a",
    expression_attribute_names: {
      "#A" => "author_username"
    },
    expression_attribute_values: {
      ":a" => author
    }
  )
)

Secondary indexes can be a powerful performance tool, and aws-record can simplify the process of managing them.

Get Involved!

Please download the gem, give it a try, and let us know what you think. This project is a work in progress, so we welcome feature requests, bug reports, and information about the kinds of problems you’d like to solve by using this gem. And, as with other SDKs and tools we produce, we’d also be happy to look at contributions.

You can find the project on GitHub at https://github.com/awslabs/aws-sdk-ruby-record

Please reach out and let us know what you think!

The AWS SDKs and Tools team invites you to the first-ever online office hour hosted by the maintainers of the AWS SDK for Ruby. It will be held via Google Hangouts at 11:00am-12:00pm PDT (UTC -7:00) on Tuesday 6/30. If you don’t have one already, you will need to create an account with Google to join the video chat.

This first office hour will be driven by customer questions. We expect to focus on questions about the SDK, but any questions related to Ruby development on AWS are welcome. We’re excited to meet you and help you be successful in developing Ruby applications on AWS!

Please register for the event, add it to your calendar, and join the office hour next Monday.

Recently, Trevor, Loren, and myself from the AWS SDK team attended RailsConf in Atlanta. We had a great time at the conference and enjoyed connecting with many of you there.

Our Rails on Amazon Web Services Workshop

At RailsConf, we ran a workshop called Deploy and Manage Ruby on Rails Apps on AWS. It was an amazing experience for us, with attendees of all experience levels gettings hands-on experience not only deploying to AWS, but learning about the tools we’ve made to help make integrations easier.

For those of you who could not make it, you can still give this workshop a try!

• Detailed step-by-step instructions, the same as we provided to attendees, are available here.
• You can also follow along with the presentation recording on YouTube.
• Code for the sample app is available on GitHub.
• If you’d like to try using Amazon Relational Database Service instead of using AWS OpsWorks managed MySQL, you can reference our blog post on that topic as well.

Continuing the Conversation

We hope to see more of you on the conference trail again soon! Apropos of this, it is worth mentioning that AWS re:Invent registration is open at the time of writing. We will be there, and we hope to see you there!

I blogged previously about downloading objects from Amazon S3 using the version 2 AWS SDK for Ruby. It was requested that I write about uploading objects as well.

Managed File Uploads

The simplest and most common task is upload a file from disk to a bucket in Amazon S3. This is very straightforward when using the resource interface for Amazon S3:

s3 = Aws::S3::Resource.new

s3.bucket('bucket-name').object('key').upload_file('/source/file/path')

You can pass additional options to the Resource constructor and to #upload_file. This expanded example demonstrates configuring the resource client, uploading a public object and then generating a URL that can be used to download the object from a browser.

s3 = Aws::S3::Resource.new(
  credentials: Aws::Credentials.new('akid', 'secret'),
  region: 'us-west-1'
)

obj = s3.bucket('bucket-name').object('key')
obj.upload_file('/source/file/path', acl:'public-read')
obj.public_url
#=> "https://bucket-name.s3-us-west-1.amazonaws.com/key"

This is the recommended method of using the SDK to upload files to a bucket. Using this approach has the following benefits:

• Manages multipart uploads for objects larger than 15MB.
• Correctly opens files in binary mode to avoid encoding issues.
• Uses multiple threads for uploading parts of large objects in parallel.

Other Methods

In addition to Aws::S3::Object#upload_file, you can upload an object using #put or using the multipart upload APIs.

PUT Object

For smaller objects, you may choose to use #put instead. The #put method accepts an optional body, which can be a string or any IO object.

obj = s3.bucket('bucket-name').object('key')

# from a string
obj.put(body:'Hello World!')

# from an IO object
File.open('/source/file', 'rb') do |file|
  obj.put(body:file)
end

Multipart APIs

I recommend you use #upload_file whenever possible. If you need to manage large object copies, then you will need to use the multipart interfaces. There are restrictions on the minimum file, and part sizes you should be aware of. Typically these are reserved for advanced use cases.

Feedback

I’d love to hear feedback. If you find the AWS SDK for Ruby lacks a utility for working with Amazon S3, I’d love to hear about it. Please feel free to open a GitHub issue or drop into our Gitter channel.

You can now use version 2 of the AWS SDK for Ruby to verify the signatures of Amazon SNS messages. To help prevent spoofing attacks, you should verify messages are sent by Amazon SNS.

The new verifier follows the documented best practices for verification, including:

• Always use HTTPS when getting the certificate from Amazon SNS.
• Validate the authenticity of the certificate.
• Verify the certificate was received from Amazon SNS.

Basic Usage

Usage is straightforward. Construct a message verifier and call one of two methods. The given message body should be the JSON document string of the message.

verifier = Aws::SNS::MessageVerifier.new

verifier.authentic?(message_body)
#=> returns true or false

verifier.authenticate!(message_body)
#=> returns true or raises a VerificationError

You can use one instance of Aws::SNS::MessageVerifier.new to verify multiple messages.

Feedback

As always, we love to hear your feedback. It helps us prioritize our development efforts. In fact, this feature was added by customer request. Feel free to join our Gitter channel or open a GitHub issue.

We’ve recently added a utility class to the AWS SDK for Ruby that makes it easy to poll an Amazon SQS queue for messages.

poller = Aws::SQS::QueuePoller.new(queue_url)

poller.poll do |msg|
  puts msg.body
end

Messages are automatically deleted from the queue at the end of the block. This tool supports receiving and deleting messages in batches, long-polling, client-side tracking of stats, and more.

Long Polling

By default, messages are received using long polling. This method will force a default :wait_time_seconds of 20 seconds. If you prefer to use the queue default wait time, then pass a nil value for :wait_time_seconds.

# disables 20 second default, use queue ReceiveMessageWaitTimeSeconds attribute
poller.poll(wait_time_seconds:nil) do |msg|
  # ...
end

When disabling :wait_time_seconds by passing nil, you must ensure the queue ReceiveMessageWaitTimeSeconds attribute is set to a non zero value, or you will be short polling. This will trigger significantly more API calls.

Batch Receiving Messages

You can specify a maximum number of messages to receive with each polling attempt via :max_number_of_messages. When this is set to a positive value, greater than 1, the block will receive an array of messages, instead of a single message.

# receives and yields up to 10 messages at a time
poller.poll(max_number_of_messages:10) do |messages|
  messages.each do |msg|
    # ...
  end
end

The maximum value for :max_number_of_messages is enforced by Amazon SQS.

Visibility Timeouts

When receiving messages, you have a fixed amount of time to process and delete each message before it is added back into the queue. This is the visibility timeout. By default, the queue’s VisibilityTimeout attribute is used. You can provide an alternative visibility timeout when polling.

# override queue visibility timeout
poller.poll(visibility_timeout:10) do |msg|
  # do work ...
end

You can reset the visibility timeout of a single message by calling #change_message_visibility. This is useful when you need more time to finish processing the message.

poller.poll do |msg|

  # do work ...

  # need more time for processing
  poller.change_message_visibility(msg, 60)

  # finish work ...

end

If you change the visibility timeout of a message to zero, it will return to the queue immediately.

Deleting Messages

Messages are deleted from the queue when the block returns normally.

poller.poll do |msg|
  # do work
end # messages deleted here

You can skip message deletion by passing skip_delete: true. This allows you to manually delete the messages using {#deletemessage}, or {#deletemessages}.

# single message
poller.poll(skip_delete: true) do |msg|
  poller.delete_message(msg) # if successful
end

# message batch
poller.poll(skip_delete: true, max_number_of_messages:10) do |messages|
  poller.delete_messages(messages)
end

Another way to manage message deletion is to throw :skip_delete from the poll block. You can use this to choose when a message, or message batch is deleted on an individual basis:

poller.poll do |msg|
  begin
    # do work
  rescue
    # unexpected error occurred while processing messages,
    # log it, and skip delete so it can be re-processed later
    throw :skip_delete
  end
end

Terminating the Polling Loop

By default, polling will continue indefinitely. You can stop the poller by providing an idle timeout or by throwing :stop_polling from the {#before_request} callback.

:idle_timeout

This is a configurable, maximum number of seconds to wait for a new message before the polling loop exists. By default, there is no idle timeout.

# stops polling after a minute of no received messages
poller.poll(idle_timeout: 60) do |msg|
  # ...
end

:stop_polling

If you want more fine-grained control, you can configure a before request callback to trigger before each long poll. Throwing :stop_polling from this callback will cause the poller to exit normally without making the next request.

# stop after processing 100 messages
poller.before_request do |stats|
  throw :stop_polling if stats.receive_message_count >= 100
end

poller.poll do |msg|
  # do work ...
end

Tracking Progress

The poller will automatically track a few statistics client-side in a PollerStats object. You can access the poller stats three ways:

• The first block argument of {#before_request}
• The second block argument of {#poll}.
• The return value from {#poll}.

Here are examples of accessing the statistics.

• Configure a {#before_request} callback.

  poller.before_reqeust do |stats|
    logger.info("requests: #{stats.request_count}")
    logger.info("messages: #{stats.received_message_count}")
    logger.info("last-timestamp: #{stats.last_message_received_at}")
  end

• Accept a second argument in the poll block, for example:

  poller.poll do |msg, stats|
    logger.info("requests: #{stats.request_count}")
    logger.info("messages: #{stats.received_message_count}")
    logger.info("last-timestamp: #{stats.last_message_received_at}")
  end

• Return value:

  stats = poller.poll(idle_timeout:10) do |msg|
    # do work ...
  end
  logger.info("requests: #{stats.request_count}")
  logger.info("messages: #{stats.received_message_count}")
  logger.info("last-timestamp: #{stats.last_message_received_at}")

Feedback

Let us know what you think about the new queue poller. Join the conversation in our Gitter channel or open a GitHub issue.

With the release of V2 of the AWS SDK for Ruby, we’ve received customer feedback asking for support for the Ruby on Rails integration features provided by V1 of the SDK.

Today, we’re excited to announce the release of the aws-sdk-rails gem, available now via RubyGems and, of course, on GitHub.

To get started, add the aws-sdk-rails gem to your Gemfile:

gem 'aws-sdk-rails', '~> 1.0'

ActionMailer and Amazon Simple Email Service (SES)

The gem will automatically configure Rails to include an :aws_sdk delivery method for ActionMailer, that uses Amazon SES as a backend. It is simple to configure Rails to use this delivery method:

# config/application.rb
config.action_mailer.delivery_method = :aws_sdk

The aws-sdk-rails gem will use the AWS SDK for Ruby V2’s SES client automatically for any mail delivery event.

Logging

The gem will automatically wire the AWS SDK for Ruby’s logger to use Rails.logger by default.

You can customize the SDK log level and an optional log formatter in a config initializer:

# config/initializers/aws-sdk.rb
# log level defaults to :info
Aws.config[:log_level] = :debug

It is important to understand that all SDK log messages are logged at the same log level. Why is this important? When you set the Rails log level, you’re muting all log messages below that log level. So, if you want to, for example, only see SDK log messages in development, you might set the SDK log level to :debug as shown above, and set the Rails logger to show debug in development.

Credentials

The AWS SDK for Ruby will attempt locate credentials by searching the following locations:

• ENV['AWS_ACCESS_KEY_ID'] and ENV['AWS_SECRET_ACCESS_KEY']
• The shared credentials ini file at ~/.aws/credentials
• From an instance profile when running on Amazon EC2

If you need to manually configure credentials, you should add them to your initializer:

# config/initializers/aws-sdk.rb
Aws.config[:credentials] = Aws::Credentials.new(access_key, secret)

Learn more about credentials in the AWS SDK for Ruby V2.

Never commit your credentials to source control. Besides being a security risk, it makes it very difficult to rotate your credentials.

Available Now

The aws-sdk-rails gem is available now.

As always, we’d love to hear your feedback, and welcome any Issues or Pull Requests at the aws-sdk-rails GitHub repo.

As we showed in previous posts, it’s easy to put JSON items into Amazon DynamoDB, retrieve specific attributes with projection expressions, and fetch only data that meet some criteria with condition expressions. Now, let’s take a look at how we can conditionally modify existing items with Update Expressions. (Note: this code uses the same ProductCatalog table we used in Parts 1 and 2).

In the following examples, we use the following helper method to perform conditional updates. It performs the UpdateItem operation with return_values set to return the old item. We also use the GetItem operation so the method can return both the old and new items for us to compare. (If the update condition in the request is not met, then the method sets the returned old item to nil.)

def do_update_item(key_id, update_exp, condition_exp, exp_attribute_values)
  begin
    old_result = @dynamodb.update_item(
      :update_expression => update_exp,
      :condition_expression => condition_exp,
      :expression_attribute_values => exp_attribute_values,
      :table_name => "ProductCatalog",
      :key => { :Id => key_id },
      :return_values => "ALL_OLD",
    ).data.attributes
  rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
    old_result = nil
    puts "Condition not met"
  end

  new_result = @dynamodb.get_item(
    :table_name => "ProductCatalog", :key => { :Id => key_id },
    :consistent_read => true
  ).data.item  

  return old_result, new_result
end

Using Conditional Update Expressions

Updates in DynamoDB are atomic. This allows applications to concurrently update items without worrying about conflicts occurring. For example, the following code demonstrates maintaining a MAX value in DynamoDB with a conditional update using SET. Note that, because DynamoDB is schema-less, we don’t need to define the HighestRating attribute beforehand. Instead, we create it on the first call.

# storing a "max" value with conditional SET
# SET attribute if doesn't exist, otherwise SET if stored highest rating < this rating
def update_highest_rating(rating)
  do_update_item(303,
    "SET HighestRating = :val",
    "attribute_not_exists(HighestRating) OR HighestRating < :val",
    {
      ":val" => rating
    }
  )
end

# multiple threads trying to SET highest value (ranging from 0 to 10)
threads = []
(0..10).to_a.shuffle.each { |i|
  # some number of "Condition not met" depending on shuffled order
  puts i
  threads[i] = Thread.new {
    update_highest_rating(i)
  }
}
threads.each {|t| t.join}

# fetch the item and examine the HighestRating stored
puts "Max = #{@dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item["HighestRating"].to_i}"   # Max = 10

We can also use update expressions to atomically maintain a count and add to a set:

# ADD to intialize/increment and add to set
threads = []
20.times do |i|
  threads[i] = Thread.new {
    do_update_item(303,
      "ADD TimesViewed :val, Tags :was_here",
      nil, # no condition expression
      {
        # Each of the 20 threads increments by 1
        ":val" => 1,

        # Each thread adds to the tag set
        # Note: type must match stored attribute's type
        ":was_here" => Set.new(["#Thread#{i}WasHere"])
      }
    )
  }
end
threads.each {|t| t.join}

# fetch the item and examine the TimesViewed attribute
item = @dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item

puts "TimesViewed = #{item["TimesViewed"].to_i}"
# TimesViewed = 20

puts "Tags = #{item["Tags"].inspect}"
# Tags = #<Set: {"#Mars", "#MarsCuriosity", "#StillRoving", ..each thread was here...}>

Similarly, we can decrement the count and remove from the set to undo our previous operations.

# Undo the views and set adding that we just performed
threads = []
20.times do |i|
  threads[i] = Thread.new {
    do_update_item(303,
      "ADD TimesViewed :val DELETE Tags :was_here",
      nil,  # no condition expression
      {
        # Each of the 20 threads decrements by 1
        ":val" => -1,

        # Each thread removes from the tag set
        # Note: type must match stored attribute's type
        ":was_here" => Set.new(["#Thread#{i}WasHere"])
      }
    )
  }
end
threads.each {|t| t.join}

# fetch the item and examine the TimesViewed attribute
item = @dynamodb.get_item(
  :table_name => "ProductCatalog", :key => { :Id => 303 }
).data.item

puts "TimesViewed = #{item["TimesViewed"].to_i}"
# TimesViewed = 0

puts "Tags = #{item["Tags"].inspect}"
# Tags = #<Set: {"#Mars", "#MarsCuriosity", "#StillRoving"}>

We can also use the REMOVE keyword to delete attributes, such as the HighestRating and TimesViewed attributes we added in the previous code.

# removing attributes from items
old_and_new = do_update_item(303,
  "REMOVE HighestRating, TimesViewed",
  nil,  # no condition expression
  nil,  # no attribute expression values
)

puts "OLD HighestRating is nil ? #{old_and_new[0]["HighestRating"] == nil}"
#=> false

puts "OLD TimesViewed is nil ? #{old_and_new[0]["TimesViewed"] == nil}"
#=> false

puts "NEW HighestRating is nil ? #{old_and_new[1]["HighestRating"] == nil}"
#=> true

puts "NEW TimesViewed is nil ? #{old_and_new[1]["TimesViewed"] == nil}"
#=> true

Conclusion

We hope this series was helpful in demonstrating expressions and how they allow you to interact with DynamoDB more flexibly than before. We’re always interested in hearing what developers would like to see in the future, so let us know what you think in the comments or through our forums!

As we showed in the previous post, it’s easy to put JSON items into Amazon DynamoDB and retrieve specific attributes with projection expressions. Condition Expressions provide a more flexible and SQL-like way to retrieve only the items you want from DynamoDB. First, let’s put a few more items into DynamoDB using a BatchWriteItem operation. (Note: this code uses the same ProductCatalog table we used in Part 1)

# add some more items
@dynamodb.batch_write_item(
  :request_items => {
    "ProductCatalog" => [

      {:put_request => { :item => {
        Id: 300,
        Title: "Sojourner",
        Description: "Mars Pathfinder robotic Mars rover",
        Price: BigDecimal.new("2.65e8"),
        LaunchDate: {
          M: 12, D: 4, Y: 1996
        },
        LostCommunicationDate: {
          M: 9, D: 27, Y: 1997
        },
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 3,
        Tags: ["#Mars", "#InStarTrekSeason4", "#InRedPlant2000", "#LostComms"],
      }}},

      {:put_request => { :item => {
        Id: 301,
        Title: "Spirit",
        Description: "Mars Exploration Rover – A",
        Price: BigDecimal.new("4.1e8"),
        LaunchDate: {
          M: 6, D: 10, Y: 2003
        },
        LostCommunicationDate: {
          M: 3, D: 22, Y: 2010
        },
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 5,
        Tags: Set.new(["#Mars", "#StuckOnMars", "#LostComms"]),
      }}},

      {:put_request => { :item => {
        Id: 302,
        Title: "Opportunity",
        Description: "Mars Exploration Rover – B",
        Price: BigDecimal.new("4.1e8"),
        LaunchDate: {
          M: 7, D: 7, Y: 2003
        },
        LostCommunicationDate: nil,
        Features: {
          Rover: true,
        },
        NumberInStock: 10,
        OrdersPlaced: 10,
        Tags: Set.new(["#Mars", "#StillRoving"]),
      }}},

      {:put_request => { :item => {
        Id: 303,
        Title: "Curiosity",
        Description: "car-sized robotic rover",
        Price: BigDecimal.new("2.5e9"),
        LaunchDate: {
          M: 11, D: 26, Y: 2011
        },
        LostCommunicationDate: nil,
        Features: {
          Rover: true,
          RoboticArm: true,
        },
        NumberInStock: 0,
        OrdersPlaced: 30,
        Tags: Set.new(["#Mars", "#MarsCuriosity", "#StillRoving"]),
      }}},

    ]
  }
)

Using Condition Expressions

We could also use condition expressions on the results of Query, but since we’re using a simple data model (only have hash key on product Id), we demonstrate this with scans. We use the following helper method to perform the scan and format the product titles returned:

def do_scan(filter_exp, exp_attribute_values)
  result = @dynamodb.scan(
    :expression_attribute_values => exp_attribute_values,
    :filter_expression => filter_exp,   # Condition Expressions are supplied through the FilterExpression parameter
    :projection_expression => "Title",
    :table_name => "ProductCatalog"
  ).data.items

  # format all retrieved titles into a single line
  return "scan retrieved: #{(result.map { |item| item["Title"] }).join(", ")}"
end

Let’s look at some example expressions and the results they return from our current ProductCatalog table:

# All products that don't have a launch month of November (11)
puts do_scan(
  "LaunchDate.M <> :m",
  {
    ":m" => 11
  }
)
# scan retrieved: 20-Bicycle 205, Opportunity, Spirit, Sojourner


# All rover products that don't have a launch month of November
puts do_scan(
  "attribute_exists(Features.Rover) AND LaunchDate.M <> :m",
  {
    ":m" => 11,
  }
)
# scan retrieved: Opportunity, Spirit, Sojourner


# Non-rovers
puts do_scan(
  "attribute_not_exists(Features.Rover)",
  nil
)
# scan retrieved: 20-Bicycle 205


# mid-range rovers or inexpensive products
puts do_scan(
  "(Price BETWEEN :low AND :high) OR Price < :verylow",
  {
    ":verylow" => BigDecimal.new("1e8"),
    ":low" => BigDecimal.new("3e8"),
    ":high" => BigDecimal.new("5e8")
  }
)
# scan retrieved: 20-Bicycle 205, Opportunity, Spirit


# within-Item referencing: more orders placed than in stock
puts do_scan(
  "OrdersPlaced > NumberInStock",
  nil
)
# scan retrieved: Curiosity


# string prefixing
puts do_scan(
  "begins_with(Title, :s)",
  {
    ":s" => "S",
  }
)
# scan retrieved: Spirit, Sojourner


# contains
puts do_scan(
  "contains(Tags, :tag1) AND contains(Tags, :tag2)",
  {
    ":tag1" => "#StuckOnMars",
    ":tag2" => "#LostComms",
  }
)
# scan retrieved: Spirit


# contains (Note: "Tags" is a list for Sojourner)
puts do_scan(
  "contains(Tags, :tag1)",
  {
    ":tag1" => "#LostComms",
  }
)
# scan retrieved: Spirit, Sojourner


# in operator
puts do_scan(
  "Id in (:id1, :id2)",
  {
    ":id1" => 302,
    ":id2" => 303,
  }
)
# scan retrieved: Curiosity, Opportunity


# equivalently, with parentheses
puts do_scan(
  "(Id = :id1) OR (Id = :id2)",
  {
    ":id1" => 302,
    ":id2" => 303,
  }
)
# scan retrieved: Curiosity, Opportunity

Next Steps

As you can see, condition expressions enable you to write more concise code to retrieve data. They also provide querying capabilities unavailable with the original access model such as within-Item references and more flexible conditions with parentheses. In an upcoming blog post, we’ll take a closer look at how we can update existing data through update expressions.