Category: Ruby
General Availability Release of the aws-record Gem
Today, we’re pleased to announce the GA release of version 1.0.0
of the aws-record
gem.
What Is aws-record?
In version 1 of the AWS SDK for Ruby, the AWS::Record
class provided a data mapping abstraction over Amazon DynamoDB operations. Earlier this year, we released the aws-record
developer preview as a separately packaged library to provide a similar data mapping abstraction for DynamoDB, built on top of the AWS SDK for Ruby version 2. After customer feedback and some more development work, we’re pleased to move the library out of developer preview to general availability.
How to Include the aws-record Gem in Your Project
The aws-record
gem is available now from RubyGems:
gem install aws-record
You can also include it in your project’s Gemfile:
# Gemfile gem 'aws-record', '~> 1.0'
This automatically includes a dependency on the aws-sdk-resources
gem, major version 2. Be sure to include the aws-sdk
or aws-sdk-resources
gem in your Gemfile if you need to lock to a specific version, like so:
# Gemfile gem 'aws-record', '~> 1.0' gem 'aws-sdk-resources', '~> 2.5'
Working with DynamoDB Tables Using the aws-record Gem
Defining an Aws::Record Model
The aws-record
gem provides the Aws::Record
module, which you can include in a class definition. This decorates your class with a variety of helper methods that can simplify interactions with Amazon DynamoDB. For example, the following model uses a variety of preset attribute definition helper methods and attribute options:
require 'aws-record' class Forum include Aws::Record string_attr :forum_uuid, hash_key: true integer_attr :post_id, range_key: true string_attr :author_username string_attr :post_title string_attr :post_body string_set_attr :tags, default_value: Set.new datetime_attr :created_at, database_attribute_name: "PostCreatedAtTime" boolean_attr :moderation, default_value: false end
Using Validation Libraries with an Aws::Record Model
The aws-record
gem does not come with a built-in validation process. Rather, it is designed to be a persistence layer, and to allow you to bring your own validation library. For example, the following model includes the popular ActiveModel::Validations
module, and has defined a set of validations that will be run when we attempt to save an item:
require 'aws-record' require 'active_model' class Forum include Aws::Record include ActiveModel::Validations string_attr :forum_uuid, hash_key: true integer_attr :post_id, range_key: true string_attr :author_username string_attr :post_title string_attr :post_body string_set_attr :tags, default_value: Set.new datetime_attr :created_at, database_attribute_name: "PostCreatedAtTime" boolean_attr :moderation, default_value: false validates_presence_of :forum_uuid, :post_id, :author_username validates_length_of :post_title, within: 4..30 validates_length_of :post_body, within: 2..5000 end
Creating a DynamoDB Table for a Model with Aws::Record::TableMigration
The aws-record
gem provides a helper class for table operations, such as migrations. If we wanted to create a table for our Forum
model in DynamoDB, we would run the following migration:
migration = Aws::Record::TableMigration.new(Forum) migration.create!( provisioned_throughput: { read_capacity_units: 5, write_capacity_units: 2 } ) migration.wait_until_available
You can write these migrations in your Rakefile
or as standalone helper scripts for your application. Because you don’t need to update your table definition for additions of non-key attributes, you may find that you’re not running migrations as often for your Aws::Record
models.
Working with DynamoDB Items Using the aws-record Gem
Creating and Persisting a New Item
Using the example model above, once it has been created in the DynamoDB remote end using Aws::Record::TableMigration
(or if it already existed in the remote end), it is simple to create and save a new item:
post = Forum.new( forum_uuid: FORUM_UUID, post_id: 1, author_username: "Author One", post_title: "Hello!", post_body: "Hello, world!" ) post.created_at = Time.now post.save # Performs a put_item call.
You can set attributes when you initialize a new item and with setter methods that are defined for you automatically.
Finding and Modifying an Item
A class-level method #find
is provided to look up items from DynamoDB using your model’s key attributes. After setting a few new attribute values, calling #save
will make an update call to DynamoDB, reflecting only the item changes you’ve made. This is important for users who are fetching items with projections (which may not include all attributes), or using single-table inheritance patterns (who may not have modeled all attributes present in a remote item), to avoid clobbering unmodeled or non-included attribute values.
post = Forum.find(forum_uuid: FORUM_UUID, post_id: 1) post.post_title = "(Removed)" post.post_body = "(Removed)" post.moderation = true post.save # Performs an update_item call on dirty attributes only.
There is also a class-level method to directly build and make an update call to DynamoDB, using key attributes to identify the item and non-key attributes to form the update expression:
Forum.update( forum_uuid: FORUM_UUID, post_id: 1, post_title: "(Removed)", post_body: "(Removed)", moderation: true )
The preceding two code examples are functionally equivalent. You’ll have the same database state after running either snippet.
A Note on Dirty Tracking
In our last example, we talked about how item updates only reflect changes to modified attributes. Users of ActiveRecord
or similar libraries will be familiar with the concept of tracking dirty attribute values, but aws-record
is a bit different. That is because DynamoDB supports collection attribute types, and in Ruby, collection types are often modified through object mutation. To properly track changes to an item when objects can be changed through mutable state, Aws::Record
items will, by default, keep deep copies of your attribute values when loading from DynamoDB. Attribute changes through mutation, like this example, will work the way you expect:
post = Forum.find(forum_uuid: FORUM_UUID, post_id: 1) post.tags.add("First") post.dirty? # => true post.save # Will call update_item with the new tags collection.
Tracking deep copies of attribute values has implications for performance and memory. You can turn off mutation tracking at the model level. If you do so, dirty tracking will still work for new object references, but will not work for mutated objects:
class NoMTModel include Aws::Record disable_mutation_tracking string_attr :key, hash_key: true string_attr :body map_attr :map end item = NoMTModel.new(key: "key", body: "body", map: {}) item.save # Will call put_item item.map[:key] = "value" item.dirty? # => false, because we won't track mutations to objects item.body = "New Body" item.dirty? # => true, because we will still notice reassignment # Will call update_item, but only update :body unless we mark map as dirty explicitly. item.save
Try the aws-record Gem Today!
We’re excited to hear about what you’re building with aws-record
. Feel free to leave your feedback in the comments, or open an issue in our GitHub repo. Read through the documentation and get started!
Introducing the Aws::Record Developer Preview
We are happy to announce that the aws-record
gem is now in Developer Preview and available for you to try.
What Is Aws::Record?
In version 1 of the AWS SDK for Ruby, the AWS::Record
class provided a data mapping abstraction over Amazon DynamoDB operations. As version 2 of the AWS SDK for Ruby was being developed, many of you asked for an updated version of the library.
The aws-record
gem provides a data mapping abstraction for DynamoDB built on top of the AWS SDK for Ruby version 2.
Using Aws::Record
You can download the aws-record
gem from RubyGems by including the --pre
flag in a gem installation:
gem install 'aws-record' --pre
You can also include it in your Gemfile. Do not include a version lock yet, so that bundler
can find the pre-release version:
# Gemfile gem 'aws-record'
Defining a Model
To create an aws-record model, include the Aws::Record
module in your class definition:
require 'aws-record' class Forum include Aws::Record end
This will decorate your class with helper methods you can use to create a model compatible with DynamoDB’s table schemas. You might define keys for your table:
require 'aws-record' class Forum include Aws::Record string_attr :forum_uuid, hash_key: true integer_attr :post_id, range_key: true end
When you use these helper methods, you do not need to worry about how to define these attributes and types in DynamoDB. The helper methods and marshaler classes are able to define your table and item operations for you. The aws-record
gem comes with predefined attribute types that cover a variety of potential use cases:
require 'aws-record' class Forum include Aws::Record string_attr :forum_uuid, hash_key: true integer_attr :post_id, range_key: true string_attr :author_username string_attr :post_title string_attr :post_body datetime_attr :created_at map_attr :post_metadata end
Creating a DynamoDB Table
The aws-record
gem provides a helper class for table operations, such as migrations. If we wanted to create a table for our Forum
model in DynamoDB, we would run the following migration:
require 'forum' # Depending on where you defined the class above. migration = Aws::Record::TableMigration.new(Forum) migration.create!( provisioned_throughput: { read_capacity_units: 10, write_capacity_units: 4 } ) migration.wait_until_available # Blocks until table creation is complete.
Operations with DynamoDB Items
With a model and table defined, we can perform operations that relate to items in our table. Let’s create a post:
require 'forum' require 'securerandom' uuid = SecureRandom.uuid post = Forum.new post.forum_uuid = uuid post.post_id = 1 post.author_username = "User1" post.post_title = "Hello!" post.post_body = "Hello Aws::Record" post.created_at = Time.now post.post_metadata = { this_is_a: "Post", types_supported_include: ["String", "Integer", "DateTime"], how_many_times_ive_done_this: 1 } post.save # Writes to the database.
This example shows us some of the types that are supported and serialized for you. Using the key we’ve defined, we can also find this object in our table:
my_post = Forum.find(forum_uuid: uuid, post_id: 1) my_post.post_title # => "Hello!" my_post.created_at # => #<DateTime: 2016-02-09T14:39:07-08:00 ((2457428j,81547s,0n),-28800s,2299161j)>
You can use the same approach to save changes or, as shown here, you can delete the item from the table:
my_post.delete! # => true
At this point, we know how to use Aws::Record
to perform key-value store operations powered by DynamoDB and have an introduction to the types available for use in our tables.
Querying, Scanning, and Collections
Because it is likely that you’re probably doing Query and Scan operations in addition to key-value operations, aws-record
provides support for integrating them with your model class.
When you include the Aws::Record
module, your model class is decorated with #query
and #scan
methods, which correspond to the AWS SDK for Ruby client operations. The response is wrapped in a collection enumerable for you. Consider the following basic scan operation:
Forum.scan # => #<Aws::Record::ItemCollection:0x007ffc293ec790 @search_method=:scan, @search_params={:table_name=>"Forum"}, @model=Forum, @client=#<Aws::DynamoDB::Client>>
No client call has been made yet: ItemCollection
instances are lazy, and only make client calls only when needed. Because they provide an enumerable interface, you can use any of Ruby’s enumerable methods on your collection, and your result page is saved:
resp = Forum.scan resp.take(1) # Makes a call to the underlying client. Returns a 'Forum' object. resp.take(1) # Same result, but does not repeat the client call.
Because the Aws::Record::ItemCollection
uses version 2 ofthe AWS SDK for Ruby, pagination support is built-in. So, if your operation requires multiple DynamoDB client calls due to response truncation, ItemCollection
will handle the calls required in your enumeration:
def author_posts Forum.scan.inject({}) do |acc, post| author = post.author_username if acc[author] acc[author] += 1 else acc[author] = 1 end acc end end
The same applies for queries. Your query result will also be provided as an enumerable ItemCollection
:
def posts_by_forum(uuid) Forum.query( key_condition_expression: "#A = :a", expression_attribute_names: { "#A" => "forum_uuid" }, expression_attribute_values: { ":a" => uuid } ) end
Given this functionality, you have the flexibility to mix and match Ruby’s enumerable functionality with DynamoDB filter expressions, for example, to curate your results. These two functions return the same set of responses:
def posts_by_author_in_forum(uuid, author) posts_by_forum(uuid).select do |post| post.author_username == author end end def posts_by_author_in_forum_with_filter(uuid, author) Forum.query( key_condition_expression: "#A = :a", filter_expression: "#B = :b", expression_attribute_names: { "#A" => "forum_uuid", "#B" => "author_username" }, expression_attribute_values: { ":a" => uuid, ":b" => author } ) end
Support for Secondary Indexes
Aws::Record
also supports both local and global secondary indexes. Consider this modified version of our Forum table:
require 'aws-record' class IndexedForum include Aws::Record string_attr :forum_uuid, hash_key: true integer_attr :post_id, range_key: true string_attr :author_username string_attr :post_title string_attr :post_body datetime_attr :created_at map_attr :post_metadata global_secondary_index(:author, hash_key: :author_username, projection: { projection_type: "INCLUDE", non_key_attributes: ["post_title"] } ) local_secondary_index(:by_date, range_key: :created_at, projection: { projection_type: "ALL" } ) end
You can see the table’s attributes are the same, but we’ve included a couple potentially useful indexes.
-
:author
: This uses the author name as a partition, which provides a way to search across forums by author user name without having to scan and filter. Take note of the projection, because your global secondary index results will only return the:forum_uuid
,:post_id
,:author_username
, and:post_title
. Other attributes will be missing from this projection, and you would have to hydrate your item by calling#reload!
on the item instance. -
:by_date
: This provides a way to sort and search within a forum by post creation date.
To create this table with secondary indexes, you create a migration like we did before:
require 'indexed_forum' migration = Aws::Record::TableMigration.new(IndexedForum) migration.create!( provisioned_throughput: { read_capacity_units: 10, write_capacity_units: 4 }, global_secondary_index_throughput: { author: { read_capacity_units: 5, write_capacity_units: 3 } } ) migration.wait_until_available
You can use either of these indexes with the query interface:
require 'indexed_forum' def search_by_author(author) IndexedForum.query( index_name: "author", key_condition_expression: "#A = :a", expression_attribute_names: { "#A" => "author_username" }, expression_attribute_values: { ":a" => author } ) )
Secondary indexes can be a powerful performance tool, and aws-record
can simplify the process of managing them.
Get Involved!
Please download the gem, give it a try, and let us know what you think. This project is a work in progress, so we welcome feature requests, bug reports, and information about the kinds of problems you’d like to solve by using this gem. And, as with other SDKs and tools we produce, we’d also be happy to look at contributions.
You can find the project on GitHub at https://github.com/awslabs/aws-sdk-ruby-record
Please reach out and let us know what you think!
AWS SDK for Ruby Office Hour
The AWS SDKs and Tools team invites you to the first-ever online office hour hosted by the maintainers of the AWS SDK for Ruby. It will be held via Google Hangouts at 11:00am-12:00pm PDT (UTC -7:00) on Tuesday 6/30. If you don’t have one already, you will need to create an account with Google to join the video chat.
This first office hour will be driven by customer questions. We expect to focus on questions about the SDK, but any questions related to Ruby development on AWS are welcome. We’re excited to meet you and help you be successful in developing Ruby applications on AWS!
Please register for the event, add it to your calendar, and join the office hour next Monday.
RailsConf 2015 Recap
Recently, Trevor, Loren, and myself from the AWS SDK team attended RailsConf in Atlanta. We had a great time at the conference and enjoyed connecting with many of you there.
Our Rails on Amazon Web Services Workshop
At RailsConf, we ran a workshop called Deploy and Manage Ruby on Rails Apps on AWS. It was an amazing experience for us, with attendees of all experience levels gettings hands-on experience not only deploying to AWS, but learning about the tools we’ve made to help make integrations easier.
For those of you who could not make it, you can still give this workshop a try!
- Detailed step-by-step instructions, the same as we provided to attendees, are available here.
- You can also follow along with the presentation recording on YouTube.
- Code for the sample app is available on GitHub.
- If you’d like to try using Amazon Relational Database Service instead of using AWS OpsWorks managed MySQL, you can reference our blog post on that topic as well.
Continuing the Conversation
We hope to see more of you on the conference trail again soon! Apropos of this, it is worth mentioning that AWS re:Invent registration is open at the time of writing. We will be there, and we hope to see you there!
Uploading Files to Amazon S3
I blogged previously about downloading objects from Amazon S3 using the version 2 AWS SDK for Ruby. It was requested that I write about uploading objects as well.
Managed File Uploads
The simplest and most common task is upload a file from disk to a bucket in Amazon S3. This is very straightforward when using the resource interface for Amazon S3:
s3 = Aws::S3::Resource.new s3.bucket('bucket-name').object('key').upload_file('/source/file/path')
You can pass additional options to the Resource
constructor and to #upload_file
. This expanded example demonstrates configuring the resource client, uploading a public object and then generating a URL that can be used to download the object from a browser.
s3 = Aws::S3::Resource.new( credentials: Aws::Credentials.new('akid', 'secret'), region: 'us-west-1' ) obj = s3.bucket('bucket-name').object('key') obj.upload_file('/source/file/path', acl:'public-read') obj.public_url #=> "https://bucket-name.s3-us-west-1.amazonaws.com/key"
This is the recommended method of using the SDK to upload files to a bucket. Using this approach has the following benefits:
- Manages multipart uploads for objects larger than 15MB.
- Correctly opens files in binary mode to avoid encoding issues.
- Uses multiple threads for uploading parts of large objects in parallel.
Other Methods
In addition to Aws::S3::Object#upload_file
, you can upload an object using #put
or using the multipart upload APIs.
PUT Object
For smaller objects, you may choose to use #put
instead. The #put
method accepts an optional body, which can be a string or any IO object.
obj = s3.bucket('bucket-name').object('key') # from a string obj.put(body:'Hello World!') # from an IO object File.open('/source/file', 'rb') do |file| obj.put(body:file) end
Multipart APIs
I recommend you use #upload_file
whenever possible. If you need to manage large object copies, then you will need to use the multipart interfaces. There are restrictions on the minimum file, and part sizes you should be aware of. Typically these are reserved for advanced use cases.
Feedback
I’d love to hear feedback. If you find the AWS SDK for Ruby lacks a utility for working with Amazon S3, I’d love to hear about it. Please feel free to open a GitHub issue or drop into our Gitter channel.
Verifying Amazon SNS Message Authenticity
You can now use version 2 of the AWS SDK for Ruby to verify the signatures of Amazon SNS messages. To help prevent spoofing attacks, you should verify messages are sent by Amazon SNS.
The new verifier follows the documented best practices for verification, including:
- Always use HTTPS when getting the certificate from Amazon SNS.
- Validate the authenticity of the certificate.
- Verify the certificate was received from Amazon SNS.
Basic Usage
Usage is straightforward. Construct a message verifier and call one of two methods. The given message body should be the JSON document string of the message.
verifier = Aws::SNS::MessageVerifier.new verifier.authentic?(message_body) #=> returns true or false verifier.authenticate!(message_body) #=> returns true or raises a VerificationError
You can use one instance of Aws::SNS::MessageVerifier.new
to verify multiple messages.
Feedback
As always, we love to hear your feedback. It helps us prioritize our development efforts. In fact, this feature was added by customer request. Feel free to join our Gitter channel or open a GitHub issue.
Polling Messages from a Amazon SQS Queue
We’ve recently added a utility class to the AWS SDK for Ruby that makes it easy to poll an Amazon SQS queue for messages.
poller = Aws::SQS::QueuePoller.new(queue_url) poller.poll do |msg| puts msg.body end
Messages are automatically deleted from the queue at the end of the block. This tool supports receiving and deleting messages in batches, long-polling, client-side tracking of stats, and more.
Long Polling
By default, messages are received using long polling. This method will force a default :wait_time_seconds
of 20 seconds. If you prefer to use the queue default wait time, then pass a nil
value for :wait_time_seconds
.
# disables 20 second default, use queue ReceiveMessageWaitTimeSeconds attribute poller.poll(wait_time_seconds:nil) do |msg| # ... end
When disabling :wait_time_seconds
by passing nil
, you must ensure the queue ReceiveMessageWaitTimeSeconds
attribute is set to a non zero value, or you will be short polling. This will trigger significantly more API calls.
Batch Receiving Messages
You can specify a maximum number of messages to receive with each polling attempt via :max_number_of_messages
. When this is set to a positive value, greater than 1, the block will receive an array of messages, instead of a single message.
# receives and yields up to 10 messages at a time poller.poll(max_number_of_messages:10) do |messages| messages.each do |msg| # ... end end
The maximum value for :max_number_of_messages
is enforced by Amazon SQS.
Visibility Timeouts
When receiving messages, you have a fixed amount of time to process and delete each message before it is added back into the queue. This is the visibility timeout. By default, the queue’s VisibilityTimeout
attribute is used. You can provide an alternative visibility timeout when polling.
# override queue visibility timeout poller.poll(visibility_timeout:10) do |msg| # do work ... end
You can reset the visibility timeout of a single message by calling #change_message_visibility
. This is useful when you need more time to finish processing the message.
poller.poll do |msg| # do work ... # need more time for processing poller.change_message_visibility(msg, 60) # finish work ... end
If you change the visibility timeout of a message to zero, it will return to the queue immediately.
Deleting Messages
Messages are deleted from the queue when the block returns normally.
poller.poll do |msg| # do work end # messages deleted here
You can skip message deletion by passing skip_delete: true
. This allows you to manually delete the messages using {#deletemessage}, or {#deletemessages}.
# single message poller.poll(skip_delete: true) do |msg| poller.delete_message(msg) # if successful end # message batch poller.poll(skip_delete: true, max_number_of_messages:10) do |messages| poller.delete_messages(messages) end
Another way to manage message deletion is to throw :skip_delete
from the poll block. You can use this to choose when a message, or message batch is deleted on an individual basis:
poller.poll do |msg| begin # do work rescue # unexpected error occurred while processing messages, # log it, and skip delete so it can be re-processed later throw :skip_delete end end
Terminating the Polling Loop
By default, polling will continue indefinitely. You can stop the poller by providing an idle timeout or by throwing :stop_polling
from the {#before_request} callback.
:idle_timeout
This is a configurable, maximum number of seconds to wait for a new message before the polling loop exists. By default, there is no idle timeout.
# stops polling after a minute of no received messages poller.poll(idle_timeout: 60) do |msg| # ... end
:stop_polling
If you want more fine-grained control, you can configure a before request callback to trigger before each long poll. Throwing :stop_polling
from this callback will cause the poller to exit normally without making the next request.
# stop after processing 100 messages poller.before_request do |stats| throw :stop_polling if stats.receive_message_count >= 100 end poller.poll do |msg| # do work ... end
Tracking Progress
The poller will automatically track a few statistics client-side in a PollerStats object. You can access the poller stats three ways:
- The first block argument of {#before_request}
- The second block argument of {#poll}.
- The return value from {#poll}.
Here are examples of accessing the statistics.
-
Configure a {#before_request} callback.
poller.before_reqeust do |stats| logger.info("requests: #{stats.request_count}") logger.info("messages: #{stats.received_message_count}") logger.info("last-timestamp: #{stats.last_message_received_at}") end
-
Accept a second argument in the poll block, for example:
poller.poll do |msg, stats| logger.info("requests: #{stats.request_count}") logger.info("messages: #{stats.received_message_count}") logger.info("last-timestamp: #{stats.last_message_received_at}") end
-
Return value:
stats = poller.poll(idle_timeout:10) do |msg| # do work ... end logger.info("requests: #{stats.request_count}") logger.info("messages: #{stats.received_message_count}") logger.info("last-timestamp: #{stats.last_message_received_at}")
Feedback
Let us know what you think about the new queue poller. Join the conversation in our Gitter channel or open a GitHub issue.
Announcing the aws-sdk-rails Gem
With the release of V2 of the AWS SDK for Ruby, we’ve received customer feedback asking for support for the Ruby on Rails integration features provided by V1 of the SDK.
Today, we’re excited to announce the release of the aws-sdk-rails gem, available now via RubyGems and, of course, on GitHub.
To get started, add the aws-sdk-rails
gem to your Gemfile:
gem 'aws-sdk-rails', '~> 1.0'
ActionMailer and Amazon Simple Email Service (SES)
The gem will automatically configure Rails to include an :aws_sdk
delivery method for ActionMailer, that uses Amazon SES as a backend. It is simple to configure Rails to use this delivery method:
# config/application.rb config.action_mailer.delivery_method = :aws_sdk
The aws-sdk-rails gem will use the AWS SDK for Ruby V2’s SES client automatically for any mail delivery event.
Logging
The gem will automatically wire the AWS SDK for Ruby’s logger to use Rails.logger
by default.
You can customize the SDK log level and an optional log formatter in a config initializer:
# config/initializers/aws-sdk.rb # log level defaults to :info Aws.config[:log_level] = :debug
It is important to understand that all SDK log messages are logged at the same log level. Why is this important? When you set the Rails log level, you’re muting all log messages below that log level. So, if you want to, for example, only see SDK log messages in development, you might set the SDK log level to :debug
as shown above, and set the Rails logger to show debug in development.
Credentials
The AWS SDK for Ruby will attempt locate credentials by searching the following locations:
-
ENV['AWS_ACCESS_KEY_ID']
andENV['AWS_SECRET_ACCESS_KEY']
- The shared credentials ini file at ~/.aws/credentials
- From an instance profile when running on Amazon EC2
If you need to manually configure credentials, you should add them to your initializer:
# config/initializers/aws-sdk.rb Aws.config[:credentials] = Aws::Credentials.new(access_key, secret)
Learn more about credentials in the AWS SDK for Ruby V2.
Never commit your credentials to source control. Besides being a security risk, it makes it very difficult to rotate your credentials.
Available Now
The aws-sdk-rails
gem is available now.
As always, we’d love to hear your feedback, and welcome any Issues or Pull Requests at the aws-sdk-rails GitHub repo.
Amazon DynamoDB Document API in Ruby (Part 3 – Update Expressions)
As we showed in previous posts, it’s easy to put JSON items into Amazon DynamoDB, retrieve specific attributes with projection expressions, and fetch only data that meet some criteria with condition expressions. Now, let’s take a look at how we can conditionally modify existing items with Update Expressions. (Note: this code uses the same ProductCatalog table we used in Parts 1 and 2).
In the following examples, we use the following helper method to perform conditional updates. It performs the UpdateItem operation with return_values set to return the old item. We also use the GetItem operation so the method can return both the old and new items for us to compare. (If the update condition in the request is not met, then the method sets the returned old item to nil.)
def do_update_item(key_id, update_exp, condition_exp, exp_attribute_values) begin old_result = @dynamodb.update_item( :update_expression => update_exp, :condition_expression => condition_exp, :expression_attribute_values => exp_attribute_values, :table_name => "ProductCatalog", :key => { :Id => key_id }, :return_values => "ALL_OLD", ).data.attributes rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException old_result = nil puts "Condition not met" end new_result = @dynamodb.get_item( :table_name => "ProductCatalog", :key => { :Id => key_id }, :consistent_read => true ).data.item return old_result, new_result end
Using Conditional Update Expressions
Updates in DynamoDB are atomic. This allows applications to concurrently update items without worrying about conflicts occurring. For example, the following code demonstrates maintaining a MAX value in DynamoDB with a conditional update using SET. Note that, because DynamoDB is schema-less, we don’t need to define the HighestRating attribute beforehand. Instead, we create it on the first call.
# storing a "max" value with conditional SET # SET attribute if doesn't exist, otherwise SET if stored highest rating < this rating def update_highest_rating(rating) do_update_item(303, "SET HighestRating = :val", "attribute_not_exists(HighestRating) OR HighestRating < :val", { ":val" => rating } ) end # multiple threads trying to SET highest value (ranging from 0 to 10) threads = [] (0..10).to_a.shuffle.each { |i| # some number of "Condition not met" depending on shuffled order puts i threads[i] = Thread.new { update_highest_rating(i) } } threads.each {|t| t.join} # fetch the item and examine the HighestRating stored puts "Max = #{@dynamodb.get_item( :table_name => "ProductCatalog", :key => { :Id => 303 } ).data.item["HighestRating"].to_i}" # Max = 10
We can also use update expressions to atomically maintain a count and add to a set:
# ADD to intialize/increment and add to set threads = [] 20.times do |i| threads[i] = Thread.new { do_update_item(303, "ADD TimesViewed :val, Tags :was_here", nil, # no condition expression { # Each of the 20 threads increments by 1 ":val" => 1, # Each thread adds to the tag set # Note: type must match stored attribute's type ":was_here" => Set.new(["#Thread#{i}WasHere"]) } ) } end threads.each {|t| t.join} # fetch the item and examine the TimesViewed attribute item = @dynamodb.get_item( :table_name => "ProductCatalog", :key => { :Id => 303 } ).data.item puts "TimesViewed = #{item["TimesViewed"].to_i}" # TimesViewed = 20 puts "Tags = #{item["Tags"].inspect}" # Tags = #<Set: {"#Mars", "#MarsCuriosity", "#StillRoving", ..each thread was here...}>
Similarly, we can decrement the count and remove from the set to undo our previous operations.
# Undo the views and set adding that we just performed threads = [] 20.times do |i| threads[i] = Thread.new { do_update_item(303, "ADD TimesViewed :val DELETE Tags :was_here", nil, # no condition expression { # Each of the 20 threads decrements by 1 ":val" => -1, # Each thread removes from the tag set # Note: type must match stored attribute's type ":was_here" => Set.new(["#Thread#{i}WasHere"]) } ) } end threads.each {|t| t.join} # fetch the item and examine the TimesViewed attribute item = @dynamodb.get_item( :table_name => "ProductCatalog", :key => { :Id => 303 } ).data.item puts "TimesViewed = #{item["TimesViewed"].to_i}" # TimesViewed = 0 puts "Tags = #{item["Tags"].inspect}" # Tags = #<Set: {"#Mars", "#MarsCuriosity", "#StillRoving"}>
We can also use the REMOVE keyword to delete attributes, such as the HighestRating and TimesViewed attributes we added in the previous code.
# removing attributes from items old_and_new = do_update_item(303, "REMOVE HighestRating, TimesViewed", nil, # no condition expression nil, # no attribute expression values ) puts "OLD HighestRating is nil ? #{old_and_new[0]["HighestRating"] == nil}" #=> false puts "OLD TimesViewed is nil ? #{old_and_new[0]["TimesViewed"] == nil}" #=> false puts "NEW HighestRating is nil ? #{old_and_new[1]["HighestRating"] == nil}" #=> true puts "NEW TimesViewed is nil ? #{old_and_new[1]["TimesViewed"] == nil}" #=> true
Conclusion
We hope this series was helpful in demonstrating expressions and how they allow you to interact with DynamoDB more flexibly than before. We’re always interested in hearing what developers would like to see in the future, so let us know what you think in the comments or through our forums!
Amazon DynamoDB Document API in Ruby (Part 2 – Condition Expressions)
As we showed in the previous post, it’s easy to put JSON items into Amazon DynamoDB and retrieve specific attributes with projection expressions. Condition Expressions provide a more flexible and SQL-like way to retrieve only the items you want from DynamoDB. First, let’s put a few more items into DynamoDB using a BatchWriteItem operation. (Note: this code uses the same ProductCatalog table we used in Part 1)
# add some more items @dynamodb.batch_write_item( :request_items => { "ProductCatalog" => [ {:put_request => { :item => { Id: 300, Title: "Sojourner", Description: "Mars Pathfinder robotic Mars rover", Price: BigDecimal.new("2.65e8"), LaunchDate: { M: 12, D: 4, Y: 1996 }, LostCommunicationDate: { M: 9, D: 27, Y: 1997 }, Features: { Rover: true, }, NumberInStock: 10, OrdersPlaced: 3, Tags: ["#Mars", "#InStarTrekSeason4", "#InRedPlant2000", "#LostComms"], }}}, {:put_request => { :item => { Id: 301, Title: "Spirit", Description: "Mars Exploration Rover – A", Price: BigDecimal.new("4.1e8"), LaunchDate: { M: 6, D: 10, Y: 2003 }, LostCommunicationDate: { M: 3, D: 22, Y: 2010 }, Features: { Rover: true, }, NumberInStock: 10, OrdersPlaced: 5, Tags: Set.new(["#Mars", "#StuckOnMars", "#LostComms"]), }}}, {:put_request => { :item => { Id: 302, Title: "Opportunity", Description: "Mars Exploration Rover – B", Price: BigDecimal.new("4.1e8"), LaunchDate: { M: 7, D: 7, Y: 2003 }, LostCommunicationDate: nil, Features: { Rover: true, }, NumberInStock: 10, OrdersPlaced: 10, Tags: Set.new(["#Mars", "#StillRoving"]), }}}, {:put_request => { :item => { Id: 303, Title: "Curiosity", Description: "car-sized robotic rover", Price: BigDecimal.new("2.5e9"), LaunchDate: { M: 11, D: 26, Y: 2011 }, LostCommunicationDate: nil, Features: { Rover: true, RoboticArm: true, }, NumberInStock: 0, OrdersPlaced: 30, Tags: Set.new(["#Mars", "#MarsCuriosity", "#StillRoving"]), }}}, ] } )
Using Condition Expressions
We could also use condition expressions on the results of Query, but since we’re using a simple data model (only have hash key on product Id), we demonstrate this with scans. We use the following helper method to perform the scan and format the product titles returned:
def do_scan(filter_exp, exp_attribute_values) result = @dynamodb.scan( :expression_attribute_values => exp_attribute_values, :filter_expression => filter_exp, # Condition Expressions are supplied through the FilterExpression parameter :projection_expression => "Title", :table_name => "ProductCatalog" ).data.items # format all retrieved titles into a single line return "scan retrieved: #{(result.map { |item| item["Title"] }).join(", ")}" end
Let’s look at some example expressions and the results they return from our current ProductCatalog table:
# All products that don't have a launch month of November (11) puts do_scan( "LaunchDate.M <> :m", { ":m" => 11 } ) # scan retrieved: 20-Bicycle 205, Opportunity, Spirit, Sojourner # All rover products that don't have a launch month of November puts do_scan( "attribute_exists(Features.Rover) AND LaunchDate.M <> :m", { ":m" => 11, } ) # scan retrieved: Opportunity, Spirit, Sojourner # Non-rovers puts do_scan( "attribute_not_exists(Features.Rover)", nil ) # scan retrieved: 20-Bicycle 205 # mid-range rovers or inexpensive products puts do_scan( "(Price BETWEEN :low AND :high) OR Price < :verylow", { ":verylow" => BigDecimal.new("1e8"), ":low" => BigDecimal.new("3e8"), ":high" => BigDecimal.new("5e8") } ) # scan retrieved: 20-Bicycle 205, Opportunity, Spirit # within-Item referencing: more orders placed than in stock puts do_scan( "OrdersPlaced > NumberInStock", nil ) # scan retrieved: Curiosity # string prefixing puts do_scan( "begins_with(Title, :s)", { ":s" => "S", } ) # scan retrieved: Spirit, Sojourner # contains puts do_scan( "contains(Tags, :tag1) AND contains(Tags, :tag2)", { ":tag1" => "#StuckOnMars", ":tag2" => "#LostComms", } ) # scan retrieved: Spirit # contains (Note: "Tags" is a list for Sojourner) puts do_scan( "contains(Tags, :tag1)", { ":tag1" => "#LostComms", } ) # scan retrieved: Spirit, Sojourner # in operator puts do_scan( "Id in (:id1, :id2)", { ":id1" => 302, ":id2" => 303, } ) # scan retrieved: Curiosity, Opportunity # equivalently, with parentheses puts do_scan( "(Id = :id1) OR (Id = :id2)", { ":id1" => 302, ":id2" => 303, } ) # scan retrieved: Curiosity, Opportunity
Next Steps
As you can see, condition expressions enable you to write more concise code to retrieve data. They also provide querying capabilities unavailable with the original access model such as within-Item references and more flexible conditions with parentheses. In an upcoming blog post, we’ll take a closer look at how we can update existing data through update expressions.