AWS Partner Network (APN) Blog

In an effort to help AWS Partner Network (APN) Partners showcase their customer obsession through AWS Certification achievement, we’ve launched APN Certification Distinctions. Historically, AWS Certifications have been an individual achievement. With APN Certification Distinctions, APN Partners will now have a chance to showcase how many active AWS Certifications their company has collectively achieved, providing additional ways to highlight the value AWS Certification brings to their customers.

APN Certification Distinctions will be distributed to qualified APN Partners in the following increments: 50, 100, 200, 300, 400, and 500. Certification data is collected via SFDC and will only count active AWS Certifications. If your company qualifies for this distinction, an APN representative will reach out to your company contact soon. Companies can showcase their distinction on their website, event kiosks, signage, customer emails, etc. Learn more about APN Certification Distinctions here.

Want to dive deep and learn more about AWS Database offerings? This webinar series will provide you an exclusive deep dive into Amazon Aurora, Amazon Redshift, and Amazon DynamoDB. These webinars feature technical sessions led by AWS solutions architects and engineers, live demonstrations, customer examples, and Q&A with AWS experts.

Check out these upcoming webinars and register to attend!

Amazon Aurora Architecture Overview

September 26, 2016 | 11:30am-12:30pm PDT

This webinar provides a deep architecture overview of Amazon Aurora. Partners attending this webinar will learn how Amazon Aurora differs from other relational database engines with special focus on features such as High Availability (HA) and 5x Performance compared to MySQL.

Register Here >>

Understanding the Aurora Storage Layer

October 3, 2016 | 11:30am-12:30pm PDT

This webinar will dive deep into the Amazon Aurora Storage Layer. Attendees will receive a technical overview of performance and availability features as well as insights into future enhancements.

Register Here >>

Amazon Aurora Migration Best Practices

October 10, 2016 | 11:30am-12:30pm PDT

This webinar will cover best practices for migrating from Oracle to Amazon Aurora. Partners attending this webinar will learn about common migration opportunities, challenges, and how to address them.

Register Here >>

Selecting an AWS Database

October 17, 2016 | 11:30am-12:30pm PDT

Amazon Aurora, Amazon Redshift, and Amazon DynamoDB are managed AWS database offerings well-suited for a variety of use cases. In this webinar, partners will learn best practices for selecting a database and how each offering fits into the broader AWS portfolio of database services.

Register Here >>

We’ll be hosting more educations webinars for APN Partners throughout the end of the year. Stay tuned to the APN Blog for more information!

By Mike Deck, Partner Solutions Architect, AWS

When AWS Lambda was launched in 2014, it unlocked an ability for AWS customers and partners to implement full-featured, scalable solutions without the need to deploy or manage any servers. I work with many SaaS partners who are now leveraging the serverless model for various components of their architecture. The first step in this journey is often to re-engineer ancillary workloads that can be easily re-implemented without servers. This can afford immediate reductions in infrastructure costs and operational surface area as well as provide valuable experience in building and running systems using this new paradigm.

Signiant, an Advanced APN Technology Partner, Digital Media Competency Partner, and Storage Competency Partner, is a textbook example of a firm who has put this pattern into practice. Over time, Signiant has re-architected its solution on AWS that leverages the bounce and delivery notifications provided by Amazon Simple Email Service (Amazon SES), via an Amazon Simple Notification Service (Amazon SNS) topic. Each iteration of the company’s system has made improvements to the scalability and operational efficiency of the solution, culminating in a simple, Lambda-based serverless architecture. In this post, I will walk through how Signiant has re-architected its SaaS solution on AWS to take advantage of the capabilities of AWS Lambda and a serverless architecture.

Solution Overview

Signiant’s SaaS solution on AWS is called Media Shuttle. This product is used pervasively within the media and entertainment industry to quickly transfer very large files. Using a simple browser plugin or mobile app, users can send or share files of any size through a simple portal. Media Shuttle takes care of the transfer acceleration, security, scalability, elasticity, and resource management on the user’s behalf.

Architecture Evolution

One key feature of Media Shuttle is its delivery notification system, built on Amazon SES. When a file becomes available, the system will send an email to the user with a secure link for downloading the content. These notification emails will occasionally bounce or be caught in spam filtering systems which prevents users from retrieving their files, and generally results in a support call from the sender to figure out why the email was never received.

To improve the support team’s ability to resolve these issues while maintaining the privacy of the sender and email content, Signiant developed a simple system for tracking email bounces that has evolved over time. The initial solution, depicted below, was to subscribe an internal email distribution list to the Amazon SNS topic that received the bounce notifications. This provided simple alerts to the support team when emails bounced and was very easy to implement, but it presented scalability problems as adoption of the product grew. Pretty soon, there were thousands of notifications flooding the support team’s inboxes, and searching for a given customer’s email quickly became cumbersome.

In the next iteration of the solution, the email list was replaced by a database-backed web application running on Amazon Elastic Compute Cloud (Amazon EC2). The bounce notifications were delivered via the Amazon SNS topic to an Amazon Simple Queue Service (Amazon SQS) queue. The application would poll the queue and update a local database that the support team could then search through a simple web UI. Shortly after this version of the system was released, SES added the ability to capture notifications for deliveries in addition to bounces. Signiant added these notifications to the system as well, so that support engineers could see successful delivery statuses in addition to bounces.

The v2 architecture shown above worked well. It was more scalable than using an email distribution list, and the search capabilities were vastly improved. Despite these functional improvements, the new system required more maintenance than the team would have liked. They now had an additional server they were running just for this process, and the database they had chosen was having difficulties managing the increasing load. To optimize the system further, the team decided to re-engineer its solution to take advantage of the benefits of AWS Lambda and a serverless architecture.

The team designed a completely serverless architecture using Lambda to host the message processing logic and Amazon DynamoDB for its database. In the current architecture, instead of a PHP process polling a queue, they have a simple Lambda function written in Python subscribed to the SNS topics fed by SES. The Lambda function was easy to develop based on their existing PHP application that processed SQS messages. The relational database has been replaced by a DynamoDB table which is trivial to scale as the number of emails tracked continues to grow.

Signiant’s current architecture for capturing email status is depicted above. While this system captures delivery status of SES emails, the pattern being employed is extremely versatile and can be applied to any event-driven data processing workflow. By moving to a serverless architecture, Signiant not only decreased the direct cost of running this system, but also removed the operational overhead of managing a one-off server for this isolated task. “Porting our previous message processor to run on Lambda was really straightforward and the new design is much simpler and more robust than our previous server-based system,” said Dave North, the Director of DevOps at Signiant. The new architecture also eliminated the scaling concerns present in the other versions of the system. Using AWS Lambda, the message processors now scale seamlessly without any additional configuration or management, and the database’s throughput can be increased with a simple parameter update.

Conclusion

In this post, I’ve walked through how Signiant has evolved its architecture over time to take advantage of a serverless architecture design. Whether event data is delivered via an SNS topic as is the case here, sent directly to Lambda through a direct service integration as is the case with Amazon Simple Storage Service (Amazon S3), or generated from your own applications using the AWS SDK, you can build systems to capture, process and report on those events using this same basic architecture.

If you would like to see working code samples for the system discussed in this post, Signiant has open sourced both the code for the Lambda functions (https://github.com/Signiant/aws-ses-recorder) and the reporting console application (https://github.com/Signiant/aws-ses-recorder-query-tool). Note that these are not managed by AWS. You can also get started building you own serverless SES notification handler using the SES notifications blueprint in the Lambda console.

Each month, our goal is to provide background on a few APN Partner solutions that Partner SA team members want to highlight. You’ll hear from the SAs themselves about what the solution does, and what they find to be interesting about the solution. In July, we told you about Opsee, Splunk, and Twistlock. While we’re admittedly a few days behind for August, we want to make sure we can provide you with an update for both August and September. In this recap, my colleagues Ian Scofield and Juan Villa discuss solutions from APN Technology Partners Freshservice, Fugue, and Zerto.[1]

Gain Additional Visibility into Your AWS Assets with Freshservice

By Ian Scofield

If your organization is taking advantage of the AWS Cloud, you’ll recognize the constant need to stay on top of all your cloud resources. Freshservice, a service desk and IT service management solution, provides a wealth of services such as incident management, a service desk software, change management, and more. Taking these features one step further, Freshservice also ships with AWS integrated right into the service desk configuration management database (CMDB). Manage your AWS resources and other assets throughout their lifecycle from the single window of your service desk.

By integrating your existing AWS account with Freshservice, you can identify questions with your various Amazon EC2, Amazon RDS, and Amazon EBS resources by associating the assets with incidents and vice versa. In addition, you can use Freshservice to help associate costs with different assets, which will help you better track spending and usage.

Through Amazon CloudWatch integration, you don’t have to juggle multiple tools to keep an eye on your cloud assets.  Freshservice helps you to focus in on critical notifications, as well as the ability to handle multiple alarms about your AWS assets within the service desk. You can convert Amazon CloudWatch alerts to Freshservice tickets to help prioritize and automate escalation tasks and to simplify resource tracking and management from a single platform. Freshservice provides efficient incident, asset, and problem management of your cloud resources and services.

You can also watch these videos about integrating your AWS account with Freshservice and using Amazon CloudWatch alerts from the Freshservice desk here. For more information about Freshservice or to try it out, visit https://freshservice.com/.

Fugue – Validate, Build, and Maintain Cloud Workload Infrastructure

By Ian Scofield

If you’re looking for a system that will allow you to focus less on building and maintaining infrastructure and more on your products, you may want to check out Fugue. Deploying and maintaining your cloud infrastructure can be complex, especially at scale or if you have to meet compliance requirements. Fugue helps solve this by validating, building, and maintaining your cloud infrastructure, which allows you to focus on the things that are relevant to your core product or service.

Fugue is described as an “infrastructure-level cloud operation system,” meaning that it allows you to declare your infrastructure in a meaningful and simple way, in addition to ensuring that it stays that way. It reduces deployment failures by allowing you to “compile” your infrastructure to detect any issues or errors in design time, rather than run time. For example, if you declare a VPC with a CIDR block that is too large, compiling your Fugue template will provide you with immediate meaningful feedback (for example, saying your CIDR block doesn’t match the /16-/28 size allowed in a VPC definition), without having to wait for the deployment to be pushed, to fail, and to roll back.

Fugue can leverage libraries that limit the AWS services that a user can interact with. This allows for fine-grained access to satisfy compliance requirements, internal limitations, cost savings, etc.  This allows you to enforce compliance at design time, before any infrastructure is created that would break compliance requirements.

Fugue continuously monitors your infrastructure to ensure that it stays exactly the way you designed it.  For example, if a security group was accidentally opened to the world, Fugue will automatically reconfigure it to its initially prescribed state.  This continuous monitoring and remediation of any modifications can eliminate drift in configuration over time due to manual modifications or other reasons, as well as continuously enforcing compliance requirements.

For a deeper look at how Fugue can improve your development, deployment, and operations  process, visit https://fugue.co/ where you can register for a launch webinar or schedule a demo to see Fugue in action.

Real Time Disaster Recovery of VMware and Hyper-V Clusters with AWS Storage Competency Partner Zerto and AWS

By Juan Villa 

Most of us have heard a horror story or two of system failures and human errors that led to costly, and sometimes irrecoverable, business disasters. When operating business critical workloads, there’s nothing more important than an effective Disaster Recovery strategy.

Does your business or organization run critical workloads on a VMware vSphere or Microsoft Hyper-V virtual machine cluster? If so, have you heard of Zerto’s Virtual Replication product with support for AWS as a target recovery site?

APN Partner Zerto, also an AWS Storage Competency Partner, provides a solution called Zerto Virtual Replication (ZVR) that enables a user to setup real-time replication of data between a source site running either vSphere or Hyper-V and the AWS cloud. To accomplish this, Zerto’s Virtual Replication streams data, as it’s being generated at the source site, to the Zerto’s Cloud Appliance (ZCA) running on AWS. The ZCA receives the streaming data and stores it in S3 while keeping a ledger for point-in-time recovery. This real-time replication allows you to achieve an RPO (Recovery Point Objective) of seconds, and a full recovery of a large production cluster in less than 1 hour.

One of the benefits of using AWS as a target site for disaster recovery is not having to pay for compute resources until you need them. This can lead to significant cost savings when compared to the traditional model of operating a fully populated data center for the sole purpose of disaster recovery. As a result, using Zerto and AWS in your disaster recovery strategy is both a cost effective and reliable way of mitigating impact to your business or organization in the event of unexpected failure.

For more information on Zerto’s Virtual Replication and Cloud Appliance please visit http://www.zerto.com/.

[1] Please note that these posts are informative in nature, and are not endorsements from AWS of the solutions highlighted, which are built, released, and managed by the third-party.

The following is a guest post from our friends at AWS Security Competency Partner Sophos. 

Fun fact: Sophos is one of the first APN Partners to be featured on “This Is My Architecture”, a new video series that highlights innovative AWS architectural solutions. Check out the Sophos video:

Many of our more security conscious customers in AWS consider controlling outbound traffic just as important as controlling inbound traffic, especially for environments that handle sensitive data. This type of control is called “egress filtering” and has long been a recommendation by security experts like the SANS Institute[1] and required by organizations that need to maintain PCI DSS compliance.

In AWS you can scale your infrastructure to support both inbound and outbound traffic demands by using Auto Scaling groups, Elastic Load Balancing (ELB), and NAT Gateway, but how do you also scale your security layer to support both ingress and egress filtering? To help customers with this need, Sophos has released a new feature called Outbound Gateway (OGW) in its Unified Threat Management (UTM) solution available on AWS. OGW extends UTM’s ability to scale and inspect inbound traffic to now scale and inspect outbound traffic as well, helping you to secure egress traffic in AWS.[2]

Sophos UTM

Sophos UTM on AWS is a security solution in the AWS Marketplace that provides a suite of tools to help you control, protect, and report on traffic entering and leaving your Virtual Private Clouds (VPCs). Sophos UTM combines multiple security tools into one solution and supports the following use cases:

• Next-Gen Firewall
• Inline Intrusion Prevention System (IPS)
• Virtual Private Network (VPN) gateway
• Advanced Threat Protection (ATP)
• Sandbox capabilities
• Web content filtering
• Web Application Firewall (WAF)
• Outbound Gateway – New

Sophos UTM integrates with AWS services like Auto Scaling, CloudWatch, and ELB to ensure inbound traffic is distributed for redundancy and scalability and also scales your security with your infrastructure to inspect your traffic before it leaves your AWS environment.

Outbound Gateway (OGW)

OGW was designed to inspect traffic leaving your VPC using a new routing mechanisms for active-passive or active-active routing. This routing allows Sophos UTM to distribute and inspect traffic leaving your private VPC subnets providing you with redundancy and an easy way to scale your security for egress filtering.

This functionality was primarily designed to help you secure Amazon WorkSpaces environments. Last year during re:Invent 2015, AWS presented how Amazon.com is providing employees, contractors, and vendors a secure remote desktop environment with Amazon WorkSpaces. As part of that move, Amazon.com showed how they used Sophos UTM to restrict network access and provide content filtering for remote users.

The following diagram shows the architecture presented in re:Invent 2015 ISM403.

Figure 1: AWS Architecture presented in re:Invent 2015 ISM403, “How Amazon.com Is Moving to Amazon WorkSpaces” – view the presentation here. 

Use Cases for Sophos UTM OGW

In order to support egress filtering, Sophos UTM has to ensure that outbound traffic can be distributed for scale and fault tolerance. OGW uses a combination of GRE tunnels, health checks, and automated route adjustments to support these requirements.

In a typical VPC, the default route for private subnets is static because the route entry points to the Elastic Network Interface (ENI) of a single instance such as a NAT device or NAT Gateway. Additionally, the route can only point to one destination, which may reduce scalability. In contrast, inbound traffic is straightforward using DNS or ELB where incoming traffic can be balanced across multiple interfaces.

To help scale outbound traffic, Sophos UTM deploys an OGW instance that acts as the default gateway for each private VPC subnet you designate. These VPC subnets can be WorkSpaces subnets or any other private VPC subnet you choose. Sophos UTM then establishes GRE tunnels from each OGW instance to the UTM Workers that reside in an Auto Scaling group.

Inbound traffic follows the normal route via ELB, which distributes traffic across multiple Sophos UTM Workers for inspection. Afterwards, the Sophos UTM workers forward traffic to your instances via an internal load balancer or to your WorkSpaces VPC subnet. However, outbound traffic now takes a new route via the GRE tunnels, which distributes the traffic across the same Sophos UTM Workers. The Sophos UTM Workers then inspect the traffic before it leaves your infrastructure. Additionally, the Sophos UTM Workers are configured in Auto Scaling groups which allow the solution to scale up and down for both inbound and outbound traffic.

The following diagram is a reference architecture using Sophos UTM for inbound and outbound filtering.

Figure 2: Reference Architecture for Sophos UTM inbound & outbound filtering

You can deploy multiple OGWs in the same VPC subnet for redundancy, and Sophos UTM will perform health checks against the OGW instances to ensure normal operation. If an OGW instance fails a health check, Sophos UTM automatically reroutes the traffic to another OGW instance, preventing any outbound interruption.

You can also route traffic for inspection between VPCs using VPC Peering since the GRE tunnels between the OGW instances and Workers support peered connections. This allows you to deploy a central set of outbound filters for many VPCs.

Getting Started

To get started using Sophos UTM OGW, first let’s review the different components used for the solution and information you’ll need beforehand:

• UTM Controller for central administration and reporting of all UTM Workers
• UTM Workers reside in an Auto Scaling group and inspect all traffic (both inbound and outbound)
• OGWs reside in VPC private subnets and act as the default gateway for all outbound traffic
• GRE Tunnels route outbound traffic from the OGWs to Workers

Next, you’ll need the following information before deploying OGW. For information on creating VPC subnets, please refer to Your VPC and Subnets.

• VPC subnet(s) in which you want to deploy OGW (We recommend using different AZs for each VPC Subnet)
• VPC subnet route table(s) for those subnets (you cannot use the route table labeled MAIN)

Once you have the VPC subnets and route tables, you’re ready to start. If you plan on deploying the OGW instances in the same VPC subnet as the UTM Workers, you can deploy OGW directly from the UTM Controller. If you plan on deploying the OGW instances in different VPC subnets, please refer to our Sophos knowledge base article.

Automatically Deploy OGWs from Sophos UTM Controller

To deploy an OGW using the Sophos UTM Controller, preform the following steps:

1. Access the UTM WebGUI (https://ip_address:4444)
2. Browse to the Network Protection> Outbound Gateway for AWS
3. Click on the New Outbound Gateway A new definition window will appear with the following fields:
   • Failover Group: defines the OGWs that will provide failover for specific subnets (Note by default all OGWs reside in the same failover group)
   • Group Name: only used if the Failover Group option is used.
   • Position: gateways can be positioned for priority, i.e., gateways listed at the top will have higher priority.
   • Resource Management: when checked this option automatically creates the OGW instance.
   • AWS Subnet ID: Enter the OGW subnet ID
   • Networks: Create or select Client Subnet definitions
   • Gateway Network Prefix: Default prefix is 240.0.0.0/8 used for the GRE tunnels. You only need to change this setting if the network address is already in use.

Figure 3: OGW details

Once you create the OGW instance, Resource Management launches the OGW CloudFormation stack, which will match the CloudFormation status section of AWS Management Console.

Figure 4: Resource Management status

Figure 5: CloudFormation stack

After the OGW instance has fully launched, the Sophos OGW definition must be enabled before it will update the VPC route table.

Figure 6: OGW status

The VPC subnet will now show an updated route table with OGW as the default gateway.

Figure 7: Route table

Once you’ve configured Sophos OGW, you’re ready to start creating rules to inspect outbound traffic. Refer to our Sophos UTM Administration Guide under the Web Protection and the Advanced Threat Protection sections for setting up rules for egress traffic.

Available Today

OGW is already available in the AWS Marketplaces and supports Pay-As-You-Go and Bring Your Own License (BYOL) licensing options. To try Sophos UTM OGW, visit the Sophos AWS Marketplace page or follow the instructions in the Sophos Quick Start Guide. Customers that are currently running Sophos UTM can follow the steps for Updating Stacks Directly to get the latest Sophos Amazon Machine Image (AMI) that supports OGW (please note that OGW is only available for Sophos UTM that supports Auto Scaling).

As always, we’d love to hear what you think about OGW or about how you’re using Sophos UTM in AWS. You can visit the Sophos user community forums or leave a comment below if you have any questions. In a follow up blog, we’ll talk about how OGW helps simplify some routing scenarios in AWS.

The content and opinions in this blog are those of the third party author and AWS is not responsible for the content or accuracy of this post.

[1] https://www.sans.org/reading-room/whitepapers/firewalls/egress-filtering-faq-1059

[2] https://d0.awsstatic.com/aws-answers/Controlling_VPC_Egress_Traffic.pdf

The following is a guest post from the AWS Quick Start team. 

AWS just published a new Quick Start that automatically deploys NGINX Plus on the AWS Cloud easily, in about 15 minutes. This new Quick Start was created by AWS solutions architects in collaboration with NGINX, Inc., an APN Technology Partner. It is the latest in a series of Quick Starts built with the AWS partner community to automate the deployment of key workloads on AWS.

NGINX Plus is a complete application delivery platform for the modern web. This platform is built on the open source NGINX, which is a web server and reverse proxy for high-traffic sites that powers over 170 million properties. NGINX Plus provides enterprise-ready features for advanced load balancing, web and mobile acceleration, application security, monitoring, and management.

The Quick Start uses an AWS CloudFormation template to automatically build the following NGINX Plus environment on AWS:

The Quick Start architecture includes:

• An Amazon Virtual Private Cloud (Amazon VPC) that spans two Availability Zones. Each Availability Zone includes two subnets (private and public).
• In the private subnets, an NGINX Plus load balancer Auto Scaling group to maintain application availability, and to provide automatic scale-up/scale-down according to conditions you define. These two NGINX Plus load balancer instances distribute traffic to the NGINX Plus application instances within the VPC in an active/active scenario.
• An ELB Classic Load Balancer that provides inbound access to the NGINX Plus load balancer Auto Scaling group via layer 4 load balancing over ports 80 and 8080.
• In the private subnets, two NGINX Plus application instances that simulate a Multi-AZ web application farm that receives traffic from the NGINX Plus load balancer Auto Scaling group. These instances are named NGINXWebApp1 and NGINXWebApp2.
• Amazon Route 53 for DNS access to the ELB and NGINXWebApp instances.
• In the public subnet, a bastion host instance running Amazon Linux for Secure Shell (SSH) access.

Deployment is easy: You subscribe to the NGINX Plus AMI in the AWS Marketplace, set up your AWS account (if you don’t already have one), launch the Quick Start, enter a few parameters, and hit Create. You can then use the URL displayed in the Outputs tab for the stack to test your deployment. The deployment guide provides step-by-step instructions and additional details.

You can use the Quick Start to build your POC or test environment. If you want to customize the default architecture, you can adjust the template settings or use the template as a baseline for your own implementation. Moving the solution to a production environment may require additional configuration.

Quick Starts are automated reference deployments for key workloads on the AWS Cloud. Each Quick Start launches, configures, and runs the AWS compute, network, storage, and other services required to deploy a specific workload on AWS, using AWS best practices for security and availability. This is the latest in a series of Quick Starts built by AWS in collaboration with AWS partners to automate the deployment of popular products and technologies on AWS.

To get started with NGINX Plus on AWS, use the following resources:

• View the deployment guide for additional details and launch links.
• Browse and launch other Quick Start reference deployments from AWS and from AWS partners.

If you’ve reviewed or deployed one of our Quick Starts, please take our survey and let us know what you think. We look forward to hearing from you.

If you’re an AWS Partner Network (APN) Consulting or Technology Partner looking to grow your AWS-based business, then chances are you’re evaluating the most effective ways to maximize your exposure to AWS customers, and go-to-market with AWS.

AWS Sponsorship is one of the best resources you can take advantage of as an APN Partner to get in front of the customers you hope to engage. AWS events attract a large number of customers, with both a business and technical focus, who have a strong desire to learn, learn, and learn some more about how they can solve their business needs with AWS and APN Partners.

Our goal is to ensure that we help you make the most of your experience sponsoring an AWS event. To that end, we’ve asked a number of AWS sponsors to share their sponsorship experience with us, and found some common themes we’d like to share with you as you look to take advantage of AWS Sponsorship.

Make the Most of Your Time – Educate Attendees

Advanced APN Technology Partner Qubole has found that AWS event attendees are often already well-versed in the benefits of the cloud; it’s then a matter of making sure the team takes advantage of the time to educate them on the specific benefits of Qubole’s solution on AWS. “Customers that attend AWS events are already looking to deploy applications in the cloud or have already deployed them, therefore we’ve found we don’t need to spend time selling the cloud benefits. Instead we concentrate on explaining why our combined value prop is a cost-effective option to deploy Big Data,” says Anselmo Barrero, Director of Business Development, Qubole.

You can take advantage of the event by bringing a deep bench of knowledgeable staff to help educate current customers and prospects. Dawn Smeaton, Head of Marketing, Cloud Workload Security, Trend Micro, explains, “AWS events attract cloud specific roles (architect, operations) that are difficult to reach at other technology events. Unlike other industry conferences, a significant percentage of attendees at AWS events will stop by our booth with sales or deep technical questions.” Think about how you can make this time count, and prove to these attendees the value of your services/solution through resources such as educational demos, case study walkthroughs, and deep dive Q&As.

Utilize an Event Plan that Focuses on Pre, During, and Post Show Marketing Activities

While it’s crucial as an AWS sponsor to have a strong physical presence at the event, it’s also important to have a strategic plan in place to generate awareness prior to the event both with prospects and current customers. Further, think about how you can capitalize on the post-show momentum by running both an email and social media campaign shortly after the conclusion of the event.

Implementing a well-executed social media plan that complements your efforts on the ground is a key component to this strategy. For example, begin tweeting about your presence at the event a few weeks before the event, ramping up your cadence in the days prior. Make sure to become familiar with the common event hashtags (such as #reinvent or #awssummit) so you can use them within each tweet you send out from your team at the event.

Advanced APN Technology Partner Trend Micro has experienced great success in utilizing social media before, during, and after sponsorship of AWS events. In fact, the company generated 1.3M impressions from social media related to AWS sponsorship in 2015. This was a 30% increase YoY.

Have a Strong Follow-Up Plan in Place

Advanced APN Technology Partner Cloudcheckr has achieved significant business growth in the last two years by strategically investing in AWS sponsorship, having grown their business 20x in the past 24 months. The company invests two-thirds of its marketing budget in AWS event sponsorship, and has found a lot of value in the customer interactions and lead follow-up. Steve Hall, Vice President of Marketing, CloudCheckr, explains, “AWS events have a much more targeted and knowledgeable attendee base. AWS event attendees are almost uniformly positioned to provide meaningful impact in a purchase decision…We consistently sponsor multiple AWS events at Gold and higher levels because it provides us with a unique opportunity to interact with a highly technical segment of our existing and potential customer base. Between the US Summits and re:Invent, AWS events deliver nearly 35,000 potential customers to our doorstep.”

Take Advantage of Event Sessions, Activities, and Expo Halls!

Being at the event affords you a fantastic opportunity to educate yourself on the latest and greatest from AWS, to schedule time for AWS Training & Certification, and to familiarize yourself with other sponsors’ solutions. If you’re an AWS Managed Service provider, for instance, don’t miss your chance to learn about APN Technology partner solutions that may complement what you do for customers. Nicole Maus, Marketing Manager, 2nd Watch (a Premier APN Consulting Partner), suggests, “Take advantage of any education that is offered at the event – use your participation as an opportunity to learn and grow your business at the same time.”

Some Closing Thoughts from AWS Sponsors…

“AWS always ensures that the content and event delivers real value and learning opportunities to attendees, which means they can attract a senior audience who keep coming back.” – Dawn Smeaton, Head of Marketing, Cloud Workload Security, Trend Micro

“AWS event sponsorship presents us with the opportunity for one-on-one interactions with the right people, at the right organizations, to help us facilitate business growth and provide a productive environment to nurture our AWS relationship.” – Nicole Maus, Marketing Manager, 2nd Watch

“AWS sponsorship offers a great opportunity to jointly promote Qubole with AWS, as well as to educate AWS attendees on the benefits of working together.” Anselmo Barrero, Director of Business Development, Qubole

“Be prepared, making sure to stay in contact with your AWS Sponsorship team, and make sure you bring sufficient staff. The event WILL be busier than you expect – every time!” – Steve Hall, VP of Marketing, CloudCheckr

There’s still time to become a Bronze sponsor at AWS re:Invent. Download the prospectus here.

APN Partners, we know that you’re regularly producing fantastic business and technical content. How would you like to highlight that content to a broad audience of AWS customers and AWS employees?

Jeff Barr, Chief Evangelist at AWS, has recently revamped the AWS Week in Review, with the goal of making it very easy for you to submit contributions. From Jeff:

The AWS Week in Review is now a GitHub project (https://github.com/aws/aws-week-in-review). I am inviting contributors (AWS fans, users, bloggers, and partners) to contribute.

Every Monday morning I will review and accept pull requests for the previous week, aiming to publish the Week in Review by 10 AM PT. In order to keep the posts focused and highly valuable, I will approve pull requests only if they meet our guidelines for style and content.

At that time I will also create a file for the week to come, so that you can populate it as you discover new and relevant content.

Learn more about how to contribute by reading Jeff’s recent post, AWS Week in Review – Coming Back With Your Help!

Go forth and contribute!

Check out an upcoming webinar from Advanced APN Technology Partner and AWS Big Data Competency Partner, Chartio, entitled, “How To Drive Exponential Growth Using Unconventional Data Sources”.

Dan Ahmadi, Director of Growth at Meteor, a large open source platform for building web and mobile apps, will discuss how Meteor uses Segment Sources, Amazon Redshift, and Chartio to combine multiple data sources to create a single-customer view.

In this webinar you will learn:

• How Meteor uses Segment to combine several data sources in Amazon Redshift
• How Meteor uses Chartio to analyze their GitHub data
• The analyses Meteor uses to track:
  • Product: trends across repositories for the Stargazer media player
  • Adoption: insights on Meteor framework usage
  • Engagement: top community contributors and what drives them
  • Growth: daily commercial growth patterns and anomalies

The webinar is on September 1st, 2016, 10:00am PT. It features Dan Ahmed, Director of Growth at Meteor, AJ Welch, Data Engineer at Chartio and JJ Nguyen, Product Marketing Manager at Segment.

Click here to register >>

Chartio was recently featured in a guest post on the Big Data Blog entitled, “How SmartNews Built a Lambda Architecture on AWS to Analyze Customer Behavior and Recommend Content”. Read the post here.

(Note: the register link will take you to a third party site, off the APN Blog. If you register for the webinar you are registering with a third party, not AWS)

This post is co-authored by Ian Scofield and Mike Ruiz, both Partner Solutions Architects at AWS 

Introduction

If you are familiar with Amazon Elastic Block Store (Amazon EBS) and the Amazon EBS volume snapshotting process, you may know that we recommend initializing – that is, reading every block – on volumes that have been created from snapshots in order to achieve maximum read performance.[1] In some cases, you can reduce initialization time by trimming volumes to discard unused blocks prior to snapshotting. This technique may be valuable if you plan to distribute snapshots or use a snapshot to build an Amazon Machine Image (AMI).

In this blog post, we will demonstrate a method to trim EBS volumes prior to snapshotting, with an eye toward optimizing the initialization process. While many SSD controllers support native TRIM operations, including Amazon Elastic Compute Cloud (Amazon EC2) Instance Store volumes, Amazon EBS does not.  Instead, our method involves making, mounting, and trimming an intermediate volume image, and then copying the trimmed image to the new volume by using a tool that preserves sparseness. We hope this method helps illuminate some factors affecting volume initialization performance and leads to insights you can use to optimize your own snapshotting and AMI creation workflows.

Amazon EBS

Amazon EBS is a block storage offering for Amazon Elastic Cloud Compute (Amazon EC2) that offers many benefits, including high availability, portability, and durability, with a variety of options to optimize storage performance and cost for your workload.  For more information on volume types and other Amazon EBS features, see Amazon EBS Product Details.

Amazon EBS Snapshots

All Amazon EBS volume types include the ability to take snapshots. A snapshot is a point-in-time copy of an Amazon EBS volume. It is common practice to take snapshots of your Amazon EBS volumes for backup/archival purposes, and snapshots are also required when building an Amazon Machine Image (AMI).

All snapshots are stored in Amazon Simple Storage Service (Amazon S3) and can be used to build new volumes anywhere in the region. Volumes created from snapshots can be larger or of a different type than the original volume.

A volume created from a snapshot will be available for use immediately, and blocks in the snapshot will be copied to the volume from Amazon S3. This behavior results in a small penalty the first time a block stored in a snapshot is read. For some environments, this behavior can result in unexpected or unpredictable performance for new volumes. To fully optimize read performance in these environments we recommend initializing the volume. Initialization means preemptively reading every block on the volume, and there are a variety of techniques available to accomplish this. Once the volume is initialized, all subsequent reads are made directly from the EBS volume and offer maximum read performance.  We can optimize the initialization process by only storing allocated, not just modified, blocks in Amazon S3.

Again, it’s worth repeating that only volumes created from snapshots have this behavior; volumes not created from a snapshot do not need to be initialized as there is no data to be read from Amazon S3.

Snapshot Contents

The first snapshot taken of a volume contains a copy of all the blocks that have been modified since the volume was created. Blocks that were allocated to the volume but have never been modified are not stored in the snapshot. Amazon EBS has no idea how the blocks are used by the instance, only that they were modified at least once. For example, if you create a file, all the blocks associated with that file are allocated and stored in every snapshot from that point forward.

A key point to note is that modified blocks are stored in the snapshot even if the file that was associated with them is eventually deleted. So if you have a file system with lots of changes – for example, file systems found on long-running instances, that involve caching or processing large backups or log files, that are used during significant operating system upgrades, or that host compilation artifacts – the snapshot size may continue to increase even if the file system size stays relatively static.  This will result in more blocks needing to be restored the first time a restored volume is read, which will result in slower initial performance.

Finally, consider the use of a common block copy utility like dd to copy an image to an Amazon EBS volume. dd used in the default way will copy every block from the image to the volume, even if the source blocks are unallocated and contain no data. Of course, Amazon EBS will correctly write zeros to these blocks and mark them as modified, and subsequent snapshots of this volume will be of the maximum possible size. Later in this post, we’ll discuss an alternate method to copy an image to a volume in a way that will preserve sparseness and avoid the allocation of empty blocks by using an updated version of dd called ddpt.

Snapshotting Examples

Let’s take a deeper look at how snapshotting might work in practice. Consider an example where you attach a new 25-GiB Amazon EBS volume to an instance, create a file system, and then write five 5-GiB files to it (see Figure A). In all, 25 GiB worth of blocks – the entire volume – have been modified. A snapshot of this volume would comprise 25 GiB of modified blocks.

Figure A: All Blocks Modified

Let’s imagine that you now remove three files from the file system. This leaves you with two 5-GiB files and 15 GiB of free space in the file system (see Figure B). However, Amazon EBS only tracks the allocation of blocks and not the current state of the file system; even though the file system now has only 10 GiB allocated, a snapshot will still be 25 GiB in length.

Figure B: Unoptimized Snapshot

If, at this point, you were to copy only the contents of the two remaining files (taking care not to copy the empty blocks) to a new volume, you would only be modifying the 10 GiB of blocks that make up the respective files (see Figure C).  The rest of the blocks would remain untouched, and the snapshot size would only be 10 GiB.

Figure C: Optimized Snapshot

Let’s apply these observations to a common use case. Consider an operating system where you have installed patches, made modifications, and removed temporary files – very typical steps in the creation of an AMI. While the size of the files remaining in the file system may only be 3 GiB, throughout that process there will be blocks that were modified but are no longer used. This will result in a snapshot that is larger than the files actually present, and a volume created from this snapshot will take longer to initialize than strictly necessary, given the files that remain.

Exploring Initialization Performance

Let’s walk through a typical snapshotting scenario and see how we can create sparse snapshots and improve initialization performance. Please note that you are responsible for any AWS usage costs that may be incurred if you decide to follow the next steps.

Setup

You’ll start out by launching an Amazon EC2 t2.micro instance with an additional 25-GiB Amazon EBS General Purpose SSD (GP2) volume, and then manually creating a partition and file system after launch.  Then you will create five 5-GiB files of random data to write to the blocks to simulate normal usage of adding and removing packages, log files, etc.

In order to generate the test files, I used the following script:

#!/bin/bash
COUNTER=1
until [ $COUNTER -eq "6" ]; do
        echo COUNTER $COUNTER
        dd if=<(openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero) of=test$COUNTER bs=1M count=5120 iflag=fullblock status=progress
        let COUNTER+=1
done

The file system is now nearly full and has touched almost every single block on the Amazon EBS volume. A snapshot taken at this point in time would be 25 GiB in length.

The next step is to remove some of the files (test1, test3, and test5) to create the scenario where blocks have had data written to them and are modified in Amazon EBS, but are no longer allocated in the file system. A snapshot taken now would still be 25 GiB in length as those three files’ respective blocks were modified.

At this point you will take a snapshot of this volume in its current state.  This represents a “dirty” volume with all blocks modified, but only ~40% of the volume is actually allocated to files. We will refer to this as the standard snapshot going forward.

Optimization

This section outlines the steps for optimizing your snapshot for future restores. You will attach another 25-GiB volume to your instance in which you will create an image of /dev/xvdf1 (our source volume with the two remaining files). Use dd to create an image of the volume that you want to optimize and prepare for a snapshot, and store this image on disk.  As we mentioned, dd will copy all the blocks in the volume and will create a 25-GiB image.

# dd if=/dev/xvdf of=/image/sparse.img bs=512 status=progress
	5671469568 bytes (5.7 GB) copied, 126.964078 s, 44.7 MB/s…

Once the image creation is complete, you’ll have an image file that you can mount using the loopback option:

# mount -o loop,discard,offset=1048576 sparse.img /loop

The final step in optimizing the image is to clean up (trim) the unallocated blocks by running the fstrim command so you can avoid allocating them in your new volume.

# fstrim /loop

At this point, you have an image file comprising only blocks that actually contain data: a sparse image. The next step is to create an Amazon EBS volume using this image. For this, you will use a modified version of dd called ddpt (ddpt). By using ddpt you are able to specify the oflag=sparse option, which respects sparseness and results in a copy that skips unallocated blocks.

Mount another 25-GiB Amazon EBS volume to receive your optimized image. Copy the optimized image to the new Amazon EBS volume, and take your final snapshot. This copy operation will result in a volume that is structurally identical to the original, but with unallocated blocks removed.

# ddpt of=/dev/xvdg if=/image/sparse.img bs=512 oflag=sparse
52428800+0 records in
20989696+0 records out
31439104 bypassed records out
time to transfer data: 2742.145369 secs at 9.79 MB/sec

You now have two snapshots: the standard snapshot that incorporates all the blocks that were touched in the initial file creation (Figure B), and the optimized snapshot that only incorporates the blocks that are used for the files that are actually present (Figure C).

Performance Test

In order to test your two snapshots and their relative performance, create a new volume from each snapshot, using a t2.micro instance for each to ensure they both have the same initial CPU credit balance. To test the performance, read the entire volume and time the total read operation by using

# time dd if=/dev/xvdf1 of=/dev/null bs=1024k status=progress  

You will immediately see that the volume created from the trimmed image is moving significantly faster because it doesn’t have to read unallocated blocks from Amazon S3.

Analysis

Both file systems contain only two files (test2 and test4 — 10 GiB of data). But as you can see in the screenshots below, there was a significant performance difference when initializing the two restored volumes. The volume that was optimized was 2.2x faster!

Unoptimized Snapshot:

# time dd if=/dev/xvdf1 of=/dev/null bs=1024k status=progress
	26836205568 bytes (27GB) copied, 5948.419841 s, 4.5 MB/s
	25599+0 records in
	25599+0 records out
	26842497024 bytes (27GB) copied, 5948.94 s, 4.5MB/s

	real 99m8.943s
	user 0m2.024s
	sys	0m10.636s

Optimized Snapshot:

# time dd if=/dev/xvdf1 of=/dev/null bs=1024k status=progress
	26828865536 bytes (27GB) copied, 2563.828358 s, 10.5 MB/s
	25599+0 records in
	25599+0 records out
	26842497024 bytes (27GB) copied, 2564.13 s, 10.5MB/s

	real 42m44.135s
        user 0m0.964s
	sys  0m10.524s

Now let’s take a look at system performance during the actual operation as reported by Amazon CloudWatch (Figure D).

Figure D: Read Throughput Comparison

In Figure D, you can see the throughput during the dd operation. The standard snapshot (orange) shows consistent performance while initializing the entire 25-GiB volume, even though only ~40% of the volume contained file system data. The optimized snapshot (blue) shows a different story. Recall that we created five files across the volume, and then removed files 1, 3, and 5, leaving those blocks unallocated. The three peaks represent the period where Amazon EBS is no longer reading the data from S3 and is instead reading directly from the Amazon EBS volume, reducing latency and increasing speed.

Figure D also clearly indicates the initialization performance for the optimized volume is twice that of the standard volume.

This next graph shows a comparison of the read latency between the two volumes (Figure E):

Figure E: Read Latency Comparison

Latency increases when restoring the blocks from Amazon S3 to the Amazon EBS volume. As you can see in Figure E, when reading where files 1, 3, and 5 were on the volume, the average is 5 ms/op for the optimized snapshot (blue) vs. 55 ms/op (orange) for the standard snapshot due to retrieving from Amazon S3.

Summary

Snapshots are an excellent tool to back up and distribute your Amazon EBS volumes, as well as an essential step in the creation of AMIs. If you have been using an EBS volume for an extended period of time or have made significant modifications since its creation, you may be paying an unnecessary penalty on the first read from the volume upon initialization. Optimizing your snapshot by trimming the file system will enable you to drastically reduce the data that is fetched by Amazon S3, which, in turn, will increase the initial performance of your volume and will help reduce storage costs.

We hope you will examine your snapshotting process and look for opportunities to use the techniques demonstrated in this post to improve initialization performance, reduce snapshot costs, and improve the launch time for your custom AMIs.

[1] NOTE: As a reminder, new EBS volumes that are not built from a snapshot receive their maximum performance the moment that they are available and do not require initialization.