AWS Partner Network (APN) Blog
Testing SaaS Solutions on AWS
Tod Golding is a Partner Solutions Architect (SA) at AWS. He is focused on SaaS.
The move to a software as a service (SaaS) delivery model is often motivated by a fundamental need for greater agility and customer responsiveness. SaaS providers often succeed and thrive based on their ability to rapidly release new features without compromising the stability of their solutions. Achieving this level of agility starts with a commitment to building a robust DevOps pipeline that includes a rich collection of automated tests. For SaaS providers, these automated tests are at the core of their ability to effectively assess the complex dimensions of multi-tenant load, performance, and security.
In this blog post, we’ll highlight the areas where SaaS can influence your approach to testing on AWS. In some cases, SaaS will simply extend your existing testing models (load, performance, and so on). In other cases, the multi-tenant nature of SaaS will introduce new considerations that will require new types of tests that exercise the SaaS-specific dimensions of your solution. The sections that follow examine each of these areas and provide insights into how expanding the scope of your tests can add value to SaaS environments.
SaaS Load/Performance Testing
In a multi-tenant universe, your tests go beyond simply ensuring that your system is healthy—tests must also assure that your system can effectively respond to unexpected variations in tenant activity that are commonly associated with SaaS systems. Your tests must be able to verify that your application’s scaling policies can respond to the continually changing peaks and valleys of resource consumption associated with SaaS environments. The reality is, the unpredictability of SaaS loads combined with the potential for cross-tenant performance degradation makes the bar for SaaS load and performance testing much higher. Customers will certainly be unhappy if their system’s performance is periodically affected by the activities of other tenants.
For SaaS, then, the scope of testing reaches beyond performance. It’s about building a suite of tests that can effectively model and evaluate how your system will respond to the expected and the unexpected. In addition to ensuring that customers have a positive experience, your tests must also consider how cost efficiently it is achieving scale. If you are over-allocating resources in response to activity, you’re likely impacting the bottom line for the business.
The following diagram represents an idealized representation of how SaaS organizations prefer to model the connection between load and resource consumption. Here, you see actual tenant consumption in blue and the allocated resources in red. In this model, you’ll notice that the application’s resources are allocated and deallocated in lockstep with tenant activity. This is every SaaS architect’s dream. Here, each tenant has a positive experience without over-committing any resources.
The patterns in this chart represent a snapshot of time on a given day. Tomorrow’s view of this same snapshot could look very different. New tenants may have signed up that are pushing the load in entirely new ways. This means your tests must consider the spectrum of load profiles to verify that changes in tenant makeup and application usage won’t somehow break your scaling policies.
Given this consumption goal and the variability of tenant activity, you’ll need to think about how your tests can evaluate your system’s ability to meet these objectives. The following list identifies some specific areas where you might augment your load and performance testing strategy in a SaaS environment:
- Cross-tenant impact tests – Create tests that simulate scenarios where a subset of your tenants place a disproportionate load on your system. The goal here is to determine how the system responds when load is not distributed evenly among tenants, and assess how this may affect overall tenant experience. If your system is decomposed into separately scalable services, you’ll want to create tests that validate the scaling policies for each service to ensure that they’re scaling on the right criteria.
- Tenant consumption tests – Create a range of load profiles (e.g., flat, spikey, random) that track both resource and tenant activity metrics, and determine the delta between consumption and tenant activity. You can ultimately use this delta as part of a monitoring policy that could identify suboptimal resource consumption. You can also use this data with other testing data to see if you’ve sized your instances correctly, have IOPS configured correctly, and are optimizing your AWS footprint.
- Tenant workflow tests – Use these tests to assess how the different workflows of your SaaS application respond to load in a multi-tenant context. The idea is to pick well-known workflows of your solution, and concentrate load on those workflows with multiple tenants to determine if these workflows create bottlenecks or over-allocation of resources in a multi-tenant setting.
- Tenant onboarding tests – As tenants sign up for your system, you want to be sure they have a positive experience and that your onboarding flow is resilient, scalable, and efficient. This is especially true if your SaaS solution provisions infrastructure during the onboarding process. You’ll want to determine that a spike in activity doesn’t overwhelm the onboarding process. This is also an area where you may have dependencies on third-party integrations (billing, for example). You’ll likely want to validate that these integrations can support their SLAs. In some cases, you may implement fallback strategies to handle potential outage for these integrations. In these cases, you’ll want to introduce tests that verify that these fault tolerance mechanisms are performing as expected.
- API throttling tests – The idea of API throttling is not unique to SaaS solutions. In general, any API you publish should include the notion of throttling. With SaaS, you also need to consider how tenants at different tiers can impose load via your API. A tenant in a free tier, for example, may not be allowed to impose the same load as a tenant in the gold tier. The main goal here is to verify that the throttling policies associated with each tier are being successfully applied and enforced.
- Data distribution tests – In most cases, SaaS tenant data will not be uniformly distributed. These variations in a tenant’s data profile can create an imbalance in your overall data footprint, and may affect both the performance and cost of your solution. To offset this dynamic, SaaS teams will typically introduce sharding policies that account for and manage these variations. Sharding policies are essential to the performance and cost profile of your solution, and, as such, they represent a prime candidate for testing. Data distribution tests allow you to verify that the sharding policies you’ve adopted will successfully distribute the different patterns of tenant data that your system may encounter. Having these tests in place early may help you avoid the high cost of migrating to a new partitioning model after you’ve already stored significant amounts of customer data.
As you can see, this test list is focused on ensuring that your SaaS solution will be able to handle load in a multi-tenant context. Load for SaaS is often unpredictable, and you will find that these tests often represent your best opportunity to uncover key load and performance issues before they impact one or all of your tenants. In some cases, these tests may also surface new points of inflection that may merit inclusion in the operational view of your system.
Tenant Isolation Testing
SaaS customers expect that every measure will be taken to ensure that their environments are secured and inaccessible by other tenants. To support this requirement, SaaS providers build in a number of policies and mechanisms to secure each tenant’s data and infrastructure. Introducing tests that continually validate the enforcement of these policies is essential to any SaaS provider.
Naturally, your isolation testing strategy will be shaped heavily by how you’ve partitioned your tenant infrastructure. Some SaaS environments run each tenant in their own isolated infrastructure while others run in a fully shared model. The mechanisms and strategies you use to validate your tenant isolation will vary based on the model you’ve adopted.
The introduction of IAM policies provides an added layer of security to your SaaS solution. At the same time, it can add a bit of complexity to your testing model. It’s often difficult to find natural mechanisms to validate that your policies are performing as expected. This is typically addressed through the introduction of test scripts and API calls that attempt to access tenant resources with specific emphasis on simulating attempts to cross-tenant boundaries.
The following diagram provides one example of this model in action. It depicts a set of resources (Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon DynamoDB items, and Amazon Simple Storage Service (Amazon S3) buckets) that belong to two tenants. To enforce isolation of these tenant resources, this solution introduces separate IAM policies that will scope and limit access to each resource.
With these policies in place, your tests must now validate the policies. Imagine, for example, that a new feature introduces a dependency on a new AWS resource. When introducing this new resource, the team happens to overlook the need to create the corresponding IAM policies to prevent cross-tenant access to that resource. Now, with good tests in place, you should be able to detect this violation. Without these tests, you have no way of knowing that your tenant isolation model is being accurately applied.
As part of isolation testing, you may also want to introduce tests that validate the scope and access of specific application and management roles. For example, SaaS providers often have separate management consoles that have varying levels of access to tenant data. You’ll want to be sure to use tests that verify that the access levels of these roles match the scoping policies for each role.
Tenant Lifecycle Testing
The management of SaaS tenants requires you to consider the full lifecycle of events that may be part of a tenant’s experience. The following diagram provides a sampling of events that are often part of the overall tenant lifecycle.
The left side of this diagram shows the actions that tenants might take, and the right side shows some of the operations that a SaaS provider’s account management team might perform in response to those tenant actions.
The tests you would introduce here would validate that the system is correctly applying the policies of the new state as tenants go through each transition. If, for example, a tenant account is suspended or deactivated, you may have policies that determine how long data is retained for that tenant. These policies may also vary based on the tier of the tenant. Your tests would need to verify that these policies are working as expected.
A tenant’s ability to change tiers also represents a good candidate for testing, because a change in tiers would also change a tenant’s ability to access features or additional resources. You’ll also want to consider the user experience for tier changes. Does the tenant need to log out and start a new session before their tier change is recognized? All of these policies represent areas that should be covered by your tier tests.
Tier Boundary Testing
SaaS solutions are typically offered in a tier-based model where SaaS providers may limit access to features, the number of users, the size of data, and so on based on the plan a tenant has selected. The system will then meter consumption and apply policies to control the experience of each tenant.
This tiering scheme is a good candidate for testing in SaaS environments. SaaS teams should create tests that validate that the boundaries of each tier are being enforced. This typically requires simulating configuration and consumption patterns that will exceed the boundary of a tier and validating that the policies associated with that boundary are correctly triggered. The policies could include everything from limiting access to sending notifications.
Fault Tolerance Testing
Fault tolerance is a general area of concern for all solutions. It’s also an area that is addressed in depth by the industry with solid guidance, frameworks, and tools. The bar for fault tolerance in SaaS applications is very high. If your customers are running on shared infrastructure and that environment is plagued by availability problems, these problems will be visible to your entire population of customers. Naturally, this can directly impact your success as a SaaS provider.
It’s beyond the scope of this blog post to dig into the various strategies for achieving better fault tolerance, but we recommend that you add this to the list of testing areas for your SaaS environment. SaaS providers should invest heavily in adopting strategies that can limit or control the scope of outages and introduce tests that validate that these mechanisms are performing as expected.
Using Cloud Constructs
Much of the testing that we’ve outlined here is made simpler and more cost effective on AWS. With AWS, you can easily spin up environments and simulate loads against those environments. This allows you to introduce tests that mimic the various flavors of load and performance you can expect in your SaaS environments. Then, when you’re done, you can tear these environments down just as quickly as you created them.
Testing with a Multi-Tenant Mindset
SaaS multi-tenancy brings with it a new set of load, performance, isolation, and agility considerations—each of which adds new dimensions to your testing mindset. This blog post provided a sampling of considerations that might shape your approach to testing in a SaaS environment. Fortunately, testing SaaS solutions is a continually evolving area with a rich collection of AWS and partner tools. These tools can support your efforts to build a robust testing strategy that enhances the experience of your customers while still allowing you to optimize the consumption of your solution.
Exciting News for Red Hat OpenShift Customers on AWS
Yesterday, Red Hat and Amazon Web Services (AWS) announced an extended strategic alliance to natively integrate access to AWS services into Red Hat OpenShift Container Platform. With this new offering, Red Hat OpenShift customers will be able to seamlessly configure, deploy, and scale AWS services like Amazon RDS, Amazon Aurora, Amazon Athena, Amazon Route 53, and AWS Elastic Load Balancing directly within the Red Hat OpenShift console. Yesterday, Red Hat and AWS demonstrated these integrations at Red Hat Summit 2017 in Boston, MA. You can view the demo below:
The Open Service Broker API provides a standardized interface for applications to interact with external, 3rd-party services. The native integration of AWS Service Brokers in Red Hat OpenShift makes it easier for Red Hat OpenShift customers to easily build applications and consume AWS services and features using simple, easy to integrate options directly from the OpenShift Container Platform web console or the OpenShift CLI.
To learn more about this announcement, read the press release here, and check out Red Hat’s blog, “AWS and Red Hat — Digging a Little Deeper“.
Learn more about Red Hat on AWS by visiting our Red Hat page.
CoreOS and Ticketmaster Collaborate to Bring AWS Application Load Balancer Support to Kubernetes
Kubernetes continues to grow in popularity on the AWS platform, and as with any popular tool or service, customers tend to identify areas where Kubernetes and AWS services can be better integrated. APN Partner CoreOS, the self-driving infrastructure company, lives at the forefront of this space and has been working to improve the Kubernetes experience on AWS.
You can observe this innovation and thought leadership in CoreOS Tectonic, an enterprise Kubernetes distribution. Tectonic provides numerous AWS integrations that enable customers to take advantage of many built-in features of the AWS platform, like AWS KMS for key management, Amazon S3 for backing up important data, Auto Scaling, IAM, and more.
One popular AWS feature that has been conspicuously unavailable to Kubernetes until now is the Application Load Balancer. This load balancing option operates at layer 7, enabling features like host, and path-based routing, TLS termination, support for WebSockets, HTTP/2, IPV6, and AWS WAF (web application firewall) features.
In collaboration with Ticketmaster, CoreOS has developed an Ingress controller. This resource can create all necessary AWS resources that will enable customers to use Application Load Balancers to route traffic to different services running on their Kubernetes controller.
A quick primer on Kubernetes Ingress
Kubernetes provides several ways of exposing your services to the Internet. One way is through a resource type called an Ingress, which is a set of rules and configuration for how traffic will be forwarded to your service. Defining an Ingress resource by itself doesn’t do anything, however, and you’ll need an Ingress controller to actually create resources on behalf of your Ingress. An Ingress controller is an application that listens to the Kubernetes API for the creation of Ingresses, and then creates the necessary resources for exposing your service.
The ALB Ingress Controller is designed to create and manage all the necessary AWS components to expose your Ingress via an Application Load Balancer. This means that when you deploy a service and expose it via an Ingress, the ALB Ingress Controller does the heavy lifting of creating an Application Load Balancer, registering your service with the target group, and creating an Amazon Route 53 Alias record to point to your Application Load Balancer. If you delete your service and its Ingress, the Ingress Controller will clean up all the associated resources. Let’s take a look at how this works.
ALB Ingress Controller Workflow
Source: https://github.com/coreos/alb-ingress-controller
These creation steps are outlined in the Github repository, but I’ll paraphrase them here to prevent too much context switching.
When an Ingress creation event from the API server is detected [1], the Ingress controller begins to create the necessary AWS resources. First, it creates the Application Load Balancer [2], and parses configuration for the load balancer from the Ingress YAML definition file. Target groups [3] are also created, one per Kubernetes service that will use the load balancer. At [4], listeners are created, which expose the service on a specific port on the load balancer. Routing rules [5] are configured, which specify the correlation between URL paths and backend target groups. Finally, Route 53 resource records [6] are created to represent the domain for your service.
Deploying your own ALB Ingress Controller
Let’s walk through a simple demo to set up our own ALB Ingress controller, spin up a sample backend service, and make sure we can actually hit our service through the load balancer at the route we configure.
First, you’ll need a Kubernetes cluster. For this demo, I’m using my Tectonic cluster, which I spun up using the Tectonic installer: https://coreos.com/tectonic/docs/latest/tutorials/installing-tectonic.html
Tectonic installs with a sensible IAM policy applied to your Kubernetes controller instances, but the ALB Ingress Controller adds new requirements, since it now has to manage Route 53 on your behalf. You can find an example IAM policy here: https://github.com/coreos/alb-ingress-controller/blob/master/examples/iam-policy.json
Since I was starting from the base Tectonic IAM role, I only needed to add the Route 53 section. My IAM policy now looks like this:
Because all Ingress controllers have a dependency on default-http-backend, we need to deploy that first.
We can do that quickly with the following:
Next, we can deploy the ALB Ingress Controller. The YAML file for deploying the controller is here: https://github.com/coreos/alb-ingress-controller
We need to customize a few environment variables in the configuration before deploying, though, so clone it locally, and in your favorite editor, take a look at the following:
These may be fairly self-explanatory, but in case they aren’t:- Set the region in which you’d like your Application Load Balancer created (this will be the same region as your Kubernetes cluster), set the cluster name to something that makes sense (this will appear in AWS logging), and perhaps consider setting AWS_DEBUG to “true” in your early experimentation so you can look at pod logs for the controller and watch which AWS API calls are being made.
Once you’ve configured the YAML file for the controller, let’s apply it:
Now you have a living and breathing Ingress controller that is listening to the Kubernetes API and waiting to do your bidding. It is ready to create AWS resources when it detects that the appropriate Ingress resources have been created. Let’s move on and do just that.
Deploying a demo service and Ingress
For this demo, we’ll use the 2048 game as our web service. We’ll use the same kubectl apply commands to apply the following resource definition .yaml files.
First we’ll create a namespace for these components to run within:
Next, we’ll create the deployment of our backend service. This will deploy the 2048 game application with five replicas across our cluster.
Using the Tectonic console, I can make sure I have my five replicas running:
Now, let’s create a service, which determines how this particular set of pods should be exposed on their respective hosts. We’ll use the following service file:
Once that’s been deployed, I can again verify the configuration in the Tectonic console:
Next, we’ll specify our Ingress. This is the configuration that triggers the ALB Ingress Controller to create resources on our behalf, so we have to provide some information about how our Application Load Balancer should be set up. Here we need to specify the scheme we want for our load balancer (internal or Internet-facing), which subnets it should live in, the security group that should be applied to it, and the Route 53 subdomain we’d like it to create for us.
After we apply this configuration, the ALB Ingress Controller sets off to work. I can open up the Amazon EC2 console and take a look at the resources created.
Here’s my ALB:
A target group has been created and associated with my Application Load Balancer, and my backend instances have been registered:
A resource record in Route 53 has been created and pointed to my Application Load Balancer:
Loading the URL in my browser resolves, as well:
Thanks for sticking with us this far! Go take a look at the ALB Ingress Controller from CoreOS today, and let us know how we can continue to work together to make your Kubernetes experience on AWS even better.
If you have any questions about this or any other aspect of Tectonic and/or Kubernetes on AWS, come hear me talk alongside CoreOS and Ticketmaster at CoreOS Fest, the Kubernetes and distributed systems conference happening May 31 and June 1 in San Francisco!
Generating Custom AWS CloudFormation Templates with Lambda to Create Cross-Account Roles
Ian Scofield is a Partner Solutions Architect (SA) at AWS.
In a previous post in our series, we showed how to use an AWS CloudFormation launch stack URL to help customers create a cross-account role in their AWS account. As mentioned in an earlier APN Blog post, a cross-account role is the recommended method to enable access to a customer account for partner integrations, and creating the role using a CloudFormation template instead of following a long series of manual steps can reduce failure rates and improve the customer onboarding experience.
In this post, we will explore the use of custom CloudFormation templates to further streamline the onboarding process.
Recall that the CloudFormation template in our previous example was static and required the customer to enter a 12-digit AWS account ID and an arcane value called an external ID. Of course, omitting or entering incorrect values results in a failed CloudFormation launch, or, worse, a useless cross-account role sitting in the customer account.
Since we already know the values of these two parameters (the partner’s AWS account ID is the parameter we want the customer to trust, and the external ID is a unique value we generate for each customer), it makes sense for us to automate template creation and set these values ahead of time on behalf of the customer.
About external IDs
The external ID is a piece of data defined in the trust policy that the partner must include when assuming a role. This allows the role to be assumed only when the correct value is passed, which specifically addresses the confused deputy problem. External IDs are a good way for APN Partners to improve the security of cross-account role handling in a SaaS solution, and should be used by APN Partners who are implementing products that use cross-account roles. For a deeper look into why external IDs are important and why APN Partners should use them, take a look at How to Use External ID When Granting Access to Your AWS Resources on the AWS Security Blog.
There are many methods for setting default values in CloudFormation templates. We’ll discuss two of these. Keep in mind that although this blog post focuses on cross-account role creation, the method of populating parameters on the fly can be used for any other components within the template. Depending on the parameter in question, one of the methods we discuss might be a better fit than the other.
The first method is to supply the partner’s account ID and external ID as the default values to CloudFormation parameters. The customer can inspect and potentially overwrite parameter values in the CloudFormation console before launching the template (Figure 1). In some cases, this level of transparency might be required so the customer is aware of the AWS Account ID they are granting access to.
However, as noted previously, incorrect values will result in the CloudFormation stack failing to launch or associate correctly, so any customer modifications to these parameters are likely to result in a failure.
Figure 1: Using default parameter values
The second method (Figure 2) doesn’t expose any parameters to the customer; instead, it hard-codes the partner’s account ID and external ID directly into the resources in the template. This helps ensure the successful creation of the role and association with the partner account, while removing any additional work for the customer.
Figure 2: Hardcoding parameter values
In both of these scenarios, how do you insert values that are unique for each customer into the template? In order for either method to work, you have to create a custom template for each customer with their unique values. This requires some additional steps in your onboarding workflow; however, the simplicity it provides to the customer and reduced chances of failure can outweigh the initial setup on your side.
To demonstrate this scenario, I created a mock portal to handle the customer onboarding experience:
Figure 3: Mock portal for onboarding
The portal requires the customer to provide their AWS account ID to associate with the uniquely generated external ID. When the user clicks Generate Custom Template, the account ID is sent to your application and invokes an AWS Lambda function. In my example, I’m using Amazon API Gateway to invoke the function, which does the following:
1. Puts an entry with the account ID into an Amazon DynamoDB table. This allows you to track customers and associate the cross-account role we’ll create later with the AWS account ID. You can also store the external ID and any other information pertaining to the customer in the DynamoDB table.
2. Generates a customized template for the user from a master template. The master template has all the necessary information and some placeholder values that you substitute with customized values:
The Lambda function downloads the template and uses a simple replace() function to replace the placeholder strings with the unique values you’ve generated for this customer.
3. Uploads the customized template to an S3 bucket with the customer’s account ID prepended to the file name to correlate templates with specific customers.
4. Sends back the S3 URL for the custom-generated template, and then displays a Launch Stack button on the portal for the customer to begin the onboarding process.
Figure 4: Launch UI
At this point, the customer clicks the Launch Stack button and begins the onboarding process for their AWS account. The stack creates the cross-account role with the provided policy embedded in the template, without the customer having to copy and paste policy documents and manually go through the role creation process.
There are a few outstanding items that would make this solution simpler still. How does the partner get the Amazon Resource Name (ARN) for the cross-account role we created? What happens to the custom template in the S3 bucket? What if the customer tears down the template without notifying the partner? We’ll continue to expand on this topic, so stay tuned for additional posts to be published in this series.
Let us know if you have any questions or comments!
Driving Big Data Innovation on AWS – ISV Highlights, April 2017
Introduction by Terry Wise, Global Vice President, Channels & Alliances at AWS
What can your data do for you? More importantly, how can insights derived from your data help you drive additional value for end customers?
Our APN partners offer services and solutions that complement what AWS has to offer. As an example, many customers are choosing to build a data lake on AWS. NorthBay is a Big Data Competency Consulting partner that helped architect and implement a data lake on AWS for Eliza Corporation. You can read details of the solution they built here. Today, I want to tell you a bit about four of our AWS Big Data Competency ISVs and what makes them unique: Alteryx, Databricks, SnapLogic, and Treasure Data.
Alteryx
AWS Big Data Competency Holder in Data Integration
How is your time spent when you embark on a new data analytics project? For many, the time required to gather, prepare, and process their data cuts into the time they can spend actually analyzing and learning from their data. Alteryx’s mission is to change the game for these analysts through the company’s self-service data analytics platform. “Alteryx Analytics provides analysts the unique ability to easily prep, blend, and analyze all of their data using a repeatable workflow, then deploy and share analytics at scale for deeper insights in hours, not weeks. Analysts love the Alteryx Analytics platform because they can connect to and cleanse data from data warehouses, cloud applications, spreadsheets, and other sources, easily join this data together, then perform analytics – predictive, statistical, and spatial – using the same intuitive user interface, without writing any code,” says Bob Laurent, VP of product marketing at Alteryx. The company’s products are used by a number of AWS customers, including Chick-fil-A, Marketo, and The National Trust.
Alteryx integrates with Amazon Redshift and provides support for Amazon Aurora and Amazon S3. Using Alteryx on AWS, users can blend data stored in the AWS Cloud, such as data stored in Redshift, with data from other sources using Alteryx’s advanced analytic workflow. Earlier this year, the company virtualized its Alteryx Server platform to make it easy for users to deploy on AWS through the AWS Marketplace. “Organizations can deploy our Alteryx Server platform in the AWS Cloud within minutes, while maintaining the enterprise-class security and scalability of our popular on-premises solution. This gives organizations a choice for how they want to quickly share critical business insights with others in their organization,” explains Laurent.
See Alteryx in action by downloading a free 14-day trial of Alteryx Designer here, or launch Alteryx Server from the AWS Marketplace here. If you’re interested in becoming an Alteryx Partner, click here. To learn more about Alteryx, visit the company’s AWS-dedicated site.
Databricks
AWS Big Data Competency Holder in Advanced Analytics
Are you looking for an efficient way to run Apache® Spark™ as you seek to create value from your data and build a sophisticated analytics solution on AWS? Then take a look at Databricks, founded by the team who created the Apache Spark project. “Databricks provides a just-in-time data platform, to simplify data integration, real-time experimentation, and robust deployment of production applications,” says John Tripier, Senior Director, Business Development at Databricks. The company’s mission is to help users of all types within an organization, from data scientists to data engineers to architects to business analysts, harness and maximize the power of Spark. Users can also take advantage of a wide range of BI tools and systems that integrate with the platform, including Tableau, Looker, and Alteryx. The company works with companies across a wide range of industries, including Capital One, 3M, NBC Universal, Edmunds.com, Viacom, and LendUp.
Databricks is hosted on AWS, and takes advantage of Amazon EC2 and Amazon S3. “Databricks is a cloud-native platform that deploys Spark clusters within the AWS accounts of our 500+ customers. We leverage the compute, storage, and security resources offered by AWS. We find AWS is a reliable and secure environment and enables fast implementation of infrastructure in regions all over the world,” says Tripier.
Want to give Databricks a spin? The company offers a free trial of their software here. Learn more about the Databricks platform here. And if you’re a Consulting Partner interested in learning more about becoming a Databricks Partner, click here. Databricks deploys in all regions, including AWS GovCloud, and is also an AWS Public Sector Partner.
SnapLogic
AWS Big Data Competency Holder in Data Integration
Where does your data come from? For most companies, particularly enterprises, the answer is, a lot of places. SnapLogic is focused on helping enterprises easily connect applications, data, and things between on-premises, cloud, and hybrid environments through its Enterprise Integration Cloud (EIC). True to its name, the company provides Snaps, which are modular collections of integration components built for a specific data source, business application, or technology. “We help customers automate business processes, accelerate analytics, and drive digital transformation,” says Ray Hines, director of strategic partners and ISVs at SnapLogic. The company works with hundreds of customers, including Adobe, Box, and Earth Networks.
The SnapLogic Enterprise Integration Cloud integrates with Amazon Redshift, Amazon DynamoDB, and Amazon RDS. “We provided pre-built integrations with these services because our customers are rapidly adopting them for their cloud data warehousing needs,” explains Hines. The company’s solution can help simplify the onboarding process for Redshift, DynamoDB, and RDS customers. For instance, Snap Patterns provide pre-built data integrations for common use cases and a number of other features (learn more here).
Care to try out SnapLogic for AWS? Click here for a 30-day free trial of SnapLogic Integration Cloud for Redshift or download the data sheet here. You can request a custom demo here. Consulting Partners, learn more about becoming a SnapLogic Partner here.
Treasure Data
AWS Big Data Competency Holder in Data Management
Are you a marketer looking for the ability to use data to provide great experiences to end customers? Are you in sales operations and are you looking to create a centralized dashboard for real-time sales data? Give life to your customer data through Treasure Data. “Treasure Data simplifies data management. Our Live Customer Data platform keeps data connected, current, and easily accessible to the people and algorithms that drive business success,” says Stephen Lee, vice president of business development at Treasure Data. “We provide a turnkey solution that collects data from 300+ sources, stores the data at scale, and provides users the tools to analyze and activate their data in their application of choice.” The company works with customers across industries including Grindr, Warner Brothers, and Dentsu.
“We deployed our solution on AWS because of the scalability, reliability, and global footprint of the AWS Cloud and the ability to deploy without having capital expenditures. With AWS, we can easily deploy our solution in new regions. We’ve also found there to be a strong support ecosystem,” says Lee. Treasure Data’s Live Customer Data Platform integrates with Amazon Redshift, Amazon S3, and Amazon Kinesis, along with many other solutions including Tableau, Chartio, Qlik, Looker, and Heroku (see all integrations and learn some nifty integration recipes). Getting started with Treasure Data is easy. “Our Solution Architects work with our new customers to get their initial data sources set up, after which our customers can be up and running in minutes,” explains Lee.
You can request a custom demo here, or simply email the team directly at info@treasuredata.com. Consulting Partners interested in becoming a Treasure Data partner can visit the company’s partner page here.
Want to learn more about big data on AWS? Click here. Bookmark the AWS Big Data Blog for a wealth of technical blog posts you can look to as you begin to take advantage of AWS for big data.
This blog is intended for educational purposes and is not an endorsement of the third-party products. Please contact the firms for details regarding performance and functionality.
Quickly Clone Amazon S3 Data at Petabyte Scale with APN Partner StorReduce
This is a guest post by Kelly Boeckman. Kelly is a Partner Manager at AWS who is focused on the Storage segment.
Benjamin Franklin once famously said, “In this world nothing can be said to be certain, except death and taxes.” It’s worth noting, however, that Ben was living in the late 1700s, and was unable to conceive of a third certainty: data growth.
Data deduplication is a critical solution for runaway data growth, as it can reduce the amount of data that needs to be stored or backed up. It also shortens backup windows, dramatically lowers infrastructure costs, and enables cost– and space-efficient data clones for testing, QA, etc. This unlocks a door to the Shangri-La of “Do-More-With-Less.”
So you’ve got your data safely deduped and stored in Amazon Simple Storage Service (Amazon S3), but surely that data can provide more value than just sitting there? Other teams across your organization could undoubtedly derive value from copies of that data. Maybe your developers could create new features or QA teams could more accurately test against your code base if they can test against a whole set of your cloud data.
Today, we want to tell you about an innovative solution in this space from APN Technology Partner StorReduce that can help you do more with your data for less. StorReduce has announced support for cloning on object storage such as S3 and Standard-Infrequent Access (Standard – IA). This support allows users and administrators to make copy-on-write copies of objects stored with StorReduce in a rapid, space-efficient manner at petabyte scale.
Who is StorReduce?
StorReduce can help enterprises storing unstructured data to S3 or Amazon Glacier on AWS to reduce the amount and cost of storage significantly. It also offers enterprises another way to migrate backup appliance data and large tape archives to AWS.
StorReduce’s scale-out deduplication software runs on the cloud or in a data center and scales to petabytes of data. The variable block length deduplication removes any redundant blocks of data before it is stored so that only one copy of each block is stored. StorReduce provides throughput of more than 1 gigabyte per second per server for both reads and writes. A StorReduce cluster can scale to tens or hundreds of servers and provide throughputs of 10s to 100s of gigabytes per second. StorReduce is suitable to deduplicate most data workloads, particularly backup, archive, Hadoop cluster backups, and general unstructured file data. Because StorReduce has read-only endpoints and a cloud-native interface, data that it migrates to the AWS Cloud can be reused with cloud services.
Object storage cloning
Storage volume cloning is a long-standing feature of most on-premises NAS and SAN devices—a well-known example is NetApp FlexClone. In these environments, cloning enables fast snapshots of data, creating many additional copies of the data. Depending on the data change rate, deduplication can dramatically reduce the footprint of those extra copies, freeing up storage resources.
StorReduce’s Object Clone brings this feature to S3 and Standard – IA. Leveraging deduplication, a clone can be created simply by copying the references to the unique blocks that have been stored with StorReduce, while leaving the data stored in S3 unchanged.
Copying large amounts of object data may take hours per copy, and – lacking data reduction – you pay for every additional copy, regardless of how much of it is unique.
With StorReduce Object Clone, you need only copy a small amount of metadata. Copying petabyte-scale buckets becomes nearly instantaneous, and multiple clones can be created in rapid succession. Additionally, secondary and subsequent copies of the data are low cost, even at petabyte scale.
How StorReduce Object Clone works
Using the StorReduce dashboard to perform a “Clone Bucket” request is as easy as creating a bucket in StorReduce or S3. Simply navigate to the source bucket to clone. Choose Clone Bucket from the Actions dropdown and specify the target bucket name and StorReduce handles the rest. It creates a duplicate bucket that is owned by the initiator of the request, and which inherits the source bucket’s data.
Clones can also be created programmatically with two API calls, and you can always trace the name of the source bucket. If the source is deleted, this does not affect the operation, the contents of the clone, or access to it.
Benefits
StorReduce Object Clone provides a variety of benefits, including:
- Increased flexibility – Provide cloud-based object storage with the same level of protection available to on-premises files or block data
- Clone entire datasets at petabyte scale as many times as needed, quickly and affordably
- Enable efficient, isolated application development and testing plus research and development against full datasets
- Simplify the cloning of data in object stores, regardless of data deduplication ratios
- Reduced management complexity – Administrators can quickly provide clones and reduce human error
- Subsets of data can be made easily administrable for unbiased application testing
- Protect against human error or malicious deletion of data in object stores
- Make time-based clones of entire buckets and assign read-only access for “locking in” protected data at critical points in time
- Reduced expense – Space-efficient clones consume less space and require less infrastructure
- Enables you to meet your compliance requirements quickly where R&D or development activities require separate pools of data
- Clone only small amounts of data in index or metadata, which reduces the clone size
- Smaller clone size reduces cost of required infrastructure to store data for the second and subsequent copies, even at petabyte scale
Use cases
StorReduce Object Clone provides anyone who is working with a large dataset in S3 (whether that data deduplicates or not) the ability to try experiments and to fail and iterate quickly. Common use cases include:
- Big data, Internet of Things, research – Clone petabyte-scale datasets so that researchers can work independently of each other in their own scratch areas.
- IT operations – Test new versions of software against your dataset in isolation.
- Developers, software testers – Clone your test datasets so that developers and testers can work with the whole dataset in insolation, not just a small subset. Roll the state back after failures and retest rapidly.
- Software quality assurance – Take a lightweight clone of an entire system at the point that it fails a test and hand it back to the developer.
- Hadoop, big data – Take point-in-time snapshots of the state of your cluster and roll back after problems.
To learn more about how AWS can help with your storage and backup needs, see the Storage and Backup details page.
Learn more about StorReduce Object Clone here. To request a StorReduce Object Clone free trial, contact info@storreduce.com.
This blog is not an endorsement of a third-party solution. It is intended for informational purposes.
Join Us for the AWS Partner Summit – Mumbai!
Come meet AWS team members and fellow APN Partners at the third-annual AWS India Partner Summit, hosted in Mumbai. The event is free and it’s exclusive to APN Partners.
The event begins with breakfast, registration, and networking opportunities at 10 am. The keynote kicks off at 11 am and features Terry Wise, Vice President, Global Alliances, Ecosystem and Channels, AWS. Following the keynote are a number of sessions running until 4:15 pm. The day closes with APN excellence awards and another opportunity to network with your industry peers and with AWS team members.
Click here to learn more. We can’t wait to see you there!
Facilitating a Migration to AWS with CloudEndure by Leveraging Automation
By Carmen Puccio and Mandus Momberg. Carmen and Mandus are AWS Partner Solutions Architects focused on Migration.
It’s no secret that migrating software and services from an on-premises environment to the cloud entails unique considerations and requirements. To provide confidence in the outcome of your migration, your migration strategy needs to scale easily. This means that a large part of your workflow must be automated. There is no shortage of documentation on why automation in the cloud is important. In this post, we will show you how to perform an automated migration utilizing AWS Advanced Technology Partner CloudEndure, with a focus on incorporating automated tests so you can be confident that your application is working as expected post-migration.
The migration of a workload from on-premises to AWS requires careful planning and precise execution. There are many different strategies for moving to the cloud, and there are also numerous tools that help facilitate migration. All migration tools share common goals: to facilitate a migration to AWS by minimizing downtime and application workload impact, and to ensure that data loss is minimized.
Customers who want to quickly move their workloads to the cloud typically follow the rehost method, i.e. lift and shift. One of the challenges when executing a rehost is the amount of time it takes to manually confirm that a migrated application is performing as expected. Migrations that incorporate automation and rapid testing pipelines to validate proper migration are not only more likely to succeed but also improve efficiency as you take advantage of repeatable processes and decrease manual verification times. (more…)
Announcing the Alexa Diabetes Challenge, Powered by Luminary Labs
By Aaron Friedman and Dario Rivera. Aaron is a Healthcare and Life Sciences Partner Solutions Architect with AWS. Dario is a Healthcare and Life Sciences Solutions Architect with AWS.
Changing lives for the better through technology is one of the most compelling reasons why both of us decided to focus our careers on merging software with healthcare and life sciences at AWS. Being able to work with AWS Partner Network (APN) partners and customers who build healthcare solutions that will affect the everyday lives of people is a task that is both equal parts challenging and rewarding. And today, we want to tell you about a new challenge that lets you to take the next step toward positively impacting human health as well: The Alexa Diabetes Challenge, sponsored by Merck & Co., Inc., Kenilworth, New Jersey, U.S.A., an APN Advanced Technology Partner.
If you’re reading this post, then there is a high probability that you know someone who has been affected by type 2 diabetes. According to the International Diabetes Federation, an estimated 415 million people around the world live with diabetes; in America alone, approximately 29.1 million Americans live with this disease. According to the Centers for Disease Control and Prevention, 1.4 million are diagnosed with type 2 diabetes every year in the US. Learning to manage this complex disease can be an overwhelming and isolating experience, especially for those newly diagnosed.
The Alexa Diabetes Challenge, sponsored by Merck & Co., calls on innovators to experiment with creating holistic voice-enabled solutions that are designed to improve the lives of those with type 2 diabetes. This pioneering challenge presents innovators with a unique opportunity to help navigate uncharted territory: experimenting with and developing new solutions for those managing a chronic condition.
Why voice?
Voice-powered experiences, such as those developers build for Amazon Alexa or Amazon Lex, can dramatically improve user experience by giving you the ability to interact with people at a more personal level. Communicating through voice, rather than just on a screen, has the potential to make it easier to empower patients to change habits and improve their overall well-being. We encourage entrants to explore a variety of voice experiences, ranging from Alexa skills to solutions built using the ecosystem of supporting AWS services, such as Amazon Lex, Amazon Polly, and AWS IoT, and other third-party technologies to develop patient-centric solutions that consider the possible impact on the entire healthcare experience.
The Challenge
This challenge will allow innovators like yourself to draw on both Merck’s expertise in drug development, epidemiology, medication adherence, observational research, and patient education programs and Amazon’s expertise in connected devices and voice interfaces.
Five finalists will be chosen to receive $25,000 each and advance to the Virtual Accelerator. At this stage, the finalists will receive expert mentorship as they develop their concepts into working prototypes. In addition, we will also host an in-person boot camp for the finalists at Amazon’s Seattle headquarters. At the end of the Virtual Accelerator, finalists will present their prototypes in-person to the judges at Demo Day at our AWS Pop-up Loft in NYC. Following Demo Day the judges will select a grand prize winner to receive $125,000.
How you can take part
Technology is changing the landscape of healthcare, and we’re very excited to work with Merck to advance the care of those with diabetes. We hope you enter the Alexa Diabetes Challenge, sponsored by Merck & Co., and use your creativity and expertise to bring ideas forward that stand to have a great impact on diabetes care for years to come.
Visit the Alexa Diabetes Challenge website to learn more and submit a concept. All submissions for this pioneering work in voice technology are due by May 22, 2017. We’re looking forward to seeing who gets chosen as a finalist and introduces a new voice in diabetes management!
The Alexa Diabetes Challenge is powered by Luminary Labs.
Announcing the Addition of AWS Service Catalog to the AWS Service Delivery Program
The AWS Service Delivery Program launched in November 2016 with one simple goal at heart: to help customers easily identify APN Partners with a proven track record of success delivering specific AWS services and who have demonstrated the ability to provide expertise in a specific service or skill area. Looking for help migrating a database to Amazon Aurora? Simply visit the Partner Solutions Finder, filter by Product, and you’ll find all of the APN Partners who’ve been validated by the AWS Service Delivery Program as holding expertise in Aurora.
A number of AWS services are included in the program, including many database services, compute services, content delivery services, security services, serverless computing services, and analytics services. The program also highlights APN Partners who deliver workloads in the AWS GovCloud (US) Region. Today, we’re excited to announce the addition of AWS Service Catalog to the program.
AWS Service Catalog + AWS Service Delivery Program = customer success
AWS Service Catalog allows organizations to create and manage catalogs of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multi-tier application architectures. With the addition of Service Catalog to the Service Delivery Program, customers can easily identify and connect with qualified AWS Consulting Partners who’ve demonstrated expertise with Service Catalog. “AWS Service Catalog is a fantastic tool for companies that want to enable developers to easily launch new AWS environments, but also need to control how environments are launched and configured,” says Stephanie Tayengco, CIO of Logicworks, a Premier APN Consulting Partner and AWS Service Catalog Delivery Partner. “Logicworks helps customers build the AWS CloudFormation stack for the products in their Service Catalog, applying the right constraints and controls so their developers can take advantage of a self-service, fully-automated, and secured environment.”
Why should APN Consulting Partners with expertise in Service Catalog join?
Joining the program provides you the ability to promote your firm as an AWS-validated expert in Service Catalog. By becoming an AWS Service Catalog Service Delivery Partner, you can increase your firm’s visibility to customers seeking your type of expertise through a number of channels, such as the AWS Service Delivery website. Additionally, you’ll be distinguished as an expert in Service Catalog in the Partner Solutions Finder and will be featured on the AWS Marketplace website. “Wipro is excited to achieve the AWS Service Delivery Partner status for the AWS Service Catalog service. With the help of AWS Service Catalog, customers can centrally manage commonly deployed IT services to improve service re-use and enhance developer productivity. Additionally, the central management of the IT Service Lifecycle can help save time and dollars which can be funneled to drive the cloud innovation agenda for digital enterprises,” says Varun Dube, General Manager – AWS practice, Wipro Ltd, a Premier APN Consulting Partner and AWS Service Catalog Delivery Partner.
What are the requirements?
In addition to meeting the minimum requirements of the program, which you can find here, your firm must pass service-specific verification of customer references and a technical review, to ensure that customers can be confident they are working with partners that provide recent and relevant experience.
Want to learn more?
Click here to learn more about the partners participating in the program, and visit the Service Delivery Program homepage to learn more about the broader program. If you would like to apply for membership, contact aws-sc-partner@amazon.com.
Join APN launch partners Wipro, Flux7, Cloudticity, Logicworks, and BizCloud Experts and apply to be a part of the AWS Service Catalog Service Delivery program.
