Experiences from AWS Summits 2016 in Chicago

Experiences from AWS Summits 2016 in Chicago

150 150 Jose R. Guay

Summit_2016

Amazon hosted their first North America summit in Chicago called AWS Summits on April 18-19 2016. The two-day free event started with a series of activities and technical bootcamps, some of which lasted the entire day. The second day was full of general sessions in which engineers from Amazon, along with some of their partners, filled our heads with the latest and greatest of the AWS platform.

stage

GameDay

 

I participated in the GameDay activity on April 18th. I must admit, I didn’t know what to expect at first. I wasn’t even sure I had everything I needed in my laptop (a must for attendees of GameDay). The activity was intended to solve some problems about implementing a system for a fictitious company called Unicorn Rentals (yes, the idea was to rent unicorns…).

When the activity started and instructions were given, I was very excited; however, excitement wearied off as soon as we started reading the material and the CEO of the company gave his “motivational speech”. I literally panicked. Fortunately, we were in teams of three people, and my two colleagues had some more experience in AWS and weren’t as lost as I was.

The developed application to rent unicorns was ready to go and we were to download it into a load balanced installation of EC2 instances. Surely enough, the problems began as soon as we started. Using Amazon Route 53 we needed to configure the identification of the team with a TXT record and access to the system with a CNAME record pointing to the load balancer. That worked for about 2 seconds and we didn’t understand why the system wasn’t responding. After about 45 minutes, and some help from the support team, we noticed the script that initializes the application in the instances had a “shutdown” command which immediately restarted the instance as soon as the application was configured and running. Finally, we were in business as soon as we removed that line from the script, but we had to reinitialize all instances, and for that, there was only one way, terminate (delete) all instances so the load balancer would create new ones and load the new script.

After the instances were loaded, our system failed again until we realized the domain was pointing to an incorrect IP address. Bummer, again. Lots of time lost in finding what was going on. Once that was fixed we needed to improve the performance of the requests. We then head to implement Amazon ElasticCache and Amazon CloudFront. Oh boy, does CloudFront takes time to initialize the distribution… It took forever! And just when we thought we were making progress, there was an announcement from the company: A new version of the application was launched and should be implemented immediately. Our system crashed immediately, requests were being dropped, it was chaos. We had to adjust the script again to load the correct version of the application so it would use ElasticCache. Finally, CloudFront was up and running and things were going well (we were actually making money).

Then the SysOps or DevOps manager made an announcement where he said they made changes to the network to “improve” the traffic. He didn’t remember what his script did but he assured us that all (or most) of it would be good. Yeah, you guessed it, once the announcement was made all systems started to fail. We head to Virtual Private Cloud (VPC) to make the modifications and it took a while but we managed to solve them.

In the end, our team was still making money but not even close as the team who got the first place. I have to say, it was really fun and I learned a lot.

Sessions

 

On day two of AWS Summits, April 19th, the general sessions began. I attended the session Building Your First Big Data Application on AWS just before the keynote. It was very interesting. The presenter created a Hadoop cluster and with the help of S3 for storage, Kinesis Firehose to collect the data from internet logs, EMR and Spark Hive to process the data and RedShift and QuickSight he presented a bunch of analytics charts, in less than 1 hour. I have to say that was impressive.

The keynote followed, Dr. Matt Wood, General Manager of Product Strategy presented the current state of AWS, the growth it has experienced and how the Amazon Cloud division is now a $10 billion business. Some of their customers such as GE, Kellogg’s, Duolingo and HERE went on stage to present their experiences implementing AWS in their companies.

keynote

Other sessions I attended were more focused on the performance of EC2 instances, Elastic Load Balancer, using Windows EC2 instances, AWS Lambda and the server-less cloud and implementing Microservices. All in all, my expectations were met, AWS has a lot of tools and power to offer to any company. I will start learning more and more about it because what I saw was really amazing.

Announcements

 

Here are some of the announcements made during the keynote:

  • Amazon Inspector now generally available. Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS. Amazon Inspector automatically assesses applications for vulnerabilities or deviations from best practices. After performing an assessment, Amazon Inspector produces a detailed list of security findings prioritized by level of severity.

 

  • Two New Low-Cost Storage Options for Amazon Elastic Block Store (EBS). Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances in the AWS Cloud.  Each Amazon EBS volume is automatically replicated within its Availability Zone to protect you from component failure, offering high availability and durability.  With this announcement, you can save up to 50% on Amazon Elastic Block Store (EBS) by launching Amazon EC2 instances and Amazon Elastic MapReduce (EMR) clusters with two new EBS volumes, Throughput Optimized HDD (st1) and Cold HDD (sc1). These new volume types offer better price performance, improved performance consistency, and higher throughput compared to our existing EBS Magnetic offering. Both are designed for throughput-intensive and big data workloads, which are characterized by having large data sets, large I/O sizes, and sequential I/O patterns. Examples of these workloads include MapReduce, Kafka, log processing, data warehouse, and ETL.

 

  • Amazon Elasticsearch Data Ingestion with Amazon Kinesis Firehose. You can now use Amazon Kinesis Firehose to stream data to your Amazon Elasticsearch domains continuously and in near real time. Amazon Kinesis Firehose automatically scales to match the throughput of your data and handles all the underlying stream management.

 

  • AWS Device Farm Remote Access is now in beta. AWS Device Farm is a test environment for Android, iOS and Web applications where you can test against real phones and tablets in the AWS cloud. You can now use AWS Device Farm for manual tasks like debugging new functionality, running manual tests, and reproducing customer issues. Simply choose an Android device and swipe, gesture, and interact with it from your web browser. This feature is currently available in beta for Android and support for iOS is coming later this year. To get started, initiate a new session in the Device Farm console, select a device, and interact with it in real time. Once your session is complete, you can view a video and logs of the session.

 

  • AWS Elastic Beanstalk added support for managed platform updates. AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker on familiar servers such as Apache, Nginx, Passenger, and IIS. You can now choose to have your AWS Elastic Beanstalk environments automatically update to the latest version of the underlying platform running your application during a specified maintenance window. Elastic Beanstalk regularly releases new versions of supported platforms (i.e., Java, PHP, Ruby, Node.js, Python, .NET, Go, and Docker) with operating system, web & application server, and language & framework updates. Previously, you had to manually initiate updates to your Elastic Beanstalk environments using the Elastic Beanstalk console, command line interface (CLI), or API. Now, you can simply select a weekly maintenance window and have Elastic Beanstalk automatically update the platform version of your environment during that window.

 

  • Amazon S3 Transfer Acceleration now generally available. Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable cloud storage. Amazon S3 is easy to use object storage, with a simple web service interface to store and retrieve any amount of data from anywhere on the web. With Amazon S3, you pay only for the storage you actually use. There is no minimum fee and no setup cost. Amazon S3 Transfer Acceleration is designed to maximize transfer speeds when you need to move data over long distances, for instance across countries or continents to your Amazon S3 bucket. It works by carrying HTTP and HTTPS traffic over a highly optimized network bridge that runs between the AWS Edge Location nearest to your clients and your Amazon S3 bucket. There are no gateway servers to manage, no firewalls to open, no special protocols or clients to integrate with, or upfront fees to pay. You simply change the S3 endpoint that your application uses to transfer data, and acceleration is automatically applied.

 

  • A new 80TB model of AWS Snowball is now available in selected regions. Snowball is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of the AWS cloud. Using Snowball addresses common challenges with large-scale data transfers including high network costs, long transfer times, and security concerns. Transferring data with Snowball is simple, fast, secure, and can be as little as one-fifth the cost of high-speed Internet. The 80 TB Snowball appliance is a larger capacity Snowball that enables more data to be securely moved in or out of AWS with fewer jobs. It is now available in the US East (N. Virginia), US West (Oregon), US West (N. California), AWS GovCloud (US), Asia Pacific (Sydney), and EU (Ireland) regions, with the rest of the world expected by the end of 2016.

 

  • Add User Sign-up and Sign-in with Cognito Identity. Amazon Cognito lets you easily add user sign-up and sign-in to your mobile and web apps. With Amazon Cognito, you can also authenticate users through social identity providers such as Facebook, Twitter, or Amazon, or by using your own identity solution. In addition, Amazon Cognito enables you to save data locally on user’s devices, allowing your applications to work even when the devices are offline. You can then synchronize data across user’s devices so that their app experience remains consistent regardless of the device they use. You can now use Cognito Identity to easily and securely add user sign-up and sign-in to your mobile and web apps. Cognito Identity can scale to support hundreds of millions of users, and is fully managed so you don’t have to worry about building, securing, and scaling a solution to handle user management and authentication.

 

  • Coming soon a new service called AWS Application Discovery Service. AWS Application Discovery Service helps Systems Integrators quickly and reliably plan application migration projects by automatically identifying applications running in on-premises data centers, their associated dependencies, and their performance profile.

 

  • New Amazon Kinesis Streams Features. Amazon Kinesis Streams enables you to build custom applications that process or analyze streaming data for specialized needs by continuously capture and store terabytes of data per hour from hundreds of thousands of sources such as website clickstreams, financial transactions, social media feeds, IT logs, and location-tracking events. Amazon is adding two new features, time-based shard iterators and shard-level metrics, to make it easy to replay and manage your Amazon Kinesis streams. Time-based iterators make your streams easily replay-able by your processing applications. Using time-based iterators, you can read records from a position in the stream corresponding to a timestamp. This gives you the ability to go back in time and replay the stream for any time window within the default 24-hour or the extended 7-day data retention period. For more information, see GetShardIterator in the Amazon Kinesis Streams API Reference. You now get ten new Amazon CloudWatch metrics for monitoring your Amazon Kinesis streams, including seven shard-level metrics. Shard-level metrics make stream management easier when using open-source processing frameworks like Apache Spark Streaming. With shard-level metrics, you can get better visibility into the performance and utilization of shards, which are units of streaming capacity. This allows you to monitor, scale, and troubleshoot your streams more easily.

Conclusion

 

Amazon offers a wide range of products and solutions to help companies of any size manage their cloud strategy. Whether you are new to cloud or an expert, you can find something that will make your life easier with their portfolio. I would encourage you to explore Elastic Beanstalk and the new AWS Lambda as a solution to running applications. In addition, the EC2 solution with virtual servers and containers can provide a solution to any size of deployments.