How Amazon Uses Explosive-Resistant Devices To Transfer Data To AWS?


Demand continues to rise from both companies and consumers that rely on remote storage and computing power accessible from anywhere. Tech giants Google, Microsoft, IBM and others are vying to be the go-to providers. But one company has remained the leader, Amazon. AWS has a commanding lead in the cloud right now. In fact, if you add up number two, three, four and five, they add up to what AWS does. Amazon Web Services is behind a lot of the technology Amazon use. From calling a Lyft to checking your video doorbell, to streaming your favorite shows. When people are watching a Prime movie, or they're watching a Netflix movie, or a Hulu movie, or others like that, they're watching it and streaming off of Amazon Web Services. The Super Bowl streams off us and also Major League Baseball and now NASCAR and Formula One racing as well. If you use Intuit to do your taxes, that runs on AWS.

 AWS has been one of Amazon's most profitable business endeavors. Last year AWS generated more than $25 billion in sales. Plus, they're still growing like a weed. Get this, they're up 47%. In the first quarter of this year, revenue climbed to $7.7 billion, up from $5.44 billion a year earlier. Amazon have over 2.2 million customers using AWS today. They're usually big companies like Goldman Sachs or Capital One. There's over 4,000 government agencies that run on us today. Companies left and right are abandoning their own data centers for Amazon's or other cloud providers. But moving all of that data online can be a challenge. The transfer fees for moving data over the network online can be quite high. And also, it can take a while if you have petabytes and petabytes and petabytes or yottabytes of data. So Amazon built physical products to make transferring large amounts of data easier. A portable data transfer device capable of operating in a war zone, called Snowball. And even a giant truck called Snowmobile to help companies migrate their data to the cloud. 

What about if we have exabytes of data? Amazon have a lot of customers who have exabytes of data. And the first thing that came to mind was, Amazon're going to need a bigger box. So why would a company need to move to a cloud service provider like AWS? Most of our customers save between 22 and 54 percent versus running all in, building their own data center, building their own networks, powering it, having people to operate it. One of the biggest reasons that people look to the cloud is not necessarily cost, but around flexibility. Developers can get access to massive amounts of compute and storage and networking resource. AWS says it has the largest global infrastructure footprint of any cloud provider, meaning it has data centers placed in regions around the globe where there is concentrated demand. It has the capacity to allow companies to tap into more server space depending on their needs. When they have a big retail day they can use a million servers, when their normal load is, say, 40 or 50 or 60 servers. And so the ability to do that is astronomically expensive to do on Prem. And that's why you see the startups growing so fast on AWS because they get the access to a Fortune 500 infrastructure for pennies on the dollar.

 For Example Netflix has always used Amazon as its cloud provider.But for a company that wants to migrate its data to the cloud, typically a massive data transfer needs to take place. Some companies have hundreds of terabytes, petabytes and even exabytes of data. For some perspective and how big that is, your average MP3 song is about three megabytes. A gigabyte is about a 1,000 megabytes, or around 300 songs. A terabyte is about a 1,000 gigabytes, or 300,000 songs. A petabytes is 1,000 terabytes, or 300 million songs, and an exabyte is around a 1,000 petabytes, or 300 billion songs. A single MP3 file might take a few seconds to transfer over the internet. 300 million or billion, however, might take a while. It's often called in IT the python eating a pig problem. So if you imagine a python, you can visualize it eating a pig. 

You get this big lump that you have to move through the python. So you have a little network and you've got a big lump to move and so for some of our customers it would've taken them years and years to upload their data over their network. Amazon has tried to solve this problem of cost and time by creating really tough hardware, called Snowballs, which people who operate data centers can connect their infrastructure to. Make copies of the data and then send those snowballs to AWS data centers so that the data can be moved more quickly. The smallest storage Snowball Amazon have is about 50 terabytes. That's 5,000 DVDs and the largest snowballs Amazon have is between 11,000 and 14,000 DVDs depending on how you compress it. Amazon work with our Lab 126 folks on the industrial design, our Kindle folks on the e-ink label. So imagine if you're shipping hundreds of these, you could easily put the wrong label, put them in the wrong box. That doesn't happen, it's all automated. It knows where it's going and it labels itself. And my boss, Charlie Bell, worked on the Space Shuttle and Amazon actually used some things off the Space Shuttle where they have to handle the shock of launch and landing.

Vass said designing the Snowball to withstand the rigors of transit was not an easy task, since it had to be highly durable as well as less than 50 pounds. Amazon actually went to our shipping partners and Amazon also went to the fulfillment center and talked to them. So from that Amazon learned a really hard problem to solve is that it had to be under 50 pounds. Amazon also wanted people to be able to check it as regular luggage. And that's actually a hard design constraint, to make something as durable as that and as dense compute and storage, in under 50 pounds. The Snowball even passed an explosives test and meets the military's requirements for being airdropped. To meet the specifications, Amazon have to drop the Snowball from 28 feet, 80 times, on all four corners and all six sides. And then because Amazon build it so robustly, Amazon is able to also pass the DoD 901 Barge Explosive Test, where you have 83 pounds of plastic explosive going off 20 feet from the device multiple times. Which is a tremendous percussion wave that would turn your insides to jello. If you were standing there it would kill you. And temperature wise, it's designed for the most extreme environments. They can operate at really high temperatures, like 140 degrees ambient temperature and really cold temperatures, like -20. And it can have unconditioned power from a generator and it'll continue to operate. For customers calmly transferring data from the safety of their office, this could all seem like overkill. But in certain instances, it's proven critical. When the volcano was going off in Hawaii, the USGS used the Snowballs. 

They had local servers and the lava was coming up on their building. And so they didn't want to lose all that extremely valuable data they've collected. They also knew the Snowballs could operate in high temperature environments. And so they shipped the Snowballs there, downloaded the data and shipped the Snowballs out. And so they were able to capture all that data without losing it. Oil rigs is another area Amazon see a lot of them. Military they're very, very popular. So they're used in forward deployed units. They're used on Navy ships. They're used in aircraft. They're used in Special Ops locations all over the world. For cybersecurity, where they're collecting network data and reacting to it locally. There's even a six micron dust filter option you can snap on the front. So if you're operating them in a desert, they can filter the sand out and not have the sand go into the device. Even Hollywood has started taking advantage of Amazon's Snow family. For studio shoots, they're shooting in 8K and 12K cameras now. And so that's a lot of data. And so they put it on the Snowballs, and you can see the screens on the front they're used for quality control. When they're done with the shoot they actually ship the Snowball back and it uploads the data into the cloud and then they post-processing it. There's another, 

I would call the upper-sell version, which is the Snowball Edge, and that's a 100 terabyte solution. Now, the interesting thing about Snowball Edge is you can actually put compute on there and actually run workloads. AWS was the first public cloud provider to make hardware like this for data transfer, but competitors have since developed similar products. Microsoft is the number two player in the public cloud market, behind AWS. It has Data Box products that have room for 1 petabyte of data, making it larger than what Google and IBM offer today. The Google Cloud, which is behind AWS and Microsoft Azure, has Transfer Appliance products. Which are storage servers that you can install inside a rack in your data center. But it's not as popular as the one that AWS is offering today. But AWS is the only company that felt like it needed to go even bigger. Snowmobile has the equivalent of 1,250 Snowballs in it. And so it's what Amazon call a 100 petabytes truck. To put in context how much data Snowmobile can take. Let's say the typical notebook is 500 gigabytes. A 100 petabytes would be 200 million notebooks that get ingested into this Mack truck. Digital Globe had this challenge where they had a huge amount of satellite imagery. They're one of the largest producers in the world of satellite imagery. And so it would have taken them about 10 years to upload it over the network. And they were actually the first customer that called us and said, hey, can't you just send a truck? And so Amazon built one. Amazon'll drive the truck up to your data center. 

Amazon have power and network fiber that will connect to your data center. Fill 'er up, and then the truck will come back, put the trailer back on the truck and Amazon'll move it back to AWS. If you think about the idea of moving an exabyte of data, if you basically assign a 10 gigabit per second line to it, which is pretty reasonable, it would take you about 26 years. Using 10 Snowmobiles, it would take you a little less than six months. AWS announced the Snowmobile by driving it onto a stage at an AWS event and people just went nuts. They were like, what? How could a cloud be a truck? And it was cool. It was innovative. But it's not like Amazon've heard about the Snowmobile being a huge business hit. It's not like the Snowmobile is what's making up most of AWS' revenue. Far from it. AWS would not disclose how many customers have used a Snowmobile to make a transfer, or how many trucks it has in service. Amazon weren't able to see the inside of the truck because the technology is safely guarded, but it's essentially a data center on wheels. Snowmobile uses what Amazon calls Zero-G racks, which suspend the system from both the top and the bottom of the truck to handle the impacts while in motion. And it has its own power and cooling. Once the transfer is complete, the truck enters transport mode. An armed guard and an escort accompany the truck as it returns to Amazon's data centers for the upload. Its location is monitored over cellular and satellite communications throughout the entire journey. Amazon intentionally left the truck devoid of branding to keep it discreet. 

When it connects to the AWS ingestion center. The data is decrypted and hashed. Validated, and then once the data is all unloaded and you validated it's unloaded, then the Snowball set up for another run for another customer. Just as Amazon has disrupted retail with its e-commerce business, the Snow products are an example of how the company has become a force to reckon with in the cloud computing industry as well. They're continually first with features. They're first with different ways of doing things, like networking. They have the most compute variation. They have the widest range of machine learning offerings too. Two years ago, Amazon deployed 1,440 new products and services. Amazon deployed 1,954, that was in 2018. It is sure Amazon will beat that in 2019. So innovation has been a key point. Security has been a key point. Our systems are the only cloud that's certified to run the intelligence agencies and DoD type clouds, top secret compartmented level of certification. And Amazon continue to have just a wide variety with the most number of databases available to our customers and the most variety of databases. The most robust storage platforms, the most number of options and compute.