Cloud Storage: Bandwidth, Storage and BIG ZIPS

Welcome to Syntax on this Monday hasty treat. We're gonna be talking all about cloud storage.

You gotta put your stuff in the cloud. You gotta be able to access it. And because of that, we're gonna be talking about storage itself JS well as bandwidth, all of the different terminologies and services you need to know in this topic. The big zips.

Big zips. Big zips and big pipes.

Gotta have them both here. So let's get into it. My My name is Scott Tolinski. I'm a developer from Denver. With me, as always, is Wes Bos.

Hey.

Yeah. I've been dipping into this a little bit lately. I've I've hosted stuff on cloud providers for probably over 10 years now.

But recently, I was looking at moving some of my stuff to a cloud provider, and I was like, oh, let me revisit this. I've been using Backblaze b two for quite a while, but let's see if there's, anything else different. And I've been noticing the Cloudflare one a lot lately as well, Cloudflare r two.

So I I dipped into it, and I learned a lot. So here to explain it all to you.

Yeah. Sick. I'm excited about that. I I've stored on all kinds of things, so I'm interested to hear what some of these other options are that I have not tried to store on yet.

This episode JS well as every episode is presented by Sentry. Sentry is the perfect place to track all of your errors and exceptions.

This could be handy if, let's say, you know, I've had situations when I ran level up tutorials where people would say they can't access the download. Maybe it's throwing an error when they try to access the download. And if you're selling a product that relies on that download, truly important that your users aren't the 1st ones to bring that to your attention. So with something like Sentry, you could set up alerts for any error that happens in your application, and you can make sure that if there is a issue with a critical part of your infrastructure, whether that is grabbing a big zip from an s three bucket and downloading it or what, that you get access to all those errors and are able to solve it. So head on over to century.ioforward/ syntax. Sign up and get 2 months for free.

Alright. Let's talk all about it. Storing data can be confusing and expensive.

You have to consider a couple things. You have to consider the cost of storing that data. You have to consider the bandwidth of people downloading it, and you also have to consider how quickly do you you need it. So let's say you have large files. So all of the ones we're gonna be talking about today are what are called object storage or blob storage. And, essentially, what that means is it's a spot to put files. It can be any type of file. It can be any hunk of a file. It could be part of a file, but you can essentially think of them as a place to store large files. So things like big zips, software downloads, log files, images, videos, user data backups, all of those types of things are often not part of your application, but you put them on a cloud storage provider because they it would be too expensive. I'll talk about that in just a second. So let's first talk about a couple different ways that we use cloud storage. I just went through all of mine, and I I took a look at, like, what am I actually storing? So Mhmm. My biggest one is I use backup backups, meaning that I have a Synology NAS locally to in my house, and I have that entire thing. Not the entire thing, actually. I have the important parts, which is my recording files, courses that I don't want on my laptop, stuff like that. I have that mirror to Backblaze b 2, and that's about 2 and a half terabytes of of data. And anytime I throw something onto my NAS, it it just auto auto syncs and uploads uploads it to Backblaze b 2 so that I have a backup backup because one backup of a file is not a backup. That's where your file lives.

Yeah. That's actually that's such a good point. And it does feel like the era of, oh, man, your drive JS all of a sudden stopped working and your your backup is toast is over, but that doesn't mean that. You shouldn't be I literally, last night on TikTok, I saw this photographer.

And, like, maybe in, like, the developer community, this is a thing of the past. But my sister's a photographer. I talked to so many of them. They have dozens of these little drives all around their house. They're probably moving around. Every at least once a year, a year and a half, I get a Sanity message from a friend of a friend.

My hard drives are clicking, and I can't turn it on. And it has wedding photos from this couple. Like, please.

Especially just like a little thing that is so easily easily dropped. This is not the backup show. Well, maybe we'll do that again, that show as Wes. But, man, what else do we put there? I put the original recordings on. So when I'm recording a course or this podcast, the raw files can be very, very large. And sometimes you wanna keep those raw files, so it's nice to have a spot to put those. I put caption files on there so I can download them and put them into my course player.

These big zips for all of my courses is another one that I do. So, like, 99% of people stream the course. Right? You got a nice experience. You get the notes along with it. You get you track your progress, but some people just want the the m p four files so they can download it and watch it locally. So in order to to allow that, I I throw them up on, one of these cloud providers and allow them to download it. Older code Bos. I have, like, basically, like, 15 years of web development of old websites that I've built, and, like, I don't delete them. I just put them up. You could put them on GitHub as well, but I just drag and drop, put the whole sucker up on a cloud provider.

And then I've been recently this is why I got into this is I've been trying to find a solution for going off of cloud app, which is like my screenshot kinda quick file hosting. You need to send somebody a file. You need to send them a video. And somebody on Twitter recommended drop share, which is really cool. It's like a essentially, a screenshot tool, also file hosting, but you pay for the app once, and then you connect it to any of these cloud providers.

And it's that seems ideal to me. So I'm I'm looking at figuring out how to move all my 15 years of screenshots and Mhmm. File uploads over to that. It does seem like a a good

approach to that. Right? Because then it's not tying you into, you know, their full infrastructure. You can still own the bucket with all of the data. Right? Exactly. Yeah. Yeah. I like that a lot. I I use cloud storage for a few things. Oh, images, maybe not something you mentioned. Images Yep. Pay. Especially in websites where users are uploading images, things can fill up really quickly. You know, back in the day in WordPress and Drupal land, you just uploaded it to your own local server.

That that was fine. But, you know, if you host these in the cloud, they can be accessed by all kinds of things and not just the current application that you're using. Either way, user images, you can create transformed images.

Video, oftentimes, you'll need to transcode video into a whole bunch of different formats, especially if you're streaming video.

Yep. Like you said, big zips. I throw my zips up on a a a Backblaze b two that we'll talk a little bit more about, but I I hosted all all my big zips there as well.

And and just really, like, that that's pretty much it for me. You know what? I don't have an off-site backup, Wes. I have a, like, a double backup system, but it's all local. So yeah.

You gotta fix that tonight. You're not gonna be able to sleep. What happens you the last podcast, you just talked about a tree falling on your your office. What happens if a tree goes right through your hard drive? What if someone breaks in and steals it?

I'm pooched. I'm pooched. Yeah. You're right. Yeah. You're right. You're right. And I did lose a 200 gigabyte LaCie hard drive back in the day, and I lost most of my video projects that I made when I was at early college, which is fine since it really sucked. Yeah.

So let's talk about some things that we're not talking about really quickly is your your Vercel, and, like, why would you want larger storage? So usually, when you host your application, you get a server, and that server will have space that you can upload files to that will run your application. Right? And they're often limited in terms of both how much space you're allowed to upload things to. So they'll give you, I don't know, like, 20 gigs or or or 10 gigs, something like that. You can upload your whole application there, and often you can put files and and images on there if you want, but the bandwidth for those things are very expensive because the people who supply you with compute are not necessarily the same people that supply you with, like, large amount of storage and bandwidth.

Often, people will just stick a CDN in front of those files that are on the same server. That's a really good solution. This is not a show about CDNs, not a show about image compression, video streaming, things like that because that's a whole another level. And often those services that are image resizing, compression, CDN distribution around the world, those things sit on top of just your raw hosted object structure, which is what we're talking about today, which is basically just a place to put

big files, big zips. Yeah. And and it's it's largely because these things really specialize in holding a lot of data, not necessarily distributing you that data. That's not their, you know, their primary task.

Exactly. So big players in this space are the they're like I think the big first one was Amazon Wes three. They kinda came out and said, hey. You got wanna put stuff somewhere, and you'll probably have have clicked on files at some point. And you see in the URL, it says, like, bucket or s 3 Scott or, US East. It's like in the URL of the Node, it often will say these words, and that's kinda how you can tell that, oh, these files are actually being hosted on Amazon Wes 3. And then along with Amazon s 3, there's a whole bunch of different options out there. Cloudflare r two, which I thought was kind of funny. They took the letter in the number before Wes 3 and named it. But then as I made this list, I realized Backblaze b 2 is also like, did they do the 2 because of Amazon Wes 3? And then What's up with that? Yeah. I I I'm assuming they did that.

And then Synology has 1, which I didn't realize, c 2. And then all the usual suspects, Google Cloud Storage, Azure, DigitalOcean, Oracle, they all have this. And often, they are all s three compatible, meaning that if you if you just take a library, like Scott was talking to me about pocket Bos, and you can you can hook it up so that it'll it'll upload files to s 3. But you can just take your Cloudflare r two credentials or your Backblaze b two credentials and pop them in there because a lot of these APIs are s 3 compatible, so they'll work with with anything.

Yeah. S 3 was certainly the big player for a very long time.

But, yeah, I find myself using Backblaze b 2 anytime I need to host anything anywhere just because I'm very familiar with it. However, Cloudflare r two. Yeah. I like Cloudflare a lot. Maybe maybe I'll try that sometime. It's a Yeah.

You're you're gonna you're gonna hear us talk a lot about Backblaze and Cloudflare both because like, I've I've used Backblaze, Amazon Pnpm, Cloudflare, Google Cloud Storage Mhmm. And Azure Store. I've used all of those, but I like Cloudflare and Backblaze the Bos, and I'll I'll tell you why in just a sec. But also this, like, bunny.net keeps popping up as well. They seem to have both, like, streaming products, CDN products, but they also have a storage product as Wes. And the prices on it seem very good. So

I've never seen Bunny.

Yeah. They That's interesting.

Every time I talk about it, somebody comes out and says, hey. Check out Bunny because they have a lot of interesting products out there.

Yeah. Oh, that's that's great. You know what? The I think the big reason that I picked Backblaze and and would even look at Cloudflare before Amazon isn't because of anything other than the dev experience of AWS.

You know? It's really easy to get going with Backblaze or Cloudflare for that matter. Things are more findable, less opaque, and it's just the nature of AWS.

The I was thinking the other day I was logged into my Backblaze, and I use Backblaze both to mirror my computers, and I use b two to host a lot of files. And their entire their entire product offering, this is a massive, massive company, is, like, 15 links in the sidebar. And, like, there's there's that's it. Organization. Come on.

It's it's so good. You use the the computer backup product?

Yes. Oh, yeah. I've been probably for 10 years now. It basically is just always and that's one thing I'm doing is I'm always pausing it before we record this episode, because it will continually back up my entire computer's hard drive to backlaze b 2. And if it were to, if I would ever lose my computer or drop it in the ocean, something like that, I'd be able to download my files. I've used it a couple times just to roll back and get older versions of files as well, which is something most of these providers do as well. If you if you upload the same file again, it keeps the older versions, which you can roll back to. Yeah. Yeah.

Speaking of dropping your computer in the ocean, do you ever think one day it would be feel real good to just, you know, frisbee that thing right out into the ocean when you're feeling really angry at it? You know? But then I would quickly go get it so it doesn't harm the environment. But yes. Sure. Well, the the yes. Okay.

This is a fictional thing. I'm not doing it. Fictional. Yeah. It's like an office space type of thing. Yes. Totally. Alright. So let's talk about we're we're gonna break it down into storage fees, bandwidth fees, retrieval fees, and operation fees or or retrieval speed. Sorry. So the first one is storage fees, and you can kind of break down the cost of of these things into how much does it cost to hold data in there month to month, and how much does it cost to get the data out if people are downloading it? Because some of these providers are cheap to hold data but more expensive to download it, and some of them are a little bit more expensive to hold data, and they go as far as being free to download them. So storage, it costs money just to hold on to data. Cloudflare is a 1 and a half cents per gig per month, whereas Backblaze is 0.6¢ per gig per month. So I'm taking a look at mine. I have 2 and a half terabytes of data being stored in Backblaze b two right Node. And because that is purely not purely, but almost entirely backup and downloads for my video courses, I actually don't incur any bandwidth, and it's just storage fees. So for 2 and a half terabytes of data is $15 per month for me to sort of hold it on there, which is that's cheap, but then you start thinking about, like like, this podcast Node. After we're done recording it, saving the high quality versions, editing it, and exporting it, every episode's probably wet. Like, we should get Randy to tell us, but it's probably 30 gigs. Right? Yeah. It's not small. If you think about 30 let's do the math right now. 30 gigs times, 150 episodes.

That's 4.5 terabytes a year for this podcast.

So that would probably cost $40 a year to host or to sorry. To hold on to GitHub even downloading it. Just to hold on to it. And, $40, not not too much, but then you start to think about like, I often think about Riverside.

You know? How much are they paying in in order to host all these files and hold on to them for forever? Because Riverside JS not it's not expensive. It's like $250 a Yarn, and we have probably

5 terabytes of data stored with them. Yeah. That's a you know, it's funny, Wes. I was just doing some clean out of my NAS because, you know, it just builds up a lot of duplicate files or things that are just, you know, stuff you didn't wanna keep anyways that just kinda made it on there. And I found a raw footage, file from a ScreenFlow project that was 200 gigabytes.

And I was just like, what

is go I was like, oh, this is taking up so much space. I I must have just recorded a very long time. I I didn't even look at it. I just deleted it. I was like, why not? If you were to pop that on Cloudflare,

200 gigs, 1 and a half cents each, That's $3 a year to hold on to that file.

That's that's kind of expensive, don't you think? Expensive.

Yeah.

Yeah.

Yeah. That's that's the truth. Oh, man. So

other ones, I won't go through all the prices. Amazon Wes three is actually quite a bit more expensive, 2 and a half cents per gig. And then at least when I did it, this was many years ago, it was also where you store it, it costs different amounts, like, because you have to pick where you're gonna store it. Right? Do you wanna put this in Virginia, or do you wanna put this in Australia? And then when when I was we'll talk about Bandwidth in just a sec, but I off I was looking at the prices, and and I have a lot of people from India download my courses.

And India is the most expensive, at least it was at the time, for people to to download from. Maybe that was their CDN product if I'm thinking about that again because, yeah. That that doesn't make sense that it would cost more depending on where they're coming from. It's just the Internet. Right? But it it would cost more if you need to put them in different different locations.

Yeah.

So the this stuff is so cheap. And, you know, we just talked about it being a little bit of expensive, but, like, it is so cheap And I think that's due to, 1, the drives that they're storing this stuff on, they're not optimized for read and write speed. Right? They're optimized for a long term storage.

Even when you buy discs for your local NAS at home, the drives that you're buying are different. They're not optimized to read and write fast.

Because of that, data has gotten so cheap that now we can just toss a whole bunch of stuff up on the on the cloud JS really what it comes down to. And it doesn't need to to be super duper fast. And if it does, then you're probably gonna be paying more for that feature. Right?

Yeah. There was a a website I was on, and they were trying to explain, like, this concept called temperature, which is, like, the faster it is to retrieve your data because some some of it does need to be retrieved extremely fast. Mhmm. The faster it is to retrieve your data, the more expensive it is. And you can even go 1 step further. And if you put a CDN in front of one of these, it's even more expensive because they're distributing copies around the world, so it's faster. And the slower it is, the the cheaper it it gets because, like, Amazon has a product. This is one of the features things, but I'll just talk about it now. Amazon has a thing called Amazon Glacier, which is offline cold storage. I think Azure has their own version of it as well where it will take longer. Like, I went to download some of our old episodes, and it takes 8 hours. You have to request it, and then 8 hours later, you get an email and says, hey. It's ready. Or 24 hours later, you get an email. This is ready for download, and that's because they are putting less frequently accessed episodes on cold storage so that it is much cheaper to hold on to, and then there's a process for getting it off of tape. You have to go into Jeff Bezos's basement and pull the VHS off the off the shelf, and they some poor person has, like, a FireWire 800 on a old Mac, and they pnpm it in and and put it on a USB stick and then run it over to you. Wow. Something like that. Yeah. That's the whole that's quite the story. If we had an animator, I would love to have somebody say that.

Let's talk about bandwidth now. So storage is one thing to think about, but the other Node is to think about bandwidth or often is referred to as egress. So when your data now needs to leave the server because it is being requested by somebody to download that file, then you also will incur some sort of costs.

Now the kind of interesting thing that has been happening lately is a lot of these cloud storage providers are giving you free egress, which is wild to think. So Cloudflare r two, I just told you, it's 3 times more expensive than Backblaze b two, but it has free egress. So if you're hosting files that need to be maybe they're they're a little bit smaller, but they're downloaded a ton. Like, you think about these websites that offer, like, NES ROMs or like, I downloaded a couple Wii games when I hacked my Wii. Yeah. I was like, who's paying to let me download this, like, 6 gig file of Wii bowling? Or I I don't know how big it was, but it was many gigs. You know? So I often think about that. I'm like, that's I know how much this Scott. That's kinda expensive

Yeah. To to to incur the bandwidth Scott, but maybe they're using Cloudflare r two. And is it just ads that's covering the cost of that? And it feels crazy, but they must be getting enough traffic to make it worth it. I I I do wanna say, before we move on off of egress here as a concept, if you've ever heard the word ingress JS the opposite. So ingress is data coming into the system.

Egress Mhmm. Is data coming out of the system.

You have an egress window at your house that's for leaving in case of an emergency. I got one right here. Yeah. So egress, ingress. Those are are 2 words that it's like those are kinda jargony words that you may have heard here and there, but ingress, in. Egress out.

Why is it not called outgress? We don't know. Yeah. That's actually a good idea. Yeah.

So Cloudflare, free bandwidth, free egress. Backblaze b two, it it kinda has this really interesting thing Wes they'll give you 3 times your storage in free bandwidth.

So if you I'm hosting 2 and a half terabytes, so that means I have 7 and a half terabytes of free bandwidth. So, like, honestly, I'm gonna say right Node, it's very hard to beat back Blaze b 2.

And they're, like, just like a wicked all around company just because, like, there's no weirdness around them. There's no, like, marketing fluff or anything like that. It just it's a really good company. So if you have lots and if you use lots and lots and lots of bandwidth, probably Cloudflare r two is is for you. But if you're storing stuff long term and you have, like for example, my because I store so much backups, my actual course downloads never incur any bandwidth because I'm sort of offsetting them, in that regard.

Yeah. Backblaze does 1 or 2 things. It does them very well. They store things. They back up things for

sure. Amazon Wes is 9¢ a gig for bandwidth, which is very expensive. I'm surprised a lot of people. That's the other thing about it is there's a lot of people on Amazon Wes 3, but they can't necessarily get off of it because if you have 70 terabytes of user images, how much is it gonna cost to move those all off of Amazon? You're gonna incur the bandwidth for everything and and then moving it off. So sometimes a lot of people say it's cheaper just to stay on this than it would be to to use Amazon Wes three and and to move it off and pay the egress fees for that. It's also a pain in the ass to

even update all the paths now. You have all these paths that presumably have all now changed. Right? I had to do that with LevelUp because s three, I found to be very slow when I was hosting my courses there. I just moved into Backblaze. It was gonna be cheaper and faster for me.

That was a total pain. It was a massive pain to to swap out all of those those paths. Oh, yeah. Yeah. Yeah. Here here's my tip for that is don't use the Amazon Wes three URLs.

Put a custom domain name in front of them.

Because if you do ever change them, at least you can write like a worker or something like that to jump in between and and rewrite those paths. Still pain, though. Right? It's huge pain. That's a lot of work, and it's likely not worth it for a lot of people. Totally. There's also this thing called the Bandwidth Alliance, which gives you either free or reduced cost of egress between service providers.

So talk about fat pipes.

A lot of these cloud providers got fat pipes between each other. So, for example, Backblaze and Cloudflare, they have, like, a a little string behind the offices where they can call each other directly, little tin can.

And they say, alright. Well, if somebody asks me for your data or if I ask for your data, it's gonna be free.

And that's really cool because if you are doing if you have a worker on Cloudflare, but your files are stored on on Backblaze and you need to download the file in order to to work with it in a worker, that's free because it's it's just going data center to data center, and it's not really costing them that much because they got a fat pnpm,

between the 2 of them. You're really killing the, metaphor today.

Thank you. Thank you. Who else is part of this? Azure and Google Cloud are discounted, they say. Mhmm. And then there's free I'm trying to see any of the big ones out there. There's a list of automatic is a big one, which is kinda interesting. So I guess, like, if you've got a massive WordPress site and you want to move your WordPress files to to Cloudflare, you could or vice Vercel, DreamHost, Backblaze, Oracle, and then there's a bunch I have never heard of here. Last thing we have here is just operation fees. These also cost money. So every time that you create a what's called a bucket, which is kinda like a folder in these cloud providers, that's that's an action. Right? It's it's an API hit. Create a bucket. Right? Mhmm. Every time you upload a chunk to a file, that's an action. Right? Every time you see if a file exists, that's an action. Every time you, delete a file, that's an action. Most of those actions, except for, like, deleting files, they charge you by the million, for those actions. So there's not a a a ton of them, but every single request of a URL, it has to pull up that file that lives behind that URL, and that is one action.

And the cost for those are are relatively minimal. What do we got here? Cloudflare puts them into class a and class b operations, which is class a is like upload, list a file, list files in a bucket, list all of my buckets.

And then class b is, like, check if a file is there or or or get the file size, but not download the file.

So those are 1,000,000 for $4.50 a month, and then 10,000,000 of the class b is 36¢ a month. So if you have a 1,000,000 images that are downloaded, that will cost you $4.50 for the initial request, but then the the bandwidth would be free. Yeah. Most people Scott something that they're worried too much. I I think and, also, like, if this if if you have such a scale that big, then you're going to basketball games and having dinners with the the sales reps

at these these companies. Basketball

games. Yeah. You're you're negotiating, like, your own terms on these things. You're not taking the pricing page carte blanche.

Mhmm.

So that's that's all we got today for cloud storage, providers, Bandwidths, and big zips.

Hopefully, you learned a thing or two, and we will catch you later.

Peace.

Peace.