Understanding Modern Cloud Architecture on AWS: A Concepts Series

Table of Contents

1. What Do You Need To Know?
2. Our Learning Scenario
3. What Kind of App Does This Work For?
4. First Things First - A Productive Development Environment
5. What Technologies Should We Use With It?
6. What Makes A Good Environment?
7. Watch The Series Here

Introduction

Welcome to the Understanding Modern Cloud Architecture on AWS blog series! Over the next couple of weeks I'm going to be diving into the components of modern technology stacks built in the cloud, specifically on AWS. Simply put, if someone came to me and asked, "Hey Cole...

How would you architect and build a modern application on AWS?

...well, this series and what we'll be talking about in it is the answer to that question. Now we're going to take a look at this topic from a 50,000 ft view. What I mean by that is we're not going to be peeling through code and technical step-by-step guides. I know that's tempting, and path of many developers when it comes to learning something new, but quite frankly it's the slowest way to get this particular topic under your belt simply due to how large it is. Diving right into a brand-new subject, without any idea as to what any of it is or does...well that'd be like trying to drive to a city you've never been to without looking at a map. Sure you might get there eventually but, without that clear 50,000 ft view of where you're headed, it's going to take you a lot longer. And you may wind up in Scranton, Pennsylvania instead of San Francisco, California.

Now, because you're here, I'm going to assume that you know of AWS. You know that it's dominating the public cloud market share and had about 47% of it in 2019. By the end of 2020's first quarter, I'd bet it's even greater. Because of this, I'm not going to waste a bunch of time trying to sell you on it or convince you that it's the right choice. Unless a radical mistake, political change, or natural disaster happens...AWS is going to be your best, safest, and stable bet if you're looking to get into the cloud. And I don't just mean for small hobby projects, or new ventures...I mean the entire rainbow of companies since enterprises and high-powered startups alike also use AWS. Netflix and Dropbox are perfect examples of companies that have practically built empires on top of AWS. But I'll leave the rest of selling it to the marketing departments.

One final thing before we get started - understand that the way we'll be approaching is not the "one true way" that renders all other paths as e-heresy. Contrary to what the most heated talking heads or social news commentary might preach, there's more than one way to create something scalable and all of those ways can be right. The architecture we'll be covering is one that will work for 90%+ of the firms out there considering that (a) most applications have very similar needs and (b) 98.2% of firms in the US have fewer than 100 workers which implies a smaller technology team which implies less resources to pour into development and operations. This architecture covers both of those points because it's made to scale to around ~1,000,000 users AND it's made to be operated by small teams.

What Do You Need To Know?

So, what do you need to know to get through this series? Well, really only one thing: Some general experience in the world wide web

Have you worked in the tech field at all? Have you built a simple website or application? Do you know what a server is? Have you ever even heard of AWS? Honestly, this series has been made to keep the amount of what you need to know pretty low. Admins, developers, and managers alike can get a lot out of this series. However if you're not in the tech field at all and this is your first conversation about anything tech, web, or cloud related...well, this is probably going to be a bit beyond your reach.

That all being said, you don't need to be some super programming or ultra sysadmin. All you need to do is really just sit back, relax, and read, or watch if you check out the YouTube version of this series. Most of what we'll be doing is a conversational exploration of concepts, ideas, and the REASONS behind why certain technologies are chosen. The goal here is to give you an awareness and understanding of all of the fabulous and modern tools out there that can be used to build applications in the cloud. Otherwise, if the only tool you have is a hammer...well everything's a nail. Similarly, if all you know is Heroku or traditional data centers, the solutions you can imagine and things you can build with them are far fewer.

Our Learning Scenario

Okay, let's talk about how we're going to learn our modern architecture on AWS. Instead of me rattling off documentation like a bad chat bot, what we'll do instead is build this thing out organically throughout the series.

"What?"

Well, what I mean by that is I could go ahead and cover auto scaling server fleets and containerization in the next post...but that's not a realistic flow. If you were starting this application from scratch with little knowledge of what it will come next, let alone how to work with AWS in the first place, you're not going to start off with the magnum opus of infrastructures. Instead, you're probably going to start with this:

Just your computer to start writing code on, and some users that you'd ultimately like to use whatever it is you're building.

And so our approach here is going to be to start simply. What's the first thing you'd need to do? Well, you'd probably code your application or service...and then realize you need to allow teams to work on this thing together. After that, the next thing would be to get it up on a server to allow clients to connect to it. And, after that, the need to set up a database to persist anything for your application comes up. And so on. However, you probably wouldn't start off with a containerized continuous deployment pipeline, right? When do you even need something like that? Why would you even do it in the first place?

Those questions, and a ton of others that come up in this process, are the reason we're taking this approach. I want you to see the decision making and the WHYs behind setting up the parts we do and what pain / need / cost creates the situation where you'd naturally arrive at setting that stuff up. Since this is how we build systems in real life, it's also is how we'll build the architecture.

Now what we'll ultimately wind up at will look like this:

An entire architecture that diagrams the flows of the developers pushing new updates to your application's code, how the clients access that application, and what's going on in the background. Throughout the process we'll cover the tools in general and the services that I recommend for getting the job done.

Now, we're not going to go into the technical nitty gritty. We're here to talk about strategy, tactics, and the authentic reasons behind the choices we make and the tools we select. Since I'll link to every service and tool we discuss, I'm not going to waste time reading the documentation definition of each thing. We want to build a map that can show you how to navigate cloud architectures and give you a compass of principles so that you can think independently.

What Kind of Application Does This Architecture Work For?

So here, let me go ahead and answer a question you already might have:

What kind of applications would this architecture work for?

Well, the short answer is 90% of all applications out there. Why? Because this architecture is the general pattern for almost any web-based application. And, just in case you want to ask "why" again, let me ask you a question: How different is any given web application from the next one?

After all, they all need servers...and a database...and static storage...they all need, well, a LOT of the same things. Even if you go to the application code, even though we have our custom code, the majority of that code is going to be the underlying frameworks or interpreters. What makes any given web application different from another one is, objectively speaking, quite small. And when we take this to hosting, scaling, and infrastructure, the differences are even smaller. Because of this, the architecture we're going to cover in this series can be applied to most things. Not only because of what we just talked about but because, nowadays, pretty much everyone has a web application of some sort.

Okay, now one last thing: Because this architecture is so flexible and universal, it is absolutely worth learning. For years, when I was primarily programming, I just brushed all this stuff off. I thought, "Eh, I don't need this stuff. It'll probably change by the time I'm done learning it anyway."

But the thing is this: if you learn this stuff, the web is no longer mysterious. When you can truly take an application end-to-end, things just make so much more sense. If you're strictly a developer or strictly operations, it can be hard to see the whole picture. And if you can't see the whole picture, well your ability innovate and create new things will be a bit handicapped.

But the second thing is a major reason why this stuff is so useful: it's evergreen . What do I mean by that? I mean that the stuff we're going to talk about doesn't radically overhaul itself every couple months. Just like how the SQL language has stayed the same for decades, the way you build out cloud architectures will also remain quite stable. And, as technologists, investing in skills that don't have an identity crisis every few months is how we can create some stability for ourselves.

And of course, when you take all of this and add to it the fact that companies want more and more and more cloud (though honestly I'm not even sure that half of them know what they mean when they say that), it's one of the best things you can add to your skill set.

First Things First - A Productive Development Environment

Every application has to start somewhere, which usually is on a solo developer's machine as they getting things set up. Even if it's for a larger team, there's usually a few folks who scaffold out the initial way of things first. The point is that applications usually start off small.

And small is also where good architecture begins.

Say your company comes to you and tells you the plans for a new web application that's going to be their APP OF APPS. They know they're going to have a team put together for it and, as you can guess, they ultimately want it on AWS. So...what do you do from here?

Well, before we can start constructing a productive development environment, we need to address what technologies we'll use to create the application. This topic, if you've seen any developer forum online, is an area riddled with battles, squabbles, and wars between developers everywhere. What's best?

Now, I'm bringing this up now because this first step is a very heated and opinionated topic within teams. And for solo developers or technical entrepreneurs, getting lost in the X vs. Y discussion is usually the death of the project. It can kill a project before it even begins. At the end of the day, this type of debate is a waste of time. Unless you're the next Google quite frankly, any framework or programming language is going to be suitable. It just is. They can all get the same things done. Yes, some will be FAR better suited for than others depending on the situation but they can all do what you need them to do.

That being said, I'm going to make a recommendation here, not based on opinion but on the raw state of things. If you're looking to build something on the web, you can't go wrong with JavaScript based technologies and frameworks. This has nothing to do with my preference or tastes. It's as simple as this: as long as the JavaScript is the sole scripting language of browsers, it'll always have a place. And since Google's V8 powers Node.js (so server side JavaScript) and since it's also a part of the Linux Foundation, it's definitely not going anywhere. In fact, it's only going to continue to grow.

Because of its widespread availability, massive community, and continually growing track record, you can't really go wrong with it. JavaScript also comes with the benefit that you can think in ONE language. You can write your servers, clients, data-crunching tasks, and administrative scripts in ONE language. We have enough moving parts as it is so, when there's a simple option available, it's nice break from the usual paralysis analysis.

And it's not all that bad either. If you take a look at the Most Popular Technologies section of the Stack Overflow Developer Survey from last year, you'll see that JavaScript is the most popular language in general in both categories of All Respondents and Professional Developers. But like I said, this isn't about love because if we go to the Most Loved section , we see it drops to number 11. However we do see TypeScript (which is just an embellished version of JavaScript) sits in 3rd. But if we go to the Most Wanted section on the same page, JavaScript is ranked 2nd just after Python. Love and desire aren't what I'd base my decision off of. And, even if you go with something like Python, if it's a web application you're still going to be dealing with JavaScript on the front-end. Honestly, if it's on the web, JavaScript is going to be part of it either way.

What Technologies Should We Use With It?

So more than likely, JavaScript is going to be part of our architecture. What else after that? Well, for server side and scripting, Node.js is the only real JavaScript option. React JS is a solid front-end framework of choice. Now what you choose to work with on TOP of those technologies is a rabbit hole we won't go down in this series. Alice jumped down the NPM package hole once and was never heard from again. Some say she discovered the Tower of Babel and was Webpacked into a minified module where her consciousness was scrambled and mixed in with millions of lost developer souls...and if that made sense to you be sure to level up.

Anyway, I'd say THE solid front-end because out of the 3 major front-end frameworks (Angular, React, and Vue), React is the only one being used internally by its creators on a core product. In this case, that's Facebook. When a company is using its framework, you can be assured that their development of it is based on real problems and needs, not just Ivory Tower theory crafting and not just solutions for small teams and small apps. Yes, I know I just praised Facebook, but despite their morally gray shenanigans they sure do craft excellent developer tools.

Now, whatever you wind up choosing, what you ultimately want is a set of technologies that are battle-tested and stable .

We talked about why JavaScript fits this because of how widespread it is and as long as you use this as your compass, you should be fine. What you don't want is some crazy bleeding edge technology because when there's a messed up bug keeping you up until 3am and you can't find an answer online (since no one's used it or done anything SERIOUS with it), you'll regret not choosing something with a better track record.

What Makes a Good Environment?

ANYWAY. The war of the developers has finally ended and your team has decided on a technology stack. I mean, at least one developer in your team is probably writing a manifesto about why you should all switch to something else but let's move on. With your technologies picked out, you can finally start thinking about what you need to do to create a development environment. Since it's very likely you've set one up before, we're not going to go deep into those specifics.

However, we are going to walk through the aspects that make a good development environment. Of course, making sure the language and any dependencies are available on your machine (and your team members' machines) is the first step. But, if you stop there, it's going to get pretty messy if everyone's not on the same page.

All of our machines come with...uhh, emotional baggage...which is all the junk from previous projects, past installs, age, operating system bugs, etc. It can be a huge stop gap in terms of onboarding new members AND helping existing members work on different parts of the application. This then brings us to the 3 core qualities that you want in your development environment.

The Qualities of a Productive Development Environment

#1 - Portability

The first is quality is Portability. While saying "It works on my machine" can be pretty funny, it's also a great way to give junior developers panic attacks. Jokes aside, you want your development environment to be portable so that you can allow new developers to come in and work on this thing immediately. Furthermore, if you change machines or upgrade your operating system, you don't want to be stuck days on end trying to get things back to where you can start working again.

Now, in terms of how to keep a development environment portable, a lot of it's going to come down to keeping the idea of portability in mind. Is the framework you've chosen to work with compatible on all operating systems? Do you have a list of all dependencies that your development environment needs to get up and working? Do you have a list of set steps that can walk someone through on how to set it up on their machine? Or even better, do you have a script that can do all of this for you? In the workshops I run, we cover how to create ultimate portability by using Docker but we'll get into how that can help later on.

#2 - Version Control

So, beyond portability, next up is going to be Version Control. This is a non-negotiable. If you don't know what version control is, you can kind of think about it like a video game that lets you create save files, giving you a checkpoint in your progress so you can stop, come back, and resume where you left off.

Say you're playing a video game and you get to a decision point - open a red box or open a blue box. Well, if you want to make it really easy, save the game and open the red box. After that, you can reload the game from your save and open the blue box if you'd like to. If you really wanted to play the heck out of the game, you could just keep both of those game saves and play them separately to see what happens.

Now this is kind of an oversimplified analogy, but essentially that's what version control lets you do with code. And it's important for the same reason that, if your little brother gets on to play, you'd tell him to use a different game save so that he doesn't mess with your progress. Similarly, developers will check out different branches of code so that they don't step on each others' toes.

Of course the main difference is that, unlike saving a game, the things developers are doing on their branches can be merged together when they're ready.

As far as what tools to use for it, Git is by and far the best choice. Hands down. I'm not even going to go beyond that. And, since you're going to have other people working on this code as well, Github is an incredible way to unify your codebase, and it's version control, in one place. With your codebase on GitHub, your whole team can work on it simultaneously from anywhere.

#3 - Automated Testing

Okay, and though there's no end to what you can do to make a productive development environment, the last key feature is Automated Testing. In a nutshell, for this, you just get your developers to write code that will test their code. That statement might be a bit confusing but in an ideal scenario, when your developers create a feature, they'll also create a set of tests that TEST if that feature works.

Now, contrary to popular opinion, this is less to test to see if the thing works and more to prevent people from breaking it in the future.

What I mean by that is that if Jerry the developer makes a new feature on this app and he's written some tests for that feature, in this case the tests ensures that the application prints text in blue. Then Bob comes in to update it. Well, if Bob comes in to do some small update, without tests, you better hope that his changes don't break other things. Maybe his update created some sort of side effect that prints the text as red. But thankfully, Jerry's written a test for that which makes sure the text is printed blue. Because of this, when Bob makes his changes and runs the tests for the whole app - it'll point out to him that he's messed up the text color.

AS YOU CAN IMAGINE...this is a MASSIVE safety net for your team and application. Yes, in a perfect world, developers will pay close attention to the work they do and the changes. But in reality, we're only human, and someone's going to do something wrong. And if you've got a solid suite of tests, you'll be able to catch it before your users do.

So, our qualities of a productive development environment come down to three things:

1. Portability
2. Version Control
3. Automated Testing

Now, WHY is this our concern? After all, we want to learn why we're doing what we do in this series. Well, these qualities set us up to seamlessly move features from our own machine to the infrastructure we'll ultimately build. And if we want to eventually create a process that allows developers to just write code, push it to GitHub, and have it automatically deploy to AWS, we need these three things.

Also a non-technical reason is that if you're stepping into the world of DevOps, or if you're the manager or lead engineer, part of your job is to remove the barriers in front of developers so that they can focus on making the app better. And finally, if you're the person managing your infrastructure, it's going to make your life a LOT easier. It doesn't matter how invincible your infrastructure is...if a developer is pushing out sloppy, slow, untested code, it's going to make your life a nightmare. Because when everything goes down, they're going to be asking YOU about it, not the developer. At least not until you figure out that it was the developer who did it.

Okay, and with that, your company now has a solid workstation to start building the application, as a team, in a productive stride. But we've got to show it to users, right? What's our next step with that? Well, the natural next thing on the to-do list is to get it up on a server so that's what we'll talk about in the next post.

Watch The Series Here

If you'd like to watch instead of read, the Understanding Modern Cloud Architecture series is also on YouTube! Check out the links below for the sections covered in this blog post.

Video 1 - The Introduction to the Series (for sections 1-2)

Video 2 - Our Learning Scenario (for sections 3-4)

Video 3 - A Productive Development Environment (for sections 5-7)