Containers Are Not the Future

I saw this tweet the other day and it prompted me to start arranging some more thoughts on the subject - putting pen to paper so to speak.

As I'll touch below you don't see engineers making statements like this that often in public as the container crowd will attack anyone that defies their worldview.

I read hackernews and I very much try to avoid opening up the comments section, however, every now and then I bow to the temptation and I'll see someone ranting about how they are ripping containers out of production because of security or performance or whatever the reason is (there's only like 10,000 reasons).

What's interesting though when these views are expressed is that, of late, I'm starting to see questions such as "I'm honestly curious - what are the alternatives?".

For a while I had no clue what the person asking the question was actually asking. What do you mean what are the alternatives? You spin up a linux box and call it a day - done.

That's when I realized that there has been an entire generation of software engineers that have grown up on this madness. Imagine if you were 22 or 23 back in 2013 (that's only 7 years ago) but in a world of software that's practically a lifetime. Hell that's long enough to graduate from college and earn your "senior engineer" spurs. You could've spent all of your 20s only deploying containers and not knowing any other methods even existed. Case in point - the term "ingress" is used in the k8s community as if it's some sort of product category and lo-and-behold nginx is used as a main component. Go figure. You'll see people asking questions like "What would you use for ingress?". What they are basically asking is how would you configure nginx as a reverse proxy.

I don't know if the education is missing or if the marketing is blotting out the sun or what, however, it's painfully clear now that we have failed to teach this cohort of engineers certain skillsets.

Resume Driven Development

A very interesting paradox going on is that a lot of kubernetes users are learning/deploying kubernetes clusters via "resume-driven development" and VPs of engineering and other leaders are letting them so that they can retain them and do what is already a very hard job to do - hire engineers.

Directors and VPs and other leaders are effectively throwing them a bone to keep them employed at their companies - not necessarily for technical/cost benefits. Many people in positions of management have privately admitted this.

I'll let you in on a dirty little secret. Software engineers in the SF Bay Area are spoiled rotten. Ok, you already knew that. What you might not have known is that those who call themselves devops, or sysadmins or SREs, if you are young and hip, have a skill that most software engineers are for some strange reason missing - they can deploy applications to production.

The secret here though is that this is also a perk in disguise because they hold the keys to the land of production.

However, this "perk" has started to fracture the software engineering ecosystem noticeably. This divestment of linux skills is going to hurt companies quite a lot in the future if they don't course correct.

Simon, presciently, noticed this was happening almost two years ago.

I think we are starting to see this accelerate lately. The tribal split is for real.

Security is the Death Knell of Containers

Containers and kubernetes have always had a very very bad rap when it comes to security but it just keeps getting worse. As demonstrated in this recent RSA conference video about advanced persistent threats in kubernetes it is clear that kubernetes represents an unheralded target for attackers. Traditionally attacking one host just gives you access to that host and while it might be easier to own other network addressable hosts you aren't immediately given the keys to the entire kingdom. That's not the case with k8s. You own one host - even the most minor application and you are done. Your entire infrastructure is up for game.

We've known for years that security was eventually going to take down the container ecosystem. It is simply broken by default and can not be fixed with any assortment of tools or architectural changes.

In this recent FOSDEM video, Kris Nova breaks into a kubernetes cluster to show how easy it is and then shows how to audit the break-in:

When you hear people talking on twitter or linkedin about "devsecops" in conjunction with security - it's a complete and utter joke. Kubernetes and security do not belong in the same sentence - at all.

As a side note, I think people should really drill into the fact that Kris is a developer evangelist in the k8s community and this is (at least) the second time she referred to kubernetes as a "clusterfuck" in the title of a talk. If their own developer evangelists are going to conferences and calling the software a "clusterfuck" why on earth would you bring this clusterfuck into your organization?

Why On Earth Would You Bring this ClusterFuck into Your Organization?

The problem here is that containers and by extension, k8s architecture, is broken by default. Despite what any vendor claims, there is no way to secure a kubernetes cluster. It never had security in design to begin with. It was a house of cards, with a foundation of sand, hawked by carnival barkers.

There were actually quite a few really tweetable quotes at FOSDEM this year such as the one in the last few minutes of this talk :

"The kernel developers view of the docker community is that in the rare case they can actually formulate the question correctly they usually don't understand the answer."

It was all a Game

Imagine you didn't read the prior paragraph though. After years of conditioning their audience on the " correct " way of provisioning and utilizing kubernetes clusters the big G introduced new pricing changes that made waves.

This predictably generated a complete and utter meltdown on companies that were already hooked on the GKE drug. I don't know why anyone should have been caught off guard by this - maybe it was the promises from Google stating they would never do this.

Even the analysts are starting to note that the kubernetes marketing swindle clock is ticking:

The Friction

Here's the rub - most containers and k8s folks are very averse to the idea of anything else being a part of the infrastructure paradigm. The problem is, these people have spent at least 5 or more years building their personal brand around the k8s mythology, telling everyone and their mother that this is the end all be all, the alpha and omega, and it is the best thing since sliced bread. Scratch that - it is better than sliced bread, home-made, with butter on top. It's a real blow to someone's ego if you tell them that there is life beyond k8s.

MicroVMs and NanoVMs

Google and AWS both know and acknowledge containers and kubernetes are broken and are not the future.

That's why they are hard at work improving things through marketing. I mean "open source", like firecracker and gVisor . Of course firecracker only works on very expensive "bare metal" instances (not really bare metal) and gVisor, well, I for one don't even know how that project got an "ok" at Google. Ptrace? Seriously? gVisor is another one of those projects where evangelists claims Google to have been running it for years in production environments when it is most clearly a replacement for something that existed internally that was somewhat similar.

Truth, lies, and cloud native.

Obviously, we have been preaching this general design pattern, a single program on an isolated vm, for some time now - to the extent that we felt it necessary to make our own kernel . There's a lot to be said for having a "stripped down system", ala bottlerocket , but there is even more to be said for ripping out components of a system that simply don't belong in the virtualized world of 2020. Users? Who needs them. Multiple processes? I thought the intent was to run only one thing? Interactive remote login? GTFO.

Now just to be completely clear - all of our engineers are happy, heavy linux users and have been for quite some time - maybe longer than some container users have been alive. This kernel is not designed to replace linux as linux is used for many things this kernel never seeks to address - for instance we would never provision Nanos directly on a real computer. Nor will we ever add ssh support to it (even if we could which we can't). Nor will we ever run it as a desktop. This is not the year of Enlightenment. Its sole purpose for existence is to be deployed in server environments.

More and more engineers are starting to embrace the general pattern and state the obvious:

It should be stated that this Ian is with the gVisor project. How many Ian's are we going to have in this post? Even Kelsey Hightower, a.k.a "the developer evangelist godfather" has gotten in on the action recently proclaiming:

It sounds a lot like real cloud native to me - you know - orchestrating the base unit of the cloud - the virtual machine. Colin might have the best thoughts here though:

For those of you who don't know - Colin is an extremely talented engineer, but I gotta love the quip from Matthew here as he flatly states out that kubernetes is not Borg. Borg is of course, the internal scheduler Google uses. One of the many cloud native lies is that kubernetes powers google. Many googlers/xooglers have came out bashing the marketing engine of k8s over this lie.

Conclusion

Containers are clearly not the future.

You will eventually need to migrate completely off of containers and kubernetes and we are here to help. We believe that microvms, particularly in the more precise form of unikernels are going to become a dominant server pattern for the future as the world hurtles towards ever more pervasiveness of software bringing private cloud/datacenters to prominence again (actually they've always had more than 90% market share) and newer edge deployments.

If you are wanting to see what is next or are just disillusioned by the cloud native world and want your organization to return to normal reach out. We can help you rid your organization of the scourge that is k8s/containers - we think this is such an anti-pattern that we will help you transition off and you don't even have to use unikernels.

Containers are not the future.