The Death of Hype: What's Next for Scala

Standardizing IO Interfaces for Scala Libraries

A recent tweet by a friend of mine noted how the public interest in the Scala programming language seems to have plateaued or waned, which matches my feeling of the latest trends and zeitgeist. This blog post will go into why I think that has happened, where Scala stands now, and what the future holds for the Scala community.

This blogpost was inspired by the following tweet:

It's a shame to see friends from Typesafe/Lightbend being laid off, but as I've always said, it's hard to sell free stuff.
I do wonder about #Scala's adoption now. It's not Clojure-dead, but it seemed to peak around 2016. I have zero data to back that, though.

• Jamie Allen

Jamie notes that public interest in the Scala programming language seems to have waned: less interest in meetups, less interest in conferences, and so on. This more or less matches what I am seeing as well. While the Scala-specific conferences targeting enthusiasts are still going strong (or were, until COVID19 shut down the global economy) I think there is no doubt that there is much less talk about Scala in the broader engineering community.

I think that this reduction in public interest is real, but also expected given the typical "Hype Cycle" that follows any new technology:

Early on, Scala rode a wave of hype that frankly surprised even me: hype around pushing syntactic boundaries, hype around reactive architectures, hype around functional programming, hype around the Apache Spark project. Much of that hype has since died down, and there was a period of backlash and negativity both within the Scala community and outside of it. Since then, even the backlash has faded, and what is left is a reasonable, boring language steadily advancing and providing a great platform for general software engineering.

The Past: the Peak of Inflated Expectations

While Scala was initially created in 2004, the language only really started taking off in the early 2010s:

• ~2009: Twitter started picking up the language for production systems

• ~2010: the first tagged release of the Scalaz functional programming library

• ~2011: Lightbend, a company pushing Scala and related technologies, was founded

• ~2014: Apache Spark was written in Scala

• ~2015: first tagged release of the Cats functional programming library

While there was no doubt activity in the Scala community before this, these were no doubt major milestones that kickstarted interest in the language. With large-scale commercial adoption, commercial support, and a killer big-data product like Apache Spark, it is not surprising that there was a surge in public interest. A lot of these efforts went into the following domains:

Pushing Syntactic Boundaries

A lot of early Scala interest was in how flexible the language was: it had extension methods, operator overloading, implicit constructors/conversions, and a very flexible implicit-parameter feature that could be used in all sorts of ways. This flexibility was liberating, as it opened up all sorts of domain-specific-languages and programming styles that would be unheard of in other languages.

These were exemplified by a number of projects, such as the Databinder Dispatch HTTP client, Scala-Graph data structure library, or SBT build tool:

// Making a HTTP request and processing the output in Databinder Dispatch
executer(:/(host, port) / target << reqBody >- { fromRespStr })

// Constructing a simple directed graph with labelled edges in Scala Graph
val flights = Graph(
	(jfc ~+#> fra)(Flight("LH 400" ,10 o 25, 8 h 20)),
	(fra ~+#> dme)(Flight("LH 1444", 7 o 50, 3 h 10))

// Append the contents of a URL to a File after filtering through grep using SBT
url("http://databinder.net/dispatch/About") #> "grep JSON" #>> file("About_JSON") !

// Search for uses of null in the source directory using SBT
"find src -name *.scala -exec grep null {} ;"  #|  "xargs test -z"  #&&  "echo null-free"  #||  "echo null detected"  !

Many of these libraries have evolved since the early days and have since moved to more boring, less interesting APIs. It turns out that naming operators >- ~+#> or #&& , while possible, was not always the wisest thing to do. Nevertheless, there is no doubt that pushing the boundaries of Scala's syntax using operators and domain-specific-languages was one of the big points of interest for early Scala.

Reactive Architectures

Reactive architectures, pushed largely by Lightbend via their Akka Actor framework , was another big effort. This was a way of writing code that was very different from "normal" code that someone might earn in school, but allowed high levels of concurrency and performance beyond what is achievable writing code in a traditional style.

In its initial incantation, Akka Actors sacrificed a lot of what Scala what it is: a total lack of type-safety between actors, prohibiting blocking APIs, and many other idiosyncracies. Working with Akka Actors were basically their own language, embedded in Scala, but with their own conventions, syntax, and semantics.

More recently, with Typed Actors and Reactive Streams, the Akka developer experience has shifted to be much closer to the type-checked experience you would expect in typical Scala programs. I think there's also been a shift in the community mood: rather than treating reactive architectures like a general-purpose solution to all problems, they are now being treated as one possible approach for one specific class of problems. That mostly matches my own experience using these techniques in my own applications.

Functional Programming

Functional programming in Scala is best exemplified by the Scalaz and Cats projects. These two libraries attempt to bring much of the functional programming techniques from Haskell to the Scala language: a heavy focus on concepts like Monads or Applicatives, eschewing the object-oriented side of the language for a more pure-functional style.

The two libraries have different styles, and both remain heavily used by portions of the community today. The evangelism has died down, to some extent, but usage remains strong and everyone recognizes functional programming as one possible style in which to write your Scala applications.

Apache Spark

The last big project that drove Scala's early hype was Apache Spark. A distributed big-data process framework, Apache Spark leveraged Scala's functional style and serializable lambdas to take common collection like map , filter , and reduce and run them distributed over a compute cluster processing huge amounts of code. Apache Spark allowed big data processing pipelines to be written in orders of magnitude less code and running in orders of magnitude less time than the earlier Hadoop APIs. Spark proved an instant hit.

Apache Spark was Scala's killer app. A huge fraction of the Scala community is only using Scala because they need to inter-operate with Spark in some way.

Apache Spark now supports developer APIs in many languages: SQL, R, with Python being the most popular. The original collections-like APIs have been largely superseded by a more SQL-like interface that allows better optimizability, an important factor when working with the huge datasets that Apache Spark is often used for. Nevertheless the Apache Spark codebase remains mostly Scala to this day.

In the Trough of Disillusionment

Since the initial spark, there definitely has been some backlash against Scala as the initial hype has died down. Some early contributors have left the community, and there was some inter-personal unhappiness. This was not just in aggregate, but on an individual level as well: I personally know many individuals who got tired of the things they were hyping, and moved on with a slightly sour taste in their mouths.

It's hard to pin-point an exact time where the hype died down, perhaps around 2016-2018.

Even on a technical level, not all was well. While many organizations used these techniques successfully in their respective niches, others have regretted going all-in on actors and all-in on functional programming. They were finding that these techniques don't fit their problem as well as they would have liked, and had to spend some time backing out of their all-in investment in a particular programming paradigm. I personally know many organizations who have had to make this adjustment.

Even back then, in the early 2010s, I was surprised at the amount of hype around some of these technologies. The usage of operators in some of these libraries was clearly through the roof, and sometimes turned even simple tasks into complex brain-teasers. The amount of hype I saw around reactive architectures and functional programming did not match the specialized use cases I found that they fit best: every Scala conference had keynote speakers evangelizing both of these approaches as the be-all end-all future of software architecture.

It surprises me not at all that over time, others would join me in realizing the Trough of Disillusionment .

The Present: on the Slope of Enlightenment

Scala had its hype in the early 2010s, followed by a backlash against that hype in the mid-late 2010s. Where that does that leave Scala now?

A Steadily Growing Community

Despite the apparent downturn in public interest, usage of Scala continues to increase. I maintain a large number of open source libraries, and the number of unique IPs downloading these libraries from the Maven Central package repository has grown just over 2x over the last year. While there is some variation in the number of downloads, or the metrics for each individual package, overall they match this trend quite closely.

While this growth isn't the exponential rocket ship that some projects are hoping for, this is a steady increase in usage that matches my own subjective experience: usage of Scala continues to increase even as the hype has died away. If we can maintain this 2x growth every year, I think the Scala community would continue to be in pretty good shape!

If you look at the most recent Redmonk Rankings or Tiobe Rankings , they both show Scala as stably hovering just outside the clique of mainstream languages, ranked 13 and 28 respectively. Uncommon, but not obscure.

With the hype and the backlash both distant memories, we now see Scala for what it is a flexible language supporting multiple programming styles, each with their own tradeoffs and use cases. A lot of the early religiosity and conflict has transformed into people using Scala for steady work, enjoying the language in the way they each choose.

A Solid Platform for Production Usage

Despite the reduction in hype, the Scala language is in a lot better place than it used to be:

• The SBT build tool has improved by leaps and bounds in the last few years: massively faster and easier to use.

• If you don't like SBT, you now have a choice: many organizations use Bazel or Pants with Scala, while my own Mill build tool is stable and widely used

• New tools like Coursier are forming a new foundation for the ecosystem: used by tools from Ammonite to SBT to Mill

• The METALS project brings Scala support to VSCode, Sublime Text, and any other editor with language-server support

• Intellij's own Scala plugin continues to improve: faster, more accurate, and with support for new tools like Ammonite scripts or the Mill build tool.

• The new collections library in Scala 2.13 cleans up a huge number of old warts and significantly improves performance in many cases (e.g. 25-40% speedup on Sjsonnet )

• "Polish" features like pattern matching on strings or warning suppression continue to make life better for everyone

• The compiler itself is massively sped up, with code compiling literally twice as fast as it just three years ago

While the language specification has hardly changed at all in the past few years, the amount of quality-of-life improvements that have landed is hard to overstate. If you had talked to any Scala-using engineering organization about halving their compile times back in 2017, they would have showered you with money to make it happen!

Each of the these individually has significantly improved my own personal experience writing Scala, and there are many more improvements that I cannot comprehensively list here.

A Broad and Varied Ecosystem

While the hype around the original four efforts has waned, these still form large and active sub-communities within the Scala ecosystem. For example while not everyone may want to use pure functional programming, and any individual may not want to use it all the time, the libraries and ecosystem surrounding it are nevertheless varied, high-quality, and well-maintained. When you do encounter an occasion to use pure functional programming, you will find everything you need.

In a way, the hype has served its purpose: by introducing a large number of people to a particular paradigm, it helps achieve the critical mass for some fraction of those people to sustain that sub-community going forward.

The broad range of styles in Scala can be thought of as a boon: rather than considering it a way of splitting the community apart, it is a way of bringing together communities that would otherwise never interact with each other. Without Scala, these communities may be disparately working in Haskell, Python, Java, and Go systems. By focusing on what's shared, rather than what's different, Scala allows these communities to collaborate on the things that benefit them all while still preserving the things that make each sub-community unique.

I personally do not default to architecting my code reactively using Actors, but I have shipped production code that heavily uses Actors when the situation called for them. I do not default to pure functional programming techniques, but I have used pretty esoteric constructs like Free Applicatives when useful. I think the community as a whole has matured in a similar way: appreciating the different styles, and using them where appropriate, without the dogmatism that plague the early days of Scala

The Future

So what exciting things do I think lie in Scala's Future? To me, the Scala language itself is in a relatively good state: an afternoon writing modern Java code is enough to relieve me of any illusions that other languages have "caught up". Where the greatest potential lies is in introducing Scala to new sets of people, or new sets of use cases, that up to this point have been completely out of bounds.

I think there are two main things that stand out to me:

Usability

In the initial four Scala efforts, one thing was missing from all of them: the ability for a new developer to start on a codebase and be immediately productive. It doesn't matter whether you were using an operator-heavy dispatch style, a actor-heavy reactive style, or a pure-FP Cats/Scalaz style: someone trying to onboard was going to have a hard time.

Why does that need to be the case?

As a counter-example, developers generally find Python an easy language to get started with: "executable pseudocode" they call it. New programmers learn Python. Non-programmers learn Python. When people write code on a whiteboard with a marker, they often write Python. While Python's performance, concurrency, or large-scale maintenability are questionable, its ease of getting started is unmatched.

Why can't Scala be as easy to get started with as Python?

Answering that question is the goal of the entire suite of libraries that I maintain. Many of my libraries are carbon-copies, shameless clones of their Python equivalents. For example:

• Ammonite , a clone of the IPython REPL

• uPickle , a clone of the Python json module

• PPrint , a clone of the Python pprint module

• Requests-Scala , a clone of Kenneth Rietz' Requests module

• OS-Lib , a clone of the Python os , sys , and shutil modules

• Cask , a clone of the Armin Ronacher's Flask library

It turns out that you can make Scala as easy to get started with as Python. And for all of Pythons weaknesses - performance, concurrency, maintainability - Scala already does excellently. Scala has the potential to be the language with the usability and flexiblity of Python with the performance and scalability of Java or Go, and type-safe maintainability far beyond any of those options. Quite literally the best of both worlds!

While there will always be use cases where high-concurrency reactive architectures or pure-functional programming are the best tool for the job, I hope going forward that developers will be able to onboard onto the Scala language in an easy and straightforward way, and only need to learn those more advanced techniques when the need for it comes.

Scala Native

Scala.js has demonstrated the feasiblity of using Scala on an alternate platform: compiled to Javascript instead of on the JVM, it has a vibrant ecosystem and community, and brings the Scala language into the browser, somewhere it could never be used before.

Scala-Native promises an even bigger revolution: by compiling Scala to self-contained binary executables, that would allow the Scala language to be used in scenarios where the startup time and resource overhead of a JVM or Javascript runtime is unacceptable. While Scala.js brought Scala to the browser, Scala-Native promises to bring Scala to a host of new environments:

• Command-line tools like git , ls , rsync , etc. which cannot tolerate the startup time of the JVM.

• Sidecar processes or background daemons, where the significant memory overhead of the JVM makes using Scala problematic

• Mobile development on iOS, running on iPhones and iPads, one of the most ubiquitous consumer computing devices in the world

• Desktop applications, where the JVM startup time is too slow or bundling the JVM too bulky

• Deep integrations with native libraries, like the Tensorflow machine learning framework

These sorts of programs have been traditionally written in unsafe languages like C or C++. More recent implementations are often written in Rust or Go, and there's a whole cottage industry of not-C programming languages like Nim, Zig, or Crystal also trying to fill this space. Scala-Native promises to be able to fill this space, not with a brand-new language, but with an already-popular language with an existing rich ecosystem of libraries and tooling.

Scala-Native will be much more difficult to make successful than Scala.js was. In particular, Scala.js could focus on the compiler and then delegate to an existing Javascript runtime, while Scala-Native has to build its own runtime from first principles: threading, garbage-collection, event-loops, and so on. This is addition to the formidable amount of compiler work necessary. Nevertheless, if successful, this would be a massive boon for broadening the use cases for which the Scala language can be used.

Conclusion

While amount of hype around the Scala language has definitely died down over the years, the usage seems to be growing at a steady clip, and the experience of using the language has been improving rapidly. This is exactly what you would expect as a language progresses through the hype cycle, and moves from a fad-driven community to a value-driven community. Scala developers are using the language, not as an end unto itself, but instead as a solid tool they can use to accomplish their non-Scala-related technical or business objectives.

In effect, this is what it means for the community to mature.

Going forward, I hope we can see more of this. More focus on outcomes and value, rather than hype. More weighing of tradeoffs, rather than dogmatism. More celebrating the shared core, rather than arguing over the differences. And more 2x compiler speedups!

I think the way we 10x the Scala community is not by spending 10x the effort bikeshedding over the existing language syntax, 10x the effort arguing about functional vs object oriented programming, or 10x the effort trying to make our already-very-type-safe code even more type-safe. It is instead by inclusively expanding the language to reach those developers we could never reach before: the newbies, the non-programmers, the command-line-tool folks, the iOS developers, and so on. If we look at the 2019 Jetbrains Developer Survey for Python , we can see the breadth of different domains in which the Python language is used:

Currently Scala is overwhelmingly used for backend services and/or Spark data pipelines, but there is no reason the above use cases couldn't be satisfied by Scala as well. It's up to the community to make using Scala easy and broadly-usable enough that anyone, not just the grizzled backend/data engineers, are able to pick up Scala and and use it while enjoying the conciseness, performance, and type-safety that those within the Scala community already know and love.

About the Author: Haoyi is a software engineer, an early contributor to Scala.js , and the author of many open-source Scala tools such as theAmmonite REPL and FastParse .

If you've enjoyed this blog, or enjoyed using Haoyi's other open source libraries, please chip in (or get your Company to chip in!) via Patreon so he can continue his open-source work