What's New in Lua 5.4

July 15, 2020

This article was contributed by Ben Hoyt

Lua version 5.4 was released at the end of June; it is the fifteenth major version of the lightweight scripting language since its creation in 1993. New in 5.4 is a generational mode for the garbage collector, which performs better for programs with lots of short-lived allocations. The language now supports "attributes" on local variables, allowing developers to mark variables as constant ( const ) or resources as closeable ( close ). There were also significant performance improvements over 5.3 along with a host of minor changes.

Lua is a programming language optimized for embedding inside other applications, with notable users such as Redis and Adobe Lightroom. It has been used as a scripting language for many computer games , including big names such as World of Warcraft and Angry Birds; Lua was the most-used scripting language in a 2009 survey of the game industry . Part of the reason Lua is good for embedding is because it is small: in these days of multi-megabyte downloads for even the simplest applications, the entire Lua 5.4 distribution (source plus docs) is a 349KB archive. To build a Lua interpreter with the default configuration, a developer can type make and wait about five seconds for compilation — the result is a self-contained 200-300KB binary.

Major versions of Lua are released every few years, not on any particular release cycle. The previous major version , 5.3, was released over five years ago, in January 2015, with the addition of a separate integer type (previously Lua used only floating-point numbers), bitwise operators, a basic UTF-8 library, and many minor features.

Language changes

One of the interesting new features in Lua 5.4 is the addition of local variable "attributes" . When declaring a local (block-scoped) variable, a developer can add <const> or <close> after a variable name to give it that attribute. The const attribute is straightforward: similar to const in C, it means that the specified variable cannot be reassigned after the initialization in its declaration. The const attribute does not make a data structure immutable: a developer is not prevented from changing entries in a table stored in a const variable, but the variable name cannot be assigned again. The const attribute provides a small amount of compile-time safety, as the compiler will give an error if a constant is accidentally reassigned:

do
        local x <const> = 42
        x = x+1
    end
    -- ERROR: attempt to assign to const variable 'x'

Perhaps more useful (though with similarly unusual syntax) is the close attribute. This tells Lua to call the object's __close() " metamethod " when the variable goes out of scope. Similar to RAII in C++ or the with statement in Python, it is a way to ensure that memory is freed, files are closed, or other resources are shut down in a deterministic way. For example, the file object returned by the built-in io.open() function can be used with <close> :

do
        local f <close> = io.open("/etc/fstab", "r")
        -- read from file 'f'
    end
    -- file is automatically closed here

The <close> attribute can also be used with user-defined objects:

function new_thing()
        local thing = {}
        setmetatable(thing, {
            __close = function()
                          print("thing closed")
                      end
        })
        return thing
    end

    do
        local x <close> = new_thing()
        print("use thing")
    end
    -- "thing closed" is printed here after "use thing"

Previously, developers would have to use the __gc() metamethod for this purpose, but that is only called when the object is garbage collected some time later, not deterministically at the end of the block.

Generational GC

Version 5.4 also brings a new generational garbage collection (GC) mode, which performs better for certain kinds of programs where objects usually have a short lifetime. A generational GC — based on the observation that "most objects die young" — scans "young" objects frequently and frees them if they are not referenced, but scans older objects (those that are still referenced after one or more GC passes) less frequently. Interestingly, Roberto Ierusalimschy (one of the creators of Lua) noted in 2017 that Lua previously had a generational GC:

Lua has an incremental garbage collector since 5.1. It was the generational collector that was introduced in 5.2 as an experiment and then removed in 5.3. It will come again in 5.4, this time probably to stay.

Ierusalimschy gave a talk in 2019 ( PDF slides and YouTube video ) that goes into more detail about how incremental GC works, as well as why the 5.2 generational GC didn't perform that well, and what the Lua team replaced it with. In 5.2's version, objects only had to survive a single GC cycle (collector pass) before becoming "old", but in 5.4 they have to survive two GC cycles, which is a more accurate model for real-world Lua programs. The two-cycle approach is more complicated to implement but gives better GC performance for many programs — though not all. Ierusalimschy notes that programs which build large data structures won't benefit. Possibly for that reason, the Lua team didn't change the default: in 5.4 the default is still to use the incremental collector; a developer needs to add " collectgarbage("generational") " to their program in order to turn on the generational GC.

On the lua-l mailing list, Gé Weijers described how the generational GC, with its "minor collections" (frequent GC passes to collect young objects) ties into the new <close> feature (which used to be called " toclose "):

The garbage collector in 5.4 implements a generational mode. If an object survives the minor collections it may take a very, very long time before its __gc metamethod gets called after is becomes inaccessible, especially if your program mostly creates short lived objects. This makes __gc less useful as a poor man's RAII replacement.

The new "toclose" feature is much more useful to release resources and unlock locks in a timely matter.

Faster

One of the unsung features in 5.4, is a significantly faster interpreter, though the release notes have overlooked it. In a test I did on my 64-bit macOS machine using Gabriel de Quadros Ligneul's Lua Benchmarks suite, I found that version 5.4 was an average of 40% faster than version 5.3 across 11 benchmarks included in the suite:

Similar gains are shown in Elmar Klausmeier's performance comparison . Admittedly, both of these are rather artificial benchmarks — when using Lua in something like a game engine, performance-sensitive code like graphics or matrix multiplication will no doubt be written in C. Still, an improvement of this magnitude for number-centric code (which most of these benchmarks are) is not to be scoffed at. Dibyendu Majumdar described some of the reasons for these improvements on the lua-l mailing list back in 2018: 5.4 added new and optimized bytecode instructions for numeric operations that Lua's parser can use when it can infer that the types involved are numbers. For example, GETI and SETI are two new instructions that are used for table lookups when the index is a constant integer.

Those who need much higher performance can use Mike Pall's LuaJIT , a just-in-time compiler for Lua 5.1 that is significantly faster than the stock Lua interpreter. However, LuaJIT hasn't added any of Lua's new features since version 5.1 (which was released in 2006). Doing so would be quite an undertaking due to many breaking changes, including new scoping rules in 5.2 and the new integer type in 5.3. For this reason, Pall has been a vocal critic of the backward-incompatible changes that the Lua team makes.

This does seem to be a real problem, and not only with obscure edge cases: two of the benchmarks in the Lua Benchmarks suite failed in 5.4 with a "C stack overflow" error (though they work fine in 5.3), so I had to remove them before running it. The ack and fixpoint-fact benchmarks fail, presumably due to different handling of recursive tail calls in 5.4. Most of the incompatibilities in 5.4 are documented , but the length of that list may still cause a fair bit of pain for those trying to upgrade large Lua scripts. My guess is that this is why tools that need long-term stability, like Redis and World of Warcraft, lock in a specific older version of Lua (in the case of both of those, version 5.1). It seems like there's something of a split in the community, with some who stick to 5.1 because it has a JIT compiler and because the changes since then are relatively minor.

Incompatibilities between Lua versions may also contribute to the problem of Lua not having a unified standard library, which LWNwrote about back in February. If a library author has to do a bunch of work to upgrade when a new Lua version comes out, they may be less likely to keep it up to date. That makes it more likely that someone will create a fork that works on the new Lua version or simply write a new library.

Smaller changes

In addition to the larger changes, Lua 5.4 adds many smaller features, including a new random number generator using the xoshiro256** algorithm instead of using the underlying C library's rand() function. There is now a simple warning system used when there's an error in a finalizer or __close() method. Also added is the ability for Lua values with " userdata " to have multiple user values (userdata is a pointer to a memory block created with the Lua C API, so this feature allows objects created by C extensions to have multiple memory blocks associated with them).

There were some minor changes in semantics as well: slightly different handling of edge cases with wrap-around in for loops and adjustment of string-to-number coercion for integers (for example, "10"+1 is the integer 11 in 5.4, but the floating-point number 11.0 in 5.3).

Overall, Lua seems like a good language for its domain (embedding into larger systems or applications); the release of 5.4 shows that it is receiving continual improvements from the core team. Lua has no clear roadmap , so it's hard to know at this early stage what changes are being planned for 5.5, or when it is likely to be released (Lua developer Pierre Chapuis even speculates the next version may be " a very impacting change " with a 6.0 version number). In any event, the new features in 5.4 will probably be fairly minor for most users, but the performance improvements will prove to be a nice win.

to post comments)