Superlifter: A DataLoader for Clojure

From the DataLoader README :

DataLoader allows you to decouple unrelated parts of your application without sacrificing the performance of batch data-loading. While the loader presents an API that loads individual values, all concurrent requests will be coalesced and presented to your batch loading function. This allows your application to safely distribute data fetching requirements throughout your application and maintain minimal outgoing data requests.

Start a superlifter per query

A superlifter should live for the lifetime of a query, ensuring consistent results for duplicate reads and capturing and executing all fetches. In lacinia the queries are processed using interceptor chains, so superlifter provides an interceptor that starts a superlifter with your options:

(require '[superlifter.api :as s])
(require '[superlifter.lacinia :refer [inject-superlifter with-superlifter]])
(require '[com.walmartlabs.lacinia.pedestal :as lacinia])

(def lacinia-opts {:graphiql true})

(def superlifter-args
  {:buckets {:default {:triggers {:interval {:interval 10}}}}
   :urania-opts {:env {:db {"abc-123" {:name "Lyra"
                                       :age 11}
                            "def-234" {:name "Pantalaimon"
                                       :age 11}
                            "ghi-345" {:name "Iorek"
                                       :age 41}}}}})

(def service (lacinia/service-map
              (compile-schema)
              (assoc lacinia-opts
                     :interceptors (into [(inject-superlifter superlifter-args)]
                                         (lacinia/default-interceptors (compile-schema) lacinia-opts)))))

Note also that we are injecting data so you can see the data we are working with, but normally this would be a database connection or http library. Our default bucket performs a fetch every 10ms of everything that’s in it at the time. You can add buckets later at runtime for specific purposes - I’ll cover this later on.

Describe your fetches

We now need a way to describe our fetches before we perform them, so that we have a chance of introspecting them in order to deduplicate and batch before executing. The urania library is used for this, so you can implement those protocols if you want the full power of urania, but superlifter provides a abstraction over them with def-fetcher and def-superfetcher that gives a simpler API.

;; def-fetcher - a convenience macro like defrecord for things which cannot be combined
(s/def-fetcher FetchPets []
  (fn [_this env]
    (map (fn [id] {:id id}) (keys (:db env)))))

;; def-superfetcher - a convenience macro like defrecord for combinable things
(s/def-superfetcher FetchPet [id]
  (fn [many env]
    (log/info "Combining request for" (count many) "pets")
    (map (:db env) (map :id many))))

See how the def-superfetcher implementation has the opportunity to combine multiple FetchPet requests into one. This is where we take advantage of any bulk APIs to speed things up!

Chuck 'em in the (time) bucket

We can now write our resolvers to put our fetches into the superlifter bucket:

(defn- resolve-pets [context _args _parent]
  (with-superlifter context
    (s/enqueue! (->FetchPets))))

(defn- resolve-pet-details [context _args {:keys [id]}]
  (with-superlifter context
    (s/enqueue! :pet-details (->FetchPet id))))

The resolve-pets resolver puts a fetch into the bucket which will be performed within the next 10ms. It returns 3 pet ids, after which the resolve-pet-details resolver will run 3 times. Instead of performing 3 separate fetches, those three fetches go into the bucket and are combined into one fetch which will be performed within the next 10ms.

A time bucket of 10ms means you will add up to 10ms to each layer of the query. 10ms might be relatively small but will add up quickly if you query is many layers deep, but making it smaller may mean your fetches span multiple buckets and you lose some efficiency. One of the worst-case scenarios is shown below, with yellow indicating wasted time.

Size buckets

In some cases (like when fetching our pet details) we already know how many fetches we are going to make because our parent has given us a list. You can create a new bucket in superlifter for this purpose with a particular size, rather than time window. The fetch is triggered when the number of items in the bucket meets the threshold, saving the cost of waiting for an interval to elapse:

(def superlifter-args
  {:buckets {:default {:triggers {:queue-size {:threshold 1}}}}})

(defn- resolve-pets [context _args _parent]
  (with-superlifter context
    (-> (s/enqueue! (->FetchPets))
        (s/add-bucket! :pet-details
                       (fn [pet-ids]
                         {:triggers {:queue-size {:threshold (count pet-ids)}
                                     :interval {:interval 50}}})))))

(defn- resolve-pet-details [context _args {:keys [id]}]
  (with-superlifter context
    (s/enqueue! :pet-details (->FetchPet id))))

The default bucket has a queue size of 1. The resolve-pets resolver puts its fetch in the default bucket, which will be performed immediately, and then adds a new bucket called :pet-details which has a size of 3, because we know we will fetch details for the 3 pets we have found.

The resolve-pet-details resolver then puts all its fetches into the :pet-details bucket and as soon as the third resolver has run the combined fetch will be performed immediately. No time lost and maximum efficiency!