High-Performance Schemas in Clojure/Script with Malli 1/2
IntroductionLink to Introduction
It's over a year since the initial launch of Malli, the data-driven data specification library for Clojure/Script. Since then, it has become a de-facto tool in our projects and we are happy to see it getting adopted in the Clojure community. While writing this, Malli has 110 forks, over 60 contributors, several high-quality community-built add-ons, and over 750k downloads on Clojars.
This post walks through the performance engineering done in the last few revisions.
PerformanceLink to Performance
Inspired by the performance of Reitit, we wanted to make Malli fast too. We have benchmarked Malli against both idiomatic hand-written Clojure code and some well-known libraries like clojure.spec and Plumatic Schema.
Initially, only validation and value transformation were optimized for performance, but over time, more performance targets have been added, currently including the following:
- Validation
- Explaining
- Value Transformation
- Parsing and Unparsing
- Inferring Schemas
- Schema Creation
- Schema Worker Creation
- Transforming Schemas
Besides just counting the CPU cycles, the following are also important:
- Library load time (on CLJ)
- Bundle-size (on CLJS)
- Memory Usage
Let's start our journey with a sample domain model.
The Mushroom ForestLink to The Mushroom Forest
This was a great year to find delicious mushrooms in the (Finnish) forests. Here's some:
(def valid1
{:name "Kanttarelli"
:eatability 3
:data {:taste [:peppery :fruity :apricot]
:latin "Cantharellus cibarius"}})
(def valid2
{:name "Suppilovahvero"
:eatability 3
:data {:taste [:sweet :sour]}})
(def valid3
{:name "Punakärpässieni"
:eatability -2
:data nil})
Punakärpässieni is actually poisonous, so don't eat it.
Besides sample data, we need a domain model. We can use the Schema Inferring to pull the schema out of the samples:
(require '[malli.provider :as mp])
(def schema (mp/provide [valid1 valid2 valid3]))
schema
;[:map
; [:name string?]
; [:eatability int?]
; [:data [:maybe [:map
; [:taste [:vector keyword?]]
; [:latin {:optional true} string?]]]]]
The returned schema looks legit, :data can be nil and :latin key is optional. Let's also create an invalid mushroom to ensure our test domain works correctly:
(def invalid1
{:name "Känsätuhkelo"
:eatability 1
:data {:taste ["mild"]}})
Checking the values:
(require '[malli.core :as m])
(m/validate schema valid1) ; => true
(m/validate schema valid2) ; => true
(m/validate schema valid3) ; => true
(m/validate schema invalid) ; => false
All set, let's dive into the performance.
ValidationLink to Validation
We are using Criterium to run the tests. There is also jmh-clj, but Criterium is much easier to use and gives good-enough results. The test harness:
(require '[criterium.core :as cc])
(defn bench-validate! [valid?]
(assert (every? valid? [valid1 valid2 valid3]))
(assert (not (valid? invalid)))
(cc/quick-bench (valid? valid1)))
Starting with idiomatic Clojure:
;; 440ns
(bench-validate!
(fn [{:keys [name eatability data] :as mushroom}]
(and (map? mushroom)
(int? eatability)
(string? name)
(or (nil? data)
(let [{:keys [taste latin]} data]
(and (map? data)
(vector? taste)
(every? keyword? taste)
(if latin (string? latin) true)))))))
That's pretty fast and compact. Same with Malli:
;; 160ns
(bench-validate! (m/validator schema))
Thanks to the optimized validation engine of Malli, it's over 2x faster than the idiomatic hand-written Clojure. Also, as the validation is derived from the schema, it's always correct and up-to-date.
Validation using clojure.spec:
(require '[clojure.spec.alpha :as s])
(s/def ::name string?)
(s/def ::eatability int?)
(s/def ::taste (s/coll-of keyword? :kind vector?))
(s/def ::latin string?)
(s/def ::data (s/nilable (s/keys :req-un [::taste], :opt-un [::lating])))
(s/def ::mushroom (s/keys :req-un [::name ::eatability ::data]))
;; 950ns
(bench-validate!
(fn [x]
(let [res (s/conform ::mushroom x)]
(not (s/invalid? res)))))
And with Plumatic Schema:
(require '[schema.core :as schema])
(def Mushroom
{:name schema/Str
:eatability schema/Int
:data (schema/maybe {:taste [schema/Keyword]
(schema/optional-key :latin) schema/Str})})
;; 1500ns
(bench-validate!
(let [check (schema/checker Mushroom)]
(fn [x] (nil? (check x)))))
Malli is the fastest here. You should not take the micro-benchmarks too seriously and always conduct your tests with your data models to see the differences. If the performance differences are in order(s) of magnitude, it starts to matter.
TransformationLink to Transformation
Mushrooms are entering our system over the wire as JSON. To decode the values from JSON to Clojure, we have to apply a set of transformations. Here's an example of a JSON kanttarelli, parsed from string using jsonista:
(require '[jsonista.core :as j])
(def json
(-> valid1
(j/write-value-as-string)
(j/read-value j/keyword-keys-object-mapper)))
json
;{:name "Kanttarelli"
; :eatability 3
; :data {:taste ["peppery" "fruity" "apricot"]
; :latin "Cantharellus cibarius"}}
JSON can't handle the keywords, otherwise, it looks correct. A new benchmark suite:
(defn bench-transform! [decode]
(assert (= valid1 (decode json)))
(cc/quick-bench (decode json)))
With idiomatic Clojure:
;; 520ns
(bench-transform!
(fn [{:keys [data] :as mushroom}]
(cond-> mushroom
data (update :data (fn [{:keys [taste] :as data}]
(cond-> data
taste (update :taste (partial mapv keyword))))))))
Despite we only need to transform one value, the transforming function is far from being simple as it has to handle the potential missing values too. We could have used Specter or Meander to simplify the code, but neither can be considered as idiomatic Clojure.
FlameGraph looks calm:
Same transformation with Malli:
(require '[malli.transform :as mt])
;; 250ns
(bench-transform! (m/decoder schema (mt/json-transformer)))
Again, 2x faster. Why would you want to build and maintain the transformation functions manually if you can derive them from a schema and get better performance?
Clojure.spec doesn't support runtime transformations, but we can use spec-tools for the job:
(require '[spec-tools.core :as st])
;; 55000ns
(bench-transform! #(st/coerce ::mushroom % st/json-transformer))
That's two orders of magnitude slower than hand-written Clojure. As spec is built around a global registry, spec-tools walks the spec at runtime to ensure that correct registry references are used. FlameGraph looks like the Himalayas.
Plumatic Schema can cache the transformation functions is much faster than spec-tools:
(require '[schema.coerce :as sc])
;; 3200ns
(bench-transform! (sc/coercer Mushroom sc/json-coercion-matcher))
But wait, Plumatic defines coercion as a process of both transforming and validating in a single sweep. We should test that with Malli too to get a fair comparison.
(defn coercer [schema transformer]
(let [valid? (m/validator schema)
decode (m/decoder schema transformer)
explain (m/explainer schema)]
(fn [x]
(let [res (decode x)]
(cond-> res (not (valid? res)) (explain))))))
;; 470ns
(bench-transform! (coercer schema (mt/json-transformer)))
Malli needs to do two sweeps over the data to get the same functionality, but having two fast sweeps is still an order of magnitude faster than the one sweep of Plumatic in this test.
Inferring SchemasLink to Inferring Schemas
The original Schema Inferrer in Malli was really slow. Since 0.7.0, it is orders of magnitude faster, thanks to a clear separation of ahead-of-time and actual runtime computations. And it's still just 71 lines of code.
(defn bench-provide! [provide]
(let [samples [valid1 valid2 valid3]]
(cc/quick-bench (provide samples))))
;; 79000µs (0.5.1)
(bench-provide! mp/provide)
;; 280µs (0.7.0) -> 280x
(bench-provide! (mp/provider))
The performance is now in the same ballpark as with spec-provider:
(require '[spec-provider.provider :as sp])
;; 440µs
(bench-provide! #(sp/infer-specs % ::mushroom))
There is still much more we can do to make the Malli providers faster (and better).
Creating Schemas and WorkersLink to Creating Schemas and Workers
One of the main reasons Malli is so fast is that we have pushed computations from runtime (e.g., validation) into the schema and worker creation phase. For the JVM, it's ok to pay a one-time cost to enable a fast runtime. Still, for single-threaded environments like JavaScript, the initial cost of building everything does matter.
In 0.7.0, we measured and optimized the creation phase making most of the operations 1-2 orders of magnitude faster. Here are the relevant parts of that work:
- new
EntryParserprotocol for parsing with primitive arrays and supporting laziness - patching
EntrySchemavalues without re-parsing - new
SchemaASTprotocol for parse-free schema creation - new
Cachedprotocol for memoizing validators, parsers, explainers, and generators - removed all calls to
satisfy?, go vote up CLJ-1814 - support for pass-through walking of schemas
Schema creation:
;; 57µs -> 5µs (11x)
(m/schema schema)
;[:map
; [:name string?]
; [:eatability int?]
; [:data [:maybe [:map
; [:taste [:vector keyword?]]
; [:latin {:optional true} string?]]]]]
Schema creation, using the new Schema AST:
(def ast (m/ast schema))
ast
;{:type :map,
; :keys {:name {:order 0
; :value {:type string?}},
; :eatability {:order 1
; :value {:type int?}},
; :data {:order 2,
; :value {:type :maybe,
; :child {:type :map,
; :keys {:taste {:order 0
; :value {:type :vector
; :child {:type keyword?}}},
; :latin {:order 1
; :value {:type string?}
; :properties {:optional true}}}}}}}}
;; 160ns (300x, lazy)
(m/from-ast ast)
Schema Walking, using the identity walker (no-op) as an example:
(def Mushroom (m/schema schema))
;; 28µs -> 2µs (14x)
(m/walk Mushroom (m/schema-walker identity))
;[:map
; [:name string?]
; [:eatability int?]
; [:data [:maybe [:map
; [:taste [:vector keyword?]]
; [:latin {:optional true} string?]]]]]
Closing a nested schema:
(require '[malli.util :as mu])
;; 55µs -> 5.8µs (9x)
(mu/closed-schema Mushroom)
;[:map {:closed true}
; [:name string?]
; [:eatability int?]
; [:data [:maybe [:map {:closed true}
; [:taste [:vector keyword?]]
; [:latin {:optional true} string?]]]]]
Removing key from a map-schema:
;; 22µs -> 1.2µs (18x)
(mu/dissoc Mushroom :data)
;[:map
; [:name string?]
; [:eatability int?]]
Transparent caching of schema workers:
;; 60µs -> 190ns (320x)
(m/validate Mushroom valid1)
; => true
Schema creation performance starts to be ok, but there's still room for improvement. We could push all the schema transformation work from initialization time into development time and "freeze" the final schemas as Schema AST, which is the fastest way to load the schemas. This could also reduce the js-bundle size as the transformation utilities could be DCE'd from production builds.
When Performance MattersLink to When Performance Matters
Performance is usually not the first concern when selecting a data specification library for Clojure, but when it matters, Malli is there for you. Malli is currently as fast or faster than hand-written idiomatic Clojure on validation and value transformation while still being fully declarative. The new 0.7.0 version is a significant internal rewrite for making schema creation, inferring, and schema transformation order(s) of magnitude faster.
Big thanks to Clojurists Together for funding the Malli development. Also, special thanks to all contributors, especially to Ben Sless for some crazy performance improvements.
For discussion and getting help, there is #malli in Clojurians Slack.
