Metosin

Muuntaja, a boring library everyone should use

The Boring Company makes flamethrowers, we make boring libraries! Here's one. It's almost two years old, does nothing new but what it does, it tries to do well. The library is Muuntaja, a message formatting and negotiation library for Clojure.

The thing

We write Clojure apps that require reading and writing data to external formats. For this, we use different formatters like Cheshire, transit-clj and clojure.edn. For http & websockets, we use components like middleware and interceptors to wrap the formatters, both on the server and on the client side. Formatters and components are all bit different: some use explicit option maps, some are extended via protocols or multimethods. Some support only Streams, some only Strings, some mostly anything. It's a mess.

The rescue

Muuntaja is an easy-to-extend Clojure library providing a simple api for encoding and decoding data. It ships with a set of pre-configured formatters.

[metosin/muuntaja "0.6.0-alpha1"]

Creating a Muuntaja instance, with defaults:

(require '[muuntaja.core :as m])

;; defaults
(def m (m/create))

(m/encodes m)
; #{"application/json"
;   "application/transit+msgpack"
;   "application/transit+json"
;   "application/edn"}

(m/decodes m)
; #{"application/json"
;   "application/transit+msgpack"
;   "application/transit+json"
;   "application/edn"}

We can now encode (and decode) data:

(->> {:olipa "kerran"}
     (m/encode m "application/json"))
; #object[java.io.ByteArrayInputStream]

By default, We get an InputStream out. byte-arrays and lazy StreamableResponse are also supported. All encoded values can be slurp'd to get the string representation:

(->> {:olipa "kerran"}
     (m/encode m "application/json")
     (slurp))
; "{\"olipa\":\"kerran\"}"

It works both ways:

(->> {:olipa "kerran"}
     (m/encode m "application/json")
     (m/decode m "application/json"))
; {:olipa "kerran"}

Configuration

New formats can be added via options:

;; [metosin/muuntaja-yaml "0.6.0-alpha1"]
(require '[muuntaja.format.yaml :as yaml])

(def m
  (m/create
    (-> m/default-options
       (m/install yaml/format))))

(m/encodes m)
; #{"application/json"
;   "application/x-yaml"
;   "application/transit+msgpack"
;   "application/transit+json"
;   "application/edn"}

(->> {:olipa "kerran"}
     (m/encode m "application/x-yaml")
     (slurp))
; "{olipa: kerran}\n"

Example with more configuration:

(def m
  (m/create
    (-> m/default-options
        ;; set Transit readers & writers
        (update-in [:formats "application/transit+json"]
                   merge {:decoder-opts {:handlers transit/readers}
                          :encoder-opts {:handlers transit/writers}})
        ;; return byte-array by default to support NIO
        (assoc :return :bytes)
        ;; support for YAML
        (m/install yaml/format))))

All state of the default formats is captured within the Muuntaja instance making it effectively immutable. It can be safely used within the application.

See all configuration options.

HTTP

For Ring, there is a set of middleware for content-negotiation and request & response formatting.

Simplest thing that works:

(require '[muuntaja.middleware :as middleware])

(defn echo [request]
  {:status 200
   :body (:body-params request)})

(def m (m/create))

(def app (middleware/wrap-format echo m))

(defn request [body]
  {:headers
   {"content-type" "application/edn"
    "accept" "application/json"}
   :body (m/encode m "application/edn" body)})

(->> {:kikka 42} (request) (app))
; {:status 200
;  :body #object[java.io.ByteArrayInputStream]
;  :muuntaja/format "application/json"
;  :headers {"Content-Type" "application/json; charset=utf-8"}}

We can user Muuntaja also in the client side (only Clojure for now):

(->> {:kikka 42}
     (request)
     (app)
     :body
     (m/decode m "application/json"))
; {:kikka 42}

Pedestal-style interceptors are also supported, via muuntaja.interceptor namespace.

Performance

We tried to make Muuntaja as fast as possible: bounded cache for content-negotiation results, no extra copying of data, support for byte-arrays to enable NIO, protocol-based dispatch etc. Here's a difference between Compojure-api JSON echo before and after Muuntaja, perceived by the Ring adapter:

The Bad

We have pushed all the configuration into one place, but there is no documentation of the format options. Some formats have :keywords?, some :keywordize. So, it's still a mess, but a contained one.

Could we do something for this? Sure. We could use clojure.spec to describe to options and use tools like spell-spec and Expound to help with error messages. PRs related to this are most welcome!

The End

[metosin/muuntaja "0.6.0-alpha1"] has been just released. It's a big release, as it replaces Cheshire with Jsonista, changes the extension api for new formats (with a fail-fast assertion for the old syntax) and splits things into multiple modules. Latest Compojure-api 2.0.0-alpha21 uses this version. Libraries like Duct and Luminus are currently on latest stabile 0.5.0 version. Looking forward to getting the 0.6.0 out.

Comments welcome.

Tommi