## Prologue

[Malli](https://github.com/metosin/malli) is a high-performance, data-driven data specification library for Clojure. It is widely used in the Clojure Community having millions of downloads and an active [channel](https://clojurians.slack.com/archives/CLDK6MFMK) on Slack. This is the fifth post on Malli, focusing on new features of the `0.14.0` version. Older posts include:

* [Malli, Data-Driven Schemas for Clojure/Script](https://www.metosin.fi/blog/malli)
* [Structure and Interpretation of Malli Regex Schemas](https://www.metosin.fi/blog/malli-regex-schemas)
* [High-Performance Schemas in Clojure/Script with Malli 1/2](https://www.metosin.fi/blog/high-performance-schemas-in-clojurescript-with-malli-1-2)
* [Transforming data with Malli and Meander](https://www.metosin.fi/blog/transforming-data-with-malli-and-meander)

## Malli Schemas

If you are new to Malli or to Clojure, here is a sample code to define Schemas and validate values against them:

```clojure
(require '[malli.core :as m])

(def UserId :string)

(def Address 
  [:map
   [:street :string]
   [:latlon [:tuple :double :double]]])

(def User
  [:map
   [:id UserId]
   [:address Address]])

(m/validate
 User
 {:id 123
  :address {:street "Hämeenkatu 13"
            :latlon [61.4980155, 23.7640067]})
; => true
```

Malli supports also human and machine-readable error messages, value transformation, value generation, inferring schemas from values, schema serialization, function schemas, static type linting and much more. See [README](https://github.com/metosin/malli?tab=readme-ov-file#malli) for all features.

## New Features in 0.14.0

The new version is released today, containing small improvements and fixes, but also two important new features:

1. New Development Mode
2. Support for Var Schema References

## New Development Mode

This is big. Malli has had pretty errors for a long time, but now pretty errors cover all development time errors - printing descriptive errors and hints on how to fix them. This is the way all libraries should be built. 

Coercion in normal mode:

```clojure
(m/coerce [:enum "S" "M" "L"] "XL")
; Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
; :malli.core/coercion
```

Same in the new development mode:

```clojure
;; start the development mode
((requiring-resolve 'malli.dev/start!))
; malli: dev-mode started

(m/coerce [:enum "S" "M" "L"] "XL")

-- Schema Error ----------------------------------------------------- user:16 --

Value:

  "XL"

Errors:

  ["should be either S, M or L"]

Schema:

  [:enum "S" "M" "L"]

More information:

  https://cljdoc.org/d/metosin/malli/CURRENT

--------------------------------------------------------------------------------
; Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
; :malli.core/coercion
```

Schema creation error:

```clojure
(m/schema [:map [:name :string?]])

-- Schema Creation Error -------------------------------------------- user:18 --

Invalid Schema

  :string?

Did you mean

  string?
  :string

More information:

  https://cljdoc.org/d/metosin/malli/CURRENT

--------------------------------------------------------------------------------
; Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
; :malli.core/invalid-schema

```

Invalid reference type:

```clojure
(m/schema [:ref 'id])

-- Schema Error ----------------------------------------------------- user:20 --

Invalid Reference

  [:ref id]

Reason

  Reference should be one of the following:

  - a qualified keyword, [:ref :user/id]
  - a qualified symbol,  [:ref 'user/id]
  - a string,            [:ref "user/id"]
  - a Var,               [:ref #'user/id]

More information:

  https://cljdoc.org/d/metosin/malli/CURRENT

--------------------------------------------------------------------------------
; Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
; :malli.core/invalid-ref
```

There is a clean separation between production and development modes:

* Production
    * Exceptions are thrown with spesific `:type` and context information as `ex-data`
    * No string formatting, nothing extra, just throw, fail fast and early
    * Helps to keep bundle sizes on ClojureScript smaller (Malli starts from 2kb gzipped)
* Development
    * Each exception `:type` (or class) can register its own error report using `malli.dev.virhe/-format` multimethod
    * Uses [fipp](https://github.com/brandonbloom/fipp) & [virhe](https://github.com/metosin/virhe) under the hood and has a custom EDN printer for Clojure/Script
    * All of malli is available - humanized error messages, generated sample data etc.
    * Pretty error reports are printed **before** exceptions are thrown so the client application will get exactly the same exception as in production mode

## Support for Var Schema References

Malli supports multiple ways for defining and reusing schemas:

1. Schemas as Vars and Values - *the [plumatic](https://github.com/plumatic/schema) way*
2. Schemas via a Global Registry - *the [clojure.spec](https://clojure.org/about/spec) way*
3. Schemas via Local Registries - *the data way*

The `User` example above uses the first one - Schemas as defined using `def` and they are embedded as values:

```clojure
(m/form User)
; [:map
;  [:id :int]
;  [:address [:map
;             [:street :string]
;             [:latlon [:tuple :double :double]]]]]
```

With `0.14.0`, Vars are now first-class schema references. This enables both recursive schemas and allows parent schema to control whether to inline schemas or not.


```clojure
(def User2
  [:map
   [:id UserId] ;; embeded
   [:address #'Address] ;; reference
   [:friends [:set [:ref #'User2]]]]) ;; recursive reference

(m/form User2)
; [:map
;  [:id :string]
;  [:address #'Address]
;  [:friends [:set [:ref #'user/User]]]]
```

Testing the recursion:

```clojure
(require '[malli.generator :as mg])

(mg/generate User2)
; {:id "6cJ7zpi8VX",
;  :address {:street "18Eqw3x8w151K633H73D"
;            :latlon [-0.759521484375 3.693115234375]},
;  :friends #{{:id "1"
;              :address {:street "e"
;                        :latlon [-1.25 0.5]}
;              :friends #{}}}}
```

There is also a new helper to embed the non-recursive refecences into the parent, like what  [Prettify TypeScript](https://twitter.com/mattpocockuk/status/1740652264006512724) does for TS.

```clojure
(m/deref-recursive User2)
; [:map
;  [:id :string]
;  [:address [:map
;             [:street :string]
;             [:latlon [:tuple :double :double]]]]
;  [:friends [:set [:ref #'user/User2]]]] ;; recursive, can't inline!
```

Var references can be serialized, but the deserializion is disabled by default.

```clojure
(require '[malli.edn :as edn])

(-> User2 
    (edn/write-string)
    (edn/read-string)) ;; fails!
```

8-bit ascii colored message seen through [Cursive](https://cursive-ide.com/) IDE:

![Malli Var Deserialization Error](posts/malli-var-deserialization-error.png)

## Schema Registries

If using Vars is not your thing, you can still use global or local schema registry instead. Below is the same example using a local (serializable) schema registry.

```clojure
(def User3
  [:schema
   {:registry {"id" :string
               "address" [:map
                          [:street :string]
                          [:latlon [:tuple :double :double]]]
               "user" [:map
                       [:id "id"]
                       [:address "address"]
                       [:friends [:set [:ref "user"]]]]}}
   "user"])

(m/deref-recursive User3)
;[:map
; [:id :string]
; [:address [:map 
;            [:street :string] 
;            [:latlon [:tuple :double :double]]]]
; [:friends [:set [:ref "user"]]]]
```

Round-robin to [EDN](https://github.com/edn-format/edn) string and back:

```clojure
(->> User3 
     (edn/write-string)  ;; serialize 
     (edn/read-string)   ;; deserialize
     (mg/generate)       ;; example value
     (m/validate User2)) ;; validate
; => true
```

## Going Forward

[Malli](https://github.com/metosin/malli) is an inspiring project to develop. We (and many others) use it as the go-to tool with Clojure: it works, it's good, it's under active development, and it's fun to develop and easy to extend. Thanks to [Clojurists Together](https://www.clojuriststogether.org/) long term funding for 2024, I'm planning to explore how far can we go with a dynamically typed lisp and data-driven schemas. Join the [discussion](https://clojurians.slack.com/archives/CLDK6MFMK) to get involved.

Changelog of the new `0.14.0` version is found [here](https://github.com/metosin/malli/blob/master/CHANGELOG.md#0140-2024-01-16).

Malli, Data Modelling for Clojure Developers

OpenAPI ([docs][openapi-docs], [official spec][openapi-spec])
is a standard format for documenting HTTP APIs. It is
the successor of [Swagger][swagger], also known as OpenAPI 2.

With OpenAPI documentation, you can provide nice interactive docs for
the users of your API ([here's an example using Swagger UI][example]).
They can also automatically generate client code for your API using a
tool like [openapi-generator][openapi-generator] or [the Clojure
library Martian][martian]

[issues]: https://github.com/metosin/reitit/issues
[slack]: https://clojurians.slack.com/archives/C7YF1SBT3
[openapi-spec]: https://learn.openapis.org/specification/
[openapi-docs]: https://spec.openapis.org/oas/v3.1.0
[swagger]: https://swagger.io/
[openapi-generator]: https://openapi-generator.tech/
[martian]: https://github.com/oliyh/martian
[example]: https://petstore3.swagger.io/

## Cool Features in OpenAPI 3

Why should you care about OpenAPI 3 when Swagger works just fine? Some
of the additions that OpenAPI 3 brings might be crucial for
documenting your API properly.

Most significantly, OpenAPI 3 brings full
[JSON Schema](https://json-schema.org/) support. This means that you can
express richer schemas, for example via the operators `oneOf`, `anyOf`
and `allOf`. These operators were present as partially-supported
extensions in Swagger, but in JSON Schema and OpenAPI 3, they have
full support.

Relatedly, you can use _discriminators_ to pick which schema applies
to a value based on a field. This is crucial for big union types like
events. Here's a nice
[blog post on these new features](https://redocly.com/docs/resources/discriminator/).
Note that as of this release, Reitit does not know how to
generate discriminators automatically for you.

OpenAPI 3 also lets you specify different schemas for different
request/response content types. Also, you can specify multiple named
examples for parameters. You can find examples of both of these in the
[Reitit example discussed below](https://github.com/metosin/reitit/tree/master/examples/http-swagger).

On the authentication side, OpenAPI 3 has added support for cookie
parameters & authentication, and better OAuth / OpenID Connect
support. See [the spec](https://spec.openapis.org/oas/v3.1.0#security-scheme-object)
for more info.

## Example

Generating OpenAPI 3 documentation from Reitit is straightforward.
Just use `create-openapi-handler` to define a route. The OpenAPI docs
will be generated based on the route data like `:summary`,
`:parameters`, etc.

```clj
[["/openapi.json"
  {:get {:no-doc true
         :openapi {:info {:title "my-api"
                          :description "reitit + openapi3"
                          :version "0.0.1"}
                   :tags [{:name "api"
                           :description "the api"}]}
         :handler (openapi/create-openapi-handler)}}]

 ["/api"
  {:tags ["api"]}

  ["/hello"
   {:get {:summary "hello"
          :handler (fn [_]
                     {:status 200
                      :body "hello world"})}}]

  ["/plus"
   {:post {:summary "plus with malli body parameters"
           :description "description here"
           :parameters {:body {:x :int, :y :int}}
           :responses {200 {:body {:total :int}}}
           :handler (fn [request]
                      (let [{:keys [x y]} (-> request :parameters :body)]
                        {:status 200
                         :body {:total (+ x y)}}))}}]]]
```

Here's what
[Swagger UI](https://swagger.io/tools/swagger-ui/)
looks like for this API:

<img src="../../img/posts/reitit-openapi-swagger-ui.png">

The
[ring-malli-swagger](https://github.com/metosin/reitit/tree/master/examples/ring-malli-swagger)
and
[ring-spec-swagger](https://github.com/metosin/reitit/tree/master/examples/ring-spec-swagger)
examples in the Reitit repository has been extended to serve OpenAPI 3
& Swagger specs side-by-side. Check them out!

## Migrating A Reitit App to OpenAPI 3

Just like in the previous example, you can easily serve Swagger and
OpenAPI 3 specs side-by-side for a transition period.

To migrate your Reitit app to OpenAPI 3, do the following:

1. Add a `reitit.openapi/create-openapi-handler` route. Here's a minimal example:
```clj
["/openapi.json"
 {:get {:handler (openapi/create-openapi-handler)
        :openapi {:info {:title "my nice api" :version "0.0.1"}}
        :no-doc true}}]
```
2. See how your OpenAPI 3 spec looks like in Swagger UI, and validate
   it using a tool like
   [openapi-schema-validator](https://www.npmjs.com/package/@seriousme/openapi-schema-validator).
3. If you have additional fields under your swagger handler (e.g.
   `:securityDefinitions`), port them to OpenAPI 3 format, and add
   them under the `:openapi` key in the route you just added.
4. If you have custom `:swagger` data on your other routes, convert
   them to `:openapi`
5. Use `:json-schema/default` and `:json-schema/example` in addition
   to `:swagger/default` and `:swagger/example` in your schema metadata.
   - For Plumatic Schema, the new keys are `:openapi/default` and
     `:openapi/example` instead.

## Thanks

It takes a village to raise a library, as they say. I'd like to thank
everybody who made this Reitit update happen.

First of all, thanks to [Solita](https://www.solita.fi) for sponsoring
the delivery of the OpenAPI 3 feature and thus my part of the work.
Without their nudging and financial support, this release would've
happened a _lot_ later.

Secondly, thanks to current and former Metosin programmers
[@ikitommi](https://github.com/ikitommi),
[@Deraen](https://github.com/Deraen),
[@kalekale](https://github.com/kalekale), and
[@miikka](https://github.com/miikka)
for their work on the Metosin libraries
[Reitit](https://github.com/metosin/reitit),
[Malli](https://github.com/metosin/malli),
[Spec-tools](https://github.com/metosin/spec-tools), and
[Schema-tools](https://github.com/metosin/schema-tools).

The foundation of the work was transforming Malli, Spec and Plumatic
Schema schemas into [JSON Schema](https://json-schema.org/), which
OpenAPI 3 builds on.

The initial JSON Schema support for Spec (via spec-tools) and Malli
was implemented by Metosin.

[@rrudakov](https://github.com/rrudakov) contributed OpenAPI support for spec-tools in
[PR #236](https://github.com/metosin/spec-tools/pull/236)
and schema-tools in
[PR #63](https://github.com/metosin/schema-tools/pull/63).

Bringing all of these tools together in Reitit was a long process,
with lots of discussion among the community in
[Reitit issue #84](https://github.com/metosin/reitit/issues/84)
and some valiant first implementation efforts by
[@czan](https://github.com/czan) and
[@Koura](https://github.com/Koura).

Finally, things started to come together last fall when
[@souenzzo](https://github.com/souenzzo)
(from [CyCognito](https://www.cycognito.com/))
submitted an initial version of OpenAPI 3 support in
[Reitit PR #563](https://github.com/metosin/reitit/pull/563)

I picked their work up this spring, added some missing pieces like
docs, tests, and coercion and
[multipart uploads](https://github.com/metosin/reitit/issues/591).

Finally, I wanted to make sure we support mutual recursion of schemas
properly. Luckily,
[@cap10morgan](https://github.com/cap10morgan)
was working on some fixes related to that
[on the Swagger side](https://github.com/metosin/malli/issues/464),
so we could help each other.
[Malli issue #868](https://github.com/metosin/malli/issues/868)
tracks all the fixes I made.

**A big thank you to everybody!**


. Please test it and report any problems on 

. We'll have a proper 0.7.0 release soon!

OpenAPI 3 Support for Reitit


A month ago I gave my first meetup talk ever. For those who are thinking of
doing this or have been there know it’s not necessarily easy to make the
decision to go talk to a bunch of seasoned programmers about _any_ topic. The
bar definitely feels high even when it shouldn’t be.

At the end of October, my colleague invited me to talk about this curious side
project of mine in a Clojure meetup that would be held in Oulu, in the northern
half of Finland. Little wonder as I'd been blabbering about my side project
during friday coffee sessions, demoing it to colleagues, and thus had gathered
interest in the topic. I slept on the decision and ended up accepting the
invitation.

The meetup was set to be held on December 1st and was hosted by the kind
[Enhancell](https://enhancell.com/) folks within their cozy office premises in
the heart of Oulu city.

## Clock starts ticking

<img src="../../img/posts/clojure-meetup-oulu-2022-onmyway.jpg" style="width:100%;">

The countdown began with four weeks on the clock. I decided on my title right
after receiving the invitation. For me, it was important to have an explicit
title so that the contents of that talk would be clear beforehand and potential
guests could make their RSVP decisions more easily. The title of my talk was:

> Creating an experimental GraphQL formatter using Clojure, Instaparse, and GraalVM. 

I knew I had to put some buzzwords in there, so I put all of them there. Some
old, some new - all good, as they say!

At end of the week, I started preparing my presentation at a high-level using
markdown Gist and bullet points. This allowed me to create the _narrative_
without spending too much time on the graphics, typography, demo code, and
other embellishments at this point. Crafting the bullet points took me one
friday evening.

## Three weeks left

At this point, I still had ample amount of time to get things right. Yes, even
my editor code style schemes.

But I still had to get my bullet points right as I knew my draft was still very
rough. I spent the entire next week fixing all my typos, bending the narrative
to my will, and thinking about the lessons in all of this. This retrospective
process involved going through the code and commits from the beginning and
reflecting on my feelings about the problem/solution space, and also digging up
my notes on different experiments and reflecting on those.

Another week began, and I already knew I had to start fleshing out my slides.
The sand was sifting through the hourglass and I didn't have any memes on my
powerpoint yet. 

And to add to my bikeshed woes, no way people would be reading markdown Gist
with enhanced grandpa font sizing (if only I had enough time to learn emacs org
mode)! Where were those company slide templates again? Which template was the
least stale out of the three?

Pretty soon I would have to start throwing that clay into something far more
beautiful. So that's what I started doing. 

At the end of the week, I had something that I could call "graphical" (in lieu
of a better word). I also wrote some [demo
code](https://github.com/ilmoraunio/graphqlfmt-clj-experiment/blob/meetup-demo/demo/evolution_of_the_ast.clj)
to showcase parts from the Abstract Syntax Tree processing pipeline that I had
built for my formatter, and then continued working on my slides, spit-polishing
the graphical parts, making sure I was using the right font sizes consistently
everywhere, etc. You know, the _important_ stuff.

## Final week: the hustle edition

On the final week, I solely focused on presentation notes, refining both the
content of the slides and rehearsing what my notes would sound like when I
actually presented it.

My goal was to aim for a maximum of 20-to-30 minutes which I thought would be a
good meetup talk length. I basically adjusted this to the time we had in total
and the number of talks we would have, plus adding into the consideration the
fact that people probably wanted to socialise as well.

Being on stage can make you forget ALL kinds of lines, improvised or not. It
was very important to write those notes down. For me, this was extra important
as I have a habit of forgetting a lot of words and things when under stress.

Initially, I’d set myself the goal not to improvise too much during rehearsal.
I only did it during times when I really couldn’t come up with anything that I
could write down in my notes. If something good came up during practise, I
would write that down. Repeat ad nauseam.

My final rehearsal in the hotel room before the meetup was strictly on-script
only with **very** minor changes to my slides and presentation notes. Also,
before the presentation, I added some very quick and small amendments to the
notes. Nothing too fundamental.

## Stepping into the fray

<img src="../../img/posts/clojure-meetup-oulu-2022-intro.jpg" style="width:100%;">

I already knew well in advance that I couldn't _just_ read from the script. I
would have to up the ad-libbing during the actual talk to make it more
interesting for the crowd.

In fact, when I watched **Jarppe Länsiö**, a Metosin colleague of mine, present
the **Architect’s Case** where he explained the architectural rationale for
selecting Clojure I became more confident of my decision. His talk was a tour
de force and contained little if any notes (or any that I saw at least). He
made it _appear_ as if he had improvised the whole talk. It was fun and
engaging, not least because of its blitzy tempo but also because some new war
stories were shared.

After Jarppe's talk, it was my turn to talk. While I’d clocked my presentation
during rehearsals to stop somewhere during the 20-to-30-minute mark, my actual
talk took me 43 minutes! And with the after-discussion I went well beyond
overtime, clocking in at 50 minutes.

<img src="../../img/posts/clojure-meetup-oulu-2022-ilmo.jpg" style="width:100%;">
Me, tuning the ad-lib to 11

The presentation itself went without major problems and there were some
questions during the presentation as well as after the presentation. There were
also a couple moments where I struggled a bit in trying to explain the concept
of Instaparse in a beginner-friendly way. Something I had not planned in my
notes. I also didn't expect that there would be people who hadn't really done
much with Clojure, but I probably should have!

## Show must go on

Besides these two presentations, we also heard from **Martín Varela** about his
historical path towards Clojure from the academic world and his current work
where he’d, among other things, produced a whitepaper pinpointing anomaly
events in logs from stability testing using N-grams or Deep-Learning as part of
his normal day-to-day programming work. This was insightful and also gave me
some ideas on how to leverage something like this, only simpler, for structured
logging data as well (something which the whitepaper did not focus on
primarily).

<img src="../../img/posts/clojure-meetup-oulu-2022-martin.jpg" style="width:100%;">
Martin Varela, opening up on the perceived difficulties of ze parentheses

All in all, the Clojure Meetup Oulu was really nice with friendly people from
diverse backgrounds. The guests had really good questions throughout the
presentations and they seemed to have enjoyed their time there! The hosts were
also super nice to us even when we went slightly overtime.

And finally, let’s not forget the second most important bit here. The food. We
had vegan and gluten-free pizza for all plus inclusive drinking options.

After the meetup, we switched locations to discuss on the meetup topics but
also other programming-related topics such as emacs and technical metal. All in
all, what a blast!

## Reflection

If you’ve never held a talk at a meetup before, I warmly recommend doing so as
it’s a great experience. It’s not necessary to set a high bar for yourself or
strive for perfection nor do you need to write code for as long as I did. 

All talks will be different and interesting in their own way, both in terms of
content but also presentation style. The most important part is that you feel
you want to talk about it.

For me, this was a unique and inclusive first-time experience. I definitely
encourage everyone to try it out once you’ve reached a certain level of
comfortability to start blabbering about something in front of a small crowd. :-)

<br />
<hr />

_Here is the link to my presentation slides:
https://www.slideshare.net/metosin/creating-an-experimental-graphql-formatter-using-clojure-instaparse-and-graalvm_

_And also, if you’d like to join our company where we focus on all things
Clojure (be it meetups, [naked-performance
programming](https://www.youtube.com/watch?v=3SSHjKT3ZmA), or Clojure
consulting), let’s start talking: rekry@metosin.fi_


My First Meetup Talk


```
;;Merry Christmas from
;;Metosin! Run this
;;hand-baked Clojure
;;cookie to find our
;;wish for this holiday
(reduce reduce reduce[,
[reduce[comp][eval,,,,,
replace]][{\e,,,,,,,,,,
(comment)\h,,,,,,,,,,,,
assoc\i,,,,,,,,,,,,,,,,
str},,,,,,,,,,,,,,,,,,,
"herewith""tr,,,,,,,,,,
wdpuocs",,,,,,,,,,,,,,"
reduce"]]),,,,,,,,,,,,,
```


A Christmas Card in Clojure

Related posts

Malli, Data Modelling for Clojure Developers

OpenAPI 3 Support for Reitit

My First Meetup Talk

Contact