njson

A JSON handling framework aiming for convenience and brevity.

NJSON aims to make it extremely convenient for you to decode, validate, destructure, process, and encode JSON data, in the minimum keystrokes/minutes possible.

Getting started

Clone the Git repository:

git clone --recursive https://github.com/atlas-engineer/njson ~/common-lisp/njson

Load NJSON (with a jzon backend) in the REPL:

;; Show ASDF where NJSON is.
(asdf:load-asd #p"/path/to/checkout/njson.asd")
;; Load it with ASDF.
(asdf:load-system :njson/jzon)
;; Alternatively, load it with Quicklisp.
(ql:quickload :njson/jzon)

And start parsing right away, be it from file:

(njson:decode #p"/path/to/njson/checkout/tests/test.json")
;; => #(1 3.8 T NIL :NULL "foo" #(1 2 3) #("bar" T :NULL 1000000)
;;      #<HASH-TABLE :TEST EQUAL :COUNT 1 {100EAB1383}>
;;      #<HASH-TABLE :TEST EQUAL :COUNT 3 {100EAB16D3}>)

or from string:

(njson:decode "[\"hello\", 5]")
;; => #("hello", 5)

or other specializeable types. Default methods support:

• pathnames,
• strings,
• streams.

Running tests

Given NJSON backend-agnostic nature, you can only test every particular backend against the uniform set of tests that NJSON provides. So, to test jzon backend, you can do:

(asdf:test-system :njson/jzon)

And, for the CL-JSON backend,

(asdf:test-system :njson/cl-json)

What NJSON is not (and what it is, instead)

NJSON is not a JSON parsing library.

It’s one level higher: it’s a convenience wrapper around your JSON parser of choice. NJSON is made in such a way so as to be usable with almost any JSON library out there. The bundled backends are jzon (reliable, though new), and CL-JSON (fuzzy yet battle-proven).

• To make NJSON support your preferred JSON parser, you have to specialize as little as two methods: decode-from-stream and encode-to-stream. If you care about correctness or proper type dispatching, you may also define (en|de)code-(to|from)-string and (en|de)code-(to|from)-file.

NJSON is not propagating unnecessary dependencies on you.

The core (njson ASDF system) has no dependencies due to specifying only the generics to implement.

Every other dependency is optional and depends on which backend you want to use for parsing.

NJSON is not the fastest JSON handling solution out there.

Plug-n-play usability and type variety are much higher of a priority than the performance. The types NJSON returns from its methods (and that your own methods extending NJSON should expect/return) are:

• Lisp real-s for JSON numbers.
• Lisp strings for JSON strings.
• :null for JSON null.
• t for true and nil for false.
• Vectors for JSON arrays.
• Hash-tables for JSON objects.

With this basic (yet disjoint) set of types, you can easily typecase over NJSON output and make informed decisions about the JSON you have. Even if it’s some couple of CPU work milliseconds slower than handling raw lists. It’s faster in human work seconds, which are much more valuable.

NJSON is not minimalist.

NJSON has strict requirements on the returned data, but this strictness enables a rich set of JSON-handling primitives/helpers. You can

• (:use #:cl #:njson) in your packages if you want short and convenient JSON operations there. It’s safe, because NJSON shadows no symbols from CL.
• Or you can define a package local nickname for :njson/aliases to be a mere j: (using trivial-package-local-nicknames), so that even shorter helpers (just a couple of characters longer than the regular CL constructs) are available:

(trivial-package-local-nicknames:add-package-local-nickname :j :njson/aliases :YOUR-PACKAGE)
;; And then use it like.
(j:get ...)
(j:decode ...)
(j:if ...)
(j:match ...)

See the next section for the functions/macros NJSON exports.

API

FUNCTION njson:jget (alias: njson/aliases:get)

Gets the value from the JSON object/array indexed by a certain key. Note that the second value is a boolean denoting whether the entry under key is found (like in gethash).

(defvar data (njson:decode "{\"key\": 5, \"second-key\": [1, 2, false]}"))
(njson:jget "key" data)
;; => 5, T

;; Index using sequence:
(njson:jget '("second-key" 1) data)
;; => 2, T

;; Index using JSON Pointer (as pathname):
(njson:jget #p"/second-key/0" data)
;; => 1, T

;; Modify the element in place:
(setf (njson:jget #p"/second-key/0" data) 3)
;; Another indexing syntax, for no particular reason:
(njson:jget #("second-key" 0) data)
;; => 3, T

Note the pathname indexing—it uses the JSON Pointer syntax for indexing convenience.

FUNCTION njson:jget* (alias: njson/aliases:get*)

A stricter version of jget that throws no-key error when there’s nothing under the given key in the provided object.

Will be merged into jget with the next major release.

FUNCTION njson:jcopy (alias: njson/aliases:copy)

Copies the whole thing it’s passed, no mater the nesting, into a fresh new equal object. Makes all the arrays adjustable and fillable for further possibly destructive use.

(defvar data (njson:jget "key" (njson:decode "{\"key\": 5}")))
;; => 5, T
(njson:jget "key" (njson:jcopy data))
;; => 5, T

FUNCTION njson:jkeys (alias: njson/aliases:keys)

Gets all the keys present in the passed object. Integer keys for arrays, string keys for object, error for anything else.

(njson:jkeys (njson:decode "{\"a\": 1, \"b\": 2}"))
;; ("a" "b")
(njson:jkeys (njson:decode "[\"a\", \"b\"]"))
;; (0 1)

FUNCTIONS njson:ensure-array, njson:ensure-object (aliases: njson/aliases:ensure-array, njson/aliases:ensure-object)

Ensure that the passed object is turned into array or object (respectively). If :convert-objects is provided in njson:ensure-array, it creates an array with all the values of object, discarding keys.

(njson:ensure-array #(1 2 3))
;; #(1 2 3)
(njson:ensure-array 3)
;; #(3)
(njson:ensure-array (njson:decode "{\"a\": 3}"))
;; #(#<hash-table>)
(njson:ensure-array (njson:decode "{\"a\": 3}") :convert-objects t)
;; #(3)

(njson:ensure-object "key" #<hash-table>)
;; #<hash-table>
(njson:ensure-object "key" 3)
;; #<hash-table> with "key": 3
(njson:ensure-object "key" #(1 2 3))
;; #<hash-table> with "key": #(1 2 3)

FUNCTION njson:jtruep (aliases: njson:jtrue-p, njson:jtrue?, njson:truep, njson:true-p, njson:true?)

Checks whether the given value is true (in other words, neither false, nor null) per JSON.

All the macros below utilize it, so, if you want to change the behavior of those, specialize this function.

MACRO njson:jwhen (alias: njson/aliases:when)

A regular CL when made aware of JSON’s null and false.

(njson:jwhen (njson:decode "null")
  "This is never returned.")
;; nil
(njson:jwhen (njson:decode "5")
  "This is always returned.")
;; "This is always returned"

MACRO njson:if (alias: njson/aliases:if)

A regular Lisp if aware of JSON truths and lies.

(njson:jif (njson:decode "5")
           "This is always returned."
           "This is never returned.")
;; "This is always returned"

MACRO njson:jor, njson:jand, njson:jnot (and aliases: njson/aliases:or, njson/aliases:and, njson/aliases:not)

Regular Lisp logic operators, with awareness of JSON values.

MACRO njson:jbind (alias njson/aliases:bind)

Destructures a JSON object against the provided destructuring pattern. This is most useful for deeply nested JSON structures often returned from old/corporate APIs. One example of such APIs is the Reddit one. To get to the title of the post, one has to go through half a dozen layers of nested objects and arrays:

[{"kind": "Listing",
  "data": {"children": [{"kind": "t3",
                         "data": {"approved_at_utc": null,
                                  "subreddit": "programming",
                                  ...
                                  // Finally, a title!
                                  "title": "Henry Baker: Meta-circular semantics for Common Lisp special forms",
                                  "link_flair_richtext": [],
                                  "subreddit_name_prefixed": "r/programming",
                                  ...}}]
           ...}}
 ...]

One needs a strong destructuring facility with type checking to move through this mess of JSON data. jbind is exactly this facility. Here’s how accessing the title of Reddit post would look like (array patterns access JSON arrays, list patterns access JSON objects) with jbind:

(njson:jbind #(("data" ("children" #(("data" ("title" title))))))
    ;; Dexador is not a dependency of NJSON, so load it separately
    (njson:decode
     (dex:get
      "https://www.reddit.com/r/programming/comments/6er9d/henry_baker_metacircular_semantics_for_common.json"))
  title)
;; "Henry Baker: Meta-circular semantics for Common Lisp special forms"

See documentation for more examples.

MACRO njson:jmatch (alias njson/aliases:match)

Matches/destructures the provided form against patterns one by one, and executes the body of the successfully matching one with the bindings it established. Every pattern and body is essentially a jbind with checking for destructuring success. The use-case is dispatching over API responses that differ in structure.

Telegram Bot API, for example, has disjoint contents for error responses and success responses:

• Error responses have “ok” key set to false, and keys called “description” and “error_code”.
• Successful responses have “ok” set to true and “result” as the payload they return.

Given these restrictions, we can jmatch the result of Bot API:

(njson:jmatch
 parsed-api-data
 (("ok" :true "result" result)
  (values t result))
 (("ok" :false "error_code" _ "description" description)
  (values nil description))
 (t (error "Malformed data!")))

After parsing the data, we have clear value distinctions:

• On success, return (VALUES (EQL T) *) with the payload.
• On error, return (VALUES NULL &OPTIONAL STRING).
• And in the exceptional case of malformed data, error out.

jmatch (and jbind) also checks the value matching (see the "ok" :true and "ok" :false parts) with arbitrary JSON atomic type (number, string, :true (for T), :false (for NIL), and :null). Arrays and lists are destructuring patterns already, so any value in them can be equality-checked.

ERROR njson:jerror

An umbrella class for all the NJSON errors. If you want to play unsafe, simply ignore all of NJSON errors:

(handler-case
    (njson:jget ...)
  ;; Or j:error if you nicknamed njson/aliases.
  (njson:jerror ()
    nil))

ERROR njson:encode-to-stream-not-implemented, njson:decode-from-stream-not-implemented

These get thrown when the JSON parsing back-end does not define methods for njson:encode-to-stream and njson:decode-from-stream. These are the bare minimum a backend should have to work. Adding the string and file methods is nice, but not required.

ERROR njson:invalid-key

This gets thrown when you try to index objects with integer indices and arrays with string keys. Because such an indexing wouldn’t make sense.

To allow string indexing for arrays (to make "1" be recognized as a valid index), you can patch the njson:jget method for string indices:

(defmethod njson:jget :around ((index string) (object array))
  (if (every #'digit-char-p index)
      (njson:jget (parse-integer index) object)
      (call-next-method)))

ERROR njson:non-indexable

It doesn’t make sense to index a number. This error reinforces the idea.

ERROR njson:invalid-pointer

This error is JSON Pointer specific. It’s thrown when there’s something wrong with the pointer syntax.

ERROR njson:no-key

This error is thrown in njson:jget* when the indexed object doesn’t have the key it’s indexed with.

ERROR njson:value-mismatch

Some value validated in njson:jbind didn’t match the expected value.

ERROR njson:deprecated

Marks a certain function as deprecated.