jq
notesGuy Lapalme , RALI-DIRO, Université de Montréal, December 2024.
[Version française de ce document]
jq
is a command line tool that allows you to manipulate.json
or .jsonl
files for validation, formatting, information extraction or transformation. It can be considered the counterpart of sed
or awk
for text files containing tab-separated line information. The system is fast and efficient for processing large files, it can also be used in streaming mode .
If the input file contains many JSON objects (this is typically the case with .jsonl
files), the program is applied to each object and each result is computed separately. A program argument (--slurp
or -s
) will make the computation behave as if all objects are on a list of objects. Global operations such as sorting or duplicate removal can thus be applied.
The official documentation presents a tutorial with a few examples and a manual that lists all the functions illustrated with short, often simplistic, examples, but it does not explain their use on interesting tasks.
The syntax of jq
expressions is daunting at first, as it relies on composing filters (point-free programming) rather than using a more usual functional notation. jq
is a full-fledged programming language with control structures, variables, function definitions and even modules. Given that all my applications have been limited to a single expression at the command line, this document focuses on this aspect of the language. This type of expression allows performing very useful tasks.
There are several tutorials on the web, including this one, which is excellent, but now full of advertisements. However, they don't emphasize the parts that I think are most useful for the use cases I've encountered, i.e., extracting or transforming JSON values in files.
This document shows the concepts used in jq
programming and a short list of functions that were most useful to me. I then show examples of expressions applied to two file organizations: one with a single JSON object and another in JSONL containing many JSON structures. Finally I will give a few links that I found the most interesting among the hundreds on the web that often more or less reuse the same simplistic examples.
This document will therefore serve as a memory aid even if, now that I have written it, I am less likely to need it, but I hope that it will be useful to others.
jq
expression is a filter that transforms its input into an output. All elements of the jq
language are filters, even a number or string, which can be considered as filters which ignore their input and always produce the same value.
Once installed, jq
is called from the command line: jq
[options] '
filter'
file , the filter being surrounded by single apostrophes to prevent its interpretation by the shell before being passed to jq
.
A filter is applied to a JSON data stream. jq
parses this stream as a sequence of space-separated JSON values that are validated and given as input to the filter one by one. The filter's output(s) are sent to stdout as a sequence of JSON values separated by a newline . By default, the JSON value is displayed with an indentation to highlight the JSON structure (pretty-print). This indentation is not applied if the -c option
is specified.
Remember that in addition to primitive values (number, string, boolean, null
), JSON provides two data structures:
object : a set of pairs key :
value separated by commas inside curly braces; the keys must be unique within a single object and there is no ordering between the pairs.
array : values separated by commas surrounded by square brackets that can be indexed by position starting at 0.
The following is a selection of functions from the hundreds described in the manual .
.
: identity : returns the input as is: this filter might seem useless, but when applied it to a JSON file, it validates it and outputs an indented version.
f1 | f2
: filter composition : redirects the output of filter f1 to the input of f2 .
f1 , f2
: filter concatenation : the output of f2 is added at the end of the output of f1 .
.["key"]
: object index : applied to an object, returns the value associated with key if it appears in the object, otherwise null
. If key is a simple identifier, we can simplify the notation to .key.
.[number]
: array index : applied to an array, returns the value at the position indicated by the number, number can be negative to index from the end.
.[]
: value generation : applied to an object, returns all values of the object as separate results; .key | .[]
can also be written ..key[]
.
..
: recursive descent : returns all nested values , often combined with |.key?
to get all values associated with key
at all nested levels with the object.
{…}
: construction of an object : collects the content which must be of the form key :
value in a single object. key is often a character string, but can also be an expression (e.g. a filter) if put in parentheses.
[…]
: array construction : combines the content into an array
(…)
: grouping : groups a subexpression according to the usual conventions
select(f)
: selection : keep only the elements for which f
returns true
strings
: keep only values that are character strings; there are similar filters for other kinds of values, e.g. nulls
, numbers
or values
(non-null
).
keys
: keys of an object : returns an array with the keys of the input object. An array of the values of an object is obtained with .[]
(see above).
to_entries
: key-value pairs of an object : transforms an object {keyi:valuei}
into an array of objects of the form {"key":keyi:"value":valuei}
del(key)
: deletion : remove the key and associated value in an object
f1 + f2
: addition of filters : depends on the filter types: numbers: arithmetic addition; string or array: concatenation; objects: fusion.
f1 - f2
: subtraction of filters : depends on the types of filters: numbers: arithmetic subtraction; array: removes from f1 the elements of f2 .
reduce results as $e (init; combination involving $e)
: combine all elements ($e) from a series of results within a single object initialized with init
==
!=
>
>=
<
<=
: comparison according to the usual conventions
and
or
not
: combination of booleans
test(regex)
: check with a regular expression
startswith(str)
endswith(str)
: check : does the start or end of string match str .
split(str)
join(str)
: conversion string <=>array : array of substrings between str, create a string from an array by adding str between each element.
Tip: Try these examples with jqTerm or jqplay, which allow you to run them directly in a web browser and see the results immediately.
.json
file [source]The following JSON structure represents an excerpt from an English dictionary whose keys are words. Each word is associated with an object which describes information according to its category ( A
: adjective, N
: noun or V
: verb). Each category is associated with a table of inflections ( tab
) and, for some, if the word is common (basic
). A noun can be countable (yes
), non-countable (no
) or both .
x1{"comfortable":{"A":{"tab":"a1"},
2 "basic":true},
3 "comic":{"A":{"tab":"a1"},
4 "N":{"tab":"n1",
5 "cnt":"yes"}},
6 "comical":{"A":{"tab":"a1"}},
7 "comically":{"Adv":{"tab":"b1"}},
8 "coming":{"A":{"tab":"a1"},
9 "N":{"tab":"n1",
10 "cnt":"yes"},
11 "basic":true},
12 "comity":{"N":{"tab":"n5",
13 "cnt":"no"}},
14 "comma":{"N":{"tab":"n1",
15 "cnt":"yes"}},
16 "command":{"N":{"tab":"n1",
17 "cnt":"both"},
18 "V":{"tab":"v1"},
19 "basic":true},
20 "commandant":{"N":{"tab":"n1",
21 "cnt":"yes"}},
22 "commandment":{"N":{"tab":"n1",
23 "cnt":"yes"}},
24 "commando":{"N":{"tab":"n2",
25 "cnt":"yes"}},
26 "commemorate":{"V":{"tab":"v3"}},
27 "commemoration":{"N":{"tab":"n1",
28 "cnt":"both"}},
29 "commemorative":{"A":{"tab":"a1"}}}
30
keys
: array of all words ⇒
61[
2 "comfortable",
3 "comic",
4
5 "commemorative"
6]
to_entries[]|select(.value.basic)|.key
: list of words with the basic
indicator. Result on several lines ⇒
31"comfortable"
2"coming"
3"command"
As select
must be applied on a series of values, the input object must first be transformed into a list of entries {"key":..i:"value":...i}
. Only entries whose value
part contains the basic
key are kept; the last filter only displays the key
field.
to_entries[]|{(.key):.value|del(.basic)}
: list of JSON structures whose key is the word, but where the basic
field has been removed .⇒
191{
2 "comfortable": {
3 "A": {
4 "tab": "a1"
5 }
6 }
7}
8{
9 "comic": {
10 "A": {
11 "tab": "a1"
12 },
13 "N": {
14 "tab": "n1",
15 "cnt": "yes"
16 }
17 }
18}
19...
For each entry of the original object, a new object is created whose key is the key
field of the entry and its value is the value field from which the basic field
is removed, when present.
[ .. | .tab? | strings ] | unique
returns the liste of unique values of the tab
field in the file. ⇒ (displayed with option -c
)
11["a1","n1","n2","n5","v1","v3"]
All levels of the objects are traversed, when the tab
fields appear, its value is returned. strings
only keeps the string values thus removing null
s and any object that might be associated with the tab
field at any level. unique
remove duplicate values, but as it must be applied on an array, the previous filters must be wrapped between square brackets to build an array from their result.
reduce (.. | .tab? | strings) as $w ({}; .[$w]+=1)
creates an object whose keys are the different values for the tab key; the associated value is the number of times this value occurs in the input.
91{
2 "a1": 5,
3 "n1": 7,
4 "b1": 1,
5 "n5": 1,
6 "v1": 1,
7 "n2": 1,
8 "v3": 1
9}
Combination by reduce of the values produced by the previous example starting from an initially empty object. For each result $w, its value is incremented. When the value does not exist, it is initialized to 0 before incrementation.
.jsonl
fileEach line of a JSON Lines file is a JSON object describing a word.
Note: This file could have been created from the previous one with the expression to_entries[] | {word:.key}+.value
141{"word":"comfortable","A":{"tab":"a1"},"basic":true}
2{"word":"comic","A":{"tab":"a1"},"N":{"tab":"n1","cnt":"yes"}}
3{"word":"comical","A":{"tab":"a1"}}
4{"word":"comically","Adv":{"tab":"b1"}}
5{"word":"coming","A":{"tab":"a1"},"N":{"tab":"n1","cnt":"yes"},"basic":true}
6{"word":"comity","N":{"tab":"n5","cnt":"no"}}
7{"word":"comma","N":{"tab":"n1","cnt":"yes"}}
8{"word":"command","N":{"tab":"n1","cnt":"both"},"V":{"tab":"v1"},"basic":true}
9{"word":"commandant","N":{"tab":"n1","cnt":"yes"}}
10{"word":"commandment","N":{"tab":"n1","cnt":"yes"}}
11{"word":"commando","N":{"tab":"n2","cnt":"yes"}}
12{"word":"commemorate","V":{"tab":"v3"}}
13{"word":"commemoration","N":{"tab":"n1","cnt":"both"}}
14{"word":"commemorative","A":{"tab":"a1"}}
As there are many JSON objects in the same file, the jq
expression is applied to each, each result being concatenated on the output. A JSON object can span more than one line provided its structure is well-formed.
.word
: all words ⇒
51"comfortable"
2"comic"
3"comical"
4"comically"
5
select(.basic)|.word
: words with the basic indicator
⇒
31"comfortable"
2"coming"
3"command"
{(.word):.|del(.basic)}
: list of JSON structures whose key is the word, but where the basic
field has been removed. ⇒ output using option (-c
ou --compact-output
) with no indentation.
31{"comfortable":{"word":"comfortable","A":{"tab":"a1"}}}
2{"comic":{"word":"comic","A":{"tab":"a1"},"N":{"tab":"n1","cnt":"yes"}}}
3
.. | .tab? | strings
: all values of the tab
key
51"a1"
2"a1"
3"n1"
4"a1"
5
In this output, all values are the result of the application of the expression on independent JSON values, so they are independent from each other. To only keep unique values, the JSON file should be consider as a single object: a list containing these objects which can be obtained with the option --slurp
(-s
).
[.[] | .. | .tab? | strings ] | unique
called with c -s
, This produces a list of unique values of the values for the tab
key in the file. ⇒ (displayed with option -c
)
11["a1","n1","n2","n5","v1","v3"]
reduce (.. | .tab? | strings) as $w ({}; .[$w]+=1)
returns an object with keys that are unique to the key “tab” in the JSON and a count of how often each key appears. This expression combining many values must be called with option -s
.
91{
2 "a1": 5,
3 "n1": 7,
4 "b1": 1,
5 "n5": 1,
6 "v1": 1,
7 "n2": 1,
8 "v3": 1
9}
Combination by reduce of the values produced by the previous example starting from an initially empty object. For each result $w, its value is incremented. When the value does not exist, it is initialized to 0 before incrementation.
Note: For the inquisitive reader : with the -s option
, the expression
11reduce . [] as $elem ( {} ; .+ { ($elem|.word) : $elem|del(.word) } )
recreates from .jsonl
file the JSON structure from the previous section using the following method: it starts from an empty object and then for each element of the array, we merge an object having for key the value of the key word
and, for value, all the properties of the element except the word
field.
Exploring jq : High-level tips for using jq
in conjunction with other systems.
jq Language Description: more formal definition of the jq
language with many interesting insights
The Ultimate Interactive JQ Guide : 25 interactive and well explained examples
Examples of more complex jq
programs:
On-line test of expressions