jq notes

Guy Lapalme , RALI-DIRO, Université de Montréal, December 2024.

[Version française de ce document]

jq is a command line tool that allows you to manipulate.json or .jsonl files for validation, formatting, information extraction or transformation. It can be considered the counterpart of sed or awk for text files containing tab-separated line information. The system is fast and efficient for processing large files, it can also be used in streaming mode .

If the input file contains many JSON objects (this is typically the case with .jsonl files), the program is applied to each object and each result is computed separately. A program argument (--slurp or -s) will make the computation behave as if all objects are on a list of objects. Global operations such as sorting or duplicate removal can thus be applied.

The official documentation presents a tutorial with a few examples and a manual that lists all the functions illustrated with short, often simplistic, examples, but it does not explain their use on interesting tasks.

The syntax of jq expressions is daunting at first, as it relies on composing filters (point-free programming) rather than using a more usual functional notation. jq is a full-fledged programming language with control structures, variables, function definitions and even modules. Given that all my applications have been limited to a single expression at the command line, this document focuses on this aspect of the language. This type of expression allows performing very useful tasks.

There are several tutorials on the web, including this one, which is excellent, but now full of advertisements. However, they don't emphasize the parts that I think are most useful for the use cases I've encountered, i.e., extracting or transforming JSON values in files.

This document shows the concepts used in jq programming and a short list of functions that were most useful to me. I then show examples of expressions applied to two file organizations: one with a single JSON object and another in JSONL containing many JSON structures. Finally I will give a few links that I found the most interesting among the hundreds on the web that often more or less reuse the same simplistic examples.

This document will therefore serve as a memory aid even if, now that I have written it, I am less likely to need it, but I hope that it will be useful to others.

Concepts

jq expression is a filter that transforms its input into an output. All elements of the jq language are filters, even a number or string, which can be considered as filters which ignore their input and always produce the same value.

Once installed, jq is called from the command line: jq [options] 'filter' file , the filter being surrounded by single apostrophes to prevent its interpretation by the shell before being passed to jq.

A filter is applied to a JSON data stream. jq parses this stream as a sequence of space-separated JSON values that are validated and given as input to the filter one by one. The filter's output(s) are sent to stdout as a sequence of JSON values separated by a newline . By default, the JSON value is displayed with an indentation to highlight the JSON structure (pretty-print). This indentation is not applied if the -c option is specified.

Remember that in addition to primitive values (number, string, boolean, null), JSON provides two data structures:

The following is a selection of functions from the hundreds described in the manual .

Filters

Groupings

Filters

Tests

String processing

Practical examples

Tip: Try these examples with jqTerm or jqplay, which allow you to run them directly in a web browser and see the results immediately.

.json file [source]

The following JSON structure represents an excerpt from an English dictionary whose keys are words. Each word is associated with an object which describes information according to its category ( A: adjective, N: noun or V: verb). Each category is associated with a table of inflections ( tab) and, for some, if the word is common (basic). A noun can be countable (yes), non-countable (no) or both .

Operations [source]

.jsonl file

Each line of a JSON Lines file is a JSON object describing a word.

Note: This file could have been created from the previous one with the expression to_entries[] | {word:.key}+.value

As there are many JSON objects in the same file, the jq expression is applied to each, each result being concatenated on the output. A JSON object can span more than one line provided its structure is well-formed.

Operations [source]

Note: For the inquisitive reader : with the -s option , the expression

recreates from .jsonl file the JSON structure from the previous section using the following method: it starts from an empty object and then for each element of the array, we merge an object having for key the value of the key word and, for value, all the properties of the element except the word field.

Interesting links