Native API¶

The native simdjson API offers significant performance improvements over the builtin-compatible API if only part of a document is of interest.

Objects and arrays are returned as fake dicts (Object) and lists (Array) that delay the creation of Python objects until they are accessed.

Proxies and iterators remain valid after parser reuse or destruction. Each retains its entire document until the last reference is released. Documents are not cached.

Duplicate object keys¶

The native Object proxy preserves duplicate keys in the JSON document. Key lookup, get(), and JSON pointer lookup return the first matching value. Iteration through keys(), values(), and items() includes every field, and len() counts fields, including duplicates.

Converting to Python dictionaries with as_dict(), recursive=True, or simdjson.load() / simdjson.loads() keeps the last value for each key, as the standard library’s JSON decoder does. For example, with {"a": 1, "a": 2}, the proxy’s obj['a'] is 1 and len(obj) is 2, while obj.as_dict() is {'a': 2}.

Streaming records¶

parse_many() reads NDJSON from memory; load_many() reads a path or binary file. Both return a DocumentStream yielding ordinary Python values. Each nonblank line must contain one complete JSON value. LF and CRLF are supported, and the final newline is optional. Invalid or oversized records raise ValueError during iteration.

import simdjson

with simdjson.load_many('records.ndjson') as records:
    for record in records:
        process(record)

batch_size must be between 32 and max_capacity bytes. Each nonblank line must be smaller than this in UTF-8 bytes, including whitespace but excluding LF/CRLF and an initial BOM. Blank lines have no size limit. parse_many() copies the entire input; load_many() uses buffers proportional to batch_size.

simdjson.parse_many(src, *, batch_size=1000000)[source]¶

Parse NDJSON from memory, yielding independent Python values.

See Streaming records for format and size limits.

simdjson.load_many(source, *, batch_size=1000000)[source]¶

Read NDJSON from a path or binary file using bounded input chunks.

See Streaming records for format and size limits.

Classes¶

class simdjson.Parser¶

Parse JSON documents while reusing scratch buffers.

Parameters:: max_capacity – Maximum input document size in bytes, not a total-memory limit. [default: SIMDJSON_MAXSIZE_BYTES]

get_implementations(supported_by_runtime=True)¶

A list of available parser implementations in the form of [(name, description),…].

By default, this only returns the implementations that are usable on the current runtime. Setting supported_by_runtime to False will instead return all the implementations _compiled_ into this build of simdjson.

implementation¶

The active parser Implementation as (name, description). Can be any value from implementations. The best Implementation for your current platform will be picked by default.

Can be set to the name of any valid Implementation to globally change underlying Parser Implementation, such as to disable AVX-512 if it is causing down-clocking.

load(path, recursive=False)¶

Load a JSON document from the file system path path.

Existing proxies remain valid when the parser is reused or destroyed.

Parameters:

path – A filesystem path.
recursive – Recursively turn the document into real python objects instead of pysimdjson proxies.

load_many(source, *, batch_size=1000000)¶

Read NDJSON from a path or binary file, yielding independent Python values.

Returns a DocumentStream using bounded reads; caller-owned files stay open. See Streaming records for format and size limits.

parse(src, recursive=False)¶

Parse the given JSON document.

The source document may be a str, bytes, bytearray, or any other object that implements the buffer protocol.

Performance

While you can pass quite a few things to this method to be parsed, simple bytes will almost always be the fastest.

Existing proxies remain valid when the parser is reused or destroyed.

Parameters:

src – The document to parse.
recursive – Recursively turn the document into real python objects instead of pysimdjson proxies. [default: False]

parse_many(src, *, batch_size=1000000)¶

Parse NDJSON from memory, yielding independent Python values.

Returns a DocumentStream owning a copy of the input. See Streaming records for format and size limits.

class simdjson.Array¶

A proxy object that behaves much like a real list().

Python objects are not created until an element in the list is accessed. When you only need a subset of an Array, this can be much faster than converting an entire array (and all of its children) into real Python objects.

as_buffer(*, of_type)¶

Copies the contents of a homogeneous array to an object that can be used as a buffer. This means it can be used as input for numpy.frombuffer, bytearray, memoryview, etc.

When n-dimensional arrays are encountered, this method will recursively flatten them.

Note

The object returned by this method contains a copy of the Array’s data. Thus, it’s safe to use even after the Array or Parser are destroyed or reused.

Parameters:: of_type – One of ‘d’ (double), ‘i’ (signed 64-bit integer) or ‘u’ (unsigned 64-bit integer).

as_list()¶: Convert this Array to a regular python list, recursively converting any objects/lists it finds.

at_pointer(json_pointer)¶: Get the value at the given JSON pointer.

mini¶

Returns the minified JSON representation of this Array.

Return type:: bytes

parser¶: The originating Parser, or None after it is destroyed.

class simdjson.Object¶

A proxy object that behaves much like a real dict().

Python objects are not created until an element in the Object is accessed. When you only need a subset of an Object, this can be much faster than converting an entire Object (and all of its children) into real Python objects.

as_dict()¶: Convert this Object to a regular python dictionary, recursively converting any objects or lists it finds.

at_pointer(json_pointer)¶: Get the value at the given JSON pointer.

get(key, default=None)¶: Return the value of key, or default if the key does not exist.

items()¶: Returns an iterator over all the (key, value) pairs in this Object.

keys()¶: Returns an iterator over all keys in this Object.

mini¶

Returns the minified JSON representation of this Object.

Return type:: bytes

parser¶: The originating Parser, or None after it is destroyed.

values()¶: Returns an iterator over of all values in this Object.

class simdjson.DocumentStream¶

An iterator and context manager for NDJSON records.

close()¶: Release resources, leaving caller-provided files open.

closed¶: Whether this stream has released its resources.

Constants¶

simdjson.MAXSIZE_BYTES: int = 4294967295¶: The maximum document size (in bytes) supported by simdjson.

simdjson.PADDING: int = 32¶

The amount of padding needed in a buffer to parse JSON.

In general, pysimdjson takes care of padding for you and you do not need to worry about this.

simdjson.VERSION: str¶: The version of the embedded simdjson library.