Skip to content
Standard libraryZig 0.17.0Official documentation
Sourcestd/json/Scanner.zig

Source line 1

The lowest level parsing API in this package; supports streaming input with a low memory footprint. The memory requirement is O(d) where d is the nesting depth of [] or {} containers in the input. Specifically d/8 bytes are required for this purpose, with some extra buffer according to the implementation of std.ArrayList.

This scanner can emit partial tokens; see std.json.Token. The input to this class is a sequence of input buffers that you must supply one at a time. Call feedInput() with the first buffer, then call next() repeatedly until error.BufferUnderrun is returned. Then call feedInput() again and so forth. Call endInput() when the last input buffer has been given to feedInput(), either immediately after calling feedInput(), or when error.BufferUnderrun requests more data and there is no more. Be sure to call next() after calling endInput() until Token.end_of_document has been returned.

Notes on standards compliance: https://datatracker.ietf.org/doc/html/rfc8259

  • RFC 8259 requires JSON documents be valid UTF-8, but makes an allowance for systems that are "part of a closed ecosystem". I have no idea what that's supposed to mean in the context of a standard specification. This implementation requires inputs to be valid UTF-8.
  • RFC 8259 contradicts itself regarding whether lowercase is allowed in \u hex digits, but this is probably a bug in the spec, and it's clear that lowercase is meant to be allowed. (RFC 5234 defines HEXDIG to only allow uppercase.)
  • When RFC 8259 refers to a "character", I assume they really mean a "Unicode scalar value". See http://www.unicode.org/glossary/#unicode_scalar_value .
  • RFC 8259 doesn't explicitly disallow unpaired surrogate halves in \u escape sequences, but vaguely implies that \u escapes are for encoding Unicode "characters" (i.e. Unicode scalar values?), which would mean that unpaired surrogate halves are forbidden. By contrast ECMA-404 (a competing(/compatible?) JSON standard, which JavaScript's JSON.parse() conforms to) explicitly allows unpaired surrogate halves. This implementation forbids unpaired surrogate halves in \u sequences. If a high surrogate half appears in a \u sequence, then a low surrogate half must immediately follow in \u notation.
  • RFC 8259 allows implementations to "accept non-JSON forms or extensions". This implementation does not accept any of that.
  • RFC 8259 allows implementations to put limits on "the size of texts", "the maximum depth of nesting", "the range and precision of numbers", and "the length and character contents of strings". This low-level implementation does not limit these, except where noted above, and except that nesting depth requires memory allocation. Note that this low-level API does not interpret numbers numerically, but simply emits their source form for some higher level code to make sense of.
  • This low-level implementation allows duplicate object keys, and key/value pairs are emitted in the order they appear in the input.

Fields

state: State = .value
string_is_object_key: bool = false
stack: BitStack
value_start: usize = undefined
utf16_code_units: [2]u16 = undefined
input: []const u8 = ""
cursor: usize = 0
is_end_of_input: bool = false
diagnostics: ?*Diagnostics = null

std.json.Scanner.Allocator

alias

Source line 48

Alias of std.mem.Allocator

std.json.Scanner.AllocError

error-set

Source line 116

Errors

OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong

std.json.Scanner.AllocIntoArrayListError

error-set

Source line 119

Errors

BufferUnderrun
OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong AllocError

std.json.Scanner.allocNextIntoArrayList

function

Source line 175

Equivalent to allocNextIntoArrayListMax(value_list, when, default_max_value_len);

pub fn allocNextIntoArrayList(self: *@This(), value_list: *std.array_list.Managed(u8), when: AllocWhen) AllocIntoArrayListError!?[]const u8

Parameters

self: *@This()
value_list: *std.array_list.Managed(u8)
when: AllocWhen

Errors

BufferUnderrun
OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong AllocError

std.json.Scanner.allocNextIntoArrayListMax

function

Source line 188

The next token type must be either .number or .string. See peekNextTokenType(). When allocation is not necessary with .alloc_if_needed, this method returns the content slice from the input buffer, and value_list is not touched. When allocation is necessary or with .alloc_always, this method concatenates partial tokens into the given value_list, and returns null once the final .number or .string token has been written into it. In case of an error.BufferUnderrun, partial values will be left in the given value_list. The given value_list is never reset by this method, so an error.BufferUnderrun situation can be resumed by passing the same array list in again. This method does not indicate whether the token content being returned is for a .number or .string token type; the caller of this method is expected to know which type of token is being processed.

pub fn allocNextIntoArrayListMax(self: *@This(), value_list: *std.array_list.Managed(u8), when: AllocWhen, max_value_len: usize) AllocIntoArrayListError!?[]const u8

Parameters

self: *@This()
value_list: *std.array_list.Managed(u8)
when: AllocWhen
max_value_len: usize

Errors

BufferUnderrun
OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong AllocError

std.json.Scanner.AllocWhen

container

Source line 1567

See the documentation for std.json.Token.

Fields

alloc_if_needed
alloc_always

std.json.Scanner.appendSlice

function

Source line 1748

fn appendSlice(list: *std.array_list.Managed(u8), buf: []const u8, max_value_len: usize) !void

Parameters

list: *std.array_list.Managed(u8)
buf: []const u8
max_value_len: usize

std.json.Scanner.ARRAY_MODE

constant

Source line 1746

std.json.Scanner.assert

alias

Source line 49

Alias of std.debug.assert

std.json.Scanner.BitStack

alias

Source line 50

Alias of std.BitStack

std.json.Scanner.default_buffer_size

constant

Source line 1422

Used by json.reader.

std.json.Scanner.default_max_value_len

constant

Source line 1571

For security, the maximum size allocated to store a single string or number value is limited to 4MiB by default. This limit can be specified by calling nextAllocMax() instead of nextAlloc().

std.json.Scanner.deinit

function

Source line 83

pub fn deinit(self: *@This()) void

Parameters

self: *@This()

std.json.Scanner.Diagnostics

container

Source line 1546

To enable diagnostics, declare var diagnostics = Diagnostics{}; then call source.enableDiagnostics(&diagnostics); where source is either a std.json.Reader or a std.json.Scanner that has just been initialized. At any time, notably just after an error, call getLine(), getColumn(), and/or getByteOffset() to get meaningful information from this.

Fields

line_number: u64 = 1
line_start_cursor: usize = @as(usize, @bitCast(@as(isize, -1)))
total_bytes_before_current_input: u64 = 0
cursor_pointer: *const usize = undefined

std.json.Scanner.Diagnostics.getByteOffset

function

Source line 1561

Starts at 0. Measures the byte offset since the start of the input.

pub fn getByteOffset(self: *const @This()) u64

Parameters

self: *const @This()

std.json.Scanner.Diagnostics.getColumn

function

Source line 1557

Starts at 1.

pub fn getColumn(self: *const @This()) u64

Parameters

self: *const @This()

std.json.Scanner.Diagnostics.getLine

function

Source line 1553

Starts at 1.

pub fn getLine(self: *const @This()) u64

Parameters

self: *const @This()

std.json.Scanner.enableDiagnostics

function

Source line 88

pub fn enableDiagnostics(self: *@This(), diagnostics: *Diagnostics) void

Parameters

self: *@This()
diagnostics: *Diagnostics

std.json.Scanner.endInput

function

Source line 111

Call this when you will no longer call feedInput() anymore. This can be called either immediately after the last feedInput(), or at any time afterward, such as when getting error.BufferUnderrun from next(). Don't forget to call next*() after endInput() until you get .end_of_document.

pub fn endInput(self: *@This()) void

Parameters

self: *@This()

std.json.Scanner.endOfBufferInNumber

function

Source line 1338

fn endOfBufferInNumber(self: *@This(), allow_end: bool) !Token

Parameters

self: *@This()
allow_end: bool

std.json.Scanner.endOfBufferInString

function

Source line 1349

fn endOfBufferInString(self: *@This()) !Token

Parameters

self: *@This()

std.json.Scanner.ensureTotalStackCapacity

function

Source line 316

Pre allocate memory to hold the given number of nesting levels. stackHeight() up to the given number will not cause allocations.

pub fn ensureTotalStackCapacity(self: *@This(), height: usize) Allocator.Error!void

Parameters

self: *@This()
height: usize

Errors

OutOfMemory

std.json.Scanner.Error

error-set

Source line 1419

The parsing errors are divided into two categories:

  • SyntaxError is for clearly malformed JSON documents, such as giving an input document that isn't JSON at all.
  • UnexpectedEndOfInput is for signaling that everything's been valid so far, but the input appears to be truncated for some reason. Note that a completely empty (or whitespace-only) input will give UnexpectedEndOfInput.

Errors

SyntaxError
UnexpectedEndOfInput

std.json.Scanner.expectByte

function

Source line 1270

fn expectByte(self: *const @This()) !u8

Parameters

self: *const @This()

std.json.Scanner.feedInput

function

Source line 95

Call this whenever you get error.BufferUnderrun from next(). When there is no more input to provide, call endInput().

pub fn feedInput(self: *@This(), input: []const u8) void

Parameters

self: *@This()
input: []const u8

std.json.Scanner.initCompleteInput

function

Source line 76

Use this if your input is a single slice. This is effectively equivalent to:

initStreaming(allocator);
feedInput(complete_input);
endInput();
pub fn initCompleteInput(allocator: Allocator, complete_input: []const u8) @This()

Parameters

allocator: Allocator
complete_input: []const u8

std.json.Scanner.initStreaming

function

Source line 64

The allocator is only used to track [] and {} nesting levels.

pub fn initStreaming(allocator: Allocator) @This()

Parameters

allocator: Allocator

std.json.Scanner.isNumberFormattedLikeAnInteger

function

Source line 1759

For the slice you get from a Token.number or Token.allocated_number, this function returns true if the number doesn't contain any fraction or exponent components, and is not -0. Note, the numeric value encoded by the value may still be an integer, such as 1.0. This function is meant to give a hint about whether integer parsing or float parsing should be used on the value. This function will not give meaningful results on non-numeric input.

pub fn isNumberFormattedLikeAnInteger(value: []const u8) bool

Parameters

value: []const u8

std.json.Scanner.next

function

Source line 321

See std.json.Token for documentation of this function.

pub fn next(self: *@This()) NextError!Token

Parameters

self: *@This()

Errors

BufferUnderrun
OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.nextAlloc

function

Source line 124

Equivalent to nextAllocMax(allocator, when, default_max_value_len); This function is only available after endInput() (or initCompleteInput()) has been called. See also std.json.Token for documentation of nextAlloc*() function behavior.

pub fn nextAlloc(self: *@This(), allocator: Allocator, when: AllocWhen) AllocError!Token

Parameters

self: *@This()
allocator: Allocator
when: AllocWhen

Errors

OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong

std.json.Scanner.nextAllocMax

function

Source line 130

This function is only available after endInput() (or initCompleteInput()) has been called. See also std.json.Token for documentation of nextAlloc*() function behavior.

pub fn nextAllocMax(self: *@This(), allocator: Allocator, when: AllocWhen, max_value_len: usize) AllocError!Token

Parameters

self: *@This()
allocator: Allocator
when: AllocWhen
max_value_len: usize

Errors

OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong

std.json.Scanner.NextError

error-set

Source line 115

Errors

BufferUnderrun
OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.OBJECT_MODE

constant

Source line 1745

std.json.Scanner.partialStringCodepoint

function

Source line 1382

fn partialStringCodepoint(code_point: u21) Token

Parameters

code_point: u21

std.json.Scanner.PeekError

error-set

Source line 117

Errors

BufferUnderrun
SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.peekNextTokenType

function

Source line 1092

Seeks ahead in the input until the first byte of the next token (or the end of the input) determines which type of token will be returned from the next next*() call. This function is idempotent, only advancing past commas, colons, and inter-token whitespace.

pub fn peekNextTokenType(self: *@This()) PeekError!TokenType

Parameters

self: *@This()

Errors

BufferUnderrun
SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.Reader

container

Source line 1574

All next*() methods here handle error.BufferUnderrun from std.json.Scanner, and then read from the reader.

Fields

scanner: Scanner
reader: *std.Io.Reader

std.json.Scanner.Reader.AllocError

error-set

Source line 1597

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong

std.json.Scanner.Reader.allocNextIntoArrayList

function

Source line 1641

Equivalent to allocNextIntoArrayListMax(value_list, when, default_max_value_len);

pub fn allocNextIntoArrayList(self: *@This(), value_list: *std.array_list.Managed(u8), when: AllocWhen) Reader.AllocError!?[]const u8

Parameters

self: *@This()
value_list: *std.array_list.Managed(u8)
when: AllocWhen

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong

std.json.Scanner.Reader.allocNextIntoArrayListMax

function

Source line 1645

Calls std.json.Scanner.allocNextIntoArrayListMax and handles error.BufferUnderrun.

pub fn allocNextIntoArrayListMax(self: *@This(), value_list: *std.array_list.Managed(u8), when: AllocWhen, max_value_len: usize) Reader.AllocError!?[]const u8

Parameters

self: *@This()
value_list: *std.array_list.Managed(u8)
when: AllocWhen
max_value_len: usize

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong

std.json.Scanner.Reader.deinit

function

Source line 1585

pub fn deinit(self: *@This()) void

Parameters

self: *@This()

std.json.Scanner.Reader.enableDiagnostics

function

Source line 1591

Calls std.json.Scanner.enableDiagnostics.

pub fn enableDiagnostics(self: *@This(), diagnostics: *Diagnostics) void

Parameters

self: *@This()
diagnostics: *Diagnostics

std.json.Scanner.Reader.ensureTotalStackCapacity

function

Source line 1705

Calls std.json.Scanner.ensureTotalStackCapacity.

pub fn ensureTotalStackCapacity(self: *@This(), height: usize) Allocator.Error!void

Parameters

self: *@This()
height: usize

Errors

OutOfMemory

std.json.Scanner.Reader.init

function

Source line 1579

The allocator is only used to track [] and {} nesting levels.

pub fn init(allocator: Allocator, io_reader: *std.Io.Reader) @This()

Parameters

allocator: Allocator
io_reader: *std.Io.Reader

std.json.Scanner.Reader.next

function

Source line 1710

See std.json.Token for documentation of this function.

pub fn next(self: *@This()) Reader.NextError!Token

Parameters

self: *@This()

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.Reader.nextAlloc

function

Source line 1602

Equivalent to nextAllocMax(allocator, when, default_max_value_len); See also std.json.Token for documentation of nextAlloc*() function behavior.

pub fn nextAlloc(self: *@This(), allocator: Allocator, when: AllocWhen) Reader.AllocError!Token

Parameters

self: *@This()
allocator: Allocator
when: AllocWhen

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong

std.json.Scanner.Reader.nextAllocMax

function

Source line 1606

See also std.json.Token for documentation of nextAlloc*() function behavior.

pub fn nextAllocMax(self: *@This(), allocator: Allocator, when: AllocWhen, max_value_len: usize) Reader.AllocError!Token

Parameters

self: *@This()
allocator: Allocator
when: AllocWhen
max_value_len: usize

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error
ValueTooLong

std.json.Scanner.Reader.NextError

error-set

Source line 1595

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.Reader.PeekError

error-set

Source line 1598

Errors

EndOfStream Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.Reader.peekNextTokenType

function

Source line 1723

See std.json.Scanner.peekNextTokenType().

pub fn peekNextTokenType(self: *@This()) Reader.PeekError!TokenType

Parameters

self: *@This()

Errors

EndOfStream Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.Reader.refillBuffer

function

Source line 1735

fn refillBuffer(self: *@This()) std.Io.Reader.Error!void

Parameters

self: *@This()

Errors

EndOfStream
ReadFailed

See the Reader implementation for detailed diagnostics.

std.json.Scanner.Reader.SkipError

alias

Source line 1596

Alias of std.json.Scanner.Reader.NextError

std.json.Scanner.Reader.skipUntilStackHeight

function

Source line 1688

Like std.json.Scanner.skipUntilStackHeight() but handles error.BufferUnderrun.

pub fn skipUntilStackHeight(self: *@This(), terminal_stack_height: usize) Reader.NextError!void

Parameters

self: *@This()
terminal_stack_height: usize

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.Reader.skipValue

function

Source line 1658

Like std.json.Scanner.skipValue, but handles error.BufferUnderrun.

pub fn skipValue(self: *@This()) Reader.SkipError!void

Parameters

self: *@This()

Errors

EndOfStream Error
OutOfMemory Error
ReadFailed Error

See the Reader implementation for detailed diagnostics.

SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.Reader.stackHeight

function

Source line 1701

Calls std.json.Scanner.stackHeight.

pub fn stackHeight(self: *const @This()) usize

Parameters

self: *const @This()

std.json.Scanner.Scanner

alias

Source line 45

Alias of std.json.Scanner

std.json.Scanner.SkipError

error-set

Source line 118

Errors

OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.skipUntilStackHeight

function

Source line 297

Skip tokens until an .object_end or .array_end token results in a stackHeight() equal the given stack height. Unlike skipValue(), this function is available in streaming mode.

pub fn skipUntilStackHeight(self: *@This(), terminal_stack_height: usize) NextError!void

Parameters

self: *@This()
terminal_stack_height: usize

Errors

BufferUnderrun
OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.skipValue

function

Source line 252

This function is only available after endInput() (or initCompleteInput()) has been called. If the next token type is .object_begin or .array_begin, this function calls next() repeatedly until the corresponding .object_end or .array_end is found. If the next token type is .number or .string, this function calls next() repeatedly until the (non .partial_*) .number or .string token is found. If the next token type is .true, .false, or .null, this function calls next() once. The next token type must not be .object_end, .array_end, or .end_of_document; see peekNextTokenType().

pub fn skipValue(self: *@This()) SkipError!void

Parameters

self: *@This()

Errors

OutOfMemory Error
SyntaxError Error
UnexpectedEndOfInput Error

std.json.Scanner.skipWhitespace

function

Source line 1279

fn skipWhitespace(self: *@This()) void

Parameters

self: *@This()

std.json.Scanner.skipWhitespaceCheckEnd

function

Source line 1303

fn skipWhitespaceCheckEnd(self: *@This()) !bool

Parameters

self: *@This()

std.json.Scanner.skipWhitespaceExpectByte

function

Source line 1298

fn skipWhitespaceExpectByte(self: *@This()) !u8

Parameters

self: *@This()

std.json.Scanner.stackHeight

function

Source line 310

The depth of {} or [] nesting levels at the current position.

pub fn stackHeight(self: *const @This()) usize

Parameters

self: *const @This()

std.json.Scanner.State

container

Source line 1218

Fields

value
post_value
object_start
object_post_comma
array_start
number_minus
number_leading_zero
number_int
number_post_dot
number_frac
number_post_e
number_post_e_sign
number_exp
string
string_backslash
string_backslash_u
string_backslash_u_1
string_backslash_u_2
string_backslash_u_3
string_surrogate_half
string_surrogate_half_backslash
string_surrogate_half_backslash_u
string_surrogate_half_backslash_u_1
string_surrogate_half_backslash_u_2
string_surrogate_half_backslash_u_3
string_utf8_last_byte
string_utf8_second_to_last_byte
string_utf8_second_to_last_byte_guard_against_overlong
string_utf8_second_to_last_byte_guard_against_surrogate_half
string_utf8_third_to_last_byte
string_utf8_third_to_last_byte_guard_against_overlong
string_utf8_third_to_last_byte_guard_against_too_large
literal_t
literal_tr
literal_tru
literal_f
literal_fa
literal_fal
literal_fals
literal_n
literal_nu
literal_nul

std.json.Scanner.std

alias

Source line 46

Alias of std

std.json.Scanner.takeValueSlice

function

Source line 1321

fn takeValueSlice(self: *@This()) []const u8

Parameters

self: *@This()

std.json.Scanner.takeValueSliceMinusTrailingOffset

function

Source line 1326

fn takeValueSliceMinusTrailingOffset(self: *@This(), trailing_negative_offset: usize) []const u8

Parameters

self: *@This()
trailing_negative_offset: usize

std.json.Scanner.Token

container

Source line 1503

The tokens emitted by std.json.Scanner and std.json.Reader .next*() functions follow this grammar:

 <document> = <value> .end_of_document
 <value> =
   | <object>
   | <array>
   | <number>
   | <string>
   | .true
   | .false
   | .null
 <object> = .object_begin ( <string> <value> )* .object_end
 <array> = .array_begin ( <value> )* .array_end
 <number> = <It depends. See below.>
 <string> = <It depends. See below.>

What you get for <number> and <string> values depends on which next*() method you call:

next():
 <number> = ( .partial_number )* .number
 <string> = ( <partial_string> )* .string
 <partial_string> =
   | .partial_string
   | .partial_string_escaped_1
   | .partial_string_escaped_2
   | .partial_string_escaped_3
   | .partial_string_escaped_4

nextAlloc*(..., .alloc_always):
 <number> = .allocated_number
 <string> = .allocated_string

nextAlloc*(..., .alloc_if_needed):
 <number> =
   | .number
   | .allocated_number
 <string> =
   | .string
   | .allocated_string

For all tokens with a []const u8, []u8, or [n]u8 payload, the payload represents the content of the value. For number values, this is the representation of the number exactly as it appears in the input. For strings, this is the content of the string after resolving escape sequences.

For .allocated_number and .allocated_string, the []u8 payloads are allocations made with the given allocator. You are responsible for managing that memory. json.Reader.deinit() does not free those allocations.

The .partial_* tokens indicate that a value spans multiple input buffers or that a string contains escape sequences. To get a complete value in memory, you need to concatenate the values yourself. Calling nextAlloc*() does this for you, and returns an .allocated_* token with the result.

For tokens with a []const u8 payload, the payload is a slice into the current input buffer. The memory may become undefined during the next call to json.Scanner.feedInput() or any json.Reader method whose return error set includes json.Error. To keep the value persistently, it recommended to make a copy or to use .alloc_always, which makes a copy for you.

Note that .number and .string tokens that follow .partial_* tokens may have 0 length to indicate that the previously partial value is completed with no additional bytes. (This can happen when the break between input buffers happens to land on the exact end of a value. E.g. "[1234", "]".) .partial_* tokens never have 0 length.

The recommended strategy for using the different next*() methods is something like this:

When you're expecting an object key, use .alloc_if_needed. You often don't need a copy of the key string to persist; you might just check which field it is. In the case that the key happens to require an allocation, free it immediately after checking it.

When you're expecting a meaningful string value (such as on the right of a :), use .alloc_always in order to keep the value valid throughout parsing the rest of the document.

When you're expecting a number value, use .alloc_if_needed. You're probably going to be parsing the string representation of the number into a numeric representation, so you need the complete string representation only temporarily.

When you're skipping an unrecognized value, use skipValue().

Fields

object_begin
object_end
array_begin
array_end
true
false
null
number: []const u8
partial_number: []const u8
allocated_number: []u8
string: []const u8
partial_string: []const u8
partial_string_escaped_1: [1]u8
partial_string_escaped_2: [2]u8
partial_string_escaped_3: [3]u8
partial_string_escaped_4: [4]u8
allocated_string: []u8
end_of_document

std.json.Scanner.TokenType

container

Source line 1529

This is only used in peekNextTokenType() and gives a categorization based on the first byte of the next token that will be emitted from a next*() call.

Fields

object_begin
object_end
array_begin
array_end
true
false
null
number
string
end_of_document

std.json.Scanner.validate

function

Source line 1397

Scan the input and check for malformed JSON. On SyntaxError or UnexpectedEndOfInput, returns false. Returns any errors from the allocator as-is, which is unlikely, but can be caused by extreme nesting depth in the input.

pub fn validate(allocator: Allocator, s: []const u8) Allocator.Error!bool

Parameters

allocator: Allocator
s: []const u8

Errors

OutOfMemory
Source code

std/json/Scanner.zig

//! The lowest level parsing API in this package;
//! supports streaming input with a low memory footprint.
//! The memory requirement is `O(d)` where d is the nesting depth of `[]` or `{}` containers in the input.
//! Specifically `d/8` bytes are required for this purpose,
//! with some extra buffer according to the implementation of `std.ArrayList`.
//!
//! This scanner can emit partial tokens; see `std.json.Token`.
//! The input to this class is a sequence of input buffers that you must supply one at a time.
//! Call `feedInput()` with the first buffer, then call `next()` repeatedly until `error.BufferUnderrun` is returned.
//! Then call `feedInput()` again and so forth.
//! Call `endInput()` when the last input buffer has been given to `feedInput()`, either immediately after calling `feedInput()`,
//! or when `error.BufferUnderrun` requests more data and there is no more.
//! Be sure to call `next()` after calling `endInput()` until `Token.end_of_document` has been returned.
//!
//! Notes on standards compliance: https://datatracker.ietf.org/doc/html/rfc8259
//! * RFC 8259 requires JSON documents be valid UTF-8,
//!   but makes an allowance for systems that are "part of a closed ecosystem".
//!   I have no idea what that's supposed to mean in the context of a standard specification.
//!   This implementation requires inputs to be valid UTF-8.
//! * RFC 8259 contradicts itself regarding whether lowercase is allowed in \u hex digits,
//!   but this is probably a bug in the spec, and it's clear that lowercase is meant to be allowed.
//!   (RFC 5234 defines HEXDIG to only allow uppercase.)
//! * When RFC 8259 refers to a "character", I assume they really mean a "Unicode scalar value".
//!   See http://www.unicode.org/glossary/#unicode_scalar_value .
//! * RFC 8259 doesn't explicitly disallow unpaired surrogate halves in \u escape sequences,
//!   but vaguely implies that \u escapes are for encoding Unicode "characters" (i.e. Unicode scalar values?),
//!   which would mean that unpaired surrogate halves are forbidden.
//!   By contrast ECMA-404 (a competing(/compatible?) JSON standard, which JavaScript's JSON.parse() conforms to)
//!   explicitly allows unpaired surrogate halves.
//!   This implementation forbids unpaired surrogate halves in \u sequences.
//!   If a high surrogate half appears in a \u sequence,
//!   then a low surrogate half must immediately follow in \u notation.
//! * RFC 8259 allows implementations to "accept non-JSON forms or extensions".
//!   This implementation does not accept any of that.
//! * RFC 8259 allows implementations to put limits on "the size of texts",
//!   "the maximum depth of nesting", "the range and precision of numbers",
//!   and "the length and character contents of strings".
//!   This low-level implementation does not limit these,
//!   except where noted above, and except that nesting depth requires memory allocation.
//!   Note that this low-level API does not interpret numbers numerically,
//!   but simply emits their source form for some higher level code to make sense of.
//! * This low-level implementation allows duplicate object keys,
//!   and key/value pairs are emitted in the order they appear in the input.

const Scanner = @This();
const std = @import("std");

const Allocator = std.mem.Allocator;
const assert = std.debug.assert;
const BitStack = std.BitStack;

state: State = .value,
string_is_object_key: bool = false,
stack: BitStack,
value_start: usize = undefined,
utf16_code_units: [2]u16 = undefined,

input: []const u8 = "",
cursor: usize = 0,
is_end_of_input: bool = false,
diagnostics: ?*Diagnostics = null,

/// The allocator is only used to track `[]` and `{}` nesting levels.
pub fn initStreaming(allocator: Allocator) @This() {
    return .{
        .stack = BitStack.init(allocator),
    };
}
/// Use this if your input is a single slice.
/// This is effectively equivalent to:
/// ```
/// initStreaming(allocator);
/// feedInput(complete_input);
/// endInput();
/// ```
pub fn initCompleteInput(allocator: Allocator, complete_input: []const u8) @This() {
    return .{
        .stack = BitStack.init(allocator),
        .input = complete_input,
        .is_end_of_input = true,
    };
}
pub fn deinit(self: *@This()) void {
    self.stack.deinit();
    self.* = undefined;
}

pub fn enableDiagnostics(self: *@This(), diagnostics: *Diagnostics) void {
    diagnostics.cursor_pointer = &self.cursor;
    self.diagnostics = diagnostics;
}

/// Call this whenever you get `error.BufferUnderrun` from `next()`.
/// When there is no more input to provide, call `endInput()`.
pub fn feedInput(self: *@This(), input: []const u8) void {
    assert(self.cursor == self.input.len); // Not done with the last input slice.
    if (self.diagnostics) |diag| {
        diag.total_bytes_before_current_input += self.input.len;
        // This usually goes "negative" to measure how far before the beginning
        // of the new buffer the current line started.
        diag.line_start_cursor -%= self.cursor;
    }
    self.input = input;
    self.cursor = 0;
    self.value_start = 0;
}
/// Call this when you will no longer call `feedInput()` anymore.
/// This can be called either immediately after the last `feedInput()`,
/// or at any time afterward, such as when getting `error.BufferUnderrun` from `next()`.
/// Don't forget to call `next*()` after `endInput()` until you get `.end_of_document`.
pub fn endInput(self: *@This()) void {
    self.is_end_of_input = true;
}

pub const NextError = Error || Allocator.Error || error{BufferUnderrun};
pub const AllocError = Error || Allocator.Error || error{ValueTooLong};
pub const PeekError = Error || error{BufferUnderrun};
pub const SkipError = Error || Allocator.Error;
pub const AllocIntoArrayListError = AllocError || error{BufferUnderrun};

/// Equivalent to `nextAllocMax(allocator, when, default_max_value_len);`
/// This function is only available after `endInput()` (or `initCompleteInput()`) has been called.
/// See also `std.json.Token` for documentation of `nextAlloc*()` function behavior.
pub fn nextAlloc(self: *@This(), allocator: Allocator, when: AllocWhen) AllocError!Token {
    return self.nextAllocMax(allocator, when, default_max_value_len);
}

/// This function is only available after `endInput()` (or `initCompleteInput()`) has been called.
/// See also `std.json.Token` for documentation of `nextAlloc*()` function behavior.
pub fn nextAllocMax(self: *@This(), allocator: Allocator, when: AllocWhen, max_value_len: usize) AllocError!Token {
    assert(self.is_end_of_input); // This function is not available in streaming mode.
    const token_type = self.peekNextTokenType() catch |e| switch (e) {
        error.BufferUnderrun => unreachable,
        else => |err| return err,
    };
    switch (token_type) {
        .number, .string => {
            var value_list = std.array_list.Managed(u8).init(allocator);
            errdefer {
                value_list.deinit();
            }
            if (self.allocNextIntoArrayListMax(&value_list, when, max_value_len) catch |e| switch (e) {
                error.BufferUnderrun => unreachable,
                else => |err| return err,
            }) |slice| {
                return if (token_type == .number)
                    Token{ .number = slice }
                else
                    Token{ .string = slice };
            } else {
                return if (token_type == .number)
                    Token{ .allocated_number = try value_list.toOwnedSlice() }
                else
                    Token{ .allocated_string = try value_list.toOwnedSlice() };
            }
        },

        // Simple tokens never alloc.
        .object_begin,
        .object_end,
        .array_begin,
        .array_end,
        .true,
        .false,
        .null,
        .end_of_document,
        => return self.next() catch |e| switch (e) {
            error.BufferUnderrun => unreachable,
            else => |err| return err,
        },
    }
}

/// Equivalent to `allocNextIntoArrayListMax(value_list, when, default_max_value_len);`
pub fn allocNextIntoArrayList(self: *@This(), value_list: *std.array_list.Managed(u8), when: AllocWhen) AllocIntoArrayListError!?[]const u8 {
    return self.allocNextIntoArrayListMax(value_list, when, default_max_value_len);
}
/// The next token type must be either `.number` or `.string`. See `peekNextTokenType()`.
/// When allocation is not necessary with `.alloc_if_needed`,
/// this method returns the content slice from the input buffer, and `value_list` is not touched.
/// When allocation is necessary or with `.alloc_always`, this method concatenates partial tokens into the given `value_list`,
/// and returns `null` once the final `.number` or `.string` token has been written into it.
/// In case of an `error.BufferUnderrun`, partial values will be left in the given value_list.
/// The given `value_list` is never reset by this method, so an `error.BufferUnderrun` situation
/// can be resumed by passing the same array list in again.
/// This method does not indicate whether the token content being returned is for a `.number` or `.string` token type;
/// the caller of this method is expected to know which type of token is being processed.
pub fn allocNextIntoArrayListMax(self: *@This(), value_list: *std.array_list.Managed(u8), when: AllocWhen, max_value_len: usize) AllocIntoArrayListError!?[]const u8 {
    while (true) {
        const token = try self.next();
        switch (token) {
            // Accumulate partial values.
            .partial_number, .partial_string => |slice| {
                try appendSlice(value_list, slice, max_value_len);
            },
            .partial_string_escaped_1 => |buf| {
                try appendSlice(value_list, buf[0..], max_value_len);
            },
            .partial_string_escaped_2 => |buf| {
                try appendSlice(value_list, buf[0..], max_value_len);
            },
            .partial_string_escaped_3 => |buf| {
                try appendSlice(value_list, buf[0..], max_value_len);
            },
            .partial_string_escaped_4 => |buf| {
                try appendSlice(value_list, buf[0..], max_value_len);
            },

            // Return complete values.
            .number => |slice| {
                if (when == .alloc_if_needed and value_list.items.len == 0) {
                    // No alloc necessary.
                    return slice;
                }
                try appendSlice(value_list, slice, max_value_len);
                // The token is complete.
                return null;
            },
            .string => |slice| {
                if (when == .alloc_if_needed and value_list.items.len == 0) {
                    // No alloc necessary.
                    return slice;
                }
                try appendSlice(value_list, slice, max_value_len);
                // The token is complete.
                return null;
            },

            .object_begin,
            .object_end,
            .array_begin,
            .array_end,
            .true,
            .false,
            .null,
            .end_of_document,
            => unreachable, // Only .number and .string token types are allowed here. Check peekNextTokenType() before calling this.

            .allocated_number, .allocated_string => unreachable,
        }
    }
}

/// This function is only available after `endInput()` (or `initCompleteInput()`) has been called.
/// If the next token type is `.object_begin` or `.array_begin`,
/// this function calls `next()` repeatedly until the corresponding `.object_end` or `.array_end` is found.
/// If the next token type is `.number` or `.string`,
/// this function calls `next()` repeatedly until the (non `.partial_*`) `.number` or `.string` token is found.
/// If the next token type is `.true`, `.false`, or `.null`, this function calls `next()` once.
/// The next token type must not be `.object_end`, `.array_end`, or `.end_of_document`;
/// see `peekNextTokenType()`.
pub fn skipValue(self: *@This()) SkipError!void {
    assert(self.is_end_of_input); // This function is not available in streaming mode.
    switch (self.peekNextTokenType() catch |e| switch (e) {
        error.BufferUnderrun => unreachable,
        else => |err| return err,
    }) {
        .object_begin, .array_begin => {
            self.skipUntilStackHeight(self.stackHeight()) catch |e| switch (e) {
                error.BufferUnderrun => unreachable,
                else => |err| return err,
            };
        },
        .number, .string => {
            while (true) {
                switch (self.next() catch |e| switch (e) {
                    error.BufferUnderrun => unreachable,
                    else => |err| return err,
                }) {
                    .partial_number,
                    .partial_string,
                    .partial_string_escaped_1,
                    .partial_string_escaped_2,
                    .partial_string_escaped_3,
                    .partial_string_escaped_4,
                    => continue,

                    .number, .string => break,

                    else => unreachable,
                }
            }
        },
        .true, .false, .null => {
            _ = self.next() catch |e| switch (e) {
                error.BufferUnderrun => unreachable,
                else => |err| return err,
            };
        },

        .object_end, .array_end, .end_of_document => unreachable, // Attempt to skip a non-value token.
    }
}

/// Skip tokens until an `.object_end` or `.array_end` token results in a `stackHeight()` equal the given stack height.
/// Unlike `skipValue()`, this function is available in streaming mode.
pub fn skipUntilStackHeight(self: *@This(), terminal_stack_height: usize) NextError!void {
    while (true) {
        switch (try self.next()) {
            .object_end, .array_end => {
                if (self.stackHeight() == terminal_stack_height) break;
            },
            .end_of_document => unreachable,
            else => continue,
        }
    }
}

/// The depth of `{}` or `[]` nesting levels at the current position.
pub fn stackHeight(self: *const @This()) usize {
    return self.stack.bit_len;
}

/// Pre allocate memory to hold the given number of nesting levels.
/// `stackHeight()` up to the given number will not cause allocations.
pub fn ensureTotalStackCapacity(self: *@This(), height: usize) Allocator.Error!void {
    try self.stack.ensureTotalCapacity(height);
}

/// See `std.json.Token` for documentation of this function.
pub fn next(self: *@This()) NextError!Token {
    state_loop: while (true) {
        switch (self.state) {
            .value => {
                switch (try self.skipWhitespaceExpectByte()) {
                    // Object, Array
                    '{' => {
                        try self.stack.push(OBJECT_MODE);
                        self.cursor += 1;
                        self.state = .object_start;
                        return .object_begin;
                    },
                    '[' => {
                        try self.stack.push(ARRAY_MODE);
                        self.cursor += 1;
                        self.state = .array_start;
                        return .array_begin;
                    },

                    // String
                    '"' => {
                        self.cursor += 1;
                        self.value_start = self.cursor;
                        self.state = .string;
                        continue :state_loop;
                    },

                    // Number
                    '1'...'9' => {
                        self.value_start = self.cursor;
                        self.cursor += 1;
                        self.state = .number_int;
                        continue :state_loop;
                    },
                    '0' => {
                        self.value_start = self.cursor;
                        self.cursor += 1;
                        self.state = .number_leading_zero;
                        continue :state_loop;
                    },
                    '-' => {
                        self.value_start = self.cursor;
                        self.cursor += 1;
                        self.state = .number_minus;
                        continue :state_loop;
                    },

                    // literal values
                    't' => {
                        self.cursor += 1;
                        self.state = .literal_t;
                        continue :state_loop;
                    },
                    'f' => {
                        self.cursor += 1;
                        self.state = .literal_f;
                        continue :state_loop;
                    },
                    'n' => {
                        self.cursor += 1;
                        self.state = .literal_n;
                        continue :state_loop;
                    },

                    else => return error.SyntaxError,
                }
            },

            .post_value => {
                if (try self.skipWhitespaceCheckEnd()) return .end_of_document;

                const c = self.input[self.cursor];
                if (self.string_is_object_key) {
                    self.string_is_object_key = false;
                    switch (c) {
                        ':' => {
                            self.cursor += 1;
                            self.state = .value;
                            continue :state_loop;
                        },
                        else => return error.SyntaxError,
                    }
                }

                switch (c) {
                    '}' => {
                        if (self.stack.pop() != OBJECT_MODE) return error.SyntaxError;
                        self.cursor += 1;
                        // stay in .post_value state.
                        return .object_end;
                    },
                    ']' => {
                        if (self.stack.pop() != ARRAY_MODE) return error.SyntaxError;
                        self.cursor += 1;
                        // stay in .post_value state.
                        return .array_end;
                    },
                    ',' => {
                        switch (self.stack.peek()) {
                            OBJECT_MODE => {
                                self.state = .object_post_comma;
                            },
                            ARRAY_MODE => {
                                self.state = .value;
                            },
                        }
                        self.cursor += 1;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },

            .object_start => {
                switch (try self.skipWhitespaceExpectByte()) {
                    '"' => {
                        self.cursor += 1;
                        self.value_start = self.cursor;
                        self.state = .string;
                        self.string_is_object_key = true;
                        continue :state_loop;
                    },
                    '}' => {
                        self.cursor += 1;
                        _ = self.stack.pop();
                        self.state = .post_value;
                        return .object_end;
                    },
                    else => return error.SyntaxError,
                }
            },
            .object_post_comma => {
                switch (try self.skipWhitespaceExpectByte()) {
                    '"' => {
                        self.cursor += 1;
                        self.value_start = self.cursor;
                        self.state = .string;
                        self.string_is_object_key = true;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },

            .array_start => {
                switch (try self.skipWhitespaceExpectByte()) {
                    ']' => {
                        self.cursor += 1;
                        _ = self.stack.pop();
                        self.state = .post_value;
                        return .array_end;
                    },
                    else => {
                        self.state = .value;
                        continue :state_loop;
                    },
                }
            },

            .number_minus => {
                if (self.cursor >= self.input.len) return self.endOfBufferInNumber(false);
                switch (self.input[self.cursor]) {
                    '0' => {
                        self.cursor += 1;
                        self.state = .number_leading_zero;
                        continue :state_loop;
                    },
                    '1'...'9' => {
                        self.cursor += 1;
                        self.state = .number_int;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .number_leading_zero => {
                if (self.cursor >= self.input.len) return self.endOfBufferInNumber(true);
                switch (self.input[self.cursor]) {
                    '.' => {
                        self.cursor += 1;
                        self.state = .number_post_dot;
                        continue :state_loop;
                    },
                    'e', 'E' => {
                        self.cursor += 1;
                        self.state = .number_post_e;
                        continue :state_loop;
                    },
                    else => {
                        self.state = .post_value;
                        return Token{ .number = self.takeValueSlice() };
                    },
                }
            },
            .number_int => {
                while (self.cursor < self.input.len) : (self.cursor += 1) {
                    switch (self.input[self.cursor]) {
                        '0'...'9' => continue,
                        '.' => {
                            self.cursor += 1;
                            self.state = .number_post_dot;
                            continue :state_loop;
                        },
                        'e', 'E' => {
                            self.cursor += 1;
                            self.state = .number_post_e;
                            continue :state_loop;
                        },
                        else => {
                            self.state = .post_value;
                            return Token{ .number = self.takeValueSlice() };
                        },
                    }
                }
                return self.endOfBufferInNumber(true);
            },
            .number_post_dot => {
                if (self.cursor >= self.input.len) return self.endOfBufferInNumber(false);
                switch (self.input[self.cursor]) {
                    '0'...'9' => {
                        self.cursor += 1;
                        self.state = .number_frac;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .number_frac => {
                while (self.cursor < self.input.len) : (self.cursor += 1) {
                    switch (self.input[self.cursor]) {
                        '0'...'9' => continue,
                        'e', 'E' => {
                            self.cursor += 1;
                            self.state = .number_post_e;
                            continue :state_loop;
                        },
                        else => {
                            self.state = .post_value;
                            return Token{ .number = self.takeValueSlice() };
                        },
                    }
                }
                return self.endOfBufferInNumber(true);
            },
            .number_post_e => {
                if (self.cursor >= self.input.len) return self.endOfBufferInNumber(false);
                switch (self.input[self.cursor]) {
                    '0'...'9' => {
                        self.cursor += 1;
                        self.state = .number_exp;
                        continue :state_loop;
                    },
                    '+', '-' => {
                        self.cursor += 1;
                        self.state = .number_post_e_sign;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .number_post_e_sign => {
                if (self.cursor >= self.input.len) return self.endOfBufferInNumber(false);
                switch (self.input[self.cursor]) {
                    '0'...'9' => {
                        self.cursor += 1;
                        self.state = .number_exp;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .number_exp => {
                while (self.cursor < self.input.len) : (self.cursor += 1) {
                    switch (self.input[self.cursor]) {
                        '0'...'9' => continue,
                        else => {
                            self.state = .post_value;
                            return Token{ .number = self.takeValueSlice() };
                        },
                    }
                }
                return self.endOfBufferInNumber(true);
            },

            .string => {
                while (self.cursor < self.input.len) : (self.cursor += 1) {
                    switch (self.input[self.cursor]) {
                        0...0x1f => return error.SyntaxError, // Bare ASCII control code in string.

                        // ASCII plain text.
                        0x20...('"' - 1), ('"' + 1)...('\\' - 1), ('\\' + 1)...0x7F => continue,

                        // Special characters.
                        '"' => {
                            const result = Token{ .string = self.takeValueSlice() };
                            self.cursor += 1;
                            self.state = .post_value;
                            return result;
                        },
                        '\\' => {
                            const slice = self.takeValueSlice();
                            self.cursor += 1;
                            self.state = .string_backslash;
                            if (slice.len > 0) return Token{ .partial_string = slice };
                            continue :state_loop;
                        },

                        // UTF-8 validation.
                        // See http://unicode.org/mail-arch/unicode-ml/y2003-m02/att-0467/01-The_Algorithm_to_Valide_an_UTF-8_String
                        0xC2...0xDF => {
                            self.cursor += 1;
                            self.state = .string_utf8_last_byte;
                            continue :state_loop;
                        },
                        0xE0 => {
                            self.cursor += 1;
                            self.state = .string_utf8_second_to_last_byte_guard_against_overlong;
                            continue :state_loop;
                        },
                        0xE1...0xEC, 0xEE...0xEF => {
                            self.cursor += 1;
                            self.state = .string_utf8_second_to_last_byte;
                            continue :state_loop;
                        },
                        0xED => {
                            self.cursor += 1;
                            self.state = .string_utf8_second_to_last_byte_guard_against_surrogate_half;
                            continue :state_loop;
                        },
                        0xF0 => {
                            self.cursor += 1;
                            self.state = .string_utf8_third_to_last_byte_guard_against_overlong;
                            continue :state_loop;
                        },
                        0xF1...0xF3 => {
                            self.cursor += 1;
                            self.state = .string_utf8_third_to_last_byte;
                            continue :state_loop;
                        },
                        0xF4 => {
                            self.cursor += 1;
                            self.state = .string_utf8_third_to_last_byte_guard_against_too_large;
                            continue :state_loop;
                        },
                        0x80...0xC1, 0xF5...0xFF => return error.SyntaxError, // Invalid UTF-8.
                    }
                }
                if (self.is_end_of_input) return error.UnexpectedEndOfInput;
                const slice = self.takeValueSlice();
                if (slice.len > 0) return Token{ .partial_string = slice };
                return error.BufferUnderrun;
            },
            .string_backslash => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    '"', '\\', '/' => {
                        // Since these characters now represent themselves literally,
                        // we can simply begin the next plaintext slice here.
                        self.value_start = self.cursor;
                        self.cursor += 1;
                        self.state = .string;
                        continue :state_loop;
                    },
                    'b' => {
                        self.cursor += 1;
                        self.value_start = self.cursor;
                        self.state = .string;
                        return Token{ .partial_string_escaped_1 = [_]u8{0x08} };
                    },
                    'f' => {
                        self.cursor += 1;
                        self.value_start = self.cursor;
                        self.state = .string;
                        return Token{ .partial_string_escaped_1 = [_]u8{0x0c} };
                    },
                    'n' => {
                        self.cursor += 1;
                        self.value_start = self.cursor;
                        self.state = .string;
                        return Token{ .partial_string_escaped_1 = [_]u8{'\n'} };
                    },
                    'r' => {
                        self.cursor += 1;
                        self.value_start = self.cursor;
                        self.state = .string;
                        return Token{ .partial_string_escaped_1 = [_]u8{'\r'} };
                    },
                    't' => {
                        self.cursor += 1;
                        self.value_start = self.cursor;
                        self.state = .string;
                        return Token{ .partial_string_escaped_1 = [_]u8{'\t'} };
                    },
                    'u' => {
                        self.cursor += 1;
                        self.state = .string_backslash_u;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .string_backslash_u => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                const c = self.input[self.cursor];
                switch (c) {
                    '0'...'9' => {
                        self.utf16_code_units[0] = @as(u16, c - '0') << 12;
                    },
                    'A'...'F' => {
                        self.utf16_code_units[0] = @as(u16, c - 'A' + 10) << 12;
                    },
                    'a'...'f' => {
                        self.utf16_code_units[0] = @as(u16, c - 'a' + 10) << 12;
                    },
                    else => return error.SyntaxError,
                }
                self.cursor += 1;
                self.state = .string_backslash_u_1;
                continue :state_loop;
            },
            .string_backslash_u_1 => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                const c = self.input[self.cursor];
                switch (c) {
                    '0'...'9' => {
                        self.utf16_code_units[0] |= @as(u16, c - '0') << 8;
                    },
                    'A'...'F' => {
                        self.utf16_code_units[0] |= @as(u16, c - 'A' + 10) << 8;
                    },
                    'a'...'f' => {
                        self.utf16_code_units[0] |= @as(u16, c - 'a' + 10) << 8;
                    },
                    else => return error.SyntaxError,
                }
                self.cursor += 1;
                self.state = .string_backslash_u_2;
                continue :state_loop;
            },
            .string_backslash_u_2 => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                const c = self.input[self.cursor];
                switch (c) {
                    '0'...'9' => {
                        self.utf16_code_units[0] |= @as(u16, c - '0') << 4;
                    },
                    'A'...'F' => {
                        self.utf16_code_units[0] |= @as(u16, c - 'A' + 10) << 4;
                    },
                    'a'...'f' => {
                        self.utf16_code_units[0] |= @as(u16, c - 'a' + 10) << 4;
                    },
                    else => return error.SyntaxError,
                }
                self.cursor += 1;
                self.state = .string_backslash_u_3;
                continue :state_loop;
            },
            .string_backslash_u_3 => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                const c = self.input[self.cursor];
                switch (c) {
                    '0'...'9' => {
                        self.utf16_code_units[0] |= c - '0';
                    },
                    'A'...'F' => {
                        self.utf16_code_units[0] |= c - 'A' + 10;
                    },
                    'a'...'f' => {
                        self.utf16_code_units[0] |= c - 'a' + 10;
                    },
                    else => return error.SyntaxError,
                }
                self.cursor += 1;
                if (std.unicode.utf16IsHighSurrogate(self.utf16_code_units[0])) {
                    self.state = .string_surrogate_half;
                    continue :state_loop;
                } else if (std.unicode.utf16IsLowSurrogate(self.utf16_code_units[0])) {
                    return error.SyntaxError; // Unexpected low surrogate half.
                } else {
                    self.value_start = self.cursor;
                    self.state = .string;
                    return partialStringCodepoint(self.utf16_code_units[0]);
                }
            },
            .string_surrogate_half => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    '\\' => {
                        self.cursor += 1;
                        self.state = .string_surrogate_half_backslash;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Expected low surrogate half.
                }
            },
            .string_surrogate_half_backslash => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    'u' => {
                        self.cursor += 1;
                        self.state = .string_surrogate_half_backslash_u;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Expected low surrogate half.
                }
            },
            .string_surrogate_half_backslash_u => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    'D', 'd' => {
                        self.cursor += 1;
                        self.utf16_code_units[1] = 0xD << 12;
                        self.state = .string_surrogate_half_backslash_u_1;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Expected low surrogate half.
                }
            },
            .string_surrogate_half_backslash_u_1 => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                const c = self.input[self.cursor];
                switch (c) {
                    'C'...'F' => {
                        self.cursor += 1;
                        self.utf16_code_units[1] |= @as(u16, c - 'A' + 10) << 8;
                        self.state = .string_surrogate_half_backslash_u_2;
                        continue :state_loop;
                    },
                    'c'...'f' => {
                        self.cursor += 1;
                        self.utf16_code_units[1] |= @as(u16, c - 'a' + 10) << 8;
                        self.state = .string_surrogate_half_backslash_u_2;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Expected low surrogate half.
                }
            },
            .string_surrogate_half_backslash_u_2 => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                const c = self.input[self.cursor];
                switch (c) {
                    '0'...'9' => {
                        self.cursor += 1;
                        self.utf16_code_units[1] |= @as(u16, c - '0') << 4;
                        self.state = .string_surrogate_half_backslash_u_3;
                        continue :state_loop;
                    },
                    'A'...'F' => {
                        self.cursor += 1;
                        self.utf16_code_units[1] |= @as(u16, c - 'A' + 10) << 4;
                        self.state = .string_surrogate_half_backslash_u_3;
                        continue :state_loop;
                    },
                    'a'...'f' => {
                        self.cursor += 1;
                        self.utf16_code_units[1] |= @as(u16, c - 'a' + 10) << 4;
                        self.state = .string_surrogate_half_backslash_u_3;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .string_surrogate_half_backslash_u_3 => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                const c = self.input[self.cursor];
                switch (c) {
                    '0'...'9' => {
                        self.utf16_code_units[1] |= c - '0';
                    },
                    'A'...'F' => {
                        self.utf16_code_units[1] |= c - 'A' + 10;
                    },
                    'a'...'f' => {
                        self.utf16_code_units[1] |= c - 'a' + 10;
                    },
                    else => return error.SyntaxError,
                }
                self.cursor += 1;
                self.value_start = self.cursor;
                self.state = .string;
                const code_point = std.unicode.utf16DecodeSurrogatePair(&self.utf16_code_units) catch unreachable;
                return partialStringCodepoint(code_point);
            },

            .string_utf8_last_byte => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    0x80...0xBF => {
                        self.cursor += 1;
                        self.state = .string;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Invalid UTF-8.
                }
            },
            .string_utf8_second_to_last_byte => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    0x80...0xBF => {
                        self.cursor += 1;
                        self.state = .string_utf8_last_byte;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Invalid UTF-8.
                }
            },
            .string_utf8_second_to_last_byte_guard_against_overlong => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    0xA0...0xBF => {
                        self.cursor += 1;
                        self.state = .string_utf8_last_byte;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Invalid UTF-8.
                }
            },
            .string_utf8_second_to_last_byte_guard_against_surrogate_half => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    0x80...0x9F => {
                        self.cursor += 1;
                        self.state = .string_utf8_last_byte;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Invalid UTF-8.
                }
            },
            .string_utf8_third_to_last_byte => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    0x80...0xBF => {
                        self.cursor += 1;
                        self.state = .string_utf8_second_to_last_byte;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Invalid UTF-8.
                }
            },
            .string_utf8_third_to_last_byte_guard_against_overlong => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    0x90...0xBF => {
                        self.cursor += 1;
                        self.state = .string_utf8_second_to_last_byte;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Invalid UTF-8.
                }
            },
            .string_utf8_third_to_last_byte_guard_against_too_large => {
                if (self.cursor >= self.input.len) return self.endOfBufferInString();
                switch (self.input[self.cursor]) {
                    0x80...0x8F => {
                        self.cursor += 1;
                        self.state = .string_utf8_second_to_last_byte;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError, // Invalid UTF-8.
                }
            },

            .literal_t => {
                switch (try self.expectByte()) {
                    'r' => {
                        self.cursor += 1;
                        self.state = .literal_tr;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_tr => {
                switch (try self.expectByte()) {
                    'u' => {
                        self.cursor += 1;
                        self.state = .literal_tru;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_tru => {
                switch (try self.expectByte()) {
                    'e' => {
                        self.cursor += 1;
                        self.state = .post_value;
                        return .true;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_f => {
                switch (try self.expectByte()) {
                    'a' => {
                        self.cursor += 1;
                        self.state = .literal_fa;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_fa => {
                switch (try self.expectByte()) {
                    'l' => {
                        self.cursor += 1;
                        self.state = .literal_fal;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_fal => {
                switch (try self.expectByte()) {
                    's' => {
                        self.cursor += 1;
                        self.state = .literal_fals;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_fals => {
                switch (try self.expectByte()) {
                    'e' => {
                        self.cursor += 1;
                        self.state = .post_value;
                        return .false;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_n => {
                switch (try self.expectByte()) {
                    'u' => {
                        self.cursor += 1;
                        self.state = .literal_nu;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_nu => {
                switch (try self.expectByte()) {
                    'l' => {
                        self.cursor += 1;
                        self.state = .literal_nul;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },
            .literal_nul => {
                switch (try self.expectByte()) {
                    'l' => {
                        self.cursor += 1;
                        self.state = .post_value;
                        return .null;
                    },
                    else => return error.SyntaxError,
                }
            },
        }
        unreachable;
    }
}

/// Seeks ahead in the input until the first byte of the next token (or the end of the input)
/// determines which type of token will be returned from the next `next*()` call.
/// This function is idempotent, only advancing past commas, colons, and inter-token whitespace.
pub fn peekNextTokenType(self: *@This()) PeekError!TokenType {
    state_loop: while (true) {
        switch (self.state) {
            .value => {
                switch (try self.skipWhitespaceExpectByte()) {
                    '{' => return .object_begin,
                    '[' => return .array_begin,
                    '"' => return .string,
                    '-', '0'...'9' => return .number,
                    't' => return .true,
                    'f' => return .false,
                    'n' => return .null,
                    else => return error.SyntaxError,
                }
            },

            .post_value => {
                if (try self.skipWhitespaceCheckEnd()) return .end_of_document;

                const c = self.input[self.cursor];
                if (self.string_is_object_key) {
                    self.string_is_object_key = false;
                    switch (c) {
                        ':' => {
                            self.cursor += 1;
                            self.state = .value;
                            continue :state_loop;
                        },
                        else => return error.SyntaxError,
                    }
                }

                switch (c) {
                    '}' => return .object_end,
                    ']' => return .array_end,
                    ',' => {
                        switch (self.stack.peek()) {
                            OBJECT_MODE => {
                                self.state = .object_post_comma;
                            },
                            ARRAY_MODE => {
                                self.state = .value;
                            },
                        }
                        self.cursor += 1;
                        continue :state_loop;
                    },
                    else => return error.SyntaxError,
                }
            },

            .object_start => {
                switch (try self.skipWhitespaceExpectByte()) {
                    '"' => return .string,
                    '}' => return .object_end,
                    else => return error.SyntaxError,
                }
            },
            .object_post_comma => {
                switch (try self.skipWhitespaceExpectByte()) {
                    '"' => return .string,
                    else => return error.SyntaxError,
                }
            },

            .array_start => {
                switch (try self.skipWhitespaceExpectByte()) {
                    ']' => return .array_end,
                    else => {
                        self.state = .value;
                        continue :state_loop;
                    },
                }
            },

            .number_minus,
            .number_leading_zero,
            .number_int,
            .number_post_dot,
            .number_frac,
            .number_post_e,
            .number_post_e_sign,
            .number_exp,
            => return .number,

            .string,
            .string_backslash,
            .string_backslash_u,
            .string_backslash_u_1,
            .string_backslash_u_2,
            .string_backslash_u_3,
            .string_surrogate_half,
            .string_surrogate_half_backslash,
            .string_surrogate_half_backslash_u,
            .string_surrogate_half_backslash_u_1,
            .string_surrogate_half_backslash_u_2,
            .string_surrogate_half_backslash_u_3,
            => return .string,

            .string_utf8_last_byte,
            .string_utf8_second_to_last_byte,
            .string_utf8_second_to_last_byte_guard_against_overlong,
            .string_utf8_second_to_last_byte_guard_against_surrogate_half,
            .string_utf8_third_to_last_byte,
            .string_utf8_third_to_last_byte_guard_against_overlong,
            .string_utf8_third_to_last_byte_guard_against_too_large,
            => return .string,

            .literal_t,
            .literal_tr,
            .literal_tru,
            => return .true,
            .literal_f,
            .literal_fa,
            .literal_fal,
            .literal_fals,
            => return .false,
            .literal_n,
            .literal_nu,
            .literal_nul,
            => return .null,
        }
        unreachable;
    }
}

const State = enum {
    value,
    post_value,

    object_start,
    object_post_comma,

    array_start,

    number_minus,
    number_leading_zero,
    number_int,
    number_post_dot,
    number_frac,
    number_post_e,
    number_post_e_sign,
    number_exp,

    string,
    string_backslash,
    string_backslash_u,
    string_backslash_u_1,
    string_backslash_u_2,
    string_backslash_u_3,
    string_surrogate_half,
    string_surrogate_half_backslash,
    string_surrogate_half_backslash_u,
    string_surrogate_half_backslash_u_1,
    string_surrogate_half_backslash_u_2,
    string_surrogate_half_backslash_u_3,

    // From http://unicode.org/mail-arch/unicode-ml/y2003-m02/att-0467/01-The_Algorithm_to_Valide_an_UTF-8_String
    string_utf8_last_byte, // State A
    string_utf8_second_to_last_byte, // State B
    string_utf8_second_to_last_byte_guard_against_overlong, // State C
    string_utf8_second_to_last_byte_guard_against_surrogate_half, // State D
    string_utf8_third_to_last_byte, // State E
    string_utf8_third_to_last_byte_guard_against_overlong, // State F
    string_utf8_third_to_last_byte_guard_against_too_large, // State G

    literal_t,
    literal_tr,
    literal_tru,
    literal_f,
    literal_fa,
    literal_fal,
    literal_fals,
    literal_n,
    literal_nu,
    literal_nul,
};

fn expectByte(self: *const @This()) !u8 {
    if (self.cursor < self.input.len) {
        return self.input[self.cursor];
    }
    // No byte.
    if (self.is_end_of_input) return error.UnexpectedEndOfInput;
    return error.BufferUnderrun;
}

fn skipWhitespace(self: *@This()) void {
    while (self.cursor < self.input.len) : (self.cursor += 1) {
        switch (self.input[self.cursor]) {
            // Whitespace
            ' ', '\t', '\r' => continue,
            '\n' => {
                if (self.diagnostics) |diag| {
                    diag.line_number += 1;
                    // This will count the newline itself,
                    // which means a straight-forward subtraction will give a 1-based column number.
                    diag.line_start_cursor = self.cursor;
                }
                continue;
            },
            else => return,
        }
    }
}

fn skipWhitespaceExpectByte(self: *@This()) !u8 {
    self.skipWhitespace();
    return self.expectByte();
}

fn skipWhitespaceCheckEnd(self: *@This()) !bool {
    self.skipWhitespace();
    if (self.cursor >= self.input.len) {
        // End of buffer.
        if (self.is_end_of_input) {
            // End of everything.
            if (self.stackHeight() == 0) {
                // We did it!
                return true;
            }
            return error.UnexpectedEndOfInput;
        }
        return error.BufferUnderrun;
    }
    if (self.stackHeight() == 0) return error.SyntaxError;
    return false;
}

fn takeValueSlice(self: *@This()) []const u8 {
    const slice = self.input[self.value_start..self.cursor];
    self.value_start = self.cursor;
    return slice;
}
fn takeValueSliceMinusTrailingOffset(self: *@This(), trailing_negative_offset: usize) []const u8 {
    // Check if the escape sequence started before the current input buffer.
    // (The algebra here is awkward to avoid unsigned underflow,
    //  but it's just making sure the slice on the next line isn't UB.)
    if (self.cursor <= self.value_start + trailing_negative_offset) return "";
    const slice = self.input[self.value_start .. self.cursor - trailing_negative_offset];
    // When trailing_negative_offset is non-zero, setting self.value_start doesn't matter,
    // because we always set it again while emitting the .partial_string_escaped_*.
    self.value_start = self.cursor;
    return slice;
}

fn endOfBufferInNumber(self: *@This(), allow_end: bool) !Token {
    const slice = self.takeValueSlice();
    if (self.is_end_of_input) {
        if (!allow_end) return error.UnexpectedEndOfInput;
        self.state = .post_value;
        return Token{ .number = slice };
    }
    if (slice.len == 0) return error.BufferUnderrun;
    return Token{ .partial_number = slice };
}

fn endOfBufferInString(self: *@This()) !Token {
    if (self.is_end_of_input) return error.UnexpectedEndOfInput;
    const slice = self.takeValueSliceMinusTrailingOffset(switch (self.state) {
        // Don't include the escape sequence in the partial string.
        .string_backslash => 1,
        .string_backslash_u => 2,
        .string_backslash_u_1 => 3,
        .string_backslash_u_2 => 4,
        .string_backslash_u_3 => 5,
        .string_surrogate_half => 6,
        .string_surrogate_half_backslash => 7,
        .string_surrogate_half_backslash_u => 8,
        .string_surrogate_half_backslash_u_1 => 9,
        .string_surrogate_half_backslash_u_2 => 10,
        .string_surrogate_half_backslash_u_3 => 11,

        // Include everything up to the cursor otherwise.
        .string,
        .string_utf8_last_byte,
        .string_utf8_second_to_last_byte,
        .string_utf8_second_to_last_byte_guard_against_overlong,
        .string_utf8_second_to_last_byte_guard_against_surrogate_half,
        .string_utf8_third_to_last_byte,
        .string_utf8_third_to_last_byte_guard_against_overlong,
        .string_utf8_third_to_last_byte_guard_against_too_large,
        => 0,

        else => unreachable,
    });
    if (slice.len == 0) return error.BufferUnderrun;
    return Token{ .partial_string = slice };
}

fn partialStringCodepoint(code_point: u21) Token {
    var buf: [4]u8 = undefined;
    switch (std.unicode.utf8Encode(code_point, &buf) catch unreachable) {
        1 => return Token{ .partial_string_escaped_1 = buf[0..1].* },
        2 => return Token{ .partial_string_escaped_2 = buf[0..2].* },
        3 => return Token{ .partial_string_escaped_3 = buf[0..3].* },
        4 => return Token{ .partial_string_escaped_4 = buf[0..4].* },
        else => unreachable,
    }
}

/// Scan the input and check for malformed JSON.
/// On `SyntaxError` or `UnexpectedEndOfInput`, returns `false`.
/// Returns any errors from the allocator as-is, which is unlikely,
/// but can be caused by extreme nesting depth in the input.
pub fn validate(allocator: Allocator, s: []const u8) Allocator.Error!bool {
    var scanner = Scanner.initCompleteInput(allocator, s);
    defer scanner.deinit();

    while (true) {
        const token = scanner.next() catch |err| switch (err) {
            error.SyntaxError, error.UnexpectedEndOfInput => return false,
            error.OutOfMemory => |e| return e,
            error.BufferUnderrun => unreachable,
        };
        if (token == .end_of_document) break;
    }

    return true;
}

/// The parsing errors are divided into two categories:
///  * `SyntaxError` is for clearly malformed JSON documents,
///    such as giving an input document that isn't JSON at all.
///  * `UnexpectedEndOfInput` is for signaling that everything's been
///    valid so far, but the input appears to be truncated for some reason.
/// Note that a completely empty (or whitespace-only) input will give `UnexpectedEndOfInput`.
pub const Error = error{ SyntaxError, UnexpectedEndOfInput };

/// Used by `json.reader`.
pub const default_buffer_size = 0x1000;

/// The tokens emitted by `std.json.Scanner` and `std.json.Reader` `.next*()` functions follow this grammar:
/// ```
///  <document> = <value> .end_of_document
///  <value> =
///    | <object>
///    | <array>
///    | <number>
///    | <string>
///    | .true
///    | .false
///    | .null
///  <object> = .object_begin ( <string> <value> )* .object_end
///  <array> = .array_begin ( <value> )* .array_end
///  <number> = <It depends. See below.>
///  <string> = <It depends. See below.>
/// ```
///
/// What you get for `<number>` and `<string>` values depends on which `next*()` method you call:
///
/// ```
/// next():
///  <number> = ( .partial_number )* .number
///  <string> = ( <partial_string> )* .string
///  <partial_string> =
///    | .partial_string
///    | .partial_string_escaped_1
///    | .partial_string_escaped_2
///    | .partial_string_escaped_3
///    | .partial_string_escaped_4
///
/// nextAlloc*(..., .alloc_always):
///  <number> = .allocated_number
///  <string> = .allocated_string
///
/// nextAlloc*(..., .alloc_if_needed):
///  <number> =
///    | .number
///    | .allocated_number
///  <string> =
///    | .string
///    | .allocated_string
/// ```
///
/// For all tokens with a `[]const u8`, `[]u8`, or `[n]u8` payload, the payload represents the content of the value.
/// For number values, this is the representation of the number exactly as it appears in the input.
/// For strings, this is the content of the string after resolving escape sequences.
///
/// For `.allocated_number` and `.allocated_string`, the `[]u8` payloads are allocations made with the given allocator.
/// You are responsible for managing that memory. `json.Reader.deinit()` does *not* free those allocations.
///
/// The `.partial_*` tokens indicate that a value spans multiple input buffers or that a string contains escape sequences.
/// To get a complete value in memory, you need to concatenate the values yourself.
/// Calling `nextAlloc*()` does this for you, and returns an `.allocated_*` token with the result.
///
/// For tokens with a `[]const u8` payload, the payload is a slice into the current input buffer.
/// The memory may become undefined during the next call to `json.Scanner.feedInput()`
/// or any `json.Reader` method whose return error set includes `json.Error`.
/// To keep the value persistently, it recommended to make a copy or to use `.alloc_always`,
/// which makes a copy for you.
///
/// Note that `.number` and `.string` tokens that follow `.partial_*` tokens may have `0` length to indicate that
/// the previously partial value is completed with no additional bytes.
/// (This can happen when the break between input buffers happens to land on the exact end of a value. E.g. `"[1234"`, `"]"`.)
/// `.partial_*` tokens never have `0` length.
///
/// The recommended strategy for using the different `next*()` methods is something like this:
///
/// When you're expecting an object key, use `.alloc_if_needed`.
/// You often don't need a copy of the key string to persist; you might just check which field it is.
/// In the case that the key happens to require an allocation, free it immediately after checking it.
///
/// When you're expecting a meaningful string value (such as on the right of a `:`),
/// use `.alloc_always` in order to keep the value valid throughout parsing the rest of the document.
///
/// When you're expecting a number value, use `.alloc_if_needed`.
/// You're probably going to be parsing the string representation of the number into a numeric representation,
/// so you need the complete string representation only temporarily.
///
/// When you're skipping an unrecognized value, use `skipValue()`.
pub const Token = union(enum) {
    object_begin,
    object_end,
    array_begin,
    array_end,

    true,
    false,
    null,

    number: []const u8,
    partial_number: []const u8,
    allocated_number: []u8,

    string: []const u8,
    partial_string: []const u8,
    partial_string_escaped_1: [1]u8,
    partial_string_escaped_2: [2]u8,
    partial_string_escaped_3: [3]u8,
    partial_string_escaped_4: [4]u8,
    allocated_string: []u8,

    end_of_document,
};

/// This is only used in `peekNextTokenType()` and gives a categorization based on the first byte of the next token that will be emitted from a `next*()` call.
pub const TokenType = enum {
    object_begin,
    object_end,
    array_begin,
    array_end,
    true,
    false,
    null,
    number,
    string,
    end_of_document,
};

/// To enable diagnostics, declare `var diagnostics = Diagnostics{};` then call `source.enableDiagnostics(&diagnostics);`
/// where `source` is either a `std.json.Reader` or a `std.json.Scanner` that has just been initialized.
/// At any time, notably just after an error, call `getLine()`, `getColumn()`, and/or `getByteOffset()`
/// to get meaningful information from this.
pub const Diagnostics = struct {
    line_number: u64 = 1,
    line_start_cursor: usize = @as(usize, @bitCast(@as(isize, -1))), // Start just "before" the input buffer to get a 1-based column for line 1.
    total_bytes_before_current_input: u64 = 0,
    cursor_pointer: *const usize = undefined,

    /// Starts at 1.
    pub fn getLine(self: *const @This()) u64 {
        return self.line_number;
    }
    /// Starts at 1.
    pub fn getColumn(self: *const @This()) u64 {
        return self.cursor_pointer.* -% self.line_start_cursor;
    }
    /// Starts at 0. Measures the byte offset since the start of the input.
    pub fn getByteOffset(self: *const @This()) u64 {
        return self.total_bytes_before_current_input + self.cursor_pointer.*;
    }
};

/// See the documentation for `std.json.Token`.
pub const AllocWhen = enum { alloc_if_needed, alloc_always };

/// For security, the maximum size allocated to store a single string or number value is limited to 4MiB by default.
/// This limit can be specified by calling `nextAllocMax()` instead of `nextAlloc()`.
pub const default_max_value_len = 4 * 1024 * 1024;

/// All `next*()` methods here handle `error.BufferUnderrun` from `std.json.Scanner`, and then read from the reader.
pub const Reader = struct {
    scanner: Scanner,
    reader: *std.Io.Reader,

    /// The allocator is only used to track `[]` and `{}` nesting levels.
    pub fn init(allocator: Allocator, io_reader: *std.Io.Reader) @This() {
        return .{
            .scanner = Scanner.initStreaming(allocator),
            .reader = io_reader,
        };
    }
    pub fn deinit(self: *@This()) void {
        self.scanner.deinit();
        self.* = undefined;
    }

    /// Calls `std.json.Scanner.enableDiagnostics`.
    pub fn enableDiagnostics(self: *@This(), diagnostics: *Diagnostics) void {
        self.scanner.enableDiagnostics(diagnostics);
    }

    pub const NextError = std.Io.Reader.Error || Error || Allocator.Error;
    pub const SkipError = Reader.NextError;
    pub const AllocError = Reader.NextError || error{ValueTooLong};
    pub const PeekError = std.Io.Reader.Error || Error;

    /// Equivalent to `nextAllocMax(allocator, when, default_max_value_len);`
    /// See also `std.json.Token` for documentation of `nextAlloc*()` function behavior.
    pub fn nextAlloc(self: *@This(), allocator: Allocator, when: AllocWhen) Reader.AllocError!Token {
        return self.nextAllocMax(allocator, when, default_max_value_len);
    }
    /// See also `std.json.Token` for documentation of `nextAlloc*()` function behavior.
    pub fn nextAllocMax(self: *@This(), allocator: Allocator, when: AllocWhen, max_value_len: usize) Reader.AllocError!Token {
        const token_type = try self.peekNextTokenType();
        switch (token_type) {
            .number, .string => {
                var value_list = std.array_list.Managed(u8).init(allocator);
                errdefer {
                    value_list.deinit();
                }
                if (try self.allocNextIntoArrayListMax(&value_list, when, max_value_len)) |slice| {
                    return if (token_type == .number)
                        Token{ .number = slice }
                    else
                        Token{ .string = slice };
                } else {
                    return if (token_type == .number)
                        Token{ .allocated_number = try value_list.toOwnedSlice() }
                    else
                        Token{ .allocated_string = try value_list.toOwnedSlice() };
                }
            },

            // Simple tokens never alloc.
            .object_begin,
            .object_end,
            .array_begin,
            .array_end,
            .true,
            .false,
            .null,
            .end_of_document,
            => return try self.next(),
        }
    }

    /// Equivalent to `allocNextIntoArrayListMax(value_list, when, default_max_value_len);`
    pub fn allocNextIntoArrayList(self: *@This(), value_list: *std.array_list.Managed(u8), when: AllocWhen) Reader.AllocError!?[]const u8 {
        return self.allocNextIntoArrayListMax(value_list, when, default_max_value_len);
    }
    /// Calls `std.json.Scanner.allocNextIntoArrayListMax` and handles `error.BufferUnderrun`.
    pub fn allocNextIntoArrayListMax(self: *@This(), value_list: *std.array_list.Managed(u8), when: AllocWhen, max_value_len: usize) Reader.AllocError!?[]const u8 {
        while (true) {
            return self.scanner.allocNextIntoArrayListMax(value_list, when, max_value_len) catch |err| switch (err) {
                error.BufferUnderrun => {
                    try self.refillBuffer();
                    continue;
                },
                else => |other_err| return other_err,
            };
        }
    }

    /// Like `std.json.Scanner.skipValue`, but handles `error.BufferUnderrun`.
    pub fn skipValue(self: *@This()) Reader.SkipError!void {
        switch (try self.peekNextTokenType()) {
            .object_begin, .array_begin => {
                try self.skipUntilStackHeight(self.stackHeight());
            },
            .number, .string => {
                while (true) {
                    switch (try self.next()) {
                        .partial_number,
                        .partial_string,
                        .partial_string_escaped_1,
                        .partial_string_escaped_2,
                        .partial_string_escaped_3,
                        .partial_string_escaped_4,
                        => continue,

                        .number, .string => break,

                        else => unreachable,
                    }
                }
            },
            .true, .false, .null => {
                _ = try self.next();
            },

            .object_end, .array_end, .end_of_document => unreachable, // Attempt to skip a non-value token.
        }
    }
    /// Like `std.json.Scanner.skipUntilStackHeight()` but handles `error.BufferUnderrun`.
    pub fn skipUntilStackHeight(self: *@This(), terminal_stack_height: usize) Reader.NextError!void {
        while (true) {
            return self.scanner.skipUntilStackHeight(terminal_stack_height) catch |err| switch (err) {
                error.BufferUnderrun => {
                    try self.refillBuffer();
                    continue;
                },
                else => |other_err| return other_err,
            };
        }
    }

    /// Calls `std.json.Scanner.stackHeight`.
    pub fn stackHeight(self: *const @This()) usize {
        return self.scanner.stackHeight();
    }
    /// Calls `std.json.Scanner.ensureTotalStackCapacity`.
    pub fn ensureTotalStackCapacity(self: *@This(), height: usize) Allocator.Error!void {
        try self.scanner.ensureTotalStackCapacity(height);
    }

    /// See `std.json.Token` for documentation of this function.
    pub fn next(self: *@This()) Reader.NextError!Token {
        while (true) {
            return self.scanner.next() catch |err| switch (err) {
                error.BufferUnderrun => {
                    try self.refillBuffer();
                    continue;
                },
                else => |other_err| return other_err,
            };
        }
    }

    /// See `std.json.Scanner.peekNextTokenType()`.
    pub fn peekNextTokenType(self: *@This()) Reader.PeekError!TokenType {
        while (true) {
            return self.scanner.peekNextTokenType() catch |err| switch (err) {
                error.BufferUnderrun => {
                    try self.refillBuffer();
                    continue;
                },
                else => |other_err| return other_err,
            };
        }
    }

    fn refillBuffer(self: *@This()) std.Io.Reader.Error!void {
        const input = self.reader.peekGreedy(1) catch |err| switch (err) {
            error.ReadFailed => |e| return e,
            error.EndOfStream => return self.scanner.endInput(),
        };
        self.reader.toss(input.len);
        self.scanner.feedInput(input);
    }
};

const OBJECT_MODE = 0;
const ARRAY_MODE = 1;

fn appendSlice(list: *std.array_list.Managed(u8), buf: []const u8, max_value_len: usize) !void {
    const new_len = std.math.add(usize, list.items.len, buf.len) catch return error.ValueTooLong;
    if (new_len > max_value_len) return error.ValueTooLong;
    try list.appendSlice(buf);
}

/// For the slice you get from a `Token.number` or `Token.allocated_number`,
/// this function returns true if the number doesn't contain any fraction or exponent components, and is not `-0`.
/// Note, the numeric value encoded by the value may still be an integer, such as `1.0`.
/// This function is meant to give a hint about whether integer parsing or float parsing should be used on the value.
/// This function will not give meaningful results on non-numeric input.
pub fn isNumberFormattedLikeAnInteger(value: []const u8) bool {
    if (std.mem.eql(u8, value, "-0")) return false;
    return std.mem.findAny(u8, value, ".eE") == null;
}

test {
    _ = @import("./scanner_test.zig");
}