LYTOK: Lightweight Token Object Notation - An Innovative Data Serialization Standard

LYTOK (Lightweight Token Object Notation) is a hybrid serialization standard crafted to enhance data exchange efficiency for Artificial Intelligence and high-concurrency systems. Featuring the official file extension .ltk, LYTOK addresses the limitations of traditional formats, offering a streamlined approach for modern applications.

Key Advantages of LYTOK

The dominance of JSON in data interchange presents obstacles in machine processing, especially with the increasing demands of AI applications. LYTOK emerges as a solution to these inefficiencies by providing:

• Reduced Context Cost: Unlike JSON, which repeatedly includes keys in array elements, LYTOK minimizes redundant data, optimizing space within model context windows.
• Explicit Type Differentiation: JSON's inherent ambiguity regarding data types is problematic for accurate interpretations. LYTOK supports distinct types, eliminating the need for additional transformations that may impact performance.

Operating Modes and Lexicon

LYTOK adapts its syntax based on data homogeneity to maximize space savings, operating primarily in three modes:

1. Uniform Arrays (#Schema)

This mode offers peak efficiency, using a header to define the positional order. For example:

#User[2]:id|nombre::

This indicates that two records will follow. Typing in the schema enables precision:

edad#|activo?|  

2. Simple Arrays (#, ?, @, &)

Designed for collections of a single data type:

• Numbers: #1;2;3;4
• Dates: @2025-01-01:00:00:00Z;2025-02-01:00:00:00Z

3. Mixed Arrays (*)

Accommodate heterogeneous data, with each value assigned a type prefix:

*#123;?T;texto;^

Where ^ represents NULL.

Advanced Data Types

LYTOK inherently supports data types often ignored by other text formats, ensuring precision without compromise:

Formatting and Structure

When utilizing the Formatted mode, readability is prioritized through specific formatting rules, including:

• Indentation: Four spaces per level.
• Support for nested maps and arrays with explicit structural rules.
• General separation guidelines: Different separators are used based on the data structure, ensuring clarity and organization.

Example of a Complex Formatted Structure:

#lote[2]:
  id|fecha|stock_total#|es_importado?|es_certificado?|fabricante{
    nombre|registro|direccion{
      calle|ciudad|zip#
    }
  }|datosLaboratorio{
    fecha_prueba|direccion{
      calle|edificio|referencia
    }
  }|certificaciones[
    nombre_cert|valida_hasta
  ]|sucursales[
    nombre|ciudad|empleados[
      nombre|apellido|datos{
        calle|numero#|telefono#|correo
      }
    ]
  ]::

Project Structure

The repository contains essential components:

• /spec: Formal EBNF grammar defining the self-delimitation rules.
• /compliance: Official test suite for validating third-party parsers.
• /assets: Visual identity elements and data flow diagrams.
• /examples: Real-world use case .ltk files.

Available SDKs

• JavaScript: lytok-js

LYTOK is designed to meet the growing need for a smarter and leaner data transport solution, empowering developers to harness the full potential of data serialization in high-performance environments.