meon

meon is a declarative flat parsing engine designed specifically for text formats, streamlining the parsing process without the overhead of an abstract syntax tree (AST). By leveraging a structure of arrays (SoA), it offers efficient parsing through contiguous memory blocks that significantly enhance performance, particularly in high-speed applications.

Key Features:

• Flat Output Structure: Instead of traditional tree structures, meon outputs a struct-of-arrays, ensuring that data is organized in such a way that it minimizes cache misses and maximizes CPU performance during parsing operations. This design approach optimizes memory access patterns and allows for efficient iteration over parsed elements.

  MarkdownContent {
      source:         &[u8],  // Original bytes
      texts:          Vec<Span>,  // Plain text runs
      bolds:          Vec<Span>,  // Bold text spans
      italics:        Vec<Span>,  // Italic text spans
      codes:          Vec<Span>,  // Inline code spans
      links:          Vec<Link>,  // Links and images
      headings:       Vec<(Heading, Span)>,  // Heading elements
      fenced_codes:   Vec<Span>,  // Fenced code sections
      bullet_items:   Vec<(BulletItem, Span)>,  // Bullet points
  }
  

• Zero-Copy Spans: Every element in the output is represented as a Span, defined by its start and end byte offsets into the original source. This ensures zero-copy access—data remains intact and unmodified unless explicitly requested, thus maintaining integrity and efficiency in operations.

  // Example of resolving a span to a string slice without copying  
  let src = b"**bold** and *italic*\n";
  let c = MarkdownParser::parse(src);
  let text: &str = c.str(c.bolds[0]).unwrap();
  assert_eq!(text, "bold");
  

• Context-Free Extraction: meon allows for the extraction of specific elements without parsing the entire structure. By providing standalone iterators, it enables focused parsing tasks that require minimal computational resources.

  // Standalone iteration example for bold spans  
  for span in MarkdownParser::find_bolds(src) {
      println!("{}", std::str::from_utf8(&src[span.start as usize..span.end as usize]).unwrap());
  }
  

• Declarative Grammar Definition: The parsing format is highly customizable, requiring only a single invocation to define the grammar. The engine compiles this grammar into a parser at build time, ensuring that parsing is both efficient and tailored for specific needs.

  use meon::define_parser;
  define_parser!(MyFormat {
      sep = b' ', eol = b'\n', tab = b'\t', escape = b'\\', max_nest = 4;
      // Define inline rules  
  });
  

Repository Overview

The meon repository includes the core parsing engine, macros for grammar definition, various grammar implementations (such as Markdown and JSON), benchmarks, and fuzz testing harnesses. This structure not only demonstrates meon's capabilities with practical examples but also serves as a testing ground for its performance in real-world scenarios.

• Modules:
  • meon: Core parsing functionalities including runtime macros.
  • meon-macros: Provides a define_parser! procedural macro for easy grammar definition.
  • meon-md: Implements a grammar for parsing a subset of Markdown.
  • meon-json: Implements a grammar for efficiently parsing JSON structures.

Performance Enhancements

meon supports feature flags like avx2 and avx512, enabling SIMD capabilities that further optimize search operations, allowing the engine to fully utilize modern hardware capabilities.

By concentrating on a flat parsing structure and eliminating unnecessary abstractions, meon offers an innovative solution for applications encountering performance bottlenecks in text parsing, making it a valuable asset for developers looking to enhance their text processing capabilities.