Grimoire — A compact assistant for Russian language…

Pitch

Description

Comments

About this project

Grimoire is a compact dual dynamic language model designed for Russian tasks. With 35.8M parameters, it offers a unique architecture that includes a Qwen2 decoder layer and a recurrent depth factory for enhanced processing. While still a research prototype, it supports tool invocation while acknowledging the challenges of accuracy and context reliability.

Grimoire is an experimental lightweight Russian-language model featuring a compact architecture with 35.8 million parameters. This dual dynamic language model includes a recurrent depth factory that generates per-head weights, and utilizes a compact vocabulary of 11,264 tokens. It is designed to facilitate efficient tool usage via a calculate and web_search invocation protocol.

Overview

Grimoire is structured as a single-file implementation in build_neuroset.py, containing the model architecture, training routines, and chat functionalities, while evaluation scripts are provided separately. The current status of this project is identified as an experimental prototype. While Grimoire is capable of generating tool calls and performing basic calculations, it may produce incorrect requests, misrepresent retrieved information, or deliver responses that deviate from the intended tasks. The reliability of information sources in context does not guarantee the accuracy of final answers.

Model Specifications

Attribute	Value
Primary checkpoint	`embrelgrim/grimoire`
Parameter count	35,849,352
Model dimension	`d_model=896`, with 14 heads × 64
Base block	Layer 6 from `Qwen/Qwen2.5-0.5B-Instruct`
Recurrent depth	8 steps with reused base layer
Training dataset size	Approximately 65,490 samples / 40,030,505 tokens
Target training time	Around 8 hours on Radeon RX 6600 (not inclusive of pre-training)

Functionality

The model is tailored for various functionalities, including arithmetic calculations and web searches utilizing a local Wikipedia database if available. Please note that the performance of these features may vary significantly based on the context and specific queries:

Calculate: Parses arithmetic expressions without relying on Python's eval.
Web Search: Utilizes a local Wikipedia corpus and fallback options for broader queries.

Usage Example

The following commands illustrate basic interactions with the model:

python build_neuroset.py --mode chat --run runs/assistant8
python build_neuroset.py --mode chat --run runs/assistant8 --prompt "сколько будет 47 * 13?"
python build_neuroset.py --mode bench --run runs/assistant8

Data Gathering and Training Configuration

Grimoire does not construct a dataset by itself. Users can specify their training and validation JSONL files through command-line arguments, and these files must be prepared in advance:

python build_neuroset.py --preset qwen36m  --data-dir YOUR_DATA_DIR --out runs/base --hours 5 --max-seq-len 768 --lr 2e-4 --warmup 400 --yes

Limitations

While Grimoire shows potential for specific tasks, it also has notable limitations which include:

The risk of generating inaccurate answers or misinterpretations.
A dependency on the availability of the tool and query context, notably when dealing with less common tokens and distributions.
Inherent weaknesses in maintaining a reliable multi-turn dialogue capability.

Conclusion

Grimoire is a pioneering attempt at creating a compact and efficient Russian-language LLM framework, which holds promise for tool-assisted functionalities. However, further refinement and testing are required to improve its reliability and contextual understanding. For those interested in model development and experimentation with language processing assistants, Grimoire serves as a valuable starting point.