SRT

SRT (Semiotic-Reflexive Transformer Adapter Architecture) is a groundbreaking tool designed to enhance frozen causal language models with semiotic awareness. This lightweight module integrates seamlessly with existing architectures, allowing models to better understand and navigate the complexities of meaning within language. The SRT-Adapter uniquely converts divergence from hidden states into actionable insights, effectively tracking reflexive awareness and providing the capability to inject semiotic corrections into the language processing stream.

Architecture Overview

The architecture of SRT is composed of several layers, where components such as the Backbone Embeddings, frozen layers, and various semiotic modules work in tandem. This modular design includes:

tokens ──► Backbone Embeddings (native, frozen)
               │
         ┌─────┴─────┐
         │  Layer 0-6 │  (frozen)
         └─────┬─────┘
               │
         ┌─────┴─────┐
  ┌─────►│  Layer 7   │──────► MAH₁ reads divergence ──► RRM step
  │      └─────┬─────┘
  │            │
  │      ┌─────┴─────┐
  │      │ Layer 8-13 │  (frozen)
  │      └─────┬─────┘
  │            │
  │      ┌─────┴─────┐
  ├─────►│  Layer 14  │──────► MAH₂ reads ──► RRM step ──► inject
  │      └─────┬─────┘                                       │
  │            │◄────────────────────────────────────────────┘
  │      ┌─────┴─────┐
  │      │ Layer 15-20│  (frozen, with semiotic correction)
  │      └─────┬─────┘
  │            │
  │      ┌─────┴─────┐
  └─────►│  Layer 21  │──────► MAH₃ reads ──► RRM step ──► inject
         └─────┬─────┘                                       │
               │◄────────────────────────────────────────────┘
         ┌─────┴─────┐
         │ Layer 22-27│  (frozen, with semiotic correction)
         └─────┬─────┘
               │
         Backbone LM Head (native, frozen) ──► logits + CE loss
               │
         BEN (from RRM meta-state) ──► r̂, regime, modulation

Key Features

1. Zero Cross-Entropy Degradation: Maintains the integrity of native embeddings and language model heads, starting at pretrained quality.
2. Efficient Parameter Management: Training involves approximately 14.6 million trainable parameters without altering the fully frozen 7 billion parameter backbone, enabling quick training times.
3. Unsupervised Community Discovery: Facilitates discovery of discourse-trajectory structures without relying on hardcoded labels.
4. Backbone-Agnostic: Fully compatible with various HuggingFace AutoModelForCausalLM implementations like Qwen, LLaMA, Mistral, Phi, and more.
5. Portability: Adapter weights are lightweight (only 44MB), allowing for easy integration with different compatible backbones during inference.

Modules Breakdown

Theoretical Foundation

Grounded in the semiotics of C.S. Peirce, SRT empowers language models to recognize that signs can have multifaceted meanings across different contexts. For instance, the model can discern that "freedom" can be interpreted differently in varying political discourses.

SRT promotes reflexivity within models, enhancing their ability to self-assess how interpretations shift and change over discourse trajectories. This adaptive capability is essential for advanced natural language understanding applications.

Further information on SRT, including technical details and research references, can be found in Lancaster (2025), available in various formats within this repository. Released checkpoints for practical use include srt-adapter-v8a and srt-adapter-v1.0.

This project contributes significantly to the field of natural language processing by integrating cutting-edge semiotic techniques with state-of-the-art language models.