SOSM: A Novel Approach to Language Models

A researcher has released the source code for the Self-Organizing State Model (SOSM) project, a research initiative that remained unfinished due to personal reasons. The goal is to receive technical feedback, constructive criticism, and potentially find someone interested in continuing the project's development.

SOSM proposes an alternative to the Transformer architecture, leveraging a graph-based system instead of traditional dense attention. The model separates semantic representation from temporal pattern learning and introduces a hierarchical attribution mechanism to improve interpretability.

Key Components of SOSM

The SOSM architecture is modular and relies on several components:

  • Semantic Representation Module (MU): Manages the representation of meaning.
  • Temporal Pattern Learner (TEMPORAL): Models sequence progression and context flow.
  • Hierarchical K-1 Self-Learning System: Provides interpretability, enabling analysis of predictions.

A More Efficient Model

Unlike Transformers that compare every token with all others, SOSM uses a graph-based connection mechanism. This approach limits computation to only the most relevant tokens, enabling selective reasoning and improved efficiency. The researcher invites the community to provide feedback and evaluate the potential of SOSM, hoping it may be useful for further research or development.