A new approach to knowledge graphs

A new study introduces an architecture for joint training of models on sentences and structured data, keeping knowledge and language representations distinct. The model treats knowledge graphs and hypergraphs as structured instances with predefined roles, encoding them into a key-value repository.

Model architecture

A language transformer accesses this repository through attention mechanisms. Attention is conditioned by journey-based role transport, which unifies the traversal of knowledge graphs with edge labels, hyperedge traversal, and sentence structure. The architecture includes a dual stream, hierarchical layer groups with instance-local, neighborhood, and global mixing attention, retrieval over a separate repository, and multi-task objectives that include masked language modeling, link prediction, and role-consistency denoising.

Separation between linguistic context and structured knowledge

The result is an explicit and inspectable separation between linguistic context and structured knowledge, while still enabling tight alignment through cross-attention. This approach could improve the interpretability and controllability of language models, opening new avenues for applications where knowledge understanding is critical.