The Forward-Forward Algorithm and the Goodness Function Challenge

The Forward-Forward (FF) algorithm emerges as a promising alternative to traditional backpropagation for training neural networks. Characterized by its biological plausibility, the FF method operates by training the network layer by layer, utilizing a local "goodness function" to distinguish between "positive" and "negative" data. Since its introduction, the "sum-of-squares" (SoS) function has served as the default choice for this evaluation.

However, research has begun to systematically explore the design space of goodness functions, analyzing both which activations to measure and how to aggregate them. This investigation is crucial for optimizing the effectiveness and efficiency of neural network training, especially in contexts where computational resources may be a constraint.

Innovations in Selective Measurement and Adaptive Sparsity

A significant innovation is the introduction of "top-k goodness," a function that evaluates only the k most active neurons within a layer. This selective approach has been shown to substantially outperform the SoS function, leading to a 22.6 percentage point improvement in accuracy on the Fashion-MNIST dataset. Further gains were achieved with the introduction of "entmax-weighted energy," which replaces rigid "top-k" selection with a learnable sparse weighting based on the alpha-entmax transformation, ensuring additional performance benefits.

In parallel, the "separate label feature forwarding" (FFCL) technique has been adopted, which injects class hypotheses at every layer through a dedicated projection, rather than concatenating them only at the input. The combination of these ideas – "top-k goodness," "entmax-weighted energy," and FFCL – allowed for an accuracy of 87.1% on Fashion-MNIST with a 4x2000 architecture, a 30.7 percentage point improvement over the SoS baseline, achieved by modifying only the goodness function and the label pathway.

The Critical Role of Sparsity in FF Network Design

Through a series of controlled experiments, involving 11 different goodness functions, two architectures, and a sparsity spectrum analysis for both k and alpha, a fundamental principle emerged: sparsity in the goodness function represents the most important design choice in Forward-Forward networks.

Specifically, the research highlighted that adaptive sparsity, with an alpha value of approximately 1.5, offers superior performance compared to both fully dense and fully sparse alternatives. This suggests that it is not merely the reduction in the number of active neurons that is crucial, but rather the ability to dynamically select and weight the most relevant activations.

Implications for AI Efficiency and Deployments

These developments in the Forward-Forward algorithm have significant implications for the efficiency of artificial intelligence systems. Fundamental algorithmic improvements such as "top-k goodness" and adaptive sparsity can potentially reduce the computational requirements for model training and inference. This is particularly relevant for organizations considering on-premise deployments or edge environments, where hardware resources are often limited and TCO is a critical factor.

The ability to achieve superior performance with sparser activation could translate into lower VRAM consumption, higher throughput, or reduced latency, making models more accessible and sustainable. For those evaluating on-premise deployments, AI-RADAR offers analytical frameworks on /llm-onpremise to assess the trade-offs between algorithmic efficiency, hardware requirements, and operational costs, contributing to informed decisions for AI/LLM workloads.