Training AI models in cybersecurity with vast datasets offers significant opportunities to mimic real-world behaviors effectively. However, challenges like data drift and scarcity of labeled data lead to frequent updates of models and the risk of overfitting.

Innovative Strategies with LLMs

To address these challenges, the research focuses on using parameter-efficient fine-tuning techniques for pre-trained language models, combining compactors with various layer freezing strategies. To further enhance the capabilities of these pre-trained models, the study introduces two strategies that leverage large language models (LLMs).

Labeling and Fallback

In the first strategy, LLMs are used as data-labeling tools, generating labels for unlabeled data. In the second strategy, LLMs serve as fallback mechanisms for predictions with low confidence scores. Comprehensive experimental analysis of the proposed strategies is performed on different downstream tasks specific to the cybersecurity domain.

Results and Implications

The results empirically demonstrate that by combining parameter-efficient pre-trained models with LLMs, it is possible to improve the reliability and robustness of the models, making them more suitable for real-world cybersecurity applications.