LLMs grappling with meme humor

Memes have become a fundamental element of online communication, often used as interactive replies within conversations. A recent study focused on the ability of large language models (LLMs) to select appropriate and humorous memes in response to specific contexts.

Researchers introduced MaMe-Re (Manga Meme Reply Benchmark), a benchmark consisting of 100,000 pairs of openly licensed Japanese manga panels and social media posts, annotated by humans. The analysis revealed that LLMs show some ability to capture complex social elements such as exaggeration, but struggle to distinguish subtle differences in wit among semantically similar candidates.

The role of images and future challenges

A surprising result is that the inclusion of visual information did not improve the performance of the models, suggesting a difficulty in integrating the understanding of visual content with contextual humor. Although LLMs can match human judgments in controlled settings, selecting contextually humorous replies remains an open challenge for current models.

For those evaluating on-premise deployments, there are trade-offs to consider. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.