A user is preparing to invest in a local LLM setup as a personal and learning project, once they start their new job in August. Coming from a theoretical university background, they are seeking practical experience.

Key Questions

The user poses several crucial questions for anyone approaching the world of LLMs locally:

  • What hardware should be prioritized?
  • Which inference stack should be chosen to start with?
  • What beginner mistakes should be avoided?
  • Which models are actually usable on consumer GPUs?

The user recognizes the fragmentation of available information and hopes to obtain consolidated guidance from those who already have experience with local setups.