llama.cpp

Getting started with llama.cpp is straightforward. Here are several ways to install it on your machine: Install llama.cpp using brew, nix, winget, or conda-forge Run with Docker - see our Docker documentation Download pre-built binaries from the releases page Build from source by cloning this repository - check out our build guide Once installed, you'll need a model to work with. Head to the Obtaining and quantizing models section to learn more. The main goal of llama.cpp is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud. Plain C/C++ implementation without any dependencies Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks AVX, AVX2, AVX512 and AMX support for x86 architectures RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA) Vulkan and SYCL backend support CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity The llama.cpp project is the main playground for developing new features for the ggml library. (to have a project listed here, it should clearly state that it depends on llama.cpp ) AI Sublime Text plugin (MIT) BonzAI App (proprietary) cztomsik/ava (MIT) Dot (GPL) eva (MIT) iohub/collama (Apache-2.0) janhq/jan (AGPL) johnbean393/Sidekick (MIT) KanTV (Apache-2.0) KodiBot (GPL) llama.vim (MIT) LARS (AGPL) Llama Assistant (GPL) LlamaLib (Apache-2.0) LLMFarm (MIT) LLMUnity (MIT) LMStudio (proprietary) LocalAI (MIT) LostRuins/koboldcpp (AGPL) MindMac (proprietary) MindWorkAI/AI-Studio (FSL-1.1-MIT) Mobile-Artificial-Intelligence/maid (MIT) Mozilla-Ocho/llamafile (Apache-2.0) nat/openplayground (MIT) nomic-ai/gpt4all (MIT) ollama/ollama (MIT) oobabooga/text-generation-webui (AGPL) PocketPal AI (MIT) psugihara/FreeChat (MIT) ptsochantaris/emeltal (MIT) pythops/tenere (AGPL) ramalama (MIT) semperai/amica (MIT) withcatai/catai (MIT) Autopen (GPL) Tools akx/ggify – download PyTorch models from Hugging Face Hub and convert them to GGML akx/ollama-dl – download models from the Ollama library to be used directly with llama.cpp crashr/gppm – launch llama.cpp instances utilizing NVIDIA Tesla P40 or P100 GPUs with reduced idle power consumption gpustack/gguf-parser - review/check the GGUF file and estimate the memory usage Styled Lines (proprietary licensed, async wrapper of inference part for game development in Unity3d with pre-built Mobile and Web platform wrappers and a model example) unslothai/unsloth – 🦥 exports/saves fine-tuned and trained models to GGUF (Apache-2.0) Infrastructure Paddler - Open-source LLMOps platform for hosting and scaling AI in your own infrastructure GPUStack - Manage GPU clusters for running LLMs llama cpp canister - llama.cpp as a smart contract on the Internet Computer, using WebAssembly llama-swap - transparent proxy that adds automatic model switching with llama-server Kalavai - Crowdsource end to end LLM deployment at any scale llmaz - ☸️ Easy, advanced inference platform for large language models on Kubernetes. LLMKube - Kubernetes operator for llama.cpp with multi-GPU and Apple Silicon Metal support" Metal Apple Silicon BLAS All BLIS All SYCL Intel GPU [OpenVINO [In Progress]](docs/backend/OPENVINO.md) Intel CPUs, GPUs, and NPUs MUSA Moore Threads GPU CUDA Nvidia GPU HIP AMD GPU ZenDNN AMD CPU Vulkan GPU CANN Ascend NPU OpenCL Adreno GPU IBM zDNN IBM Z & LinuxONE WebGPU All RPC All [Hexagon [In Progress]](docs/backend/snapdragon/README.md) Snapdragon VirtGPU VirtGPU APIR The Hugging Face platform hosts a number of LLMs compatible with llama.cpp : Trending LLaMA You can either manually download the GGUF file or directly use any llama.cpp -compatible models from Hugging Face