Try Llama2 Locally at No Cost

Run Llama locally for free

Outdated: Written in July 2023. llama.cpp replaced GGML .bin models with GGUF, renamed server and main to llama-server and llama-cli, and replaced the Makefile build with CMake. The repository moved to github.com/ggml-org/llama.cpp. The post is kept as an archive; do not follow the commands as written.

Today I will unravel the process of running Llama 2, a recent LLM from Meta, for free! Let's get started with this powerful tool.

I will assume a minimal level of familiarity with the command line and also that you have a basic machine with at least 8GB of RAM and without a GPU. If you have a GPU, you can use it to speed up the inference process (explore the official docs).

Clone the Repository

First and foremost, we need to clone the repository. Using the GitHub llama.cpp, we may clone the repository on our machine. Of course, we need to have Git installed for this to work.

After the clone, we follow the build instructions to compile the code and make the program executable.

Build Instructions

We have two different options for building the llama.cpp project:

Using 'make'

For Linux or macOS systems, we can simply build the program in the terminal using:

make

For Windows, we do the following:

1. Download the latest version of w64devkit.
2. Extract 'w64devkit' on your computer.
3. Run 'w64devkit.exe'.
4. Use the 'cd' command to get to the 'llama.cpp' directory.
5. From here we are all set to run:

   make
   

Using CMake

mkdir build
cd build
cmake ..
cmake --build . --config Release

Download Llama 2

Proceed to download the latest release of Llama 2 (a quantized version brought by TheBloke) from here. We look for what best suits our system's RAM following the table provided here.

Note: The GGML versions are used to run the model on the CPU, while the GPTQ versions are used to run the model on GPU. However, with the GGML versions it may offload some layers to the GPU, which will speed up the inference process. Explore the official docs for more information.

Update: The GGML versions are deprecated. Use the new GGUF versions instead.

For 8GB of RAM, it is recommended to begin with the llama-2-7b-chat.ggmlv3.q4_K_S.bin version.

Following the download, put the obtained file in the models/7B directory.

Spring Into Action

All set? Start the server!

For Unix-based systems (such as Linux and macOS):

./server -m models/7B/llama-2-7b-chat.ggmlv3.q4_K_S.bin -c 2048

For Windows systems:

server.exe -m models\7B\llama-2-7b-chat.ggmlv3.q4_K_S.bin -c 2048

That's all there is to it!

Get Engaged with Llama 2

Once everything is set into place, simply type into the web browser "http://127.0.0.1:8080".

Now, start inference with your Llama 2. Observe how human the interactions seem, get plenty of useful information, and simply have fun!

In a time when the role of AI is amplifying and human-like chatbots are soaring in popularity, mastering the ground-level implementation of tools like Llama 2 can be overwhelming with the expansive possibilities of AI systems. So, go ahead and apply this simple demonstration to host Llama 2 on your device.

Try More

The downloaded model is a 7B model, which means it has 7 billion parameters. A larger model is usually more capable. However, a larger model also needs more RAM. If you have a GPU, you can use it to speed up the inference.

Different models have different capabilities. For example, the Chat model is fine-tuned to have conversations, while the basic model is unrestricted and may generate more diverse content.