{"id":991,"date":"2023-08-07T15:15:44","date_gmt":"2023-08-07T15:15:44","guid":{"rendered":"https:\/\/ai-box.eu\/?p=991"},"modified":"2023-08-08T11:31:44","modified_gmt":"2023-08-08T11:31:44","slug":"chatbot-llama-2-70b-run-locally-in-a-jupyter-notebook","status":"publish","type":"post","link":"https:\/\/ai-box.eu\/en\/news\/chatbot-llama-2-70b-run-locally-in-a-jupyter-notebook\/991\/","title":{"rendered":"Chatbot Llama 2 70B – run locally in a jupyter notebook"},"content":{"rendered":"

With this article I would like to help you to run locally under Ubuntu 22.04 Llama 2 in the 70B version on a NVIDIA A6000. Of course this is not easy because the classic Llama 2 70B model needs a GPU memory of about 280GB in the Float32 bit version. But if you quantize the model to INT4 bit then it fits into the memory of the NVIDIA A6000 with its 48GB GPU Ram. The model then occupies about 40GB of ram and runs at a decent speed.<\/p>\n

I followed the video tutorial by James Briggs on YouTube which explains very well how to get the model running. However, this tutorial lacks the setup of the environment under Ubuntu where everything is set up. This is where I had my minor challenges until everything was up and running. Therefore my tutorial starts with setting up the runtime environment. After that everything works without problems.<\/p>\n

Here is the link to the YouTube video of James Briggs: Llama 2 in LangChain \u2014 FIRST Open Source Conversational Agent!<\/a><\/p>\n

Hardware Note:<\/strong> In order to set up the Llama 2 70B model on your computer as described in this guide, you will need approximately 200GB of free memory and a video card with 48GB of video RAM \/ GPU RAM.<\/p>\n

\n
\n

Table of Contents<\/p>\nToggle<\/span><\/path><\/svg><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n