Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 3-3
Download and Run Additional Models Locally Since you already downloaded models locally in Phase 4,...
Read More
Category: Top story
Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 2-3
Configuring vLLM for Production Deployment (More Complex) For production use, I want to run the...
Read More
Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 1-3
After showing you how to install Ollama, Open WebUI, ComfyUI, and LLaMA Factory on the Gigabyte AI...
Read More
Installing LLaMA Factory on Gigabyte AI TOP ATOM: Fine-tuning Language Models with LoRA and QLoRA – Part 2-2
Phase 7: Start Fine-Tuning Training Before I start the training, I might need to log in to the...
Read More
Installing LLaMA Factory on Gigabyte AI TOP ATOM: Fine-tuning Language Models with LoRA and QLoRA – Part 1-2
After showing how to install Ollama, Open WebUI, and ComfyUI on the Gigabyte AI TOP ATOM in my...
Read More
Installing ComfyUI on Gigabyte AI TOP ATOM: AI Image Generation with Stable Diffusion on Your Own Network
After showing you how to install Ollama and Open WebUI on the Gigabyte AI TOP ATOM in my previous...
Read More
Open WebUI on the Gigabyte AI TOP ATOM: ChatGPT-like Chat Interface for Your Own Network
After showing you in the last post how to install Ollama on the Gigabyte AI TOP ATOM and configure...
Read More
Ollama on the Gigabyte AI TOP ATOM: Central LLM Server for the Entire Network
Anyone experimenting with Large Language Models knows the problem: local hardware is often...
Read More
How to Scale Ollama with Two or More GPUs – Parallel Instances for Maximum Performance
Anyone operating a powerful server with two NVIDIA GPUs (such as two RTX A6000s) quickly...
Read More
SongBloom Assistant: Orchestrating Your Own AI Music with Ollama and Tokens
Artificial intelligence is fundamentally changing music production. With tools like SongBloom from...
Read More
Automating Enchanted Children’s Stories with AI & n8n
Imagine receiving a beautifully crafted children’s story by email—complete with vivid...
Read More
NVIDIA’s RTX 5090 Performance Issue: Missing Render Units Discovered
NVIDIA’s Blackwell GPUs are currently facing significant issues due to a hardware defect...
Read More
- Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 3-3
Download and Run Additional Models Locally Since you already downloaded models locally in Phase 4, you can now switch between different models without having to redownload them. To use another already downloaded model, simply stop the current container and restart it with the desired model. First, I stop the current container (if it is still… Read more: Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 3-3 - Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 2-3
Configuring vLLM for Production Deployment (More Complex) For production use, I want to run the container in the background and ensure that it starts automatically after a reboot. Additionally, I mount the Hugging Face cache so that models do not have to be redownloaded every time the container starts. This saves you the time otherwise… Read more: Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 2-3 - Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 1-3
After showing you how to install Ollama, Open WebUI, ComfyUI, and LLaMA Factory on the Gigabyte AI TOP ATOM in my previous posts, now comes something for everyone who needs maximum performance when running Large Language Models: vLLM – a high-performance inference engine specifically designed to run LLMs with maximum throughput and minimal memory consumption.… Read more: Install vLLM on Gigabyte AI TOP ATOM: High-Performance LLM Inference with OpenAI-Compatible API – Part 1-3 - Installing LLaMA Factory on Gigabyte AI TOP ATOM: Fine-tuning Language Models with LoRA and QLoRA – Part 2-2
Phase 7: Start Fine-Tuning Training Before I start the training, I might need to log in to the Hugging Face Hub if the model is gated (has access restrictions). For public models, this is not necessary: Command: huggingface-cli login You will be asked for your Hugging Face token. You can find this in your Hugging… Read more: Installing LLaMA Factory on Gigabyte AI TOP ATOM: Fine-tuning Language Models with LoRA and QLoRA – Part 2-2 - Installing LLaMA Factory on Gigabyte AI TOP ATOM: Fine-tuning Language Models with LoRA and QLoRA – Part 1-2After showing how to install Ollama, Open WebUI, and ComfyUI on the Gigabyte AI TOP ATOM in my previous posts, now comes something for everyone who wants to adapt their own language models and make them individual: LLaMA Factory – an open-source framework that simplifies the fine-tuning of Large Language Models and supports methods such… Read more: Installing LLaMA Factory on Gigabyte AI TOP ATOM: Fine-tuning Language Models with LoRA and QLoRA – Part 1-2
The tutorial offers a clear and practical guide for setting up and running the Tensorflow Object Detection Training Suite. Could…
This works using an very old laptop with old GPU >>> print(torch.cuda.is_available()) True >>> print(torch.version.cuda) 12.6 >>> print(torch.cuda.device_count()) 1 >>>…
Hello Valentin, I will not share anything related to my work on detecting mines or UXO's. Best regards, Maker
Hello, We are a group of students at ESILV working on a project that aim to prove the availability of…