TensorRT-LLM in Numbers: FP16 vs. FP8 on the RTX A6000 Ada
In the first three posts of this little series I explained why I’m tackling TensorRT-LLM on...
Read More
Author: Maker
TensorRT-LLM Pipeline: Building Persistent Engines with FP16 and FP8
In the second part I set up TensorRT-LLM inside a Docker container and validated with TinyLlama...
Read More
TensorRT-LLM on Ubuntu 24.04: Setup with Docker and Helper Scripts
In the previous part I described why I’m tackling TensorRT-LLM on my RTX A6000 Ada: as...
Read More
TensorRT-LLM on the RTX A6000 Ada: Preparing for the Edge-LLM Ecosystem
With TensorRT Edge-LLM, NVIDIA has released a C++-only inference framework for embedded platforms....
Read More
ESP-IDF v5.5.4 einrichten und ESP-Claw bauen – Schritt für Schritt
After I introduced the vision in Part 1 of my ESP-Claw series – bringing a local AI agent to life...
Read MoreMein neues Buch
Souveräne KI
Strategie und Aufbau der eigenen Open-Source-Infrastruktur
Eigene LLMs betreiben, unabhängig von der Cloud. Von der Hardware bis zum laufenden System.
Jetzt bei Amazon bestellenRoboter-Autos mit dem ESP32
Bauen, programmieren, steuern
Vom ersten Chassis bis zur WLAN-Steuerung. Schritt für Schritt zum eigenen Roboter-Auto.
Jetzt bei Amazon bestellenBuch und E-Book, Rheinwerk Verlag
Werbung / Affiliate-Link
- Local Voice Agent: Wiring ASR, LLM and TTS into a Loop with NVIDIA Pipecat Locally
Every building block is now available locally: ASR (Parakeet, Canary), TTS (Magpie), an LLM via my Ollama server and the orchestrator (NAT, Part 5). Now I’m connecting them into a continuous, interruptible speech loop. This will be my first small local voice agent. What I’m aiming for is a kind of general-purpose agent: I speak,… Read more: Local Voice Agent: Wiring ASR, LLM and TTS into a Loop with NVIDIA Pipecat Locally - NVIDIA NeMo Agent Toolkit (NAT): set up the agent orchestrator locally
The two halves of a voice agent are in place: with Parakeet (Part 2) and Canary (Part 3) the agent listens, with Magpie (Part 4) it answers. What’s still missing is the brain: the layer that turns recognized text into a decision and triggers the matching answer or action. That’s exactly what I take on… Read more: NVIDIA NeMo Agent Toolkit (NAT): set up the agent orchestrator locally - NVIDIA Magpie TTS locally: German speech output as a microservice
The speech recognition is in place: with Parakeet (Part 2) and Canary (Part 3) I covered the input direction, speech to text, natively on NVIDIA. Now comes the opposite direction, the voice output. In this post I run NVIDIA Magpie TTS as a local NIM and have German text read aloud naturally. This is the… Read more: NVIDIA Magpie TTS locally: German speech output as a microservice - NVIDIA Canary locally: multilingual speech recognition and translation as a NIM
In Part 2 I set up Parakeet as a streaming-capable ASR NIM for German and ran it as a live service with low latency. In this post I take on the sister model: NVIDIA Canary as a NIM. Canary does not shine at latency but at accuracy, and it can do something Parakeet cannot: translate.… Read more: NVIDIA Canary locally: multilingual speech recognition and translation as a NIM - NVIDIA NIM locally: running German speech recognition as a microservice
In my last post I ran NVIDIA Nemotron ASR Streaming directly with NeMo locally. That was the “bare” route via the framework. In this post I go one step further and dive into NVIDIA NIM. NIM stands for NVIDIA Inference Microservices, the microservice variant NVIDIA uses to ship its models as ready-made, optimized containers. The… Read more: NVIDIA NIM locally: running German speech recognition as a microservice
The tutorial offers a clear and practical guide for setting up and running the Tensorflow Object Detection Training Suite. Could…
This works using an very old laptop with old GPU >>> print(torch.cuda.is_available()) True >>> print(torch.version.cuda) 12.6 >>> print(torch.cuda.device_count()) 1 >>>…
Hello Valentin, I will not share anything related to my work on detecting mines or UXO's. Best regards, Maker
Hello, We are a group of students at ESILV working on a project that aim to prove the availability of…