{"id":2255,"date":"2026-05-16T03:22:06","date_gmt":"2026-05-16T03:22:06","guid":{"rendered":"https:\/\/ai-box.eu\/?p=2255"},"modified":"2026-05-16T11:27:50","modified_gmt":"2026-05-16T11:27:50","slug":"tensorrt-llm-on-the-rtx-a6000-ada-preparing-for-the-edge-llm-ecosystem","status":"publish","type":"post","link":"https:\/\/ai-box.eu\/en\/top-story-en\/tensorrt-llm-on-the-rtx-a6000-ada-preparing-for-the-edge-llm-ecosystem\/2255\/","title":{"rendered":"TensorRT-LLM on the RTX A6000 Ada: Preparing for the Edge-LLM Ecosystem"},"content":{"rendered":"

With TensorRT Edge-LLM<\/strong>, NVIDIA has released a C++-only inference framework for embedded platforms. It is designed for the Jetson Thor, DRIVE Thor or the MediaTek CX1 platform. These are exactly the kinds of platforms on which the exciting part of Edge AI will happen over the coming years: language models, vision-language models and vision-language-action models running on local, low-power hardware without any cloud connection. For anyone taking Edge Physical AI<\/strong> and Sovereign AI<\/strong> seriously, this is the next logical step after local inference servers built around classical GPUs. With TensorRT Edge-LLM<\/strong>, NVIDIA has released a C++-only inference framework for embedded platforms. Primarily intended for NVIDIA’s own Jetson Thor<\/strong> and DRIVE AGX Thor<\/strong> platforms, it is already being adopted by partners, including MediaTek<\/strong> for their CX1 SoC<\/strong>, Bosch<\/strong> for the AI-powered Cockpit<\/strong>, and ThunderSoft<\/strong> for the AIBOX platform<\/strong>. These are exactly the kinds of platforms on which the exciting part of Edge AI will happen over the coming years: language models, vision-language models and vision-language-action models running on local, low-power hardware without any cloud connection. For anyone taking Edge Physical AI<\/strong> and Sovereign AI<\/strong> seriously, this is the next logical step after local inference servers built around classical GPUs. The problem: I have neither a Jetson Thor nor a DRIVE Thor.<\/strong> Both are expensive, not easy to get hold of, and outside the reach of a typical home setup. What I do have is an NVIDIA RTX A6000 Ada (SM89)<\/strong> in an Ubuntu server. Edge-LLM doesn’t officially run on it, because discrete GPUs are listed in the support matrix as “unofficial, experimental”. I only have a few first-generation Jetson Nanos with 4 GB of RAM, left over from my robot car projects from the years 2020 to 2021.<\/p>\r\n

\"Donkey<\/a>

Donkey Car – Jetson Nano<\/p><\/div>\r\n

Rather than waiting for the hardware question to resolve itself, I decided to work through the conceptually related big sibling project<\/strong>: TensorRT-LLM<\/strong>. It shares almost its entire architecture with Edge-LLM, but runs on datacenter GPUs \u2014 and therefore also on a professional workstation card like my Ada. That way I build up the skills that will later transfer 1:1 to the Jetson Thor, as soon as the hardware becomes affordable in my hobby environment. This blog post is part 1 of a four-part series<\/strong> in which I document the complete journey: motivation, installation, build pipeline with different quantization formats, and finally the real measurements together with all the pitfalls I ran into along the way. I first became aware of the topic through this article from NVIDIA: Accelerating LLM and VLM Inference for Automotive and Robotics with NVIDIA TensorRT Edge-LLM<\/a><\/p>\r\n

\n
\n

Table of Contents<\/p>\nToggle<\/span><\/path><\/svg><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n