{"id":2262,"date":"2026-05-16T07:26:08","date_gmt":"2026-05-16T07:26:08","guid":{"rendered":"https:\/\/ai-box.eu\/?p=2262"},"modified":"2026-05-16T11:30:17","modified_gmt":"2026-05-16T11:30:17","slug":"tensorrt-llm-pipeline-building-persistent-engines-with-fp16-and-fp8","status":"publish","type":"post","link":"https:\/\/ai-box.eu\/en\/top-story-en\/tensorrt-llm-pipeline-building-persistent-engines-with-fp16-and-fp8\/2262\/","title":{"rendered":"TensorRT-LLM Pipeline: Building Persistent Engines with FP16 and FP8"},"content":{"rendered":"

In the second part<\/a> I set up TensorRT-LLM inside a Docker container and validated with TinyLlama that the pipeline basically runs. In doing so, one important detail came to light: the default API from tensorrt_llm import LLM<\/code> uses the PyTorch backend<\/strong> and does not<\/strong> produce a deployable engine file. For the classic build-once-deploy-many pattern, and in order to try out an architecture that will later also work on Edge-LLM, I need the TensorRT backend<\/strong> and its two-stage build workflow.<\/p>\n

In this part I’ll walk through, step by step together with you, the building of two persistent engines<\/strong> for Qwen2.5-7B-Instruct: once in FP16 as a baseline, once in FP8 to activate the hardware Transformer Engine on the Ada architecture. Both as deployable .engine<\/code> files, both reproducible with build scripts.<\/p>\n

\n
\n

Table of Contents<\/p>\nToggle<\/span><\/path><\/svg><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n