TensorRT-LLM
NVIDIA’s open-source library for optimizing and accelerating Large Language Model inference on GPUs.
Why consider TensorRT-LLM?
TensorRT-LLM is NVIDIA’s official open-source inference acceleration library for LLMs on NVIDIA GPUs. Built on TensorRT deep learning compilers, it implements FP8/INT4 quantization, in-flight batching, paged KV caching, and multi-GPU tensor parallelism for massive inference throughput.
Learn TensorRT-LLM by building
Practical setup notes, real use cases, and copy-ready examples in one focused guide.
In this guide2 sections
Overview of TensorRT-LLM
TensorRT-LLM maximizes token generation speeds on NVIDIA H100, A100, and RTX GPUs with custom CUDA kernels.
Quickstart
pip install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.comTensorRT-LLM is licensed under the Apache License Version 2.0.
Related tools
More options with a similar category or technology profile.
diskus
Minimal, fast alternative to du -sh written in Rust using multi-threaded directory traversal.
peco
Simplistic interactive filtering tool for Unix pipelines, process lists, and file trees.
Dapr CLI
Command-line tool for managing Dapr distributed application runtime environments and sidecars.
Freeze
Generate beautiful image screenshots and SVGs of code snippets and terminal outputs.