LMDeploy

Toolkit for compressing and serving LLMs
Inference serverInference serverApache-2.0Self-hostedPythonAdvanced
🛡 Published security advisories
A high count usually means the project is heavily examined. A low count usually means it is little examined — not that it is safe.
No public activity signals for this tool — it has no open repository we can measure.
OSAI Pulse ⓘ ★★★★★★★★★★ /100 signals tracked
🐳 Docker pulls 📦 PyPI downloads / month 📦 npm downloads / month 🚀 Latest release ·
Visit website →View on GitHub

What is LMDeploy?

LMDeploy is a toolkit for compressing, quantizing and serving LLMs with high request throughput via its TurboMind engine.

Why people choose LMDeploy

LMDeploy at a glance

CategoryInference server
TypeInference server
LicenseApache-2.0
Runs locallySelf-hosted
Built withPython
Skill levelAdvanced
Best forteams optimizing quantized serving

Open-source alternatives to LMDeploy

Other open-source inference server tools worth comparing:

vLLMHigh-throughput serving for productionTGIHugging Face's production text serverSGLangFast serving with structured outputsAphrodite EngineHigh-throughput LLM servingTensorRT-LLMPeak throughput on NVIDIA GPUsOpenLLMServe any open model as an OpenAI API in one commandKTransformersRun huge MoE models on one consumer GPURay ServeScale model serving across a clusterBentoMLPackage any model into a production API

LMDeploy head-to-head

LMDeploy vs vLLMLMDeploy vs TGILMDeploy vs SGLangLMDeploy vs Aphrodite EngineLMDeploy vs TensorRT-LLMLMDeploy vs OpenLLMLMDeploy vs KTransformersLMDeploy vs Ray ServeLMDeploy vs BentoML

FAQ

Is LMDeploy free?

LMDeploy is free and open-source (Apache-2.0 license), so you can use, self-host and modify it at no cost.

Can I run LMDeploy locally?

Yes. LMDeploy is designed to run on your own machine or server, keeping your data private.

What is the best alternative to LMDeploy?

Popular open-source alternatives include vLLM, TGI, SGLang. See the comparisons above to choose.

Explore more open-source AI

Browse the full directory of open-source AI tools, models and projects — updated daily.

Browse all tools →