3090显卡Ktransformer本地部署deepseek R1:70B

效果完成视频：

配置参考

64G内存
i9 10100K cpu
3090 gpu

依赖安装

Linux-x86_64 ， gcc, g++ and cmak，cuda

sudo apt-get update
sudo apt-get install gcc g++ cmake ninja-build
方法一：
# CUDA下载地址：
https://developer.nvidia.com/cuda-toolkit-archive
# 下载.run格式的安装包后：
chmod +x xxx.run # 增加执⾏权限
./xxx.run # 运⾏安装包
# cuDNN下载地址：
https://developer.nvidia.com/cudnn
# 先解压， 后将动态链接库和头⽂件放⼊相应⽬录
mv cuda/include/* /usr/local/cuda/include/
chmod +x cuda/lib64/* && mv cuda/lib64/* /usr/local/cuda/lib64/
echo "export LD_LIBRARY_PATH=/usr/local/cuda/lib64/:${LD_LIBRARY_PATH} \n" >> 
~/.bashrc
source ~/.bashrc && ldconfig
方法二：推荐
conda install cudatoolkit==12.1
conda install cudnn==9.7.1

安装cuda

sudo mkdir -p ~/miniconda3
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
pip3 install packaging ninja cpufeature numpy
sudo wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linuxx86_64.sh -O ~/miniconda3/miniconda.sh
sudo bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
sudo rm ~/miniconda3/miniconda.sh
source ~/miniconda3/bin/activate
conda init --all

配置环境

conda create --name ktransformers python=3.10 -y
source ~/.bashrc
conda activate ktransformers 
pip install packaging ninja cpufeature numpy

下载deepseek R1 70B

https://huggingface.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF
推荐5bit量化版本

下载ktransoformer

https://github.com/kvcache-ai/ktransformers

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers

开始安装

# Adding CUDA to PATH
export PATH=/usr/local/cuda-12.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH
export CUDA_PATH=/usr/local/cuda
# 源码安装
# chmod +x ./install.sh
bash install.sh
# 安装包
sudo wget https://github.com/kvcache-ai/ktransformers/releases/download/v0.2.2rc1/ktransformers-0.2.2rc1+cu121torch23avx2-cp310-cp310-linux_x86_64.whl
pip install ktransformers-0.2.1+cu121torch23avx512-cp310-cp310-
linux_x86_64.whl

运行

mkdir DeepSeek-R1-14B-GGUF
cd DeepSeek-R1-14B-GGUF
# 下载GGUF⽂件
sudo wget https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-14BGGUF/blob/main/DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf -O DeepSeek-R1-
14B.gguf
cd .. # Move to repo's root dir
# ⾃动下载模型⽂件
python -m ktransformers