nerf-slam论文复现
nerf-slam实现三维重建 详细的在我文档里面(有图片步骤)
Table of Contents
InstallDownload DatasetsRunCitationLicenseAcknowledgmentsContact
Install
Clone repo with submodules:
git clone https://github.com/ToniRV/NeRF-SLAM.git --recurse-submodules
git submodule update --init --recursive
From this point on, use a virtual environment... Install torch (see here for other versions):
# CUDA 11.3
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
Pip install requirements:
pip install -r requirements.txt
pip install -r ./thirdparty/gtsam/python/requirements.txt
Compile ngp (you need cmake>3.22):
先升级cmake到3.22版本
查看当前cmake版本
cmake --version
下载:
wget https://cmake.org/files/v3.22/cmake-3.22.0-rc2-linux-x86_64.tar.gz --no-check-certificate
解压
tar -zxvf cmake-3.22.0-rc2-linux-x86_64.tar.gz
编译安装
cd cmake-3.22.0-rc2-linux-x86_64/
cmake .
make
sudo make install
安装3.9版本
wget https://cmake.org/files/v3.9/cmake-3.9.2.tar.gz --no-check-certificate
解压、安装新版本
tar -xvf cmake-3.9.2.tar.gz
cd cmake-3.9.2
./configure
sudo make && make install
sudo make
sudo make install
cmake --version
hash -r
cmake --version
安装3.26版本
从https://cmake.org/download/官网下载新版本
解压、安装新版本
tar -xvf cmake-3.26.3.tar.gz
cd cmake-3.26.3/
./configure
sudo make
sudo make install
cmake --version
hash -r
cmake --version
进入nerf-slam目录
cmake ./thirdparty/instant-ngp -B build_ngp
报Could NOT find GLEW (missing: GLEW_INCLUDE_DIRS GLEW_LIBRARIES)错误
sudo apt install libglew-dev
重新
cmake ./thirdparty/instant-ngp -B build_ngp
cmake --build build_ngp --config RelWithDebInfo -j
Compile gtsam and enable the python wrapper:
cmake ./thirdparty/gtsam -DGTSAM_BUILD_PYTHON=1 -B build_gtsam
报Could NOT find Boost (missing: Boost_INCLUDE_DIR serialization system filesystem thread program_options date_time timer chrono regex) (Required is at least version "1.65")错误
这里介绍使用源代码编译安装的方式，以Boost_1.73.0为例
1. 下载源代码
wget https://boostorg.jfrog.io/artifactory/main/release/1.73.0/source/boost_1_73_0.tar.bz2 --no-check-certificate
2. 编译安装
tar xvf boost_1_73_0.tar.bz2
cd boost_1_73_0
./bootstrap.sh
sudo ./b2 --buildtype=complete install
boost有大量的库其实可以选择安装指定的库，这里为了方便我选择完全安装。
安装boost库时我直接按照默认的路径安装，最后会在/usr/local/lib目录下安装所有编译的libboost*库。如果需要指定路径，可以在编译时添加"–prefix=xxx"的路径参数，这样的话在安装后还需要手动将该路径添加到环境变量中。
如果系统中有多个python版本，可以通过"–with-python=python*"来指定python版本，默认情况下使用系统默认的版本。
装好boost之后重新
cmake ./thirdparty/gtsam -DGTSAM_BUILD_PYTHON=1 -B build_gtsam
cmake --build build_gtsam --config RelWithDebInfo -j
怀疑是eigen库的问题
eigen安装
命令安装
sudo apt-get install libeigen3-dev
重新
cmake --build build_gtsam --config RelWithDebInfo -j
依然报错
还是不行
把文件夹thirdparty里面的gtsam注释,然后git clone更新
git clone https://github.com/ToniRV/gtsam-1
对文件夹进行重命名
回到主目录
重新
cmake --build build_gtsam --config RelWithDebInfo -j
cd build_gtsam
make python-install
make python-test
报ImportError: libboost_chrono.so.1.73.0: cannot open shared object file: No such file or directory错误
首先确定电脑是否已经安装libboost_chrono.so,输入命令locate libboost_chrono
把缺少文件的文件夹的都复制上
sudo cp -r /usr/local/lib/libboost_chrono.so.1.73.0 /usr/local/lib/cmake/boost_chrono-1.73.0/
sudo cp -r /usr/local/lib/libboost_chrono.so.1.73.0 /home/user/linzejun01/cmake-3.26.3/Tests/RunCMake/FindBoost/CMakePackageFixtures/boost_chrono-1.70.0/
这样还是不行
执行：sudo ldconfig /usr/local/lib/
即可。路径替换为boost安装路径。
重新运行
make python-test
进入主目录
Install:
sudo python3 setup.py install
Download Sample Data
This will just download one of the replica scenes:
./scripts/download_replica_sample.bash
将下载的数据集解压
Run
python3 ./examples/slam_demo.py --dataset_dir=./inzejun_Datasets/office0 --dataset_name=nerf --buffer=100 --slam --parallel_run --img_stride=2 --fusion='nerf' --multi_gpu --gui
报ModuleNotFoundError: No module named 'colored_glog'错误
pip install colored_glog
重新运行
python3 ./examples/slam_demo.py --dataset_dir=./inzejun_Datasets/office0 --dataset_name=nerf --buffer=100 --slam --parallel_run --img_stride=2 --fusion='nerf' --multi_gpu --gui
路径少了一个l
python3 ./examples/slam_demo.py --dataset_dir=./linzejun_Datasets/office0 --dataset_name=nerf --buffer=100 --slam --parallel_run --img_stride=2 --fusion='nerf' --multi_gpu --gui
报AttributeError: type object 'gtsam.gtsam.Pose3' has no attribute 'identity'错
进入报错的代码:
将报错的代码首字母改为大写
This repo also implements Sigma-Fusion: just change --fusion='sigma' to run that.
/home/user/linzejun01/linzejun_mutiply_view01/NeRF-SLAM/linzejun_Datasets/office0
FAQ
GPU Memory
This is a GPU memory intensive pipeline, to monitor your GPU usage, I'd recommend to use nvitop. Install nvitop in a local env:
pip3 install --upgrade nvitop
Keep it running on a terminal, and monitor GPU memory usage:
nvitop --monitor
If you consistently see "out-of-memory" errors, you may either need to change parameters or buy better GPUs :). The memory consuming parts of this pipeline are:
Frame to frame correlation volumes (but can be avoided using on-the-fly correlation computation).Volumetric rendering (intrinsically memory intensive, tricks exist, but ultimately we need to move to light fields or some better representation (OpenVDB?)).
Installation issues
Gtsam not working: check that the python wrapper is installed, check instructions here: gtsam_python. Make sure you use our gtsam fork, which exposes more of gtsam's functionality to python.Gtsam's dependency is not really needed, I just used to experiment adding IMU and/or stereo cameras, and have an easier interface to build factor-graphs. This didn't quite work though, because the network seemed to have a concept of scale, and it didn't quite work when updating poses/landmarks and then optical flow.Somehow the parser converts this to const std::vector
Did you forget to `#include
Citation
@article{rosinol2022nerf,
title={NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields},
author={Rosinol, Antoni and Leonard, John J and Carlone, Luca},
journal={arXiv preprint arXiv:2210.13641},
year={2022}
}
License
This repo is BSD Licensed. It reimplements parts of Droid-SLAM (BSD Licensed). Our changes to instant-NGP (Nvidia License) are released in our fork of instant-ngp (branch feature/nerf_slam) and added here as a thirdparty dependency using git submodules.
Acknowledgments
This work has been possible thanks to the open-source code from Droid-SLAM and Instant-NGP, as well as the open-source datasets Replica and Cube-Diorama.
Contact
I have many ideas on how to improve this approach, but I just graduated so I won't have much time to do another PhD... If you are interested in building on top of this, feel free to reach out :) [email protected]