ubuntu22.04搭建qt交叉编译环境

本文主要介绍在ubuntu22.04主机上搭建qt交叉编译环境教程，方便在上位机开发下位机应用程序

一、编译环境和开发版本介绍

• 小编的主机电脑环境介绍：

$ uname -a
Linux yz-MS-7E06 6.8.0-45-generic #45~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 11 15:25:05 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

• 下位机开发板：reComputer J4012 - Edge AI Computer with NVIDIA® Jetson™ Orin™ NX 16GB
• 开发板使用的SDK版本JetPack 6.1 ，对应的Linux Kernel 5.15

二、准备下载的文件

• 准备下载qt源代码，小编使用的是Qt5.15.15版本，下载链接
  
• 准备下位机交叉编译工具链，下载链接
  
• 准备下位机对应版本的rootfs文件，如下图所示：
  
• 准备安装的qtCreate软件版本是10.0.2，下载链接
  
• 下载qt编译的配置文件在附件
• 在主目录home文件夹新建qt5.15.15文件夹，命令如下

$ pwd
/home/yz/qt5.15.15
$ ls
aarch64--glibc--stable-2022.08-1               qt-everywhere-opensource-src-5.15.15.tar.xz
aarch64--glibc--stable-2022.08-1.tar.bz2       qt-everywhere-src-5.15.15
linux-jetson-nx                                sysroot
linux-jetson-nx.zip                            Tegra_Linux_Sample-Root-Filesystem_R36.4.0_aarch64.tbz2
qt-creator-opensource-linux-x86_64-10.0.2.run

三、配置和编译

• 将linux-jetson-nx文件夹移动到device文件夹内

$ mv linux-jetson-nx ./qt-everywhere-src-5.15.15/qtbase/mkspecs/devices/

• 调整rootfs里面的软链接由直接路径为相对路径，运行下面的python代码

#!/usr/bin/env python
import sys
import os
# Take a sysroot directory and turn all the abolute symlinks and turn them into
# relative ones such that the sysroot is usable within another system.
if len(sys.argv) != 2:
    print("Usage is " + sys.argv[0] + "")
    sys.exit(1)

topdir = sys.argv[1]
topdir = os.path.abspath(topdir)

def handlelink(filep, subdir):
    link = os.readlink(filep)
    if link[0] != "/":
        return
    if link.startswith(topdir):
        return
    #print("Replacing %s with %s for %s" % (link, topdir+link, filep))
    print("Replacing %s with %s for %s" % (link, os.path.relpath(topdir+link, subdir), filep))
    os.unlink(filep)
    os.symlink(os.path.relpath(topdir+link, subdir), filep)

for subdir, dirs, files in os.walk(topdir):
    for f in files:
        filep = os.path.join(subdir, f)
        if os.path.islink(filep):
            #print("Considering %s" % filep)
            handlelink(filep, subdir)

运行如下命令：

$ ./sysroot-relativelinks.py ./sysroot

• 添加交叉工具链到全局路径中

vim ~/.bashrc
export PATH=$PATH:~/qt5.15.15/aarch64--glibc--stable-2022.08-1/bin
source ~/.bashrc

• 新建文件夹qt5build，执行脚本命令

#!/bin/bash
echo "start build"
../qt-everywhere-src-5.15.15/configure -release -opengl es2 \
        -device linux-jetson-nx \
        -device-option CROSS_COMPILE=aarch64-linux- \
        -sysroot $HOME/qt5.15.15/sysroot \
        -prefix /usr/local/qt5 \
        -opensource -confirm-license \
        -skip qtscript \
        -skip wayland \
        -skip qtwebengine \
        -force-debug-info -skip qtlocation \
        -nomake examples \
        -nomake tests \
        -make libs \
        -pkg-config -no-use-gold-linker -v

• 运行编译命令 make -j24，大概25分钟左右编译完毕
• 执行安装命令 make install

四、编译中遇到的问题

小编在编译中碰见如下的问题，希望给大家提供一种解决方案

• 出现 Scrt1.o: No such file or directory 报错退出，具体错误信息如下

> /home/yz/qt5.15.15/aarch64--glibc--stable-2022.08-1/bin/../lib/gcc/aarch64-buildroot-linux-gnu/11.3.0/../../../../aarch64-buildroot-linux-gnu/bin/ld: cannot find Scrt1.o: No such file or directory
> /home/yz/qt5.15.15/aarch64--glibc--stable-2022.08-1/bin/../lib/gcc/aarch64-buildroot-linux-gnu/11.3.0/../../../../aarch64-buildroot-linux-gnu/bin/ld: cannot find crti.o: No such file or directory
> > /home/yz/qt5.15.15/aarch64--glibc--stable-2022.08-1/bin/../lib/gcc/aarch64-buildroot-linux-gnu/11.3.0/../../../../aarch64-buildroot-linux-gnu/bin/ld: cannot find Scrt1.o: No such file or directory

问题分析:在sysroot中找不到对应的链接，可以进行手动链接
cd ./sysroot/usr/lib
ln -sf ./aarch64-linux-gnu/crti.o crti.o
ln -sf ./aarch64-linux-gnu/crt1.o crt1.o
ln -sf ./aarch64-linux-gnu/crtn.o crtn.o
ln -sf ./aarch64-linux-gnu/Scrt1.o Scrt1.o

五、上位机qtCreat配置

• 选择Kit界面，配置Sysroot, Compiler, QtVersion 分别对应如下路径：

/home/yz/qt5.15.15/sysroot
/home/yz/qt5.15.15/aarch64--glibc--stable-2022.08-1/bin/aarch64-linux-gcc
/home/yz/qt5.15.15/aarch64--glibc--stable-2022.08-1/bin/aarch64-linux-g++
/home/yz/qt5.15.15/sysroot/usr/local/qt5/bin/qmake

• 新建一个测试工程，生成的qmake编译命令如下
  
• 从上图可以看出对应配置是 -spec devices/linux-jetson-nx
• 通过file命令查看编译的信息如下：

$ file untitled 
untitled: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

六、一个cuda的交叉编译工程例子

• 项目工程文件的结构

$ tree
.
├── hello.cu
├── hello.cuh
├── main.cpp
├── mainwindow.cpp
├── mainwindow.h
├── mainwindow.ui
└── MemsTest.pro

• qmake的配置

QT       += core gui
greaterThan(QT_MAJOR_VERSION, 4): QT += widgets
CONFIG += c++11
DEFINES += QT_DEPRECATED_WARNINGS
#DEFINES += QT_DISABLE_DEPRECATED_BEFORE=0x060000    # disables all the APIs deprecated before Qt 6.0.0
SOURCES += \
    main.cpp \
    mainwindow.cpp
HEADERS += \
    hello.cuh \
    mainwindow.h
FORMS += \
    mainwindow.ui
# Default rules for deployment.
target.path = /opt/$${TARGET}/bin
!isEmpty(target.path): INSTALLS += target
###########################CUDA CONFIG####################################
#配置cuda的头文件和相关库的路径
LIBS += -L'/usr/local/cuda-11.4/targets/aarch64-linux/lib' -lcudart
INCLUDEPATH += '/usr/local/cuda-11.4/targets/aarch64-linux/include'
DEPENDPATH += '/usr/local/cuda-11.4/targets/aarch64-linux/lib'
INCLUDEPATH += './'
CUDA_DIR="/usr/local/cuda-11.4"
CUDA_SDK="/usr/local/cuda-11.4"
QMAKE_LIBDIR += $$CUDA_DIR/targets/aarch64-linux/lib
NVCCFLAGS = --use_fast_math
CUDA_INC = $$join(INCLUDEPATH,'" -I"','-I"','"')
# MSVCRT link option (static or dynamic, it must be the same with your Qt SDK link option)
CUDA_OBJECTS_DIR = ./
CUDA_SOURCES+=./hello.cu
CONFIG(debug, debug|release) {
    # Debug mode
    cuda_d.input = CUDA_SOURCES
    cuda_d.output = $$CUDA_OBJECTS_DIR/${QMAKE_FILE_BASE}.o
    cuda_d.commands = $$CUDA_DIR/bin/nvcc --device-debug --debug -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 -ccbin $$QMAKE_CXX \
                           -c ${QMAKE_FILE_NAME} -o ${QMAKE_FILE_OUT}
    cuda_d.dependency_type = TYPE_C
    QMAKE_EXTRA_COMPILERS += cuda_d
}
else {
# Release mode
    cuda.input = CUDA_SOURCES
    cuda.output = $$CUDA_OBJECTS_DIR/${QMAKE_FILE_BASE}.o
    cuda.commands = $$CUDA_DIR/bin/nvcc --cudart=static -ccbin $$QMAKE_CXX -gencode arch=compute_86,code=sm_86 -gencode arch=compute_86,code=compute_86 \
                        -c ${QMAKE_FILE_NAME} -o ${QMAKE_FILE_OUT}
    cuda.dependency_type = TYPE_C
    QMAKE_EXTRA_COMPILERS += cuda
}

• hello.cu文件

#include 
#include 
#include "hello.cuh"

__global__ void hello(){
    int i = blockDim.x * blockIdx.x + threadIdx.x;
    printf("hello %d\n",i);
}
int test_cuda(void){
    hello<<<1, 3>>>();
    cudaDeviceSynchronize();
    printf("Done \n");
    return 0;
}

• main.cpp文件内容

#include "mainwindow.h"

#include 
#include "hello.cuh"
#include 
#include 
#include 

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);

    test_cuda();

    int runtimeVersion;
    int deviceCount;
    cudaError_t err = cudaRuntimeGetVersion(&runtimeVersion);
    if (err == cudaSuccess) {
        std::cout << "CUDA Runtime Version: " << (runtimeVersion / 1000) << "."
                  << (runtimeVersion % 100) << std::endl;
    } else {
        std::cerr << "Failed to get CUDA runtime version: "
                  << cudaGetErrorString(err) << std::endl;
    }
    err = cudaGetDeviceCount(&deviceCount);
    if (err == cudaSuccess) {
        std::cout << "Detected " << deviceCount << " CUDA Capable device(s)."
                  << std::endl;
    } else {
        std::cerr << "Error: " << cudaGetErrorString(err)
                  << " when trying to get CUDA device count." << std::endl;
    }

    MainWindow w;
    w.show();
    return a.exec();
}

• 编译成果物属性

 file MemsTest 
MemsTest: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

七、解决SDK编译环境移植问题

当将编译环境sysroot拷贝到另外的电脑开发环境中，需要修改相关的qt.conf来解决qmake识别不到版本问题
在sysroot/usr/local/qt5/bin配置qt.conf配置如下：

[Paths]
Prefix=/usr/local/qt5
HostPrefix=~/test/sysroot/usr/local/qt5
Sysroot=~/test/sysroot
TargetSpec=devices/linux-jetson-nx

注意上面的HostPrefix和Sysroot与SDK路径一致

八、参考

https://gitcode.com/YYRAN_ZZU/QtJetsonCrossCompilationExample.git