docker containerd 交叉编译 移植到arm64v8

可以把docker移植到嵌入式的aarch64 linux系统中，比如hi3559等芯片

 

一. docker编译

docker以及所依赖的containerd、runc等可能没法自己交叉编译，没有相关文档，并且东西太多太复杂，不过好在docker官方为不同平台提供编译好的程序，其中包括：
containerd、ctr、docker-init、dockerd、containerd-shim、docker、docker-proxy、runc

相关说明：
https://docs.docker.com/engine/install/binaries/

其中包括下载地址：
https://download.docker.com/linux/static/stable/aarch64/

二. 移植iptable

如果系统中没有iptable命令，则需要移植

0.打开内核选项：

Networking support ---> Networking options ---> Network packet filtering framework (Netfilter)
选中Core Netfilter Configuration以及下面所有的，IP: Netfilter Configuration以及下面所有的。

 

iptable又依赖libmnl、libnftnl，需要按顺序编译：

1.libmnl

git clone https://git.netfilter.org/libmnl/
./configure CC=aarch64-himix100-linux-gcc --host=aarch64-himix100-linux --prefix=$PWD/bin --enable-static
make
make install

2.libnftnl

git clone https://git.netfilter.org/libnftnl/
LIBMNL_CFLAGS="-I/home/a/share/libmnl/bin/include" \
LIBMNL_LIBS="-L/home/a/share/libmnl/bin/lib -lmnl" \
./configure CC=aarch64-himix100-linux-gcc --host=aarch64-himix100-linux --prefix=$PWD/bin --enable-static
make
make install

3.iptables

git clone https://git.netfilter.org/iptables/
libmnl_CFLAGS="-I/home/a/share/libmnl/bin/include" \
libmnl_LIBS="-L/home/a/share/libmnl/bin/lib -lmnl" \
libnftnl_CFLAGS="-I/home/a/share/libnftnl/bin/include" \
libnftnl_LIBS="-L/home/a/share/libnftnl/bin/lib -lnftnl" \
./configure CC=aarch64-himix100-linux-gcc --host=aarch64-himix100-linux --prefix=$PWD/bin --enable-static --without-cli
make
make install

最后把 iptables/bin 目录下生成的东西放到板子上。

三. 内核配置

内核需要打开很多选项，具体见此文章最后的docker依赖的内核选项列表。

此列表是用这个脚本检测并生成的 https://github.com/moby/moby/blob/master/contrib/check-config.sh

 

四. 运行服务

1.配置eth0和lo：
ifconfig lo 127.0.0.1 netmask 255.255.255.0

2.挂载cgroup并重启
vi /etc/fstab
在最后一行增加以下配置信息：
none        /sys/fs/cgroup        cgroup    defaults    0    0

3. 把下载的docker安装包里的文件都放在 /usr/bin 下
4. 先执行 containerd &，然后执行 dockerd &

五. 运行镜像

1.下载镜像

一般要在单板上用 docker pull arm64v8/ubuntu 来下载镜像，还要配置好 /etc/docker/daemon.json，如果pull有问题的话可以先用另外的方法下载镜像：

1. 找一个正式的ubuntu系统通过上面的命令下载镜像
2. 用命令 docker save arm64v8/ubuntu > ubuntu_image.tar 来保存镜像，把ubuntu_image.tar放到单板上
3. 在单板上用 docker load < ./ubuntu_image.tar 来导入镜像，这时候 docker image ls 就能看到这个镜像了

2.执行 docker run -it arm64v8/ubuntu /bin/sh 就能运行容器，并进入这个容器的shell环境

 

 

附：docker依赖的内核选项:

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: missing
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: missing
- CONFIG_POSIX_MQUEUE: enabled
 
Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: missing
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled (as module)
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled (as module)
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing