containerd中文翻译系列(十九)cri插件

cri插件包含的内容比较多，阅读之前请深呼吸三次、三次、三次。

CRI 插件的架构

本小节介绍了 containerd 的 cri 插件的架构。

该插件是 Kubernetes 容器运行时接口（CRI） 的实现。Containerd与Kubelet在同一个节点上运行。containerd内部的cri插件处理来自Kubelet的所有CRI服务请求，并使用containerd内部机制来管理容器和容器映像。

cri 插件使用 containerd 管理整个容器生命周期和所有容器镜像。如下所示，cri\通过 CNI（另一个 CNCF 项目）管理 pod 网络。

让我们以 Kubelet 创建单容器 pod 为例，演示 cri 插件是如何工作的：

• Kubelet 通过 CRI 运行时服务 API 调用 cri 插件来创建 pod；
• cri 创建 pod 的网络命名空间，然后使用 CNI 对其进行配置；
• cri 使用 containerd 内部创建并启动一个特殊的暂停容器（沙盒容器），并将该容器置于 pod 的 cgroups 和命名空间内（为简洁起见，省略步骤）
• 如果节点上没有镜像，cri会进一步使用containerd拉取镜像；
• 然后，Kubelet 通过 CRI 运行时服务 API 调用 cri ，使用拉取的容器镜像在 pod 内创建并启动应用容器；
• 最后，"cri "会使用内部containerd 来创建应用容器，将其放入 pod 的 cgroups 和命名空间中，然后启动 pod 的新应用容器。
  完成这些步骤后，一个 pod 及其相应的应用容器就创建完成并开始运行了。

CRI 插件配置指南

这一章节介绍 CRI 插件配置。
CRI 插件配置是 containerd 配置（默认
路径： /etc/containerd/config.toml`）。

了解有关 containerd 配置的更多信息。，请参阅 here

请注意，[plugins. "io.containerd.grpc.v1.cri"] 部分是 CRI 特有的、其他 containerd 客户端（如 ctr、nerdctl 和 Docker/Moby）无法识别。

基本配置

cgroup 驱动程序

容器 d 和 Kubernetes 默认使用传统的 cgroupfs 驱动程序来管理 cgroup、
建议在基于 systemd 的主机上使用 systemd 驱动程序，以符合
cgroups 的"单写入器 "规则。

要配置 containerd 以使用 systemd 驱动程序，请在 /etc/containerd/config.toml中设置以下选项：

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

除 containerd 外，还必须配置 KubeletConfiguration 以使用 “systemd” cgroup 驱动程序。
KubeletConfiguration" 通常位于 /var/lib/kubelet/config.yaml:

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: "systemd"

kubeadm 用户还应参阅 kubeadm 文档。

注意：Kubernetes v1.28 支持自动检测 cgroup 驱动程序，
作为一项 alpha 功能。启用 KubeletCgroupDriverFromCRI kubelet 功能后，kubelet 会自动从 CRI运行时检测到 cgroup 驱动程序，因此不需要上述的 KubeletConfiguration 配置步骤。
在确定 cgroup 驱动程序时，containerd 会使用基于 runc 的运行时中的 SystemdCgroup 设置。
从默认运行时类开始。如果没有配置基于 runc 的运行时类，containerd
则依赖于基于确定 systemd 是否正在运行的自动检测。
请注意，所有基于 runc 的运行时类都应配置为具有
相同的 SystemdCgroup 设置，以避免意外行为。
containerd v2.0 及更高版本中支持为 kubelet 自动配置cgroup驱动程序的功能。

快照程序

默认快照器设置为 overlayfs（类似于 Docker 的 overlay2 存储驱动程序）：

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"

有关其他支持的快照器，请参阅 此处。

运行时类

下面的示例将自定义运行时注册到 containerd 中：

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd]
  default_runtime_name = "crun"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
    # crun: https://github.com/containers/crun
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.crun.options]
        BinaryName = "/usr/local/bin/crun"
    # gVisor: https://gvisor.dev/
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor]
      runtime_type = "io.containerd.runsc.v1"
    # Kata Containers: https://katacontainers.io/
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
      runtime_type = "io.containerd.kata.v2"

此外，您还必须通过集群管理员角色将以下 RuntimeClass 资源安装到集群中：

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: crun
handler: crun
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: gvisor
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata

要将运行时类应用到 pod，请设置 .spec.runtimeClassName:

apiVersion: v1
kind: Pod
spec:
  runtimeClassName: crun

另请参阅 Kubernetes 文档。

完整配置

各配置项的说明和默认值如下：

# Use config version 2 to enable new configuration fields.
# Config file is parsed as version 1 by default.
# Version 2 uses long plugin names, i.e. "io.containerd.grpc.v1.cri" vs "cri".
version = 2

# The 'plugins."io.containerd.grpc.v1.cri"' table contains all of the server options.
[plugins."io.containerd.grpc.v1.cri"]

  # disable_tcp_service disables serving CRI on the TCP server.
  # Note that a TCP server is enabled for containerd if TCPAddress is set in section [grpc].
  disable_tcp_service = true

  # stream_server_address is the ip address streaming server is listening on.
  stream_server_address = "127.0.0.1"

  # stream_server_port is the port streaming server is listening on.
  stream_server_port = "0"

  # stream_idle_timeout is the maximum time a streaming connection can be
  # idle before the connection is automatically closed.
  # The string is in the golang duration format, see:
  #   https://golang.org/pkg/time/#ParseDuration
  stream_idle_timeout = "4h"

  # enable_selinux indicates to enable the selinux support.
  enable_selinux = false

  # selinux_category_range allows the upper bound on the category range to be set.
  # if not specified or set to 0, defaults to 1024 from the selinux package.
  selinux_category_range = 1024

  # sandbox_image is the image used by sandbox container.
  sandbox_image = "registry.k8s.io/pause:3.9"

  # stats_collect_period is the period (in seconds) of snapshots stats collection.
  stats_collect_period = 10

  # enable_tls_streaming enables the TLS streaming support.
  # It generates a self-sign certificate unless the following x509_key_pair_streaming are both set.
  enable_tls_streaming = false

  # tolerate_missing_hugetlb_controller if set to false will error out on create/update
  # container requests with huge page limits if the cgroup controller for hugepages is not present.
  # This helps with supporting Kubernetes <=1.18 out of the box. (default is `true`)
  tolerate_missing_hugetlb_controller = true

  # ignore_image_defined_volumes ignores volumes defined by the image. Useful for better resource
	# isolation, security and early detection of issues in the mount configuration when using
	# ReadOnlyRootFilesystem since containers won't silently mount a temporary volume.
  ignore_image_defined_volumes = false

  # netns_mounts_under_state_dir places all mounts for network namespaces under StateDir/netns
  # instead of being placed under the hardcoded directory /var/run/netns. Changing this setting
  # requires that all containers are deleted.
  netns_mounts_under_state_dir = false

  # 'plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming' contains a x509 valid key pair to stream with tls.
  [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
    # tls_cert_file is the filepath to the certificate paired with the "tls_key_file"
    tls_cert_file = ""

    # tls_key_file is the filepath to the private key paired with the "tls_cert_file"
    tls_key_file = ""

  # max_container_log_line_size is the maximum log line size in bytes for a container.
  # Log line longer than the limit will be split into multiple lines. -1 means no
  # limit.
  max_container_log_line_size = 16384

  # disable_cgroup indicates to disable the cgroup support.
  # This is useful when the daemon does not have permission to access cgroup.
  disable_cgroup = false

  # disable_apparmor indicates to disable the apparmor support.
  # This is useful when the daemon does not have permission to access apparmor.
  disable_apparmor = false

  # restrict_oom_score_adj indicates to limit the lower bound of OOMScoreAdj to
  # the containerd's current OOMScoreAdj.
  # This is useful when the containerd does not have permission to decrease OOMScoreAdj.
  restrict_oom_score_adj = false

  # max_concurrent_downloads restricts the number of concurrent downloads for each image.
  max_concurrent_downloads = 3

  # disable_proc_mount disables Kubernetes ProcMount support. This MUST be set to `true`
  # when using containerd with Kubernetes <=1.11.
  disable_proc_mount = false

  # unset_seccomp_profile is the seccomp profile containerd/cri will use if the seccomp
  # profile requested over CRI is unset (or nil) for a pod/container (otherwise if this field is not set the
  # default unset profile will map to `unconfined`)
    # Note: The default unset seccomp profile should not be confused with the seccomp profile
    # used in CRI when the runtime default seccomp profile is requested. In the later case, the
    # default is set by the following code (https://github.com/containerd/containerd/blob/main/contrib/seccomp/seccomp_default.go).
    # To summarize, there are two different seccomp defaults, the unset default used when the CRI request is
    # set to nil or `unconfined`, and the default used when the runtime default seccomp profile is requested.
  unset_seccomp_profile = ""

  # enable_unprivileged_ports configures net.ipv4.ip_unprivileged_port_start=0
  # for all containers which are not using host network
  # and if it is not overwritten by PodSandboxConfig
  # Note that currently default is set to disabled but target change it in future, see:
  #   [k8s discussion](https://github.com/kubernetes/kubernetes/issues/102612)
  enable_unprivileged_ports = false

  # enable_unprivileged_icmp configures net.ipv4.ping_group_range="0 2147483647"
  # for all containers which are not using host network, are not running in user namespace
  # and if it is not overwritten by PodSandboxConfig
  # Note that currently default is set to disabled but target change it in future together with enable_unprivileged_ports
  enable_unprivileged_icmp = false

  # enable_cdi enables support of the Container Device Interface (CDI)
  # For more details about CDI and the syntax of CDI Spec files please refer to
  # https://tags.cncf.io/container-device-interface.
  # TODO: Deprecate this option when either Dynamic Resource Allocation(DRA)
  # or CDI support for the Device Plugins are graduated to GA.
  # `Dynamic Resource Allocation` KEP:
  # https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3063-dynamic-resource-allocation
  # `Add CDI devices to device plugin API` KEP:
  # https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4009-add-cdi-devices-to-device-plugin-api
  enable_cdi = true

  # cdi_spec_dirs is the list of directories to scan for CDI spec files
  # For more details about CDI configuration please refer to
  # https://tags.cncf.io/container-device-interface#containerd-configuration
  cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

  # drain_exec_sync_io_timeout is the maximum duration to wait for ExecSync API'
  # IO EOF event after exec init process exits. A zero value means there is no
  # timeout.
  #
  # The string is in the golang duration format, see:
  #    https://golang.org/pkg/time/#ParseDuration
  #
  # For example, the value can be '5h', '2h30m', '10s'.
  drain_exec_sync_io_timeout = "0s"

  # 'plugins."io.containerd.grpc.v1.cri".containerd' contains config related to containerd
  [plugins."io.containerd.grpc.v1.cri".containerd]

    # snapshotter is the default snapshotter used by containerd
    # for all runtimes, if not overridden by an experimental runtime's snapshotter config.
    snapshotter = "overlayfs"

    # no_pivot disables pivot-root (linux only), required when running a container in a RamDisk with runc.
    # This only works for runtime type "io.containerd.runtime.v1.linux".
    no_pivot = false

    # disable_snapshot_annotations disables to pass additional annotations (image
    # related information) to snapshotters. These annotations are required by
    # stargz snapshotter (https://github.com/containerd/stargz-snapshotter)
    # changed to default true with https://github.com/containerd/containerd/pull/4665 and subsequent service refreshes.
    disable_snapshot_annotations = true

    # discard_unpacked_layers allows GC to remove layers from the content store after
    # successfully unpacking these layers to the snapshotter.
    discard_unpacked_layers = false

    # default_runtime_name is the default runtime name to use.
    default_runtime_name = "runc"

    # ignore_blockio_not_enabled_errors disables blockio related
    # errors when blockio support has not been enabled. By default,
    # trying to set the blockio class of a container via annotations
    # produces an error if blockio hasn't been enabled.  This config
    # option practically enables a "soft" mode for blockio where these
    # errors are ignored and the container gets no blockio class.
    ignore_blockio_not_enabled_errors = false

    # ignore_rdt_not_enabled_errors disables RDT related errors when RDT
    # support has not been enabled. Intel RDT is a technology for cache and
    # memory bandwidth management. By default, trying to set the RDT class of
    # a container via annotations produces an error if RDT hasn't been enabled.
    # This config option practically enables a "soft" mode for RDT where these
    # errors are ignored and the container gets no RDT class.
    ignore_rdt_not_enabled_errors = false

    # 'plugins."io.containerd.grpc.v1.cri".containerd.default_runtime' is the runtime to use in containerd.
    # DEPRECATED: use `default_runtime_name` and `plugins."io.containerd.grpc.v1.cri".containerd.runtimes` instead.
    [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]

    # 'plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime' is a runtime to run untrusted workloads on it.
    # DEPRECATED: use `untrusted` runtime in `plugins."io.containerd.grpc.v1.cri".containerd.runtimes` instead.
    [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]

    # 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes' is a map from CRI RuntimeHandler strings, which specify types
    # of runtime configurations, to the matching configurations.
    # In this example, 'runc' is the RuntimeHandler string to match.
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      # runtime_type is the runtime type to use in containerd.
      # The default value is "io.containerd.runc.v2" since containerd 1.4.
      # The default value was "io.containerd.runc.v1" in containerd 1.3, "io.containerd.runtime.v1.linux" in prior releases.
      runtime_type = "io.containerd.runc.v2"

      # pod_annotations is a list of pod annotations passed to both pod
      # sandbox as well as container OCI annotations. Pod_annotations also
      # supports golang path match pattern - https://golang.org/pkg/path/#Match.
      # e.g. ["runc.com.*"], ["*.runc.com"], ["runc.com/*"].
      #
      # For the naming convention of annotation keys, please reference:
      # * Kubernetes: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set
      # * OCI: https://github.com/opencontainers/image-spec/blob/main/annotations.md
      pod_annotations = []

      # container_annotations is a list of container annotations passed through to the OCI config of the containers.
      # Container annotations in CRI are usually generated by other Kubernetes node components (i.e., not users).
      # Currently, only device plugins populate the annotations.
      container_annotations = []

      # privileged_without_host_devices allows overloading the default behaviour of passing host
      # devices through to privileged containers. This is useful when using a runtime where it does
      # not make sense to pass host devices to the container when privileged. Defaults to false -
      # i.e pass host devices through to privileged containers.
      privileged_without_host_devices = false

      # privileged_without_host_devices_all_devices_allowed allows the allowlisting of all devices when
      # privileged_without_host_devices is enabled.
      # In plain privileged mode all host device nodes are added to the container's spec and all devices
      # are put in the container's device allowlist. This flags is for the modification of the privileged_without_host_devices
      # option so that even when no host devices are implicitly added to the container, all devices allowlisting is still enabled.
      # Requires privileged_without_host_devices to be enabled. Defaults to false.
      privileged_without_host_devices_all_devices_allowed = false

      # base_runtime_spec is a file path to a JSON file with the OCI spec that will be used as the base spec that all
      # container's are created from.
      # Use containerd's `ctr oci spec > /etc/containerd/cri-base.json` to output initial spec file.
      # Spec files are loaded at launch, so containerd daemon must be restarted on any changes to refresh default specs.
      # Still running containers and restarted containers will still be using the original spec from which that container was created.
      base_runtime_spec = ""

      # conf_dir is the directory in which the admin places a CNI conf.
      # this allows a different CNI conf for the network stack when a different runtime is being used.
      cni_conf_dir = "/etc/cni/net.d"

      # cni_max_conf_num specifies the maximum number of CNI plugin config files to
      # load from the CNI config directory. By default, only 1 CNI plugin config
      # file will be loaded. If you want to load multiple CNI plugin config files
      # set max_conf_num to the number desired. Setting cni_max_config_num to 0 is
      # interpreted as no limit is desired and will result in all CNI plugin
      # config files being loaded from the CNI config directory.
      cni_max_conf_num = 1

      # snapshotter overrides the global default snapshotter to a runtime specific value.
      # Please be aware that overriding the default snapshotter on a runtime basis is currently an experimental feature.
      # See https://github.com/containerd/containerd/issues/6657 for context.
      snapshotter = ""

      # treat_ro_mounts_as_rro ("Enabled"|"IfPossible"|"Disabled")
      # treats read-only mounts as recursive read-only mounts.
      # An empty string means "IfPossible".
      # "Enabled" requires Linux kernel v5.12 or later.
      # Introduced in containerd v2.0.
      # This configuration does not apply to non-volume mounts such as "/sys/fs/cgroup".
      treat_ro_mounts_as_rro = ""

      # 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options' is options specific to
      # "io.containerd.runc.v1" and "io.containerd.runc.v2". Its corresponding options type is:
      #   https://github.com/containerd/containerd/blob/v1.3.2/runtime/v2/runc/options/oci.pb.go#L26 .
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        # NoPivotRoot disables pivot root when creating a container.
        NoPivotRoot = false

        # NoNewKeyring disables new keyring for the container.
        NoNewKeyring = false

        # ShimCgroup places the shim in a cgroup.
        ShimCgroup = ""

        # IoUid sets the I/O's pipes uid.
        IoUid = 0

        # IoGid sets the I/O's pipes gid.
        IoGid = 0

        # BinaryName is the binary name of the runc binary.
        BinaryName = ""

        # Root is the runc root directory.
        Root = ""

        # SystemdCgroup enables systemd cgroups.
        SystemdCgroup = false

        # CriuImagePath is the criu image path
        CriuImagePath = ""

        # CriuWorkPath is the criu work path.
        CriuWorkPath = ""

  # 'plugins."io.containerd.grpc.v1.cri".cni' contains config related to cni
  [plugins."io.containerd.grpc.v1.cri".cni]
    # bin_dir is the directory in which the binaries for the plugin is kept.
    bin_dir = "/opt/cni/bin"

    # conf_dir is the directory in which the admin places a CNI conf.
    conf_dir = "/etc/cni/net.d"

    # max_conf_num specifies the maximum number of CNI plugin config files to
    # load from the CNI config directory. By default, only 1 CNI plugin config
    # file will be loaded. If you want to load multiple CNI plugin config files
    # set max_conf_num to the number desired. Setting max_config_num to 0 is
    # interpreted as no limit is desired and will result in all CNI plugin
    # config files being loaded from the CNI config directory.
    max_conf_num = 1

    # conf_template is the file path of golang template used to generate
    # cni config.
    # If this is set, containerd will generate a cni config file from the
    # template. Otherwise, containerd will wait for the system admin or cni
    # daemon to drop the config file into the conf_dir.
    # See the "CNI Config Template" section for more details.
    conf_template = ""
    # ip_pref specifies the strategy to use when selecting the main IP address for a pod.
    # options include:
    # * ipv4, "" - (default) select the first ipv4 address
    # * ipv6 - select the first ipv6 address
    # * cni - use the order returned by the CNI plugins, returning the first IP address from the results
    ip_pref = "ipv4"

  # 'plugins."io.containerd.grpc.v1.cri".image_decryption' contains config related
  # to handling decryption of encrypted container images.
  [plugins."io.containerd.grpc.v1.cri".image_decryption]
    # key_model defines the name of the key model used for how the cri obtains
    # keys used for decryption of encrypted container images.
    # The [decryption document](https://github.com/containerd/containerd/blob/main/docs/cri/decryption.md)
    # contains additional information about the key models available.
    #
    # Set of available string options: {"", "node"}
    # Omission of this field defaults to the empty string "", which indicates no key model,
    # disabling image decryption.
    #
    # In order to use the decryption feature, additional configurations must be made.
    # The [decryption document](https://github.com/containerd/containerd/blob/main/docs/cri/decryption.md)
    # provides information of how to set up stream processors and the containerd imgcrypt decoder
    # with the appropriate key models.
    #
    # Additional information:
    # * Stream processors: https://github.com/containerd/containerd/blob/main/docs/stream_processors.md
    # * Containerd imgcrypt: https://github.com/containerd/imgcrypt
    key_model = "node"

  # 'plugins."io.containerd.grpc.v1.cri".registry' contains config related to
  # the registry
  [plugins."io.containerd.grpc.v1.cri".registry]
    # config_path specifies a directory to look for the registry hosts configuration.
    #
    # The cri plugin will look for and use config_path/host-namespace/hosts.toml
    #   configs if present OR load certificate files as laid out in the Docker/Moby
    #   specific layout https://docs.docker.com/engine/security/certificates/
    #
    # If config_path is not provided defaults are used.
    #
    # *** registry.configs and registry.mirrors that were a part of containerd 1.4
    # are now DEPRECATED and will only be used if the config_path is not specified.
    config_path = ""

注册表配置

下面是一个默认注册表主机配置的简单示例。设置
在 containerd 的 config.toml 中设置 config_path = "/etc/containerd/certs.d"。
在配置路径下创建一个目录树，其中包括 docker.io 作为要配置的主机命名空间的目录。然后在docker.io目录中添加一个hosts.toml文件。
文件来配置主机命名空间。文件应该是这样的:

$ tree /etc/containerd/certs.d
/etc/containerd/certs.d
└── docker.io
   └── hosts.toml

$ cat /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"

[host."https://registry-1.docker.io"]
 capabilities = ["pull", "resolve"]

指定自定义证书：

$ cat /etc/containerd/certs.d/192.168.12.34:5000/hosts.toml
server = "https://192.168.12.34:5000"

[host."https://192.168.12.34:5000"]
  ca = "/path/to/ca.crt"

更多信息请参见 docs/hosts.md。

不受信任的工作负载

运行不受信任工作负载的推荐方法是使用 Kubernetes 1.12引入的
运行时类 api选择plugin."io.containerd.grpc.v1.cri.containerd.runtimes" 中配置为运行不信任工作负载的 RuntimeHandlers。

但是，如果使用传统的 io.kubernetes.cri.untrusted-workloadpod 注解
来请求使用不受信任工作负载的运行时运行 pod，那么 RuntimeHandler
必须首先定义plugins. "io.containerd.grpc.v1.cri.containerd.runtrusted"。
当注解 io.kubernetes.cri.untrusted-workload 设置为 true 时，untrusted运行时将被使用。请参阅
使用kata容器创建不受信任的 pod。

CNI 配置模板

理想情况下，cni 配置应由系统管理员或 cni 守护进程（如 calico、weaveworks 等）放置。
不过，这对没有 cni 守护进程来放置 cni 配置的情况很有用。

cni 配置模板使用 golang
template格式。目前支持的
值是

• .PodCIDR 是分配给节点的第一个 CIDR 的字符串。
• .PodCIDRRanges 是分配给节点的所有 CIDR 的字符串数组。它
  通常用于
  dualstack支持。
• .Routes 是一个包含所有所需路由的字符串数组。它通常用于
  通常用于支持双协议栈或单协议栈，但 IPv4 或 IPv6 则在运行时决定。

golang 模板动作](https://golang.org/pkg/text/template/#hdr-Actions)
可用于渲染 cni 配置。例如，您可以使用以下
模板在 CNI 配置中添加双协议栈的 CIDR 和路由：

"ipam": {
  "type": "host-local",
  "ranges": [{{range $i, $range := .PodCIDRRanges}}{{if $i}}, {{end}}[{"subnet": "{{$range}}"}]{{end}}],
  "routes": [{{range $i, $route := .Routes}}{{if $i}}, {{end}}{"dst": "{{$route}}"}{{end}}]
}

过时

CRI 插件的配置选项遵循 Kubernetes Deprecation
面向管理员的 CLI 组件 "政策。

总之，当一个配置选项被宣布弃用时：

• 在 6 个月或 1 个版本（以时间较长者为准）内保持其功能；
• 使用时发出警告。

配置镜像解密

本小节介绍为 containerd 配置加密容器镜像解密的方法，以便与 cri 插件一起使用。

加密容器镜像

加密容器镜像是包含加密 blob 的 OCI 镜像。这些加密镜像可以通过使用 containerd/imgcrypt project 创建。要解密这些镜像，"containerd "运行时会使用从 "cri "传递过来的信息，如密钥、选项和加密元数据。

"节点"密钥模型

加密根据与密钥相关联的模型将信任与实体联系起来。我们称之为密钥模型。当我们想把密钥的信任度与集群中的节点联系起来时，就是这样一种用例。在这种情况下，我们称之为 "节点 "或 "主机 "密钥模型。未来的工作将包括更多的密钥模型，以促进其他信任关联（即多租户）。

"节点 "密钥模型 使用案例

在这种模式下，加密与工作节点绑定。这里的用例围绕着镜像只能在受信任的主机上解密这一想法展开。使用这种模式，各种基于节点的技术都有助于引导对工作节点的信任并执行安全密钥分发（即 TPM、主机认证、安全/测量引导）。在这种情况下，运行时能够获取必要的解密密钥。使用 imgcrypt 中的 [--decryption-keys-path 标志就是一个例子。

为 "节点 "密钥模型配置镜像解密

这是自 containerd v1.5 以来的默认模型。

对于 containerd v1.4，您需要将以下配置添加到 /etc/containerd/config.toml，并手动重启 containerd 服务。

version = 2

[plugins."io.containerd.grpc.v1.cri".image_decryption]
  key_model = "node"

[stream_processors]
  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"
    path = "ctd-decoder"
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env= ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    returns = "application/vnd.oci.image.layer.v1.tar"
    path = "ctd-decoder"
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env= ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]

在本例中，容器图像解密被设置为使用 "节点 "密钥模型。
此外，解密 stream_processors的配置与 containerd/imgcrypt project 中指定的一样，并配置了附加字段 --decryption-keys-path，以指定解密密钥在节点本地的位置。

$OCICRYPT_KEYPROVIDER_CONFIG` 环境变量用于 ocicrypt keyprovider protocol

配置镜像注册表

本小节介绍为 containerd 配置镜像注册表的方法，以便与 cri 插件一起使用。

NOTE: 本文档中之前描述的 registry.mirrors 和 registry.configs
已被删除。如 the cri config 所述，现在应使用以下配置

[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

配置注册表凭据

NOTE: registry.configs.*.auth 已过时，还没有除了在主机
配置文件中存储未加密的secret的相应方法。在
有合适的secret管理替代插件出现后才会删除它。
在 1.x 版本中，包括 1.6 LTS 版本中仍会支持。

要为特定注册表配置证书，请创建/修改
/etc/containerd/config.toml"：

# explicitly use v2 config format
version = 2

# The registry host has to be a domain name or IP. Port number is also
# needed if the default HTTPS or HTTP port is not used.
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
  username = ""
  password = ""
  auth = ""
  identitytoken = ""

每个字段的含义与.docker/config.json中的相应字段相同。

请注意，CRI 传递的 auth 配置优先于此配置。
该配置中的注册凭据只有在 auth 配置未由 Kubernetes 通过 CRI 指定时，才会使用此配置中的注册凭据。

修改此配置后，需要重新启动 containerd 服务。

配置注册表凭据示例 - 使用服务账户密钥身份验证的 GCR

如果尚未设置 Google Container Registry (GCR)，则需要执行以下步骤：

• 创建谷歌云平台（GCP）账户和项目（如果尚未创建）（请参阅 GCP 入门
• 为您的项目启用 GCR（请参阅 Container Registry 快速入门
• 为验证 GCR：创建 服务账户和 JSON 密钥
• 需要从 GCP 控制台将 JSON 密钥文件下载到系统中。
• 用于访问 GCR 存储： 向 GCR 存储桶添加具有存储管理访问权限的服务账户（请参阅 授予权限）。

有关上述步骤的详细信息，请参阅 推送和提取镜像。

注意：JSON 密钥文件是一个多行文件，将其内容用作文件外的密钥会很麻烦。因此值得生成该文件的单行格式输出。一种方法是使用 jq 工具，如下所示： jq -c . key.json

在将 GCR 挂接到 containerd 之前，最好先从终端确认自己可以通过 GCR 进行身份验证并访问存储。这可以通过登录 GCR 和向其推送映像，即可验证这一点：

docker login -u _json_key -p "$(cat key.json)" gcr.io

docker pull busybox

docker tag busybox gcr.io/your-gcp-project-id/busybox

docker push gcr.io/your-gcp-project-id/busybox

docker logout gcr.io

既然知道可以从终端访问 GCR，现在就该试试 containerd 了。

编辑 containerd 配置（默认位置为 /etc/containerd/config.toml)
以添加用于gcr.io域镜像拉取的 JSON 密钥
请求：

version = 2

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["https://registry-1.docker.io"]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
      endpoint = ["https://gcr.io"]
  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io".auth]
      username = "_json_key"
      password = 'paste output from jq'

注意：_json_key 的 username 表示将使用 JSON 密钥验证。

重启containerd:

service containerd restart

使用 crictl 从 GCR 中提取图像：

$ sudo crictl pull gcr.io/your-gcp-project-id/busybox

DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gcr.io/your-gcr-instance-id/busybox,},Auth:nil,SandboxConfig:nil,}
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,}
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42

---

注意：本文档中使用的配置语法是自 containerd 1.3 以来推荐使用的第 2 版。有关以前的配置格式，请参考 https://github.com/containerd/cri/blob/release/1.2/docs/registry.md。

CRI 插件测试指南

本小节假定您已经设置了开发环境（go、git、github.com/containerd/containerd repo 等）。

在发送拉取请求之前，您至少应确保您的更改已通过代码验证、单元测试、集成测试和 CRI 验证测试。

构建

按照 building 说明进行。

CRI 集成测试

• 运行所有 CRI 集成测试：

make cri-integration

• 运行特定的 CRI 集成测试：使用 FOCUS 参数指定测试用例。

# 运行与测试字符串  匹配的 CRI 集成测试
FOCUS=<TEST_NAME> make cri-integration

示例:

FOCUS=TestContainerListStats make cri-integration

CRI 验证测试

CRI验证测试 是一个测试框架，用于验证容器运行时接口（CRI）的实现，如带有cri插件的containerd，是否满足管理pod沙箱、容器、镜像等所需的所有要求。

通过 CRI 验证测试，无需设置 Kubernetes 组件或运行 Kubernetes 端到端测试，就能验证 containerd 的 CRI 一致性。

• 安装依赖项。
• 运行上文内置了 cri 插件的 containerd：

containerd -l debug

有关 CRI 验证测试的 更多信息。

节点 E2E 测试

Node e2e test是一个测试 Kubernetes 节点级功能（如管理 pod、挂载卷等）的测试框架。它使用 Kubelet 和其他一些最基本的依赖项启动本地集群，并针对本地集群运行节点功能测试。

目前，e2e-node 测试支持通过 github 上的 Pull Request 注释进行。
在拉取请求的注释中输入"/test all"，即可查看已通过 prow bot 与 GCE 托管的 kubernetes 测试服务集成的测试选项列表。
输入 /test pull-containerd-node-e2e 将在你的拉取请求提交上启动节点 e2e 测试运行。

有关 Kubernetes 节点 e2e 测试的 更多信息。

CRICTL 用户指南

本小节假定您已经安装并运行了带有 cri 插件的 containerd 。

本文档面向希望调试、检查和管理 pod 的开发人员、
容器和容器映像的开发人员。

在生成针对本文档、containerd、containerd/cri、
或 crictl 生成问题之前，请确保该问题尚未提交过。

安装 crictl

如果您尚未安装 crictl，请安装与您正在使用的
与您正在使用的 cri 插件兼容的版本。如果您是用户，您的部署应该已为您安装了 crictl。如果没有，请从发布的压缩包中获取。
如果您是开发人员，当前的 crictl 版本将在 此处 中指定。
已包含一条辅助命令，用于在正确的版本上安装依赖项：

$ make install-deps

• 注意：名为 /etc/crictl.yaml 的文件用于配置 crictl
  的文件，这样就不必重复指定用于将 crictl和
  容器运行时进行连接的sock：

$ cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true

下载并检查容器镜像

pull 命令会告诉容器运行时从容器注册中心下载容器镜像。

$ crictl pull busybox
  ...
$ crictl inspecti busybox
  ... displays information about the image.

注意： 如果在运行 crictl 命令时遇到类似于下面的错
时（而你的 containerd 实例已经在运行）

crictl info
FATA[0000] getting status of runtime failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

这可能是因为您使用了不正确的 containerd 配置（可能是
来自 Docker 安装）。您需要更新正在运行的 containerd 实例的配置。一种方法如下：

$ mv /etc/containerd/config.toml /etc/containerd/config.bak
$ containerd config default > /etc/containerd/config.toml

直接加载容器镜像

将镜像加载到容器运行时的另一种方法是使用 load
命令。使用加载命令，您可以从文件中将容器镜像注入容器
运行时。首先，你需要创建一个容器镜像压缩包。例如
例如，为使用 Docker 的暂停容器创建镜像压缩包：

$ docker pull registry.k8s.io/pause:3.9
  3.9: Pulling from pause
  7582c2cc65ef: Pull complete
  Digest: sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097
  Status: Downloaded newer image for registry.k8s.io/pause:3.9
  registry.k8s.io/pause:3.9
$ docker save registry.k8s.io/pause:3.9 -o pause.tar

然后使用 ctr 将容器镜像加载到容器运行时：

# cri 插件使用 "k8s.io "容器名称空间
$ sudo ctr -n=k8s.io images import pause.tar
  Loaded image: registry.k8s.io/pause:3.9

列出镜像并检查暂停镜像：

$ sudo crictl images
IMAGE                       TAG                 IMAGE ID            SIZE
docker.io/library/busybox   latest              f6e427c148a76       728kB
registry.k8s.io/pause            3.9                 e6f181688397       311kB
$ sudo crictl inspecti e6f181688397
  ... displays information about the pause image.
$ sudo crictl inspecti registry.k8s.io/pause:3.9
  ... displays information about the pause image.

运行 pod 沙盒（使用配置文件）

$ cat sandbox-config.json
{
    "metadata": {
        "name": "nginx-sandbox",
        "namespace": "default",
        "attempt": 1,
        "uid": "hdishd83djaidwnduwk28bcsb"
    },
    "linux": {
    }
}

$ crictl runp sandbox-config.json
e1c83b0b8d481d4af8ba98d5f7812577fc175a37b10dc824335951f52addbb4e
$ crictl pods
PODSANDBOX ID       CREATED             STATE               NAME               NAMESPACE          ATTEMPT
e1c83b0b8d481       2 hours ago         SANDBOX_READY       nginx-sandbox      default            1
$ crictl inspectp e1c8
  ... displays information about the pod and the pod sandbox pause container.

• 注意：如上所述，如果 ID 是唯一的，则可以使用截断的 ID。
• 管理 pod 的其他命令包括：stops ID 用于停止运行中的 pod，rmp ID 用于删除 pod 沙盒。

在 Pod 沙盒中创建并运行容器（使用配置文件）

$ cat container-config.json
{
  "metadata": {
      "name": "busybox"
  },
  "image":{
      "image": "busybox"
  },
  "command": [
      "top"
  ],
  "linux": {
  }
}

$ crictl create e1c83 container-config.json sandbox-config.json
0a2c761303163f2acaaeaee07d2ba143ee4cea7e3bde3d32190e2a36525c8a05
$ crictl ps -a
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT
0a2c761303163       docker.io/busybox   2 hours ago         CONTAINER_CREATED   busybox             0
$ crictl start 0a2c
0a2c761303163f2acaaeaee07d2ba143ee4cea7e3bde3d32190e2a36525c8a05
$ crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT
0a2c761303163       docker.io/busybox   2 hours ago         CONTAINER_RUNNING   busybox             0
$ crictl inspect 0a2c7
  ... show detailed information about the container

在容器中执行命令

$ crictl exec -i -t 0a2c ls
bin   dev   etc   home  proc  root  sys   tmp   usr   var

显示容器的统计信息

$ crictl stats
CONTAINER           CPU %               MEM                 DISK              INODES
0a2c761303163f      0.00                983kB             16.38kB             6

• 其他管理容器的命令包括：stop ID 用于停止运行中的容器和 rm ID 用于移除容器。

显示版本信息

$ crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.7.0
RuntimeApiVersion:  v1

显示 Containerd 和 CRI 插件的状态和配置信息

$ crictl info
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "/opt/cni/bin"
    ],
    "PluginConfDir": "/etc/cni/net.d",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": []
  },
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "runc",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "privileged_without_host_devices_all_devices_allowed": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0,
        "snapshotter": "",
        "sandboxMode": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "privileged_without_host_devices_all_devices_allowed": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0,
        "snapshotter": "",
        "sandboxMode": ""
      },
      "runtimes": {
        "runc": {
          "runtimeType": "io.containerd.runc.v2",
          "runtimePath": "",
          "runtimeEngine": "",
          "PodAnnotations": [],
          "ContainerAnnotations": [],
          "runtimeRoot": "",
          "options": {
            "BinaryName": "",
            "CriuImagePath": "",
            "CriuPath": "",
            "CriuWorkPath": "",
            "IoGid": 0,
            "IoUid": 0,
            "NoNewKeyring": false,
            "NoPivotRoot": false,
            "Root": "",
            "ShimCgroup": "",
            "SystemdCgroup": false
          },
          "privileged_without_host_devices": false,
          "privileged_without_host_devices_all_devices_allowed": false,
          "baseRuntimeSpec": "",
          "cniConfDir": "",
          "cniMaxConfNum": 0,
          "snapshotter": "",
          "sandboxMode": "podsandbox"
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false,
      "ignoreBlockIONotEnabledErrors": false,
      "ignoreRdtNotEnabledErrors": false
    },
    "cni": {
      "binDir": "/opt/cni/bin",
      "confDir": "/etc/cni/net.d",
      "maxConfNum": 1,
      "setupSerially": false,
      "confTemplate": "",
      "ipPref": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": {},
      "configs": {},
      "auths": {},
      "headers": {}
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 1024,
    "sandboxImage": "registry.k8s.io/pause:3.9",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": true,
    "disableHugetlbController": true,
    "device_ownership_from_security_context": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "enableUnprivilegedPorts": false,
    "enableUnprivilegedICMP": false,
    "enableCDI": false,
    "cdiSpecDirs": [
      "/etc/cdi",
      "/var/run/cdi"
    ],
    "imagePullProgressTimeout": "1m0s",
    "drainExecSyncIOTimeout": "0s",
    "containerdRootDir": "/var/lib/containerd",
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.20.3",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK"
}

更多信息

参见 此处