OpenShift 4 - 利用 File Integrity Operator 实现对集群节点进行入侵检测（附视频）
《OpenShift / RHEL / DevSecOps 汇总目录》
说明：本文已经在OpenShift 4.9环境中验证
文章目录
- File Integrity Operator 功能介绍
- 安装 File Integrity Operator
- 创建 FileIntegrity 对象
- 修改节点配置文件
- 确认文件完整性发生变化
- 接受对节点文件的修改
- 通过Prometheus监控
- 定制 File Integrity 配置
- 演示视频
- 参考
File Integrity Operator 功能介绍
OpenShift 的 File Integrity Operator 可以在集群节点上持续地运行文件完整性检查。它部署了一个 DaemonSet，在每个节点上初始化并运行有特权的AIDE（Advanced Intrusion Detection Environment - 高级入侵检测环境）容器，提供自 DaemonSet 初始运行以后被修改的文件记录。
File Integrity Operator 根据从配置文件中找到的正则表达式规则创建一个数据库。一旦数据库被初始化，它就可以用来验证文件的完整性。它有几种消息摘要算法用于检查文件的完整性，另外文件属性也可以被检查出不一致的地方。
安装 File Integrity Operator
在 OpenShift 上使用缺省配置安装 File Integrity Operator ，缺省会将资源安装在 openshift-file-integrity 项目下。
安装后可以看到 File Integrity Operator 实例：
创建 FileIntegrity 对象
- 在 OpenShift 中根据以下内容创建FileIntegrity对象。其中 spec.config.gracePeriod 是两次 AIDE 完整性检查之间暂停的秒数。由于在节点中频繁进行 AIDE 检查需要大量资源，因此可以指定较长的时间间隔（默认为 900 秒）。
apiVersion: fileintegrity.openshift.io/v1alpha1
kind: FileIntegrity
metadata:
name: example-fileintegrity
namespace: openshift-file-integrity
spec:
config:
gracePeriod: 900
debug: false
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- 如果没有指定，FileIntegrity 会在所有节点运行一个 Daemonset Pod 以进行完整性检查。执行命令可以看到 Daemonset 运行在集群的 5 个节点中。
$ oc get daemonset -n openshift-file-integrity
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
aide-example-fileintegrity 5 5 5 5 5 <none> 7h41m
$ oc get pod -n openshift-file-integrity -o wide -l app=aide-example-fileintegrity
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
aide-example-fileintegrity-2vpbq 1/1 Running 1 7h45m 10.128.2.15 ip-10-0-241-123.us-east-2.compute.internal <none> <none>
aide-example-fileintegrity-czf4w 1/1 Running 1 7h45m 10.131.0.11 ip-10-0-134-47.us-east-2.compute.internal <none> <none>
aide-example-fileintegrity-vx5q7 1/1 Running 1 7h45m 10.128.0.6 ip-10-0-173-213.us-east-2.compute.internal <none> <none>
aide-example-fileintegrity-wd4pm 1/1 Running 1 7h45m 10.129.0.10 ip-10-0-173-152.us-east-2.compute.internal <none> <none>
aide-example-fileintegrity-wx42d 1/1 Running 1 7h45m 10.130.0.8 ip-10-0-135-39.us-east-2.compute.internal <none> <none>
- 查看内容为 “FileIntegrityStatus” 的事件，可以看到当前已经是 “Active”。
$ oc get event -n openshift-file-integrity --field-selector reason=FileIntegrityStatus
LAST SEEN TYPE REASON OBJECT MESSAGE
2m28s Normal FileIntegrityStatus fileintegrity/example-fileintegrity Pending
2m28s Normal FileIntegrityStatus fileintegrity/example-fileintegrity Initializing
2m13s Normal FileIntegrityStatus fileintegrity/example-fileintegrity Active
- 在example-fileintegrity完成首次扫描后，可以执行以下命令确认所有节点的 fileintegritynodestatus 的状态都是 “Succeeded”，这说明被监控节点的目标文件没有变化。
$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME NODE STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal ip-10-0-134-47.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal ip-10-0-135-39.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal ip-10-0-173-152.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal ip-10-0-173-213.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal ip-10-0-241-123.us-east-2.compute.internal Succeeded
- 另外查看事件也能看到此时所有目标都 “no changes to node”。
$ oc get events -n openshift-file-integrity --field-selector reason=NodeIntegrityStatus
LAST SEEN TYPE REASON OBJECT MESSAGE
7m4s Normal NodeIntegrityStatus fileintegrity/example-fileintegrity no changes to node ip-10-0-134-47.us-east-2.compute.internal
7m3s Normal NodeIntegrityStatus fileintegrity/example-fileintegrity no changes to node ip-10-0-241-123.us-east-2.compute.internal
7m2s Normal NodeIntegrityStatus fileintegrity/example-fileintegrity no changes to node ip-10-0-173-152.us-east-2.compute.internal
7m Normal NodeIntegrityStatus fileintegrity/example-fileintegrity no changes to node ip-10-0-135-39.us-east-2.compute.internal
6m59s Normal NodeIntegrityStatus fileintegrity/example-fileintegrity no changes to node ip-10-0-173-213.us-east-2.compute.internal
修改节点配置文件
- 进入到一个Node的RHCOS内部。
$ NODE=ip-10-0-135-39.us-east-2.compute.internal
$ NODESTATUS=example-fileintegrity-${NODE}
$ oc debug node/${NODE}
- 执行命令，更新 “** /host/etc/resolv.conf**” 文件。
sh-4.2# echo "# integrity test" >> /host/etc/resolv.conf
sh-4.2# exit
确认文件完整性发生变化
- 执行以下命令，或在 OpenShift 控制台上查看 example-fileintegrity 的事件。确认提示有一个节点的完整性检查失败，其中完整性变化是：“a:0,c:1,r:0”，即：0个新增（a）；1个变化（c）；0个删除（r）。
$ oc get events -n openshift-file-integrity --field-selector reason=NodeIntegrityStatus,type=Warning
LAST SEEN TYPE REASON OBJECT MESSAGE
17m Warning NodeIntegrityStatus fileintegrity/example-fileintegrity node ip-10-0-135-39.us-east-2.compute.internal has changed! a:0,c:1,r:0 log:openshift-file-integrity/aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed
2. 还可执行以下命令，确认其中一个的 STATUS 已经变为 “Failed”。
$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME NODE STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal ip-10-0-134-47.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal ip-10-0-135-39.us-east-2.compute.internal Failed
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal ip-10-0-173-152.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal ip-10-0-173-213.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal ip-10-0-241-123.us-east-2.compute.internal Succeeded
- 执行以下命令，可以在该节点的 fileintegritynodestatus 中查到记录的完整性检查结果。可以看到完整性检查开始是 “Secceeded”，然后变成了“Failed”，这是由 1 个 filesChanged 引起的。File Integrity Operator 将完整性检查详细结果存放在 “openshift-file-integrity” 项目中名为 “aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed” 的 ConfigMap 对象中了。
$ oc get fileintegritynodestatus/${NODESTATUS} -n openshift-file-integrity -ojsonpath='{.results}' | jq -r
[
{
"condition": "Succeeded",
"lastProbeTime": "2022-03-05T03:19:42Z"
},
{
"condition": "Failed",
"filesChanged": 1,
"lastProbeTime": "2022-03-05T03:35:27Z",
"resultConfigMapName": "aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed",
"resultConfigMapNamespace": "openshift-file-integrity"
}
]
- 从 ConfigMap 对象中查看完整性检查结果失败说明，其中提示 “/hostroot/etc/resolv.conf” 发生了变化。
$ oc describe cm aide-${NODESTATUS}-failed
Name: aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed
Namespace: openshift-file-integrity
Labels: file-integrity.openshift.io/node=ip-10-0-135-39.us-east-2.compute.internal
file-integrity.openshift.io/owner=example-fileintegrity
file-integrity.openshift.io/result-log=
Annotations: file-integrity.openshift.io/files-added: 0
file-integrity.openshift.io/files-changed: 1
file-integrity.openshift.io/files-removed: 0
Data
====
integritylog:
----
Start timestamp: 2022-03-05 03:50:27 +0000 (AIDE 0.16)
AIDE found differences between database and filesystem!!
Summary:
Total number of entries: 35211
Added entries: 0
Removed entries: 0
Changed entries: 1
---------------------------------------------------
Changed entries:
---------------------------------------------------
f ... .C... : /hostroot/etc/resolv.conf
---------------------------------------------------
Detailed information about changes:
---------------------------------------------------
File: /hostroot/etc/resolv.conf
SHA512 : Xl2pzxjmRPtW8bl6Kj49SkKOSBVJgsCI | 4M9rtSPrVj1Q34k0r/j3sZlreZT6uieC
Fw3VVMAbuGgHwINR5Fi5CYGuLhNmmDNh | 850ZVM9KcJ5DYB5Gf0IGbuojvIF1exdI
dC7kQIKcAjFwB3Ib+UaOzw== | vXZwgl8BvjmcezDPzmTGPA==
---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------
/hostroot/etc/kubernetes/aide.db.gz
MD5 : OeZ96l1VZFyigc/hL7vv5w==
SHA1 : a8EFf5Ylgi1h86rq549XFFea6CM=
RMD160 : SV7z0d1kjNxzSRn0Pu+6Hps02Ew=
TIGER : HHLY/HeLmpTjF3/bNyTjcZ3+/AlDfYrK
SHA256 : DsDlSlIxzysKry78/mYualTEt6BqtjJJ
nIoRhHwy1jo=
SHA512 : iIIJZZKx36suzJG2BsdG78jAonAm1EpI
ZKmz40Gw5h6eWIu88+VyFlStpkOLCeFn
3j8uZda/6Qgste6zBQV/Sg==
End timestamp: 2022-03-05 03:51:10 +0000 (run time: 0m 43s)
BinaryData
====
Events: <none>
接受对节点文件的修改
如果这个修改是必要的，则可以接受这一修改。
- 执行命令重置完整性对比数据库。
$ oc annotate fileintegrity/example-fileintegrity file-integrity.openshift.io/re-init=
- 然后查看由 FileIntegrityStatus 引发的事件。
$ oc get events -n openshift-file-integrity --field-selector reason=FileIntegrityStatus
LAST SEEN TYPE REASON OBJECT MESSAGE
54s Normal FileIntegrityStatus fileintegrity/example-fileintegrity Initializing
47s Normal FileIntegrityStatus fileintegrity/example-fileintegrity Active
- 可以再次进入前面操作的 RHCOS，确认已经在 “/host/etc/kubernetes/” 目录中对 “aide.db.gz”和“aide.log”进行了备份。
sh-4.4# chroot /host
sh-4.4# ls -lR /etc/kubernetes/aide.*
-rw-------. 1 root root 1953565 Mar 5 10:47 /etc/kubernetes/aide.db.gz
-rw-------. 1 root root 1953538 Mar 5 10:46 /etc/kubernetes/aide.db.gz.backup-20220305T10_46_29
-rw-------. 1 root root 1953565 Mar 5 10:47 /etc/kubernetes/aide.db.gz.new
-rw-------. 1 root root 1012 Mar 5 10:31 /etc/kubernetes/aide.log
-rw-------. 1 root root 1012 Mar 5 10:46 /etc/kubernetes/aide.log.backup-20220305T10_46_29
-rw-------. 1 root root 778 Mar 5 10:47 /etc/kubernetes/aide.log.new
- 过一段时间后每个节点的 DaemonSet 会完成一次新的完整性对比。再此过程会中会基于修改后的文件重新建完整性对比数据库，这样对比状态又恢复成 “Succeeded” 了。
$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME NODE STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal ip-10-0-134-47.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal ip-10-0-135-39.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal ip-10-0-173-152.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal ip-10-0-173-213.us-east-2.compute.internal Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal ip-10-0-241-123.us-east-2.compute.internal Succeeded
通过Prometheus监控
File Integrity 提供了 Prometheus 接口，运行以下命令可以从 Prometheus 接口访问到当前集群文件完整性的状态。其中：
“file_integrity_operator_node_failed gauge” 是以往完整性失败的情况。
“file_integrity_operator_node_status_total counter” 是当前集群节点完整性的统计。
$ oc run --rm -i --restart=Never --image=registry.fedoraproject.org/fedora-minimal:latest -n openshift-file-integrity metrics-test -- bash -c 'curl -ks -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://metrics.openshift-file-integrity.svc:8585/metrics-fio'
If you don't see a command prompt, try pressing enter.
# HELP file_integrity_operator_node_failed A gauge that is set to 1 when a node has unresolved integrity failures, and 0 when it is healthy
# TYPE file_integrity_operator_node_failed gauge
file_integrity_operator_node_failed{node="ip-10-0-134-47.us-east-2.compute.internal"} 1
file_integrity_operator_node_failed{node="ip-10-0-135-39.us-east-2.compute.internal"} 0
# HELP file_integrity_operator_node_status_total The total number of FileIntegrityNodeStatus transitions, per condition and node
# TYPE file_integrity_operator_node_status_total counter
file_integrity_operator_node_status_total{condition="Failed",node="ip-10-0-134-47.us-east-2.compute.internal"} 1
file_integrity_operator_node_status_total{condition="Succeeded",node="ip-10-0-135-39.us-east-2.compute.internal"} 1
。。。
定制 File Integrity 配置
- 当创建一个 FileIntegrity，它的详细配置保存在对应名称的 ConfigMap 对象中。可以执行以下命令查看该 ConfigMap 的内容，其中使用正则表达式列出了所有受 FileIntegrity 保护的目标，默认配置包含以下目录中的文件 “/root”、“/boot”、“/usr”、“/etc”，而不包含的目录文件有 “/var”、“/opt”、“/etc/”。
$ oc describe cm/example-fileintegrity -n openshift-file-integrity
Name: example-fileintegrity
Namespace: openshift-file-integrity
Labels: file-integrity.openshift.io/aide-conf=
file-integrity.openshift.io/owner=example-fileintegrity
Annotations: <none>
Data
====
aide.conf:
----
@@define DBDIR /hostroot/etc/kubernetes
@@define LOGDIR /hostroot/etc/kubernetes
database=file:@@{DBDIR}/aide.db.gz
database_out=file:@@{DBDIR}/aide.db.gz.new
gzip_dbout=yes
verbose=5
report_url=file:@@{LOGDIR}/aide.log.new
report_url=stdout
PERMS = p+u+g+acl+selinux+xattrs
CONTENT_EX = sha512+ftype+p+u+g+n+acl+selinux+xattrs
/hostroot/boot/ CONTENT_EX
/hostroot/root/\..* PERMS
/hostroot/root/ CONTENT_EX
!/hostroot/root/\.kube
!/hostroot/usr/src/
!/hostroot/usr/tmp/
/hostroot/usr/ CONTENT_EX
# OpenShift specific excludes
!/hostroot/opt/
!/hostroot/var
!/hostroot/etc/NetworkManager/system-connections/
!/hostroot/etc/mtab$
!/hostroot/etc/.*~
!/hostroot/etc/kubernetes/static-pod-resources
!/hostroot/etc/kubernetes/aide.*
!/hostroot/etc/kubernetes/manifests
!/hostroot/etc/docker/certs.d
!/hostroot/etc/selinux/targeted
!/hostroot/etc/openvswitch/conf.db
!/hostroot/etc/kubernetes/cni/net.d/*
!/hostroot/etc/machine-config-daemon/currentconfig$
!/hostroot/etc/pki/ca-trust/extracted/java/cacerts$
!/hostroot/etc/cvo/updatepayloads
# Catch everything else in /etc
/hostroot/etc/ CONTENT_EX
BinaryData
====
Events: <none>
- 可以将该 ConfigMap 内容导出成文件。
$ oc extract cm/example-fileintegrity -n openshift-file-integrity --keys=aide.conf
- 修改后再根据修改后的文件生成一个新的 ConfigMap 对象。
$ oc create cm master-aide-conf -n openshift-file-integrity --from-file=aide.conf
- 最后可以根据以下YAML只在 “master” 节点上根据建的 “master-aide-conf” 配置创建 FileIntegrity 对象。
node-role.kubernetes.io/master: “” 在所有 master 节点上调度 AIDE；name 和 namespace 属性指向配置映射
apiVersion: fileintegrity.openshift.io/v1alpha1
kind: FileIntegrity
metadata:
name: master-fileintegrity
namespace: openshift-file-integrity
spec:
nodeSelector:
node-role.kubernetes.io/master: ""
config:
name: master-aide-conf
namespace: openshift-file-integrity
演示视频
视频
参考
https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.9/html-single/security_and_compliance/index
https://github.com/openshift/file-integrity-operator
https://aide.github.io/