Linux应急响应 - 31 自动化排查脚本

Linux应急响应
, ,

自动化应急响应排查脚本的设计与实现

应急响应现场时间紧迫，自动化脚本是保证排查全面、不遗漏的关键。本节从设计原则到完整脚本，覆盖模块化 IR 脚本和主流开源工具对比

关联：02-排查命令速查 | 00-学习路线

为什么需要自动化

时间压力

应急响应的黄金时间有限，手动逐项排查容易遗漏

攻击者可能仍在系统中活动，需要快速完成证据收集

多台主机排查时，手动操作不可扩展

标准化与一致性

不同分析师手动排查的结果可能不一致

自动化脚本保证每次检查项目完全相同

结构化输出便于后续分析和对比

典型应用场景

安全事件初始响应 — 5 分钟内完成全面快照

批量扫描 — 100+ 台服务器同时排查

日常安全巡检 — 定期自动运行

入职交接 — 新人也能执行标准化排查

2. 脚本设计原则

2.1 只读原则 (Read-Only)

绝对不修改目标系统 — 这是最重要的原则

不写任何文件到目标系统磁盘 (输出到外部存储或网络)

不 kill 进程、不删除文件、不修改配置

不安装软件包、不加载内核模块

1
2
3
4
5
6
7
8
# 好: 只读取信息
ps auxwwf > "$OUTPUT/ps.txt"
cat /etc/passwd > "$OUTPUT/passwd.txt"

# 坏: 修改系统
kill -9 $suspicious_pid           # 不要杀进程
rm /tmp/malware                   # 不要删文件
apt install -y some-tool          # 不要装软件

2.2 模块化设计

每个检查项独立成函数，可单独运行

失败的模块不影响其他模块执行

便于根据场景选择运行哪些模块

1
2
3
4
5
6
7
8
9
10
11
# 模块化结构示例
check_system_info()   { ... }
check_users()         { ... }
check_processes()     { ... }
check_network()       { ... }
check_filesystem()    { ... }
check_crontabs()      { ... }
check_services()      { ... }
check_backdoors()     { ... }
collect_logs()        { ... }
generate_report()     { ... }

2.3 格式化输出与异常标记

使用统一的输出格式，关键发现用颜色/标记突出

1
2
3
4
5
6
7
8
9
10
11
# 颜色定义
RED='\033[0;31m'
YELLOW='\033[1;33m'
GREEN='\033[0;32m'
NC='\033[0m'

# 输出函数
info()    { echo -e "[${GREEN}INFO${NC}] $1"; }
warn()    { echo -e "[${YELLOW}WARN${NC}] $1"; }
alert()   { echo -e "[${RED}ALERT${NC}] $1"; }
section() { echo -e "\n========== $1 =========="; }

2.4 跨平台兼容

同时支持 Debian/Ubuntu 和 CentOS/RHEL 系列

检测可用命令，优雅降级

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 检测发行版
if [ -f /etc/debian_version ]; then
    DISTRO="debian"
elif [ -f /etc/redhat-release ]; then
    DISTRO="redhat"
else
    DISTRO="unknown"
fi

# 命令兼容
if command -v ss &>/dev/null; then
    ss -antup > "$OUTPUT/ss.txt"
elif command -v netstat &>/dev/null; then
    netstat -antup > "$OUTPUT/netstat.txt"
fi

3. 完整 IR 排查脚本

脚本位于项目目录: /Users/lucy/Desktop/AI/应急响应/scripts/ir-collector.sh

以下为各模块的详细实现

3.1 脚本框架与初始化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#!/bin/bash
#====================================================================
# ir-collector.sh — Linux 应急响应自动化排查脚本
# 用途: 快速收集系统关键信息，标记异常项
# 使用: sudo bash ir-collector.sh [output_dir]
# 注意: 本脚本为只读操作，不会修改目标系统
#====================================================================

set -euo pipefail

# === 颜色定义 ===
RED='\033[0;31m'
YELLOW='\033[1;33m'
GREEN='\033[0;32m'
BLUE='\033[0;34m'
BOLD='\033[1m'
NC='\033[0m'

# === 输出函数 ===
info()    { echo -e "[${GREEN}*${NC}] $1"; }
warn()    { echo -e "[${YELLOW}!${NC}] $1"; }
alert()   { echo -e "[${RED}!!!${NC}] $1"; }
section() {
    echo ""
    echo -e "${BOLD}================================================================${NC}"
    echo -e "${BOLD}  $1${NC}"
    echo -e "${BOLD}================================================================${NC}"
}

# === 权限检查 ===
if [ "$(id -u)" -ne 0 ]; then
    echo "[-] This script must be run as root"
    exit 1
fi

# === 输出目录 ===
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
HOSTNAME=$(hostname)
OUTPUT_DIR="${1:-/tmp/ir_${HOSTNAME}_${TIMESTAMP}}"
mkdir -p "$OUTPUT_DIR"

# === 开始时间 ===
START_TIME=$(date +%s)
info "IR Collector started at $(date)"
info "Output directory: $OUTPUT_DIR"
info "Hostname: $HOSTNAME"
echo ""

3.2 系统信息模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
check_system_info() {
    section "System Information"
    local out="$OUTPUT_DIR/01_system_info.txt"
    
    {
        echo "=== Hostname ==="
        hostname
        
        echo -e "\n=== Date/Time ==="
        date -u
        echo "Timezone: $(timedatectl 2>/dev/null | grep 'Time zone' || cat /etc/timezone 2>/dev/null || echo 'unknown')"
        
        echo -e "\n=== Uptime ==="
        uptime
        
        echo -e "\n=== Kernel ==="
        uname -a
        
        echo -e "\n=== OS Release ==="
        cat /etc/os-release 2>/dev/null || cat /etc/redhat-release 2>/dev/null
        
        echo -e "\n=== CPU Info ==="
        lscpu | head -20
        
        echo -e "\n=== Memory ==="
        free -h
        
        echo -e "\n=== Disk Usage ==="
        df -hT
        
        echo -e "\n=== Mount Points ==="
        mount | column -t
        
        echo -e "\n=== Last Reboot ==="
        last reboot | head -5
        
        echo -e "\n=== Loaded Kernel Modules ==="
        lsmod
        
        echo -e "\n=== SELinux/AppArmor Status ==="
        getenforce 2>/dev/null || echo "SELinux not available"
        aa-status 2>/dev/null || echo "AppArmor not available"
    } > "$out" 2>&1
    
    info "System info collected -> $out"
}

3.3 用户账户审计模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
check_users() {
    section "User Account Audit"
    local out="$OUTPUT_DIR/02_users.txt"
    
    {
        echo "=== /etc/passwd (UID=0 accounts) ==="
        awk -F: '$3==0 {print "[ALERT] UID 0 account: "$0}' /etc/passwd
        
        echo -e "\n=== All users with login shell ==="
        grep -vE '(nologin|false|sync|halt|shutdown)$' /etc/passwd
        
        echo -e "\n=== /etc/shadow (check for empty passwords) ==="
        awk -F: '($2=="" || $2=="!") {print "[WARN] "$1" has empty/locked password"}' /etc/shadow 2>/dev/null
        awk -F: '($2!="*" && $2!="!" && $2!="!!" && $2!="") {print "[INFO] "$1" has password set"}' /etc/shadow 2>/dev/null
        
        echo -e "\n=== /etc/sudoers ==="
        cat /etc/sudoers 2>/dev/null | grep -v '^#' | grep -v '^$'
        
        echo -e "\n=== /etc/sudoers.d/ ==="
        ls -la /etc/sudoers.d/ 2>/dev/null
        cat /etc/sudoers.d/* 2>/dev/null | grep -v '^#' | grep -v '^$'
        
        echo -e "\n=== Currently Logged In Users ==="
        w
        
        echo -e "\n=== Last Logins ==="
        lastlog | grep -v "Never"
        
        echo -e "\n=== Failed Login Attempts ==="
        lastb 2>/dev/null | head -20 || echo "lastb not available"
        
        echo -e "\n=== SSH Authorized Keys ==="
        for home in /root /home/*; do
            if [ -f "$home/.ssh/authorized_keys" ]; then
                echo "--- $home/.ssh/authorized_keys ---"
                cat "$home/.ssh/authorized_keys"
                echo ""
            fi
        done
        
        echo -e "\n=== Recently Created Users (passwd mtime) ==="
        ls -la /etc/passwd /etc/shadow /etc/group
        
        echo -e "\n=== /etc/group (sudo/wheel members) ==="
        grep -E '^(sudo|wheel|admin):' /etc/group
    } > "$out" 2>&1
    
    # 终端告警
    uid0_count=$(awk -F: '$3==0' /etc/passwd | wc -l)
    if [ "$uid0_count" -gt 1 ]; then
        alert "Found $uid0_count accounts with UID=0!"
    fi
    
    info "User audit completed -> $out"
}

3.4 进程排查模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
check_processes() {
    section "Process Investigation"
    local out="$OUTPUT_DIR/03_processes.txt"
    
    {
        echo "=== Process Tree ==="
        ps auxwwf
        
        echo -e "\n=== Top CPU Consumers ==="
        ps aux --sort=-%cpu | head -20
        
        echo -e "\n=== Top Memory Consumers ==="
        ps aux --sort=-%mem | head -20
        
        echo -e "\n=== Processes Running from /tmp, /dev/shm, /var/tmp ==="
        ls -la /proc/*/exe 2>/dev/null | grep -E '(tmp|shm|dev)' && \
            echo "[ALERT] Processes running from suspicious locations!" || \
            echo "[OK] No processes from tmp directories"
        
        echo -e "\n=== Processes with Deleted Executables ==="
        ls -la /proc/*/exe 2>/dev/null | grep '(deleted)' && \
            echo "[ALERT] Found processes with deleted executables!" || \
            echo "[OK] No deleted executables"
        
        echo -e "\n=== Hidden Processes (comparing /proc vs ps) ==="
        # /proc 中存在但 ps 看不到的进程可能被 rootkit 隐藏
        comm -23 \
            <(ls /proc/ | grep -E '^[0-9]+$' | sort -n) \
            <(ps -eo pid --no-headers | awk '{print $1}' | sort -n) 2>/dev/null
        
        echo -e "\n=== Processes Listening on Network ==="
        if command -v ss &>/dev/null; then
            ss -tlnp
        else
            netstat -tlnp
        fi
        
        echo -e "\n=== /proc/*/cmdline for Suspicious Processes ==="
        for pid in $(ls /proc/ | grep -E '^[0-9]+$'); do
            cmdline=$(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ')
            if echo "$cmdline" | grep -qiE '(miner|xmrig|kdevtmpfsi|kinsing|\.hidden|base64)'; then
                echo "[ALERT] PID $pid: $cmdline"
            fi
        done
        
        echo -e "\n=== /proc/*/environ for Suspicious Vars ==="
        for pid in $(ls /proc/ | grep -E '^[0-9]+$'); do
            env_str=$(cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n')
            if echo "$env_str" | grep -qi 'LD_PRELOAD'; then
                echo "[ALERT] PID $pid has LD_PRELOAD set:"
                echo "$env_str" | grep -i 'LD_PRELOAD'
            fi
        done
        
        echo -e "\n=== Open Files (lsof summary) ==="
        lsof -nPl 2>/dev/null | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
    } > "$out" 2>&1
    
    info "Process investigation completed -> $out"
}

3.5 网络排查模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
check_network() {
    section "Network Investigation"
    local out="$OUTPUT_DIR/04_network.txt"
    
    {
        echo "=== Network Interfaces ==="
        ip addr show
        
        echo -e "\n=== Routing Table ==="
        ip route show
        
        echo -e "\n=== ARP Table ==="
        ip neigh show
        
        echo -e "\n=== All TCP/UDP Connections ==="
        if command -v ss &>/dev/null; then
            ss -antup
        else
            netstat -antup
        fi
        
        echo -e "\n=== Listening Ports ==="
        if command -v ss &>/dev/null; then
            ss -tlnp
            echo ""
            ss -ulnp
        else
            netstat -tlnp
            echo ""
            netstat -ulnp
        fi
        
        echo -e "\n=== ESTABLISHED Connections to External IPs ==="
        if command -v ss &>/dev/null; then
            ss -tnp state established | grep -vE '(127\.0\.0\.1|::1|10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)'
        fi
        
        echo -e "\n=== DNS Configuration ==="
        cat /etc/resolv.conf
        
        echo -e "\n=== /etc/hosts ==="
        cat /etc/hosts
        
        echo -e "\n=== Firewall Rules (iptables) ==="
        iptables -L -n -v 2>/dev/null || echo "iptables not available"
        
        echo -e "\n=== Firewall Rules (nftables) ==="
        nft list ruleset 2>/dev/null || echo "nftables not available"
        
        echo -e "\n=== Promiscuous Mode Check ==="
        ip link show | grep -i promisc && \
            echo "[ALERT] Promiscuous mode detected (possible sniffer)!" || \
            echo "[OK] No promiscuous interfaces"
    } > "$out" 2>&1
    
    info "Network investigation completed -> $out"
}

3.6 文件系统排查模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
check_filesystem() {
    section "Filesystem Investigation"
    local out="$OUTPUT_DIR/05_filesystem.txt"
    
    {
        echo "=== Recently Modified Files (7 days, /etc/) ==="
        find /etc/ -type f -mtime -7 -ls 2>/dev/null | sort -k11
        
        echo -e "\n=== Recently Modified Files (7 days, /var/www/) ==="
        find /var/www/ -type f -mtime -7 -ls 2>/dev/null | sort -k11
        
        echo -e "\n=== Recently Modified Files (3 days, /tmp/) ==="
        find /tmp/ -type f -mtime -3 -ls 2>/dev/null | sort -k11
        
        echo -e "\n=== SUID Files ==="
        find / -perm -4000 -type f -ls 2>/dev/null
        
        echo -e "\n=== SGID Files ==="
        find / -perm -2000 -type f -ls 2>/dev/null
        
        echo -e "\n=== World-Writable Files (excl /proc, /sys, /dev) ==="
        find / -path /proc -prune -o -path /sys -prune -o -path /dev -prune \
            -o -type f -perm -0002 -ls 2>/dev/null | grep -v -E '(/proc|/sys|/dev)'
        
        echo -e "\n=== Hidden Files in /tmp, /var/tmp, /dev/shm ==="
        find /tmp/ /var/tmp/ /dev/shm/ -name ".*" -ls 2>/dev/null
        
        echo -e "\n=== Executable Files in /tmp, /var/tmp, /dev/shm ==="
        find /tmp/ /var/tmp/ /dev/shm/ -type f -executable -ls 2>/dev/null
        
        echo -e "\n=== Large Files (>100MB) in /tmp ==="
        find /tmp/ /var/tmp/ -type f -size +100M -ls 2>/dev/null
        
        echo -e "\n=== Immutable Files (chattr +i) ==="
        lsattr -R /etc/ 2>/dev/null | grep -E '^....i'
        lsattr -R /usr/bin/ 2>/dev/null | grep -E '^....i'
        lsattr -R /tmp/ 2>/dev/null | grep -E '^....i'
        
        echo -e "\n=== RPM/DPKG Verification ==="
        if command -v rpm &>/dev/null; then
            echo "--- RPM Verify (modified system files) ---"
            rpm -Va 2>/dev/null | grep -vE '^\.{8}' | head -50
        elif command -v dpkg &>/dev/null; then
            echo "--- DPKG Verify ---"
            dpkg --verify 2>/dev/null | head -50
        fi
    } > "$out" 2>&1
    
    info "Filesystem investigation completed -> $out"
}

更多文件系统取证技巧详见 06-文件系统取证

3.7 计划任务审计模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
check_crontabs() {
    section "Scheduled Tasks Audit"
    local out="$OUTPUT_DIR/06_crontabs.txt"
    
    {
        echo "=== System Crontab ==="
        cat /etc/crontab 2>/dev/null
        
        echo -e "\n=== /etc/cron.d/ ==="
        ls -la /etc/cron.d/ 2>/dev/null
        for f in /etc/cron.d/*; do
            echo "--- $f ---"
            cat "$f" 2>/dev/null
            echo ""
        done
        
        echo -e "\n=== /etc/cron.hourly/ ==="
        ls -la /etc/cron.hourly/ 2>/dev/null
        
        echo -e "\n=== /etc/cron.daily/ ==="
        ls -la /etc/cron.daily/ 2>/dev/null
        
        echo -e "\n=== /etc/cron.weekly/ ==="
        ls -la /etc/cron.weekly/ 2>/dev/null
        
        echo -e "\n=== /etc/cron.monthly/ ==="
        ls -la /etc/cron.monthly/ 2>/dev/null
        
        echo -e "\n=== User Crontabs ==="
        for user in $(cut -d: -f1 /etc/passwd); do
            crontab_content=$(crontab -l -u "$user" 2>/dev/null)
            if [ -n "$crontab_content" ]; then
                echo "--- User: $user ---"
                echo "$crontab_content"
                echo ""
            fi
        done
        
        echo -e "\n=== /var/spool/cron/ ==="
        ls -la /var/spool/cron/ 2>/dev/null
        ls -la /var/spool/cron/crontabs/ 2>/dev/null
        for f in /var/spool/cron/* /var/spool/cron/crontabs/*; do
            if [ -f "$f" ]; then
                echo "--- $f ---"
                cat "$f" 2>/dev/null
                echo ""
            fi
        done
        
        echo -e "\n=== Systemd Timers ==="
        systemctl list-timers --all --no-pager 2>/dev/null
        
        echo -e "\n=== at Jobs ==="
        atq 2>/dev/null || echo "at not available"
        ls -la /var/spool/at/ 2>/dev/null
        
        echo -e "\n=== Suspicious Crontab Entries ==="
        # 检查包含下载执行模式的 crontab
        grep -rE '(wget|curl).*\|(ba)?sh' \
            /etc/crontab /etc/cron.d/ /var/spool/cron/ 2>/dev/null && \
            echo "[ALERT] Found download-and-execute in crontab!" || \
            echo "[OK] No obvious malicious crontab entries"
    } > "$out" 2>&1
    
    info "Crontab audit completed -> $out"
}

3.8 服务审计模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
check_services() {
    section "Service Audit"
    local out="$OUTPUT_DIR/07_services.txt"
    
    {
        echo "=== Systemd Services (enabled) ==="
        systemctl list-unit-files --type=service --state=enabled --no-pager 2>/dev/null
        
        echo -e "\n=== Systemd Services (running) ==="
        systemctl list-units --type=service --state=running --no-pager 2>/dev/null
        
        echo -e "\n=== Failed Services ==="
        systemctl list-units --type=service --state=failed --no-pager 2>/dev/null
        
        echo -e "\n=== init.d Services ==="
        ls -la /etc/init.d/ 2>/dev/null
        
        echo -e "\n=== rc.local ==="
        cat /etc/rc.local 2>/dev/null
        
        echo -e "\n=== Recently Modified Service Files ==="
        find /etc/systemd/system/ /usr/lib/systemd/system/ \
            -name "*.service" -mtime -30 -ls 2>/dev/null
        
        echo -e "\n=== Non-standard Service Files ==="
        # 查找不属于软件包的 service 文件
        find /etc/systemd/system/ -name "*.service" -type f 2>/dev/null | \
            while read svc; do
                if command -v rpm &>/dev/null; then
                    rpm -qf "$svc" 2>/dev/null || echo "[WARN] Unpackaged: $svc"
                elif command -v dpkg &>/dev/null; then
                    dpkg -S "$svc" 2>/dev/null || echo "[WARN] Unpackaged: $svc"
                fi
            done
        
        echo -e "\n=== Docker Containers (if present) ==="
        docker ps -a 2>/dev/null || echo "Docker not available"
    } > "$out" 2>&1
    
    info "Service audit completed -> $out"
}

3.9 后门检测模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
check_backdoors() {
    section "Backdoor Detection"
    local out="$OUTPUT_DIR/08_backdoors.txt"
    
    {
        # --- LD_PRELOAD 检测 ---
        echo "=== LD_PRELOAD Checks ==="
        
        echo "--- /etc/ld.so.preload ---"
        if [ -f /etc/ld.so.preload ]; then
            echo "[ALERT] /etc/ld.so.preload exists!"
            cat /etc/ld.so.preload
        else
            echo "[OK] /etc/ld.so.preload not found"
        fi
        
        echo -e "\n--- LD_PRELOAD in environment ---"
        grep -r "LD_PRELOAD" /etc/environment /etc/profile /etc/profile.d/ \
            /etc/bashrc /etc/bash.bashrc 2>/dev/null && \
            echo "[ALERT] LD_PRELOAD found in system environment!" || \
            echo "[OK] No LD_PRELOAD in system environment"
        
        echo -e "\n--- Process LD_PRELOAD ---"
        for pid in $(ls /proc/ | grep -E '^[0-9]+$'); do
            preload=$(cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n' | grep LD_PRELOAD)
            if [ -n "$preload" ]; then
                cmdline=$(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ')
                echo "[ALERT] PID $pid ($cmdline): $preload"
            fi
        done
        
        # --- SSH 后门检测 ---
        echo -e "\n=== SSH Backdoor Checks ==="
        
        echo "--- SSH authorized_keys anomalies ---"
        for home in /root /home/*; do
            ak="$home/.ssh/authorized_keys"
            if [ -f "$ak" ]; then
                # 检查 command= 限制 (可能是正常的，也可能是后门)
                grep -n 'command=' "$ak" 2>/dev/null && \
                    echo "[WARN] Found command= in $ak"
                # 检查 no-pty 等限制
                grep -n 'no-pty\|permitopen' "$ak" 2>/dev/null
                # 统计 key 数量
                count=$(wc -l < "$ak")
                echo "[INFO] $ak: $count keys"
            fi
        done
        
        echo -e "\n--- SSH config backdoors ---"
        grep -E '(PermitRootLogin|AuthorizedKeysFile|PasswordAuthentication)' \
            /etc/ssh/sshd_config 2>/dev/null
        
        echo -e "\n--- SSH wrapper check ---"
        file $(which sshd 2>/dev/null) 2>/dev/null
        md5sum $(which sshd 2>/dev/null) 2>/dev/null
        
        # --- SUID 后门检测 ---
        echo -e "\n=== SUID Backdoor Checks ==="
        echo "--- Unusual SUID files ---"
        # 已知合法的 SUID 列表
        known_suid="/usr/bin/passwd /usr/bin/sudo /usr/bin/su /usr/bin/newgrp"
        known_suid="$known_suid /usr/bin/gpasswd /usr/bin/chsh /usr/bin/chfn"
        known_suid="$known_suid /usr/bin/mount /usr/bin/umount /usr/bin/ping"
        known_suid="$known_suid /usr/bin/crontab /usr/bin/at /usr/bin/pkexec"
        
        find / -perm -4000 -type f 2>/dev/null | while read f; do
            if ! echo "$known_suid" | grep -q "$f"; then
                echo "[WARN] Non-standard SUID: $(ls -la "$f")"
            fi
        done
        
        # --- PAM 后门检测 ---
        echo -e "\n=== PAM Backdoor Checks ==="
        echo "--- Modified PAM modules ---"
        find /lib/security/ /lib64/security/ /usr/lib/security/ \
            -name "*.so" -mtime -30 -ls 2>/dev/null
        # 检查 PAM 配置
        grep -rn "pam_exec\|pam_script" /etc/pam.d/ 2>/dev/null && \
            echo "[WARN] Found pam_exec/pam_script in PAM config!" || \
            echo "[OK] No pam_exec/pam_script"
        
        # --- 别名后门检测 ---
        echo -e "\n=== Alias Backdoor Checks ==="
        for home in /root /home/*; do
            for rc in .bashrc .bash_profile .profile .zshrc; do
                if [ -f "$home/$rc" ]; then
                    suspicious=$(grep -n 'alias\s\+\(sudo\|su\|ssh\|ls\|ps\|netstat\)=' "$home/$rc" 2>/dev/null)
                    if [ -n "$suspicious" ]; then
                        echo "[WARN] Suspicious alias in $home/$rc:"
                        echo "$suspicious"
                    fi
                fi
            done
        done
        
        # --- 启动项后门 ---
        echo -e "\n=== Startup Backdoor Checks ==="
        echo "--- /etc/profile.d/ scripts ---"
        ls -la /etc/profile.d/ 2>/dev/null
        for f in /etc/profile.d/*.sh; do
            if [ -f "$f" ]; then
                # 检查可疑内容
                if grep -qiE '(wget|curl|nc |ncat|python.*socket|bash.*-i)' "$f" 2>/dev/null; then
                    echo "[ALERT] Suspicious content in $f:"
                    grep -iE '(wget|curl|nc |ncat|python.*socket|bash.*-i)' "$f"
                fi
            fi
        done
        
        echo -e "\n--- /etc/rc.local ---"
        if [ -f /etc/rc.local ]; then
            cat /etc/rc.local
        fi
        
        # --- 动态链接库后门 ---
        echo -e "\n=== Shared Library Backdoor Checks ==="
        echo "--- /etc/ld.so.conf.d/ ---"
        cat /etc/ld.so.conf 2>/dev/null
        ls -la /etc/ld.so.conf.d/ 2>/dev/null
        cat /etc/ld.so.conf.d/* 2>/dev/null
    } > "$out" 2>&1
    
    info "Backdoor detection completed -> $out"
}

3.10 日志收集模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
collect_logs() {
    section "Log Collection"
    local out="$OUTPUT_DIR/09_logs"
    mkdir -p "$out"
    
    # 拷贝关键日志
    cp -a /var/log/auth.log* "$out/" 2>/dev/null
    cp -a /var/log/secure* "$out/" 2>/dev/null
    cp -a /var/log/syslog* "$out/" 2>/dev/null
    cp -a /var/log/messages* "$out/" 2>/dev/null
    cp -a /var/log/cron* "$out/" 2>/dev/null
    cp -a /var/log/kern.log* "$out/" 2>/dev/null
    cp -a /var/log/lastlog "$out/" 2>/dev/null
    cp -a /var/log/wtmp* "$out/" 2>/dev/null
    cp -a /var/log/btmp* "$out/" 2>/dev/null
    
    # Web 服务器日志
    cp -a /var/log/nginx/ "$out/nginx/" 2>/dev/null
    cp -a /var/log/apache2/ "$out/apache2/" 2>/dev/null
    cp -a /var/log/httpd/ "$out/httpd/" 2>/dev/null
    
    # journalctl 导出
    journalctl --since "7 days ago" --no-pager > "$out/journal_7days.txt" 2>/dev/null
    
    # 审计日志
    cp -a /var/log/audit/ "$out/audit/" 2>/dev/null
    
    # 日志摘要
    {
        echo "=== Log File Sizes ==="
        du -sh /var/log/* 2>/dev/null | sort -rh | head -20
        
        echo -e "\n=== SSH Login Summary (last 7 days) ==="
        grep "Accepted\|Failed" /var/log/auth.log 2>/dev/null | tail -50
        grep "Accepted\|Failed" /var/log/secure 2>/dev/null | tail -50
        
        echo -e "\n=== sudo Usage ==="
        grep "sudo:" /var/log/auth.log 2>/dev/null | tail -30
        grep "sudo:" /var/log/secure 2>/dev/null | tail -30
        
        echo -e "\n=== Log Tampering Check ==="
        # 检查日志是否有时间跳跃 (可能被清除后重新开始)
        for log in /var/log/auth.log /var/log/secure /var/log/syslog /var/log/messages; do
            if [ -f "$log" ]; then
                first=$(head -1 "$log" | awk '{print $1,$2,$3}')
                last=$(tail -1 "$log" | awk '{print $1,$2,$3}')
                echo "$log: first=$first, last=$last"
            fi
        done
    } > "$out/log_summary.txt" 2>&1
    
    info "Log collection completed -> $out/"
}

3.11 报告生成模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
generate_report() {
    section "Generating Report"
    local report="$OUTPUT_DIR/00_SUMMARY_REPORT.txt"
    
    {
        echo "================================================================"
        echo "  Linux IR Collection Report"
        echo "================================================================"
        echo "Hostname:     $HOSTNAME"
        echo "Date:         $(date -u)"
        echo "Collector:    ir-collector.sh v1.0"
        echo "Output Dir:   $OUTPUT_DIR"
        echo ""
        
        END_TIME=$(date +%s)
        DURATION=$((END_TIME - START_TIME))
        echo "Collection Time: ${DURATION} seconds"
        echo ""
        
        echo "================================================================"
        echo "  ALERTS & WARNINGS"
        echo "================================================================"
        # 汇总所有文件中的告警
        grep -rh "\[ALERT\]" "$OUTPUT_DIR"/*.txt 2>/dev/null | sort -u
        echo ""
        grep -rh "\[WARN\]" "$OUTPUT_DIR"/*.txt 2>/dev/null | sort -u
        echo ""
        
        echo "================================================================"
        echo "  Collected Files"
        echo "================================================================"
        ls -la "$OUTPUT_DIR"/
        echo ""
        du -sh "$OUTPUT_DIR"
    } > "$report" 2>&1
    
    # 计算所有收集文件的 hash
    sha256sum "$OUTPUT_DIR"/* 2>/dev/null > "$OUTPUT_DIR/checksums.sha256"
    
    # 打包
    tar czf "${OUTPUT_DIR}.tar.gz" -C "$(dirname "$OUTPUT_DIR")" "$(basename "$OUTPUT_DIR")"
    sha256sum "${OUTPUT_DIR}.tar.gz" > "${OUTPUT_DIR}.tar.gz.sha256"
    
    info "Report generated -> $report"
    info "Archive: ${OUTPUT_DIR}.tar.gz"
    info "Archive SHA256: $(cat "${OUTPUT_DIR}.tar.gz.sha256")"
}

3.12 主函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
main() {
    info "========================================="
    info "  Linux IR Collector v1.0"
    info "========================================="
    
    check_system_info
    check_users
    check_processes
    check_network
    check_filesystem
    check_crontabs
    check_services
    check_backdoors
    collect_logs
    generate_report
    
    echo ""
    section "Collection Complete"
    alert "Review ALERTS in: $OUTPUT_DIR/00_SUMMARY_REPORT.txt"
    info "Total output: $(du -sh "$OUTPUT_DIR" | awk '{print $1}')"
    info "Archive: ${OUTPUT_DIR}.tar.gz"
}

# 执行
main "$@"

更多排查命令详见 02-排查命令速查

4. 开源工具对比

4.1 LinPEAS — 权限提升枚举

项目 说明
用途 Linux 权限提升路径枚举 (红队视角，应急时可反向利用)
地址 github.com/carlospolop/PEASS-ng
安装 curl -L https://github.com/carlospolop/PEASS-ng/releases/latest/download/linpeas.sh -o linpeas.sh
使用 chmod +x linpeas.sh && ./linpeas.sh -a 2>&1 | tee linpeas_output.txt 
1
2
3
4
# 常用选项
./linpeas.sh -a          # 全面扫描
./linpeas.sh -s          # 静默模式 (减少输出)
./linpeas.sh -e /tmp/out # 导出结果

优点:

检查项极其全面 (SUID、Capabilities、内核漏洞、配置错误等)

输出带颜色标记，红色/黄色高亮风险项

单文件，无需安装依赖

缺点:

红队工具，可能被 EDR/AV 检测

输出量极大，需要经验筛选

部分检查涉及主动探测

4.2 linux-smart-enumeration (LSE)

项目 说明
用途 Linux 安全枚举，比 LinPEAS 更简洁
地址 github.com/diego-treitos/linux-smart-enumeration
安装 curl -L https://github.com/diego-treitos/linux-smart-enumeration/releases/latest/download/lse.sh -o lse.sh
使用 chmod +x lse.sh && ./lse.sh -l 2 
1
2
3
4
5
# 等级控制
./lse.sh -l 0   # 只显示重要发现
./lse.sh -l 1   # 显示有趣的信息
./lse.sh -l 2   # 显示所有信息
./lse.sh -s      # 选择特定部分

优点:

输出分级，可控制详细程度

比 LinPEAS 更轻量

POSIX sh 兼容，不依赖 bash

缺点:

检查项不如 LinPEAS 全面

更新频率较低

4.3 Lynis — 安全审计

项目 说明
用途 系统安全审计和加固评估
地址 github.com/CISOfy/lynis
安装 sudo apt install lynis 或 git clone https://github.com/CISOfy/lynis.git
使用 sudo lynis audit system 
1
2
3
4
5
6
7
8
9
10
11
12
# 完整审计
sudo lynis audit system --no-colors 2>&1 | tee lynis_report.txt

# 只检查特定类别
sudo lynis audit system --tests-from-group "malware"
sudo lynis audit system --tests-from-group "authentication"

# 查看建议
grep "suggestion" /var/log/lynis.log

# 输出安全评分
grep "hardening_index" /var/log/lynis-report.dat

优点:

专业安全审计工具，不是 “黑客工具”

输出安全评分 (Hardening Index)

提供具体加固建议

支持合规检查 (PCI DSS, HIPAA 等)

缺点:

偏向安全加固评估，不专注于应急响应

企业版功能需要付费

4.4 ClamAV — 恶意软件扫描

项目 说明
用途 开源杀毒引擎
地址 github.com/Cisco-Talos/clamav
安装 sudo apt install clamav clamav-daemon
使用 sudo clamscan -r /var/www/ 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 更新病毒库 (重要!)
sudo systemctl stop clamav-freshclam
sudo freshclam
sudo systemctl start clamav-freshclam

# 扫描 Web 目录
sudo clamscan -r -i /var/www/html/ 2>&1 | tee clamscan_web.txt
# -r: 递归
# -i: 只显示感染文件

# 扫描临时目录
sudo clamscan -r -i /tmp/ /var/tmp/ /dev/shm/

# 使用 ClamD (守护进程模式，更快)
sudo clamdscan -m /var/www/html/

# 配合自定义 YARA 规则
sudo clamscan -r -d /path/to/custom.yar /var/www/

优点:

开源免费，病毒库定期更新

支持自定义 YARA 规则

可以作为守护进程实时扫描

缺点:

检测率不如商业杀毒软件

对 Linux 恶意软件检测能力有限

全盘扫描速度较慢

4.5 对比总结

工具 侧重 速度 输出质量 适用阶段
ir-collector.sh IR 证据收集 快 结构化 应急响应初期
LinPEAS 提权路径枚举 中 颜色标记 深入排查
LSE 安全枚举 快 分级输出 快速检查
Lynis 安全审计 慢 专业报告 加固评估
ClamAV 恶意软件扫描 慢 简洁 恶意文件检测
YARA 自定义特征匹配 快 精准 针对性检测

5. 定制化与集成

5.1 根据环境定制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# === 自定义配置文件 ir-collector.conf ===

# 要扫描的 Web 目录
WEB_DIRS="/var/www /opt/webapp /data/www"

# 要排除的目录 (大文件、已知安全)
EXCLUDE_DIRS="/proc /sys /dev /run /snap"

# 已知合法的 SUID 文件 (白名单)
SUID_WHITELIST="/usr/bin/passwd /usr/bin/sudo /usr/bin/su"

# 关注的时间范围 (天)
DAYS_BACK=7

# 输出格式: text / json
OUTPUT_FORMAT="text"

# 是否收集日志文件副本
COLLECT_LOGS=true

# 最大输出大小 (MB)
MAX_OUTPUT_SIZE=500

5.2 JSON 输出 (便于 SIEM 集成)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# JSON 格式输出函数
json_alert() {
    local category="$1"
    local message="$2"
    local detail="$3"
    echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"host\":\"$HOSTNAME\",\"severity\":\"alert\",\"category\":\"$category\",\"message\":\"$message\",\"detail\":\"$detail\"}"
}

# 使用示例
json_alert "user" "UID 0 account found" "toor:x:0:0::/root:/bin/bash"
# 输出:
# {"timestamp":"2024-01-15T14:30:00Z","host":"web-01","severity":"alert",
#  "category":"user","message":"UID 0 account found",
#  "detail":"toor:x:0:0::/root:/bin/bash"}

5.3 与 SOAR 集成

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 将结果发送到 SOAR 平台 (以 TheHive 为例)
send_to_thehive() {
    local api_url="http://thehive.internal:9000/api"
    local api_key="YOUR_API_KEY"
    local case_id="$1"
    local file="$2"
    
    curl -s -X POST "$api_url/case/$case_id/artifact" \
        -H "Authorization: Bearer $api_key" \
        -F "data=@$file" \
        -F '_json={"dataType":"file","message":"IR Collection from '$HOSTNAME'"}'
}

# 使用
# send_to_thehive "~123456" "${OUTPUT_DIR}.tar.gz"

5.4 批量部署

1
2
# 使用 Ansible 批量在多台服务器上运行
# playbook: ir-scan.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
---
- name: Emergency IR Scan
  hosts: all_servers
  become: yes
  tasks:
    - name: Copy IR collector script
      copy:
        src: scripts/ir-collector.sh
        dest: /tmp/ir-collector.sh
        mode: '0755'

    - name: Run IR collector
      command: /tmp/ir-collector.sh /tmp/ir_output
      timeout: 600

    - name: Fetch results
      fetch:
        src: "/tmp/ir_output.tar.gz"
        dest: "results/{{ inventory_hostname }}/"
        flat: yes

    - name: Cleanup
      file:
        path: "{{ item }}"
        state: absent
      loop:
        - /tmp/ir-collector.sh
        - /tmp/ir_output
        - /tmp/ir_output.tar.gz
1
2
3
4
5
6
7
8
9
10
# 执行批量扫描
ansible-playbook -i inventory.ini ir-scan.yml

# 汇总分析所有结果
for host_dir in results/*/; do
    echo "=== $(basename "$host_dir") ==="
    tar xzf "$host_dir/ir_output.tar.gz" -C "$host_dir"
    grep -rh "\[ALERT\]" "$host_dir" 2>/dev/null
    echo ""
done

6. 完整模块化脚本代码

以下是将上述所有模块整合后的完整可运行脚本骨架

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
#!/bin/bash
#====================================================================
# ir-collector.sh — Linux Emergency Response Collection Script
# Version: 1.0
# Usage:   sudo bash ir-collector.sh [output_dir]
# Author:  IR Team
# Note:    READ-ONLY — does NOT modify the target system
#====================================================================

set -uo pipefail

# === 全局变量 ===
VERSION="1.0"
RED='\033[0;31m'; YELLOW='\033[1;33m'; GREEN='\033[0;32m'
BOLD='\033[1m'; NC='\033[0m'

info()    { echo -e "[${GREEN}*${NC}] $1"; }
warn()    { echo -e "[${YELLOW}!${NC}] $1"; }
alert()   { echo -e "[${RED}!!!${NC}] $1"; }
section() { echo -e "\n${BOLD}===== $1 =====${NC}"; }

# === 检查 root ===
[ "$(id -u)" -ne 0 ] && { echo "Run as root"; exit 1; }

# === 输出目录 ===
TS=$(date +%Y%m%d_%H%M%S)
HN=$(hostname)
OUT="${1:-/tmp/ir_${HN}_${TS}}"
mkdir -p "$OUT"
START=$(date +%s)

info "IR Collector v${VERSION} started — $(date)"
info "Output: $OUT"

# === 加载模块 ===
# 将上述 check_system_info, check_users, check_processes,
# check_network, check_filesystem, check_crontabs,
# check_services, check_backdoors, collect_logs,
# generate_report 函数粘贴到此处

# === 主流程 ===
main() {
  check_system_info
  check_users
  check_processes
  check_network
  check_filesystem
  check_crontabs
  check_services
  check_backdoors
  collect_logs
  generate_report
  
  section "DONE"
  info "Review: $OUT/00_SUMMARY_REPORT.txt"
  info "Archive: ${OUT}.tar.gz"
}

main "$@"

完整脚本文件位于: scripts/ir-collector.sh

建议配合 00-学习路线 中的实验环境进行练习

相关章节

02-排查命令速查 — 手动排查命令参考

00-学习路线 — 完整学习路径

29-取证工具 — 数字取证工具 (LiME, Volatility, Sleuthkit)

30-YARA规则 — YARA 规则编写与使用

上一章 目录 下一章
30-YARA规则 Linux应急响应 Lab-README