ARM64下 Kdump & Crash

大家好，我是讯享网，很高兴认识大家。

1.工具准备

1.1 kexec

kexec 是一个 Linux 内核到内核的引导加载程序，可以帮助从第一个内核的上下文引导到第二个内核。kexec会关闭第一个内核，绕过BIOS或固件阶段，并跳转到第二个内核。在没有 BIOS 阶段的情况下，重新启动变得更快。

下载最新的kexec-tools源码包

tar xvpzf kexec-tools.tar.gz

讯享网

进入到kexec-tools目录中，并进行配置（静态编译）

讯享网LDFLAGS=-static ./configure ARCH=arm64 --build=x86_64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu --without-xen

编译

make

生成的工具即在build目录下，可拷贝至文件系统中

1.2 crash

crash是redhat的工程师开发的，主要用来离线分析linux内核转存文件，它整合了gdb工具，功能非常强大。可以查看堆栈，dmesg日志，内核数据结构，反汇编等等。crash支持多种工具生成的转存文件格式，如kdump，LKCD，netdump和diskdump，而且还可以分析虚拟机Xen和Kvm上生成的内核转存文件。同时crash还可以调试运行时系统，直接运行crash即可，ubuntu下内核映象存放在/proc/kcore。

下载

crash不同版本下载地址

解压编译

讯享网$ tar -vxzf crash-7.3.0.tar.gz $ cd crash-7.3.0/ $ make target=arm64

首次编译时总是报一个错误，我的PC系统是ubuntu20.04，安装一些工具后解决

sudo apt-get install libaio-dev libncurses5-dev zlib1g-dev liblzma-dev flex bison byacc

验证

讯享网ycy@DESKTOP-IDHR9T5:~/tool/crash-7.3.0$ ./crash --buildinfo build_command: crash build_data: Wed Jun 9 11:41:35 CST 2021 by uid=1000(ycy) on DESKTOP-IDHR9T5 build_target: ARM64 build_version: 7.3.0 compiler version: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0

2.整体流程

2.1 使能配置项

CONFIG_KEXEC=y
CONFIG_SYSFS=y
CONFIG_DEBUG_INFO=y
CONFIG_CRASH_DUMP=y
CONFIG_PROC_VMCORE=y

2.2 预留内存

在内核命令行中传递以下启动项，其中 @X 是可选的。

crashkernel=Y[@X] // Y: 内核的大小 // X: 内核的起始地址（起始地址必须与2MiB (0x)对齐）

通过cat /proc/iomem 可以查看预留内存是否成

讯享网-1efffffff : System RAM 1a0000000-1a7ffffff : Crash kernel 1eab38000-1eadfffff : reserved 1eae11000-1efdfffff : reserved 1efe30000-1efe4ffff : reserved 1efe50000-1efe50fff : reserved 1efe53000-1efe53fff : reserved 1efe54000-1eff6ffff : reserved 1eff70000-1efffffff : reserved

2.3 加载内核

kexec -p <dump-capture-kernel-Image> \ --initrd=<initrd-for-dump-capture-kernel> \ --append="root=<root-dev> <arch-specific-options>" //下面是当时使用的特定于arch的命令行选项 For arm64: "1 maxcpus=1 reset_devices"

2.4 检查捕获内核加载状况

讯享网//查看捕获内核加载状态 0：未加载 1：已加载 root@a1000:/# cat /sys/kernel/kexec_crash_loaded //查看捕获内核加载大小 root@a1000:/# cat /sys/kernel/kexec_crash_size

2.5 启动捕获内核

在如前所述成功加载转储-捕获内核之后，如果触发系统崩溃，系统将重新引导到转储-捕获内核。触发点位于panic()、die()、die_nmi()和sysrq处理程序(ALT-SysRq-c)中。

讯享网

可通过以下指令触发panic测试：

root@a1000:~# echo c > /proc/sysrq-trigger

2.6 拷贝vmcore

将proc目录下的vmcore文件拷贝至你的存储设备中即可，vmcore就是第一个内核崩溃时的内存镜像

提供一个脚本：

讯享网#!/bin/sh KDUMP_PATH="/home" function save_dump_file() { # datename=$(date +%Y-%m-%d-%H:%M:%S) coredir="${KDUMP_PATH}/kdump" mkdir -p $coredir if [ ! -f /sbin/vmcore-dmesg ];then echo “Skipping saving vmcore-dmesg.txt.File /sbin/vmcore-dmesg is not present” return; fi echo “kdump: saving vmcore-dmesg.txt to $coredir” if [ ! -f "$coredir/vmcore_1" ]; then /sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_1.txt cp /proc/vmcore $coredir/vmcore_1 else if [ ! -f "$coredir/vmcore_2" ]; then /sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_2.txt cp /proc/vmcore $coredir/vmcore_2 else rm $coredir/vmcore_1 $coredir/vmcore-dmesg_1.txt mv $coredir/vmcore_2 $coredir/vmcore_1 mv $coredir/vmcore-dmesg_2.txt $coredir/vmcore-dmesg_1.txt /sbin/vmcore-dmesg /proc/vmcore > $coredir/vmcore-dmesg_2.txt cp /proc/vmcore $coredir/vmcore_2 fi fi } if [ -s /proc/vmcore ];then echo “save core dump file…” save_dump_file reboot else # kexec setting /sbin/kexec -p --append="earlycon=uart8250,mmio32,0x console=ttyS0,n8 memreserve=64M@0xf rw boot_delay=32 1 maxcpus=1 reset_devices root=/dev/mmcblk0p9 boot_delay=32 rootwait init=/linuxrc" --dtb=/home/root/bsta1000-fadb.dtb /home/root/Image fi

3.Crash使用详解

3.1 背景知识

3.2 使用方法

crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]     (dumpfile form)
crash [OPTION]... [NAMELIST]                          (live system form)

ex:
ycy@DESKTOP-IDHR9T5:~/tool/crash-7.3.0$ ./crash vmlinux ~/te/vmcore

crash 7.3.0
Copyright (C) 2002-2021  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-elf-linux"...
... ...

crash>

crash在加载内核转存文件是会输出系统基本信息，如出问题的进程（bash - 2613），系统内存大小（7.9GB），系统架构（x86_64）等等，可以看到这个dump是sysrq触发的一个panic系统崩溃。

讯享网This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-elf-linux"... KERNEL: vmlinux DUMPFILE: /home/ycy/te/vmcore CPUS: 8 DATE: Sun May 16 16:50:05 CST 2021 UPTIME: 00:15:09 LOAD AVERAGE: 5.06, 4.79, 3.15 TASKS: 274 NODENAME: a1000 RELEASE: 4.19.66+1.0.0 VERSION: #2 SMP Tue Jun 15 09:27:09 CST 2021 MACHINE: aarch64 (unknown Mhz) MEMORY: 2.6 GB PANIC: "sysrq: SysRq : Trigger a crash" PID: 641 COMMAND: "sh" TASK: ffff8001df128dc0 [THREAD_INFO: ffff8001df128dc0] CPU: 5 STATE: TASK_RUNNING (SYSRQ) crash>

3.3 常用命令

Crash命令列表

命令	功能

| 指针快捷健
alias | 命令快捷键
ascii | ASCII码转换和码表
bpf | eBPF - extended Berkeley Filter
bt | 堆栈查看
btop | 地址页表转换
dev | 设备数据查询
dis | 反汇编
eval | 计算器
exit | 退出
extend | 命令扩展
files | 打开的文件查看
foreach | 循环查看
fuser | 文件使用者查看
gdb | 调用gdb执行命令
help | 帮助
ipcs | 查看system V IPC工具
irq | 查看irq数据
kmem | 查看Kernel内存
list | 查看链表
log | 查看系统消息缓存
mach | 查看平台信息
mod | 加载符号表
mount | Mount文件系统数据
net | 网络命令
p | 查看数据结构
ps | 查看进程状态信息
pte | 查看页表
ptob | 页表地址转换
ptov | 物理地址虚拟地址转换
rd | 查看内存
repeat | 重复执行
runq | 查看run queue上的线程
search | 搜索内存
set | 设置线程环境和Crash内部变量
sig | 查询线程消息
struct | 查询结构体
swap | 查看swap信息
sym | 符号和虚拟地址转换
sys | 查看系统信息
task | 查看task_struct和thread_thread信息
timer | 查看timer队列
tree | 查看radix树和rb树
union | 查看union结构体
vm | 查看虚拟内存
vtop | 虚拟地址物理地址转换
waitq | 查看wait queue上的进程
whatis | 符号表查询
wr | 改写内存
q | 退出

以下对常用命令进行详细介绍：

3.3.1 查看堆栈 bt

bt:默认会dump问题线程的场景

crash> bt PID: 641 TASK: ffff8001df128dc0 CPU: 5 COMMAND: "sh" #0 [8caa7f99223ac900] machine_kexec at ffff0000082185ec PC: 0000ffff8552f2ec LR: 0000ffff854dd6e0 SP: 0000ffffff551f40 X29: 0000ffffff551f40 X28: 0000000000000000 X27: 0000000000000001 X26: 00000000004cd320 X25: 000000000050b000 X24: 0000000000000002 X23: 00000000020b54e0 X22: 0000000000000002 X21: 0000ffff855d2560 X20: 00000000020b54e0 X19: 0000000000000001 X18: 0000000000000fff X17: 0000ffff854da1f0 X16: 00000000005030f0 X15: 0000000000000020 X14: 0000000000000000 X13: 0000000000000018 X12: 0000000000000000 X11: 0000000000000000 X10: 0000ffffff551f20 X9: 0000000000000001 X8: 0000000000000040 X7: 0000000000000063 X6: 0000000000000001 X5: 0000000000015551 X4: 0000000000000000 X3: 0000ffff855d6a78 X2: 0000000000000002 X1: 00000000020b54e0 X0: 0000000000000001 ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE:

-t/-T:把整个stack的信息dump出来

讯享网crash> bt -t PID: 641 TASK: ffff8001df128dc0 CPU: 5 COMMAND: "sh" START: machine_kexec at ffff0000082185f0 [ffff8001e450b6a0] machine_kexec at ffff0000082185f0 [ffff8001e450b6e8] __crash_kexec at ffff0000082ba048 [ffff8001e450b738] crash_kexec at ffff0000082ba15c [ffff8001e450b868] __handle_sysrq at ffff00000872a080 [ffff8001e450b878] sysrq_handle_crash at ffff0000087295f4 [ffff8001e450b8b0] sysrq_handle_crash at ffff0000087295f4 [ffff8001e450b8c8] die at ffff00000820ce00 [ffff8001e450b8f8] die_kernel_fault at ffff00000821ae10 [ffff8001e450b938] __do_kernel_fault at ffff00000821aea8 [ffff8001e450b968] do_page_fault at ffff00000821b0d0 [ffff8001e450b998] do_translation_fault at ffff00000821b3d8 [ffff8001e450ba18] cont_add at ffff00000827a9e8 [ffff8001e450ba28] __up_console_sem at ffff00000827abf8 [ffff8001e450ba48] console_unlock at ffff00000827b3e0 [ffff8001e450ba68] vprintk_emit at ffff00000827cb6c [ffff8001e450ba98] do_mem_abort at ffff00000 [ffff8001e450baa8] el1_ia at ffff0000082030d0 [ffff8001e450bac8] sysrq_handle_crash at ffff0000087295f4 [ffff8001e450bad8] vprintk_emit at ffff00000827ca64 [ffff8001e450bae8] vprintk_default at ffff00000827cd60 [ffff8001e450bb38] _cond_resched at ffff000008d32bb0 [ffff8001e450bc60] __handle_sysrq at ffff00000872a080 [ffff8001e450bc70] sysrq_handle_crash at ffff0000087295f4 [ffff8001e450bca8] sysrq_handle_crash at ffff0000087295f4 [ffff8001e450bcb8] write_sysrq_trigger at ffff00000872a65c [ffff8001e450bcf8] proc_reg_write at ffff00000840ffa0 [ffff8001e450bd18] __vfs_write at ffff000008390ea8 [ffff8001e450bd38] vfs_write at ffff0000083911a4 [ffff8001e450bd58] f_dupfd at ffff0000083b48d8 [ffff8001e450bd78] _cond_resched at ffff000008d32bb0 [ffff8001e450bd98] __sb_start_write at ffff0000083934c0 [ffff8001e450bda8] vfs_write at ffff00000 [ffff8001e450bdd8] ksys_write at ffff00000c [ffff8001e450be18] __arm64_sys_write at ffff00000 [ffff8001e450be48] __arm64_sys_close at ffff00000838e59c [ffff8001e450be68] el0_svc_common at ffff0000082139f4 [ffff8001e450be78] el0_svc_handler at ffff000008213b04 [ffff8001e450beb8] el0_svc at ffff000008203f88

-slf:加参数显示函数偏移，函数所在的文件和每一帧的具体内容，从而对照源码和汇编代码，查看函数入参和局部变量

crash> bt -slf PID: 641 TASK: ffff8001df128dc0 CPU: 5 COMMAND: "sh" #0 [8caa7f99223ac900] machine_kexec+60 at ffff0000082185ec /home/ycy/kernel_tag/linux_kernel-Linux-v1.4/build/../arch/arm64/kernel/machine_kexec.c: 158 PC: 0000ffff8552f2ec LR: 0000ffff854dd6e0 SP: 0000ffffff551f40 X29: 0000ffffff551f40 X28: 0000000000000000 X27: 0000000000000001 X26: 00000000004cd320 X25: 000000000050b000 X24: 0000000000000002 X23: 00000000020b54e0 X22: 0000000000000002 X21: 0000ffff855d2560 X20: 00000000020b54e0 X19: 0000000000000001 X18: 0000000000000fff X17: 0000ffff854da1f0 X16: 00000000005030f0 X15: 0000000000000020 X14: 0000000000000000 X13: 0000000000000018 X12: 0000000000000000 X11: 0000000000000000 X10: 0000ffffff551f20 X9: 0000000000000001 X8: 0000000000000040 X7: 0000000000000063 X6: 0000000000000001 X5: 0000000000015551 X4: 0000000000000000 X3: 0000ffff855d6a78 X2: 0000000000000002 X1: 00000000020b54e0 X0: 0000000000000001 ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE:

-c/-a:查看指定cpu的堆栈或所有当前CPU

讯享网crash> bt -c 2 PID: 0 TASK: ffff8001e9f044c0 CPU: 2 COMMAND: "swapper/2" #0 [ffff8001eab77cc0] crash_save_cpu at ffff0000082ba430 #1 [ffff8001eab77e80] handle_IPI at ffff000008212f08 #2 [ffff8001eab77ec0] gic_handle_irq at ffff00000c --- <IRQ stack> --- #3 [ffff8001e9f1bf50] el1_irq at ffff00000c PC: ffff000008206f18 [arch_cpu_idle+16] LR: ffff000008206f14 [arch_cpu_idle+12] SP: ffff8001e9f1bf60 PSTATE: 60c00009 X29: ffff8001e9f1bf60 X28: 0000000000000000 X27: 0000000000000000 X26: 0000000000000000 X25: 0000000000000000 X24: 0000000000000000 X23: ffff8001e9f044c0 X22: ffff8001e9f044c0 X21: ffff0000092da748 X20: 0000000000000002 X19: ffff0000092da6c8 X18: 0000000000000000 X17: 0000000000000000 X16: 0000000000000000 X15: 0000000000000400 X14: 0000000000000400 X13: 0000000000000001 X12: 0000000000000000 X11: 000000000000bc06 X10: 00000000000009e0 X9: ffff8001e9f1beb0 X8: ffff8001e9f04f00 X7: 0000000000000000 X6: 0000000000000002 X5: 000000000000003f X4: ffff8001e9f1bed0 X3: ffff0000092f04c0 X2: 0000000000000000 X1: 00000000002e1ab8 X0: 0000000000000008 #4 [ffff8001e9f1bf60] arch_cpu_idle at ffff000008206f14 #5 [ffff8001e9f1bf70] do_idle at ffff00000 #6 [ffff8001e9f1bfb0] cpu_startup_entry at ffff00000 #7 [ffff8001e9f1bfd0] secondary_start_kernel at ffff0000082129cc

ps:也可以用set命令来改变线程环境，从而查看别的cpu上的堆栈情况

3.3.2 查看堆栈 log

log命令可以用来查看系统的日志，“log-a”可以读取还没有从内核日志缓存到用户空间日志缓存的日志。也可以重定向到文件（log > logfile）。

crash> log [ 7.] Mali: [ 7.] Mali device driver loaded [ 7.] cacheinfo: Unable to detect cache hierarchy for CPU 0 [ 7.] brd: module loaded [ 7.] loop: module loaded [ 7.] zram: Added device: zram0 [ 7.] null: module loaded [ 7.] dummy-irq: no IRQ given. Use irq=N [ 7.] i2c-core: driver [at24] registered [ 7.] i2c-core: driver [eeprom] registered .......

3.3.3 反汇编 dis -r/-f

显示从给定地址到例程结束的所有指令。（-f 正向，-r 反向）

讯享网crash> dis -f ffff00000c d0xffff00000c <el1_irq+172>: blr x1 0xffff00000 <el1_irq+176>: mov sp, x19 0xffff00000 <el1_irq+180>: msr daifset, #0xf 0xffff00000 <el1_irq+184>: ldr x20, [sp,#288] 0xffff00000c <el1_irq+188>: str x20, [x28,#8] 0xffff00000 <el1_irq+192>: ldp x21, x22, [sp,#256] 0xffff00000 <el1_irq+196>: msr elr_el1, x21 0xffff00000 <el1_irq+200>: msr spsr_el1, x22 0xffff00000c <el1_irq+204>: ldp x0, x1, [sp] 0xffff00000 <el1_irq+208>: ldp x2, x3, [sp,#16] 0xffff00000 <el1_irq+212>: ldp x4, x5, [sp,#32] 0xffff00000 <el1_irq+216>: ldp x6, x7, [sp,#48] 0xffff00000c <el1_irq+220>: ldp x8, x9, [sp,#64] 0xffff00000 <el1_irq+224>: ldp x10, x11, [sp,#80] 0xffff00000 <el1_irq+228>: ldp x12, x13, [sp,#96] 0xffff00000 <el1_irq+232>: ldp x14, x15, [sp,#112] 0xffff00000c <el1_irq+236>: ldp x16, x17, [sp,#128] 0xffff00000 <el1_irq+240>: ldp x18, x19, [sp,#144] 0xffff00000 <el1_irq+244>: ldp x20, x21, [sp,#160] 0xffff00000 <el1_irq+248>: ldp x22, x23, [sp,#176] 0xffff00000c <el1_irq+252>: ldp x24, x25, [sp,#192] 0xffff00000 <el1_irq+256>: ldp x26, x27, [sp,#208] 0xffff00000 <el1_irq+260>: ldp x28, x29, [sp,#224] 0xffff00000 <el1_irq+264>: ldr x30, [sp,#240] 0xffff00000c <el1_irq+268>: add sp, sp, #0x140 0xffff00000 <el1_irq+272>: eret

3.3.4 查看和搜索内存 rd

rd:查看内存内容

crash> rd ffff000008d40070 32 ffff000008d40070: e694c 2e34206e6f Linux version 4. ffff000008d40080: 2e312b36362e3931 02e30 19.66+1.0.0 (ycy ffff000008d40090: 504f544b 4492d @DESKTOP-IDHR9T5 ffff000008d400a0: 82029 39206e6f ) (gcc version 9 ffff000008d400b0: e332e 332ee75 .3.0 (Ubuntu 9.3 ffff000008d400c0: d302e 2e30327ee .0-17ubuntu1~20. ffff000008d400d0: 93430 04d53 04)) #2 SMP Tue ffff000008d400e0: e754a a37323a39 Jun 15 09:27:09 ffff000008d400f0: 45343 000000000000000a CST 2021........ ffff000008d40100: 0000000000000000 0000000000000000 ................ ffff000008d40110: 0000000000000000 0000000000000000 ................ ffff000008d40120: 0000000000000000 0000000000000000 ................ ffff000008d40130: 0000000000000000 0000000000000000 ................ ffff000008d40140: 0000000000000000 0000000000000000 ................ ffff000008d40150: 0000000000000000 0000000000000000 ................ ffff000008d40160: 0000000000000000 0000000000000000 ................

-a linux_banner:查看版本信息

讯享网crash> rd -a linux_banner ffff000008d40070: Linux version 4.19.66+1.0.0 (ycy@DESKTOP-IDHR9T5) (gcc versi ffff000008d400ac: on 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #2 SMP Tue Jun 15 0 ffff000008d400e8: 9:27:09 CST 2021

-s:打印符号表

crash> rd ffff8001e450b6a0 32 -s ffff8001e450b6a0: machine_kexec+64 0000000000000004 ffff8001e450b6b0: 0000000000000004 000000000000003f ffff8001e450b6c0: ffff8001a0000040 0000000000000001 ffff8001e450b6d0: ffff8001a00000f8 d61f0220aa1f03e3 ffff8001e450b6e0: ffff8001e450b730 __crash_kexec+120 ffff8001e450b6f0: __stack_chk_guard crashk_res ffff8001e450b700: ffff8001e450bb70 posix_timers_hashtable+2040 ffff8001e450b710: ffff8001e450b778 0000000000000000 ffff8001e450b720: 00000000 posix_timers_hashtable+2040 ffff8001e450b730: ffff8001e450b8c0 crash_kexec+108 ffff8001e450b740: ffffffffffffffff __compound_literal.26+8 ffff8001e450b750: ffff8001e450bb70 0000000000000000 ffff8001e450b760: 0000ffffffffffff kallsyms_token_index+6456 ffff8001e450b770: ffff8001e450b860 0000000000000001 ffff8001e450b780: 0000000000000000 0000000000000000 ffff8001e450b790: 0000000000000006 0000000000000001

-e:查看指定内存区域内容

讯享网crash> rd ffff8001e450b6a0 -e ffff8001e450b6f0 ffff8001e450b6a0: ffff0000082185f0 0000000000000004 ..!............. ffff8001e450b6b0: 0000000000000004 000000000000003f ........?....... ffff8001e450b6c0: ffff8001a0000040 0000000000000001 @............... ffff8001e450b6d0: ffff8001a00000f8 d61f0220aa1f03e3 ............ ... ffff8001e450b6e0: ffff8001e450b730 ffff0000082ba048 0.P.....H.+.....

search:搜索指定内存

crash> search -s ffff8001e450b6a0 -e ffff8001e450b6e0 d61f0220aa1f03e3 ffff8001e450b6d8: d61f0220aa1f03e3

3.3.5 查看线程状态 ps

ps:查看所有线程状态

讯享网crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffff0000092e5240 RU 0.0 0 0 [swapper/0] > 0 0 1 ffff8001e9f06040 RU 0.0 0 0 [swapper/1] > 0 0 2 ffff8001e9f044c0 RU 0.0 0 0 [swapper/2] > 0 0 3 ffff8001e9f02940 RU 0.0 0 0 [swapper/3] > 0 0 4 ffff8001e9f00000 RU 0.0 0 0 [swapper/4] 0 0 5 ffff8001e9f03700 RU 0.0 0 0 [swapper/5] > 0 0 6 ffff8001e9f01b80 RU 0.0 0 0 [swapper/6] > 0 0 7 ffff8001e9f3b700 RU 0.0 0 0 [swapper/7] 1 0 5 ffff8001e9ec6040 IN 0.1  6688 systemd 2 0 3 ffff8001e9ec44c0 IN 0.0 0 0 [kthreadd] 3 2 0 ffff8001e9ec2940 ID 0.0 0 0 [rcu_gp] 4 2 0 ffff8001e9ec0000 ID 0.0 0 0 [rcu_par_gp] 7 2 3 ffff8001e9ec5280 ID 0.0 0 0 [kworker/u16:0] ... ...

-p:查看父线程树

crash> ps -p 167 PID: 0 TASK: ffff0000092e5240 CPU: 0 COMMAND: "swapper/0" PID: 2 TASK: ffff8001e9ec44c0 CPU: 3 COMMAND: "kthreadd" PID: 167 TASK: ffff8001e6a5ee00 CPU: 5 COMMAND: "bst_lw0"

-c:查看子线程

讯享网crash> ps -c 2 PID: 2 TASK: ffff8001e9ec44c0 CPU: 3 COMMAND: "kthreadd" PID: 3 TASK: ffff8001e9ec2940 CPU: 0 COMMAND: "rcu_gp" PID: 4 TASK: ffff8001e9ec0000 CPU: 0 COMMAND: "rcu_par_gp" PID: 7 TASK: ffff8001e9ec5280 CPU: 3 COMMAND: "kworker/u16:0" PID: 8 TASK: ffff8001e9ec0dc0 CPU: 0 COMMAND: "mm_percpu_wq" PID: 9 TASK: ffff8001e9ec6e00 CPU: 0 COMMAND: "ksoftirqd/0" PID: 10 TASK: ffff8001e9f05280 CPU: 1 COMMAND: "rcu_sched" PID: 11 TASK: ffff8001e9f00dc0 CPU: 0 COMMAND: "rcu_bh" PID: 12 TASK: ffff8001e9f06e00 CPU: 0 COMMAND: "migration/0" ... ...

-t:查看线程运行时间

crash> ps -t 167 PID: 167 TASK: ffff8001e6a5ee00 CPU: 5 COMMAND: "bst_lw0" RUN TIME: 00:15:01 START TIME:  UTIME: 0 STIME: 0

-A:查看活动线程

讯享网crash> ps -A PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffff0000092e5240 RU 0.0 0 0 [swapper/0] > 0 0 1 ffff8001e9f06040 RU 0.0 0 0 [swapper/1] > 0 0 2 ffff8001e9f044c0 RU 0.0 0 0 [swapper/2] > 0 0 3 ffff8001e9f02940 RU 0.0 0 0 [swapper/3] > 0 0 4 ffff8001e9f00000 RU 0.0 0 0 [swapper/4] > 0 0 6 ffff8001e9f01b80 RU 0.0 0 0 [swapper/6] > 0 0 7 ffff8001e9f3b700 RU 0.0 0 0 [swapper/7] > 641 478 5 ffff8001df128dc0 RU 0.0 7352 3764 sh

-k:查看内核线程

crash> ps -k PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffff0000092e5240 RU 0.0 0 0 [swapper/0] > 0 0 1 ffff8001e9f06040 RU 0.0 0 0 [swapper/1] > 0 0 2 ffff8001e9f044c0 RU 0.0 0 0 [swapper/2] > 0 0 3 ffff8001e9f02940 RU 0.0 0 0 [swapper/3] > 0 0 4 ffff8001e9f00000 RU 0.0 0 0 [swapper/4] 0 0 5 ffff8001e9f03700 RU 0.0 0 0 [swapper/5] > 0 0 6 ffff8001e9f01b80 RU 0.0 0 0 [swapper/6] > 0 0 7 ffff8001e9f3b700 RU 0.0 0 0 [swapper/7] 2 0 3 ffff8001e9ec44c0 IN 0.0 0 0 [kthreadd] 3 2 0 ffff8001e9ec2940 ID 0.0 0 0 [rcu_gp] 4 2 0 ffff8001e9ec0000 ID 0.0 0 0 [rcu_par_gp] ... ...

-u:查看用户态线程

讯享网crash> ps -u PID PPID CPU TASK ST %MEM VSZ RSS COMM 1 0 5 ffff8001e9ec6040 IN 0.1  6688 systemd 259 1 1 ffff8001e86be040 IN 0.1 25836 10456 systemd-journal 277 1 1 ffff8001e4468dc0 IN 0.0 13860 3152 systemd-udevd 372 1 6 ffff8001e97a9b80 IN 0.1 7868 4276 systemd-network 378 1 5 ffff8001e584c4c0 IN 0.1 7516 4132 systemd-resolve 379 1 1 ffff8001e446e040 IN 0.0 81452 3936 systemd-timesyn 381 1 4 ffff8001e IN 0.1 7492 4076 systemd-logind 382 1 4 ffff8001e IN 0.0 3128 1620 klogd 407 1 7 ffff8001e IN 0.0 3128 1596 syslogd 408 1 7 ffff8001e81044c0 IN 0.1  9028 SafetyService 409 1 1 ffff8001e IN 0.0 4648 3184 dbus-daemon ... ...

-l:查看最后运行时间戳

crash> ps -l [4] [IN] PID: 535 TASK: ffff8001df37c4c0 CPU: 2 COMMAND: "SecurityService" [9] [IN] PID: 507 TASK: ffff8001e584a940 CPU: 3 COMMAND: "CameraService" [6] [IN] PID: 510 TASK: ffff8001e5848dc0 CPU: 7 COMMAND: "SecurityService" [0] [IN] PID: 514 TASK: ffff8001df18e040 CPU: 6 COMMAND: "PersistenceServ" [7] [IN] PID: 561 TASK: ffff8001dd951b80 CPU: 0 COMMAND: "UpgradeService" [8] [IN] PID: 470 TASK: ffff8001e2aa2940 CPU: 2 COMMAND: "DiagnoseService" [4] [IN] PID: 490 TASK: ffff8001e0778dc0 CPU: 0 COMMAND: "SensorService" ... ...

-r:查看线程资源限制

讯享网crash> ps -r 167 PID: 167 TASK: ffff8001e6a5ee00 CPU: 5 COMMAND: "bst_lw0" RLIMIT CURRENT MAXIMUM CPU (unlimited) (unlimited) FSIZE (unlimited) (unlimited) DATA (unlimited) (unlimited) STACK  (unlimited) CORE 0 (unlimited) RSS (unlimited) (unlimited) NPROC 6630 6630 NOFILE 1024 4096 MEMLOCK 65536 65536 AS (unlimited) (unlimited) LOCKS (unlimited) (unlimited) SIGPENDING 6630 6630 MSGQUEUE   NICE 0 0 RTPRIO 0 0 RTTIME (unlimited) (unlimited)

3.3.6 Context切换 set

切换panic线程

crash> set -p PID: 641 COMMAND: "sh" TASK: ffff8001df128dc0 [THREAD_INFO: ffff8001df128dc0] CPU: 5 STATE: TASK_RUNNING (SYSRQ) crash> set ffff8001df128dc0 PID: 641 COMMAND: "sh" TASK: ffff8001df128dc0 [THREAD_INFO: ffff8001df128dc0] CPU: 5 STATE: TASK_RUNNING (SYSRQ)

3.3.7 加载module符号表 mod

mod:查看当前加载的module

讯享网crash> mod MODULE NAME BASE SIZE OBJECT FILE ffff000000d42040 panic_module ffff000000d40000 16384 (not loaded) [CONFIG_KALLSYMS]

-S:加载所有module符号表