Worker node not coming up with the new kernel after reboot

[ravicorning]

I'm on CentOs 7.8.2003, and using the latest OEK, 20.12.2

Following are the last logs after i run the ./deploy_sh to configure the nodes. Seems the new kernel is taking the host into a bad state...any ideas what might be going wrong ?

Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-1127.19.1.rt56.1116.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1127.19.1.rt56.1116.el7.x86_64.img
Found linux image: /boot/vmlinuz-3.10.0-1127.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-1127.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-cd2908905321447789efff0871aa4a65
Found initrd image: /boot/initramfs-0-rescue-cd2908905321447789efff0871aa4a65.img
done

TASK [machine_setup/grub : get current CMDLINE] ********************************************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/grub/tasks/main.yml:40
ok: [node01] => {
"changed": false,
"cmd": [
"cat",
"/proc/cmdline"
],
"delta": "0:00:00.004990",
"end": "2021-03-04 13:55:11.746850",
"rc": 0,
"start": "2021-03-04 13:55:11.741860"
}

STDOUT:

BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8

TASK [machine_setup/grub : request reboot (will be performed by conditional_reboot role)] ************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/grub/tasks/main.yml:46
ok: [node01] => {
"ansible_facts": {
"perform_reboot": [
"kernel",
"grub"
]
},
"changed": false
}

TASK [machine_setup/configure_tuned : install tuned packages] ******************************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:5
ok: [node01] => {
"attempts": 1,
"changed": false,
"rc": 0,
"results": [
"tuned-2.11.0-8.el7.noarch providing tuned-2.11.0-8.el7 is already installed",
"tuned-profiles-realtime-2.11.0-8.el7.noarch providing /root/.ansible/tmp/ansible-moduletmp-1614894912.58-33w9XE/tuned-profiles-realtime-2.11.0-8.el7.noarchV2C6y2.rpm is already installed"
]
}

TASK [machine_setup/configure_tuned : install tuned packages - offline mode] ***************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:13
skipping: [node01] => {
"changed": false,
"skip_reason": "Conditional result was False"
}

TASK [machine_setup/configure_tuned : find isolcpus value if applied as grub parameter] ****************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:23
skipping: [node01] => {
"changed": false,
"skip_reason": "Conditional result was False"
}

TASK [machine_setup/configure_tuned : change isolcpus tuned setting] ***********************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:26
skipping: [node01] => {
"changed": false,
"skip_reason": "Conditional result was False"
}

TASK [machine_setup/configure_tuned : copy tuned variables into the file] ******************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:31
changed: [node01] => {
"changed": true,
"checksum": "3de7712038e7523848d66a80cc78bbed32d726a3",
"dest": "/etc/tuned/realtime-variables.conf",
"gid": 0,
"group": "root",
"md5sum": "fd84ed0a56591d879fddd2265ca1954b",
"mode": "0644",
"owner": "root",
"secontext": "system_u:object_r:tuned_etc_t:s0",
"size": 41,
"src": "/root/.ansible/tmp/ansible-tmp-1614894881.43-21337-212133244989112/source",
"state": "file",
"uid": 0
}

TASK [machine_setup/configure_tuned : apply tuned profile] *********************************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:40
changed: [node01] => {
"changed": true,
"cmd": [
"tuned-adm",
"profile",
"realtime"
],
"delta": "0:00:02.487842",
"end": "2021-03-04 13:55:17.010233",
"rc": 0,
"start": "2021-03-04 13:55:14.522391"
}

TASK [machine_setup/configure_tuned : get tuned CMDLINE] ***********************************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:45
ok: [node01] => {
"changed": false,
"cmd": [
"awk",
"-F"",
"/TUNED_BOOT_CMDLINE/ {print $2}",
"/etc/tuned/bootcmdline"
],
"delta": "0:00:00.002240",
"end": "2021-03-04 13:55:17.285015",
"rc": 0,
"start": "2021-03-04 13:55:17.282775"
}

STDOUT:

skew_tick=1 isolcpus=2-3 intel_pstate=disable nosoftlockup

TASK [machine_setup/configure_tuned : get current CMDLINE] *********************************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:50
ok: [node01] => {
"changed": false,
"cmd": [
"cat",
"/proc/cmdline"
],
"delta": "0:00:00.001627",
"end": "2021-03-04 13:55:17.544039",
"rc": 0,
"start": "2021-03-04 13:55:17.542412"
}

STDOUT:

BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8

TASK [machine_setup/configure_tuned : request reboot (will be performed by conditional_reboot role)] *************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:56
ok: [node01] => {
"ansible_facts": {
"perform_reboot": [
"kernel",
"grub",
"tuned"
]
},
"changed": false
}

TASK [machine_setup/conditional_reboot : display reboot requesters] ************************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/conditional_reboot/tasks/main.yml:8
ok: [node01] => {}

MSG:

Performing reboot requested by: [u'kernel', u'grub', u'tuned']

TASK [machine_setup/conditional_reboot : reboot the machine] *******************************************************************************************************************
task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/conditional_reboot/tasks/main.yml:11

[aniket-intel]

Hi Ravi,

Can you please try removing mce=off from the file oek/flavors/flexran/edgenode_group.yml in line 35, and try the deploy script again after cleaning up the system? Let me know if that works.