I am developing a kernel model that has two kernel threads - A and B. Lets' assume kernel thread A is producer and kernel thread B is a consumer. These threads are pinned to cores 37 and 38 in a 40 core x86 machine. The kernel version is 5.6.x and the kernel boot params are shows below:
$cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.6.3 root=/dev/mapper/cl-root isolcpus=managed_irq,nohz,domain,36,37,38,39 rcu_nocbs=36,37,38,39 nohz_full=36,37,38,39 rcu_nocb_poll nmi_watchdog=0Note that the kernel threads are running on cpuid 37 and 38 and are marked as isolated CPUs. In the above setup, producer thread every 1us or so, inserts an entry to circular queue while consumer thread pops the entry from the queue infinitely with udelay(5) seconds. My expectation is since I have disabled rcu_nocbs and also have added nohz_full for the cores, I should not observe a hard/soft lockup trigger by the watchdog timer on these cores. However, for some reason, watchdog is still enabled on these threads causing kernel panic reporting following error in dmesg:
[ +0.998369] BUG: workqueue lockup - pool cpus=38 node=0 flags=0x0 nice=0 stuck for 39s![ +0.000347] Showing busy workqueues and worker pools:[ +0.000331] workqueue events: flags=0x0[ +0.000337] pwq 76: cpus=38 node=0 flags=0x0 nice=0 active=1/256 refcnt=2[ +0.000340] pending: drm_fb_helper_dirty_work [drm_kms_helper][ +0.000356] pwq 60: cpus=30 node=0 flags=0x0 nice=0 active=1/256 refcnt=2[ +0.000341] in-flight: 3231:do_free_init[ +0.000766] workqueue mm_percpu_wq: flags=0x8[ +0.000353] pwq 76: cpus=38 node=0 flags=0x0 nice=0 active=1/256 refcnt=2[ +0.000363] pending: vmstat_update[ +0.017176] rcu: INFO: rcu_sched self-detected stall on CPU[ +0.000393] rcu: 38-....: (59506 ticks this GP) idle=2e2/1/0x4000000000000002 softirq=55/55 fqs=14483 [ +0.000411] (t=60001 jiffies g=22617 q=103939)[ +0.000412] NMI backtrace for cpu 38[ +0.000414] CPU: 38 PID: 6068 Comm: two Tainted: P O 5.6.3 #1[ +0.000433] Call Trace:[ +0.000431] <IRQ>[ +0.000433] dump_stack+0x66/0x90[ +0.000431] nmi_cpu_backtrace.cold.8+0x13/0x4f[ +0.000438] ? lapic_can_unplug_cpu.cold.28+0x37/0x37[ +0.000440] nmi_trigger_cpumask_backtrace+0xde/0xe0[ +0.000443] rcu_dump_cpu_stacks+0x9c/0xca[ +0.000442] rcu_sched_clock_irq.cold.88+0x1e7/0x3d5[ +0.000447] ? tick_sched_do_timer+0x70/0x70[ +0.000448] ? tick_sched_do_timer+0x70/0x70[ +0.000433] update_process_times+0x24/0x50[ +0.000424] tick_sched_handle+0x22/0x60[ +0.000421] tick_sched_timer+0x37/0x70[ +0.000413] __hrtimer_run_queues+0x100/0x280[ +0.000440] hrtimer_interrupt+0x100/0x220[ +0.000438] smp_apic_timer_interrupt+0x6a/0x130[ +0.000398] apic_timer_interrupt+0xf/0x20[ +0.000387] </IRQ>[ +0.000386] RIP: 0010:kthread_should_stop+0x12/0x30[ +0.000383] Code: 41 bc ea ff ff ff e9 20 fc ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 65 48 8b 04 25 c0 8b 01 00 f6 40 26 20 <74> 11 48 8b 80 80 09 00 00 48 8b 00 48 d1 e8 83 e0 01 c3 0f 0b eb[ +0.000807] RSP: 0000:ffffa45e02dc3ed0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13[ +0.000402] RAX: ffff98826d683c80 RBX: ffffffffc040a380 RCX: 0000000000000000[ +0.000410] RDX: 0000000000000000 RSI: ffff9882bff996b8 RDI: ffff9882bff996b8[ +0.000414] RBP: 000000000000003d R08: 0000000000003690 R09: 0000000000aaaaaa[ +0.000419] R10: 0000000000000000 R11: 0000000000000001 R12: ffff9882ab008000[ +0.000419] R13: ffff9882af7c0000 R14: 0000000000000001 R15: 000000000000003d[ +0.000426] consumer+0x9c/0x150 [kthread_consumer][ +0.000425] ? 0xffffffffc0419000[ +0.000422] kthread+0x10c/0x130[ +0.000422] ? kthread_park+0x80/0x80[ +0.000422] ret_from_fork+0x1f/0x40Is there an alternative way to disable the watchdog on these specific cores? I looked at nowatchdog option and also recompiling the kernel by disabling watchdog using .config. However, both would disable watchdog across all CPUs which may not be a good idea.