Re: [Xen-devel] Need some advice on how to debug a boot hang in console_lock() in radeon driver loading

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Wed, Mar 12, 2014 at 04:34:48PM +0000, Zytaruk, Kelly wrote:
> Hi,
> 
> I recently upgraded from (Debian 7) linux 3.2.0 to 3.13.4 and I am having 
> difficulties booting.
> I have multiple versions on my machine; 3.2.0, 3.4.9, 3.13.4 and 3.13.5.  
> 3.2.0 and 3.4.9 have no problem but both 3.13.4 and 3.13.5 hang during boot. 

Could you give some details on what you have on the Xen and Linux command line?
And as well your 'lspci' output?

> 
> My system is hanging during the loading of the radeon driver.
> Specifically, the call to do_register_framebuffer() is calling console_lock().
> Within console_lock() it is calling down(&sem).  The hang is somewhere within 
> the down() call as it never returns from down().
> 
> I have put printk() calls in console_lock() as shown below (and elsewhere in 
> the radeon driver);
> 
> void console_lock(void)
> {
>         printk (KERN_ERR "console_lock() - console_locked = %d, sem count = 
> %d\n", console_locked, console_sem.count);
>         might_sleep();
>         printk (KERN_ERR "console_lock() - call down()\n");
>         down(&console_sem);
>         if (console_suspended)
>         {
>                 printk (KERN_ERR "console_lock() - console is suspended\n");

You might want to use 'xen_raw_print' at this stage. The reason is that the
printk (I think) ends up calling the console lock so you can get a deadlock.

>                 return;
>         }
>         console_locked = 1;
>         console_may_schedule = 1;
>         printk (KERN_ERR "console_lock() - acquire the mutex\n");
>         mutex_acquire(&console_lock_dep_map, 0, 0, _RET_IP_);
> }
> 
> The output to a serial port during boot is;
> 
> . . . 
> [    5.313569] console_lock() - console_locked = 0, sem count = 1
> [    5.319930] console_lock() - call down()
> [    5.325371] console_lock() - acquire the mutex
> [    5.330140] console_lock() - console_locked = 0, sem count = 1
> [    5.337362] console_lock() - call down()
> [    5.341393] console_lock() - acquire the mutex
> [ ok ] Starting the hotplug events dispatcher: udevd.
> [ ok ] Synthesizing the initial hotplug events...done.
> [....] Waiting for /dev to be fully populated...
> [    6.636204] __pci_register_driver (i2c_piix4, piix4_smbus)
> [    6.642085] __pci_register_driver (k10temp, k10temp)
> [    6.656914] __pci_register_driver (ehci_pci, ehci-pci)
> [    6.715643] __pci_register_driver (ohci_pci, ohci-pci)
> [    6.922953] __pci_register_driver (amd64_edac_mod, amd64_edac)
> [    7.815481] __pci_register_driver (snd_hda_intel, snd_hda_intel)
> [    8.066608] Enter radeon_init(void)
> [    8.070231] Exitting radeon_init(void),call drm_pci_init() on the way out
> [    8.077173] Enter drm_pci_init()
> [    8.080503] This is a Render device
> [    8.084096] __pci_register_driver (drm, radeon)
> [    8.089331] Enter radeon_driver_load_kms()
> [    8.093585] Enter radeon_device_init()
> [    8.136403] radeon 0000:00:01.0: radeon_uvd: Can't load firmware 
> "radeon/BONAIRE_uvd.bin"
> [    8.156979] Try to load uCode for KAVERI
>  [    8.185785] request_firmware (radeon/KAVERI_pfp.bin) returns 0
> [    9.208145] Exit radeon_device_init() with success
> [    9.213105] radeon_driver_load_kms - call radeon_modeset_init()
> [    9.219188] radeon_modeset_init() - call drm_mode_config_init()
> [    9.225248] radeon_modeset_init() - max_width=16384, max_height=16384
> [    9.231836] radeon_modeset_init() - call radeon_modeset_create_props()
> [    9.238519] radeon_modeset_init() - call radeon_i2c_init()
> [    9.244271] radeon_modeset_init() - call radeon_setup_enc_conn()
> [    9.251532] radeon_modeset_init() - call radeon_atom_encoder_init()
> [    9.279816] radeon_modeset_init() - call radeon_atom_disp_eng_pll_init()
> [    9.286779] radeon_modeset_init() - call radeon_hpd_init()
> [    9.292413] radeon_modeset_init() - call radeon_afmt_init()
> [    9.298132] radeon_modeset_init() - call radeon_pm_init()
> [    9.303728] radeon_modeset_init() - call radeon_fbdev_init()
> [    9.320230] Exit: drm_helper_disable_unused_functions()
> [    9.325607] drm_fb_helper_initial_config() call 
> drm_fb_helper_parse_command_line()
> [    9.333339] drm_fb_helper_initial_config() call 
> drm_fb_helper_probe_connector_modes()
> [    9.417229] drm_fb_helper_initial_config() call drm_setup_crtcs()
> [    9.423527] drm_fb_helper_initial_config() call 
> drm_fb_helper_single_fb_probe() on the way out
> [    9.432316] Enter: drm_fb_helper_single_fb_probe()
> [    9.437225] drm_fb_helper_single_fb_probe() - 3 connectors
> [    9.442832] drm_fb_helper_single_fb_probe() call probing function
> [    9.450542] drm_fb_helper_single_fb_probe() probing function returns 0
> [    9.457273] drm_fb_helper_single_fb_probe() call register_framebuffer()
> [    9.464026] Enter: register_framebuffer()
> [    9.468047] Enter: do_register_framebuffer() - call fb_check_foreignness()
> [    9.475102] do_register_framebuffer() call 
> do_remove_conflicting_framebuffers()
> [    9.482566] do_register_framebuffer() call device_create()
> [    9.488250] do_register_framebuffer() call fb_init_device()
> [    9.493959] do_register_framebuffer() call fb_var_to_videomode(true)
> [    9.504827] do_register_framebuffer() call fb_add_videomode()
> [    9.510738] do_register_framebuffer() call console_lock()
> [    9.516257] console_lock() - console_locked = 0, sem count = 1
> [    9.522220] console_lock() - call down()
> 
> The last line above is the last thing that I see from the serial port.
> It took me quite a while to get this far with my investigation and now I am 
> stuck on how to debug into the semaphore, printk calls don't work anymore.  I 
> tried putting a printk into down() and it hung very early in the boot.
> 
> Any suggestions in how to proceed or what tools I need to continue would be 
> appreciated, keeping in mind that my Linux background is still on the 
> learning curve ( I started learning Linux about 6 months ago).  I have a 
> serial port available on the system and if needed I could probably dig up a 
> PCIe 1394 firewire card as well.
> 
> I have two machines, 3.13.4 fails on one but works on the other.  The failing 
> machine is a Kaveri system, the passing machine is a Tahiti on Intel, so I 
> suspect something Kaveri specific but I don't see how the semaphore could be 
> related to Kaveri.
> 
> Thanks,
> Kelly
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxx
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel