[Xen-devel] megaraid_sas regression in linux-3.18

[Ian Jackson <ian.jackson@xxxxxxxxxxxxx>]

Our CI and automatic bisector have identified that

 24542192519d21719377d89f14654b3afd993a61
 scsi: megaraid_sas: Fix data integrity failure for JBOD (passthrough) devices

breaks booting certain machines under Xen.  See full report from the
bisector, below.

The boot messages including a lot of wrong-looking output from
megasas, can be found here:

  
http://logs.test-lab.xenproject.org/osstest/logs/102788/test-amd64-amd64-xl-qemut-debianhvm-amd64/serial-nobling1.log

(Look around timestamp `Dec  2 10:46:58.538047'.  NB this logfile
contains irrelevant contents from previous test: everything before
`Dec  2 10:46:41.330103' was with a different kernel and is not of
any interest.)


I'm afraid I know nothing about this driver.  I'm just responsible for
the CI system that found the regression.  Could someone please look at
this and figure out what is wrong ?

Thanks,
Ian.


osstest service owner writes ("[linux-3.18 bisection] complete 
test-amd64-amd64-xl-qemut-debianhvm-amd64"):
> branch xen-unstable
> xenbranch xen-unstable
> job test-amd64-amd64-xl-qemut-debianhvm-amd64
> testid xen-boot
> 
> Tree: linux 
> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
> Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> Tree: xen git://xenbits.xen.org/xen.git
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  linux 
> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
>   Bug introduced:  24542192519d21719377d89f14654b3afd993a61
>   Bug not present: c6f51aabaf400f357eebe8f8f17e8bb39fc033dc
>   Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/102788/
> 
> 
>   commit 24542192519d21719377d89f14654b3afd993a61
>   Author: Kashyap Desai <kashyap.desai@xxxxxxxxxxxx>
>   Date:   Fri Oct 21 06:33:32 2016 -0700
>   
>       scsi: megaraid_sas: Fix data integrity failure for JBOD (passthrough) 
> devices
>       
>       [ Upstream commit 1e793f6fc0db920400574211c48f9157a37e3945 ]
>       
>       Commit 02b01e010afe ("megaraid_sas: return sync cache call with
>       success") modified the driver to successfully complete SYNCHRONIZE_CACHE
>       commands without passing them to the controller. Disk drive caches are
>       only explicitly managed by controller firmware when operating in RAID
>       mode. So this commit effectively disabled writeback cache flushing for
>       any drives used in JBOD mode, leading to data integrity failures.
>       
>       [mkp: clarified patch description]
>       
>       Fixes: 02b01e010afeeb49328d35650d70721d2ca3fd59
>       CC: stable@xxxxxxxxxxxxxxx
>       Signed-off-by: Kashyap Desai <kashyap.desai@xxxxxxxxxxxx>
>       Signed-off-by: Sumit Saxena <sumit.saxena@xxxxxxxxxxxx>
>       Reviewed-by: Tomas Henzl <thenzl@xxxxxxxxxx>
>       Reviewed-by: Hannes Reinecke <hare@xxxxxxxx>
>       Reviewed-by: Ewan D. Milne <emilne@xxxxxxxxxx>
>       Signed-off-by: Martin K. Petersen <martin.petersen@xxxxxxxxxx>
>       Signed-off-by: Sasha Levin <alexander.levin@xxxxxxxxxxx>
> 
> 
> For bisection revision-tuple graph see:
>    
> http://logs.test-lab.xenproject.org/osstest/results/bisect/linux-3.18/test-amd64-amd64-xl-qemut-debianhvm-amd64.xen-boot.html
> Revision IDs in each graph node refer, respectively, to the Trees above.
> 
> ----------------------------------------
> Running cs-bisection-step 
> --graph-out=/home/logs/results/bisect/linux-3.18/test-amd64-amd64-xl-qemut-debianhvm-amd64.xen-boot
>  --summary-out=tmp/102788.bisection-summary --basis-template=101675 
> --blessings=real,real-bisect linux-3.18 
> test-amd64-amd64-xl-qemut-debianhvm-amd64 xen-boot
> Searching for failure / basis pass:
>  102754 fail [host=nobling1] / 101675 [host=nocera1] 101662 [host=elbling0] 
> 101648 [host=pinot0] 101637 [host=godello1] 101623 [host=nobling0] 101603 
> [host=baroque0] 101584 [host=huxelrebe1] 101570 [host=italia1] 101561 
> [host=nocera0] 101552 ok.
> Failure / basis pass flights: 102754 / 101552
> (tree with no url: minios)
> (tree with no url: ovmf)
> (tree with no url: seabios)
> Tree: linux 
> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> Tree: linuxfirmware git://xenbits.xen.org/osstest/linux-firmware.git
> Tree: qemu git://xenbits.xen.org/qemu-xen-traditional.git
> Tree: qemuu git://xenbits.xen.org/qemu-xen.git
> Tree: xen git://xenbits.xen.org/xen.git
> Latest ac3d826bef907afe35f80ecccbcdd57223df4b88 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 4220231eb22235e757d269722b9f6a594fbcb70f 
> 99a10da1b4fee8ef7a096e5fd3608f6c15932eb0
> Basis pass 3cab355c2ff3a781b6ebe9d1a25bd4ebc1207430 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 570117996772b762e9654e58e708943a4db68b4f 
> 05e379bd279768495cdc516f17a120e30dfbcca5
> Generating revisions with ./adhoc-revtuple-generator  
> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git#3cab355c2ff3a781b6ebe9d1a25bd4ebc1207430-ac3d826bef907afe35f80ecccbcdd57223df4b88
>  
> git://xenbits.xen.org/osstest/linux-firmware.git#c530a75c1e6a472b0eb9558310b518f0dfcd8860-c530a75c1e6a472b0eb9558310b518f0dfcd8860
>  
> git://xenbits.xen.org/qemu-xen-traditional.git#c4e0d84d3c92923fdbc7fa922638d54e5e834753-c4e0d84d3c92923fdbc7fa922638d54e5e834753
>  
> git://xenbits.xen.org/qemu-xen.git#570117996772b762e9654e58e708943a4db68b4f-4220231eb22235e757d269722b9f6a594fbcb70f
>  
> git://xenbits.xen.org/xen.git#05e379bd279768495cdc516f17a120e30dfbcca5-99a10da1b4fee8ef7a096e5fd3608f6c15932eb0
> Loaded 3003 nodes in revision graph
> Searching for test results:
>  101493 [host=elbling1]
>  101532 [host=chardonnay0]
>  101515 [host=fiano0]
>  101497 [host=chardonnay1]
>  101552 pass 3cab355c2ff3a781b6ebe9d1a25bd4ebc1207430 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 570117996772b762e9654e58e708943a4db68b4f 
> 05e379bd279768495cdc516f17a120e30dfbcca5
>  101541 [host=pinot1]
>  101561 [host=nocera0]
>  101570 [host=italia1]
>  101584 [host=huxelrebe1]
>  101603 [host=baroque0]
>  101648 [host=pinot0]
>  101637 [host=godello1]
>  101623 [host=nobling0]
>  101662 [host=elbling0]
>  101675 [host=nocera1]
>  102732 fail ac3d826bef907afe35f80ecccbcdd57223df4b88 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 4220231eb22235e757d269722b9f6a594fbcb70f 
> 99a10da1b4fee8ef7a096e5fd3608f6c15932eb0
>  102782 pass c6f51aabaf400f357eebe8f8f17e8bb39fc033dc 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102753 pass 3cab355c2ff3a781b6ebe9d1a25bd4ebc1207430 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 570117996772b762e9654e58e708943a4db68b4f 
> 05e379bd279768495cdc516f17a120e30dfbcca5
>  102784 fail 24542192519d21719377d89f14654b3afd993a61 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102761 fail ac3d826bef907afe35f80ecccbcdd57223df4b88 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 4220231eb22235e757d269722b9f6a594fbcb70f 
> 99a10da1b4fee8ef7a096e5fd3608f6c15932eb0
>  102787 pass c6f51aabaf400f357eebe8f8f17e8bb39fc033dc 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102763 pass a6846cfd266b48af1ee7c3c19d5cb60477ca4469 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 6cfcdf037edadba984ccf8476b5d1e2a0940b789 
> fcb618c025f9251d7e22138f6528595037252c21
>  102764 pass da512d38ffce476a7bebb4474ed572887cbb5fb1 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102788 fail 24542192519d21719377d89f14654b3afd993a61 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102765 fail b4caa5be158ef6ba86b63d9b95546e88e4ee5438 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102754 fail ac3d826bef907afe35f80ecccbcdd57223df4b88 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 4220231eb22235e757d269722b9f6a594fbcb70f 
> 99a10da1b4fee8ef7a096e5fd3608f6c15932eb0
>  102770 pass 0b2a0a58ad22f9d6dfc641bc5ec46057493f22a5 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102771 pass 1f601388d5c150f0cef255bdb1a9296bdab9e157 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102775 pass c6f51aabaf400f357eebe8f8f17e8bb39fc033dc 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
>  102779 fail 24542192519d21719377d89f14654b3afd993a61 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
> Searching for interesting versions
>  Result found: flight 101552 (pass), for basis pass
>  Result found: flight 102732 (fail), for basis failure
>  Repro found: flight 102753 (pass), for basis pass
>  Repro found: flight 102754 (fail), for basis failure
>  0 revisions at c6f51aabaf400f357eebe8f8f17e8bb39fc033dc 
> c530a75c1e6a472b0eb9558310b518f0dfcd8860 
> c4e0d84d3c92923fdbc7fa922638d54e5e834753 
> 53307d7a9c749237386eb2ccaa709c786cb5f8a5 
> 7b5266abfdf7f842c49ce4a52d250e523bc0172d
> No revisions left to test, checking graph state.
>  Result found: flight 102775 (pass), for last pass
>  Result found: flight 102779 (fail), for first failure
>  Repro found: flight 102782 (pass), for last pass
>  Repro found: flight 102784 (fail), for first failure
>  Repro found: flight 102787 (pass), for last pass
>  Repro found: flight 102788 (fail), for first failure
> 
> *** Found and reproduced problem changeset ***
> 
>   Bug is in tree:  linux 
> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
>   Bug introduced:  24542192519d21719377d89f14654b3afd993a61
>   Bug not present: c6f51aabaf400f357eebe8f8f17e8bb39fc033dc
>   Last fail repro: http://logs.test-lab.xenproject.org/osstest/logs/102788/
> 
> 
>   commit 24542192519d21719377d89f14654b3afd993a61
>   Author: Kashyap Desai <kashyap.desai@xxxxxxxxxxxx>
>   Date:   Fri Oct 21 06:33:32 2016 -0700
>   
>       scsi: megaraid_sas: Fix data integrity failure for JBOD (passthrough) 
> devices
>       
>       [ Upstream commit 1e793f6fc0db920400574211c48f9157a37e3945 ]
>       
>       Commit 02b01e010afe ("megaraid_sas: return sync cache call with
>       success") modified the driver to successfully complete SYNCHRONIZE_CACHE
>       commands without passing them to the controller. Disk drive caches are
>       only explicitly managed by controller firmware when operating in RAID
>       mode. So this commit effectively disabled writeback cache flushing for
>       any drives used in JBOD mode, leading to data integrity failures.
>       
>       [mkp: clarified patch description]
>       
>       Fixes: 02b01e010afeeb49328d35650d70721d2ca3fd59
>       CC: stable@xxxxxxxxxxxxxxx
>       Signed-off-by: Kashyap Desai <kashyap.desai@xxxxxxxxxxxx>
>       Signed-off-by: Sumit Saxena <sumit.saxena@xxxxxxxxxxxx>
>       Reviewed-by: Tomas Henzl <thenzl@xxxxxxxxxx>
>       Reviewed-by: Hannes Reinecke <hare@xxxxxxxx>
>       Reviewed-by: Ewan D. Milne <emilne@xxxxxxxxxx>
>       Signed-off-by: Martin K. Petersen <martin.petersen@xxxxxxxxxx>
>       Signed-off-by: Sasha Levin <alexander.levin@xxxxxxxxxxx>
> 
> pnmtopng: 182 colors found
> Revision graph left in 
> /home/logs/results/bisect/linux-3.18/test-amd64-amd64-xl-qemut-debianhvm-amd64.xen-boot.{dot,ps,png,html,svg}.
> ----------------------------------------
> 102788: tolerable ALL FAIL
> 
> flight 102788 linux-3.18 real-bisect [real]
> http://logs.test-lab.xenproject.org/osstest/logs/102788/
> 
> Failures :-/ but no regressions.
> 
> Tests which did not succeed,
> including tests which could not be run:
>  test-amd64-amd64-xl-qemut-debianhvm-amd64  6 xen-boot   fail baseline 
> untested
> 
> 
> jobs:
>  test-amd64-amd64-xl-qemut-debianhvm-amd64                    fail    
> 
> 
> ------------------------------------------------------------
> sg-report-flight on osstest.test-lab.xenproject.org
> logs: /home/logs/logs
> images: /home/logs/images
> 
> Logs, config files, etc. are available at
>     http://logs.test-lab.xenproject.org/osstest/logs
> 
> Explanation of these reports, and of osstest in general, is at
>     
> http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
>     http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master
> 
> Test harness code can be found at
>     http://xenbits.xen.org/gitweb?p=osstest.git;a=summary
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel