Re: [Xen-devel] [PATCH RFC-for-4.12] tools/xen-foreign: Update python scripts to be Py3 compatible

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 3/5/19 12:31 PM, George Dunlap wrote:
> On 3/5/19 12:30 PM, Juergen Gross wrote:
>> On 05/03/2019 13:19, George Dunlap wrote:
>>> On 3/5/19 12:04 PM, Wei Liu wrote:
>>>> On Tue, Mar 05, 2019 at 12:41:53PM +0100, Juergen Gross wrote:
>>>>> On 04/03/2019 20:00, George Dunlap wrote:
>>>>>> On 3/4/19 6:31 PM, Andrew Cooper wrote:
>>>>>>> The issues are:
>>>>>>>  * dict.has_key() was completely removed in Py3
>>>>>>>  * dict.keys() is an iterable rather than list in Py3, so .sort() 
>>>>>>> doesn't work.
>>>>>>>  * list.sort(cmp=) was deprecated in Py2.4 and removed in Py3.  Replace 
>>>>>>> it
>>>>>>>    with a key= sort instead.
>>>>>>>
>>>>>>> This is all compatible with Py2.4 and later, which is when the sorted()
>>>>>>> builtin was introduced.  Tested with Py2.7 and Py3.4
>>>>>>
>>>>>> Did you test ./configure with python3?
>>>>>>
>>>>>> Loads of the python-related runes in the configure are also
>>>>>> python2-only, resulting in (at least for me) the wrong library paths
>>>>>> being generated, so the following fails:
>>>>>>
>>>>>>     PYTHON=/usr/bin/python3 ./configure
>>>>>>
>>>>>> With the following message:
>>>>>>
>>>>>>   File "<string>", line 1
>>>>>>     import distutils.sysconfig;     print
>>>>>> distutils.sysconfig.get_config_var("VERSION")
>>>>>>                                                   ^
>>>>>> SyntaxError: invalid syntax
>>>>>> checking for python3-config... /home/gdunlap/anaconda3/bin/python3-config
>>>>>> checking Python.h usability... yes
>>>>>> checking Python.h presence... yes
>>>>>> checking for Python.h... yes
>>>>>> checking for PyArg_ParseTuple in -lpython... no
>>>>>> configure: error: Unable to find a suitable python development library
>>>>>> configure: error: ./configure failed for tools
>>>>>>
>>>>>> Note the error with the VERSION above; that results in looking for
>>>>>> `-lpython` rather than `-lpython3.7`, which results in the following
>>>>>> line being executed:
>>>>>>
>>>>>> configure:7468: checking for PyArg_ParseTuple in -lpython
>>>>>> configure:7493: gcc -o conftest -g -O2 -g -O2
>>>>>> -I/persistent/local/home/dunlapg/anaconda3/include/python3.7m
>>>>>> -I/persistent/local/home/dunlapg/anaconda3/include/python3.7m
>>>>>> -Wno-unused-result -Wsign-compare -march=nocona -mtune=haswell
>>>>>> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe
>>>>>> -fdebug-prefix-map==/usr/local/src/conda/-
>>>>>> -fdebug-prefix-map==/usr/local/src/conda-prefix -fuse-linker-plugin
>>>>>> -ffat-lto-objects -flto-partition=none -flto -flto -fuse-linker-plugin
>>>>>> -ffat-lto-objects -flto-partition=none -g -DNDEBUG -fwrapv -O3 -Wall
>>>>>> -L/persistent/local/home/dunlapg/anaconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu
>>>>>> -L/persistent/local/home/dunlapg/anaconda3/lib -lpython3.7m -lpthread
>>>>>> -ldl  -lutil -lrt -lm  -Xlinker -export-dynamic conftest.c -lpython
>>>>>> -llzma  -lpython3.7m -lpthread -ldl  -lutil -lrt -lm  -llzma  >&5
>>>>>> /usr/bin/ld: cannot find -lpython
>>>>>> collect2: error: ld returned 1 exit status
>>>>>>
>>>>>> The following fixes it for me:
>>>>>>
>>>>>> diff --git a/tools/configure b/tools/configure
>>>>>> index acc857510e..d7e21db959 100755
>>>>>> --- a/tools/configure
>>>>>> +++ b/tools/configure
>>>>>> @@ -6953,7 +6953,7 @@ if echo "$PYTHON" | grep -q "^/"; then :
>>>>>>      PYTHON=`basename $PYTHONPATH`
>>>>>>
>>>>>>  elif test -z "$PYTHON"; then :
>>>>>> -  PYTHON="python"
>>>>>> +  PYTHON="python2"
>>>>>>  else
>>>>>>    as_fn_error $? "PYTHON specified, but is not an absolute path"
>>>>>> "$LINENO" 5
>>>>>>  fi
>>>>>> diff --git a/tools/configure.ac b/tools/configure.ac
>>>>>> index 1499344ce6..adf70aeb33 100644
>>>>>> --- a/tools/configure.ac
>>>>>> +++ b/tools/configure.ac
>>>>>> @@ -355,7 +355,7 @@ esac
>>>>>>  AS_IF([echo "$PYTHON" | grep -q "^/"], [
>>>>>>      PYTHONPATH=$PYTHON
>>>>>>      PYTHON=`basename $PYTHONPATH`
>>>>>> -],[test -z "$PYTHON"], [PYTHON="python"],
>>>>>> +],[test -z "$PYTHON"], [PYTHON="python2"],
>>>>>>  [AC_MSG_ERROR([PYTHON specified, but is not an absolute path])])
>>>>>>  AX_PATH_PROG_OR_FAIL([PYTHONPATH], [$PYTHON])
>>>>>>  AX_CHECK_PYTHON_VERSION([2], [3])
>>>>>>
>>>>>> ..but:
>>>>>> 1) I'm not sure if we can guarantee that `python2` will exist on all
>>>>>> systems we support,
>>>>>> 2) We probably want to support python3 at some point anyway.
>>>>>>
>>>>>> On the other hand, Arch has python -> python3, but (apparently) has
>>>>>> since 2011 [1]; so there's an argument to be made that this isn't
>>>>>> something that needs to be solved for 4.12.
>>>>>
>>>>> Which raises the question whether Andrew's patch should be included
>>>>> in 4.12. I'm tempted to refuse my Rab as I don't see why we need to
>>>>> be python3 compliant in the build system while configure isn't.
>>>>>
>>>>> Thoughts?
>>>>
>>>> I plan to have a closer look at python 3 compatibility issue today. But
>>>> the outcome is unlikely to be simple -- most likely there will be a few
>>>> patches. I don't think they will be suitable for 4.12.
>>>>
>>>> We can document the python 2 requirement for 4.12.
>>>
>>> Right, but there are two issues:
>>> 1. Build system requires python2
>>> 2. Build system assumes that 'python' == 'python2', and fails in strange
>>> ways when it's not
>>>
>>> I agree #1 isn't something to rush before the release.  Any opinions on #2?
>>>
>>> Options are:
>>>
>>> A. Simply document that if 'python'  isn't 'python2' (such as on Arch
>>> Linux, or if you've got anaconda python installed), you need to
>>> explicitly set PYTHON=/path/to/python2
>>>
>>> B. Hard-code configure to look for python2 if not specified rather than
>>> python.
>>>
>>> C. Have configure detect when python is python3, and look for python2,
>>> failing if it's not available.
>>>
>>> Evaluation:
>>>
>>> Option A isn't a great UI, but the situation has been like this for a
>>> long time, so it's probably not the worst thing ever to wait for the
>>> next release to fix things.  Also, serves you right for using such a
>>> broken distro. :-)
>>>
>>> Option B is dead-simple; I just don't know how far back it will be
>>> compatible.
>>>
>>> Option C is the most robust, but more work (though probably not a huge
>>> amount).
>>>
>>> Thoughts?
>>
>> I'm hoping to have 4.12 finished rather soon (maybe one further RC?).
>>
>> For this reason I'd like to include fixes for regressions only from now
>> on. This would imply option A.
> 
> Works for me.

I'll send a patch to README (which mentions python) saying that we
require python2; and saying how to build correctly if python points to
python3.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel