Xen project Mailing List

Hello,

Forgot to CC the Xen-devel mailing list.

I have changed the code to add the tests, and now it is like you need to provide the whole link of the mbox file rather than just the file name.

So, $python3 createjson.py --mboxhttp://lists.xenproject.org/archives/html/mbox/advisory-board-2013-05-2014-02--output new.json should work instead of

$ python3 createjson.py --mbox advisory-board-2014-02 --output new.json

I forgot to update the README file, and I have done that now. I will try k['data']['Message-ID'] instead, and I would update soon. I'm working on the testing part and I'll complete it soon.

Priya V

Amrita University

LinkedIn | GitHub | Bitbucket

On Mon, Apr 11, 2016 at 1:23 PM, Jesus M. Gonzalez-Barahona <jgb@xxxxxxxxxxxx> wrote:

On Fri, 2016-04-08 at 19:33 +0530, Priya wrote:
> Hello,
>
> I tried running the same command in new version of perceval. I found
> the following missing message id errors in perceval_mbox_parse.log
> file. I am working on the testing part and I will be able to finish
> it in one or two days.
>
> You can see the errors here [1]
>
> [1]:http://imgur.com/yVsIoCT

Hi, Priya. I'm not sure about what exactly is causing your messages,
since I cannot reproduce them (see below). But I still suspect that
they may happen because in current versions of Perceval the data parsed
from an mbox is no longer stored as first level key/data in the
dictionary returned by Perceval for each message, but in data for key
"data", which is itself a dictionary.

In particular, in the code:

-------------
for k in msg_json:
                    try:
                        if key == k['Message-ID'].strip('<>'):
                            k['property'] = key
-------------

probably you should be checking for k['data']['Message-ID'] instead of
just k['Message-ID'].

Please, have a look at how recent versions of Perceval produce the
dictionaries for each message...

But as I said, I cannot reproduce your error. When running your most
recent code right now (9a5abc47bbab3b06550) with the most recent
Perceval/master code (53efc14001c806f0452) I get:

--------------------
(perceval)jgb@expisito:~/src/outreachy/Dashboard/dashboard$ python3
createjson.py --mbox advisory-board-2014-02 --output new.json
Traceback (most recent call last):
File "createjson.py", line 96, in <module>
    main()
File "createjson.py", line 92, in main
    mparser.create_json(args.mbox,args.output)
File "createjson.py", line 59, in create_json
    messages = th.message_details(mbox_files)
File "/home/jgb/src/outreachy/Dashboard/dashboard/jwzthreading_r.py",
line 338, in message_details
    urllib.request.urlretrieve(filename, 'mbox')
File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
File "/usr/lib/python3.4/urllib/request.py", line 449, in open
    req = Request(fullurl, data)
File "/usr/lib/python3.4/urllib/request.py", line 267, in __init__
    self.full_url = url
File "/usr/lib/python3.4/urllib/request.py", line 293, in full_url
    self._parse()
File "/usr/lib/python3.4/urllib/request.py", line 322, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'advisory-board-2014-02'
-----------------------------

Could you please try to checkout and install exactly the same version
of Perceval I'm using, and see if you get the same error? And if the
above problem with the format returned by Perceval persists, maybe you
can fix that too.

Saludos,

Jesus.

--
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah

Re: [Xen-devel] Regarding Outreachy project on Improving CR Dashboard