Re: Alternative Hashtbl and lwt yield-like functionality questions

[Thomas Gazagnaire <thomas.gazagnaire@xxxxxxxxx>]

>> I don't get it. Why do you need to read the data you are producing while 
>> your test is running ?
> 
> I am running a code that simulates the functionality of a
> learning_switch. The switch will inspect mac layer header and discover
> on which port it sees which mac address. So the Hashtbl needs to
> insert information while it reads data but also use that information
> to forward frames.

Ok I see. Thx for the explanation.

>>> In any case, what I did was to
>>> reduce the size of the experiment. It was though a surprise to me how
>>> inefficient memory-wise is the Hashtbl in comparison to c structures.
>> 
>> it is really depending on the data-structure you are using. If you have a 
>> know in advance the size of your data, and you want to have constant time 
>> read/write, you can use arrays. If your data is not bound, the only way it 
>> to amortize your costs. I'm curious to know which kind of C structure does a 
>> better job here.
> 
> I am comparing against a similar implementation of the code that uses
> the boost hashtbl. While the test runs, the c++ code create a very
> small footprint, while the ocaml code starts bloating. As an example
> for the previous experiment a mirage vm will run out of memory after
> 20 seconds of insertion, while the c++ code will consume only a few
> megs. I guess this is the cost you pay when you want automatic memory
> management.

The default hashing algorithm is known to generate a lot of collisions on some 
data-structure. Maybe you've hit such a case. It's strange to have so much 
allocation. Anyway, I'm interested to look at your code if it is available 
somewhere :-)

--
Thomas

>>> On 14 December 2011 14:04, Thomas Gazagnaire
>>> <thomas.gazagnaire@xxxxxxxxx> wrote:
>>>> How long does you test longs ? Do you really want to store your data in a 
>>>> structured way ?
>>>> What you can do to avoid too much allocation during your test is to 
>>>> allocate an array each second (say of size 70K), and once filled, to store 
>>>> it in a mutable list and allocate a new array to continue your test. When 
>>>> you test is finished, you can analyze your mutable list and build your 
>>>> association table using the data-structure you want.
>>> 
>>> 
>>> 
>>> --
>>> Charalampos Rotsos
>>> PhD student
>>> The University of Cambridge
>>> Computer Laboratory
>>> William Gates Building
>>> JJ Thomson Avenue
>>> Cambridge
>>> CB3 0FD
>>> 
>>> Phone: +44-(0) 1223 767032
>>> Email: cr409@xxxxxxxxxxxx
>> 
>> 
> 
> 
> 
> -- 
> Charalampos Rotsos
> PhD student
> The University of Cambridge
> Computer Laboratory
> William Gates Building
> JJ Thomson Avenue
> Cambridge
> CB3 0FD
> 
> Phone: +44-(0) 1223 767032
> Email: cr409@xxxxxxxxxxxx