Re: Alternative Hashtbl and lwt yield-like functionality questions

[Thomas Gazagnaire <thomas.gazagnaire@xxxxxxxxx>]

> Firstly, I am writing a regression test for mirage in which I need to
> store a large number of entries in an Hash table. The test will create
> unique entries in the table with a rate of   ~70K netries per second.
> My experiment so far shows that the performance of the Hashtbl
> insertions start to drop after a few seconds, as the memory management
> will start to consume cycles in order to allocate new buckets and
> possible rearrange the the distribution of the hashed values. The key
> I am using is a record of a string and a 64bit int. The string
> represents a mac address. Is any one aware of some other ocaml module
> that could satisfy the basic requirements of a hash table and would
> possible scale to large numbers of insertions?

How long does you test longs ? Do you really want to store your data in a 
structured way ?
What you can do to avoid too much allocation during your test is to allocate an 
array each second (say of size 70K), and once filled, to store it in a mutable 
list and allocate a new array to continue your test. When you test is finished, 
you can analyze your mutable list and build your association table using the 
data-structure you want.

--
Thomas

> Secondly, I am currently using the lwt threading mechanism in order to
> handle all the asynvronous functionality of my code. I have a minor
> problem with the scheduler though as it tends to stick with a single
> thread during a test and doesn't schedule any other as long as it has
> data to process. Can somebody more experienced with lwt point me some
> calls or configuration parameters that would allow either a thread to
> yield explicitly, or implement a more fair scheduling policy.
> 
> thanks in advance for any replies.
> 
> 
> -- 
> Charalampos Rotsos
> PhD student
> The University of Cambridge
> Computer Laboratory
> William Gates Building
> JJ Thomson Avenue
> Cambridge
> CB3 0FD
> 
> Phone: +44-(0) 1223 767032
> Email: cr409@xxxxxxxxxxxx
>