Multi-thread read performance

[olsonanl]

I've been experimenting with NuDB and ran into the following issue - I would like to have multiple worker threads access the database concurrently, entirely read-only. There seems to be a bottleneck where I only get 2-3x speedups; intel profiler showed what looks to be locking overhead. Should I be able to see higher concurrency for this? Or perhaps am I seeing I/O read rate issues (the data file is in /dev/shm ramdisk for this testing)

Thanks,
Bob

[vinniefalco]

yes you should be seeing higher concurrency if there are more than 3 threads.

[olsonanl]

OK, thanks. I'll do some more digging, including putting the data back on the SSD volume. Do you think it's possible the performance characteristics of the ramdisk may be at fault?

[olsonanl]

I'm seeing a speedup of about 3.7 on 6 threads. The serial version of the code is seeing about 60% system CPU; I wonder if I'm just hitting disk overheads. Data records are small; 8 byte keys and 10 byte records. 600M entries.

(Prior version of this code loaded up an in-memory hashtable from a memory map; trying to get the app working with as little startup overhead as possible without the current solution of requiring either vmtouch-locked data or persistent server)

[vinniefalco]

Do you think it's possible the performance characteristics of the ramdisk may be at fault?

I don't know