Question: How is max_mem_for_data calculated?

[seongju]

Where does the 0.6 come from? My understanding is that the two 1000s are to convert MB to bytes, but I can't really think of where 0.6 would come from. Is it just an arbitrary number to prevent reaching the memory limit of lambda?

lambda-refarch-mapreduce/src/python/lambdautils.py

Line 131 in ed3ae93

max_mem_for_data = 0.6 * lambda_memory * 1000 * 1000;

[Rayz0r]

I haven't checked with Sunil, but I suspect it is. In the single node test from AMPLABS each key is actually very small. I have tested with 128MB memory and because each key still fits in 76MB (128MB*0.6) the test still works fine. However for people with different size keys we should really parameterize it around the lambdaMemory variable.