out of memory exception caused by big array in java -


i'm implenmenting algorithm based on probabilistic latent semantic indexing(plsa) , paper hereand need 4 dimension array named p_z_d_wt_wv, z topic, d document, wt text word, wv visual word,and number of each dimension 12, 7000,100, 500, , array double array, need 32g memory!! allocate memory way below, , demonstration number of wt , wv in each document different.

p_z_d_wt_wv = new double[12][7000][][];  for( int t = 0; t < 12; ++t)  {      for( int d = 0; d < 7000; ++d )      {          p_z_d_wt_wv[t][d] = new double[100][500];     }  } 

when run code, has out of memory problem. first, why code run out of memory? are memory allocated consecutively if array allocated in way? because java have memory limit consecutive memory? if so, what's limit?

second, can solve problem supposed memory of server big enough. know can change float array, there other solutions?

if need of memory, well, need of memory.

there alternatives:

  1. you using memory mapped files.

  2. if array has lot of zeros in it, store sparse matrix representation (don't explicitly store 0s).

  3. if don't need whole thing in memory @ once, store in sort of persistent storage (file, database, etc) , access parts need @ given time.


Comments

Popular posts from this blog

javascript - Count length of each class -

What design pattern is this code in Javascript? -

hadoop - Restrict secondarynamenode to be installed and run on any other node in the cluster -