[tlug] how to tune reiser4 for millions of files?

[Michal Hajek <hajek1@example.com>]

Hello, 

is there any hope to manage nearly 7 million of files in one directory 
by tuning reiser4 fs?

Situation: somebody wrote incredibly stupid application which analyzes
some measurement data by producing 7032034 small text files into one
directory. Now we need to analyze that data "en block", that is, we
cannot split it into many dirs. Also we need it somewhat soon. That is,
hopefuly before the proces of rewriting the above mentioned stupid
application comes to an end (could take months). Right now I employed
reiser4 fs on a scsi disk array in default configuration. Still `ls -l`
takes more than 6 hours - more accurately, after 6 hours I lost my
patience and killed it.

So I wonder if by moving that whole dir to another place (it totals to
approximately 74GB) and tuning some parameters of reiser4 fs, I could 
get noticably better performance.

I measure the performance by iotop, and top mostly (feel free to give
advice on this). 

one example:

$iotop -o 
5738 root     1818.83 K/s       0 B/s  0.00 % 97.30 % evn421 40,47,52,57,0 /mnt/polea/out/data-0000001

(97.30 %  is io waiting time - I suggest it is because fs deals with
millions of files )

$top
 5738 root      20   0 2018m 2.0g  488 D    4 12.6  16:31.31 evn421  
 (4 = cpu usage of "evn421")

also top shows
 Cpu0  :  0.7%us,  1.7%sy,  0.0%ni,  0.0%id, 97.0%wa,  0.3%hi,  0.3%si, 0.0%st
 

Best regards
Michal