[Liblas-devel] I/O performance? -- your help requested!

[David Burken]

I see I need a faster machine:

$  uname -a
Linux tazz 2.6.35.10-74.fc14.x86_64 #1 SMP Thu Dec 23 16:04:50 UTC 2010 
x86_64 x86_64 x86_64 GNU/Linux
model name    : Intel(R) Core(TM)2 Quad  CPU   Q9450  @ 2.66GHz
MemTotal:        4056188 kB


$ time laszippertest -n 1000000
Settings:
   num_points=1000000, use_iostream=false, run_forever=false, 
use_random=false, user_seed=1296923027
laszipper wrote 70000000 bytes in 0.47 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.27 seconds
laszipper wrote 38271321 bytes in 1.6 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.4 seconds
Finished 1 runs


real    0m4.769s
user    0m4.379s
sys    0m0.373s
$ time laszippertest -n 1000000
Settings:
   num_points=1000000, use_iostream=false, run_forever=false, 
use_random=false, user_seed=1296923035
laszipper wrote 70000000 bytes in 0.47 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.27 seconds
laszipper wrote 38271321 bytes in 1.55 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.33 seconds
Finished 1 runs


real    0m4.857s
user    0m4.298s
sys    0m0.358s
$ time laszippertest -n 1000000
Settings:
   num_points=1000000, use_iostream=false, run_forever=false, 
use_random=false, user_seed=1296923043
laszipper wrote 70000000 bytes in 0.49 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.26 seconds
laszipper wrote 38271321 bytes in 1.56 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.32 seconds
Finished 1 runs


real    0m4.659s
user    0m4.291s
sys    0m0.364s
$ time laszippertest -n 1000000 -s
Settings:
   num_points=1000000, use_iostream=true, run_forever=false, 
use_random=false, user_seed=1296923051
laszipper wrote 70000000 bytes in 0.53 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.34 seconds
laszipper wrote 38271321 bytes in 1.69 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.55 seconds
Finished 1 runs


real    0m5.342s
user    0m4.626s
sys    0m0.516s
$ time laszippertest -n 1000000 -s
Settings:
   num_points=1000000, use_iostream=true, run_forever=false, 
use_random=false, user_seed=1296923058
laszipper wrote 70000000 bytes in 0.49 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.36 seconds
laszipper wrote 38271321 bytes in 1.72 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.46 seconds
Finished 1 runs


real    0m5.073s
user    0m4.556s
sys    0m0.515s
$ time laszippertest -n 1000000 -s
Settings:
   num_points=1000000, use_iostream=true, run_forever=false, 
use_random=false, user_seed=1296923064
laszipper wrote 70000000 bytes in 0.51 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.35 seconds
laszipper wrote 38271321 bytes in 1.69 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.41 seconds
Finished 1 runs


real    0m5.577s
user    0m4.487s
sys    0m0.508s

On 02/04/2011 11:39 AM, Michael P. Gerlek wrote:
> Do you have 5-10 minutes to spare today?
>
> Your libLAS team (well, me anyway) is wondering about I/O performance of the
> liblas kit -- specifically, when doing binary reading and writing, is there
> any fundamental performance difference between using C-style FILE* I/O and
> C++-style stream I/O?  And if streams are better, would boost's stream be
> better still?  If you google around a bit, you'll find lots of contradictory
> (and sometimes overly passionate) statements about this topic.  At the end
> of the day, though, the consensus seems to be that:
>
>    (1) you need to be "smart" if you're using C++ I/O -- it is easy to shoot
> yourself in the foot
>    (2) modern C++ streams are implemented on top of the native OS APIs
>    (3) under Visual Studio, FILE* operations and streams are both implemented
> using the win32 APIs, but streams have an additional lock (that is claimed
> by some to be not needed)
>
> and, most importantly,
>
>    (4) performance varies greatly with different I/O patterns, e.g. large
> sequential block reads vs small random reads
>
> Very fortunately, we happen to already have a rough, 1st-order I/O
> performance test built into the laszip tree.  If you have that tree built
> (http://hg.liblas.org/zip), in Release mode, could you please send me the
> results of running the "laszippertest" test app, as follows?
>
>      time ./laszippertest -n 1000000
>      time ./laszippertest -n 1000000
>      time ./laszippertest -n 1000000
>      time ./laszippertest -n 1000000 -s
>      time ./laszippertest -n 1000000 -s
>      time ./laszippertest -n 1000000 -s
>
> The first three runs will encode and decode 1 million random points using
> FILEs, and the second three will do it with streams.  This is not a perfect
> test, but it represents something approximating the real I/O footprint or
> traces that liblas uses.
>
> Oh, and be sure to include the kind of platform (processor speed, compiler,
> OS) you're running it on.
>
> Thanks much!
>
> -mpg
>
>
> _______________________________________________
> Liblas-devel mailing list
> Liblas-devel at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/liblas-devel
>