[Liblas-devel] I/O performance? -- your help requested!

[Michael P. Gerlek]

Your box is too fast. :-(

 

Could you maybe do the runs again, but with "-n 100000000" (100x larger)?

 

-mpg

 

 

From: Smith, Michael ERDC-CRREL-NH [mailto:michael.smith at usace.army.mil] 
Sent: Saturday, February 05, 2011 4:48 AM
To: mpg at flaxen.com; liblas-devel at lists.osgeo.org
Subject: Re: [Liblas-devel] I/O performance? -- your help requested!

 

lidar at lidarora1 tmp]$ uname -a
Linux lidarora1 2.6.32-100.0.19.el5 #1 SMP Fri Sep 17 17:51:41 EDT 2010
x86_64 x86_64 x86_64 GNU/Linux

2 CPU, 4cores/cpu + hyperthreading = 16 virtual cores 

processor    : 15
vendor_id    : GenuineIntel
cpu family    : 6
model        : 26
model name    : Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping    : 5
cpu MHz        : 1600.000
cache size    : 8192 KB



[lidar at lidarora1 tmp]$ time laszippertest -n 1000000
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.18 seconds

real    0m0.379s
user    0m0.349s
sys    0m0.029s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds

real    0m0.366s
user    0m0.336s
sys    0m0.029s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds

real    0m0.366s
user    0m0.338s
sys    0m0.026s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds

real    0m0.364s
user    0m0.338s
sys    0m0.024s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds

real    0m0.369s
user    0m0.339s
sys    0m0.028s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds

real    0m0.364s
user    0m0.335s
sys    0m0.027s



-- 
Michael Smith
Remote Sensing/GIS Center
US Army Corps of Engineers


On 2/4/11 11:39 AM, "Michael P. Gerlek" <mpg at flaxen.com> wrote:

Do you have 5-10 minutes to spare today?

Your libLAS team (well, me anyway) is wondering about I/O performance of the
liblas kit -- specifically, when doing binary reading and writing, is there
any fundamental performance difference between using C-style FILE* I/O and
C++-style stream I/O?  And if streams are better, would boost's stream be
better still?  If you google around a bit, you'll find lots of contradictory
(and sometimes overly passionate) statements about this topic.  At the end
of the day, though, the consensus seems to be that:

  (1) you need to be "smart" if you're using C++ I/O -- it is easy to shoot
yourself in the foot
  (2) modern C++ streams are implemented on top of the native OS APIs
  (3) under Visual Studio, FILE* operations and streams are both implemented
using the win32 APIs, but streams have an additional lock (that is claimed
by some to be not needed)

and, most importantly,

  (4) performance varies greatly with different I/O patterns, e.g. large
sequential block reads vs small random reads

Very fortunately, we happen to already have a rough, 1st-order I/O
performance test built into the laszip tree.  If you have that tree built
(http://hg.liblas.org/zip), in Release mode, could you please send me the
results of running the "laszippertest" test app, as follows?

    time ./laszippertest -n 1000000
    time ./laszippertest -n 1000000
    time ./laszippertest -n 1000000
    time ./laszippertest -n 1000000 -s
    time ./laszippertest -n 1000000 -s
    time ./laszippertest -n 1000000 -s

The first three runs will encode and decode 1 million random points using
FILEs, and the second three will do it with streams.  This is not a perfect
test, but it represents something approximating the real I/O footprint or
traces that liblas uses.

Oh, and be sure to include the kind of platform (processor speed, compiler,
OS) you're running it on.

Thanks much!

-mpg


_______________________________________________
Liblas-devel mailing list
Liblas-devel at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/liblas-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/liblas-devel/attachments/20110207/78ea3919/attachment.html