[Liblas-devel] I/O performance? -- your help requested!
David Burken
dburken at comcast.net
Mon Feb 7 13:00:39 EST 2011
OK I feel better now. I see Michael's and mine are about the same.
I did 100x and something went out of range I think:
$ time laszippertest -n 100000000
Settings:
num_points=100000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297100531
laszipper wrote -1589934592 bytes in 58.12 seconds
Note the negative, also had to control-c.
6 10x below and 2 million points. Hope this helps...
[dburken at tazz build]$ time laszippertest -n 10000000
Settings:
num_points=10000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297100699
laszipper wrote 700000000 bytes in 5.85 seconds
SUCCESS: lasunzipper read 700000000 bytes in 3.85 seconds
laszipper wrote 381625423 bytes in 17.62 seconds
SUCCESS: lasunzipper read 381625422 bytes in 28.49 seconds
Finished 1 runs
real 0m59.820s
user 0m51.617s
sys 0m5.002s
[dburken at tazz build]$
[dburken at tazz build]$ time laszippertest -n 10000000
Settings:
num_points=10000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297100974
laszipper wrote 700000000 bytes in 4.91 seconds
SUCCESS: lasunzipper read 700000000 bytes in 2.63 seconds
laszipper wrote 381625423 bytes in 18.51 seconds
SUCCESS: lasunzipper read 381625422 bytes in 30.89 seconds
Finished 1 runs
real 0m58.074s
user 0m53.394s
sys 0m3.840s
[dburken at tazz build]$ time laszippertest -n 10000000
Settings:
num_points=10000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297101035
laszipper wrote 700000000 bytes in 4.79 seconds
SUCCESS: lasunzipper read 700000000 bytes in 2.67 seconds
laszipper wrote 381625423 bytes in 20.48 seconds
SUCCESS: lasunzipper read 381625422 bytes in 23.44 seconds
Finished 1 runs
real 0m52.201s
user 0m47.626s
sys 0m4.031s
[dburken at tazz build]$ time laszippertest -n 10000000 -s
Settings:
num_points=10000000, use_iostream=true, run_forever=false,
use_random=false, user_seed=1297101129
laszipper wrote 700000000 bytes in 6.81 seconds
SUCCESS: lasunzipper read 700000000 bytes in 4.76 seconds
laszipper wrote 381625423 bytes in 21.66 seconds
SUCCESS: lasunzipper read 381625422 bytes in 28.8 seconds
Finished 1 runs
real 1m2.768s
user 0m55.831s
sys 0m6.590s
[dburken at tazz build]$ time laszippertest -n 10000000 -s
Settings:
num_points=10000000, use_iostream=true, run_forever=false,
use_random=false, user_seed=1297101209
laszipper wrote 700000000 bytes in 5.09 seconds
SUCCESS: lasunzipper read 700000000 bytes in 3.53 seconds
laszipper wrote 381625423 bytes in 17.39 seconds
SUCCESS: lasunzipper read 381625422 bytes in 27.96 seconds
Finished 1 runs
real 0m54.278s
user 0m49.023s
sys 0m5.235s
[dburken at tazz build]$ time laszippertest -n 10000000 -s
Settings:
num_points=10000000, use_iostream=true, run_forever=false,
use_random=false, user_seed=1297101272
laszipper wrote 700000000 bytes in 5.2 seconds
SUCCESS: lasunzipper read 700000000 bytes in 3.58 seconds
laszipper wrote 381625423 bytes in 17.02 seconds
SUCCESS: lasunzipper read 381625422 bytes in 24.59 seconds
Finished 1 runs
real 0m50.899s
user 0m45.558s
sys 0m5.113s
[dburken at tazz build]$ time laszippertest -n 1000000
Settings:
num_points=1000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297101353
laszipper wrote 70000000 bytes in 0.47 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.27 seconds
laszipper wrote 38271321 bytes in 1.55 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.45 seconds
Finished 1 runs
real 0m5.060s
user 0m4.435s
sys 0m0.594s
[dburken at tazz build]$ time laszippertest -n 1000000 -s
Settings:
num_points=1000000, use_iostream=true, run_forever=false,
use_random=false, user_seed=1297101361
laszipper wrote 70000000 bytes in 0.5 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.36 seconds
laszipper wrote 38271321 bytes in 1.69 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.48 seconds
Finished 1 runs
On 02/07/2011 12:02 PM, Michael P. Gerlek wrote:
>
> Your box is too fast. :-(
>
> Could you maybe do the runs again, but with "--n 100000000" (100x larger)?
>
> -mpg
>
> *From:*Smith, Michael ERDC-CRREL-NH [mailto:michael.smith at usace.army.mil]
> *Sent:* Saturday, February 05, 2011 4:48 AM
> *To:* mpg at flaxen.com; liblas-devel at lists.osgeo.org
> *Subject:* Re: [Liblas-devel] I/O performance? -- your help requested!
>
> lidar at lidarora1 tmp]$ uname -a
> Linux lidarora1 2.6.32-100.0.19.el5 #1 SMP Fri Sep 17 17:51:41 EDT
> 2010 x86_64 x86_64 x86_64 GNU/Linux
>
> 2 CPU, 4cores/cpu + hyperthreading = 16 virtual cores
>
> processor : 15
> vendor_id : GenuineIntel
> cpu family : 6
> model : 26
> model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
> stepping : 5
> cpu MHz : 1600.000
> cache size : 8192 KB
>
>
>
> [lidar at lidarora1 tmp]$ time laszippertest -n 1000000
> (skipping range coder test)
> laszipper1 wrote 7000000 bytes in 0.03 seconds
> laszipper2 wrote 3898142 bytes in 0.13 seconds
> SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
> SUCCESS: lasunzipper2 read 3898142 bytes in 0.18 seconds
>
> real 0m0.379s
> user 0m0.349s
> sys 0m0.029s
> [lidar at lidarora1 tmp]$ time laszippertest -n 1000000
> (skipping range coder test)
> laszipper1 wrote 7000000 bytes in 0.03 seconds
> laszipper2 wrote 3898142 bytes in 0.13 seconds
> SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
> SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds
>
> real 0m0.366s
> user 0m0.336s
> sys 0m0.029s
> [lidar at lidarora1 tmp]$ time laszippertest -n 1000000
> (skipping range coder test)
> laszipper1 wrote 7000000 bytes in 0.03 seconds
> laszipper2 wrote 3898142 bytes in 0.13 seconds
> SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
> SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds
>
> real 0m0.366s
> user 0m0.338s
> sys 0m0.026s
> [lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
> (skipping range coder test)
> laszipper1 wrote 7000000 bytes in 0.03 seconds
> laszipper2 wrote 3898142 bytes in 0.13 seconds
> SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
> SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds
>
> real 0m0.364s
> user 0m0.338s
> sys 0m0.024s
> [lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
> (skipping range coder test)
> laszipper1 wrote 7000000 bytes in 0.03 seconds
> laszipper2 wrote 3898142 bytes in 0.13 seconds
> SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
> SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds
>
> real 0m0.369s
> user 0m0.339s
> sys 0m0.028s
> [lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
> (skipping range coder test)
> laszipper1 wrote 7000000 bytes in 0.03 seconds
> laszipper2 wrote 3898142 bytes in 0.13 seconds
> SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
> SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds
>
> real 0m0.364s
> user 0m0.335s
> sys 0m0.027s
>
>
>
> --
> Michael Smith
> Remote Sensing/GIS Center
> US Army Corps of Engineers
>
>
> On 2/4/11 11:39 AM, "Michael P. Gerlek" <mpg at flaxen.com> wrote:
>
> Do you have 5-10 minutes to spare today?
>
> Your libLAS team (well, me anyway) is wondering about I/O performance
> of the
> liblas kit -- specifically, when doing binary reading and writing, is
> there
> any fundamental performance difference between using C-style FILE* I/O and
> C++-style stream I/O? And if streams are better, would boost's stream be
> better still? If you google around a bit, you'll find lots of
> contradictory
> (and sometimes overly passionate) statements about this topic. At the end
> of the day, though, the consensus seems to be that:
>
> (1) you need to be "smart" if you're using C++ I/O -- it is easy to
> shoot
> yourself in the foot
> (2) modern C++ streams are implemented on top of the native OS APIs
> (3) under Visual Studio, FILE* operations and streams are both
> implemented
> using the win32 APIs, but streams have an additional lock (that is claimed
> by some to be not needed)
>
> and, most importantly,
>
> (4) performance varies greatly with different I/O patterns, e.g. large
> sequential block reads vs small random reads
>
> Very fortunately, we happen to already have a rough, 1st-order I/O
> performance test built into the laszip tree. If you have that tree built
> (http://hg.liblas.org/zip), in Release mode, could you please send me the
> results of running the "laszippertest" test app, as follows?
>
> time ./laszippertest -n 1000000
> time ./laszippertest -n 1000000
> time ./laszippertest -n 1000000
> time ./laszippertest -n 1000000 -s
> time ./laszippertest -n 1000000 -s
> time ./laszippertest -n 1000000 -s
>
> The first three runs will encode and decode 1 million random points using
> FILEs, and the second three will do it with streams. This is not a
> perfect
> test, but it represents something approximating the real I/O footprint or
> traces that liblas uses.
>
> Oh, and be sure to include the kind of platform (processor speed,
> compiler,
> OS) you're running it on.
>
> Thanks much!
>
> -mpg
>
>
> _______________________________________________
> Liblas-devel mailing list
> Liblas-devel at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/liblas-devel
>
>
> _______________________________________________
> Liblas-devel mailing list
> Liblas-devel at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/liblas-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/liblas-devel/attachments/20110207/aacfcf20/attachment-0001.html
More information about the Liblas-devel
mailing list