[Liblas-devel] I/O performance? -- your help requested!
Michael P. Gerlek
mpg at flaxen.com
Mon Feb 7 13:07:38 EST 2011
> I did 100x and something went out of range I think:
I'll look into that, thanks.
Anyone else have any numbers to contribute? I'll roll up the results
tomorrow.
-mpg
From: liblas-devel-bounces at lists.osgeo.org
[mailto:liblas-devel-bounces at lists.osgeo.org] On Behalf Of David Burken
Sent: Monday, February 07, 2011 10:01 AM
To: liblas-devel at lists.osgeo.org
Subject: Re: [Liblas-devel] I/O performance? -- your help requested!
OK I feel better now. I see Michael's and mine are about the same.
I did 100x and something went out of range I think:
$ time laszippertest -n 100000000
Settings:
num_points=100000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297100531
laszipper wrote -1589934592 bytes in 58.12 seconds
Note the negative, also had to control-c.
6 10x below and 2 million points. Hope this helps...
[dburken at tazz build]$ time laszippertest -n 10000000
Settings:
num_points=10000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297100699
laszipper wrote 700000000 bytes in 5.85 seconds
SUCCESS: lasunzipper read 700000000 bytes in 3.85 seconds
laszipper wrote 381625423 bytes in 17.62 seconds
SUCCESS: lasunzipper read 381625422 bytes in 28.49 seconds
Finished 1 runs
real 0m59.820s
user 0m51.617s
sys 0m5.002s
[dburken at tazz build]$
[dburken at tazz build]$ time laszippertest -n 10000000
Settings:
num_points=10000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297100974
laszipper wrote 700000000 bytes in 4.91 seconds
SUCCESS: lasunzipper read 700000000 bytes in 2.63 seconds
laszipper wrote 381625423 bytes in 18.51 seconds
SUCCESS: lasunzipper read 381625422 bytes in 30.89 seconds
Finished 1 runs
real 0m58.074s
user 0m53.394s
sys 0m3.840s
[dburken at tazz build]$ time laszippertest -n 10000000
Settings:
num_points=10000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297101035
laszipper wrote 700000000 bytes in 4.79 seconds
SUCCESS: lasunzipper read 700000000 bytes in 2.67 seconds
laszipper wrote 381625423 bytes in 20.48 seconds
SUCCESS: lasunzipper read 381625422 bytes in 23.44 seconds
Finished 1 runs
real 0m52.201s
user 0m47.626s
sys 0m4.031s
[dburken at tazz build]$ time laszippertest -n 10000000 -s
Settings:
num_points=10000000, use_iostream=true, run_forever=false,
use_random=false, user_seed=1297101129
laszipper wrote 700000000 bytes in 6.81 seconds
SUCCESS: lasunzipper read 700000000 bytes in 4.76 seconds
laszipper wrote 381625423 bytes in 21.66 seconds
SUCCESS: lasunzipper read 381625422 bytes in 28.8 seconds
Finished 1 runs
real 1m2.768s
user 0m55.831s
sys 0m6.590s
[dburken at tazz build]$ time laszippertest -n 10000000 -s
Settings:
num_points=10000000, use_iostream=true, run_forever=false,
use_random=false, user_seed=1297101209
laszipper wrote 700000000 bytes in 5.09 seconds
SUCCESS: lasunzipper read 700000000 bytes in 3.53 seconds
laszipper wrote 381625423 bytes in 17.39 seconds
SUCCESS: lasunzipper read 381625422 bytes in 27.96 seconds
Finished 1 runs
real 0m54.278s
user 0m49.023s
sys 0m5.235s
[dburken at tazz build]$ time laszippertest -n 10000000 -s
Settings:
num_points=10000000, use_iostream=true, run_forever=false,
use_random=false, user_seed=1297101272
laszipper wrote 700000000 bytes in 5.2 seconds
SUCCESS: lasunzipper read 700000000 bytes in 3.58 seconds
laszipper wrote 381625423 bytes in 17.02 seconds
SUCCESS: lasunzipper read 381625422 bytes in 24.59 seconds
Finished 1 runs
real 0m50.899s
user 0m45.558s
sys 0m5.113s
[dburken at tazz build]$ time laszippertest -n 1000000
Settings:
num_points=1000000, use_iostream=false, run_forever=false,
use_random=false, user_seed=1297101353
laszipper wrote 70000000 bytes in 0.47 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.27 seconds
laszipper wrote 38271321 bytes in 1.55 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.45 seconds
Finished 1 runs
real 0m5.060s
user 0m4.435s
sys 0m0.594s
[dburken at tazz build]$ time laszippertest -n 1000000 -s
Settings:
num_points=1000000, use_iostream=true, run_forever=false,
use_random=false, user_seed=1297101361
laszipper wrote 70000000 bytes in 0.5 seconds
SUCCESS: lasunzipper read 70000000 bytes in 0.36 seconds
laszipper wrote 38271321 bytes in 1.69 seconds
SUCCESS: lasunzipper read 38271321 bytes in 2.48 seconds
Finished 1 runs
On 02/07/2011 12:02 PM, Michael P. Gerlek wrote:
Your box is too fast. :-(
Could you maybe do the runs again, but with "-n 100000000" (100x larger)?
-mpg
From: Smith, Michael ERDC-CRREL-NH [mailto:michael.smith at usace.army.mil]
Sent: Saturday, February 05, 2011 4:48 AM
To: mpg at flaxen.com; liblas-devel at lists.osgeo.org
Subject: Re: [Liblas-devel] I/O performance? -- your help requested!
lidar at lidarora1 tmp]$ uname -a
Linux lidarora1 2.6.32-100.0.19.el5 #1 SMP Fri Sep 17 17:51:41 EDT 2010
x86_64 x86_64 x86_64 GNU/Linux
2 CPU, 4cores/cpu + hyperthreading = 16 virtual cores
processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.18 seconds
real 0m0.379s
user 0m0.349s
sys 0m0.029s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds
real 0m0.366s
user 0m0.336s
sys 0m0.029s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds
real 0m0.366s
user 0m0.338s
sys 0m0.026s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds
real 0m0.364s
user 0m0.338s
sys 0m0.024s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds
real 0m0.369s
user 0m0.339s
sys 0m0.028s
[lidar at lidarora1 tmp]$ time laszippertest -n 1000000 -s
(skipping range coder test)
laszipper1 wrote 7000000 bytes in 0.03 seconds
laszipper2 wrote 3898142 bytes in 0.13 seconds
SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds
SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds
real 0m0.364s
user 0m0.335s
sys 0m0.027s
--
Michael Smith
Remote Sensing/GIS Center
US Army Corps of Engineers
On 2/4/11 11:39 AM, "Michael P. Gerlek" <mpg at flaxen.com> wrote:
Do you have 5-10 minutes to spare today?
Your libLAS team (well, me anyway) is wondering about I/O performance of the
liblas kit -- specifically, when doing binary reading and writing, is there
any fundamental performance difference between using C-style FILE* I/O and
C++-style stream I/O? And if streams are better, would boost's stream be
better still? If you google around a bit, you'll find lots of contradictory
(and sometimes overly passionate) statements about this topic. At the end
of the day, though, the consensus seems to be that:
(1) you need to be "smart" if you're using C++ I/O -- it is easy to shoot
yourself in the foot
(2) modern C++ streams are implemented on top of the native OS APIs
(3) under Visual Studio, FILE* operations and streams are both implemented
using the win32 APIs, but streams have an additional lock (that is claimed
by some to be not needed)
and, most importantly,
(4) performance varies greatly with different I/O patterns, e.g. large
sequential block reads vs small random reads
Very fortunately, we happen to already have a rough, 1st-order I/O
performance test built into the laszip tree. If you have that tree built
(http://hg.liblas.org/zip), in Release mode, could you please send me the
results of running the "laszippertest" test app, as follows?
time ./laszippertest -n 1000000
time ./laszippertest -n 1000000
time ./laszippertest -n 1000000
time ./laszippertest -n 1000000 -s
time ./laszippertest -n 1000000 -s
time ./laszippertest -n 1000000 -s
The first three runs will encode and decode 1 million random points using
FILEs, and the second three will do it with streams. This is not a perfect
test, but it represents something approximating the real I/O footprint or
traces that liblas uses.
Oh, and be sure to include the kind of platform (processor speed, compiler,
OS) you're running it on.
Thanks much!
-mpg
_______________________________________________
Liblas-devel mailing list
Liblas-devel at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/liblas-devel
_______________________________________________
Liblas-devel mailing list
Liblas-devel at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/liblas-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/liblas-devel/attachments/20110207/7878e25c/attachment.html
More information about the Liblas-devel
mailing list