<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><title>Re: [Liblas-devel] I/O performance? -- your help requested!</title><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>> I did 100x and something went out of range I think:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’ll look into that, thanks.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Anyone else have any numbers to contribute? I’ll roll up the results tomorrow.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>-mpg<o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> liblas-devel-bounces@lists.osgeo.org [mailto:liblas-devel-bounces@lists.osgeo.org] <b>On Behalf Of </b>David Burken<br><b>Sent:</b> Monday, February 07, 2011 10:01 AM<br><b>To:</b> liblas-devel@lists.osgeo.org<br><b>Subject:</b> Re: [Liblas-devel] I/O performance? -- your help requested!<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>OK I feel better now. I see Michael's and mine are about the same.<br><br>I did 100x and something went out of range I think:<br><br>$ time laszippertest -n 100000000<br>Settings:<br> num_points=100000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297100531<br>laszipper wrote -1589934592 bytes in 58.12 seconds<br><br>Note the negative, also had to control-c.<br><br>6 10x below and 2 million points. Hope this helps...<br><br>[dburken@tazz build]$ time laszippertest -n 10000000<br>Settings:<br> num_points=10000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297100699<br>laszipper wrote 700000000 bytes in 5.85 seconds<br>SUCCESS: lasunzipper read 700000000 bytes in 3.85 seconds<br>laszipper wrote 381625423 bytes in 17.62 seconds<br>SUCCESS: lasunzipper read 381625422 bytes in 28.49 seconds<br>Finished 1 runs<br><br><br>real 0m59.820s<br>user 0m51.617s<br>sys 0m5.002s<br>[dburken@tazz build]$ <br>[dburken@tazz build]$ time laszippertest -n 10000000<br>Settings:<br> num_points=10000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297100974<br>laszipper wrote 700000000 bytes in 4.91 seconds<br>SUCCESS: lasunzipper read 700000000 bytes in 2.63 seconds<br>laszipper wrote 381625423 bytes in 18.51 seconds<br>SUCCESS: lasunzipper read 381625422 bytes in 30.89 seconds<br>Finished 1 runs<br><br><br>real 0m58.074s<br>user 0m53.394s<br>sys 0m3.840s<br>[dburken@tazz build]$ time laszippertest -n 10000000<br>Settings:<br> num_points=10000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297101035<br>laszipper wrote 700000000 bytes in 4.79 seconds<br>SUCCESS: lasunzipper read 700000000 bytes in 2.67 seconds<br>laszipper wrote 381625423 bytes in 20.48 seconds<br>SUCCESS: lasunzipper read 381625422 bytes in 23.44 seconds<br>Finished 1 runs<br><br><br>real 0m52.201s<br>user 0m47.626s<br>sys 0m4.031s<br>[dburken@tazz build]$ time laszippertest -n 10000000 -s<br>Settings:<br> num_points=10000000, use_iostream=true, run_forever=false, use_random=false, user_seed=1297101129<br>laszipper wrote 700000000 bytes in 6.81 seconds<br>SUCCESS: lasunzipper read 700000000 bytes in 4.76 seconds<br>laszipper wrote 381625423 bytes in 21.66 seconds<br>SUCCESS: lasunzipper read 381625422 bytes in 28.8 seconds<br>Finished 1 runs<br><br><br>real 1m2.768s<br>user 0m55.831s<br>sys 0m6.590s<br>[dburken@tazz build]$ time laszippertest -n 10000000 -s<br>Settings:<br> num_points=10000000, use_iostream=true, run_forever=false, use_random=false, user_seed=1297101209<br>laszipper wrote 700000000 bytes in 5.09 seconds<br>SUCCESS: lasunzipper read 700000000 bytes in 3.53 seconds<br>laszipper wrote 381625423 bytes in 17.39 seconds<br>SUCCESS: lasunzipper read 381625422 bytes in 27.96 seconds<br>Finished 1 runs<br><br><br>real 0m54.278s<br>user 0m49.023s<br>sys 0m5.235s<br>[dburken@tazz build]$ time laszippertest -n 10000000 -s<br>Settings:<br> num_points=10000000, use_iostream=true, run_forever=false, use_random=false, user_seed=1297101272<br>laszipper wrote 700000000 bytes in 5.2 seconds<br>SUCCESS: lasunzipper read 700000000 bytes in 3.58 seconds<br>laszipper wrote 381625423 bytes in 17.02 seconds<br>SUCCESS: lasunzipper read 381625422 bytes in 24.59 seconds<br>Finished 1 runs<br><br><br>real 0m50.899s<br>user 0m45.558s<br>sys 0m5.113s<br>[dburken@tazz build]$ time laszippertest -n 1000000<br>Settings:<br> num_points=1000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297101353<br>laszipper wrote 70000000 bytes in 0.47 seconds<br>SUCCESS: lasunzipper read 70000000 bytes in 0.27 seconds<br>laszipper wrote 38271321 bytes in 1.55 seconds<br>SUCCESS: lasunzipper read 38271321 bytes in 2.45 seconds<br>Finished 1 runs<br><br><br>real 0m5.060s<br>user 0m4.435s<br>sys 0m0.594s<br>[dburken@tazz build]$ time laszippertest -n 1000000 -s<br>Settings:<br> num_points=1000000, use_iostream=true, run_forever=false, use_random=false, user_seed=1297101361<br>laszipper wrote 70000000 bytes in 0.5 seconds<br>SUCCESS: lasunzipper read 70000000 bytes in 0.36 seconds<br>laszipper wrote 38271321 bytes in 1.69 seconds<br>SUCCESS: lasunzipper read 38271321 bytes in 2.48 seconds<br>Finished 1 runs<br><br><br><br><br>On 02/07/2011 12:02 PM, Michael P. Gerlek wrote: <o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Your box is too fast. :-(</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Could you maybe do the runs again, but with “–n 100000000” (100x larger)?</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>-mpg</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> </span><o:p></o:p></p><div style='border:none;border-left:solid windowtext 1.5pt;padding:0in 0in 0in 4.0pt;border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color blue'><div><div style='border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:-moz-use-text-color -moz-use-text-color'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Smith, Michael ERDC-CRREL-NH [<a href="mailto:michael.smith@usace.army.mil">mailto:michael.smith@usace.army.mil</a>] <br><b>Sent:</b> Saturday, February 05, 2011 4:48 AM<br><b>To:</b> <a href="mailto:mpg@flaxen.com">mpg@flaxen.com</a>; <a href="mailto:liblas-devel@lists.osgeo.org">liblas-devel@lists.osgeo.org</a><br><b>Subject:</b> Re: [Liblas-devel] I/O performance? -- your help requested!</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>lidar@lidarora1 tmp]$ uname -a<br>Linux lidarora1 2.6.32-100.0.19.el5 #1 SMP Fri Sep 17 17:51:41 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux<br><br>2 CPU, 4cores/cpu + hyperthreading = 16 virtual cores <br><br>processor : 15<br>vendor_id : GenuineIntel<br>cpu family : 6<br>model : 26<br>model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz<br>stepping : 5<br>cpu MHz : 1600.000<br>cache size : 8192 KB<br><br><br><br>[lidar@lidarora1 tmp]$ time laszippertest -n 1000000<br>(skipping range coder test)<br>laszipper1 wrote 7000000 bytes in 0.03 seconds<br>laszipper2 wrote 3898142 bytes in 0.13 seconds<br>SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds<br>SUCCESS: lasunzipper2 read 3898142 bytes in 0.18 seconds<br><br>real 0m0.379s<br>user 0m0.349s<br>sys 0m0.029s<br>[lidar@lidarora1 tmp]$ time laszippertest -n 1000000<br>(skipping range coder test)<br>laszipper1 wrote 7000000 bytes in 0.03 seconds<br>laszipper2 wrote 3898142 bytes in 0.13 seconds<br>SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds<br>SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds<br><br>real 0m0.366s<br>user 0m0.336s<br>sys 0m0.029s<br>[lidar@lidarora1 tmp]$ time laszippertest -n 1000000<br>(skipping range coder test)<br>laszipper1 wrote 7000000 bytes in 0.03 seconds<br>laszipper2 wrote 3898142 bytes in 0.13 seconds<br>SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds<br>SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds<br><br>real 0m0.366s<br>user 0m0.338s<br>sys 0m0.026s<br>[lidar@lidarora1 tmp]$ time laszippertest -n 1000000 -s<br>(skipping range coder test)<br>laszipper1 wrote 7000000 bytes in 0.03 seconds<br>laszipper2 wrote 3898142 bytes in 0.13 seconds<br>SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds<br>SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds<br><br>real 0m0.364s<br>user 0m0.338s<br>sys 0m0.024s<br>[lidar@lidarora1 tmp]$ time laszippertest -n 1000000 -s<br>(skipping range coder test)<br>laszipper1 wrote 7000000 bytes in 0.03 seconds<br>laszipper2 wrote 3898142 bytes in 0.13 seconds<br>SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds<br>SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds<br><br>real 0m0.369s<br>user 0m0.339s<br>sys 0m0.028s<br>[lidar@lidarora1 tmp]$ time laszippertest -n 1000000 -s<br>(skipping range coder test)<br>laszipper1 wrote 7000000 bytes in 0.03 seconds<br>laszipper2 wrote 3898142 bytes in 0.13 seconds<br>SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds<br>SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds<br><br>real 0m0.364s<br>user 0m0.335s<br>sys 0m0.027s<br><br><br><br>-- <br>Michael Smith<br>Remote Sensing/GIS Center<br>US Army Corps of Engineers<br><br><br>On 2/4/11 11:39 AM, "Michael P. Gerlek" <<a href="mpg@flaxen.com">mpg@flaxen.com</a>> wrote:</span><o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Do you have 5-10 minutes to spare today?<br><br>Your libLAS team (well, me anyway) is wondering about I/O performance of the<br>liblas kit -- specifically, when doing binary reading and writing, is there<br>any fundamental performance difference between using C-style FILE* I/O and<br>C++-style stream I/O? And if streams are better, would boost's stream be<br>better still? If you google around a bit, you'll find lots of contradictory<br>(and sometimes overly passionate) statements about this topic. At the end<br>of the day, though, the consensus seems to be that:<br><br> (1) you need to be "smart" if you're using C++ I/O -- it is easy to shoot<br>yourself in the foot<br> (2) modern C++ streams are implemented on top of the native OS APIs<br> (3) under Visual Studio, FILE* operations and streams are both implemented<br>using the win32 APIs, but streams have an additional lock (that is claimed<br>by some to be not needed)<br><br>and, most importantly,<br><br> (4) performance varies greatly with different I/O patterns, e.g. large<br>sequential block reads vs small random reads<br><br>Very fortunately, we happen to already have a rough, 1st-order I/O<br>performance test built into the laszip tree. If you have that tree built<br>(<a href="http://hg.liblas.org/zip">http://hg.liblas.org/zip</a>), in Release mode, could you please send me the<br>results of running the "laszippertest" test app, as follows?<br><br> time ./laszippertest -n 1000000<br> time ./laszippertest -n 1000000<br> time ./laszippertest -n 1000000<br> time ./laszippertest -n 1000000 -s<br> time ./laszippertest -n 1000000 -s<br> time ./laszippertest -n 1000000 -s<br><br>The first three runs will encode and decode 1 million random points using<br>FILEs, and the second three will do it with streams. This is not a perfect<br>test, but it represents something approximating the real I/O footprint or<br>traces that liblas uses.<br><br>Oh, and be sure to include the kind of platform (processor speed, compiler,<br>OS) you're running it on.<br><br>Thanks much!<br><br>-mpg<br><br><br>_______________________________________________<br>Liblas-devel mailing list<br><a href="Liblas-devel@lists.osgeo.org">Liblas-devel@lists.osgeo.org</a><br><a href="http://lists.osgeo.org/mailman/listinfo/liblas-devel">http://lists.osgeo.org/mailman/listinfo/liblas-devel</a></span><o:p></o:p></p></div><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre>_______________________________________________<o:p></o:p></pre><pre>Liblas-devel mailing list<o:p></o:p></pre><pre><a href="mailto:Liblas-devel@lists.osgeo.org">Liblas-devel@lists.osgeo.org</a><o:p></o:p></pre><pre><a href="http://lists.osgeo.org/mailman/listinfo/liblas-devel">http://lists.osgeo.org/mailman/listinfo/liblas-devel</a><o:p></o:p></pre></div></div></body></html>