<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.Shkpostityyli19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">My colleague did make a test about the general overhead of the GDAL docker when GDAL is not involved at all. Overhead in this test seems to be close to 0.4 sec
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#444444;background:white">ubuntu@t:/etc$ time docker exec -it 742a044b8ef1 uname -a</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#444444"><br>
<span style="background:white">Linux 742a044b8ef1 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux</span><br>
<br>
<span style="background:white">real 0m0.399s</span><br>
<span style="background:white">user 0m0.032s</span><br>
<span style="background:white">sys 0m0.023s<o:p></o:p></span></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#444444;background:white"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#444444;background:white">-Jukka Rahkonen-</span><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Lähettäjä:</b> Brent Wood <pcreso@yahoo.com> <br>
<b>Lähetetty:</b> keskiviikko 20. marraskuuta 2019 1.12<br>
<b>Vastaanottaja:</b> gdal-dev@lists.osgeo.org; Rahkonen Jukka (MML) <jukka.rahkonen@maanmittauslaitos.fi><br>
<b>Aihe:</b> Re: [gdal-dev] gdalwarp overhead on linux but not on windows<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif">Ummm...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif">How much is due to the very different version of GDAL vs docker overhead?
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif">Any chance of using the same GDAL version for both tests?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif">Brent Wood<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
</div>
</div>
<div id="ydp13ca23d6yahoo_quoted_4771880266">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">On Wednesday, November 20, 2019, 11:20:50 AM GMT+13, jratike80 <<a href="mailto:jukka.rahkonen@maanmittauslaitos.fi">jukka.rahkonen@maanmittauslaitos.fi</a>>
wrote: <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hi,<br>
<br>
I have been told that Docker has overhead with everything, with these<br>
numbers as evidence:<br>
<br>
<a href="mailto:ubuntu@t" target="_blank">ubuntu@t</a>:/etc$ time gdalinfo --version<br>
GDAL 2.2.3, released 2017/11/20<br>
<br>
real 0m0.030s<br>
user 0m0.017s<br>
sys 0m0.013s<br>
<br>
<a href="mailto:ubuntu@t" target="_blank">ubuntu@t</a>:/etc$ time docker exec -it 742a044b8ef1 gdalinfo --version<br>
GDAL 3.1.0dev-19f34295c8c310b6ecb27d2d7aa7ac0a2a2a3d9f, released 2019/11/05<br>
<br>
real 0m0.386s<br>
user 0m0.023s<br>
sys 0m0.030s<br>
<br>
-Jukka Rahkonen-<br>
<br>
<br>
alex-6 wrote<br>
> Hi guys,<br>
> <br>
> I've been playing quite a lot with "wrapper_GDALWarpDestName" and a .net <br>
> core wrapper (<a href="https://github.com/MaxRev-Dev/gdal.netcore/" target="_blank">https://github.com/MaxRev-Dev/gdal.netcore/</a>).<br>
> Everything is running fine, I'm quite happy to see the same behavior on <br>
> windows and linux. Sometimes linux is faster for reading and writing <br>
> raster, this is great ! .NET Core is (now) a very good option for cross <br>
> platform dev. Anyway...<br>
> <br>
> When calling wrapper_GDALWarpDestName, I'm experiencing an overhead of <br>
> 350 to 500 ms on linux that I don't see at all on Windows.<br>
> To narrow down the issue, I tried many warp options (-multi; -wo <br>
> NUM_THREADS=4; -wm 2000), output drivers (MEM, VRT, NWT_GRD), input file <br>
> format (MEM, VRT, NWT_GRD), input and output projections. I also checked <br>
> Gdal.GetCacheUsed(), Gdal.HasThreadSupport() and Gdal.GetCacheMax(), <br>
> same values between linux and windows. Nothing helps.<br>
> <br>
> Even with VRT format output, this overhead is present which proves this <br>
> is not related to processing. I bet for some sort of initialization of <br>
> the warp algorithm with file reading... because 400 ms is far from <br>
> nothing. But I'm wondering why this is occurring for every single call <br>
> on linux and not on windows... Any option I could try ? My next step is <br>
> to profile the linux process but I'm not good at that so any help there <br>
> would be great too.<br>
> <br>
> <br>
> OS (version): Windows 10 (1903)<br>
> Docker:<br>
> Server: Docker Engine - Community<br>
> Engine:<br>
> Version: 19.03.2<br>
> API version: 1.40 (minimum version 1.12)<br>
> Go version: go1.12.8<br>
> Git commit: 6a30dfc<br>
> Built: Thu Aug 29 05:32:21 2019<br>
> OS/Arch: linux/amd64<br>
> Experimental: false<br>
> <br>
> Linux in docker file: debian 9 <br>
> (mcr.microsoft.com/dotnet/core/runtime:2.2-stretch-slim)<br>
> Package version (core): [e.g. 3.0.1.25]<br>
> Package version (runtime windows): [e.g. 3.0.1.25]<br>
> Package version (runtime linux): [e.g. 3.0.1.2]<br>
> <br>
> This is the kind of code I'm running:<br>
> <br>
> namespace GdalCoreTest<br>
> {<br>
> class Program<br>
> {<br>
> static void Main(string[] args)<br>
> {<br>
> GdalBase.ConfigureAll();<br>
> Console.WriteLine("GDAL configured");<br>
> Console.WriteLine(string.Join('\n',<br>
> "GDAL Version: " + Gdal.VersionInfo("RELEASE_NAME"),<br>
> "GDAL INFO: " + Gdal.VersionInfo("")));<br>
> WarpFile();<br>
> WarpFile();<br>
> WarpFile();<br>
> }<br>
> <br>
> public static void WarpFile()<br>
> {<br>
> Stopwatch sw = new Stopwatch();<br>
> <br>
> int sizeX = 1024, sizeY = 1024;<br>
> var driver = Gdal.GetDriverByName("MEM");<br>
> string path = $"/vsimem/test.mem";<br>
> string vrtFilePath = null;<br>
> using(Dataset ds = driver.Create(path, sizeX, sizeY, 1, <br>
> DataType.GDT_UInt32, new string[0]))<br>
> {<br>
> try<br>
> {<br>
> //**************<br>
> //create a grid<br>
> ds.SetGeoTransform(new double[]{0, 30, 0, 0, 0, -30});<br>
> <br>
> var ttt = new SpatialReference(null);<br>
> ttt.ImportFromProj4("+proj=longlat +datum=WGS84");<br>
> ttt.ExportToWkt(out string latLongWkt, new string[0]);<br>
> ds.SetProjection(latLongWkt);<br>
> <br>
> //write data in the buffer<br>
> int[] buffer = new int[sizeX * sizeY];<br>
> for (int x = 0; x < sizeX; x++)<br>
> {<br>
> for (int y = 0; y < sizeY; y++)<br>
> {<br>
> buffer[y * sizeX + x] = x + y;<br>
> }<br>
> }<br>
> <br>
> ds.GetRasterBand(1).WriteRaster(0, 0, sizeX, sizeY, buffer, sizeX, <br>
> sizeY, 0, 0);<br>
> <br>
> <br>
> //**************<br>
> //Prepare for warping<br>
> IntPtr[] ptr = {Dataset.getCPtr(ds).Handle};<br>
> GCHandle gcHandle = GCHandle.Alloc(ptr, GCHandleType.Pinned);<br>
> var dss = new <br>
> SWIGTYPE_p_p_GDALDatasetShadow(gcHandle.AddrOfPinnedObject(), false, <br>
> null);<br>
> vrtFilePath = $"/vsimem/tile.vrt";<br>
> <br>
> //Warp<br>
> sw.Start();<br>
> using (Dataset dst = Gdal.wrapper_GDALWarpDestName(vrtFilePath, 1, <br>
> dss, new GDALWarpAppOptions(new[]{"-of", "VRT" }), null,null))<br>
> {<br>
> sw.Stop();<br>
> }<br>
> }<br>
> finally<br>
> {<br>
> Gdal.Unlink(path);<br>
> Gdal.Unlink(vrtFilePath);<br>
> }<br>
> }<br>
> Console.Out.WriteLine($"Warp duration = {sw.ElapsedTicks} ticks <br>
> ({sw.ElapsedMilliseconds} ms).");<br>
> }<br>
> }<br>
> }<br>
> <br>
> <br>
> This is the output I'm getting on windows:<br>
> <br>
> GDAL configured<br>
> GDAL Version: 3.0.1<br>
> GDAL INFO: GDAL 3.0.1, released 2019/06/28<br>
> Warp duration = 221267 ticks (22 ms).<br>
> Warp duration = 133395 ticks (13 ms).<br>
> Warp duration = 155402 ticks (15 ms).<br>
> <br>
> This is the output I'm getting on docker/linux:<br>
> <br>
> GDAL configured<br>
> GDAL Version: 3.0.1<br>
> GDAL INFO: GDAL 3.0.1, released 2019/06/28<br>
> Warp duration = 395911725 ticks (395 ms).<br>
> Warp duration = 368923383 ticks (368 ms).<br>
> Warp duration = 373055090 ticks (373 ms).<br>
> <br>
> Thanks,<br>
> Alex.<br>
> _______________________________________________<br>
> gdal-dev mailing list<br>
<br>
> <a href="mailto:gdal-dev@.osgeo" target="_blank">gdal-dev@.osgeo</a><br>
<br>
> <a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
<br>
<br>
<br>
<br>
<br>
--<br>
Sent from: <a href="http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html" target="_blank">
http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html</a><o:p></o:p></span></p>
<div id="ydp13ca23d6yqtfd29434">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>