<html><head></head><body><div class="ydpd98c45cayahoo-style-wrap" style="font-family:verdana, helvetica, sans-serif;font-size:16px;"><div></div>
<div dir="ltr" data-setdir="false">Ummm...</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">How much is due to the very different version of GDAL vs docker overhead? <br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Any chance of using the same GDAL version for both tests?<br></div><div><br></div><div><br></div><div dir="ltr" data-setdir="false">Brent Wood<br></div><div><br></div>
</div><div id="ydp13ca23d6yahoo_quoted_4771880266" class="ydp13ca23d6yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Wednesday, November 20, 2019, 11:20:50 AM GMT+13, jratike80 <jukka.rahkonen@maanmittauslaitos.fi> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div dir="ltr">Hi,<br clear="none"><br clear="none">I have been told that Docker has overhead with everything, with these<br clear="none">numbers as evidence:<br clear="none"><br clear="none"><a shape="rect" href="mailto:ubuntu@t" rel="nofollow" target="_blank">ubuntu@t</a>:/etc$ time gdalinfo --version<br clear="none">GDAL 2.2.3, released 2017/11/20<br clear="none"><br clear="none">real 0m0.030s<br clear="none">user 0m0.017s<br clear="none">sys 0m0.013s<br clear="none"><br clear="none"><a shape="rect" href="mailto:ubuntu@t" rel="nofollow" target="_blank">ubuntu@t</a>:/etc$ time docker exec -it 742a044b8ef1 gdalinfo --version<br clear="none">GDAL 3.1.0dev-19f34295c8c310b6ecb27d2d7aa7ac0a2a2a3d9f, released 2019/11/05<br clear="none"><br clear="none">real 0m0.386s<br clear="none">user 0m0.023s<br clear="none">sys 0m0.030s<br clear="none"><br clear="none">-Jukka Rahkonen-<br clear="none"><br clear="none"><br clear="none">alex-6 wrote<br clear="none">> Hi guys,<br clear="none">> <br clear="none">> I've been playing quite a lot with "wrapper_GDALWarpDestName" and a .net <br clear="none">> core wrapper (<a shape="rect" href="https://github.com/MaxRev-Dev/gdal.netcore/" rel="nofollow" target="_blank">https://github.com/MaxRev-Dev/gdal.netcore/</a>).<br clear="none">> Everything is running fine, I'm quite happy to see the same behavior on <br clear="none">> windows and linux. Sometimes linux is faster for reading and writing <br clear="none">> raster, this is great ! .NET Core is (now) a very good option for cross <br clear="none">> platform dev. Anyway...<br clear="none">> <br clear="none">> When calling wrapper_GDALWarpDestName, I'm experiencing an overhead of <br clear="none">> 350 to 500 ms on linux that I don't see at all on Windows.<br clear="none">> To narrow down the issue, I tried many warp options (-multi; -wo <br clear="none">> NUM_THREADS=4; -wm 2000), output drivers (MEM, VRT, NWT_GRD), input file <br clear="none">> format (MEM, VRT, NWT_GRD), input and output projections. I also checked <br clear="none">> Gdal.GetCacheUsed(), Gdal.HasThreadSupport() and Gdal.GetCacheMax(), <br clear="none">> same values between linux and windows. Nothing helps.<br clear="none">> <br clear="none">> Even with VRT format output, this overhead is present which proves this <br clear="none">> is not related to processing. I bet for some sort of initialization of <br clear="none">> the warp algorithm with file reading... because 400 ms is far from <br clear="none">> nothing. But I'm wondering why this is occurring for every single call <br clear="none">> on linux and not on windows... Any option I could try ? My next step is <br clear="none">> to profile the linux process but I'm not good at that so any help there <br clear="none">> would be great too.<br clear="none">> <br clear="none">> <br clear="none">> OS (version): Windows 10 (1903)<br clear="none">> Docker:<br clear="none">> Server: Docker Engine - Community<br clear="none">> Engine:<br clear="none">> Version: 19.03.2<br clear="none">> API version: 1.40 (minimum version 1.12)<br clear="none">> Go version: go1.12.8<br clear="none">> Git commit: 6a30dfc<br clear="none">> Built: Thu Aug 29 05:32:21 2019<br clear="none">> OS/Arch: linux/amd64<br clear="none">> Experimental: false<br clear="none">> <br clear="none">> Linux in docker file: debian 9 <br clear="none">> (mcr.microsoft.com/dotnet/core/runtime:2.2-stretch-slim)<br clear="none">> Package version (core): [e.g. 3.0.1.25]<br clear="none">> Package version (runtime windows): [e.g. 3.0.1.25]<br clear="none">> Package version (runtime linux): [e.g. 3.0.1.2]<br clear="none">> <br clear="none">> This is the kind of code I'm running:<br clear="none">> <br clear="none">> namespace GdalCoreTest<br clear="none">> {<br clear="none">> class Program<br clear="none">> {<br clear="none">> static void Main(string[] args)<br clear="none">> {<br clear="none">> GdalBase.ConfigureAll();<br clear="none">> Console.WriteLine("GDAL configured");<br clear="none">> Console.WriteLine(string.Join('\n',<br clear="none">> "GDAL Version: " + Gdal.VersionInfo("RELEASE_NAME"),<br clear="none">> "GDAL INFO: " + Gdal.VersionInfo("")));<br clear="none">> WarpFile();<br clear="none">> WarpFile();<br clear="none">> WarpFile();<br clear="none">> }<br clear="none">> <br clear="none">> public static void WarpFile()<br clear="none">> {<br clear="none">> Stopwatch sw = new Stopwatch();<br clear="none">> <br clear="none">> int sizeX = 1024, sizeY = 1024;<br clear="none">> var driver = Gdal.GetDriverByName("MEM");<br clear="none">> string path = $"/vsimem/test.mem";<br clear="none">> string vrtFilePath = null;<br clear="none">> using(Dataset ds = driver.Create(path, sizeX, sizeY, 1, <br clear="none">> DataType.GDT_UInt32, new string[0]))<br clear="none">> {<br clear="none">> try<br clear="none">> {<br clear="none">> //**************<br clear="none">> //create a grid<br clear="none">> ds.SetGeoTransform(new double[]{0, 30, 0, 0, 0, -30});<br clear="none">> <br clear="none">> var ttt = new SpatialReference(null);<br clear="none">> ttt.ImportFromProj4("+proj=longlat +datum=WGS84");<br clear="none">> ttt.ExportToWkt(out string latLongWkt, new string[0]);<br clear="none">> ds.SetProjection(latLongWkt);<br clear="none">> <br clear="none">> //write data in the buffer<br clear="none">> int[] buffer = new int[sizeX * sizeY];<br clear="none">> for (int x = 0; x < sizeX; x++)<br clear="none">> {<br clear="none">> for (int y = 0; y < sizeY; y++)<br clear="none">> {<br clear="none">> buffer[y * sizeX + x] = x + y;<br clear="none">> }<br clear="none">> }<br clear="none">> <br clear="none">> ds.GetRasterBand(1).WriteRaster(0, 0, sizeX, sizeY, buffer, sizeX, <br clear="none">> sizeY, 0, 0);<br clear="none">> <br clear="none">> <br clear="none">> //**************<br clear="none">> //Prepare for warping<br clear="none">> IntPtr[] ptr = {Dataset.getCPtr(ds).Handle};<br clear="none">> GCHandle gcHandle = GCHandle.Alloc(ptr, GCHandleType.Pinned);<br clear="none">> var dss = new <br clear="none">> SWIGTYPE_p_p_GDALDatasetShadow(gcHandle.AddrOfPinnedObject(), false, <br clear="none">> null);<br clear="none">> vrtFilePath = $"/vsimem/tile.vrt";<br clear="none">> <br clear="none">> //Warp<br clear="none">> sw.Start();<br clear="none">> using (Dataset dst = Gdal.wrapper_GDALWarpDestName(vrtFilePath, 1, <br clear="none">> dss, new GDALWarpAppOptions(new[]{"-of", "VRT" }), null,null))<br clear="none">> {<br clear="none">> sw.Stop();<br clear="none">> }<br clear="none">> }<br clear="none">> finally<br clear="none">> {<br clear="none">> Gdal.Unlink(path);<br clear="none">> Gdal.Unlink(vrtFilePath);<br clear="none">> }<br clear="none">> }<br clear="none">> Console.Out.WriteLine($"Warp duration = {sw.ElapsedTicks} ticks <br clear="none">> ({sw.ElapsedMilliseconds} ms).");<br clear="none">> }<br clear="none">> }<br clear="none">> }<br clear="none">> <br clear="none">> <br clear="none">> This is the output I'm getting on windows:<br clear="none">> <br clear="none">> GDAL configured<br clear="none">> GDAL Version: 3.0.1<br clear="none">> GDAL INFO: GDAL 3.0.1, released 2019/06/28<br clear="none">> Warp duration = 221267 ticks (22 ms).<br clear="none">> Warp duration = 133395 ticks (13 ms).<br clear="none">> Warp duration = 155402 ticks (15 ms).<br clear="none">> <br clear="none">> This is the output I'm getting on docker/linux:<br clear="none">> <br clear="none">> GDAL configured<br clear="none">> GDAL Version: 3.0.1<br clear="none">> GDAL INFO: GDAL 3.0.1, released 2019/06/28<br clear="none">> Warp duration = 395911725 ticks (395 ms).<br clear="none">> Warp duration = 368923383 ticks (368 ms).<br clear="none">> Warp duration = 373055090 ticks (373 ms).<br clear="none">> <br clear="none">> Thanks,<br clear="none">> Alex.<br clear="none">> _______________________________________________<br clear="none">> gdal-dev mailing list<br clear="none"><br clear="none">> <a shape="rect" href="mailto:gdal-dev@.osgeo" rel="nofollow" target="_blank">gdal-dev@.osgeo</a><br clear="none"><br clear="none">> <a shape="rect" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="nofollow" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br clear="none"><br clear="none"><br clear="none"><br clear="none"><br clear="none"><br clear="none">--<br clear="none">Sent from: <a shape="rect" href="http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html" rel="nofollow" target="_blank">http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html</a><div class="ydp13ca23d6yqt8183077292" id="ydp13ca23d6yqtfd29434"><br clear="none">_______________________________________________<br clear="none">gdal-dev mailing list<br clear="none"><a shape="rect" href="mailto:gdal-dev@lists.osgeo.org" rel="nofollow" target="_blank">gdal-dev@lists.osgeo.org</a><br clear="none"><a shape="rect" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="nofollow" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a></div></div></div>
</div>
</div></body></html>