[gdal-dev] gdalwarp overhead on linux but not on windows
Brent Wood
pcreso at yahoo.com
Tue Nov 19 15:11:48 PST 2019
Ummm...
How much is due to the very different version of GDAL vs docker overhead?
Any chance of using the same GDAL version for both tests?
Brent Wood
On Wednesday, November 20, 2019, 11:20:50 AM GMT+13, jratike80 <jukka.rahkonen at maanmittauslaitos.fi> wrote:
Hi,
I have been told that Docker has overhead with everything, with these
numbers as evidence:
ubuntu at t:/etc$ time gdalinfo --version
GDAL 2.2.3, released 2017/11/20
real 0m0.030s
user 0m0.017s
sys 0m0.013s
ubuntu at t:/etc$ time docker exec -it 742a044b8ef1 gdalinfo --version
GDAL 3.1.0dev-19f34295c8c310b6ecb27d2d7aa7ac0a2a2a3d9f, released 2019/11/05
real 0m0.386s
user 0m0.023s
sys 0m0.030s
-Jukka Rahkonen-
alex-6 wrote
> Hi guys,
>
> I've been playing quite a lot with "wrapper_GDALWarpDestName" and a .net
> core wrapper (https://github.com/MaxRev-Dev/gdal.netcore/).
> Everything is running fine, I'm quite happy to see the same behavior on
> windows and linux. Sometimes linux is faster for reading and writing
> raster, this is great ! .NET Core is (now) a very good option for cross
> platform dev. Anyway...
>
> When calling wrapper_GDALWarpDestName, I'm experiencing an overhead of
> 350 to 500 ms on linux that I don't see at all on Windows.
> To narrow down the issue, I tried many warp options (-multi; -wo
> NUM_THREADS=4; -wm 2000), output drivers (MEM, VRT, NWT_GRD), input file
> format (MEM, VRT, NWT_GRD), input and output projections. I also checked
> Gdal.GetCacheUsed(), Gdal.HasThreadSupport() and Gdal.GetCacheMax(),
> same values between linux and windows. Nothing helps.
>
> Even with VRT format output, this overhead is present which proves this
> is not related to processing. I bet for some sort of initialization of
> the warp algorithm with file reading... because 400 ms is far from
> nothing. But I'm wondering why this is occurring for every single call
> on linux and not on windows... Any option I could try ? My next step is
> to profile the linux process but I'm not good at that so any help there
> would be great too.
>
>
> OS (version): Windows 10 (1903)
> Docker:
> Server: Docker Engine - Community
> Engine:
> Version: 19.03.2
> API version: 1.40 (minimum version 1.12)
> Go version: go1.12.8
> Git commit: 6a30dfc
> Built: Thu Aug 29 05:32:21 2019
> OS/Arch: linux/amd64
> Experimental: false
>
> Linux in docker file: debian 9
> (mcr.microsoft.com/dotnet/core/runtime:2.2-stretch-slim)
> Package version (core): [e.g. 3.0.1.25]
> Package version (runtime windows): [e.g. 3.0.1.25]
> Package version (runtime linux): [e.g. 3.0.1.2]
>
> This is the kind of code I'm running:
>
> namespace GdalCoreTest
> {
> class Program
> {
> static void Main(string[] args)
> {
> GdalBase.ConfigureAll();
> Console.WriteLine("GDAL configured");
> Console.WriteLine(string.Join('\n',
> "GDAL Version: " + Gdal.VersionInfo("RELEASE_NAME"),
> "GDAL INFO: " + Gdal.VersionInfo("")));
> WarpFile();
> WarpFile();
> WarpFile();
> }
>
> public static void WarpFile()
> {
> Stopwatch sw = new Stopwatch();
>
> int sizeX = 1024, sizeY = 1024;
> var driver = Gdal.GetDriverByName("MEM");
> string path = $"/vsimem/test.mem";
> string vrtFilePath = null;
> using(Dataset ds = driver.Create(path, sizeX, sizeY, 1,
> DataType.GDT_UInt32, new string[0]))
> {
> try
> {
> //**************
> //create a grid
> ds.SetGeoTransform(new double[]{0, 30, 0, 0, 0, -30});
>
> var ttt = new SpatialReference(null);
> ttt.ImportFromProj4("+proj=longlat +datum=WGS84");
> ttt.ExportToWkt(out string latLongWkt, new string[0]);
> ds.SetProjection(latLongWkt);
>
> //write data in the buffer
> int[] buffer = new int[sizeX * sizeY];
> for (int x = 0; x < sizeX; x++)
> {
> for (int y = 0; y < sizeY; y++)
> {
> buffer[y * sizeX + x] = x + y;
> }
> }
>
> ds.GetRasterBand(1).WriteRaster(0, 0, sizeX, sizeY, buffer, sizeX,
> sizeY, 0, 0);
>
>
> //**************
> //Prepare for warping
> IntPtr[] ptr = {Dataset.getCPtr(ds).Handle};
> GCHandle gcHandle = GCHandle.Alloc(ptr, GCHandleType.Pinned);
> var dss = new
> SWIGTYPE_p_p_GDALDatasetShadow(gcHandle.AddrOfPinnedObject(), false,
> null);
> vrtFilePath = $"/vsimem/tile.vrt";
>
> //Warp
> sw.Start();
> using (Dataset dst = Gdal.wrapper_GDALWarpDestName(vrtFilePath, 1,
> dss, new GDALWarpAppOptions(new[]{"-of", "VRT" }), null,null))
> {
> sw.Stop();
> }
> }
> finally
> {
> Gdal.Unlink(path);
> Gdal.Unlink(vrtFilePath);
> }
> }
> Console.Out.WriteLine($"Warp duration = {sw.ElapsedTicks} ticks
> ({sw.ElapsedMilliseconds} ms).");
> }
> }
> }
>
>
> This is the output I'm getting on windows:
>
> GDAL configured
> GDAL Version: 3.0.1
> GDAL INFO: GDAL 3.0.1, released 2019/06/28
> Warp duration = 221267 ticks (22 ms).
> Warp duration = 133395 ticks (13 ms).
> Warp duration = 155402 ticks (15 ms).
>
> This is the output I'm getting on docker/linux:
>
> GDAL configured
> GDAL Version: 3.0.1
> GDAL INFO: GDAL 3.0.1, released 2019/06/28
> Warp duration = 395911725 ticks (395 ms).
> Warp duration = 368923383 ticks (368 ms).
> Warp duration = 373055090 ticks (373 ms).
>
> Thanks,
> Alex.
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at .osgeo
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20191119/3cd84718/attachment-0001.html>
More information about the gdal-dev
mailing list