[gdal-dev] gdalwarp overhead on linux but not on windows

[jratike80]

Hi,

I have been told that Docker has overhead with everything, with these
numbers as evidence:

ubuntu at t:/etc$ time gdalinfo --version
GDAL 2.2.3, released 2017/11/20

real 0m0.030s
user 0m0.017s
sys 0m0.013s

ubuntu at t:/etc$ time docker exec -it 742a044b8ef1 gdalinfo --version
GDAL 3.1.0dev-19f34295c8c310b6ecb27d2d7aa7ac0a2a2a3d9f, released 2019/11/05

real 0m0.386s
user 0m0.023s
sys 0m0.030s

-Jukka Rahkonen-


alex-6 wrote
> Hi guys,
> 
> I've been playing quite a lot with "wrapper_GDALWarpDestName" and a .net 
> core wrapper (https://github.com/MaxRev-Dev/gdal.netcore/).
> Everything is running fine, I'm quite happy to see the same behavior on 
> windows and linux. Sometimes linux is faster for reading and writing 
> raster, this is great ! .NET Core is (now) a very good option for cross 
> platform dev. Anyway...
> 
> When calling wrapper_GDALWarpDestName,  I'm experiencing an overhead of 
> 350 to 500 ms on linux that I don't see at all on Windows.
> To narrow down the issue, I tried many warp options (-multi; -wo 
> NUM_THREADS=4; -wm 2000), output drivers (MEM, VRT, NWT_GRD), input file 
> format (MEM, VRT, NWT_GRD), input and output projections. I also checked 
> Gdal.GetCacheUsed(), Gdal.HasThreadSupport() and Gdal.GetCacheMax(), 
> same values between linux and windows. Nothing helps.
> 
> Even with VRT format output, this overhead is present which proves this 
> is not related to processing. I bet for some sort of initialization of 
> the warp algorithm with file reading... because 400 ms is far from 
> nothing. But I'm wondering why this is occurring for every single call 
> on linux and not on windows... Any option I could try ? My next step is 
> to profile the linux process but I'm not good at that so any help there 
> would be great too.
> 
> 
>      OS (version): Windows 10 (1903)
>      Docker:
>      Server: Docker Engine - Community
>      Engine:
>      Version: 19.03.2
>      API version: 1.40 (minimum version 1.12)
>      Go version: go1.12.8
>      Git commit: 6a30dfc
>      Built: Thu Aug 29 05:32:21 2019
>      OS/Arch: linux/amd64
>      Experimental: false
> 
>      Linux in docker file: debian 9 
> (mcr.microsoft.com/dotnet/core/runtime:2.2-stretch-slim)
>      Package version (core): [e.g. 3.0.1.25]
>      Package version (runtime windows): [e.g. 3.0.1.25]
>      Package version (runtime linux): [e.g. 3.0.1.2]
> 
> This is the kind of code I'm running:
> 
> namespace GdalCoreTest
> {
>   class Program
>   {
>    static void Main(string[] args)
>    {
> 	GdalBase.ConfigureAll();
> 	Console.WriteLine("GDAL configured");
> 	Console.WriteLine(string.Join('\n',
> 		"GDAL Version: " + Gdal.VersionInfo("RELEASE_NAME"),
> 		"GDAL INFO: " + Gdal.VersionInfo("")));
> 	WarpFile();
> 	WarpFile();
> 	WarpFile();
>    }
> 
>    public static void WarpFile()
>    {
> 	Stopwatch sw = new Stopwatch();
> 
> 	int sizeX = 1024, sizeY = 1024;
> 	var driver = Gdal.GetDriverByName("MEM");
> 	string path = $"/vsimem/test.mem";
> 	string vrtFilePath = null;
> 	using(Dataset ds = driver.Create(path, sizeX, sizeY, 1, 
> DataType.GDT_UInt32, new string[0]))
> 	{
> 	  try
> 	  {
> 	  //**************
> 	  //create a grid
> 		ds.SetGeoTransform(new double[]{0, 30, 0, 0, 0, -30});
> 
> 		var ttt = new SpatialReference(null);
> 		ttt.ImportFromProj4("+proj=longlat +datum=WGS84");
> 		ttt.ExportToWkt(out string latLongWkt, new string[0]);
> 		ds.SetProjection(latLongWkt);
> 
> 		//write data in the buffer
> 		int[] buffer = new int[sizeX * sizeY];
> 		for (int x = 0; x < sizeX; x++)
> 		{
> 		  for (int y = 0; y < sizeY; y++)
> 		  {
> 			buffer[y * sizeX + x] = x + y;
> 		  }
> 		}
> 
> 		ds.GetRasterBand(1).WriteRaster(0, 0, sizeX, sizeY, buffer, sizeX, 
> sizeY, 0, 0);
> 
> 
> 		//**************
> 		//Prepare for warping
> 		IntPtr[] ptr = {Dataset.getCPtr(ds).Handle};
> 		GCHandle gcHandle = GCHandle.Alloc(ptr, GCHandleType.Pinned);
> 		var dss = new 
> SWIGTYPE_p_p_GDALDatasetShadow(gcHandle.AddrOfPinnedObject(), false, 
> null);
> 		vrtFilePath = $"/vsimem/tile.vrt";
> 
> 	  //Warp
> 		sw.Start();
> 		using (Dataset dst = Gdal.wrapper_GDALWarpDestName(vrtFilePath, 1, 
> dss, new GDALWarpAppOptions(new[]{"-of", "VRT" }), null,null))
> 		{
> 		  sw.Stop();
> 		}
> 	  }
> 	  finally
> 	  {
> 		Gdal.Unlink(path);
> 		Gdal.Unlink(vrtFilePath);
> 	  }
> 	}
>      Console.Out.WriteLine($"Warp duration = {sw.ElapsedTicks} ticks 
> ({sw.ElapsedMilliseconds} ms).");
>    }
>   }
> }
> 
> 
> This is the output I'm getting on windows:
> 
> GDAL configured
> GDAL Version: 3.0.1
> GDAL INFO: GDAL 3.0.1, released 2019/06/28
> Warp duration = 221267 ticks (22 ms).
> Warp duration = 133395 ticks (13 ms).
> Warp duration = 155402 ticks (15 ms).
> 
> This is the output I'm getting on docker/linux:
> 
> GDAL configured
> GDAL Version: 3.0.1
> GDAL INFO: GDAL 3.0.1, released 2019/06/28
> Warp duration = 395911725 ticks (395 ms).
> Warp duration = 368923383 ticks (368 ms).
> Warp duration = 373055090 ticks (373 ms).
> 
> Thanks,
> Alex.
> _______________________________________________
> gdal-dev mailing list

> gdal-dev at .osgeo

> https://lists.osgeo.org/mailman/listinfo/gdal-dev





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html