[gdal-dev] gdalwarp overhead on linux but not on windows

[alex at vautier.biz]

Hi guys,

I've been playing quite a lot with "wrapper_GDALWarpDestName" and a .net 
core wrapper (https://github.com/MaxRev-Dev/gdal.netcore/).
Everything is running fine, I'm quite happy to see the same behavior on 
windows and linux. Sometimes linux is faster for reading and writing 
raster, this is great ! .NET Core is (now) a very good option for cross 
platform dev. Anyway...

When calling wrapper_GDALWarpDestName,  I'm experiencing an overhead of 
350 to 500 ms on linux that I don't see at all on Windows.
To narrow down the issue, I tried many warp options (-multi; -wo 
NUM_THREADS=4; -wm 2000), output drivers (MEM, VRT, NWT_GRD), input file 
format (MEM, VRT, NWT_GRD), input and output projections. I also checked 
Gdal.GetCacheUsed(), Gdal.HasThreadSupport() and Gdal.GetCacheMax(), 
same values between linux and windows. Nothing helps.

Even with VRT format output, this overhead is present which proves this 
is not related to processing. I bet for some sort of initialization of 
the warp algorithm with file reading... because 400 ms is far from 
nothing. But I'm wondering why this is occurring for every single call 
on linux and not on windows... Any option I could try ? My next step is 
to profile the linux process but I'm not good at that so any help there 
would be great too.


     OS (version): Windows 10 (1903)
     Docker:
     Server: Docker Engine - Community
     Engine:
     Version: 19.03.2
     API version: 1.40 (minimum version 1.12)
     Go version: go1.12.8
     Git commit: 6a30dfc
     Built: Thu Aug 29 05:32:21 2019
     OS/Arch: linux/amd64
     Experimental: false

     Linux in docker file: debian 9 
(mcr.microsoft.com/dotnet/core/runtime:2.2-stretch-slim)
     Package version (core): [e.g. 3.0.1.25]
     Package version (runtime windows): [e.g. 3.0.1.25]
     Package version (runtime linux): [e.g. 3.0.1.2]

This is the kind of code I'm running:

namespace GdalCoreTest
{
  class Program
  {
   static void Main(string[] args)
   {
	GdalBase.ConfigureAll();
	Console.WriteLine("GDAL configured");
	Console.WriteLine(string.Join('\n',
		"GDAL Version: " + Gdal.VersionInfo("RELEASE_NAME"),
		"GDAL INFO: " + Gdal.VersionInfo("")));
	WarpFile();
	WarpFile();
	WarpFile();
   }

   public static void WarpFile()
   {
	Stopwatch sw = new Stopwatch();

	int sizeX = 1024, sizeY = 1024;
	var driver = Gdal.GetDriverByName("MEM");
	string path = $"/vsimem/test.mem";
	string vrtFilePath = null;
	using(Dataset ds = driver.Create(path, sizeX, sizeY, 1, 
DataType.GDT_UInt32, new string[0]))
	{
	  try
	  {
	  //**************
	  //create a grid
		ds.SetGeoTransform(new double[]{0, 30, 0, 0, 0, -30});

		var ttt = new SpatialReference(null);
		ttt.ImportFromProj4("+proj=longlat +datum=WGS84");
		ttt.ExportToWkt(out string latLongWkt, new string[0]);
		ds.SetProjection(latLongWkt);

		//write data in the buffer
		int[] buffer = new int[sizeX * sizeY];
		for (int x = 0; x < sizeX; x++)
		{
		  for (int y = 0; y < sizeY; y++)
		  {
			buffer[y * sizeX + x] = x + y;
		  }
		}

		ds.GetRasterBand(1).WriteRaster(0, 0, sizeX, sizeY, buffer, sizeX, 
sizeY, 0, 0);


		//**************
		//Prepare for warping
		IntPtr[] ptr = {Dataset.getCPtr(ds).Handle};
		GCHandle gcHandle = GCHandle.Alloc(ptr, GCHandleType.Pinned);
		var dss = new 
SWIGTYPE_p_p_GDALDatasetShadow(gcHandle.AddrOfPinnedObject(), false, 
null);
		vrtFilePath = $"/vsimem/tile.vrt";

	  //Warp
		sw.Start();
		using (Dataset dst = Gdal.wrapper_GDALWarpDestName(vrtFilePath, 1, 
dss, new GDALWarpAppOptions(new[]{"-of", "VRT" }), null,null))
		{
		  sw.Stop();
		}
	  }
	  finally
	  {
		Gdal.Unlink(path);
		Gdal.Unlink(vrtFilePath);
	  }
	}
     Console.Out.WriteLine($"Warp duration = {sw.ElapsedTicks} ticks 
({sw.ElapsedMilliseconds} ms).");
   }
  }
}


This is the output I'm getting on windows:

GDAL configured
GDAL Version: 3.0.1
GDAL INFO: GDAL 3.0.1, released 2019/06/28
Warp duration = 221267 ticks (22 ms).
Warp duration = 133395 ticks (13 ms).
Warp duration = 155402 ticks (15 ms).

This is the output I'm getting on docker/linux:

GDAL configured
GDAL Version: 3.0.1
GDAL INFO: GDAL 3.0.1, released 2019/06/28
Warp duration = 395911725 ticks (395 ms).
Warp duration = 368923383 ticks (368 ms).
Warp duration = 373055090 ticks (373 ms).

Thanks,
Alex.