[gdal-dev] GDAL + Docker with NFS

[Michael Smith]

We haven’t had that behavior, we use gdal.Open on many large files inside docker and this isn’t an issue we’ve had. We haven’t specifically used EFS, we use NFS and an S3 fuse layer with docker and have had very good performance with the python bindings. 

Note that we haven’t used privileged mode though, we aren’t using selinux, and we use the same uid/gid inside the docker container as owns the files. So that might be something to look at. 

Michael Smith
Remote Sensing/GIS Center
US Army Corps of Engineers

> On Mar 18, 2018, at 8:09 AM, Ouwen Huang <ouwen.oh at gmail.com> wrote:
> 
> I feel quite silly, but it seems that the `gdal.Open` was actually opening the file from the docker mounted NFS, just very slowly. I left it to run for 2 minutes or so and it returned.
> 
> But this is weird behavior because on the host a `*Dataset` was returned instantly, and within the docker container running `cp /mnt/efs/<my_geo.tiff>  <docker_local_dir` is also instant. The 2min latency seems to only occur within a docker mounted network filesystem for the `gdal.Open` command.
> 
> Some other findings, the `/mnt/efs/` directory i'm using has several files (100k+). When I move the geotiff under a new directory within EFS (`mnt/efs/test`), the `gdal.Open` command is instant within the docker container.
> 
> Is there a reason why `gdal.Open` is slow for a cluttered directory but unix `cp` is not?
> 
>> On Sun, Mar 18, 2018 at 2:51 AM Ouwen Huang <ouwen.oh at gmail.com> wrote:
>> Hey Michael,
>> 
>> Thanks for the quick response, I've been bashing my head against this for a while.
>> Unfortunately restarting the docker daemon did not work for me. I can access the EFS from within docker using regular unix commands like `cp`, and `cat`, even with GDAL `VSIFOpenL` and `VSIFReadL`. However, `gdal.Open` will just hang without giving an error or returning.
>> 
>> I'm using the following mount command from amazon EFS if this helps debug anything:
>> `sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 <fs-id>.efs.us-west-2.amazonaws.com:/ /mnt/efs`
>> 
>> I'm glad that you have a working version of this though, this had worked for me in the past but I did not keep track of my versioning.
>> 
>> What host os version, docker version and gdal version are you using?
>> 
>> - Ouwen
>> 
>>> On Sun, Mar 18, 2018 at 2:31 AM Michael Smith <michael.smith.erdc at gmail.com> wrote:
>>> Have you restarted the docker daemon after mounting the EFS drive to the local OS? If not, do that and then map the volume with -v. That worked for me.
>>> 
>>> Michael Smith
>>> Remote Sensing/GIS Center
>>> US Army Corps of Engineers
>>> 
>>> > On Mar 18, 2018, at 7:15 AM, Ouwen Huang <ouwen.oh at gmail.com> wrote:
>>> >
>>> > Hello,
>>> >
>>> > I am having some trouble loading a GeoTiff into GDAL from a docker container that is mounting a NFS (specifically amazon EFS). When mounting a volume with the `docker -v` command or mounting the NFS directly into docker under `--privileged` mode. The `gdal.Open` command hangs.
>>> >
>>> > I am able to run os level commands to open the GeoTiff of interest from within the docker container. In fact, if I copy from the NFS to a local directory, gdal works fine.
>>> >
>>> > I'm trying to understand what might be causing the `gdal.Open` command to hang. I've tested that `VSIFOpenL` and `VSIFReadL` work with the GeoTiff within the docker container so I'm very confused what is happening.
>>> >
>>> > I've tried different versions of GDAL (1.10 - 2.3) and differing versions of docker with no luck.
>>> >
>>> > Any help would be greatly appreciated!
>>> >
>>> > Best,
>>> > Ouwen Huang
>>> > _______________________________________________
>>> > gdal-dev mailing list
>>> > gdal-dev at lists.osgeo.org
>>> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>> -- 
>> Duke MD/PhD Candidate
>> Ycombinator Fellow
>> cell: 210.860.3915
>> http://ouwen.io/
> -- 
> Duke MD/PhD Candidate
> Ycombinator Fellow
> cell: 210.860.3915
> http://ouwen.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20180318/e1c85a6c/attachment.html>