[gdal-dev] GDAL + Docker with NFS

[Ouwen Huang]

I feel quite silly, but it seems that the `gdal.Open` was actually opening
the file from the docker mounted NFS, just very slowly. I left it to run
for 2 minutes or so and it returned.

But this is weird behavior because on the host a `*Dataset` was returned
instantly, and within the docker container running `cp
/mnt/efs/<my_geo.tiff>  <docker_local_dir` is also instant. The 2min
latency seems to only occur within a docker mounted network filesystem for
the `gdal.Open` command.

Some other findings, the `/mnt/efs/` directory i'm using has several files
(100k+). When I move the geotiff under a new directory within EFS
(`mnt/efs/test`), the `gdal.Open` command is instant within the docker
container.

Is there a reason why `gdal.Open` is slow for a cluttered directory but
unix `cp` is not?

On Sun, Mar 18, 2018 at 2:51 AM Ouwen Huang <ouwen.oh at gmail.com> wrote:

> Hey Michael,
>
> Thanks for the quick response, I've been bashing my head against this for
> a while.
> Unfortunately restarting the docker daemon did not work for me. I can
> access the EFS from within docker using regular unix commands like `cp`,
> and `cat`, even with GDAL `VSIFOpenL` and `VSIFReadL`. However,
> `gdal.Open` will just hang without giving an error or returning.
>
> I'm using the following mount command from amazon EFS if this helps debug
> anything:
> `sudo mount -t nfs4 -o
> nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2
> <fs-id>.efs.us-west-2.amazonaws.com:/ /mnt/efs`
>
> I'm glad that you have a working version of this though, this had worked
> for me in the past but I did not keep track of my versioning.
>
> What host os version, docker version and gdal version are you using?
>
> - Ouwen
>
> On Sun, Mar 18, 2018 at 2:31 AM Michael Smith <
> michael.smith.erdc at gmail.com> wrote:
>
>> Have you restarted the docker daemon after mounting the EFS drive to the
>> local OS? If not, do that and then map the volume with -v. That worked for
>> me.
>>
>> Michael Smith
>> Remote Sensing/GIS Center
>> US Army Corps of Engineers
>>
>> > On Mar 18, 2018, at 7:15 AM, Ouwen Huang <ouwen.oh at gmail.com> wrote:
>> >
>> > Hello,
>> >
>> > I am having some trouble loading a GeoTiff into GDAL from a docker
>> container that is mounting a NFS (specifically amazon EFS). When mounting a
>> volume with the `docker -v` command or mounting the NFS directly into
>> docker under `--privileged` mode. The `gdal.Open` command hangs.
>> >
>> > I am able to run os level commands to open the GeoTiff of interest from
>> within the docker container. In fact, if I copy from the NFS to a local
>> directory, gdal works fine.
>> >
>> > I'm trying to understand what might be causing the `gdal.Open` command
>> to hang. I've tested that `VSIFOpenL` and `VSIFReadL` work with the GeoTiff
>> within the docker container so I'm very confused what is happening.
>> >
>> > I've tried different versions of GDAL (1.10 - 2.3) and differing
>> versions of docker with no luck.
>> >
>> > Any help would be greatly appreciated!
>> >
>> > Best,
>> > Ouwen Huang
>> > _______________________________________________
>> > gdal-dev mailing list
>> > gdal-dev at lists.osgeo.org
>> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
> --
> Duke MD/PhD Candidate
> Ycombinator Fellow
> cell: 210.860.3915 <(210)%20860-3915>
> http://ouwen.io/
>
-- 
Duke MD/PhD Candidate
Ycombinator Fellow
cell: 210.860.3915
http://ouwen.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20180318/f9977e1b/attachment-0001.html>