[gdal-dev] OGR slow with network files on Windows

[Jonathan Greenberg]

This is an old(ish) thread but I wanted to follow-up since I've been
dealing with SMB2 and GIS stuff recently and came across this in my
searches.  Some of this is specific to reading files from a Samba
server (linux, typically), but modern Windows file servers have a
similar problem.  A couple of notes that we found related to this:

1) oplocks may be a problem.  By disabling oplocks in SMB1, we saw
some of these issues go away.  In smb.conf (ask your network admin
about this):
[global]
...
kernel oplocks = no

[shared folder]
...
        oplocks = no
        level2 oplocks = no
        posix locking = no
        locking = no

(don't forget to restart the server, e.g. sudo service smb restart)

If the files are being shared up by Windows, try:
http://support.microsoft.com/kb/296264 (but note SMB2 can't have
oplocks disabled, see #2 below).

2) oplocks can NOT be disabled on SMB2 as far as I can tell, so there
is no way to fix this problem in SMB2 (or SMB3, AFAIK).  If you have
two Windows machines talking to each other, they are likely to be
using SMB2 (or maybe even SMB3).  You can determine this by using
Wireshark (http://www.wireshark.org/), and capturing when you start
doing work over the network.  If you see a column that says "SMB2" it
is using SMB2 (and oplocks will be running, and likely interfering
with DB-type operations).  If it says "SMB", then it is running SMB v.
1 and, with the right tweaks, you can disable oplocks (see (1) for
linux).

3) It is possible to force a Windows machine to use SMB1 only, but be
aware there are performance hits to non-DB-type processes that will
arise.  http://support.microsoft.com/kb/2696547 gives a good overview
of how to disable it on the client-side.  If you are using a Linux
samba server, set in smb.conf:
[global]
...
max protocol = NT1
(restart smbd via e.g. sudo service smb restart, but I had to actually
reboot the client machine for this to work -- you might get
authentication errors if you don't reboot the client machine).

Hope this helps some future folks!

--j

On Mon, Apr 29, 2013 at 4:07 AM, Radim Blazek <radim.blazek at gmail.com> wrote:
> On Mon, Apr 29, 2013 at 10:55 AM, Even Rouault
> <even.rouault at mines-paris.org> wrote:
>> Selon Radim Blazek <radim.blazek at gmail.com>:
>>
>>> On Mon, Apr 29, 2013 at 10:23 AM, Even Rouault
>>> <even.rouault at mines-paris.org> wrote:
>>> > I'm also wondering if setting unconditionnaly VSI_CACHE might not have some
>>> > theoretical penalty for fast access storage when there is a lot of seeking
>>> and
>>> > reading by small amounts (I believe this is the typical I/O pattern when
>>> you
>>> > display a JPEG2000). It might be relevant to have an option to enable
>>> > caching/buffering for files only located on SMB folders.
>>>
>>> How do you recognize if a file is SMB?
>>
>> I was wondering that too and hoped you would known the answer ;-) There's
>> perhaps a Windows API for that... ?
>
> There seems to be: GetVolumePathName + GetDriveType which should be
> DRIVE_REMOTE.
> maybe also PathIsNetworkPath.
>
> It seems to me that VSI cache as buffer (small VSI_CACHE_SIZE) should
> always be used on Windows by default with network files. It should go
> to OGR, I think.
>
> Radim
>
>> FWIK SMB filenames are in the form "servershared_folderxxxx" , but it can be
>> mounted on a drive, so it becomes "X:shared_folderxxxx". Perhaps there's a way
>> of resolving this to the original name ?
>>
>> But I'm not sure it is really needed either. Perhaps the theoretical performance
>> loss is neglectable in practice. I'm just throwing thoughts.
>>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev



-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
607 South Mathews Avenue, MC 150
Urbana, IL 61801
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007