[GRASS-dev] Re: [GRASS GIS] #1447: wxGUI wingrass scripts need whitespace in path

[GRASS GIS]

#1447: wxGUI wingrass scripts need whitespace in path
------------------------------+---------------------------------------------
 Reporter:  mmetz             |       Owner:  grass-dev@…              
     Type:  defect            |      Status:  new                      
 Priority:  major             |   Milestone:  6.4.2                    
Component:  wxGUI             |     Version:  svn-releasebranch64      
 Keywords:  wingrass, spaces  |    Platform:  MSWindows XP             
      Cpu:  All               |  
------------------------------+---------------------------------------------

Comment(by mmetz):

 Replying to [comment:10 glynn]:
 > Replying to [comment:8 mmetz]:
 >
 > > The *args argument to subprocess.Popen [0] is a list of arguments.
 >
 > Note: args is a list of Popen's non-keyword arguments. args[0] should be
 a list containing the program's arguments. len(args) should always be 1;
 all arguments except for the first one ("args") should be passed as
 keyword arguments, not positional arguments.
 >
 > While subprocess.Popen()'s first argument can be a string, this feature
 probably shouldn't be used. It's too easy to get it wrong.

 The horrible hack worksforme. But I am aware that any string protection
 which is normally done by Subprocess.Popen's string conversion, would then
 need to be done by the wxGUI, and lib/python/task.py.
 >
 > > subprocess.Popen converts the arguments differently, depending on
 whether there is whitespace in an argument or not.
 >
 > subprocess.Popen() has to convert the argument list to a command string
 according to the rules by which MSVCRT parses the command string into
 arguments, so that the program's main() ends up seeing the correct argv[].
 One of those rules is that strings containing whitespace must be quoted.

 That's all fine for programs that have a main(), but shell scripts don't
 have main(). I think that is the reason why the current implementation
 works for real programs but not for scripts. MSVCRT rules do not (need to)
 cater for strings passed to cmd.exe, then to %GRASS_SH%. AFAICT, a file
 path without whitespaces is passed ok to the Windows shell (cmd.exe), but
 then further passed to %GRASS_SH%, yet another shell, and there the (by
 now unprotected) backslashes get lost if not quoted properly.
 >
 > Because the underlying Windows interface (!CreateProcess) treats a
 command line as a string, rather than as a list of strings (as is the case
 for Unix' execve()), there isn't any 100% reliable way of ensuring that
 the program gets the correct argv[] passed to its main() function
 (historically, DOS programs were often written in assembler and didn't
 have a main(), while Windows programs use !WinMain rather than main()).
 The best that you can do is to assume that the program parses the command
 line according to the MSVCRT rules (there's a URL in the subprocess.py
 file). If it doesn't, you lose.

 Same as above, 1) worksforme, 2) we could do that horrible hack for
 scripts only.
 >
 > > The result is meant as input for programs and apparently sometimes
 incompatible with "sh -c <grass.module> <arguments>". The problem for file
 input arguments arises if there is no whitespace in there because sh
 removes all backslashes left over by subprocess.Popen. The workaround is
 to convert the *args list into one long string with whitespaces, file
 paths quoted earlier when creating the command:
 >
 > Before attempting this, first please confirm that the data being passed
 to `subprocess.Popen.__init__` by gcmd.Popen() is actually correct.
 len(args) should be 1, args[0] should be a list of strings, none of those
 strings should have any quoting or escaping (beware of Python's own
 quoting/escaping if repr() is used). If this isn't the case, then the
 problem lies elsewhere, and trying to correct for it in gcmd.Popen would
 actually mean introducing an "equal but opposite" bug.

 The strings indicating file paths do have escaping under Windows within
 the wxGUI, but it's \ that is escaped to \\, looks fine to me. These
 strings are converted somewhere inside subprocess.Popen to protect
 whitespaces, but not to protect backslashes (escapes). I'm pretty sure
 that this is the problem for scripts because what the sh shell sees is
 e.g. \I and not \\I, i.e. the backslash gets lost (see above). I have
 tested in wxGUI with v.in.geonames (script) and v.in.ogr (real program)
 and there is no quoting before subprocess.Popen is called. Quoting appears
 as output of subprocess.Popen if an argument contains whitespace.

 The fix causes subprocess.Popen to protect everything because the program
 and its args are passed as one single string which means that protection
 of single arguments, e.g. input="D:\Grass_test_data\IT\It.txt" must be
 done manually.

 I am aware that this is not looking ideal. Any better ideas?
 >
 > Second, ensure that any "fix" for shell scripts DOESN'T go into 7.0. On
 Windows, 7.0 needs to work with binaries, Python scripts and (cmd.exe)
 batch files, and it needs to do so reliably. If that means that it doesn't
 work with bash scripts, tough.

 I am aware of that ;-)

 Markus M

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/1447#comment:11>
GRASS GIS <http://grass.osgeo.org>