[GRASS-dev] [GRASS GIS] #2139: g.parser does not allow newline character in option value for Python scripts

[GRASS GIS]

#2139: g.parser does not allow newline character in option value for Python
scripts
----------------------------------------------------+-----------------------
 Reporter:  wenzeslaus                              |       Owner:  grass-dev@…              
     Type:  defect                                  |      Status:  new                      
 Priority:  normal                                  |   Milestone:  7.0.0                    
Component:  Parser                                  |     Version:  svn-trunk                
 Keywords:  g.parser, scripts, newline, line break  |    Platform:  All                      
      Cpu:  Unspecified                             |  
----------------------------------------------------+-----------------------
 `g.parser` mechanism does not allow newline character when newline is in
 option value. This applies to Python modules:

 {{{
 m.proj input=test.txt proj_out="+proj=lcc..."
 proj_in="+proj=merc...+to_meter=1
 "
 }}}

 C modules are fine:

 {{{
 g.proj proj4="
 "
 }}}
 (error is ''Can't parse PROJ.4-style parameter string'' which means that
 parser allowed module to continue)

 When newline is in parameter, `g.parser -s module/name/or/path` will
 output (for the above example):

 {{{
 @ARGS_PARSED@
 flag_i=0
 flag_o=0
 flag_d=0
 flag_e=0
 flag_c=0
 opt_input=-
 opt_output=
 opt_separator= ,
 opt_proj_in=+proj=merc...+to_meter=1

 opt_proj_out=+proj=lcc...+to_meter=1
 }}}

 I'm not sure how newlines should be handled, I haven't found a
 documentation of this format.

 Anyway, the problem is in
 [source:grass/trunk/lib/python/script/core.py#L594 _parse_opts()] function
 which stops when line is empty:

 {{{
 if not line:
     break
 }}}

 If the option with newline is not the last one, Python script will get the
 `options` dictionary incomplete which causes unexpected error.

 In my case I was running `m.proj` in a Python script with the parameter
 `proj_in` obtained by `g.proj`:

 {{{
 proj_in = gcore.read_command('g.proj', flags='jf')
 ...
 proc = gcore.start_command('m.proj', input='-', separator=' , ',
                            proj_in=from_proj, proj_out=to_proj,
                            stdin=gcore.PIPE,
                            stdout=gcore.PIPE, stderr=gcore.PIPE)
 }}}

 And the error I obtained was:

 {{{
 Traceback (most recent call last):
   File "/home/vasek/dev/grass/trunk_clang/dist.i686-pc-linux-
 gnu/scripts/m.proj", line 292, in <module>
     main()
   File "/home/vasek/dev/grass/trunk_clang/dist.i686-pc-linux-
 gnu/scripts/m.proj", line 121, in main
     proj_out = options['proj_out']
 KeyError: 'proj_out'
 }}}

 Simple `proj_in.strip()` solves the problem in my case because the output
 of `g.proj` is ended with a newline.

 However, this is a general problem and if `g.parser` Python mechanism is
 not able to handle newlines `g.parser` itself should give an error message
 when user tries to input some newline. If `g.parser` Python mechanism
 should allow newlines (anywhere in the option value) for all modules, the
 output and/or the reader (`_parse_opts()` function) should be fixed
 accordingly.

 Maybe the `_parse_opts()` function should check the lines by regular
 expressions `opt_.*=` and lines which does not match should be merged with
 above option value (I'm not sure about last line, it might be empty
 always, if so, it should not be merged).

 Are there other suggestions, e.g. for some fast (in any sense)
 implementation?

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/2139>
GRASS GIS <http://grass.osgeo.org>