[GRASS-dev] ​GSoC 2018 report week 03 - GRASS GIS module for Sentinel-2 cloud and shadow detection

[Nikos Alexandris]

* Roberta Fagandini <robifagandini at gmail.com> [2018-06-04 16:14:16 +0200]:

>Hi all!
>
>Thank you, Moritz and Nikos, for your suggestions!
>
>2018-06-04 11:54 GMT+02:00 Nikos Alexandris <nik at nikosalexandris.net>:
>
>> Ciao Roberta,
>>
>> great work!
>>
>> Moritz got me faster, as I intended to chime in, with similar
>> suggestions.
>>
>> So +1 for:
>>
>> - a "real" module, as Moritz described it
>>
>
>Done! I added the folder to my github repository but I have some trouble
>compiling the module with g.extension:
>
>g.extension extension=i.sentinel.mask operation=add url=
>https://github.com/RobiFag/GRASS_clouds_and_shadows
>
>error message:
>g.extension extension=i.sentinel.mask operation=add url=
>https://github.com/RobiFag/GRASS_clouds_and_shadows
>Fetching <i.sentinel.mask> from <
>https://github.com/RobiFag/GRASS_clouds_and_shadows/archive/master.zip> (be
>patient)...
>Compiling...
>make[1]: *** No rule to make target `/tmp/grass7-roberta-701
>4/tmpe7QEso/i.sentinel.mask/scripts/i.sentinel.mask', needed
>by `script'. Stop.
>/bin/sh: 1: cannot create /usr/lib/grass74/error.log:
>Permission denied
>make: *** [i.sentinel.mask] Error 2
>ERROR: Compilation failed, sorry. Please check above error messages.
>
>any suggestion?

Maybe irrelevant, but perhaps it is best to name the script as
`i.sentinel.mask.py`?


>> - for using the metadata file, and parsing it -- my experience with
>>  Landsat and other high-resolution satellite products, showed it's best to
>>  trust the individual metadata that accompany an acquisition.
>>   Parameters may be very specific to one single acquisition.
>>  Also, a module that is designed to be "dynamic" is more
>>  future-proof and easy to maintain and update.
>>   For example, reading band names from the meta-data, as opposed to more
>>  "static" designs, i.e. expecting specific hardcoded options, such as
>>  band names, or their order.
>>
>
>I have already thought about parsing the metadatafile to retrieve the input
>bands but at the moment the module requires atmospherically corrected bands
>and if users perform the atmospheric correction on their own I don't have
>any control on naming. Moreover, the module needs to know which raster map
>corresponds to the blue band, to the green band and so on in order to apply
>the rules for clouds and shadows detection. Therefore I need users specify
>each band but maybe I can manage this issue requiring a specific suffix in
>the bands' names (e.g. blabla_blue, blabla_green, etc.). In this way, users
>have to provide only the prefix (blabla) while the module search for all
>raster maps with the specified prefix and at the same time it is able to
>recognize each band.
>
>Anyway, the aim of the next coding period is to create a python script that
>wraps in a single GRASS module the download and import phase
>(i.sentinel.download and i.sentinel.import), the atmospheric correction
>using i.atcorr and the cloud and shadow detection procedure. In particular,
>for what concerns the atmospheric correction, I'd like to implement an
>iterative procedure that executes i.atcorr for all bands of the input image
>changing accordingly the requested input parameters. This part should
>simplify the management of input maps.

One thing here: massive processing workflows like to have simple
processing building blocks. One Input -> one Process -> one Output. In
"your" case:

- Input: name of/request one band and associated
  metadata, and optionally other ancillary data.

- Process: download, import, removal of atmospheric effects to estimate
  surface reflectance

- Output: surface reflectance ("corrected" for atmospheric effects) of
  the requested input band

This building block can run in one processing units/node. Making use of many
processing units/nodes, this block can run in parallel for different
input bands.

I suggest to build the module in a way that it runs for one band or as many
bands as the user requires for his workflow. I did this in
https://gitlab.com/NikosAlexandris/i.fusion.hpf/blob/master/i.fusion.hpf.py.
I don't suggest this script as an example. Yet I think the concept "for
one or many" is useful and wanted.


As a sidenote:

I think that we don't use the term "correction"
correctly. An image acquired by a satellite-borne sensor is not "wrong"
to need a correction. Rather, we try to retouch the digital form of the
acquisition, in order to estimate the energy that was actually reflected
by the surface, before the signal travels through the atmosphere.

I studied remote sensing with Thomas Lillesand's; Ralph W.
Kiefer's book "Remote Sensing and Image Interpretation". The book
"correctly" touches this very question.

Maybe it's only a perspective, and others have another say on this.


>> - PEP8 rules and keeping module options in separate lines to increase
>> legibility
>>
>> In line with the suggestions on style, I would stress out to fully
>> spell out the names of functions, options, descriptions of flags,
>> iterables and iterators.  It will make it easier for the next reader of
>> your code.

>Yes, you are both right!! I have to clean up the code from the style point
>of view and it will be one of the tasks of this week and especially of the
>last days before the GSoC Evaluation.


>> Finally, it's a good idea to write even mini-tests, while you are
>> progressing, for the small functions that eventually develop to keep the
>> full scrip compact, functional and more legible.
>>
>> Writing a test, for your module, in the end, will be much easier.
>>

>Sorry Nikos but I don't really understand what you mean by
>"mini-tests"..are you referring to the examples in the manual page?

For a function f(x) that takes x as an input and gives you y as output,
test, for example, whether the output value 'y' is within the expected
range. Test your single functions, inside the script. Use assertions, for
example, etc. If you have 10 functions in your script, and have written
a test for each, you will save your time later on when you will revisit
the code to debug or modify.  Spend as much time as you can in testing.


Again, this is what I try to stick to. And, lastly, I am not a core
developer, nor a mentor. I am, however, interested in seeing this module
coming alive. So, apologies to the mentors that I am chimingi in.

[rest deleted]

Grazie, Nikos
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20180604/d14f8191/attachment.sig>