[GRASS-dev] ​GSoC 2018 report week 03 - GRASS GIS module for Sentinel-2 cloud and shadow detection

[Roberta Fagandini]

Hi Nikos!

2018-06-04 17:23 GMT+02:00 Nikos Alexandris <nik at nikosalexandris.net>:

> * Roberta Fagandini <robifagandini at gmail.com> [2018-06-04 16:14:16 +0200]:
>
>
> Hi all!
>>
>> Thank you, Moritz and Nikos, for your suggestions!
>>
>> 2018-06-04 11:54 GMT+02:00 Nikos Alexandris <nik at nikosalexandris.net>:
>>
>> Ciao Roberta,
>>>
>>> great work!
>>>
>>> Moritz got me faster, as I intended to chime in, with similar
>>> suggestions.
>>>
>>> So +1 for:
>>>
>>> - a "real" module, as Moritz described it
>>>
>>>
>> Done! I added the folder to my github repository but I have some trouble
>> compiling the module with g.extension:
>>
>> g.extension extension=i.sentinel.mask operation=add url=
>> https://github.com/RobiFag/GRASS_clouds_and_shadows
>>
>> error message:
>> g.extension extension=i.sentinel.mask operation=add url=
>> https://github.com/RobiFag/GRASS_clouds_and_shadows
>> Fetching <i.sentinel.mask> from <
>> https://github.com/RobiFag/GRASS_clouds_and_shadows/archive/master.zip>
>> (be
>> patient)...
>> Compiling...
>> make[1]: *** No rule to make target `/tmp/grass7-roberta-701
>> 4/tmpe7QEso/i.sentinel.mask/scripts/i.sentinel.mask', needed
>> by `script'. Stop.
>> /bin/sh: 1: cannot create /usr/lib/grass74/error.log:
>> Permission denied
>> make: *** [i.sentinel.mask] Error 2
>> ERROR: Compilation failed, sorry. Please check above error messages.
>>
>> any suggestion?
>>
>
> Maybe irrelevant, but perhaps it is best to name the script as
> `i.sentinel.mask.py`?


I named the file as you suggested but I still have some problems, now the
error message is:

Fetching <i.sentinel.mask> from <
https://github.com/RobiFag/GRASS_clouds_and_shadows/archive/master.zip> (be
patient)...
Compiling...
Installing...
make: *** No rule to make target `install'. Stop.
WARNING: Installation failed, sorry. Please check above error messages.


>
>
>
> - for using the metadata file, and parsing it -- my experience with
>>>  Landsat and other high-resolution satellite products, showed it's best
>>> to
>>>  trust the individual metadata that accompany an acquisition.
>>>   Parameters may be very specific to one single acquisition.
>>>  Also, a module that is designed to be "dynamic" is more
>>>  future-proof and easy to maintain and update.
>>>   For example, reading band names from the meta-data, as opposed to more
>>>  "static" designs, i.e. expecting specific hardcoded options, such as
>>>  band names, or their order.
>>>
>>>
>> I have already thought about parsing the metadatafile to retrieve the
>> input
>> bands but at the moment the module requires atmospherically corrected
>> bands
>> and if users perform the atmospheric correction on their own I don't have
>> any control on naming. Moreover, the module needs to know which raster map
>> corresponds to the blue band, to the green band and so on in order to
>> apply
>> the rules for clouds and shadows detection. Therefore I need users specify
>> each band but maybe I can manage this issue requiring a specific suffix in
>> the bands' names (e.g. blabla_blue, blabla_green, etc.). In this way,
>> users
>> have to provide only the prefix (blabla) while the module search for all
>> raster maps with the specified prefix and at the same time it is able to
>> recognize each band.
>>
>> Anyway, the aim of the next coding period is to create a python script
>> that
>> wraps in a single GRASS module the download and import phase
>> (i.sentinel.download and i.sentinel.import), the atmospheric correction
>> using i.atcorr and the cloud and shadow detection procedure. In
>> particular,
>> for what concerns the atmospheric correction, I'd like to implement an
>> iterative procedure that executes i.atcorr for all bands of the input
>> image
>> changing accordingly the requested input parameters. This part should
>> simplify the management of input maps.
>>
>
> One thing here: massive processing workflows like to have simple
> processing building blocks. One Input -> one Process -> one Output. In
> "your" case:
>
> - Input: name of/request one band and associated
>  metadata, and optionally other ancillary data.
>
> - Process: download, import, removal of atmospheric effects to estimate
>  surface reflectance
>
> - Output: surface reflectance ("corrected" for atmospheric effects) of
>  the requested input band
>
> This building block can run in one processing units/node. Making use of
> many
> processing units/nodes, this block can run in parallel for different
> input bands.
>
> I suggest to build the module in a way that it runs for one band or as many
> bands as the user requires for his workflow. I did this in
> https://gitlab.com/NikosAlexandris/i.fusion.hpf/blob/master/
> i.fusion.hpf.py.
> I don't suggest this script as an example. Yet I think the concept "for
> one or many" is useful and wanted.
>

Thank you! This is very interesting..
I will check your module and I will certainly take it into account even if
all the requested input bands are mandatory otherwise the clouds and
shadows detection procedures [0] will return wrong results.


>
> As a sidenote:
>
> I think that we don't use the term "correction"
> correctly. An image acquired by a satellite-borne sensor is not "wrong"
> to need a correction. Rather, we try to retouch the digital form of the
> acquisition, in order to estimate the energy that was actually reflected
> by the surface, before the signal travels through the atmosphere.
>
> I studied remote sensing with Thomas Lillesand's; Ralph W.
> Kiefer's book "Remote Sensing and Image Interpretation". The book
> "correctly" touches this very question.
>
> Maybe it's only a perspective, and others have another say on this.
>
>
> - PEP8 rules and keeping module options in separate lines to increase
>>> legibility
>>>
>>> In line with the suggestions on style, I would stress out to fully
>>> spell out the names of functions, options, descriptions of flags,
>>> iterables and iterators.  It will make it easier for the next reader of
>>> your code.
>>>
>>
> Yes, you are both right!! I have to clean up the code from the style point
>> of view and it will be one of the tasks of this week and especially of the
>> last days before the GSoC Evaluation.
>>
>
>
> Finally, it's a good idea to write even mini-tests, while you are
>>> progressing, for the small functions that eventually develop to keep the
>>> full scrip compact, functional and more legible.
>>>
>>> Writing a test, for your module, in the end, will be much easier.
>>>
>>>
> Sorry Nikos but I don't really understand what you mean by
>> "mini-tests"..are you referring to the examples in the manual page?
>>
>
> For a function f(x) that takes x as an input and gives you y as output,
> test, for example, whether the output value 'y' is within the expected
> range. Test your single functions, inside the script. Use assertions, for
> example, etc. If you have 10 functions in your script, and have written
> a test for each, you will save your time later on when you will revisit
> the code to debug or modify.  Spend as much time as you can in testing.
>

Ok, it's clearer to me now! I will certainly do it!


>
>
> Again, this is what I try to stick to. And, lastly, I am not a core
> developer, nor a mentor. I am, however, interested in seeing this module
> coming alive. So, apologies to the mentors that I am chimingi in.
>

Thanks, I hope you will continue to follow the development of the
project/module. As for me also your suggestions and feedback are very
useful!


>
> [rest deleted]
>
> Grazie, Nikos
>

Ciao
Roberta


[0]
https://drive.google.com/file/d/1KYEKvNBurBFHw1xUTLjM0PW80Z-7br81/view?usp=sharing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20180604/48bbd94e/attachment-0001.html>