[GRASS-dev] compile 6S library in GRASS

Thu Jun 28 06:36:56 PDT 2018

On 28/06/18 15:26, Roberta Fagandini wrote:
> 
> 
> 2018-06-28 14:46 GMT+02:00 Markus Metz <markus.metz.giswork at gmail.com 
> <mailto:markus.metz.giswork at gmail.com>>:
> 
> 
> 
>     On Thu, Jun 28, 2018 at 12:17 PM, Roberta Fagandini
>     <robifagandini at gmail.com <mailto:robifagandini at gmail.com>> wrote:
>      >
>      >
>      >
>      > 2018-06-28 10:26 GMT+02:00 Moritz Lennert
>     <mlennert at club.worldonline.be <mailto:mlennert at club.worldonline.be>>:
>      >>
>      >> On 27/06/18 15:40, Roberto Marzocchi wrote:
>      >>>
>      >>>
>      >>>
>      >>> Il giorno mer 27 giu 2018 alle ore 15:25 Markus Metz
>     <markus.metz.giswork at gmail.com
>     <mailto:markus.metz.giswork at gmail.com>
>     <mailto:markus.metz.giswork at gmail.com
>     <mailto:markus.metz.giswork at gmail.com>>> ha scritto:
>      >>>
>      >>>
>      >>>
>      >>>     On Wed, Jun 27, 2018 at 3:06 PM, Roberta Fagandini
>      >>>     <robifagandini at gmail.com <mailto:robifagandini at gmail.com>
>     <mailto:robifagandini at gmail.com <mailto:robifagandini at gmail.com>>>
>     wrote:
>      >>>      >
>      >>>      >
>      >>>      >
>      >>>      > 2018-06-27 14:24 GMT+02:00 Markus Metz
>      >>>     <markus.metz.giswork at gmail.com
>     <mailto:markus.metz.giswork at gmail.com>
>     <mailto:markus.metz.giswork at gmail.com
>     <mailto:markus.metz.giswork at gmail.com>>>:
>      >>>      >>
>      >>>      >>
>      >>>      >>
>      >>>      >> On Wed, Jun 27, 2018 at 2:16 PM, Roberta Fagandini
>      >>>     <robifagandini at gmail.com <mailto:robifagandini at gmail.com>
>     <mailto:robifagandini at gmail.com <mailto:robifagandini at gmail.com>>>
>     wrote:
>      >>>      >> >
>      >>>      >> > I'm trying to create an automatic procedure to
>     retrieve all
>      >>>     the input parameters for the control file of i.atcorr.
>      >>>      >> > I need py6S because I want to automatically read and
>     compute
>      >>>     AOT at 550nm from an AERONET file.
>      >>>      >>
>      >>>      >> With AOT you mean aerosol optical thickness right? This
>     is user
>      >>>     input to i.atcorr. There are various sources for AOT at
>     550nm, e.g.
>      >>>     MODIS.
>      >>>      >
>      >>>      >
>      >>>      > Right!
>      >>>      >
>      >>>      >>
>      >>>      >>
>      >>>      >>
>      >>>      >>
>      >>>      >> Anyway, it is probably much easier to to write your own
>     routine
>      >>>     to read the AERONET file and get the corresponding formula
>     for AOT
>      >>>     than to fight with 51821 lines of Fortran source code that is
>      >>>     numerically unstable. Maybe I can help with getting AOT at
>     550nm
>      >>>     from an AERONET file avoiding the Fortran version of 6S.
>      >>>      >
>      >>>      >
>      >>>      > Ok but I'm not able to create this kind of routine on my
>     own. If
>      >>>     someone, more expert than me, can help me in translating
>     the 6S/Py6S
>      >>>     functions I can go on in this way otherwise for me it's
>     easier to
>      >>>     use the fortran source code.
>      >>>
>      >>>     Apparently the AERONET data are simple CSV files. You can
>     import
>      >>>     them with r.in.xyz <http://r.in.xyz> <http://r.in.xyz> or
>     v.in.ascii.
>      >>>
>      >>>     Still, I think it is up to the user to decide on the source
>     of AOT
>      >>>     values. AERONET is one of many possible sources.
>      >>>
>      >>>
>      >>> Ok but I know that Roberta  want try to simplify the procedure
>     for atmospheric correction. In particular she would like to add an
>     option (not the only one) to read automatically the AOT at the wave
>     length of 550 nm from the AERONET file.
>      >>>
>      >>> The steps to do this IMHO are:
>      >>>
>      >>> 1) read the file (using GRASS or maybe better Python)
>      >>>
>      >>> 2) a formula to calculate the AOT550
>      >>>
>      >>> Unfortunately I have no idea about the 2nd point :-(
>      >>
>      >>
>      >> I'm not sure either, what exactly the necessary calculations are.
>      >>
>      >> Superficially reading through the relevant class in Py6S [1], I
>     understasnd that it does two things:
>      >>
>      >> 1) Get the value of AOT550
>      >> 2) Create a continuous profile by interpolating values
>      >>
>      >> If all you need is 1), then all they do (see _get_aot() method
>     at the end) is to read all the AOT_* columns and chose the value of
>     the line with the timestamp closest to the one asked for and the
>     wavelength closest to 550.
>      >>
>      >>
>      >> As the code is GPL you could probably just extract the necessary
>     parts and integrate them into your code. The only thing I'm not to
>     sure about is their use of pandas which creates yet again another
>     dependency which I don't really think necessary and desirable.
>      >
>      >
>      > Yesterday I studied the library better and I understood that it
>     only extracts the available AOT values closest to the wavelength 550
>     for the specified timestamp. No calculation is made so I'm trying to
>     write my own routine avoiding any dependency.
>      >
>      >>
>      >>
>      >>
>      >> It should be enough to extract from the file the date and time
>     (or maybe easier to work with: year and the Julian day) and all
>     AOT_* columns, parse the AOT_* columns for their wavelength value
>     and then identify the value which has closes time and wavelength to
>     the time requested and 550nm.
>      >
>      >
>      > That's exactly what I want to do and I'm working on it!
> 
>     open the file with your preferred spreadsheet program and copy the
>     corresponding value;-)
>     >  
>     >>
>     >>
>     >> Moritz
>     >
>     >
>     > I know that AERONET is not the only source and obviously in my module there will be both an option for providing manually the value, as Markus and Stefan suggested, and an option to extract the AOT from an AERONET file. I'm not an expert in MODIS data so if someone wants to give suggestions about how to retrieve the AOT from these data, I will definitely add this option too.
> 
> 
> ok..now I'm a bit confused..
> 
> 
>     about your module: it is concerned with Sentinel-2 cloud and shadow
>     detection. As a general rule, a module should do one thing, and
>     should do it right. In this sense, importing AERONET data should be
>     a separate module, and your module should do Sentinel-2 cloud and
>     shadow detection, nothing else, this is already complex enough.
> 
> 
> the module i.sentinel.mask  does only the cloud and shadow detection.
> 
> 
>     Regarding a new module e.g. v.in.aeronet:
>     It should interpolate AOT at 550nm considering the closest
>     wavelengths with valid data. Unfortunately most of the AERONET data
>     are nodata. Interpolation can be a simple linear interpolation, see
>     https://aeronet.gsfc.nasa.gov/new_web/V2/climo_new/explain/explain_new.html
>     <https://aeronet.gsfc.nasa.gov/new_web/V2/climo_new/explain/explain_new.html>
> 
>     It should perform spatial interpolation because the stations are
>     scattered with large distances in between. Spatial interpolation can
>     be done with any of the v.surf.* modules.
> 
>     It should offer temporal interpolation in case there are no data for
>     the required date. This can be done with the temporal GIS framework
>     or with r.series.lwr.
> 
> 
> Regarding AERONET, this is only a small part of another possible 
> module/script. My intention is to create an iterative procedure that 
> executes i.atcorr for all bands of the input image changing accordingly 
> the requested input parameters, this module/script should do:
> 
>   * read input bands from metadata file,
>   * write the control file retrieving as many parameters as possible
>     from the metadata
>   * add other parameters provided by users to the control file
>   * change the control file according to the input band
>   * run i.atcorr
> 
> Reading the AOT from an AERONET file,  it is only one possible option to 
> further automate the module/script. Obviously users can specify manually 
> the AOT value and they can retrieve it from their favourite source.

Concerning the Aeronet data: if it is possible to completely automate 
everything, including the download, and thus make it completely 
transparent to the user, then I think it could be interesting to code 
the extraction from the file. If, instead of just using the 
spatio-temporally and spectrally "closest" value, the module could also 
provide the different interpolations Markus writes about, then this 
would be a big plus (assuming that this really makes a significant 
difference compared to just using the closest value).

If, on the contrary, the user has to download and decompress the file, 
then I do understand Markus' point that it might be a bit overkill to 
code a procedure which is the equivalent of the user just opening the 
file and identifying manually the value to extract.

> 
> 
>     Regarding MODIS data, r.in.gdal can import MODIS data, and there is
>     r.modis
> 
>     Putting it all together:
>     that would be a wrapper script that helps to
>     1) import visibility as a map or estimate a fixed visibility value
>     2) perform atmospheric correction with i.atcorr
>     3) detect clouds and shadows
> 
> 
> My idea was to create a module (e.g. i.sentinel.preproc) that wraps the 
> dowload and import phase (i.sentinel.download and i.sentinel.import), 
> the atmosperic correction using i.atcorr (and the iterative procedure 
> that I'm trying to implement) and the cloud and shadow detection procedure.

This would be a nice module to have, yes !

Moritz