[gdal-dev] Reading Sentinel-2 Data in AWS
Bborie Park
dustymugs at gmail.com
Fri Dec 7 13:02:09 PST 2018
You may want to look at Google's store of Sentinel-2 as Google faithfully
follows the SAFE format, which looks to be what's expected in the Sentinel2
driver.
The data stored in AWS has been post-processed for SentinelHub so is not
SAFE format.
-bborie
On Fri, Dec 7, 2018 at 12:43 PM Trochim, Edward <
Edward.Trochim at radiantsolutions.com> wrote:
> Hi Everyone,
>
>
>
> I am using gdal 2.3.2 to access Sentinel-2 data on the AWS Open Data
> Registry (https://registry.opendata.aws/sentinel-2/). I am using a
> version of GDAL I built myself in an Amazon Linux docker image with the
> intention of eventually running it on AWS Lambda.
>
>
>
> I am trying to use gdal’s built-in support for Sentinel-2 (
> https://www.gdal.org/frmt_sentinel2.html) to easily read all the band
> data with vsis3 (
> https://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsis3).
> However, when I do so I get this error:
>
>
>
> # gdalinfo /vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml
> --config AWS_REQUEST_PAYER requester
>
> ERROR 4: `/vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml'
> not recognized as a supported file format.
>
> gdalinfo failed - unable to open
> '/vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml'.
>
>
>
> After taking a look at the Sentinel2 driver source code I saw that I
> needed to prepend SENTINEL2_L2A: to the path. With that I got a different
> error:
>
> # gdalinfo
> SENTINEL2_L2A:/vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml
> --config AWS_REQUEST_PAYER requester
>
> ERROR 1: Invalid syntax for SENTINEL2_L2A:
>
> gdalinfo failed - unable to open
> 'SENTINEL2_L2A:/vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml'.
>
>
>
> After some investigation I found this pull request (
> https://github.com/OSGeo/gdal/pull/1069) which seems to add support for
> the data that I’m trying to access. However, after rebuilding off the
> master branch in github I’m still getting the same errors.
>
>
>
> Accessing the band files individually seems to work correctly:
>
> # gdalinfo /vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/R10m/B02.jp2
> --config AWS_REQUEST_PAYER requester
>
> Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
>
> Files: /vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/R10m/B02.jp2
>
> Size is 10980, 10980
>
> Coordinate System is:
>
> PROJCS["WGS 84 / UTM zone 23S",
>
> …
>
>
>
> Does anyone know what I’m doing wrong?
>
>
>
> In case it is helpful here is the output of gdalinfo –formats
>
> Supported Formats:
>
> VRT -raster- (rw+v): Virtual Raster
>
> DERIVED -raster- (ro): Derived datasets using VRT pixel functions
>
> GTiff -raster- (rw+vs): GeoTIFF
>
> NITF -raster- (rw+vs): National Imagery Transmission Format
>
> RPFTOC -raster- (rovs): Raster Product Format TOC format
>
> ECRGTOC -raster- (rovs): ECRG TOC format
>
> HFA -raster- (rw+v): Erdas Imagine Images (.img)
>
> SAR_CEOS -raster- (rov): CEOS SAR Image
>
> CEOS -raster- (rov): CEOS Image
>
> JAXAPALSAR -raster- (rov): JAXA PALSAR Product Reader (Level 1.1/1.5)
>
> GFF -raster- (rov): Ground-based SAR Applications Testbed File Format
> (.gff)
>
> ELAS -raster- (rw+v): ELAS
>
> AIG -raster- (rov): Arc/Info Binary Grid
>
> AAIGrid -raster- (rwv): Arc/Info ASCII Grid
>
> GRASSASCIIGrid -raster- (rov): GRASS ASCII Grid
>
> SDTS -raster- (rov): SDTS Raster
>
> DTED -raster- (rwv): DTED Elevation Raster
>
> PNG -raster- (rwv): Portable Network Graphics
>
> JPEG -raster- (rwv): JPEG JFIF
>
> MEM -raster- (rw+): In Memory Raster
>
> JDEM -raster- (rov): Japanese DEM (.mem)
>
> GIF -raster- (rwv): Graphics Interchange Format (.gif)
>
> BIGGIF -raster- (rov): Graphics Interchange Format (.gif)
>
> ESAT -raster- (rov): Envisat Image Format
>
> BSB -raster- (rov): Maptech BSB Nautical Charts
>
> XPM -raster- (rwv): X11 PixMap Format
>
> BMP -raster- (rw+v): MS Windows Device Independent Bitmap
>
> DIMAP -raster- (rov): SPOT DIMAP
>
> AirSAR -raster- (rov): AirSAR Polarimetric Image
>
> RS2 -raster- (rovs): RadarSat 2 XML Product
>
> SAFE -raster- (rov): Sentinel-1 SAR SAFE Product
>
> PCIDSK -raster,vector- (rw+v): PCIDSK Database File
>
> PCRaster -raster- (rw+): PCRaster Raster File
>
> ILWIS -raster- (rw+v): ILWIS Raster Map
>
> SGI -raster- (rw+v): SGI Image File Format 1.0
>
> SRTMHGT -raster- (rwv): SRTMHGT File Format
>
> Leveller -raster- (rw+v): Leveller heightfield
>
> Terragen -raster- (rw+v): Terragen heightfield
>
> ISIS3 -raster- (rw+v): USGS Astrogeology ISIS cube (Version 3)
>
> ISIS2 -raster- (rw+v): USGS Astrogeology ISIS cube (Version 2)
>
> PDS -raster- (rov): NASA Planetary Data System
>
> PDS4 -raster- (rw+vs): NASA Planetary Data System 4
>
> VICAR -raster- (rov): MIPL VICAR file
>
> TIL -raster- (rov): EarthWatch .TIL
>
> ERS -raster- (rw+v): ERMapper .ers Labelled
>
> JP2OpenJPEG -raster,vector- (rwv): JPEG-2000 driver based on OpenJPEG
> library
>
> L1B -raster- (rovs): NOAA Polar Orbiter Level 1b Data Set
>
> FIT -raster- (rwv): FIT Image
>
> GRIB -raster- (rwv): GRIdded Binary (.grb, .grb2)
>
> RMF -raster- (rw+v): Raster Matrix Format
>
> WCS -raster- (rovs): OGC Web Coverage Service
>
> WMS -raster- (rwvs): OGC Web Map Service
>
> MSGN -raster- (rov): EUMETSAT Archive native (.nat)
>
> RST -raster- (rw+v): Idrisi Raster A.1
>
> INGR -raster- (rw+v): Intergraph Raster
>
> GSAG -raster- (rwv): Golden Software ASCII Grid (.grd)
>
> GSBG -raster- (rw+v): Golden Software Binary Grid (.grd)
>
> GS7BG -raster- (rw+v): Golden Software 7 Binary Grid (.grd)
>
> COSAR -raster- (rov): COSAR Annotated Binary Matrix (TerraSAR-X)
>
> TSX -raster- (rov): TerraSAR-X Product
>
> COASP -raster- (ro): DRDC COASP SAR Processor Raster
>
> R -raster- (rwv): R Object Data Store
>
> MAP -raster- (rov): OziExplorer .MAP
>
> KMLSUPEROVERLAY -raster- (rwv): Kml Super Overlay
>
> PDF -raster,vector- (w+): Geospatial PDF
>
> PLMOSAIC -raster- (ro): Planet Labs Mosaics API
>
> CALS -raster- (rwv): CALS (Type 1)
>
> WMTS -raster- (rwv): OGC Web Map Tile Service
>
> SENTINEL2 -raster- (rovs): Sentinel 2
>
> MRF -raster- (rw+v): Meta Raster Format
>
> PNM -raster- (rw+v): Portable Pixmap Format (netpbm)
>
> DOQ1 -raster- (rov): USGS DOQ (Old Style)
>
> DOQ2 -raster- (rov): USGS DOQ (New Style)
>
> PAux -raster- (rw+v): PCI .aux Labelled
>
> MFF -raster- (rw+v): Vexcel MFF Raster
>
> MFF2 -raster- (rw+): Vexcel MFF2 (HKV) Raster
>
> FujiBAS -raster- (rov): Fuji BAS Scanner Image
>
> GSC -raster- (rov): GSC Geogrid
>
> FAST -raster- (rov): EOSAT FAST Format
>
> BT -raster- (rw+v): VTP .bt (Binary Terrain) 1.3 Format
>
> LAN -raster- (rw+v): Erdas .LAN/.GIS
>
> CPG -raster- (rov): Convair PolGASP
>
> IDA -raster- (rw+v): Image Data and Analysis
>
> NDF -raster- (rov): NLAPS Data Format
>
> EIR -raster- (rov): Erdas Imagine Raw
>
> DIPEx -raster- (rov): DIPEx
>
> LCP -raster- (rwv): FARSITE v.4 Landscape File (.lcp)
>
> GTX -raster- (rw+v): NOAA Vertical Datum .GTX
>
> LOSLAS -raster- (rov): NADCON .los/.las Datum Grid Shift
>
> NTv1 -raster- (rov): NTv1 Datum Grid Shift
>
> NTv2 -raster- (rw+vs): NTv2 Datum Grid Shift
>
> CTable2 -raster- (rw+v): CTable2 Datum Grid Shift
>
> ACE2 -raster- (rov): ACE2
>
> SNODAS -raster- (rov): Snow Data Assimilation System
>
> KRO -raster- (rw+v): KOLOR Raw
>
> ROI_PAC -raster- (rw+v): ROI_PAC raster
>
> RRASTER -raster- (rw+v): R Raster
>
> BYN -raster- (rw+v): Natural Resources Canada's Geoid
>
> ARG -raster- (rwv): Azavea Raster Grid format
>
> RIK -raster- (rov): Swedish Grid RIK (.rik)
>
> USGSDEM -raster- (rwv): USGS Optional ASCII DEM (and CDED)
>
> GXF -raster- (rov): GeoSoft Grid Exchange Format
>
> NWT_GRD -raster- (rw+v): Northwood Numeric Grid Format .grd/.tab
>
> NWT_GRC -raster- (rov): Northwood Classified Grid Format .grc/.tab
>
> ADRG -raster- (rw+vs): ARC Digitized Raster Graphics
>
> SRP -raster- (rovs): Standard Raster Product (ASRP/USRP)
>
> BLX -raster- (rwv): Magellan topo (.blx)
>
> SAGA -raster- (rw+v): SAGA GIS Binary Grid (.sdat, .sg-grd-z)
>
> XYZ -raster- (rwv): ASCII Gridded XYZ
>
> HF2 -raster- (rwv): HF2/HFZ heightfield raster
>
> OZI -raster- (rov): OziExplorer Image File
>
> CTG -raster- (rov): USGS LULC Composite Theme Grid
>
> E00GRID -raster- (rov): Arc/Info Export E00 GRID
>
> ZMap -raster- (rwv): ZMap Plus Grid
>
> NGSGEOID -raster- (rov): NOAA NGS Geoid Height Grids
>
> IRIS -raster- (rov): IRIS data (.PPI, .CAPPi etc)
>
> PRF -raster- (rov): Racurs PHOTOMOD PRF
>
> RDA -raster- (ro): DigitalGlobe Raster Data Access driver
>
> EEDAI -raster- (ros): Earth Engine Data API Image
>
> SIGDEM -raster- (rwv): Scaled Integer Gridded DEM .sigdem
>
> IGNFHeightASCIIGrid -raster- (rov): IGN France height correction ASCII
> Grid
>
> CAD -raster,vector- (rovs): AutoCAD Driver
>
> PLSCENES -raster,vector- (ro): Planet Labs Scenes API
>
> GenBin -raster- (rov): Generic Binary (.hdr Labelled)
>
> ENVI -raster- (rw+v): ENVI .hdr Labelled
>
> EHdr -raster- (rw+v): ESRI .hdr Labelled
>
> ISCE -raster- (rw+v): ISCE raster
>
> HTTP -raster,vector- (ro): HTTP Fetching Wrapper
>
> This electronic communication and any attachments may contain confidential
> and proprietary information of Radiant Solutions, Inc. If you are not the
> intended recipient, or an agent or employee responsible for delivering this
> communication to the intended recipient, or if you have received this
> communication in error, please do not print, copy, retransmit, disseminate
> or otherwise use the information. Please indicate to the sender that you
> have received this communication in error, and delete the copy you
> received.
>
> Radiant Solutions reserves the right to monitor any electronic
> communication sent or received by its employees, agents or representatives.
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181207/8383089e/attachment-0001.html>
More information about the gdal-dev
mailing list