[gdal-dev] Reading Sentinel-2 Data in AWS
Trochim, Edward
Edward.Trochim at RadiantSolutions.com
Fri Dec 7 12:43:04 PST 2018
Hi Everyone,
I am using gdal 2.3.2 to access Sentinel-2 data on the AWS Open Data Registry (https://registry.opendata.aws/sentinel-2/). I am using a version of GDAL I built myself in an Amazon Linux docker image with the intention of eventually running it on AWS Lambda.
I am trying to use gdal's built-in support for Sentinel-2 (https://www.gdal.org/frmt_sentinel2.html) to easily read all the band data with vsis3 (https://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsis3). However, when I do so I get this error:
# gdalinfo /vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml --config AWS_REQUEST_PAYER requester
ERROR 4: `/vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml' not recognized as a supported file format.
gdalinfo failed - unable to open '/vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml'.
After taking a look at the Sentinel2 driver source code I saw that I needed to prepend SENTINEL2_L2A: to the path. With that I got a different error:
# gdalinfo SENTINEL2_L2A:/vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml --config AWS_REQUEST_PAYER requester
ERROR 1: Invalid syntax for SENTINEL2_L2A:
gdalinfo failed - unable to open 'SENTINEL2_L2A:/vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/metadata.xml'.
After some investigation I found this pull request (https://github.com/OSGeo/gdal/pull/1069) which seems to add support for the data that I'm trying to access. However, after rebuilding off the master branch in github I'm still getting the same errors.
Accessing the band files individually seems to work correctly:
# gdalinfo /vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/R10m/B02.jp2 --config AWS_REQUEST_PAYER requester
Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
Files: /vsis3/sentinel-s2-l2a/tiles/23/L/MD/2016/10/27/0/R10m/B02.jp2
Size is 10980, 10980
Coordinate System is:
PROJCS["WGS 84 / UTM zone 23S",
...
Does anyone know what I'm doing wrong?
In case it is helpful here is the output of gdalinfo -formats
Supported Formats:
VRT -raster- (rw+v): Virtual Raster
DERIVED -raster- (ro): Derived datasets using VRT pixel functions
GTiff -raster- (rw+vs): GeoTIFF
NITF -raster- (rw+vs): National Imagery Transmission Format
RPFTOC -raster- (rovs): Raster Product Format TOC format
ECRGTOC -raster- (rovs): ECRG TOC format
HFA -raster- (rw+v): Erdas Imagine Images (.img)
SAR_CEOS -raster- (rov): CEOS SAR Image
CEOS -raster- (rov): CEOS Image
JAXAPALSAR -raster- (rov): JAXA PALSAR Product Reader (Level 1.1/1.5)
GFF -raster- (rov): Ground-based SAR Applications Testbed File Format (.gff)
ELAS -raster- (rw+v): ELAS
AIG -raster- (rov): Arc/Info Binary Grid
AAIGrid -raster- (rwv): Arc/Info ASCII Grid
GRASSASCIIGrid -raster- (rov): GRASS ASCII Grid
SDTS -raster- (rov): SDTS Raster
DTED -raster- (rwv): DTED Elevation Raster
PNG -raster- (rwv): Portable Network Graphics
JPEG -raster- (rwv): JPEG JFIF
MEM -raster- (rw+): In Memory Raster
JDEM -raster- (rov): Japanese DEM (.mem)
GIF -raster- (rwv): Graphics Interchange Format (.gif)
BIGGIF -raster- (rov): Graphics Interchange Format (.gif)
ESAT -raster- (rov): Envisat Image Format
BSB -raster- (rov): Maptech BSB Nautical Charts
XPM -raster- (rwv): X11 PixMap Format
BMP -raster- (rw+v): MS Windows Device Independent Bitmap
DIMAP -raster- (rov): SPOT DIMAP
AirSAR -raster- (rov): AirSAR Polarimetric Image
RS2 -raster- (rovs): RadarSat 2 XML Product
SAFE -raster- (rov): Sentinel-1 SAR SAFE Product
PCIDSK -raster,vector- (rw+v): PCIDSK Database File
PCRaster -raster- (rw+): PCRaster Raster File
ILWIS -raster- (rw+v): ILWIS Raster Map
SGI -raster- (rw+v): SGI Image File Format 1.0
SRTMHGT -raster- (rwv): SRTMHGT File Format
Leveller -raster- (rw+v): Leveller heightfield
Terragen -raster- (rw+v): Terragen heightfield
ISIS3 -raster- (rw+v): USGS Astrogeology ISIS cube (Version 3)
ISIS2 -raster- (rw+v): USGS Astrogeology ISIS cube (Version 2)
PDS -raster- (rov): NASA Planetary Data System
PDS4 -raster- (rw+vs): NASA Planetary Data System 4
VICAR -raster- (rov): MIPL VICAR file
TIL -raster- (rov): EarthWatch .TIL
ERS -raster- (rw+v): ERMapper .ers Labelled
JP2OpenJPEG -raster,vector- (rwv): JPEG-2000 driver based on OpenJPEG library
L1B -raster- (rovs): NOAA Polar Orbiter Level 1b Data Set
FIT -raster- (rwv): FIT Image
GRIB -raster- (rwv): GRIdded Binary (.grb, .grb2)
RMF -raster- (rw+v): Raster Matrix Format
WCS -raster- (rovs): OGC Web Coverage Service
WMS -raster- (rwvs): OGC Web Map Service
MSGN -raster- (rov): EUMETSAT Archive native (.nat)
RST -raster- (rw+v): Idrisi Raster A.1
INGR -raster- (rw+v): Intergraph Raster
GSAG -raster- (rwv): Golden Software ASCII Grid (.grd)
GSBG -raster- (rw+v): Golden Software Binary Grid (.grd)
GS7BG -raster- (rw+v): Golden Software 7 Binary Grid (.grd)
COSAR -raster- (rov): COSAR Annotated Binary Matrix (TerraSAR-X)
TSX -raster- (rov): TerraSAR-X Product
COASP -raster- (ro): DRDC COASP SAR Processor Raster
R -raster- (rwv): R Object Data Store
MAP -raster- (rov): OziExplorer .MAP
KMLSUPEROVERLAY -raster- (rwv): Kml Super Overlay
PDF -raster,vector- (w+): Geospatial PDF
PLMOSAIC -raster- (ro): Planet Labs Mosaics API
CALS -raster- (rwv): CALS (Type 1)
WMTS -raster- (rwv): OGC Web Map Tile Service
SENTINEL2 -raster- (rovs): Sentinel 2
MRF -raster- (rw+v): Meta Raster Format
PNM -raster- (rw+v): Portable Pixmap Format (netpbm)
DOQ1 -raster- (rov): USGS DOQ (Old Style)
DOQ2 -raster- (rov): USGS DOQ (New Style)
PAux -raster- (rw+v): PCI .aux Labelled
MFF -raster- (rw+v): Vexcel MFF Raster
MFF2 -raster- (rw+): Vexcel MFF2 (HKV) Raster
FujiBAS -raster- (rov): Fuji BAS Scanner Image
GSC -raster- (rov): GSC Geogrid
FAST -raster- (rov): EOSAT FAST Format
BT -raster- (rw+v): VTP .bt (Binary Terrain) 1.3 Format
LAN -raster- (rw+v): Erdas .LAN/.GIS
CPG -raster- (rov): Convair PolGASP
IDA -raster- (rw+v): Image Data and Analysis
NDF -raster- (rov): NLAPS Data Format
EIR -raster- (rov): Erdas Imagine Raw
DIPEx -raster- (rov): DIPEx
LCP -raster- (rwv): FARSITE v.4 Landscape File (.lcp)
GTX -raster- (rw+v): NOAA Vertical Datum .GTX
LOSLAS -raster- (rov): NADCON .los/.las Datum Grid Shift
NTv1 -raster- (rov): NTv1 Datum Grid Shift
NTv2 -raster- (rw+vs): NTv2 Datum Grid Shift
CTable2 -raster- (rw+v): CTable2 Datum Grid Shift
ACE2 -raster- (rov): ACE2
SNODAS -raster- (rov): Snow Data Assimilation System
KRO -raster- (rw+v): KOLOR Raw
ROI_PAC -raster- (rw+v): ROI_PAC raster
RRASTER -raster- (rw+v): R Raster
BYN -raster- (rw+v): Natural Resources Canada's Geoid
ARG -raster- (rwv): Azavea Raster Grid format
RIK -raster- (rov): Swedish Grid RIK (.rik)
USGSDEM -raster- (rwv): USGS Optional ASCII DEM (and CDED)
GXF -raster- (rov): GeoSoft Grid Exchange Format
NWT_GRD -raster- (rw+v): Northwood Numeric Grid Format .grd/.tab
NWT_GRC -raster- (rov): Northwood Classified Grid Format .grc/.tab
ADRG -raster- (rw+vs): ARC Digitized Raster Graphics
SRP -raster- (rovs): Standard Raster Product (ASRP/USRP)
BLX -raster- (rwv): Magellan topo (.blx)
SAGA -raster- (rw+v): SAGA GIS Binary Grid (.sdat, .sg-grd-z)
XYZ -raster- (rwv): ASCII Gridded XYZ
HF2 -raster- (rwv): HF2/HFZ heightfield raster
OZI -raster- (rov): OziExplorer Image File
CTG -raster- (rov): USGS LULC Composite Theme Grid
E00GRID -raster- (rov): Arc/Info Export E00 GRID
ZMap -raster- (rwv): ZMap Plus Grid
NGSGEOID -raster- (rov): NOAA NGS Geoid Height Grids
IRIS -raster- (rov): IRIS data (.PPI, .CAPPi etc)
PRF -raster- (rov): Racurs PHOTOMOD PRF
RDA -raster- (ro): DigitalGlobe Raster Data Access driver
EEDAI -raster- (ros): Earth Engine Data API Image
SIGDEM -raster- (rwv): Scaled Integer Gridded DEM .sigdem
IGNFHeightASCIIGrid -raster- (rov): IGN France height correction ASCII Grid
CAD -raster,vector- (rovs): AutoCAD Driver
PLSCENES -raster,vector- (ro): Planet Labs Scenes API
GenBin -raster- (rov): Generic Binary (.hdr Labelled)
ENVI -raster- (rw+v): ENVI .hdr Labelled
EHdr -raster- (rw+v): ESRI .hdr Labelled
ISCE -raster- (rw+v): ISCE raster
HTTP -raster,vector- (ro): HTTP Fetching Wrapper
This electronic communication and any attachments may contain confidential and proprietary information of Radiant Solutions, Inc. If you are not the intended recipient, or an agent or employee responsible for delivering this communication to the intended recipient, or if you have received this communication in error, please do not print, copy, retransmit, disseminate or otherwise use the information. Please indicate to the sender that you have received this communication in error, and delete the copy you received.
Radiant Solutions reserves the right to monitor any electronic communication sent or received by its employees, agents or representatives.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181207/8bb31dda/attachment-0001.html>
More information about the gdal-dev
mailing list