[gdal-dev] How to get a file list from a Sentinel zip?

Wed Apr 8 11:56:21 PDT 2020

Hi Jukka,

I hit "Send" before knowing that Even had replied you.

Ivan

________________________________
From: Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi>
Sent: Wednesday, April 8, 2020 12:52 PM
To: Ivan Lucena <ivan.lucena at outlook.com>; 'gdal-dev at lists.osgeo.org' <gdal-dev at lists.osgeo.org>
Subject: Re: How to get a file list from a Sentinel zip?

Ivan Lucena <ivan.lucena at outlook.com>  wrote:

> Jukka,

>> gdalinfo S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip
>>
>> ...if I rename the zip file into "test.zip" I can get the file list.
>> gdalinfo /vsizip/test.zip

>You did more than renaming the file, you also added /vsizip/ but it might be a typo while editing the e-mail.
No, it was not a typo but I did not mention that when the name of the zip file does not match the Sentinel autorecognition then using /vsizip/ is necessary or otherwise happens this:

gdalinfo test.zip
ERROR 4: `test.zip' not recognized as a supported file format.
gdalinfo failed - unable to open 'test.zip'.

>What do you get from:

>gdalinfo /vsizip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip
This command returns the list of Sentinel raster subdatasets and the result is the same than by running gdalinfo /vsizip/

> or the SUBDATASET_N_NAME contents you showed:

>gdalinfo /vsizip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/MTD_MSIL2A.xml:10m:EPSG_32634

The name of the subdataset is actually longer and contains also prefix "SENTINEL2_L2A:" as gdalinfo also reports it. Gdalinfo about the subdataset works as it is supposed to work with command

gdalinfo SENTINEL2_L2A:/vsizip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/MTD_MSIL2A.xml:10m:EPSG_32634

What I was interested in is not the Sentinel rasters but the vector data from the Quality Indicator (QI) directory: cloud mask vectors and defective pixels mask. I can find them by listing the files from the archive with some zip utility like 7-zip on Windows. Someone who really works with Sentinel data would probably appreciate some convenient method for finding at least all the cloud masks but I am just curious because I saw this question in gis.stackexchange https://gis.stackexchange.com/questions/357540/how-to-use-gdal-vsizip-to-access-gml-files-from-sentinel-2-zip-package/357589.

-Jukka-

>I hope it helps,

>Ivan

________________________________________
From: gdal-dev <mailto:gdal-dev-bounces at lists.osgeo.org> on behalf of Rahkonen Jukka (MML) <mailto:jukka.rahkonen at maanmittauslaitos.fi>
Sent: Wednesday, April 8, 2020 10:06 AM
To: 'gdal-dev at lists.osgeo.org' <mailto:gdal-dev at lists.osgeo.org>
Subject: [gdal-dev] How to get a file list from a Sentinel zip?

Hi,

I am investigating this dataset https://scihub.copernicus.eu/dhus/odata/v1/Products('456d05ea-835f-4e6c-b957-751c155252a1')/%24value
download with test/test (890 MB)

There seems to be some automagic logic that recognizes that the zip is a Sentinel zip

gdalinfo S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip
.
Subdatasets:
  SUBDATASET_1_NAME=SENTINEL2_L2A:/vsizip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/MTD_MSIL2A.xml:10m:EPSG_32634
  SUBDATASET_1_DESC=Bands B2, B3, B4, B8 with 10m resolution, UTM 34N
  SUBDATASET_2_NAME=SENTINEL2_L2A:/vsizip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/MTD_MSIL2A.xml:20m:EPSG_32634
  SUBDATASET_2_DESC=Bands B5, B6, B7, B8A, B11, B12 with 20m resolution, UTM 34N
  SUBDATASET_3_NAME=SENTINEL2_L2A:/vsizip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/MTD_MSIL2A.xml:60m:EPSG_32634
  SUBDATASET_3_DESC=Bands B1, B9 with 60m resolution, UTM 34N
  SUBDATASET_4_NAME=SENTINEL2_L2A:/vsizip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/MTD_MSIL2A.xml:TCI:EPSG_32634
  SUBDATASET_4_DESC=True color image, UTM 34N

However, the zip file contains also vector data in gml format. Those files gets hidden by the Sentinel auto-recognizing system but if I rename the zip file into "test.zip" I can get the file list.

gdalinfo /vsizip/test.zip
ERROR 4: `/vsizip/test.zip' not recognized as a supported file format.
gdalinfo failed - unable to open '/vsizip/test.zip'.
Unable to open source `/vsizip/test.zip' directly.
The archive contains 146 files:
       /vsizip/test.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/
. . .
/vsizip/test.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/GRANULE/L2A_T34TFS_A024972_20200403T092024/QI_DATA/MSK_TECQUA_B02.gml

Now I can read the gml file with ogrinfo
ogrinfo /vsizip/test.zip/S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.SAFE/GRANULE/L2A_T34TFS_A024972_20200403T092024/QI_DATA/MSK_TECQUA_B02.gml

and the command also works also with the original name of the zip S2A_MSIL2A_20200403T092021_N0214_R093_T34TFS_20200403T114409.zip.

Two questions:
. How to read the full file list of a zip file that follows the structure and naming convention of Sentinel zips without using the rename trick?
. I must read the file list with gdalinfo because ogrinfo does not work with this kind of zip with subdirectories

ogrinfo /vsizip/test.zip
FAILURE:
Unable to open datasource `/vsizip/test.zip' with the following drivers.

Is this worth making a feature request or is there some good reason why ogrinfo cannot read the file list from zip while gdalinfo can?

-Jukka Rahkonen-

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20200408/207c621c/attachment.html>