[gdal-dev] gdalwarp Resampling Input Raster Nodata Setting
Rahkonen Jukka
jukka.rahkonen at maanmittauslaitos.fi
Mon Jun 27 23:26:59 PDT 2022
Hi,
You are reading the documentation optimistically. The -srsnodata does accept a list but the list can only be a one nodata value per band.
“Set nodata masking values for input bands (different values can be supplied for each band).” In the GDAL datamodel the nodata in rasterband is a single value https://gdal.org/api/gdalrasterband_cpp.html. In addition, all image formats do not support having a separate nodata value for each band. GeoTIFF for example is using the same one nodata value for all bands https://gdal.org/drivers/raster/gtiff.html “GDAL stores band nodata value in the non standard TIFFTAG_GDAL_NODATA ASCII tag (code 42113) for files created with the default profile GDALGeoTIFF. Note that all bands must use the same nodata value.”
There is a similar question in gis.stackexchange from couple of years ago https://gis.stackexchange.com/questions/380092/gdalwarp-assigning-more-than-one-nodata-value with an advice about how to update the raster with Python to have a single value for nodata.
-Jukka Rahkonen-
Lähettäjä: Chao Li <chaoli0394 at gmail.com>
Lähetetty: tiistai 28. kesäkuuta 2022 2.58
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
Kopio: gdal-dev at lists.osgeo.org
Aihe: Re: [gdal-dev] gdalwarp Resampling Input Raster Nodata Setting
Hi,
I read the gdalwarp synopsis: https://gdal.org/programs/gdalwarp.html<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgdal.org%2Fprograms%2Fgdalwarp.html&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7fbed8ed633f467be75a08da5898e96a%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C637919711038387709%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=olfoXnii%2FhSi6RHqcF6c%2BCfM78eFEAeRG1hJNK2VSMA%3D&reserved=0>
I find the multi values can be used, but it has to write in -srcnodata "32760 32761 32762 32763 32764 32765 32766 32767".
However, it does not work. Does anyone know how to figure it out?
Thanks
Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> 于2022年6月27日周一 20:11写道：
Hi,
Your idea is OK in theory but GeoTIFF supports only one nodata value. This is not relevant with your single band data, but it may be good to know that in GeoTIFF the nodata value is also the same for all bands. Therefore with GeoTIFF it is not possible to make for example blue 0 0 255 nodata in a RGB image even it could be handy because natural aerial and satellite images tend to have 0 0 0 and 255 255 255 also in the data pixels but not pure blue pixels.
A good solution would be to polygonise areas with pixel value >=32760 and create and add an alpha band or mask band into the image. An easier way is to update the raster with gdal_calc https://gdal.org/programs/gdal_calc.html<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgdal.org%2Fprograms%2Fgdal_calc.html&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7fbed8ed633f467be75a08da5898e96a%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C637919711038387709%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YQ0nkJ%2BQbbXSzsdNj7z6Bj2FOoU7v7KOeQqkeCWKH8E%3D&reserved=0> and update all pixel values >=32760 into single value 32767.
-Jukka Rahkonen-
Lähettäjä: Chao Li <chaoli0394 at gmail.com<mailto:chaoli0394 at gmail.com>>
Lähetetty: maanantai 27. kesäkuuta 2022 13.55
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Kopio: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: Re: [gdal-dev] gdalwarp Resampling Input Raster Nodata Setting
Hi,
You are right but when I try to check the data processing, there is no any compression. On my computer, it is similar to your situation. I guess most nodata values are over 32760, so maybe I can list all of them.
-srcnodata <32760 32761 32762 32763 32764 32765 32766 32767>. Is this the right way?
Thank you.
Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> 于2022年6月27日周一 18:14写道：
Hi,
Please send answers as reply-to-all so they will go also to the gdal-dev mailing list.
There is something odd in the source data. As you can see from the attached image there are pixels which are close to the nodata 32767 but not exactly. Huge areas have value 32766 but here and there I can also see values 32765, 32762 etc. The point in the image is at EPSG:4326 97.30902,5.18109.
I think that the source data that you clipped for the test is somehow corrupted. Could you check if you can find similar pixel values from the original image? The sample is LZW compressed but could it be that at some moment the image has been compressed with JPEG or some other lossy method?
-Jukka Rahkonen-
Lähettäjä: Chao Li <chaoli0394 at gmail.com<mailto:chaoli0394 at gmail.com>>
Lähetetty: maanantai 27. kesäkuuta 2022 11.29
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Aihe: Re: [gdal-dev] gdalwarp Resampling Input Raster Nodata Setting
Hi,
Now, seemingly, it works. I am also able to get output raster, but once you visualize this raster you will find that the boundaries between value and nodata zones are abnormal. There might be over 20,000.
However the valid values are no more than 1000. That’s my problem.
Could please check whether you have the same problem?
Thank you.
Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>于2022年6月27日 周一17:17写道：
Hi,
It seems to work for me with GDAL 3.6.0dev on Windows.
gdalwarp -t_srs EPSG:4326 -r average -ot float64 -multi MOD17A2H_GPP_2020_001.tif mod64m.tif
Creating output file that is 2969P x 2572L.
Processing MOD17A2H_GPP_2020_001.tif [1/1] : 0Using internal nodata values (e.g. 32767) for image MOD17A2H_GPP_2020_001.tif.
Copying nodata values from source MOD17A2H_GPP_2020_001.tif to destination mod64m.tif.
...10...20...30...40...50...60...70...80...90...100 - done.
gdalinfo mod64m.tif
Driver: GTiff/GeoTIFF
Files: mod64m.tif
Size is 2969, 2572
Coordinate System is:
GEOGCRS["WGS 84",
…
Band 1 Block=2969x1 Type=Float64, ColorInterp=Gray
NoData Value=32767
I don’t know why nodata is lost for you. Have you already tried to set the output nodata explicitly with -dstnodata 32767 as a workaround?
-Jukka Rahkonen-
Lähettäjä: Chao Li <chaoli0394 at gmail.com<mailto:chaoli0394 at gmail.com>>
Lähetetty: maanantai 27. kesäkuuta 2022 10.50
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Aihe: Re: [gdal-dev] gdalwarp Resampling Input Raster Nodata Setting
Dear Ms./Mr. Jukka
Thank you for your reply. I try to delete the -srcnodata 32767, but it still doesn't work.
I have attach a part of raster, in this email.
Thanks for your help
Best regards
Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> 于2022年6月27日周一 14:52写道：
Hi,
The right syntax is average -srcnodata 32767 but you should not need to use it at all because GDAL recognizes nodata from your input automatically as you can see from the gdalinfo report. Could you clip and share a small sample from your source data so that some nodata is included? You can use gdal_translate with -srswin or -projwin for that.
-Jukka Rahkonen-
Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> Puolesta Chao Li
Lähetetty: maanantai 27. kesäkuuta 2022 4.09
Vastaanottaja: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: [gdal-dev] gdalwarp Resampling Input Raster Nodata Setting
Dear GDAL experts,
I have a problem using gdalwarp.
I want to reproject a raster and change its resolution by the average method.
This is the gdalinfo of the raster
[q70176a at ito-2 ~]$ gdalinfo DP12/11_RawData/02_GGP/Gross_PP_8Days_500m_v6/GPP/MOD17A2H_GPP_2016_001.tif Driver: GTiff/GeoTIFF
Files: DP12/11_RawData/02_GGP/Gross_PP_8Days_500m_v6/GPP/MOD17A2H_GPP_2016_001.tif
Size is 86400, 31200
Coordinate System is:
PROJCRS["unknown",
BASEGEOGCRS["unknown",
DATUM["unknown",
ELLIPSOID["unknown",6371007.181,0,
LENGTHUNIT["metre",1,
ID["EPSG",9001]]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]]],
CONVERSION["Sinusoidal",
METHOD["Sinusoidal"],
PARAMETER["Longitude of natural origin",0,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["False easting",0,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",0,
LENGTHUNIT["metre",1],
ID["EPSG",8807]]],
CS[Cartesian,2],
AXIS["easting",east,
ORDER[1],
LENGTHUNIT["metre",1,
ID["EPSG",9001]]],
AXIS["northing",north,
ORDER[2],
LENGTHUNIT["metre",1,
ID["EPSG",9001]]]]
Data axis to CRS axis mapping: 1,2
Origin = (-20015109.353999998420477,7783653.637667000293732)
Pixel Size = (463.312716527771215,-463.312716527787586)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left (-20015109.354, 7783653.638) (166d17' 5.25"W, 70d 0' 0.00"N)
Lower Left (-20015109.354,-6671703.118) ( 0d 0' 0.00"E, 60d 0' 0.00"S)
Upper Right (20015109.354, 7783653.638) (166d17' 5.25"E, 70d 0' 0.00"N)
Lower Right (20015109.354,-6671703.118) ( 0d 0' 0.00"W, 60d 0' 0.00"S)
Center ( -0.000, 555975.260) ( 0d 0' 0.00"W, 5d 0' 0.00"N)
Band 1 Block=86400x1 Type=Int16, ColorInterp=Gray
NoData Value=32767
This is the gdalwarp commend:
[q70176a at ito-2 ~]$ gdalwarp -t_srs EPSG:4326 -te -180 -90 180 90 -tr 0.1 0.1 -ot Float64 -r average -srcnodata "32767" -multi -overwrite DP12/11_RawData/02_GGP/Gross_PP_8Days_500m_v6/GPP/MOD17A2H_GPP_2016_001.tif DP12/11_RawData/02_GGP/Gross_PP_8Days_500m_v6/GPP_01_test/test2.tif
I also tried
[q70176a at ito-2 ~]$ gdalwarp -t_srs EPSG:4326 -te -180 -90 180 90 -tr 0.1 0.1 -ot Float64 -r average -srcnodata 32767 -multi -overwrite DP12/11_RawData/02_GGP/Gross_PP_8Days_500m_v6/GPP/MOD17A2H_GPP_2016_001.tif DP12/11_RawData/02_GGP/Gross_PP_8Days_500m_v6/GPP_01_test/test2.tif
However, the 32767 is not set to be nodata, and be averaged in the output raster.
How can I set this?
Additionally, I am using a HPC with multinodes. With 8 nodes, I have 288. If I set -multi and -wo NUM_THREADS=val/ALL_CPUS, will all that 288 run together?
Since there nothing returns, I really do not know.
Best regards.
--
------------------------------------------------------------------------------------
Mike Li
Department of Urban and Environmental Engineering
Graduate School of Engineering
Kyushu University
744 Motooka, Nishi-ku, Fukuoka 819-0395 Japan
Tel: 090-8304-8953
E-mail: chaoli0394 at gmail.com<mailto:chaoli0394 at gmail.com>
-------------------------------------------------------------------------------------
李潮（リ チョウ）
九州大学 大学院 工学府 都市環境システム工学専攻
都市工学研究室
-------------------------------------------------------------------------------------
--
------------------------------------------------------------------------------------
Mike Li
Department of Urban and Environmental Engineering
Graduate School of Engineering
Kyushu University
744 Motooka, Nishi-ku, Fukuoka 819-0395 Japan
Tel: 090-8304-8953
E-mail: chaoli0394 at gmail.com<mailto:chaoli0394 at gmail.com>
-------------------------------------------------------------------------------------
李潮（リ チョウ）
九州大学 大学院 工学府 都市環境システム工学専攻
都市工学研究室
-------------------------------------------------------------------------------------
--
------------------------------------------------------------------------------------
Mike Li
Department of Urban and Environmental Engineering
Graduate School of Engineering
Kyushu University
744 Motooka, Nishi-ku, Fukuoka 819-0395 Japan
Tel: 090-8304-8953
E-mail: chaoli0394 at gmail.com<mailto:chaoli0394 at gmail.com>
-------------------------------------------------------------------------------------
李潮（リ チョウ）
九州大学 大学院 工学府 都市環境システム工学専攻
都市工学研究室
-------------------------------------------------------------------------------------
--
------------------------------------------------------------------------------------
Mike Li
Department of Urban and Environmental Engineering
Graduate School of Engineering
Kyushu University
744 Motooka, Nishi-ku, Fukuoka 819-0395 Japan
Tel: 090-8304-8953
E-mail: chaoli0394 at gmail.com<mailto:chaoli0394 at gmail.com>
-------------------------------------------------------------------------------------
李潮（リ チョウ）
九州大学 大学院 工学府 都市環境システム工学専攻
都市工学研究室
-------------------------------------------------------------------------------------
--
------------------------------------------------------------------------------------
Mike Li
Department of Urban and Environmental Engineering
Graduate School of Engineering
Kyushu University
744 Motooka, Nishi-ku, Fukuoka 819-0395 Japan
Tel: 090-8304-8953
E-mail: chaoli0394 at gmail.com<mailto:chaoli0394 at gmail.com>
-------------------------------------------------------------------------------------
李潮（リ チョウ）
九州大学 大学院 工学府 都市環境システム工学専攻
都市工学研究室
-------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220628/74b9ae70/attachment-0001.htm>
More information about the gdal-dev
mailing list