[gdal-dev] Standardize gdal-utils scripts return code for "no arguments"
Matt.Wilkie at yukon.ca
Matt.Wilkie at yukon.ca
Wed Apr 27 10:26:23 PDT 2022
Thank you for the study path Idan, much appreciated and I will use it.
> Bottom line, returncode 2 is from the very last line in argparse.py,
> the builtin ArgumentParser (function ArgumentParser.error()):
> `self.exit(2, _('%(prog)s: error: %(message)s\n') % args)`
Hmmm. Since return 2 is the default for argparse then it seems to we should do the same for all the python utils?
-Matt
From: Idan Miara <idanmiara at outlook.com> On Behalf Of Idan Miara
Sent: April 27, 2022 6:24 AM
To: Matt.Wilkie <Matt.Wilkie at yukon.ca>
Cc: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Standardize gdal-utils scripts return code for "no arguments"
You don't often get email from idan at miara.com<mailto:idan at miara.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Hi Matt,
Since you mentioned that you don't understand how classes work (i.e. super function)
I would highly recommend spending some time to go over some basic Python OOP, it's not complicated and I'm confident that it would be very useful.
From a quick search, maybe this link would be useful https://realpython.com/python3-object-oriented-programming/<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frealpython.com%2Fpython3-object-oriented-programming%2F&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lldjRlTrTEnWnk4lhViopt%2B2v4cmk1li2JXWDGySfO0%3D&reserved=0> but there are many other resources.
super() is mainly used in Python as a way to call the superclass' function when that you subclass (https://realpython.com/python-super/<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frealpython.com%2Fpython-super%2F&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KZ%2B4PamCCg0v1nA2hFrq1LsSeBXAN7oKCKWgLpyW%2FKU%3D&reserved=0>)
For example, in `GDALArgumentParser.__init__()` I call `super().__init__` in order to access ArgumentParser's constructor (__init__).
GDALArgumentParser is defined as `class GDALArgumentParser(argparse.ArgumentParser)`.
So GDALArgumentParser is a superclass of the built in class argparse.ArgumentParser.
The objective of this class is mainly to standardize our usage of ArgumentParser by adding some functionality that was implemented in many different utils on top of ArgumentParser.
As GDALArgumentParser is a thin superclass of ArgumentParser, If you want to better understand GDALArgumentParser I advise you to first read the docs here https://docs.python.org/3/library/argparse.html<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.python.org%2F3%2Flibrary%2Fargparse.html&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rbnjs2pgbJh5nIb8iYJE2BJQX0v65yEsopaf3iWaZ5g%3D&reserved=0>, it helped me very much when I coded GDALArgumentParser.
Bottom line, returncode 2 is from the very last line in argparse.py, the builtin ArgumentParser (function ArgumentParser.error()):
`self.exit(2, _('%(prog)s: error: %(message)s\n') % args)`
When ArgumentParser encounters an error (like calling it without arguments when arguments were required) it calls this function and exits with errorcode 2.
You could see this in action if you copy-paste and run without parameters the first example in the ArgumentParser docs (https://docs.python.org/3/library/argparse.html<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.python.org%2F3%2Flibrary%2Fargparse.html&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rbnjs2pgbJh5nIb8iYJE2BJQX0v65yEsopaf3iWaZ5g%3D&reserved=0>).
The way I found this was just using the debugger in PyCharm and stepping in until I found out what's exits the run
(If an exception is raised it makes it much easier as you can easily follow the call stack)
pyi are stub files (https://peps.python.org/pep-0484/<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeps.python.org%2Fpep-0484%2F&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YgIYRRFyd1rKQnLQukduYh71LiPtV5XltFow%2B31%2BdkU%3D&reserved=0>) and you probably encountered them when digging into Python's standard library which is not written in pure Python.
Best,
Idan
On Tue, 26 Apr 2022 at 23:06, <Matt.Wilkie at yukon.ca<mailto:Matt.Wilkie at yukon.ca>> wrote:
Here is the path I'm following in trying to deconstruct GDALArgumentParser.
With file = pc2rgb.py:
r = subprocess.run([sys.executable,
file],
shell=True,
capture_output=True,
text=True,
)
if debug:
print(f'returncode: {r.returncode}')
I get:
returncode: 2
Digging in to pct2rgb.py I see main() calling PCT2RGB class, calling pct2rgb function, which should error out with an exception.:
def main(argv=sys.argv):
return PCT2RGB().main(argv)
def doit(self, **kwargs):
return pct2rgb(**kwargs)
def doit(**kwargs):
try:
ds = pct2rgb(**kwargs)
return ds, 0
except:
return None, 1
def pct2rgb(src_filename: PathLikeOrStr, pct_filename: Optional[PathLikeOrStr], dst_filename: PathLikeOrStr,
band_number: int = 1, out_bands: int = 3, driver_name: Optional[str] = None):
# Open source file
src_ds = open_ds(src_filename)
if src_ds is None:
raise Exception(f'Unable to open {src_filename} ')
...
So I'm thinking that while PCT2RGB class is invoking GDALArgumentParser it's exiting early somehow with this 2 return code.
def get_parser(self, argv) -> GDALArgumentParser:
...
return parser
I look at auxillary/gdal_argparse.py: class GDALArgumentParser(argparse.ArgumentParser) and I see... well a bunch of stuff I only apprehend the shadowy outlines of. Looking at the return statements doesn't add light. (It doesn't help that I don't understand how classes work. There's this thing called super() but it's not defined anywhere. Shouldn't that cause an error? Oh it's from stdlib builtins.pyi. What the heck is a pyi? "...It is pitch black. You are likely to be eaten by a grue.")
» findstr "return" ..\auxiliary\gdal_argparse.py
return super().parse_args(args=args, **kwargs)
return shlex.split(arg_line, comments=True)
return self._parser
return kwargs
return kwargs
return 0
return 1
return epilog or None
So my question is: where is returncode: 2 coming from?
-Matt
From: Matt.Wilkie
Sent: April 26, 2022 1:03 PM
To: 'gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>' <gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>>
Subject: RE: Standardize gdal-utils scripts return code for "no arguments"
Hi Folks,
I've converted all the scripts that were using -1 to 1. However when I started looking at the ones returning 2 it became less clear what to do. Excepting gdal2tiles all of them are using GDALArgumentParser and I don't see where the return value is being set. Thinking this might mean the mainline utils might be using 2 for no args I checked gdal_translate and gdalwarp, but no, both of those use 1.
The scripts that return 2 are:
osgeo_utils\gdal2xyz.py
osgeo_utils\gdal_calc.py
osgeo_utils\gdal_fillnodata.py
osgeo_utils\gdal_polygonize.py
osgeo_utils\pct2rgb.py
osgeo_utils\rgb2pct.py
osgeo_utils\samples\gdallocationinfo.py
I haven’t been able to figure out the GDALArgumentParser parser class so I embarked on making everything else use 2 instead of 1. I'm questioning the wisdom of that at the moment since so many files are touched. However the process has forced me to look more closely at the many scripts and start to internalize the various patterns they use. This has been worthwhile even if the approach might get abandoned. The “make everything return 2” effort is in branch patch-5561-ret2<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaphew%2Fgdal%2Ftree%2Fpatch-5561-ret2&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G9NH8tqppVyTgxx6ubzvxZEtjCLtytJPOqcId7s5DDM%3D&reserved=0>.
The next message will have more details on how I’ve tried and failed to understand GDALArgumentParser.
-Matt
From: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> On Behalf Of Matt.Wilkie at yukon.ca<mailto:Matt.Wilkie at yukon.ca>
Sent: April 4, 2022 4:07 PM
To: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Subject: [gdal-dev] Standardize gdal-utils scripts return code for "no arguments"
Hi folks,
I’m working on “Standardize gdal-utils scripts return codes #5561” for all the scripts in swig/python/gdal-utils. Currently the scripts do not return the same status code for "was run without arguments".
It would be good for the same code meant the same thing across all the scripts in the package. Given that most of the scripts use `1` now, and that this is in line with sys,exit() docs I think it makes sense to make 1 the new standard.
At present we have (return_code, num scripts with that code):
0: 30
1: 56
2: 8
-1: 9
-1 is a special case. In the script code it’s written as `return -1` but subprocess.run() captures it as `4294967295`. Another special case is `gdal_auth.py` sample which with no arguments spawns a web authentication page in browser.
Changing the return code will mean anyone who is relying on those in their own scripts or programs will need to adjust. This seems to be a small price for the gain in harmonization across the utilities, to me. Your thoughts?
[0]: https://github.com/OSGeo/gdal/issues/5561<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOSGeo%2Fgdal%2Fissues%2F5561&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ivFPApdA%2FDSKBptJoFtavUj5DkwTyPrHomPLt2YqKiM%3D&reserved=0>
[1]: https://docs.python.org/3/library/sys.html#sys.exit<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.python.org%2F3%2Flibrary%2Fsys.html%23sys.exit&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=feIqF2ldsa%2FZzWXBWGPY6v4js3lkGxPptty9vdebozY%3D&reserved=0>, “Unix programs generally use 2 for command line syntax errors and 1 for all other kind of errors”
Matt Wilkie
Geomatics Developer & Administrator
Environment | Technology, Innovation and Mapping
T 867-667-8133 | Yukon.ca<https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fyukon.ca%2F&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=AUpiUIH7jENlcxo%2BIY%2BAx9WVdHeGvRJzzzorrFR3LE4%3D&reserved=0>
Hours: 08:30-16:30, Mon-Wed: Office, Thu: Remote, Fri: Away.
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7CMatt.Wilkie%40yukon.ca%7C44e0362e122546ade38008da28514d88%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637866628546448376%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Kz6E8GoJl5e%2BO5dfVAwVD9nm9wAZbvFBoXh0H5ZiQ3g%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220427/8d0fc873/attachment-0001.html>
More information about the gdal-dev
mailing list