[gdal-dev] Standardize gdal-utils scripts return code for "no arguments"
Idan Miara
idan at miara.com
Wed Apr 27 06:24:20 PDT 2022
Hi Matt,
Since you mentioned that you don't understand how classes work (i.e. super
function)
I would highly recommend spending some time to go over some basic Python
OOP, it's not complicated and I'm confident that it would be very useful.
>From a quick search, maybe this link would be useful
https://realpython.com/python3-object-oriented-programming/ but there are
many other resources.
super() is mainly used in Python as a way to call the superclass' function
when that you subclass (https://realpython.com/python-super/)
For example, in `GDALArgumentParser.__init__()` I call `super().__init__`
in order to access ArgumentParser's constructor (__init__).
GDALArgumentParser is defined as `class
GDALArgumentParser(argparse.ArgumentParser)`.
So GDALArgumentParser is a superclass of the built in class
argparse.ArgumentParser.
The objective of this class is mainly to standardize our usage of
ArgumentParser by adding some functionality that was implemented in many
different utils on top of ArgumentParser.
As GDALArgumentParser is a thin superclass of ArgumentParser, If you want
to better understand GDALArgumentParser I advise you to first read the docs
here https://docs.python.org/3/library/argparse.html, it helped me very
much when I coded GDALArgumentParser.
Bottom line, returncode 2 is from the very last line in argparse.py, the
builtin ArgumentParser (function ArgumentParser.error()):
`self.exit(2, _('%(prog)s: error: %(message)s\n') % args)`
When ArgumentParser encounters an error (like calling it without arguments
when arguments were required) it calls this function and exits with
errorcode 2.
You could see this in action if you copy-paste and run without parameters
the first example in the ArgumentParser docs (
https://docs.python.org/3/library/argparse.html).
The way I found this was just using the debugger in PyCharm and stepping in
until I found out what's exits the run
(If an exception is raised it makes it much easier as you can easily follow
the call stack)
pyi are stub files (https://peps.python.org/pep-0484/) and you probably
encountered them when digging into Python's standard library which is not
written in pure Python.
Best,
Idan
On Tue, 26 Apr 2022 at 23:06, <Matt.Wilkie at yukon.ca> wrote:
> *Here is the path I'm following in trying to deconstruct *
> *GDALArgumentParser.*
>
> With file = pc2rgb.py:
>
> r = subprocess.run([sys.executable,
>
> file],
>
> shell=True,
>
> capture_output=True,
>
> text=True,
>
> )
>
> if debug:
>
> print(f'returncode: {r.returncode}')
>
> I get:
>
> returncode: 2
>
> Digging in to pct2rgb.py I see main() calling PCT2RGB class, calling
> pct2rgb function, which should error out with an exception.:
>
> def main(argv=sys.argv):
>
> return PCT2RGB().main(argv)
>
> def doit(self, **kwargs):
>
> return pct2rgb(**kwargs)
>
> def doit(**kwargs):
>
> try:
>
> ds = pct2rgb(**kwargs)
>
> return ds, 0
>
> except:
>
> return None, 1
>
> def pct2rgb(src_filename: PathLikeOrStr, pct_filename:
> Optional[PathLikeOrStr], dst_filename: PathLikeOrStr,
>
> band_number: int = 1, out_bands: int = 3, driver_name:
> Optional[str] = None):
>
> # Open source file
>
> src_ds = open_ds(src_filename)
>
> if src_ds is None:
>
> raise Exception(f'Unable to open {src_filename} ')
>
> ...
>
> So I'm thinking that while PCT2RGB class is invoking GDALArgumentParser
> it's exiting early somehow with this 2 return code.
>
> def get_parser(self, argv) -> GDALArgumentParser:
>
> ...
>
> return parser
>
> I look at auxillary/gdal_argparse.py: class
> GDALArgumentParser(argparse.ArgumentParser) and I see... well a bunch of
> stuff I only apprehend the shadowy outlines of. Looking at the return
> statements doesn't add light. (It doesn't help that I don't understand how
> classes work. There's this thing called super() but it's not defined
> anywhere. Shouldn't that cause an error? Oh it's from stdlib builtins.pyi.
> What the heck is a pyi? *"...It is pitch black. You are likely to be
> eaten by a grue."*)
>
> » findstr "return" ..\auxiliary\gdal_argparse.py
>
> return super().parse_args(args=args, **kwargs)
>
> return shlex.split(arg_line, comments=True)
>
> return self._parser
>
> return kwargs
>
> return kwargs
>
> return 0
>
> return 1
>
> return epilog or None
>
> So my question is: where is returncode: 2 coming from?
>
>
>
> -Matt
>
>
>
> *From:* Matt.Wilkie
> *Sent:* April 26, 2022 1:03 PM
> *To:* 'gdal-dev at lists.osgeo.org' <gdal-dev at lists.osgeo.org>
> *Subject:* RE: Standardize gdal-utils scripts return code for "no
> arguments"
>
>
>
> Hi Folks,
>
>
>
> I've converted all the scripts that were using -1 to 1. However when I
> started looking at the ones returning 2 it became less clear what to do.
> Excepting *gdal2tiles* all of them are using GDALArgumentParser and I
> don't see where the return value is being set. Thinking this might mean the
> mainline utils might be using 2 for no args I checked *gdal_translate*
> and *gdalwarp*, but no, both of those use 1.
>
>
>
> The scripts that return 2 are:
>
>
>
> osgeo_utils\gdal2xyz.py
>
> osgeo_utils\gdal_calc.py
>
> osgeo_utils\gdal_fillnodata.py
>
> osgeo_utils\gdal_polygonize.py
>
> osgeo_utils\pct2rgb.py
>
> osgeo_utils\rgb2pct.py
>
> osgeo_utils\samples\gdallocationinfo.py
>
>
>
>
>
> I haven’t been able to figure out the GDALArgumentParser parser class so
> I embarked on making everything else use 2 instead of 1. I'm questioning
> the wisdom of that at the moment since so many files are touched. However
> the process has forced me to look more closely at the many scripts and
> start to internalize the various patterns they use. This has been
> worthwhile even if the approach might get abandoned. The “make everything
> return 2” effort is in branch patch-5561-ret2
> <https://github.com/maphew/gdal/tree/patch-5561-ret2>.
>
>
>
> The next message will have more details on how I’ve tried and failed to
> understand GDALArgumentParser.
>
>
>
> -Matt
>
>
>
> *From:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> *On Behalf Of *
> Matt.Wilkie at yukon.ca
> *Sent:* April 4, 2022 4:07 PM
> *To:* gdal-dev at lists.osgeo.org
> *Subject:* [gdal-dev] Standardize gdal-utils scripts return code for "no
> arguments"
>
>
>
> Hi folks,
>
>
>
> I’m working on “Standardize gdal-utils scripts return codes #5561” for all
> the scripts in swig/python/gdal-utils. Currently the scripts do not return
> the same status code for "was run without arguments".
>
>
>
> It would be good for the same code meant the same thing across all the
> scripts in the package. Given that most of the scripts use `1` now, and
> that this is in line with sys,exit() docs I think it makes sense to make 1
> the new standard.
>
>
>
> At present we have (return_code, num scripts with that code):
>
>
>
> 0: 30
>
> 1: 56
>
> 2: 8
>
> -1: 9
>
>
>
> -1 is a special case. In the script code it’s written as `return -1` but
> subprocess.run() captures it as `4294967295`. Another special case is
> `gdal_auth.py` sample which with no arguments spawns a web authentication
> page in browser.
>
>
>
> Changing the return code will mean anyone who is relying on those in their
> own scripts or programs will need to adjust. This seems to be a small price
> for the gain in harmonization across the utilities, to me. Your thoughts?
>
>
>
>
>
> [0]: https://github.com/OSGeo/gdal/issues/5561
> <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOSGeo%2Fgdal%2Fissues%2F5561&data=04%7C01%7Cmatt.wilkie%40yukon.ca%7Cc3e8dba890e54a879a4808da168fdceb%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637847105002756645%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cJaW4TxL68DJMI8KTEjVjQCIzsN1Oa9ShUtILmy1%2B2g%3D&reserved=0>
>
> [1]: https://docs.python.org/3/library/sys.html#sys.exit
> <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.python.org%2F3%2Flibrary%2Fsys.html%23sys.exit&data=04%7C01%7Cmatt.wilkie%40yukon.ca%7Cc3e8dba890e54a879a4808da168fdceb%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637847105002756645%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pHA7JH0IS7tjePzMXO1MzvO6svuPVHVR9uQgPqxOtMY%3D&reserved=0>,
> “Unix programs generally use 2 for command line syntax errors and 1 for all
> other kind of errors”
>
>
>
> *Matt Wilkie*
>
> Geomatics Developer & Administrator
>
> Environment | Technology, Innovation and Mapping
>
> T 867-667-8133 | *Yukon.ca
> <https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fyukon.ca%2F&data=04%7C01%7Cmatt.wilkie%40yukon.ca%7Cc3e8dba890e54a879a4808da168fdceb%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637847105002756645%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=S08L%2BQsPMBZOc2%2Bb3wnWS8W08a0ejZpHRC1QAMfFEyk%3D&reserved=0>*
>
> *Hours: 08:30-16:30, Mon-Wed: Office, Thu: Remote, Fri: Away.*
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220427/610b9241/attachment-0001.html>
More information about the gdal-dev
mailing list