<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><div dir="ltr">Hi Matt,<div><br></div><div>Since you mentioned that you don't understand how classes work (i.e. super function) <br></div><div><div>I would highly recommend spending some time to go over some basic Python OOP, it's not complicated and I'm confident that it would be very useful.</div><div>From a quick search, maybe this link would be useful <a href="https://realpython.com/python3-object-oriented-programming/">https://realpython.com/python3-object-oriented-programming/</a> but there are many other resources.</div><div></div><div>super() is mainly used in Python as a way to call the superclass' function when that you subclass (<a href="https://realpython.com/python-super/">https://realpython.com/python-super/</a>)<br></div><div>For example, in `GDALArgumentParser.__init__()` I call `super().__init__` in order to access ArgumentParser's constructor (__init__).</div></div><div><div><br></div></div><div>GDALArgumentParser is defined as `class GDALArgumentParser(argparse.ArgumentParser)`.</div><div>So GDALArgumentParser is a superclass of the built in class argparse.ArgumentParser.</div><div>The objective of this class is mainly to standardize our usage of ArgumentParser by adding some functionality that was implemented in many different utils on top of ArgumentParser.<br></div><div><div>As GDALArgumentParser is a thin superclass of ArgumentParser, If you want to better understand GDALArgumentParser I advise you to first read the docs here <a href="https://docs.python.org/3/library/argparse.html">https://docs.python.org/3/library/argparse.html</a>, it helped me very much when I coded GDALArgumentParser.</div></div><div><br></div><div><div><div>Bottom line, returncode 2 is from the very last line in argparse.py, the builtin ArgumentParser (function ArgumentParser.error()):</div><div>`self.exit(2, _('%(prog)s: error: %(message)s\n') % args)`<br></div><div>When ArgumentParser encounters an error (like calling it without arguments when arguments were required) it calls this function and exits with errorcode 2.<br></div><div>You could see this in action if you copy-paste and run without parameters the first example in the ArgumentParser docs (<a href="https://docs.python.org/3/library/argparse.html">https://docs.python.org/3/library/argparse.html</a>).</div></div><div><br></div><div>The way I found this was just using the debugger in PyCharm and stepping in until I found out what's exits the run </div><div>(If an exception is raised it makes it much easier as you can easily follow the call stack) </div><div><br></div><div><div>pyi are stub files (<a href="https://peps.python.org/pep-0484/">https://peps.python.org/pep-0484/</a>) and you probably encountered them when digging into Python's standard library which is not written in pure Python.<br></div><div><br></div><div>Best,</div><div>Idan</div><div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 26 Apr 2022 at 23:06, <<a href="mailto:Matt.Wilkie@yukon.ca">Matt.Wilkie@yukon.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-CA">
<div class="gmail-m_7422459703816551646WordSection1">
<p class="MsoNormal"><i><span style="font-size:12pt;font-family:"Times New Roman",serif">Here is the path I'm following in trying to deconstruct
</span></i><i><span style="color:rgb(31,73,125)">GDALArgumentParser.</span></i><i><span style="font-size:12pt;font-family:"Times New Roman",serif"><u></u><u></u></span></i></p>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif">With
</span><span style="font-size:10pt;font-family:"Courier New"">file = pc2rgb.py</span><span style="font-size:12pt;font-family:"Times New Roman",serif">:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> r = subprocess.run([sys.executable,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> file],<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> shell=True,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> capture_output=True,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> text=True,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> )<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> if debug:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> print(f'returncode: {r.returncode}')<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif">I get:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New"">returncode: 2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif">Digging in to pct2rgb.py I see main() calling PCT2RGB class, calling pct2rgb function,
which should error out with an exception.:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New"">def main(argv=sys.argv):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return PCT2RGB().main(argv)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> def doit(self, **kwargs):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return pct2rgb(**kwargs)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New"">def doit(**kwargs):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> try:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> ds = pct2rgb(**kwargs)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return ds, 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> except:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return None, 1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New"">def pct2rgb(src_filename: PathLikeOrStr, pct_filename: Optional[PathLikeOrStr], dst_filename: PathLikeOrStr,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> band_number: int = 1, out_bands: int = 3, driver_name: Optional[str] = None):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> # Open source file<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> src_ds = open_ds(src_filename)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> if src_ds is None:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> raise Exception(f'Unable to open {src_filename} ')<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> ...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif">So I'm thinking that while PCT2RGB class is invoking GDALArgumentParser it's exiting
early somehow with this 2 return code.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New"">def get_parser(self, argv) -> GDALArgumentParser:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> ...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return parser<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif">I look at
</span><span style="font-size:10pt;font-family:"Courier New"">auxillary/gdal_argparse.py</span><span style="font-size:12pt;font-family:"Times New Roman",serif">:
</span><span style="font-size:10pt;font-family:"Courier New"">class GDALArgumentParser(argparse.ArgumentParser)</span><span style="font-size:12pt;font-family:"Times New Roman",serif"> and I see... well
a bunch of stuff I only apprehend the shadowy outlines of. Looking at the </span>
<span style="font-size:10pt;font-family:"Courier New"">return</span><span style="font-size:12pt;font-family:"Times New Roman",serif"> statements doesn't add light. (It doesn't help that I don't understand
how classes work. There's this thing called </span><span style="font-size:10pt;font-family:"Courier New"">super()</span><span style="font-size:12pt;font-family:"Times New Roman",serif"> but it's not
defined anywhere. Shouldn't that cause an error? Oh it's from stdlib builtins.pyi. What the heck is a pyi?
<i>"...It is pitch black. You are likely to be eaten by a grue."</i>)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New"">» findstr "return" ..\auxiliary\gdal_argparse.py<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return super().parse_args(args=args, **kwargs)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return shlex.split(arg_line, comments=True)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return self._parser<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return kwargs<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return kwargs<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return 1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Courier New""> return epilog or None<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"Times New Roman",serif">So my question is: where is
</span><span style="font-size:10pt;font-family:"Courier New"">returncode: 2</span><span style="font-size:12pt;font-family:"Times New Roman",serif"> coming from?
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;color:rgb(31,73,125)">-Matt</span><i><span style="font-size:10pt;color:rgb(31,73,125)"><u></u><u></u></span></i></p>
</div>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Matt.Wilkie
<br>
<b>Sent:</b> April 26, 2022 1:03 PM<br>
<b>To:</b> '<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>' <<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>><br>
<b>Subject:</b> RE: Standardize gdal-utils scripts return code for "no arguments"<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Hi Folks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">I've converted all the scripts that were using -1 to 1. However when I started looking at the ones returning 2 it became less clear what to do. Excepting
<i>gdal2tiles</i> all of them are using GDALArgumentParser and I don't see where the return value is being set. Thinking this might mean the mainline utils might be using 2 for no args I checked
<i>gdal_translate</i> and <i>gdalwarp</i>, but no, both of those use 1.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">The scripts that return 2 are:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New"">osgeo_utils\gdal2xyz.py<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New"">osgeo_utils\gdal_calc.py<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New"">osgeo_utils\gdal_fillnodata.py<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New"">osgeo_utils\gdal_polygonize.py<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New"">osgeo_utils\pct2rgb.py<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New"">osgeo_utils\rgb2pct.py<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt"><span style="font-size:10pt;font-family:"Courier New"">osgeo_utils\samples\gdallocationinfo.py<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">I haven’t been able to figure out the GDALArgumentParser parser class so I embarked on making everything else use 2 instead of 1. I'm questioning the wisdom of that at the moment since so many files are touched.
However the process has forced me to look more closely at the many scripts and start to internalize the various patterns they use. This has been worthwhile even if the approach might get abandoned. The “make everything return 2” effort is in branch
<a href="https://github.com/maphew/gdal/tree/patch-5561-ret2" target="_blank">patch-5561-ret2</a>.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">The next message will have more details on how I’ve tried and failed to understand GDALArgumentParser.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;color:rgb(31,73,125)">-Matt</span><i><span style="font-size:10pt;color:rgb(31,73,125)"><u></u><u></u></span></i></p>
</div>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> gdal-dev <<a href="mailto:gdal-dev-bounces@lists.osgeo.org" target="_blank">gdal-dev-bounces@lists.osgeo.org</a>>
<b>On Behalf Of </b><a href="mailto:Matt.Wilkie@yukon.ca" target="_blank">Matt.Wilkie@yukon.ca</a><br>
<b>Sent:</b> April 4, 2022 4:07 PM<br>
<b>To:</b> <a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<b>Subject:</b> [gdal-dev] Standardize gdal-utils scripts return code for "no arguments"<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi folks,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’m working on “Standardize gdal-utils scripts return codes #5561” for all the scripts in swig/python/gdal-utils. Currently the scripts do not return the same status code for "was run without arguments".
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">It would be good for the same code meant the same thing across all the scripts in the package. Given that most of the scripts use `1` now, and that this is in line with sys,exit() docs I think it makes sense to make 1 the new standard.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">At present we have (return_code, num scripts with that code):<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">0: 30<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">1: 56<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2: 8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">-1: 9<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">-1 is a special case. In the script code it’s written as `return -1` but subprocess.run() captures it as `<span style="color:black">4294967295`. Another special case is `gdal_auth.py` sample which with no arguments
spawns a web authentication page in browser.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal">Changing the return code will mean anyone who is relying on those in their own scripts or programs will need to adjust. This seems to be a small price for the gain in harmonization across the utilities, to me. Your thoughts?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">[0]: <a href="https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOSGeo%2Fgdal%2Fissues%2F5561&data=04%7C01%7Cmatt.wilkie%40yukon.ca%7Cc3e8dba890e54a879a4808da168fdceb%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637847105002756645%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=cJaW4TxL68DJMI8KTEjVjQCIzsN1Oa9ShUtILmy1%2B2g%3D&reserved=0" target="_blank">
https://github.com/OSGeo/gdal/issues/5561</a><u></u><u></u></p>
<p class="MsoNormal">[1]: <a href="https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.python.org%2F3%2Flibrary%2Fsys.html%23sys.exit&data=04%7C01%7Cmatt.wilkie%40yukon.ca%7Cc3e8dba890e54a879a4808da168fdceb%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637847105002756645%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pHA7JH0IS7tjePzMXO1MzvO6svuPVHVR9uQgPqxOtMY%3D&reserved=0" target="_blank">
https://docs.python.org/3/library/sys.html#sys.exit</a>, “Unix programs generally use 2 for command line syntax errors and 1 for all other kind of errors”<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-top:4pt;line-height:115%;vertical-align:middle">
<b><span lang="EN-US" style="font-size:10pt;line-height:115%">Matt Wilkie</span></b><span lang="EN-US" style="font-size:10pt;line-height:115%"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt">Geomatics Developer & Administrator<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt">Environment
<span style="color:rgb(46,116,181)">|</span> Technology, Innovation and Mapping<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt">T 867-667-8133
<span style="color:rgb(46,116,181)">|</span> </span><u><span style="font-size:10pt"><a href="https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fyukon.ca%2F&data=04%7C01%7Cmatt.wilkie%40yukon.ca%7Cc3e8dba890e54a879a4808da168fdceb%7C98f515313973490abb70195aa264a2bc%7C0%7C0%7C637847105002756645%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=S08L%2BQsPMBZOc2%2Bb3wnWS8W08a0ejZpHRC1QAMfFEyk%3D&reserved=0" target="_blank"><span lang="EN-US" style="color:windowtext">Yukon.ca</span></a><u></u><u></u></span></u></p>
<p class="MsoNormal"><i><span style="font-size:10pt">Hours: 08:30-16:30, Mon-Wed: Office, Thu: Remote, Fri: Away.<u></u><u></u></span></i></p>
</div>
</div>
</div>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</blockquote></div></div>