<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you for the reply.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Nikos,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The path is correct, the space was an error in my redaction. <br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
James,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I tried to run a GDALINFO in my Linux command line using the GDAL I configured and installed and I am not able to open the raster, even trying to read it from local:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span> <b>-bash-4.2$ gdalinfo /vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif<br>
</b></span>
<div><b> ERROR 4: /vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif: No such file or directory<br>
</b></div>
<div><b> gdalinfo failed - unable to open '/vsihdfs/file:/scratch/zherena/test/write/data/rasters/hawaii.tif'.<br>
</b></div>
<b><span></span></b></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If I try the same commad withouth /vsihdfs/file:/... it works fine.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I also tried:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span> <b> -bash-4.2$ gdalinfo --version<br>
</b></span><b><span> GDAL 2.3.2, released 2018/09/21</span> </b><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Version says 2.3.2 but libraries say: libgdal.so.20.4.2 . I am not sure if I got the latest code, this is the first time I compile it myself, I used this link to download source code:
<a href="http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz" id="LPlnk713058">
http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz</a><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
And I have a doubt, when configuring I used: <b>--with-hdfs=yes</b>, made another using:
<b>--with-hdfs = /scratch/zherena/gdal/hadoop/lib/native</b>. Both tries did not work. I am not really sure what should I pass as a parameter to --with-hdfs?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This is the list of steps I used to build, I am using Java in my application where I plan to use gdal:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<ul>
<li> make clean distclean(per your suggestion)</li><li> <span>./configure --prefix=/scratch/zherena/gdal/build/gdal-2.3.2/outputb/ --with-complete=yes --with-java=yes --with-swig-java=yes --with-hdfs=/scratch/zherena/gdal/hadoop/lib/native --with-curl=/usr/bin/curl-config</span></li><li><span>make</span></li><li><span>make install</span></li><li><span>cd /swig/java</span></li><li><span>make veryclean</span></li><li><span>make</span></li><li><span>cp *.so ../../outputb/lib/</span></li><li><span>cp *.jar ../../outputb/lib/<br>
</span><span></span><br>
</li></ul>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Maybe I am missing something during compile?<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you!!<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Zazhil-ha<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> gdal-dev <gdal-dev-bounces@lists.osgeo.org> on behalf of James McClain <james.mcclain@gmail.com><br>
<b>Sent:</b> Thursday, November 22, 2018 8:27 PM<br>
<b>Cc:</b> gdal-dev@lists.osgeo.org<br>
<b>Subject:</b> Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hello,</div>
<div><br>
</div>
<div>I am the author of the vsihdfs code, I am ready and willing to help.</div>
<div><br>
</div>
<div>I just rebuilt it from current master and was able to successfully open an dataset via an HDFS URI with the GDAL Python bindings. I have a few suggestions.</div>
<div><br>
</div>
<div>First, please try putting the file into a local directory and try something like `gdalinfo /vsihdfs/file:/tmp/kahoolawe.tif` to establish a baseline.<br>
</div>
<div><br>
</div>
<div>Second, if you are using the Python bindings, please make sure that they have been built and installed (and that you are using the ones that you built rather than other ones that exist on your system). Instructions for building the Python bindings can
be found here: <a href="https://trac.osgeo.org/gdal/wiki/BuildingOnUnix">https://trac.osgeo.org/gdal/wiki/BuildingOnUnix</a> .</div>
<div><br>
</div>
<div>In my case, after building and installing the library and bindings, I was able to successfully open a dataset by starting a python REPL like this:</div>
<div><br>
</div>
<div>```bash</div>
<div>export LD_LIBRARY_PATH=$HOME/local/hadoop-2.7.7/lib/native:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:$HOME/local/gdal-master-vsihdfs/lib<br>
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64<br>
export CLASSPATH=$($HOME/local/hadoop-2.7.7/bin/hadoop classpath --glob)<br>
PYTHONPATH=$HOME/local/gdal-master-vsihdfs/lib/python2.7/site-packages python<br>
</div>
<div>```</div>
<div><br>
</div>
<div>then typing this into it:</div>
<div><br>
</div>
<div>```python</div>
<div>from osgeo import gdal, gdalconst<br>
</div>
<div>ds = gdal.Open('/vsihdfs/file:/tmp/testfile.tif', gdalconst.GA_ReadOnly)<br>
</div>
<div>```</div>
<div><br>
</div>
<div>(I do not have easy access to and HDFS cluster right at the moment, so I only tested a local HDFS URI.)</div>
<div><br>
</div>
<div>A note: After having done a build without HDFS support in the tree, I had do a `make clean distclean` before I was able to get a build with working HDFS support.</div>
<div><br>
</div>
<div>Sincerely,</div>
<div>James McClain<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr">On Thu, Nov 22, 2018 at 8:13 PM Nikos Alexandris <<a href="mailto:nik@nikosalexandris.net">nik@nikosalexandris.net</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
* ZAZHIL-HA HERENA <<a href="mailto:zazhil_ha@hotmail.com" target="_blank">zazhil_ha@hotmail.com</a>> [2018-11-22 22:35:32 +0000]:<br>
<br>
>Hello, I am not sure if I should use this mailing list to ask questions but I wanted to try, I am a developer trying to use GDAL to open rasters in HDFS.<br>
><br>
><br>
>I read in GDAL documentation that starting 2.4 it is possible to open a raster in HDFS. I downloaded and compiled the latest source code available version and the generated libraries show it is 2.4 (libgdal.so.20.4.2). I compiled with option "-with-hdfs=yes"
and "--with-java=yes".<br>
><br>
>I am trying to open a raster using:<br>
><br>
><br>
><br>
> Dataset raster = gdal.Open("/vsihdfs/hdfs://node:8020/user/hdfs /spatial_raster/input_raster/kahoolawe.tif", gdalconst.GA_ReadOnly);<br>
<br>
Is your path correct? There is a space here (in "/hfds /").<br>
<br>
Nikos<br>
<br>
><br>
><br>
>but I am getting the following error: "ERROR 4: No such file or directory"<br>
<br>
[rest deleted]<br>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a></blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">"I prayed for freedom for twenty years, but received no answer until I prayed with my legs."<br>
-- Frederick Douglass<br>
<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>