[gdal-dev] Question on how to open a raster in HDFS using GDAL

[James McClain]

Hello,

I am the author of the vsihdfs code, I am ready and willing to help.

I just rebuilt it from current master and was able to successfully open an
dataset via an HDFS URI with the GDAL Python bindings.  I have a few
suggestions.

First, please try putting the file into a local directory and try something
like `gdalinfo /vsihdfs/file:/tmp/kahoolawe.tif` to establish a baseline.

Second, if you are using the Python bindings, please make sure that they
have been built and installed (and that you are using the ones that you
built rather than other ones that exist on your system).  Instructions for
building the Python bindings can be found here:
https://trac.osgeo.org/gdal/wiki/BuildingOnUnix .

In my case, after building and installing the library and bindings, I was
able to successfully open a dataset by starting a python REPL like this:

```bash
export
LD_LIBRARY_PATH=$HOME/local/hadoop-2.7.7/lib/native:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:$HOME/local/gdal-master-vsihdfs/lib
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export CLASSPATH=$($HOME/local/hadoop-2.7.7/bin/hadoop classpath --glob)
PYTHONPATH=$HOME/local/gdal-master-vsihdfs/lib/python2.7/site-packages
python
```

then typing this into it:

```python
from osgeo import gdal, gdalconst
ds = gdal.Open('/vsihdfs/file:/tmp/testfile.tif', gdalconst.GA_ReadOnly)
```

(I do not have easy access to and HDFS cluster right at the moment, so I
only tested a local HDFS URI.)

A note: After having done a build without HDFS support in the tree, I had
do a `make clean distclean` before I was able to get a build with working
HDFS support.

Sincerely,
James McClain

On Thu, Nov 22, 2018 at 8:13 PM Nikos Alexandris <nik at nikosalexandris.net>
wrote:

> * ZAZHIL-HA HERENA <zazhil_ha at hotmail.com> [2018-11-22 22:35:32 +0000]:
>
> >Hello, I am not sure if I should use this mailing list to ask questions
> but I wanted to try, I am a developer trying to use GDAL to open rasters in
> HDFS.
> >
> >
> >I read in GDAL documentation that starting 2.4 it is possible to open a
> raster in HDFS. I downloaded and compiled the latest source code available
> version and the generated libraries show it is 2.4 (libgdal.so.20.4.2). I
> compiled with option "-with-hdfs=yes" and "--with-java=yes".
> >
> >I am trying to open a raster using:
> >
> >
> >
> >    Dataset raster = gdal.Open("/vsihdfs/hdfs://node:8020/user/hdfs
> /spatial_raster/input_raster/kahoolawe.tif", gdalconst.GA_ReadOnly);
>
> Is your path correct? There is a space here (in "/hfds /").
>
> Nikos
>
> >
> >
> >but I am getting the following error: "ERROR 4: No such file or directory"
>
> [rest deleted]
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev



-- 
"I prayed for freedom for twenty years, but received no answer until I
prayed with my legs."
     -- Frederick Douglass
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181122/464b04a8/attachment.html>