[gdal-dev] Question on how to open a raster in HDFS using GDAL

[ZAZHIL-HA HERENA]

Hi there,

Thanks everybody for your valuable help, with your suggestions I finally got it working in my cluster with Cloudera 6 distribution(Hadoop 3) with Java, I would like to list here a few things I had to do to make it work since I used Cloudera to compile:


  *   In Cloudera, Hadoop libraries are under $CLOUDERA_PATH/lib/hadoop so this is the path I use for ./configure in --with-hdfs
  *   Then ./configure is looking for  a file include/hdfs.h , this directory in Cloudera exists in different path (outside Hadoop) $CLOUDERA_PATH/include, so I had to copy it under Hadoop path.
  *   Same for libhdfs.so , make expects to find it in lib/native/libhdfs.so, for Cloudera it is in $CLOUDERA_PATH/lib64 , so I just copied it to the expected location.

I tested using command line and also my Java application and /vsihdfs/ works as expected.

Thanks!!!
Zazhil-ha
________________________________
From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of James McClain <james.mcclain at gmail.com>
Sent: Friday, November 23, 2018 3:18 PM
To: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

Hello,

It may not be finding the native HDFS libraries.  Please see the pull request https://github.com/OSGeo/gdal/pull/714 for build instructions (in particular, you may need to augment the LD_LIBRARY_PATH environment variable).

If trouble persists, I would suggest building against Apache Hadoop 2.7.6 or 2.7.7 (both of those are know to work) as an experiment.

Sincerely,
James McClian

On Fri, Nov 23, 2018 at 2:15 PM ZAZHIL-HA HERENA <zazhil_ha at hotmail.com<mailto:zazhil_ha at hotmail.com>> wrote:
Thank you so much!, now I am working on 2.4 source code but I am getting an error when trying to configure using:

./configure --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/ --with-complete=yes --with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/ --with-swig-java=yes --with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/ --with-curl=/usr/bin/curl-config

The error I get is:

      checking for HDFS in /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for hdfsConnect in -lhdfs... no
      checking for /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes
      configure: error: HDFS support not enabled.


Is there any configuration in my environment that I should consider? or maybe another distribution of Hadoop?


________________________________
From: Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>>
Sent: Friday, November 23, 2018 11:52 AM
To: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Cc: ZAZHIL-HA HERENA; James McClain; nik at nikosalexandris.net<mailto:nik at nikosalexandris.net>
Subject: Re: [gdal-dev] Question on how to open a raster in HDFS using GDAL

> Version says 2.3.2 but libraries say: libgdal.so.20.4.2 .

Libtool number (.so.20.4.2) has nothing to do with user-friendly version
number (2.3.2)

> I am not sure if I
> got the latest code, this is the first time I compile it myself, I used
> this link to download source code:
> http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz

This is the latest release, but /vsihdfs/ is in the development version, not
yet released, so download

https://github.com/OSGeo/gdal/archive/master.zip

--
Spatialys - Geospatial professional services
http://www.spatialys.com


--
"I prayed for freedom for twenty years, but received no answer until I prayed with my legs."
     -- Frederick Douglass

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181127/998625d3/attachment.html>