[gdal-dev] Question on how to open a raster in HDFS using GDAL

[James McClain]

You are very welcome, thank you for the report!

On Tue, Nov 27, 2018 at 5:44 PM ZAZHIL-HA HERENA <zazhil_ha at hotmail.com>
wrote:

> Hi there,
>
> Thanks everybody for your valuable help, with your suggestions I finally
> got it working in my cluster with Cloudera 6 distribution(Hadoop 3) with
> Java, I would like to list here a few things I had to do to make it work
> since I used Cloudera to compile:
>
>
>    - In Cloudera, Hadoop libraries are under $CLOUDERA_PATH/lib/hadoop so
>    this is the path I use for ./configure in --with-hdfs
>    - Then* ./configure* is looking for  a file *include/hdfs.h* , this
>    directory in Cloudera exists in different path (outside Hadoop)
>    $CLOUDERA_PATH/include, so I had to copy it under Hadoop path.
>    - Same for libhdfs.so , *make* expects to find it in
>    *lib/native/libhdfs.so*, for Cloudera it is in $CLOUDERA_PATH/lib64 ,
>    so I just copied it to the expected location.
>
>
> I tested using command line and also my Java application and /vsihdfs/
> works as expected.
>
> Thanks!!!
> Zazhil-ha
> ------------------------------
> *From:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of James
> McClain <james.mcclain at gmail.com>
> *Sent:* Friday, November 23, 2018 3:18 PM
> *To:* gdal-dev at lists.osgeo.org
> *Subject:* Re: [gdal-dev] Question on how to open a raster in HDFS using
> GDAL
>
> Hello,
>
> It may not be finding the native HDFS libraries.  Please see the pull
> request https://github.com/OSGeo/gdal/pull/714 for build instructions (in
> particular, you may need to augment the LD_LIBRARY_PATH environment
> variable).
>
> If trouble persists, I would suggest building against Apache Hadoop 2.7.6
> or 2.7.7 (both of those are know to work) as an experiment.
>
> Sincerely,
> James McClian
>
> On Fri, Nov 23, 2018 at 2:15 PM ZAZHIL-HA HERENA <zazhil_ha at hotmail.com>
> wrote:
>
> Thank you so much!, now I am working on 2.4 source code but I am getting
> an error when trying to configure using:
>
> *./configure
> --prefix=/scratch/zherena/gdal/build/gdal-master/gdal/outputb/
> --with-complete=yes
> --with-java=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64/
> --with-swig-java=yes
> --with-hdfs=/scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/
> --with-curl=/usr/bin/curl-config*
>
> The error I get is:
>
>
> *      checking for HDFS in
> /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678/... checking for
> hdfsConnect in -lhdfs... no *
>
> *      checking for
> /scratch/zherena/gdal/CDH-6.0.1-1.cdh6.0.1.p0.590678//include/hdfs.h... yes
> *
> *      configure: error: HDFS support not enabled.*
>
>
> Is there any configuration in my environment that I should consider? or
> maybe another distribution of Hadoop?
>
>
> ------------------------------
> *From:* Even Rouault <even.rouault at spatialys.com>
> *Sent:* Friday, November 23, 2018 11:52 AM
> *To:* gdal-dev at lists.osgeo.org
> *Cc:* ZAZHIL-HA HERENA; James McClain; nik at nikosalexandris.net
> *Subject:* Re: [gdal-dev] Question on how to open a raster in HDFS using
> GDAL
>
> > Version says 2.3.2 but libraries say: libgdal.so.20.4.2 .
>
> Libtool number (.so.20.4.2) has nothing to do with user-friendly version
> number (2.3.2)
>
> > I am not sure if I
> > got the latest code, this is the first time I compile it myself, I used
> > this link to download source code:
> > http://download.osgeo.org/gdal/CURRENT/gdal-2.3.2.tar.gz
>
> This is the latest release, but /vsihdfs/ is in the development version,
> not
> yet released, so download
>
> https://github.com/OSGeo/gdal/archive/master.zip
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
>
>
> --
> "I prayed for freedom for twenty years, but received no answer until I
> prayed with my legs."
>      -- Frederick Douglass
>
>

-- 
"I prayed for freedom for twenty years, but received no answer until I
prayed with my legs."
     -- Frederick Douglass
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181127/23bf2f77/attachment-0001.html>