[postgis-devel] GSoC Proposal : GDAL PostGIS Raster driver

Varun Saraf varunsaraf14 at gmail.com
Wed Mar 19 12:59:34 PDT 2014


Hi,

Below is the link to my proposal on "Implement write support for the GDAL
PostGIS Raster driver" :
https://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2014/varunsaraf/5629499534213120

Please have a look and let me know your suggestions in the comments section
on melange.

Regards,
Varun Saraf

On Mon, Mar 17, 2014 at 2:22 AM, Jorge Arevalo <jorgearevalo at libregis.org>wrote:

> Hello again, Varun.
>
> I've read your proposal. I think it looks great. Clear and straight to
> the point I've added just a few comments. Take a look whenever you
> have time.
>
> About future ideas, I'd mention the possibility of testing DB Manager
> with the new driver with write support. QGIS connection is really
> important. And the important fact that you have GDAL + PostGIS Raster
> working both ways. This is a really powerful tool for third parties
> that rely on PostGIS for their data and use GDAL as main tool to
> handle it. They will be able to just use GDAL to read/write data in
> PostGIS Raster, without using the loader. Check the software using
> GDAL: https://trac.osgeo.org/gdal/wiki/SoftwareUsingGdal.
>
> Anyway, you may want to ask GDAL / PostGIS users list, just asking
> this same question: use cases of GDAL + PostGIS Raster and how those
> cases will benefit from the write support in the driver. Final users
> can make you a better way of how useful could your work be.
>
>
> Best regards,
>
> Jorge
>
> On Fri, Mar 14, 2014 at 5:21 PM, Jorge Arevalo
> <jorgearevalo at libregis.org> wrote:
> > Hello Varun,
> >
> > Sorry for my delay. I've been really busy this week. I'll have a look
> > at your proposal ASAP.
> >
> > We keep in touch.
> >
> > Best regards,
> >
> > Jorge
> >
> > On Fri, Mar 14, 2014 at 12:33 PM, Varun Saraf <varunsaraf14 at gmail.com>
> wrote:
> >> Hello Sir,
> >>
> >> Below is the link to the initial draft of my proposal. Please have a
> look
> >> and let me know. I have enabled comments so you can comment there
> itself.
> >>
> >>
> https://docs.google.com/document/d/1NeTkt0uXUAm1iH2RT43WwgQOPFttlrhHa79GFIok_Ng/edit
> >>
> >> I have left the 'future ideas" section empty. Can you suggest something
> for
> >> that?
> >>
> >> Regards,
> >> Varun Saraf
> >>
> >>
> >>
> >> On Wed, Mar 12, 2014 at 9:45 PM, Varun Saraf <varunsaraf14 at gmail.com>
> wrote:
> >>>
> >>> Hello Sir,
> >>>
> >>> So let me check if I am on the right track. A broad workflow would be:
> >>>
> >>> First we load the raster 'tiles'(actually individual raster columns),
> to
> >>> be written in the db, in the buffer with the help VRT driver since it
> is the
> >>> only driver that handles 'tiles' with different dimensions,
> geo-referencing,
> >>> pixel size etc.
> >>>
> >>> Now we modify the IRasterIO method so that it writes from the buffer
> into
> >>> a region of PostGISRasterRasterBands(in the db).
> >>>
> >>> Now if I am not wrong, what all should I include in the proposal?
> >>>
> >>> Regards,
> >>> Varun Saraf
> >>>
> >>> On Sat, Mar 8, 2014 at 3:31 AM, Jorge Arevalo <
> jorgearevalo at libregis.org>
> >>> wrote:
> >>>>
> >>>> Hello Varun,
> >>>>
> >>>> On Fri, Mar 7, 2014 at 10:23 PM, Varun Saraf <varunsaraf14 at gmail.com>
> >>>> wrote:
> >>>> > Hello Sir,
> >>>> >
> >>>> > Thank you for your response. That was quite helpful.
> >>>> >
> >>>> > I have been going through the codes as well as descriptions of
> driver
> >>>> > implementation tutorial, IRasterIO and VRT driver. We need to
> convert
> >>>> > the
> >>>> > GDAL PostGIS Raster driver to read/write. But I am not sure where to
> >>>> > start.
> >>>>
> >>>> If you're finally accepted in the program, you will basically have to
> >>>> modify the IRasterIO method in PostGISRasterRasterBand class (and
> >>>> probably more methods, but this is the main one)
> >>>>
> >>>> This is a good starting point for the method:
> >>>>
> >>>>
> http://trac.osgeo.org/gdal/browser/trunk/gdal/frmts/postgisraster/postgisrasterrasterband.cpp#L326
> >>>>
> >>>> Look at the first "if" clause. If you're trying to call the driver in
> >>>> write mode, it raises an error. That part needs to be replaced with
> >>>> something useful. That's the core of the write support.
> >>>>
> >>>> If you read the driver implementation tutorial, you will know about
> >>>> IReadBlock / IWroteBlock methods. Those are the methods you need to
> >>>> implement read / write support in most of the drivers. They work with
> >>>> a single block of data (a tile). But they expect all tiles to be the
> >>>> same size and well aligned. As you probably know now, this isn't the
> >>>> case of PostGIS Raster driver. So, you need to use IRasterIO method.
> >>>> It reads/write a region of data (probably including several tiles),
> >>>> taking care of data type translation, among other things. This method,
> >>>> in PostGIS Raster driver, just allow reading, for the time being. We
> >>>> want writing support too.
> >>>>
> >>>> Check the header of this method in parent class:
> >>>>
> >>>>
> http://www.gdal.org/classGDALRasterBand.html#a5497e8d29e743ee9177202cb3f61c3c7
> >>>>
> >>>> BTW, don't worry about RasterIO / IRasterIO difference. It's
> >>>> important, but not now.
> >>>>
> >>>>
> >>>>
> >>>> > I think I also have to go through PostGIS raster2pgsql to get a
> clearer
> >>>> > picture. Can you be more specific about the TODO work you mentioned?
> >>>> > What I
> >>>> > want to know is, for example, which are the key functions in
> >>>> > raster2pgsql
> >>>> > that need to be replicated in the GDAL PostGIS Raster driver to
> >>>> > implement
> >>>> > write support?
> >>>> >
> >>>>
> >>>> That's a good catching. Yes, understanding raster2pgsql would be
> >>>> helpful. I didn't code it, but I think the functions that actually
> >>>> create the "INSERT" / "CREATE TABLE" statements are a good starting
> >>>> point. They start here:
> >>>>
> >>>>
> >>>>
> http://trac.osgeo.org/postgis/browser/trunk/raster/loader/raster2pgsql.c#L827
> >>>>
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Jorge
> >>>>
> >>>>
> >>>> > Regards,
> >>>> > Varun Saraf
> >>>> >
> >>>> >
> >>>> > On Thu, Mar 6, 2014 at 6:44 PM, Jorge Arevalo
> >>>> > <jorgearevalo at libregis.org>
> >>>> > wrote:
> >>>> >>
> >>>> >> Hello Varun,
> >>>> >>
> >>>> >> On Wed, Mar 5, 2014 at 6:45 PM, Varun Saraf <
> varunsaraf14 at gmail.com>
> >>>> >> wrote:
> >>>> >> > Hello Sir,
> >>>> >> >
> >>>> >> > I am reading the files mentioned by you earlier. I had a doubt
> >>>> >> > regarding
> >>>> >> > the
> >>>> >> > VRT driver. Can you please elaborate what do you mean by using
> >>>> >> > behind
> >>>> >> > the
> >>>> >> > curtains.
> >>>> >> >
> >>>> >>
> >>>> >> Of course.
> >>>> >>
> >>>> >> If you read about VRT format, you know that format is just a XML
> >>>> >> descriptor to tell GDAL where to find the real data. In human
> >>>> >> language, VRT tells GDAL things like: My "virtual" raster has 3
> bands:
> >>>> >>
> >>>> >> - Band 1 is built from band 1 of file file1.tif. And just this
> part of
> >>>> >> the band (a length in pixels)
> >>>> >> - Band 2 is built from band 1 of another file, file2.tif. And just
> >>>> >> this part of the band (the same lenght, in pixels)
> >>>> >> - Etc
> >>>> >>
> >>>> >> You get the point. I can build a "virtual" raster just by reading
> some
> >>>> >> bands of other real raster files, and select which bands I should
> >>>> >> read, and which pixels of those bands. VRT calls those descriptors
> >>>> >> telling which bands to read "VRT sources". So, a VRT raster is
> build
> >>>> >> reading from several VRT sources.
> >>>> >>
> >>>> >> Understanding this, remember one of the key concepts of PostGIS
> >>>> >> Raster. A PostGIS Raster table may contain several rows, each row
> >>>> >> containing one raster column. So far, so good. You may think about
> >>>> >> each of these raster columns as "tiles" of your bigger raster
> coverage
> >>>> >> (the table). Ok, but don't forget that *each tile is really a
> single
> >>>> >> raster object*. That means that each tile may have *different
> >>>> >> dimensions, georeferencing and pixel size*.
> >>>> >>
> >>>> >> So, if you want to consider a single PostGIS Raster table in the
> same
> >>>> >> way you'd consider a tiled GeoTIFF file, you have this big problem
> to
> >>>> >> face. In a GeoTIFF, all tiles have the same dimensions and pixel
> size,
> >>>> >> and they're not really georeferenced. The whole file is
> georeferenced
> >>>> >> as a set of tiles. The tiles are just "dumb" chunks of data, put
> one
> >>>> >> after the another. PostGIS Raster tiles have meaning, and nothing
> >>>> >> assures you they will behave properly, in terms of pixel size,
> >>>> >> georeferencing, grid alignment, etc.
> >>>> >>
> >>>> >> How to approach this problem? The response was: using other GDAL
> >>>> >> driver with the same problem (you know: don't repeat work already
> >>>> >> done). And this driver was VRT driver. Because VRT format, as said,
> >>>> >> can read data from different sources, and create a raster coverage
> >>>> >> from it, making all the "dirty work" behind the curtains
> (alignment,
> >>>> >> pixel size adjustment, etc).
> >>>> >>
> >>>> >> So, to take advantage of VRT driver, I just considered PostGIS
> Raster
> >>>> >> as special kind of VRT. And each single PostGIS Raster tile (raster
> >>>> >> column belonging to a single row) is a VRT source (a place to read
> >>>> >> data from, with its own georeferencing, pixel size and dimensions).
> >>>> >> So, I build a "in-memory" VRT object reading from PostGIS Raster
> table
> >>>> >> using these sources, and let VRT driver finish the job.
> >>>> >>
> >>>> >> Regarding this, there're some advanced concepts, like pre-caching
> some
> >>>> >> tiles in memory, or live calculation of raster extent, instead of
> >>>> >> reading it (how would you get the extent covered by a PostGIS
> Raster
> >>>> >> table if you don't have a single georeference, but a per-tile
> >>>> >> georeference?) Anyway, don't go so deep right now. I just want you
> to
> >>>> >> understand the reasons to use VRT driver, before trying to
> understand
> >>>> >> code. I hope this helps you.
> >>>> >>
> >>>> >> Best regards,
> >>>> >>
> >>>> >> Jorge
> >>>> >>
> >>>> >> > Regards,
> >>>> >> > Varun Saraf
> >>>> >> >
> >>>> >> >
> >>>> >> > On Wed, Feb 26, 2014 at 7:53 PM, Jorge Arevalo
> >>>> >> > <jorgearevalo at libregis.org>
> >>>> >> > wrote:
> >>>> >> >>
> >>>> >> >> Hello Varun,
> >>>> >> >>
> >>>> >> >> First of all, you need to understand the GDAL data model [1],
> and
> >>>> >> >> the GDAL driver implementation tutorial [2]. After that, you may
> >>>> >> >> want
> >>>> >> >> to read the code of the driver itself [3]. Main files are
> >>>> >> >> postgisrasterdriver.cpp, postgisrasterdataset.cpp and
> >>>> >> >> postgisrasterrasterband.cpp. It has several dark points, so,
> don't
> >>>> >> >> worry if you don't understand everything.
> >>>> >> >>
> >>>> >> >> About this specific driver, you also need to understand one main
> >>>> >> >> concept: most GDAL drivers are based in the concept of "block".
> In
> >>>> >> >> tiled datasets, 1 block = 1 tile. In not tiled datasets,
> normally 1
> >>>> >> >> block = 1 scanline. But both cases have something in common: all
> >>>> >> >> tiles
> >>>> >> >> (or scanlines) have the same dimensions (width and height).
> Think
> >>>> >> >> about the Google Maps tiles. They're all 256x256 pixels.
> >>>> >> >>
> >>>> >> >> Well, GDAL PostGIS Raster driver doesn't work that way. In
> PostGIS
> >>>> >> >> Raster, each tile of a coverage can have different dimensions,
> >>>> >> >> different pixel size and even being aligned to a different grid.
> >>>> >> >> GDAL
> >>>> >> >> driver needs to handle this, and the problem is GDAL data model
> was
> >>>> >> >> not thought for that. The only driver that works that way, apart
> >>>> >> >> from
> >>>> >> >> PostGIS Raster driver, is VRT driver. For that reason, I used it
> >>>> >> >> behind curtains. If VRT driver can solve this kind of problems,
> why
> >>>> >> >> not using it?. And for that reason, there are two classes
> >>>> >> >> (postgisrastertiledataset and postgisrastertilerasterband) that
> >>>> >> >> handle
> >>>> >> >> these "different" kind of tiles (each one with its own
> dimensions,
> >>>> >> >> pixel size, etc).
> >>>> >> >>
> >>>> >> >> Having those concepts in mind, you can start thinking about the
> >>>> >> >> TODO
> >>>> >> >> work. Basically, a GDAL driver can be read only or read/write.
> GDAL
> >>>> >> >> PostGIS Raster driver is readonly. We want to convert it in a
> >>>> >> >> read/write driver (this is: making it able to create new tables
> in
> >>>> >> >> the
> >>>> >> >> database, not just reading them). There are a few methods
> related
> >>>> >> >> with
> >>>> >> >> this. Most important are IWriteBlock and IRasterIO. For this
> >>>> >> >> specific
> >>>> >> >> driver, you'd need to work with IRasterIO method, because
> PostGIS
> >>>> >> >> Raster driver doesn't work with equally sized blocks, as said in
> >>>> >> >> the
> >>>> >> >> previous paragraph, and IWriteBlock expects that.
> >>>> >> >>
> >>>> >> >> Now, you should be ready to write a proposal. Check [4] to know
> how
> >>>> >> >> a
> >>>> >> >> student proposal should look like, and [5] to get useful tips on
> >>>> >> >> writing proposals for GSoC.
> >>>> >> >>
> >>>> >> >> Apart from that, are you interested in other projects? You can
> send
> >>>> >> >> more than one proposal to GSoC.
> >>>> >> >>
> >>>> >> >> As suggestion, the QGIS PostGIS Raster plugin needs to be
> ported to
> >>>> >> >> QGIS 2.0. This project is also related with GDAL PostGIS Raster
> >>>> >> >> driver
> >>>> >> >> and PostGIS Raster extension. Not present in QGIS ideas page,
> but I
> >>>> >> >> could guide you.
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> Best regards,
> >>>> >> >>
> >>>> >> >> Jorge
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> [1] http://www.gdal.org/gdal_datamodel.html
> >>>> >> >> [2] http://www.gdal.org/gdal_drivertut.html
> >>>> >> >> [3]
> >>>> >> >>
> http://trac.osgeo.org/gdal/browser/trunk/gdal/frmts/postgisraster
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> [4]
> http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2014/help_page#5._What_should_a_student_proposal_look
> >>>> >> >> [5]
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> http://www.di.ens.fr/~baghdadi/TXT_blog/5_advices_to_get_your_proposal_accepted.lyx.html
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> On Tue, Feb 25, 2014 at 10:46 PM, Varun Saraf
> >>>> >> >> <varunsaraf14 at gmail.com>
> >>>> >> >> wrote:
> >>>> >> >> > Hello Sir,
> >>>> >> >> >
> >>>> >> >> > Please tell me how should I go about preparing the proposal
> for
> >>>> >> >> > this
> >>>> >> >> > project.
> >>>> >> >> >
> >>>> >> >> > Regards,
> >>>> >> >> > Varun Saraf
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> > On Mon, Feb 17, 2014 at 5:12 AM, Jorge Arevalo
> >>>> >> >> > <jorgearevalo at libregis.org>
> >>>> >> >> > wrote:
> >>>> >> >> >>
> >>>> >> >> >> Hello Varun,
> >>>> >> >> >>
> >>>> >> >> >> You may want to check
> >>>> >> >> >> http://trac.osgeo.org/postgis/wiki/GoogleSummerCode, and
> >>>> >> >> >> specifically
> >>>> >> >> >> http://trac.osgeo.org/gdal/wiki/frmts_wtkraster.html
> >>>> >> >> >>
> >>>> >> >> >> Let me know if you have any problem.
> >>>> >> >> >>
> >>>> >> >> >> Best regards,
> >>>> >> >> >>
> >>>> >> >> >> Jorge
> >>>> >> >> >>
> >>>> >> >> >> On Sat, Feb 15, 2014 at 5:40 PM, Varun Saraf
> >>>> >> >> >> <varunsaraf14 at gmail.com>
> >>>> >> >> >> wrote:
> >>>> >> >> >> > Hi Sir,
> >>>> >> >> >> >
> >>>> >> >> >> > Thank you for your reply.
> >>>> >> >> >> >
> >>>> >> >> >> > In the meanwhile can you suggest something which will help
> me
> >>>> >> >> >> > familiarize
> >>>> >> >> >> > with the project.
> >>>> >> >> >> >
> >>>> >> >> >> > Regards,
> >>>> >> >> >> > Varun Saraf
> >>>> >> >> >> > Graduate Student in Computer Science
> >>>> >> >> >> > Lab for Spatial Informatics
> >>>> >> >> >> > International Institute of Information Technology
> >>>> >> >> >> > Hyderabad, India
> >>>> >> >> >> > +91-90308-74699
> >>>> >> >> >> >
> >>>> >> >> >> > On Fri, Feb 14, 2014 at 4:34 PM, Jorge Arevalo
> >>>> >> >> >> > <jorgearevalo at libregis.org>
> >>>> >> >> >> > wrote:
> >>>> >> >> >> >>
> >>>> >> >> >> >> Hello Varun,
> >>>> >> >> >> >>
> >>>> >> >> >> >> Happy to hear about your interest. I will be pleased to be
> >>>> >> >> >> >> your
> >>>> >> >> >> >> mentor. I need to do some administrative work, but we
> keep in
> >>>> >> >> >> >> touch.
> >>>> >> >> >> >>
> >>>> >> >> >> >> The student application period opens on March 10. I will
> help
> >>>> >> >> >> >> you
> >>>> >> >> >> >> to
> >>>> >> >> >> >> prepare a proposal in the meantime.
> >>>> >> >> >> >>
> >>>> >> >> >> >> Be ready for some coding fun!
> >>>> >> >> >> >>
> >>>> >> >> >> >> Best regards
> >>>> >> >> >> >>
> >>>> >> >> >> >> --
> >>>> >> >> >> >> Jorge Arevalo
> >>>> >> >> >> >> Freelance developer
> >>>> >> >> >> >>
> >>>> >> >> >> >> http://about.me/jorgeas80
> >>>> >> >> >> >>
> >>>> >> >> >> >> On Fri, Feb 14, 2014 at 11:18 AM, Varun Saraf
> >>>> >> >> >> >> <varunsaraf14 at gmail.com>
> >>>> >> >> >> >> wrote:
> >>>> >> >> >> >> > Hello,
> >>>> >> >> >> >> >
> >>>> >> >> >> >> > I am interested in the project to implement write
> support
> >>>> >> >> >> >> > for
> >>>> >> >> >> >> > the
> >>>> >> >> >> >> > PostGIS
> >>>> >> >> >> >> > raster GDAL driver. I have worked in PostGIS and GDAL
> for
> >>>> >> >> >> >> > some
> >>>> >> >> >> >> > of
> >>>> >> >> >> >> > my
> >>>> >> >> >> >> > projects during graduation. I also have a background in
> >>>> >> >> >> >> > geospatial
> >>>> >> >> >> >> > imagery.
> >>>> >> >> >> >> >
> >>>> >> >> >> >> > I have the required skill set - I am well versed in
> C/C++
> >>>> >> >> >> >> > and
> >>>> >> >> >> >> > am
> >>>> >> >> >> >> > familiar
> >>>> >> >> >> >> > with data structures, trees and algorithms. I have been
> >>>> >> >> >> >> > developing
> >>>> >> >> >> >> > in
> >>>> >> >> >> >> > C/C++
> >>>> >> >> >> >> > for past four years. Please guide me with this project.
> >>>> >> >> >> >> >
> >>>> >> >> >> >> > Thanks and Regards,
> >>>> >> >> >> >> > Varun Saraf
> >>>> >> >> >> >> > Graduate Student in Computer Science
> >>>> >> >> >> >> > Lab for Spatial Informatics
> >>>> >> >> >> >> > International Institute of Information Technology
> >>>> >> >> >> >> > Hyderabad, India
> >>>> >> >> >> >> > +91-90308-74699
> >>>> >> >> >> >> >
> >>>> >> >> >> >> >
> >>>> >> >> >> >> > On Fri, Feb 14, 2014 at 2:56 PM, Jorge Arevalo
> >>>> >> >> >> >> > <jorgearevalo at libregis.org>
> >>>> >> >> >> >> > wrote:
> >>>> >> >> >> >> >>
> >>>> >> >> >> >> >> Hi all,
> >>>> >> >> >> >> >>
> >>>> >> >> >> >> >> I'm available for the first project here
> >>>> >> >> >> >> >>
> >>>> >> >> >> >> >> http://trac.osgeo.org/postgis/wiki/WKTRasterGSoC
> >>>> >> >> >> >> >>
> >>>> >> >> >> >> >> Best regards,
> >>>> >> >> >> >> >>
> >>>> >> >> >> >> >> --
> >>>> >> >> >> >> >> Jorge Arevalo
> >>>> >> >> >> >> >> Freelance developer
> >>>> >> >> >> >> >>
> >>>> >> >> >> >> >> http://about.me/jorgeas80
> >>>> >> >> >> >> >> _______________________________________________
> >>>> >> >> >> >> >> SoC mailing list
> >>>> >> >> >> >> >> SoC at lists.osgeo.org
> >>>> >> >> >> >> >> http://lists.osgeo.org/mailman/listinfo/soc
> >>>> >> >> >> >> >
> >>>> >> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >> >
> >>>> >> >> >>
> >>>> >> >> >>
> >>>> >> >> >>
> >>>> >> >> >> --
> >>>> >> >> >> Jorge Arevalo
> >>>> >> >> >> Freelance developer
> >>>> >> >> >>
> >>>> >> >> >> http://about.me/jorgeas80
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >>
> >>>> >> >>
> >>>> >> >>
> >>>> >> >> --
> >>>> >> >> Jorge Arevalo
> >>>> >> >> Freelance developer
> >>>> >> >>
> >>>> >> >> http://about.me/jorgeas80
> >>>> >> >
> >>>> >> >
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> --
> >>>> >> Jorge Arevalo
> >>>> >> Freelance developer
> >>>> >>
> >>>> >> http://about.me/jorgeas80
> >>>> >
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Jorge Arevalo
> >>>> Freelance developer
> >>>>
> >>>> http://about.me/jorgeas80
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > Jorge Arevalo
> > Freelance developer
> >
> > http://about.me/jorgeas80
>
>
>
> --
> Jorge Arevalo
> Freelance developer
>
> http://about.me/jorgeas80
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20140320/c5c78d83/attachment.html>


More information about the postgis-devel mailing list