[postgis-devel] Regular expressions in PostGIS sources

Wed Jan 5 07:32:09 PST 2011

Hi Pierre, welcome back,

On Tue, Jan 4, 2011 at 8:28 PM, Pierre Racine
<Pierre.Racine at sbf.ulaval.ca> wrote:
> Here I am...
>
> Implementing our own operator seems like an easy solution and seems to provide elegant syntax like "SELECT 2 * rast" but this is really a misleading illusion.
>
> 1 - How are you going to specify the band on which you want to apply the calculation?
>
> 2 - How do you specify what to do when there is a nodata value?
>
> 3 - In the case of a computation implying two or more rasters, how are you going to specify what to do with the non-overlapping areas?
>
> For sure we can assume trivial default for all those cases but this would deprive us from all the flexibility provided by the currently specified function. Please have look at the written specifications and at what the function does before turning it upside down into mystical illusions...
>
> The present specked function:
>
> 1 - Solve the three precedent problems very elegantly with more flexibility provided by ArcGIS and GRASS alltogether.
>
> 2 - Use the built-in PostgreSQL expression parser allowing our users to use ALL the functions they are already familiar with (http://www.postgresql.org/docs/9.0/interactive/functions.html) (Including the VERY necessary CASE WHEN and all the logical operators). Using this bullet proof parser will prevent us from fixing thousand of bug introduced by our own implementation. Implementing a robust parser is not a simple task. There will be months before we get 10% of what is provided by the PostgreSQL parser.
>
> 3 - This specification allow users to insert their own plpgsql function in the expression...
>
> 4 - We really have tons of other problems to solve before wanting to implement our own parser. The two raster version is already VERY complex without having to complexify our life.
>
> If, really, a user want to be able to use a simple syntax, with simple rasters (one band only), to do simple mathematical operations, he can alway define its own operators by overloading one of the two complete ST_MapAlgebra functions. But we are working for people having complex problems with complex raster structures. They will consider the syntax complexity issue a minor problem when they will see what they can do with the function.
>
> If you want to reenvent the wheel write a new expression parser... This might be a very nice programming exercice but before we get all the functionality we get by using the existing PostgreSQL one, Oracle with release a lightweight, easy to use, database system...
>

Yes, fully agree with you. Actually, my idea wasn't to create a new
expression parser. I was thinking in creating a simple "calculator" in
flex/bison to perform the algebraic operations in each step inside the
main loop, once replaced the "rast" occurrences by the real cell
value.

But after working deeper in MapAlgebra function itself, following the
specs, I think we don't need such calculator. We only need to replace
"rast" occurrences for concrete values and pass the SQL expression to
PostgreSQL server by libpq execution functions.

> You do not need any regular expression stuff for replacing all occurrences of "rast" in the expression. Use a simple C function like this:
>
> http://coding.debuntu.org/c-implementing-str_replace-replace-all-occurrences-substring
>
> or
>
> http://www.velocityreviews.com/forums/t439973-how-to-replace-a-substring-in-a-string-using-c.html
>
> or isn't there already a str_replace kind of function in PostGIS, PostgreSQL that we can link with?
>
> Pierre
>

I am using a simple string replacement function in C. I couldn't find
such function in PostGIS/PostgreSQL, neither GDAL.

Best regards,

-- 
Jorge Arévalo
Internet & Mobilty Division, DEIMOS
jorge.arevalo at deimos-space.com
http://mobility.grupodeimos.com/
http://gis4free.wordpress.com
http://geohash.org/ezjqgrgzz0g

>>-----Original Message-----
>>From: postgis-devel-bounces at postgis.refractions.net [mailto:postgis-devel-
>>bounces at postgis.refractions.net] On Behalf Of Jorge Arévalo
>>Sent: 16 novembre 2010 06:52
>>To: PostGIS Development Discussion
>>Subject: Re: [postgis-devel] Regular expressions in PostGIS sources
>>
>>On Tue, Nov 16, 2010 at 12:17 AM, Paul Ramsey <pramsey at opengeo.org> wrote:
>>> OK, that's what I thought / remembered-from-foss4g.
>>> So expressions like this:
>>>
>>>  'rast + 20'
>>>  'rast / 2',
>>>  'cos(rast)'
>>>  '2*rast + 3/rast'
>>>  '5'
>>>
>>> As arguments to an ST_MapAlgebra function.
>>>
>>> I have to admit that, given what I've been working on recently, I
>>> would be tempted to suggest a simple flex/bison grammar. But since I
>>> have invested the days necessary to understand WTF flex/bison are
>>> doing that feels like a smaller investment to me than it perhaps is
>>> for you. It would, however, allow you to get things like '2 *
>>> (cos(rast)/(4-rast))' "for free" without writing a bunch of your own
>>> stack management code.
>>>
>>
>>Yes, I was thinking exactly in flex/bison, I used them some years ago,
>>and I'll need to refresh my memory. I had a look at wkt_parse.* files,
>>but I'm not sure it's a good starting point.
>>
>>
>>> (is there anything special about the 'rast' keyword in there? what
>>> happens when you want to do MapAlgebra on two things at once? eg
>>> '2*rast1 + 4*rast2')
>>>
>>
>>No, it's only an easy way of naming raster objects in database, for
>>the user. Actually, the 2-raster version of MapAlgebra will use
>>'rast1', 'rast2' keywords.
>>
>>
>>> Wait, backing up you say "The expression is any PostgreSQL valid
>>> expression returning a number. This expression can contain conditional
>>> expressions like CASE or any user-defined function." Do you have a
>>> feel for how you are going to evaluate this stuff? Presumably with
>>> hooks into the PostgreSQL backend, where the functions are actually
>>> defined? Do you truly plan on handling every single function defined
>>> in PostgreSQL or what? Is that a strict requirement or can you just
>>> define a subset and handle them in your grammar (ala Mapserver
>>> EXPRESSION syntax or CQL?)
>>>
>>> How much scope are you biting off here? Maybe you have a magic easy
>>> approach to this that I'm not seeing.
>>>
>>> Paul
>>>
>>
>>Well, I have the same doubt. I'm not sure how far Pierre wants to go.
>>I think we should allow mathematical functions and boolean and
>>relational operations with rasters. Something similar to ArcGIS raster
>>calculator
>>http://www.esri.com/software/arcgis/extensions/spatialanalyst/graphics/features/map_algebra-lg.jpg
>>
>>The specs, IMHO, are referred to a pure PL/pgSQL implementation of
>>MapAlgebra, with the SQL executor behind, that allows evaluating user
>>defined functions over raster fields, and whatever. I hadn't time to
>>discuss this topic with Pierre.
>>
>>My bet, after reading, thinking and your answers, is to implement a
>>simple grammar, like you suggested. Allow mathematical operations and
>>logical and comparison operators. Looks fine? Too difficult maybe?
>>
>>Thanks!
>>
>>--
>>Jorge Arévalo
>>Internet & Mobilty Division, DEIMOS
>>jorge.arevalo at deimos-space.com
>>http://mobility.grupodeimos.com/
>>http://gis4free.wordpress.com
>>
>>> 2010/11/15 Jorge Arévalo <jorge.arevalo at deimos-space.com>:
>>>> On Mon, Nov 15, 2010 at 9:40 PM, Paul Ramsey <pramsey at opengeo.org> wrote:
>>>>> Could you give me some richer examples of the syntax in the form of
>>>>> full SQL statements?
>>>>>
>>>>
>>>> Yes, of course. From current documentation: "The expression is any
>>>> PostgreSQL valid expression returning a number. This expression can
>>>> contain conditional expressions like CASE or any user-defined
>>>> function. In the one raster version of ST_MapAlgebra, the value of the
>>>> current pixel is expressed in the expression by "rast". E.g.
>>>> 'cos(rast)' or 'rast / 2'"
>>>>
>>>> The calls to the MapAlgebra function will follow the syntax:
>>>>
>>>> ST_MapAlgebra(rast raster, band integer, expression text,
>>>> nodatavalueexpr text, pixeltype text)
>>>>
>>>> And the expressions will be something like 'rast + 20', 'rast / 2',
>>>> 'cos(rast)', '2*rast + 3/rast', '5', etc. Then, each pixel of the
>>>> raster must be used in that operation, by replacing the word 'rast'
>>>> for its value, and storing the resulting pixel value in the output
>>>> raster. It's a basic MapAlgebra implementation, with only one raster,
>>>> by now.
>>>>
>>>> You have a PL/pgSQL implementation at
>>>> http://svn.osgeo.org/postgis/trunk/raster/scripts/plpgsql/st_mapalgebra.sql
>>>>
>>>> But we're working on a C implementation
>>>>
>>>>> 2010/11/15 Jorge Arévalo <jorge.arevalo at deimos-space.com>:
>>>>>> On Mon, Nov 15, 2010 at 7:19 PM, Paul Ramsey <pramsey at opengeo.org> wrote:
>>>>>>> There's some regex support in postgresql core. What do you need regex
>>>>>>> for in raster?
>>>>>>>
>>>>>>> P
>>>>>>
>>>>>> Hi Paul,
>>>>>>
>>>>>> For MapAlgebra implementation at core level. The ST_MapAlgebra
>>>>>> Pl/pgSQL function takes an input argument like "SELECT rast + 3",
>>>>>> meaning "add 3 to all raster's pixels", and it calls a  core function
>>>>>> (via postgresql function) to perform the operation.
>>>>>>
>>>>>> Should I parse the expression in PL/pgSQL function, at postgresql
>>>>>> level or at core level? Now, I'm working only at core level.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> --
>>>>>> Jorge Arévalo
>>>>>> Internet & Mobilty Division, DEIMOS
>>>>>> jorge.arevalo at deimos-space.com
>>>>>> http://mobility.grupodeimos.com/
>>>>>> http://gis4free.wordpress.com
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> 2010/11/15 Jorge Arévalo <jorge.arevalo at deimos-space.com>:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Is there any mechanism in PostGIS to deal with regular expressions and
>>>>>>>> string replacement at core level (I mean, C functions)? I'm going to
>>>>>>>> use string replacement in PostGIS Raster but I don't want to reinvent
>>>>>>>> the wheel, if is invented
>>>>>>>>
>>>>>>>> Many thanks in advance,
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jorge Arévalo
>>>>>>>> Internet & Mobilty Division, DEIMOS
>>>>>>>> jorge.arevalo at deimos-space.com
>>>>>>>> http://mobility.grupodeimos.com/
>>>>>>>> http://gis4free.wordpress.com
>>>>>>>> _______________________________________________
>>>>>>>> postgis-devel mailing list
>>>>>>>> postgis-devel at postgis.refractions.net
>>>>>>>> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> postgis-devel mailing list
>>>>>>> postgis-devel at postgis.refractions.net
>>>>>>> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>>>>>>>
>>>>>> _______________________________________________
>>>>>> postgis-devel mailing list
>>>>>> postgis-devel at postgis.refractions.net
>>>>>> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>>>>>>
>>>>> _______________________________________________
>>>>> postgis-devel mailing list
>>>>> postgis-devel at postgis.refractions.net
>>>>> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jorge Arévalo
>>>> Internet & Mobilty Division, DEIMOS
>>>> jorge.arevalo at deimos-space.com
>>>> http://mobility.grupodeimos.com/
>>>> http://gis4free.wordpress.com
>>>> _______________________________________________
>>>> postgis-devel mailing list
>>>> postgis-devel at postgis.refractions.net
>>>> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>>>>
>>> _______________________________________________
>>> postgis-devel mailing list
>>> postgis-devel at postgis.refractions.net
>>> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>>>
>>_______________________________________________
>>postgis-devel mailing list
>>postgis-devel at postgis.refractions.net
>>http://postgis.refractions.net/mailman/listinfo/postgis-devel
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-devel
>