[OSGeo-Discuss] Does rasdaman CE solve an open source geospatial problem?

Sat May 21 09:50:47 PDT 2016

Peter,

It's impressive that 20 TB is being served out by the community edition.
However, the point that Edzer brings up (and that I was unaware of) causes
me even more concern regarding the language used in service of promoting
the potential OSGeo version of rasdaman around it's comparison to other
projects.

Are the performance benchmarks that you were siting as sources based on
queries that would benefit from distributed processing (which is a very
large class of queries)? Are the benchmarks using the open source rasdaman
or the proprietary version? If you are going to compare yourself to a
distributed processing engine such as Apache Spark, you should be comparing
it fairly - Apache Spark does not claim to be a fast solution in a single
machine case. If you are comparing it to the proprietary version of
rasdaman, then I'm wondering why you are referencing those numbers when
talking about the open source version, which has been the subject of these
threads of conversation. Google Earth Engine may beat us all out in
performance, but that doesn't directly contribute much to the open source
geospatial community because it lacks that key openness.

If rasdaman CE does not scale horizontally, there are use cases and example
queries with which I am very confident rasdaman would be outperformed by
systems constructed at their core with Apache Spark. If I've misunderstood
something, or if this is somehow not the case, please let me know.

Best,
Rob

On Sat, May 21, 2016 at 4:07 AM, Peter Baumann <
p.baumann at jacobs-university.de> wrote:

> oh, just looking at the subject again:
>
> several service providers believe indeed rasdaman community does offer a
> significant advantage:
> - see the download figures on www.rasdaman.org
> - concretely, see www.planetserver.eu which is running rasdaman community
> on - I believe - about 20 TB of Planetary Science data.
>
> -Peter
>
>
>
> On 05/21/2016 09:56 AM, Peter Baumann wrote:
>
> Hm, first of all: this is opening a different thread, talking about
> functionality of rasdaman community. Next, it is based on assumptions -
> without details (because off topic): conclusions are wrong.
>
> But to respond to the core message emphasized in the first paragraph: I
> respectfully disagree. In particular, such a position does not benefit the
> open source community very much as I am trying to explain below.
>
> TL;DR:
>
> You have a strong expertise in Geoinformatics, I know something about
> Computer Science. This is where we can talk as professors and scientists.
> Your statement is about economics, industry etc. Having an opinion there
> (and articulate it) is fair, but in these fields our opinion weighs not
> more than anyone else's in the street. We should not attempt to attain
> importance through inapplicable roles.
>
> Let us look at a professor. They have a conveniently high salary which is
> paid by society, that is: tax payers. Nobody can influence what a professor
> does and how much return s/he generates for society.
>
> A single open source developer (or a small group, whatever) do not
> experience this convenience. They have a dream where they invest, they try
> to not make money for getting richer than a professor ;-) but merely for
> their economic survival. Some (in particular scientists) enjoy the money
> rain coming from publicly funded projects (again: the tax payer
> subsidizes), but most in the community have to struggle hard. They face
> reluctant customers, competition by the giants in the market, and many more
> obstacles.
>
> From the cosy place of a lifelong position with a secured salary and
> decent retirement funds it is easy to say that all software should be free
> like free beer (quote from below: "can be reproduced by other scientists
> without prohibitive license costs").
>
> If the open source movement cannibalizes itself it will make it all so
> easy for the big players to maintain their dominance, they will silently
> applaud. Quoting Jeroen:
> > NEVER IGNORE COMPANIES AGAIN IN OSGEO OR FOSS4G! THEY ARE NOT A THREAT,
> THEY ARE A NECESSITY.
>
> That said: It is entirely ok to have the opinion you have. Others, though,
> may disagree. I am one of those.
>
> respectfully,
> Peter
>
>
>
> On 05/20/2016 09:30 AM, Edzer Pebesma wrote:
>
> As a scientist, I teach my students that for doing science it is a
> requirement to work with open source software, because only then
> workflows are fully transparent and can be reproduced by other
> scientists without prohibitive license costs. Currently, working
> with large amounts of earth observation (EO) or climate model data
> typically requires to download these data tile by tile, stitch them
> together, and go through all of them. Array databases may simplify
> this substantially: after ingesting the tiles, they can directly
> work on the whole data as a multi-dimensinal array ("data cube").
> Computations on these array are typically embarassingly parallel,
> and scale up with the number of cores in a cluster.
>
> Rasdaman is an array data base that comes in two flavours, the open
> source community edition (CE) and the commercial enterprise edition
> (EE). The differences between the two are clear [1]. When I want to
> use rasdaman CE (open source) for scalable image analysis, I get
> stuck waiting for one core to finish everything [1]. This is not
> going to solve any problems related to computing on large data,
> and is not scalable. The bold claim that rasdaman.org opens with
> ("This worldwide leading array analytics engine distinguishes itself
> by its flexibility, performance, and scalability") is not true for
> the CE advertised. This has been mentioned in the past on mailing
> lists [2,3], but the typical answer from Peter Baumann diverts into
> other arguments. Also the benchmark graph (photo from an AGU poster)
> [4] that Peter sent this week [5] must refer to the enterprise
> edition, since Spark and Hive both scale, but rasdaman CE does not [3].
>
> I assume that on the discussions on this list, ONLY the open
> source community edition is considered, compared, and discussed,
> as a potential future OSGeo project.
>
> OSGeo supports the needs of the open source geospatial community [6].
>
> Given
>
>  * the bold claims and continuing confusion about whether,
>    and which, rasdaman is scalable,
>  * the need for OSGeo to give good advice to prospective users
>    about technologies that do scale EO data analysis,
>  * the current (unfilled!) needs of scientists for good, open source
>    software for such analysis, and
>  * the potential conflict of interest of its creator [7],
>
> I wonder wether OSGeo should recommend rasdaman CE to the open
> source geospatial community.
>
>
> [1] http://rasdaman.org/wiki/Features
> [2] https://lists.osgeo.org/pipermail/incubator/2014-October/002540.html
> [3] https://groups.google.com/forum/#!topic/rasdaman-users/66XL3tmDDQI
> [4]https://lists.osgeo.org/pipermail/discuss/attachments/20160515/49200cd4/attachment.jpg
> [5] https://lists.osgeo.org/pipermail/discuss/2016-May/016099.html
> [6] http://www.osgeo.org/content/faq/foundation_faq.html
> [7] https://lists.osgeo.org/pipermail/discuss/2016-May/016045.html
>
>
>
> _______________________________________________
> Discuss mailing listDiscuss at lists.osgeo.orghttp://lists.osgeo.org/mailman/listinfo/discuss
>
>
> --
> Dr. Peter Baumann
>  - Professor of Computer Science, Jacobs University Bremen
>    www.faculty.jacobs-university.de/pbaumann
>    mail: p.baumann at jacobs-university.de
>    tel: +49-421-200-3178, fax: +49-421-200-493178
>  - Executive Director, rasdaman GmbH Bremen (HRB 26793)
>    www.rasdaman.com, mail: baumann at rasdaman.com
>    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
> "Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
>
>
>
>
> --
> Dr. Peter Baumann
>  - Professor of Computer Science, Jacobs University Bremen
>    www.faculty.jacobs-university.de/pbaumann
>    mail: p.baumann at jacobs-university.de
>    tel: +49-421-200-3178, fax: +49-421-200-493178
>  - Executive Director, rasdaman GmbH Bremen (HRB 26793)
>    www.rasdaman.com, mail: baumann at rasdaman.com
>    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
> "Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
>
>
>
>
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/discuss/attachments/20160521/a5b64a5d/attachment.html>