<div dir="ltr">Peter,<div><br></div><div>It's impressive that 20 TB is being served out by the community edition. However, the point that Edzer brings up (and that I was unaware of) causes me even more concern regarding the language used in service of promoting the potential OSGeo version of rasdaman around it's comparison to other projects.</div><div><br></div><div>Are the performance benchmarks that you were siting as sources based on queries that would benefit from distributed processing (which is a very large class of queries)? Are the benchmarks using the open source rasdaman or the proprietary version? If you are going to compare yourself to a distributed processing engine such as Apache Spark, you should be comparing it fairly - Apache Spark does not claim to be a fast solution in a single machine case. If you are comparing it to the proprietary version of rasdaman, then I'm wondering why you are referencing those numbers when talking about the open source version, which has been the subject of these threads of conversation. Google Earth Engine may beat us all out in performance, but that doesn't directly contribute much to the open source geospatial community because it lacks that key openness.</div><div><br></div><div>If rasdaman CE does not scale horizontally, there are use cases and example queries with which I am very confident rasdaman would be outperformed by systems constructed at their core with Apache Spark. If I've misunderstood something, or if this is somehow not the case, please let me know.</div><div><br></div><div>Best,</div><div>Rob</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, May 21, 2016 at 4:07 AM, Peter Baumann <span dir="ltr"><<a href="mailto:p.baumann@jacobs-university.de" target="_blank">p.baumann@jacobs-university.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
oh, just looking at the subject again:<br>
<br>
several service providers believe indeed rasdaman community does
offer a significant advantage:<br>
- see the download figures on <a href="http://www.rasdaman.org" target="_blank">www.rasdaman.org</a><br>
- concretely, see <a href="http://www.planetserver.eu" target="_blank">www.planetserver.eu</a> which is running rasdaman
community on - I believe - about 20 TB of Planetary Science data.<span class="HOEnZb"><font color="#888888"><br>
<br>
-Peter</font></span><div><div class="h5"><br>
<br>
<br>
<div>On 05/21/2016 09:56 AM, Peter Baumann
wrote:<br>
</div>
<blockquote type="cite">
Hm, first of all: this is opening a different thread, talking
about functionality of rasdaman community. Next, it is based on
assumptions - without details (because off topic): conclusions are
wrong.<br>
<br>
But to respond to the core message emphasized in the first
paragraph: I respectfully disagree. In particular, such a position
does not benefit the open source community very much as I am
trying to explain below. <br>
<br>
TL;DR:<br>
<br>
You have a strong expertise in Geoinformatics, I know something
about Computer Science. This is where we can talk as professors
and scientists. Your statement is about economics, industry etc.
Having an opinion there (and articulate it) is fair, but in these
fields our opinion weighs not more than anyone else's in the
street. We should not attempt to attain importance through
inapplicable roles.<br>
<br>
Let us look at a professor. They have a conveniently high salary
which is paid by society, that is: tax payers. Nobody can
influence what a professor does and how much return s/he generates
for society.<br>
<br>
A single open source developer (or a small group, whatever) do not
experience this convenience. They have a dream where they invest,
they try to not make money for getting richer than a professor ;-)
but merely for their economic survival. Some (in particular
scientists) enjoy the money rain coming from publicly funded
projects (again: the tax payer subsidizes), but most in the
community have to struggle hard. They face reluctant customers,
competition by the giants in the market, and many more obstacles.<br>
<br>
From the cosy place of a lifelong position with a secured salary
and decent retirement funds it is easy to say that all software
should be free like free beer (quote from below: "can be
reproduced by other scientists without prohibitive license
costs").<br>
<br>
If the open source movement cannibalizes itself it will make it
all so easy for the big players to maintain their dominance, they
will silently applaud. Quoting Jeroen:<br>
> NEVER IGNORE COMPANIES AGAIN IN OSGEO OR FOSS4G! THEY ARE NOT
A THREAT, THEY ARE A NECESSITY.<br>
<br>
That said: It is entirely ok to have the opinion you have. Others,
though, may disagree. I am one of those.<br>
<br>
respectfully,<br>
Peter<br>
<br>
<br>
<br>
<div>On 05/20/2016 09:30 AM, Edzer Pebesma
wrote:<br>
</div>
<blockquote type="cite">
<pre>As a scientist, I teach my students that for doing science it is a
requirement to work with open source software, because only then
workflows are fully transparent and can be reproduced by other
scientists without prohibitive license costs. Currently, working
with large amounts of earth observation (EO) or climate model data
typically requires to download these data tile by tile, stitch them
together, and go through all of them. Array databases may simplify
this substantially: after ingesting the tiles, they can directly
work on the whole data as a multi-dimensinal array ("data cube").
Computations on these array are typically embarassingly parallel,
and scale up with the number of cores in a cluster.
Rasdaman is an array data base that comes in two flavours, the open
source community edition (CE) and the commercial enterprise edition
(EE). The differences between the two are clear [1]. When I want to
use rasdaman CE (open source) for scalable image analysis, I get
stuck waiting for one core to finish everything [1]. This is not
going to solve any problems related to computing on large data,
and is not scalable. The bold claim that <a href="http://rasdaman.org" target="_blank">rasdaman.org</a> opens with
("This worldwide leading array analytics engine distinguishes itself
by its flexibility, performance, and scalability") is not true for
the CE advertised. This has been mentioned in the past on mailing
lists [2,3], but the typical answer from Peter Baumann diverts into
other arguments. Also the benchmark graph (photo from an AGU poster)
[4] that Peter sent this week [5] must refer to the enterprise
edition, since Spark and Hive both scale, but rasdaman CE does not [3].
I assume that on the discussions on this list, ONLY the open
source community edition is considered, compared, and discussed,
as a potential future OSGeo project.
OSGeo supports the needs of the open source geospatial community [6].
Given
* the bold claims and continuing confusion about whether,
and which, rasdaman is scalable,
* the need for OSGeo to give good advice to prospective users
about technologies that do scale EO data analysis,
* the current (unfilled!) needs of scientists for good, open source
software for such analysis, and
* the potential conflict of interest of its creator [7],
I wonder wether OSGeo should recommend rasdaman CE to the open
source geospatial community.
[1] <a href="http://rasdaman.org/wiki/Features" target="_blank">http://rasdaman.org/wiki/Features</a>
[2] <a href="https://lists.osgeo.org/pipermail/incubator/2014-October/002540.html" target="_blank">https://lists.osgeo.org/pipermail/incubator/2014-October/002540.html</a>
[3] <a href="https://groups.google.com/forum/#%21topic/rasdaman-users/66XL3tmDDQI" target="_blank">https://groups.google.com/forum/#!topic/rasdaman-users/66XL3tmDDQI</a>
[4]
<a href="https://lists.osgeo.org/pipermail/discuss/attachments/20160515/49200cd4/attachment.jpg" target="_blank">https://lists.osgeo.org/pipermail/discuss/attachments/20160515/49200cd4/attachment.jpg</a>
[5] <a href="https://lists.osgeo.org/pipermail/discuss/2016-May/016099.html" target="_blank">https://lists.osgeo.org/pipermail/discuss/2016-May/016099.html</a>
[6] <a href="http://www.osgeo.org/content/faq/foundation_faq.html" target="_blank">http://www.osgeo.org/content/faq/foundation_faq.html</a>
[7] <a href="https://lists.osgeo.org/pipermail/discuss/2016-May/016045.html" target="_blank">https://lists.osgeo.org/pipermail/discuss/2016-May/016045.html</a>
</pre>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Discuss mailing list
<a href="mailto:Discuss@lists.osgeo.org" target="_blank">Discuss@lists.osgeo.org</a>
<a href="http://lists.osgeo.org/mailman/listinfo/discuss" target="_blank">http://lists.osgeo.org/mailman/listinfo/discuss</a></pre>
</blockquote>
<br>
<pre cols="80">--
Dr. Peter Baumann
- Professor of Computer Science, Jacobs University Bremen
<a href="http://www.faculty.jacobs-university.de/pbaumann" target="_blank">www.faculty.jacobs-university.de/pbaumann</a>
mail: <a href="mailto:p.baumann@jacobs-university.de" target="_blank">p.baumann@jacobs-university.de</a>
tel: <a href="tel:%2B49-421-200-3178" value="+494212003178" target="_blank">+49-421-200-3178</a>, fax: <a href="tel:%2B49-421-200-493178" value="+49421200493178" target="_blank">+49-421-200-493178</a>
- Executive Director, rasdaman GmbH Bremen (HRB 26793)
<a href="http://www.rasdaman.com" target="_blank">www.rasdaman.com</a>, mail: <a href="mailto:baumann@rasdaman.com" target="_blank">baumann@rasdaman.com</a>
tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: <a href="tel:%2B49-173-5837882" value="+491735837882" target="_blank">+49-173-5837882</a>
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
</pre>
</blockquote>
<br>
<pre cols="80">--
Dr. Peter Baumann
- Professor of Computer Science, Jacobs University Bremen
<a href="http://www.faculty.jacobs-university.de/pbaumann" target="_blank">www.faculty.jacobs-university.de/pbaumann</a>
mail: <a href="mailto:p.baumann@jacobs-university.de" target="_blank">p.baumann@jacobs-university.de</a>
tel: <a href="tel:%2B49-421-200-3178" value="+494212003178" target="_blank">+49-421-200-3178</a>, fax: <a href="tel:%2B49-421-200-493178" value="+49421200493178" target="_blank">+49-421-200-493178</a>
- Executive Director, rasdaman GmbH Bremen (HRB 26793)
<a href="http://www.rasdaman.com" target="_blank">www.rasdaman.com</a>, mail: <a href="mailto:baumann@rasdaman.com" target="_blank">baumann@rasdaman.com</a>
tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: <a href="tel:%2B49-173-5837882" value="+491735837882" target="_blank">+49-173-5837882</a>
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
</pre>
</div></div></div>
<br>_______________________________________________<br>
Discuss mailing list<br>
<a href="mailto:Discuss@lists.osgeo.org">Discuss@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/discuss" rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/discuss</a><br></blockquote></div><br></div>