[gdal-dev] CAD (DWG) Driver

Mon Nov 21 02:10:25 PST 2016

The DWG driver contribution is much appreciated.  It currently supports older versions of files up to R1015 (AutoCAD 2000).  The DWG format was relatively straight forward up to that version.  In 2004 the format became rather more complex, including simple xor encryption and LZ77 compression of most of the records.  In 2007 the format became considerably more complex again, including Reed-Solomon encoding.  In 2010 the format became simpler again, going back to a format more similar to the 2004 format.  Our experience here at 1Spatial is that customers are on versions 2010 or 2013.  I have not received a DWG file from an earlier version from a customer.  Therefore we have implemented a library for reading versions 2010 and later (https://github.com/westbury/dwg-lib).  This has been implemented in Java and could readily be ported to C or otherwise compiled to a DLL, though as it is currently under active development this would need some thought to avoid maintaining two copies.  Alternatively we could leave it in Java and call it much as the GDAL GeoMedia driver does for Jackcess.   We have put the code under the MIT licence so that it would be available to GDAL.  The work was implemented by reference to the Open Design Specification for .dwg files Version 5.3.  We do plan on limited writing too, but probably updates to certain elements only, no support for outputting a complete DWG file from scratch.

Nigel Westbury

-----Original Message-----
From: gdal-dev [mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of gdal-dev-request at lists.osgeo.org
Sent: 19 November 2016 19:49
To: gdal-dev at lists.osgeo.org
Subject: gdal-dev Digest, Vol 150, Issue 33

Send gdal-dev mailing list submissions to
gdal-dev at lists.osgeo.org

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.osgeo.org/mailman/listinfo/gdal-dev
or, via email, send a message with subject or body 'help' to
gdal-dev-request at lists.osgeo.org

You can reach the person managing the list at
gdal-dev-owner at lists.osgeo.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of gdal-dev digest..."

Today's Topics:

   1. Re: CAD (DWG) Driver (Even Rouault)
   2. Re: CAD (DWG) Driver (Kurt Schwehr)
   3. Re: CAD (DWG) Driver (Even Rouault)
   4. Re: CAD (DWG) Driver (Dmitry Baryshnikov)
   5. Re: identifying the fields from the .osm file (Djordje Spasic)

----------------------------------------------------------------------

Message: 1
Date: Sat, 19 Nov 2016 17:49:25 +0100
From: Even Rouault <even.rouault at spatialys.com>
To: Dmitry Baryshnikov <bishop.dev at gmail.com>, schwehr at gmail.com,
gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] CAD (DWG) Driver
Message-ID: <7873061.FYxrsrM0Qa at even-i700>
Content-Type: text/plain; charset="us-ascii"

On samedi 19 novembre 2016 19:34:02 CET Dmitry Baryshnikov wrote:
> Hi Even,
>
> The big_endian test now is ok. But osx crash at the exiting of python
> after all test finished successfully.
>
> This is error message:
>
> libc++abi.dylib: terminating with uncaught exception of type
> std::__1::system_error: mutex lock failed: Invalid argument
> ./gdal/ci/travis/osx/script.sh: line 13: 56363 Abort trap: 6
> python run_all.py
>
> I find out that this may be the result of resources free order:
> https://github.com/dmlc/mxnet/issues/309
> https://github.com/google/benchmark/issues/52
>
> No idea why this occurred then C++11 used. I'll try to use osx CI test
> disabling the CAD driver to exclude this possible issue.
>

Hum, I think this might be indeed an issue with resource free order between the GDALDestroy() destructor function (in gcore/gdaldllmain.cpp) and the static C++11 mutex Kurt introduced in
https://github.com/OSGeo/gdal/commit/a9b947d6850d496f09e668f4cd148826d45d9fa9

If this mutex gets destroyed before GDALDestroy() is called, then things might indeed crash.

I think we should remove this static C++11 mutex. Kurt ?

(Another option would be to remove the library destructor function. I've the feeling they cause more harm than good.)

--
Spatialys - Geospatial professional services http://www.spatialys.com

------------------------------

Message: 2
Date: Sat, 19 Nov 2016 10:06:26 -0800
From: Kurt Schwehr <schwehr at gmail.com>
To: Even Rouault <even.rouault at spatialys.com>
Cc: Dmitry Baryshnikov <bishop.dev at gmail.com>, gdal dev
<gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] CAD (DWG) Driver
Message-ID:
<CACmBxysE_yyRtWmOCc=A6kEPLiwdkpi4k6=HEk8jJVpAuM_9ww at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I'm really not sure.

You can certainly try removing that C++11 mutex.  It was added based on the
comment at that point in the code.  We'll just have to keep an eye out for
possible occasional flakes.  If that's the case, we'll need to carefully
work through the logic and try to figure out a solution that works for
everything.

+1 for getting the driver into the tree.  It will be easier to make
additional changes to either it or GDAL with it in the same tree.

On Sat, Nov 19, 2016 at 8:49 AM, Even Rouault <even.rouault at spatialys.com>
wrote:

> On samedi 19 novembre 2016 19:34:02 CET Dmitry Baryshnikov wrote:
> > Hi Even,
> >
> > The big_endian test now is ok. But osx crash at the exiting of python
> > after all test finished successfully.
> >
> > This is error message:
> >
> > libc++abi.dylib: terminating with uncaught exception of type
> > std::__1::system_error: mutex lock failed: Invalid argument
> > ./gdal/ci/travis/osx/script.sh: line 13: 56363 Abort trap: 6
> > python run_all.py
> >
> > I find out that this may be the result of resources free order:
> > https://github.com/dmlc/mxnet/issues/309
> > https://github.com/google/benchmark/issues/52
> >
> > No idea why this occurred then C++11 used. I'll try to use osx CI test
> > disabling the CAD driver to exclude this possible issue.
> >
>
> Hum, I think this might be indeed an issue with resource free order between
> the GDALDestroy() destructor function (in gcore/gdaldllmain.cpp) and the
> static C++11 mutex Kurt introduced in
> https://github.com/OSGeo/gdal/commit/a9b947d6850d496f09e668f4cd1488
> 26d45d9fa9
>
> If this mutex gets destroyed before GDALDestroy() is called, then things
> might
> indeed crash.
>
> I think we should remove this static C++11 mutex. Kurt ?
>
> (Another option would be to remove the library destructor function. I've
> the
> feeling they cause more harm than good.)
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>

--
--
http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161119/da36d65e/attachment-0001.html>

------------------------------

Message: 3
Date: Sat, 19 Nov 2016 19:24:16 +0100
From: Even Rouault <even.rouault at spatialys.com>
To: Kurt Schwehr <schwehr at gmail.com>
Cc: Dmitry Baryshnikov <bishop.dev at gmail.com>, gdal dev
<gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] CAD (DWG) Driver
Message-ID: <1731467.ZNQqGo50XJ at even-i700>
Content-Type: text/plain; charset="us-ascii"

On samedi 19 novembre 2016 10:06:26 CET Kurt Schwehr wrote:
> I'm really not sure.

Not sure that the crash mentionned by Dmitry is related to that ? It looks
like a really good canditate to me. That's the only C++11 mutex in GDAL, with
2  other ones in the GeoTIFF driver (and those are likely not triggered at
process termination)

>
> You can certainly try removing that C++11 mutex.  It was added based on the
> comment at that point in the code.  We'll just have to keep an eye out for
> possible occasional flakes.  If that's the case, we'll need to carefully
> work through the logic and try to figure out a solution that works for
> everything.

I doubt we will see flakes related to that. The mutex was added to prevent
races in GDALDestroyDriverManager(), but I'm not aware of code in GDAL and its
utilities that would call it from several threads. And the function has always
been documented as thread-unsafe.

>
> +1 for getting the driver into the tree.  It will be easier to make
> additional changes to either it or GDAL with it in the same tree.
>
> On Sat, Nov 19, 2016 at 8:49 AM, Even Rouault <even.rouault at spatialys.com>
>
> wrote:
> > On samedi 19 novembre 2016 19:34:02 CET Dmitry Baryshnikov wrote:
> > > Hi Even,
> > >
> > > The big_endian test now is ok. But osx crash at the exiting of python
> > > after all test finished successfully.
> > >
> > > This is error message:
> > >
> > > libc++abi.dylib: terminating with uncaught exception of type
> > > std::__1::system_error: mutex lock failed: Invalid argument
> > > ./gdal/ci/travis/osx/script.sh: line 13: 56363 Abort trap: 6
> > > python run_all.py
> > >
> > > I find out that this may be the result of resources free order:
> > > https://github.com/dmlc/mxnet/issues/309
> > > https://github.com/google/benchmark/issues/52
> > >
> > > No idea why this occurred then C++11 used. I'll try to use osx CI test
> > > disabling the CAD driver to exclude this possible issue.
> >
> > Hum, I think this might be indeed an issue with resource free order
> > between
> > the GDALDestroy() destructor function (in gcore/gdaldllmain.cpp) and the
> > static C++11 mutex Kurt introduced in
> > https://github.com/OSGeo/gdal/commit/a9b947d6850d496f09e668f4cd1488
> > 26d45d9fa9
> >
> > If this mutex gets destroyed before GDALDestroy() is called, then things
> > might
> > indeed crash.
> >
> > I think we should remove this static C++11 mutex. Kurt ?
> >
> > (Another option would be to remove the library destructor function. I've
> > the
> > feeling they cause more harm than good.)
> >
> > --
> > Spatialys - Geospatial professional services
> > http://www.spatialys.com

--
Spatialys - Geospatial professional services
http://www.spatialys.com

------------------------------

Message: 4
Date: Sat, 19 Nov 2016 22:22:13 +0300
From: Dmitry Baryshnikov <bishop.dev at gmail.com>
To: Even Rouault <even.rouault at spatialys.com>, Kurt Schwehr
<schwehr at gmail.com>
Cc: gdal dev <gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] CAD (DWG) Driver
Message-ID: <69e67501-5af0-d09a-bf2a-4faed559ad6f at gmail.com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"

I commented out the C++11 mutex in gdaldrivermanager.cpp
<https://github.com/OSGeo/gdal/pull/164/commits/d1449e2eb5a85a6ec2520ad77ee579015fe14197#diff-e3e90e0089f1b70ddf36ebe89a73a733>
- this helps. OSX CI test passed successfully.

https://travis-ci.org/OSGeo/gdal/jobs/177297023

Best regards,
     Dmitry

19.11.16 21:24, Even Rouault пишет:
> On samedi 19 novembre 2016 10:06:26 CET Kurt Schwehr wrote:
>> I'm really not sure.
> Not sure that the crash mentionned by Dmitry is related to that ? It looks
> like a really good canditate to me. That's the only C++11 mutex in GDAL, with
> 2  other ones in the GeoTIFF driver (and those are likely not triggered at
> process termination)
>
>> You can certainly try removing that C++11 mutex.  It was added based on the
>> comment at that point in the code.  We'll just have to keep an eye out for
>> possible occasional flakes.  If that's the case, we'll need to carefully
>> work through the logic and try to figure out a solution that works for
>> everything.
> I doubt we will see flakes related to that. The mutex was added to prevent
> races in GDALDestroyDriverManager(), but I'm not aware of code in GDAL and its
> utilities that would call it from several threads. And the function has always
> been documented as thread-unsafe.
>
>> +1 for getting the driver into the tree.  It will be easier to make
>> additional changes to either it or GDAL with it in the same tree.
>>
>> On Sat, Nov 19, 2016 at 8:49 AM, Even Rouault <even.rouault at spatialys.com>
>>
>> wrote:
>>> On samedi 19 novembre 2016 19:34:02 CET Dmitry Baryshnikov wrote:
>>>> Hi Even,
>>>>
>>>> The big_endian test now is ok. But osx crash at the exiting of python
>>>> after all test finished successfully.
>>>>
>>>> This is error message:
>>>>
>>>> libc++abi.dylib: terminating with uncaught exception of type
>>>> std::__1::system_error: mutex lock failed: Invalid argument
>>>> ./gdal/ci/travis/osx/script.sh: line 13: 56363 Abort trap: 6
>>>> python run_all.py
>>>>
>>>> I find out that this may be the result of resources free order:
>>>> https://github.com/dmlc/mxnet/issues/309
>>>> https://github.com/google/benchmark/issues/52
>>>>
>>>> No idea why this occurred then C++11 used. I'll try to use osx CI test
>>>> disabling the CAD driver to exclude this possible issue.
>>> Hum, I think this might be indeed an issue with resource free order
>>> between
>>> the GDALDestroy() destructor function (in gcore/gdaldllmain.cpp) and the
>>> static C++11 mutex Kurt introduced in
>>> https://github.com/OSGeo/gdal/commit/a9b947d6850d496f09e668f4cd1488
>>> 26d45d9fa9
>>>
>>> If this mutex gets destroyed before GDALDestroy() is called, then things
>>> might
>>> indeed crash.
>>>
>>> I think we should remove this static C++11 mutex. Kurt ?
>>>
>>> (Another option would be to remove the library destructor function. I've
>>> the
>>> feeling they cause more harm than good.)
>>>
>>> --
>>> Spatialys - Geospatial professional services
>>> http://www.spatialys.com
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161119/78087c6b/attachment-0001.html>

------------------------------

Message: 5
Date: Sat, 19 Nov 2016 19:45:40 +0000 (UTC)
From: Djordje Spasic <issworld2000 at yahoo.com>
To: Etienne Trimaille <etienne.trimaille at gmail.com>
Cc: "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] identifying the fields from the .osm file
Message-ID: <840769665.311796.1479584740697 at mail.yahoo.com>
Content-Type: text/plain; charset="utf-8"

Thank you Etienne.
I am very interested in your reply, but I haven't quite understand you.
The ogr2ogr.exe file is using a particular python script to read the .osm file and identify its unique fields?
Can you be a bit more specific please? Solving this issue will mean a lot to me.
Sorry for disturbance, and thank you for the clarification in advance.Regards,
Djordje

    On Saturday, November 19, 2016 5:09 PM, Etienne Trimaille <etienne.trimaille at gmail.com> wrote:

 The parsing is done using python in the plugin. We can just use the parser by calling the python file. Python is included when you install osgeo4w.Yes you can set the osmconf.ini or directly use the goal ogr api to write the output file.
On 19 Nov 2016 18:00, "Djordje Spasic" <issworld2000 at yahoo.com> wrote:

Thank you for the quick reply Etienne!

I am stuck with ogr2ogr.exe, or GDAL C# libraries. I can not use QGIS or other applications.
Indeed Even told me that identifying all fields in the .osm file, would require doing 2 read passes of the .osm file.
I thought that maybe there is a way of doing exactly that, where in the first read I would get all the fields, find only those that are unique, and then perform a second read by setting these fields in the osmconf.ini file?

    On Saturday, November 19, 2016 2:40 AM, Etienne Trimaille <etienne.trimaille at gmail.com> wrote:

 Hi,
With ogr2ogr, you can't read all keys from an OSM file.
As Even said, you have to use another loop to read all available keys in the file. You can have a look at the QGIS QuickOSM plugin. You can open a OSM file and ask the plugin to read all keys available in the file.
2016-11-19 8:30 GMT+07:00 Djordje Spasic <issworld2000 at yahoo.com>:

During the last couple of days, both Jukka and Even helped me with some .osm file issues. And I am grateful on that.

I would like to ask one more question, if that's ok:
Is it possible to open an .osm file through ogr2ogr.exe and somehow read all the fields (keys) used in that .osm file? The final list of those fields do not have to be unique, they can repeat.
Is it possible to somehow save all those fields to a .shp or .txt file through ogr2ogr.exe?

Thank you for the reply.

______________________________ _________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman /listinfo/gdal-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161119/ebd0edd4/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

------------------------------

End of gdal-dev Digest, Vol 150, Issue 33
*****************************************
 [http://www.1spatial.com/signatures/1SpatialLogo_Small.jpg] <http://www.1spatial.com>

[http://www.1spatial.com/wp-content/themes/1spatial/images/youtube.png]<http://www.youtube.com/user/1SpatialGroup> [http://www.1spatial.com/wp-content/themes/1spatial/images/facebook.png] <https://www.facebook.com/1Spatial>  [http://www.1spatial.com/wp-content/themes/1spatial/images/linkedIn.png] <http://www.linkedin.com/company/1spatial>  [http://www.1spatial.com/wp-content/themes/1spatial/images/twitter.png] <http://www.twitter.com/1spatial>

________________________________

The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorised. Any views or opinions expressed in this e-mail may be solely those of the author and are not necessarily those of 1Spatial Plc. This email has been scanned for viruses and message content. 1Spatial Group Limited is a company registered in England and Wales No.4785688. Registered Office: Tennyson House, Cambridge Business Park, Cambridge CB4 0WZ