[gdal-dev] Segmentation fault (core dumped) occurs when running ogr2ogr command on container

Tue Mar 12 02:25:47 PDT 2024

Hi,

I had a try with your dataset on Windows and I had no segmentation fault. Perhaps not surprise because I do not use the same container than you. I had some encoding problem but I believe that it is not related. But perhaps you have some use for knowing that the unzipping part is not necessary, shapefile can be read directly from the zip with /vsizip/.

ogr2ogr -f GeoJSON -oo ENCODING=MS932 d1_raw.geojson /vsizip/test.zip d1  --config CPL_DEBUG ON
GDAL: Auto register C:\OSGeo4W\apps\gdal\lib\gdalplugins\gdal_ECW_JP2ECW.dll using GDALRegister_ECW_JP2ECW.
GDAL: GDALOpen(/vsizip/test.zip, this=0000023F19871250) succeeds as ESRI Shapefile.
GDAL: GDALDriver::Create(GeoJSON,d1_raw.geojson,0,0,0,Unknown,0000000000000000)
Shape: DBF Codepage = SJIS for /vsizip/test.zip/d1.shp
Shape: Treating as encoding 'MS932'.
Shape: Cannot recode from 'MS932'. Disabling recoding
GDALVectorTranslate: 5315 features written in layer 'd1'
Shape: 5315 features read on layer 'd1'.
GDAL: GDALClose(/vsizip/test.zip, this=0000023F19871250)
GDAL: GDALClose(d1_raw.geojson, this=0000023F19876CF0)
GDAL: In GDALDestroy - unloading GDAL shared library.

-Jukka Rahkonen-

Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org> Puolesta ?? ?? via gdal-dev
Lähetetty: tiistai 12. maaliskuuta 2024 10.56
Vastaanottaja: Abel Pau <a.pau at creaf.uab.cat>; gdal-dev at lists.osgeo.org
Aihe: Re: [gdal-dev] Segmentation fault (core dumped) occurs when running ogr2ogr command on container

Hi,

I tried executing with --config CPL_DEBUG ON, but it did not provide any useful information.

I'm sending you the command that's causing the issue.
We're using curl command to obtain the Shape file from the internet.

---

  1.  Obtaining the Shape file:
$curl -f --output-dir ./ -O https://data.bodik.jp/dataset/d4b9a284-58c0-4559-9e48-b7c83a4275f0/resource/28aa8f13-7cf0-4dc5-9dbf-a77f58c3f2c8/download/shape.zip

  1.  Unzipping the ZIP file:
$unzip -o -j -d ./ ./shape.zip

③ Converting from Shape format to GeoJSON format:
$ogr2ogr -f GeoJSON -oo ENCODING=MS932 ./d1_raw.geojson ./d1.shp --config CPL_DEBUG ON
---
These commands are executed within the convert.sh called from the Dockerfile.

Best regards,

From: Abel Pau <a.pau at creaf.uab.cat<mailto:a.pau at creaf.uab.cat>>
Sent: Wednesday, March 6, 2024 5:53 PM
To: 寛座　智大 <tomohiro.kanza.306 at baycurrent.co.jp<mailto:tomohiro.kanza.306 at baycurrent.co.jp>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Subject: RE: Segmentation fault (core dumped) occurs when running ogr2ogr command on container

Hi,

I reduced the amount of memory requested. I could do that because I am developing the driver that failed.
After my mail I discovered another error, so, perhaps, this was the real cause.

4Mb it’s so small, so I is not the cause...
Perhaps you can add --config  CPL_DEBUG ON at your command line to see if this give you more information about the place where it crashes.

I’m sorry not being able to help you more.
Perhaps you can send me the shape and the command line to see if it crashes in my computer and I can find out the problem.

See you

De: 寛座　智大 <tomohiro.kanza.306 at baycurrent.co.jp<mailto:tomohiro.kanza.306 at baycurrent.co.jp>>
Enviado el: dimecres, 6 de març de 2024 7:20
Para: Abel Pau <a.pau at creaf.uab.cat<mailto:a.pau at creaf.uab.cat>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Asunto: RE: Segmentation fault (core dumped) occurs when running ogr2ogr command on container

Abel Pau,

Hi. Thank you for your response.

>I reduced this capacity 10 times and it worked.
Does this mean you reduced the size of the Shape file?
Or does it mean that you reduced the amount of memory requested?
( If it's the latter, I'd like to learn how to do it.)

Shape files are typically around 4MB, so I don't consider them to be very large files.
Additionally, I tried it on an AKS with a free memory size of over 20GB, but encountered the same issue.
It seems unlikely that there is a lack of specifications in the execution environment.

Could the issue possibly be due to a bug in the command?

From: Abel Pau <a.pau at creaf.uab.cat<mailto:a.pau at creaf.uab.cat>>
Sent: Monday, March 4, 2024 4:57 PM
To: 寛座　智大 <tomohiro.kanza.306 at baycurrent.co.jp<mailto:tomohiro.kanza.306 at baycurrent.co.jp>>
Subject: RE: Segmentation fault (core dumped) occurs when running ogr2ogr command on container

Hi,
I did similar tests with a driver I am implementing and I obtained exact the same behavior as yours.
After a few speculations I deduced that my driver was asking for too much memory for the docker capacity.
I reduced this capacity 10 times and it worked.
So, perhaps, Shape or Geojson asks for too much memory for the docker capacity.
I don’t know if it’s possible to make it bigger.

Abel.

De: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> En nombre de ?? ?? via gdal-dev
Enviado el: dilluns, 4 de març de 2024 3:30
Para: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Asunto: [gdal-dev] Segmentation fault (core dumped) occurs when running ogr2ogr command on container

I am implementing batch processing to convert Shape files, publicly available as open data,

into Geojson format within a container environment.

I installed gdal-bin on top of ubuntu:22.04 base image.

The Dockerfile I implemented is as follows:

```

FROM ubuntu:22.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt update

RUN apt -y install apt-transport-https jq unzip openjdk-17-jdk-headless curl

# Install gdal-bin

# Reference: https://mothergeo-py.readthedocs.io/en/latest/development/how-to/gdal-ubuntu-pkg.html#how-to-install-gdal-ogr-packages-on-ubuntu

RUN apt -y install software-properties-common

RUN add-apt-repository -y ppa:ubuntugis/ppa

RUN apt update

RUN apt -y install gdal-bin

COPY ./convert.sh /convert.sh

CMD ["bash","convert.sh"]

```

When executing the ogr2ogr command within convert.sh, the following error occurs:

```

Shape: DBF Codepage = SJIS for /d1.shp

Shape: Treating as encoding 'MS932'.

GDAL: GDALOpen(/d1.shp, this=0x55f1edd77f30) succeeds as ESRI Shapefile.

GDAL: QuietDelete(/d1_raw.geojson) invoking Delete()

GDAL: GDALDriver::Create(GeoJSON,/d1_raw.geojson,0,0,0,Unknown,(nil))

GDALVectorTranslate: 5315 features written in layer 'd1'

Shape: 5315 features read on layer 'd1'.

GDAL: GDALClose(/d1.shp, this=0x55f1edd77f30)

GDAL: GDALClose(/d1_raw.geojson, this=0x55f1edd75fd0)

GDAL: In GDALDestroy - unloading GDAL shared library.

Segmentation fault (core dumped)

```

I checked if convert.sh execution is possible on the VM (host OS running Docker).

The error seems to occur only when executed within the container.

How should I resolve this?

※Environment

Host OS: ubuntu 22.04

Docker Deamon: 25.0.3

gdal-bin: 3.6.4

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240312/6aa38fea/attachment-0001.htm>