[Live-demo] Converting docs to UTF encoding

[Jorge Gaspar Sanz Salinas]

On 5 August 2012 23:52, Frank Gasdorf <fgdrf at users.sourceforge.net> wrote:
> 2012/8/5 Cameron Shorter <cameron.shorter at gmail.com>:
>> Frank, good idea,
>> The place to capture this information is on this wiki page:
>> http://wiki.osgeo.org/wiki/Live_GIS_Translate
>
> Cameron, I guess the sphinx syntax thing would be better preserved in
> sphinx docs itself because here you can see what you get. IMHO its
> better for everybody to see a result of the rst rendering/theming
> process directly source and html side by side.
>
> I agree to share best practices at the place you suggested ;)
>
> Cheers,
> Frank
>
>>
>> On 06/08/12 07:26, Frank Gasdorf wrote:
>>>
>>> Alex, thanks for sharing this hint!! In the past we had several issues
>>> with other languages as well (es, el, etc). Properly we should think
>>> about a sphinx page "Creating OSGeo-Live Docs with sphinx" with DO's
>>> and DON'Ts for such cases.
>>>
>>> In this case I use notepad++ on windows or EncodingMaster on Mac OS X
>>> to convert between encodings afterwards. I absolutely agree with you,
>>> the Documentation Guilde should have your hint, independent from the
>>> doc language your are working on to choose UTF-8 File-Encoding because
>>> sphinx has set utf-8 to default source encoding
>>> (http://sphinx.pocoo.org/rest.html#source-encoding)
>>>
>>> What do you think about a documentation_guide.rst file in doc folder
>>> of svn source checkout, where we could just have templates in how to
>>> define headers, how to deal with directives and references. A pretty
>>> good example to start from is the geoserver sphinx guide
>>> (http://docs.geoserver.org/trunk/en/docguide/sphinx.html).
>>>
>>> possible  guide structure:
>>>
>>> - general sphinx syntax
>>> - how to create pages
>>>      - quickstart_template
>>>      - overview_template
>>>      - (standard_template)
>>>
>>> - How to start with tranlations
>>>   (e.g. How to start translating files (copy paste, encodings, etc)
>>>
>>> - References
>>>    - sphinx documentations
>>>
>>> Opinions?
>>>
>>> Cheers,
>>> Frank
>>>
>>>
>>>
>>> 2012/8/5 Alex Mandel <tech_dev at wildintellect.com>:
>>>>
>>>> So one issue we ran into translating Asian languages, when you do an svn
>>>> copy (or a cp) of the English and then start to change the text the file
>>>> is still encoding as Latin. Which appears fine to the end user but ends
>>>> up with ????? in the sphinx builds.
>>>>
>>>> The workaround we found is to open a blank file, choose utf8 without BOM
>>>> as the encoding, copy and paste the text from the original and then save
>>>> over the original. Note- Using Notepad++ on Windows for this.
>>>>
>>>> There must be a better way.
>>>>
>>>> Thanks,
>>>> Alex
>>>> _______________________________________________
>>>> Live-demo mailing list
>>>> Live-demo at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/live-demo
>>>> http://live.osgeo.org
>>>> http://wiki.osgeo.org/wiki/Live_GIS_Disc
>>>
>>> _______________________________________________
>>> Live-demo mailing list
>>> Live-demo at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/live-demo
>>> http://live.osgeo.org
>>> http://wiki.osgeo.org/wiki/Live_GIS_Disc
>>
>>
>>
>> --
>> Cameron Shorter
>> Geospatial Solutions Manager
>> Tel: +61 (0)2 8570 5050
>> Mob: +61 (0)419 142 254
>>
>> Think Globally, Fix Locally
>> Geospatial Solutions enhanced with Open Standards and Open Source
>> http://www.lisasoft.com
>>
>>
>> _______________________________________________
>> Live-demo mailing list
>> Live-demo at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/live-demo
>> http://live.osgeo.org
>> http://wiki.osgeo.org/wiki/Live_GIS_Disc
> _______________________________________________
> Live-demo mailing list
> Live-demo at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/live-demo
> http://live.osgeo.org
> http://wiki.osgeo.org/wiki/Live_GIS_Disc

Well I'm a Linux guy and since some of the Spanish translators were
not really "encoding aware" I ended up with some CLI to fix things,
let me share it:

1st: Find suspicious files:
=======================================================

find . -type f -name "*.rst" -exec file {} \; | grep ISO-8859   <= encoding
find . -type f -name "*.rst" -exec file {} \; | grep CRLF         <= line ending
find . -type f -name "*.rst" -exec file {} \; | grep empty        <= empty

2nd Fix the encoding, usnig iconv
=========================================

I=mapproxy_quickstart.rst; iconv --from-code=ISO-8859-1 --to-code=UTF-8
 $I > ${I}2; rm $I; mv ${I}2 $I


3rd Fix end line codes usnig dos2unix
=========================================

dos2unix sahana_quickstart.rst


-- 
Jorge Sanz
http://es.osgeo.org