Easy contributable internationalization process with Sphinx @ PyCon APAC 2016

Easy contributable internationalization process with Sphinx

Takayuki Shimizukawa

Sphinx co-maintainer

Sphinx-users.jp1

안녕하세요 .annyeonghaseyo.나는시미즈카와입니다 .

naneun shimizukawa imnida.

2

こんにちはKon-Nichi-Wa

3

Who am I

@shimizukawa

Python developer since 2004

Sphinx co-maintainerSphinx-users.jp organizerPyCon JP committeeBePROUD co, ltd. member

4

Agenda

1. Sphinx introduction & Setup for i18n

2. A easy way and Non easy ways to translate

3. Sphinx i18n feature

4. Automated translation process with several services

5. Tips6

Question

7

Question1

How many people do you use Open Source

Software?

8

Question2

How many people may have contributed to the

OSS?

9

Question3

Does the translation into familiar language

contribute to other developers?

10

11

Translator

Product Author

Translated Docs

Readers

English Docs

Translators

Tools & Services

Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for

Transifex

Services5. Transifex - translation support service6. Drone.io - continuous integration service

12

1. Sphinx introduction& Setup for i18n

13

What is Sphinx?

Sphinx is a documentation generator

Sphinx generates doc as several output formats from the reST text

markup

14


Sphinx

reSTreSTreStructuredText

(reST)reST

Parser

HTML Builder

ePub Builder

LaTeX Builder texlive

HTMLtheme

Favorite Editor

The history of Sphinx (short ver.)

15


The father of Sphinx

Too hard to maintenance

~2007

Easy to writeEasy to

maintenance2007~

Sphinx Before and After

Before

There was no standard ways to write documentsSometime, we need converting markups into other

formats

After

Generate multiple output format from single sourceIntegrated html themes make read docs easierAPI references can be integrated into narrative docsAutomated doc building and hosting by ReadTheDocs

16


Many docs are written by SphinxFor examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

17


Many docs are written by SphinxFor examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

18


SPHINXdocutils

HTML Builder HTML theme(Jinja2)

gettext Builder

*.pot

*.po

I18N

*.mo OmegaT

Pootle

Transifex

Translation Tools, Services

Favorite Editor

Sphinx i18n feature (built-in)

19


reST Parser(directive / role)

doctree(Intermediat

e)

reSTreSTreStructuredText(reST)

$ pip install sphinx

Support tools for translation

How to install Sphinx with i18n tools

To build a sphinx document

$ pip install sphinx-intl$ pip install transifex-client

20


$ git clone https://github.com/shimizukawa/deepthought.git$ cd deepthought/doc$ make html...Build finished. The HTML pages are in _build/html.

"make html" command generates html files into _build/html.

make html once

21


2. Easy contributable internationalization process with Sphinx

23

Easy contributable internationalizatio

n process with Sphinx

24

What is internationalization?

25


I18N

Easy contributable internationalization process with Sphinx

26

What is easy contributable?

27


Not a easy way (example 1/3)

The manual are provided only in the HTML files

You can rewrite HTML filesHow to follow the original?

28



The HTML manual are generated fromreST files and

docstrings in the source files

You can rewrite reST files and docstringsHow to follow the upstream?

29



OK, we are engineers. we can use GIT!

Learn gitLearn GitHubgit clone to get the codeTranslation (Yes! This is what I want to

do!)git commitgit pullSometimes you must resolve conflictgit push

30


Not easy vs easy

Not a easy way Easy way1. Learn git2. Learn GitHub3. git clone to get the

code4. Translation5. git commit, pull,

push6. Resolve conflict7. Build html your self

1. No git2. No Github3. No file4. Translation5. Update

Automatically6. No conflict7. You can get a HTML

output w/o hand-build.

31



32

Two i18n features of SphinxOutput pot files:

from reST

Input po files:to generate translated HTML

33


reST pot

reST

htmlpo

Translation flowGenerate pot

Translate pot into po

Generate Translated HTML

34


reST pot

reST

htmlpo

pot poTranslator

Pleasetranslate

Translate!Thanks for yourContribution!

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""

msgid "For example:"msgstr ""

35

Output pot files 3. Sphinx i18n feature

$ make gettext...Build finished. The message catalogs are in _build/gettext.

$ ls _build/gettextapi.pot examples.pot generated.pot index.pot

generated.pot

reST pot

37

Preparing po files to translate

$ sphinx-intl update -p _build/gettext -l koCreate: locale/ko/LC_MESSAGES/api.poCreate: locale/ko/LC_MESSAGES/examples.poCreate: locale/ko/LC_MESSAGES/generated.poCreate: locale/ko/LC_MESSAGES/index.po

At first, sphinx-intl copy pot files and rename them

pot po

sphinx-intl

$ make gettext$ sphinx-intl update -p _build/gettext -l koNot Changed: locale/ko/LC_MESSAGES/api.poUpdated: locale/ko/LC_MESSAGES/examples.po +3, -1Not Changed: locale/ma/LC_MESSAGES/generated.po

After change the document, sphinx-intl update differences


Translate po files

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr ""

generated.po

pot po

sphinx-intl

Translator

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr "JSON 형식의 :class:`str` 유형에 `òbj`` 를 직렬화 ."

generated.po

Translate by using Vim, Emacs, OmegaT, ...

38


Input po files

39


reST

htmlpo

Translated

$ make html...Build finished. The HTML pages are in _build/html.

Big picture

40


reST pot

html

po

make gettext

sphinx-intl

Translator

make html

Doc Author

TranslationCatalog

Translatedcatalog


41

Entire process to translate sphinx docs

42


reST pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

Translator

Translator

Translator

Autor / Translator

upload

Translator

clone

Translator

To Be

43


reST pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

upload

Translators

To BeAutomated

To BeParallel

clone

Translation tool typesVim / Emacs (Editors)

Edit local filesTranslation support plugins are available.

OmegaT (Translation Tools)Edit local files.Translation support features.

Transifex (Services)Edit onlineTranslation support

features.

44


po

Translators

To BeParallel

Be Parallel by using Transifex

Transifex provides...As API:Upload potDownload po

As Web console:GlossaryTranslation memoryRecommendationAuto-translation

45


po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client

po transifex-clientDownload

Translation on Transifex web console

46


Original Text

Translated Text(you should keep reST syntax)

Suggestions fromTranslation Memory (TM)

Original(pot)

Translation(po)

Copy orig to translationMachine translation

SaveReview (if needed)

Translators

Parallel1

2

4

3

56

To Be Automated

47


po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client


reST

html

make gettext

make html

Doc Author

upload

To BeAutomated

clone

To Be Automated

48


pot

Upload sphinx-intltransifex-client


reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

The procedure for automation

1. clone repository2. make gettext3. Upload pot4. Download po5. make html6. Upload html

49


pot



reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomate

d

1. pip install sphinx sphinx-intl transifex-client2. git clone https://github.com/shimizukawa/deepthought.git3. cd deepthought/doc4. sphinx-intl create-transifexrc # create ~/.transifexrc5. sphinx-intl create-txconfig # create .tx/config6. make gettext7. sphinx-intl -p _build/gettext update-txconfig-resources

# update .tx/config8. tx push -s # push pot files to

transifex9. tx pull -l ko # pull po files from

transifex10. make html SPHINXOPTS="-D language=ko"

Shell commands for automationrun.sh

$ export SPHINXINTL_TRANSIFEX_USERNAME=mice$ export SPHINXINTL_TRANSIFEX_PASSWORD=42$ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7$ export SPHINXINTL_POT_DIR=_build/gettext$ run.sh

50


The drone.io

51

WebHook

Deploy

Clone repositoryRun shell script

The drone.io is a continuous integration service.


GitHub + drone.io + S3

52

GitHub

Amazon S3

Transifex 1. Clone repository2. make gettext.3. Upload pot4. Download po5. make html6. Upload html


2

1

3

Be Automated

53

pot



reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

UploadpotDownload

poupload

WebHookclone

1

5

6make html

make gettext2

3

4

1WebHook

Automated


Automated by drone.io

54

UploadpotDownload

poupload

WebHookclone

1

56

make html

make gettext2

3

4

1WebHook

Automated


View from doc author

Doc Author doesn't require annoying procedure.

UpdateTranslation Source

Doc Author

Notify

See

Commit

55


View from doc translatorsNo gitNo fileNo conflictUpdate AutomaticallyThey can get a translated HTML output w/o hand-

build.

Translators

ParallelSee

TranslateTranslated Pages

56


The entire automated process

57

UpdateTranslation Source

Doc Author

Translators

Parallel

Notify

See

See

Publish

Translate

Commit

DownloadTranslations

Notify

Automated


Summary

Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for

Transifex

Services5. Transifex - translation support service6. Drone.io - continuous integration service

58


5. Tips

59

msgid "Sphinx translates a set of reStructuredText_ source filesinto various output formats."msgstr "Sphinx translates a set of ““`reStructuredText <http://docutils.sphinx-users.jp/web/rst.html>`_ ““source files into various output formats."

*.po

How to handle URLs

60

5. TIPS

Sphinx translates a set of reStructuredText_ source files intovarious output formats.

.. _reStructuredText: http://docutils.sourceforge.net/rst.html

*.rstHyperlink

Target

Hyperlink Target doesn’t appear in POT files.If you want to change URLs into translation

version, you can use Embedded URLs notation.

How to change language w/o rewriting conf.py

You can use SPHINXOPTS make optionIt also works with other targets not only

’html’.linkcheck target is useful if your translation

contains URLs. It will find typos in translation text.

61

5. TIPS

$ make linkcheck SPHINXOPTS="-D language=ko"

$ make html SPHINXOPTS="-D language=ko"

You can control pot file resolutionUse gettext_compat

By default ‘gettext_compat = True’.reST files under sub directories are collected

into single POT file.

62

5. TIPS

language = 'ko'locale_dirs = ['locale']gettext_compat = False # generate .pot for each .rst

doc/conf.py

Translatable sphinx html templeteSphinx HTML template is Jinja2.You can use “{% trans %}” template tag.

“make gettext” exports “trans” contents into sphinx.pot file files.

63

5. TIPS

 {%trans%}What users say:{%endtrans%} {%trans%}“Cheers for a great tool that actually makes programmers wantto write documentation!“{%endtrans%}

_templates/index.html

Example projects

Sphinx-1.4 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc15/admin

Docutils doc for "ja" translationhttps://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin

64

5. Tips

https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin






Questions?@shimizukawa

Grab me after the session :)

I’ll be in SPRINT venue too!

65

Thanks :)

66

Easy contributable internationalization process with Sphinx @ PyCon APAC 2016

Technology

Transcript of Easy contributable internationalization process with Sphinx @ PyCon APAC 2016