Prepare project to use Zanata for translations

[KurzedMetal]

I realized that with the recent refactors I made, I was changing some translation strings and they needed to be re-translated. Suddenly I realized some translations haven't been updated in a while and the project don't have an easy way to receive translation contributions without fiddling with gettext tools... Let's be real, potential contributors to translatation are pushed off if they need to actually fiddle with tons of tools instead of actually translating, most of them aren't even tech savvy or devs...

I always thought that web tools for translation like Transifex were super cool and wanted to configurate one of these services for CDDA Launcher. I checked Transifex (which is actually used by CDDA itself too), but they require an "approved" registration for open source projects as the service is usually paid. So I googled around and I found Zanata, which seems a similar open source service with less registration hassle.

I already configured everything the project needs to push and get translation from this page.
The workflow would be:

• You code stuff in CDDA Launcher, create or update translatable strings.
• Run setup.py zanata_expush to extract strings and push them to Zanata service
• Translators register on Zanata (free account and painless short steps)
• Translators do their work on a nice web interface with no tool required.
• You register, you approve them their translations (or not)
• Run setup.py zanata_pull --approved-only to get all the strings you approved in the web interface. Or no --approved-only and you get everything anyways.
• At this point, if you run setup.py create_installer it would include the new translations (no need to do anything else).

I already setup a temporary page for the project, so you could see it, register and join to check (cdda-launcher-test in Zanata). But if you want you can register your own and just change the configuration files (just so you can get the ownership of it, since it's your project)

If you wanna create your own, you only need to: create an account, create a project, add a version, update the zanata.xml file and configure your client (take a good look at that link, it's how you setup the client in your PC) and run setup.py zanata_expush. If you want to boostrap your new project and upload existing translations too you need to execute setup.py zanata_push --push-translations too (this is only for an initial setup, because it overwrites translations in the server)

A note worth mentioning: after I had a good progress in the configuration and almost everything working, I noticed some bad news around, Zanata was an open source project sponsored by Red Hat, but it seems they pulled the main devs from the projects a few months ago. The Zanata project seems mature and still hosted, so at least for now it works and it's better than you had before. But if you wanna try other service or maybe try getting a Transifex account for this project, maybe I could help setting it up too.

[remyroy]

That seems like a nice idea. Let me check the current state of Zanata.

[remyroy]

I contacted Transifex and they said that we qualify for an Open Source license. We can use their services for free. What do you think? Should we go with them? They say they have the largest translation community.

[KurzedMetal]

Yes, Transifex is probably better. I doubt the Zanata project is going down any time soon, and the feature set does seem currently stable, but it would be better to be in the safer side.

I'll check some Transifex documentation and see how we can push/pull Translations (if they even have such an API, which I bet they do)

[KurzedMetal]

Can you create an organization / project for CDDA Launcher and invite me?
Seems that configuring project requires those to be already created (tx init with their client).

Also, If you give me "Project Maintainer" Role could probably let me do some tests (I wanna add some new setup.py commands) and push initial data.

From what I read so far, the workflow is similar:

• install the client
• initialize project with: tx init (this creates a ~\.transifexrc with you auth token too)
• push untranslated strings with tx push --no-interactive -s (add -t for translations too, probably useful only in the initial setup)
• pull translated strings with tx pull --no-interactive -a --mode onlyreviewed (or onlytranslated if you wanna accept every translation from contributors without reviews)

[remyroy]

I'll create the organization in the next few minutes.

[remyroy]

What is your Transfinex username? If you don't have one, just create one by using their "Try for free" link I guess. It would probably be a good idea to reuse your github login with their system.

[KurzedMetal]

Same as here: KurzedMetal

[remyroy]

Transifex seems to have some kind of Github integration where Transifex can push and pull directly on Github. I'm trying to activate it. That might help. If not let me know.

[KurzedMetal]

Yes, you can configure Transifex to automatically check changes in a hosted file (like the .pot file in github master branch), and they check it twice a day.
But I was trying to create some setup.py commands so you can push the untranslated strings whenever you want on demand.

Your choice, both will probably work, you can probably even use both I guess.

[remyroy]

I was just trying to setup the GitHub integration but I don't think I succeeded. I see that you started adding some resources. I'll let you explore more of it and come back later.

[KurzedMetal]

Yes, I was testing the tx commands manually and they worked great. It pushed everything already except for 3 plural Russian strings because it seems Transifex expected more strings for them.

I'll use those tx commands to create setup.py commands, those will be handy, and you can use it whenever you want, I'm pretty sure they won't interfere from configuring automatic updates too, if that's what you want.

Anyways, I should have something soon, as it's very similar as what I had with Zanata.

[remyroy]

Good stuff. There are various ways of configuring the GitHub integration. I've set it to create pull requests instead of directly pushing into the repository.

We might also need to tweak the .gitignore file to keep that pot file around.

[KurzedMetal]

I created the new commands for Transifex, so now, the workflow should be like:

• You code stuff in CDDA Launcher, create or update translatable strings.
• Run setup.py translation_expush to extract strings and push them to Transifex service
• Translators navigate to the project Dashboard and click "Help translate CDDA Game Launcher" to register.
• Translators do their work on a nice web interface with no tool required.
• Someone review the translations (or not)
• Run setup.py translation_pull --reviewed-only to get all the reviewed strings in the web interface. Or no --reviewed-only and you get every translated anyways (reviewed or not).
• At this point, if you run setup.py create_installer it would include the new translations (no need to do anything else).

I also left around the code/configuration for Zanata, in case you want to use them in the future, but I could purge all reference of it if you want, your choice.

As for the automated way to push new strings to Transifex, I realized that it would require you to run exup_messages command manually anyways, it wouldn't be much difference from running translation_push. So, unless there's an automated way to extract messages each time you merge a PR or push a commit, I don't see the "automatic push" to be of much use.

[remyroy]

Yeah, that is a good idea. We could only translate that single simple phrase for everyone and keep the canonical license in English.

[KurzedMetal]

Ok, I'm dead serious now :P that was the last push. ;P

EDIT: Oh I noticed you managed to find where to add translators.
Also, I wonder if the translators have to ask permissions to translate stuff (like being added to teams) or they can do it without any permissions.

[remyroy]

There was an option to enable anyone to join the team. I checked it.

[KurzedMetal]

I would prefer to remove the Zanata stuff if we are to use Transifex. If you want to keep the Zanata stuff, just create a seperate branch in your GitHub repo and we'll be able to revisit it later on if we need to.

I totally missed that before.

If you don't squash the PR commits, the Zanata stuff will stay in the history anyways.

[KurzedMetal]

It seems I hit a babel bug when using translation_pull with the change where I added the MIT License summary. I used the text "100% free" and it seems babel is reading that as a string format "%f" (yes even if it has a space between % and f because it's actual valid python syntax) and it is slapping the "python-format" hint to the text, which makes it expect having valid %x formatting (it's actually breaking without that flag too, it seems a behavior from the compile_catalog)

Due to this, I'm getting the error: cddagl\locale\es\LC_MESSAGES\cddagl.po:470: incompatible format for placeholder 1: 'f' and 's' are not compatible.

I'd really hate workaround it by changing the message, I was checking if I could pass some option to babel to avoid this but I couldn't find it so far. IMO we really don't need the %x formatting ever (I think you used that format syntax in a single line in the whole code and we could change it).

EDIT: To summarize, babel is not handling correctly strings with % symbols in it.
EDIT2: I couldn't find a way to control this behavior in the compile_catalog command, it always check these % string formats no matter what, so I workarounded it by using an HTML entity instead of a plain %, since that text is used as HTML. Keep in mind that using % in strings could lead to this issue.

[remyroy]

I'm confused about this part in i18n.py:

def load_gettext_locale(locale_dir, locale, domain='cddagl'):
    """Load gettext into the application with specified locale file.

    Fallback to default untranslated strings if locale file is not found.
    """
    if locale == 'en':
        return load_gettext_no_locale()

Why are we not loading all the locales if en is selected?

Why is 'en' the value for when we cannot find the locales? Would it not be possible for 'en' to be a normal value and we still need to load the locales at that point in load_gettext_locale?

[KurzedMetal]

Until now, the code depended on an empty po to have english translations.

When 'en' was loaded, it was actually provided with an empty file (well, actually not "empty", but with no translations, but an empty file would have the same effect). That made the code use the translations found in the source code instead of the non-existent translations in the empty .po.

With two small changes, I made the code not depend on the empty .po file for english to actually load empty English translations, it uses load_gettext_no_locale() which is exactly the same as giving it an empty file (meaning it directly uses the strings in the source code).

Basically, that change add two lines of code to avoid having an useless empty file. There's no point in having a .po file with no translations.

[remyroy]

English shows twice in the Language settings. It should only show once.

[KurzedMetal]

That issue happened because you still have the empty english .mo file around when I removed it locally. I already deleted the .po file from the repo, but there's nothing I could do for the .mo because it's git-ignored.

Since the .mo is ignored, I can't actually delete it or anything, so I improved the get_available_locales to avoid duplicates. But in reality, the issue is that you have a file that shouldn't be there, and that git clean -X -f would delete.

[remyroy]

Alright, good to know.

[remyroy]

Seems good. Merging.

[chuanyueyouxia]

May I translate it into Chinese？I have already requested on transifex.

[remyroy]

@chuanyueyouxia Sure! Let me check it out on Transifex and setup Chinese there.

[remyroy]

@chuanyueyouxia There are various forms of Chinese in the localization world. I know that you requested zh_CN but would it make sense to do others as well like one of these:

• zh_SG for Singapore
• zh_TW for Taiwan
• zh_HK for Hong Kong
• zh_HANS for (generic) simplified Chinese characters
• zh_HANT for (generic) traditional Chinese characters

I know that there can be some subtle differences from region to region but it might not be worth to do them. For instance, I'm French Canadian. We have this fr_CA code here but there are multiple french variants like fr_FR (French from France), fr-BE (French in Belgium), etc. Since there are not enough differences, I decided to go for the generic fr code. Let me know what you think about the Chinese language for this.

Also, I would need the email address or the username you used to create your profile on Transifex so I can add you to the Chinese translation team as a translator.

Thanks

[remyroy]

I think I was able to add you to the correct team with the proper role on Transifex now so I will not need your username or email.

If you need help with the translation or if there is something you don't understand, let us know. It can be somewhat difficult with the various codes and markups that are used. We'll be happy to help you out @chuanyueyouxia !

[chuanyueyouxia]

@remyroy Thanks!
Oh, I see.There are many forms of Chinese in the localization world.Chinese can be written as simplified Chinese and traditional Chinese.
IN CDDA game, Chinese zh_CN is simplified Chinese, Chinese zh_TW for Taiwan is traditional Chinese.
Maybe we can use general code?

[remyroy]

I think zh_HANS might be the generic simplified one that we are looking for. I'll switch to that one on Transifex and add you there on that team too.

[remyroy]

@chuanyueyouxia The switch to zh_HANS from zh_CN is completed on Transifex. You should be good now to start translating.

Thanks

[chuanyueyouxia]

Thanks a lot, @remyroy . You've been great！