Internationalization: Converting to PHP Gettext

Gettext would be great as a default localisation system. I have a lot of PHP apps that already use PHP gettext for translations and they work perfectly. What would be great is if OpenCart would use gettext so that I could reuse all my existing gettext message stores. Gettext has been around a while and it does a great job as a translation system. I wish all PHP apps, carts, and frameworks would use PHP gettext instead of trying to reinvent the wheel.



In taking a look through the code base, one way to change over to gettext might be to modify the language system to only have a single language directory instead of a language directory for each language. So instead of this: You would have this: And then in each of the files under 'language' instead of bare language strings like this: You would have PHP gettext strings like this: And then we can let gettext resolve the translations. Based on HTTP_ACCEPT_LANGUAGE or any other way. It's easy to configure gettext to do this.

And then you can run xgettext over your OpenCart files and it will extract all of your translatable strings into a messages.po file. Then you put your translation in it and run 'msgfmt' to gen the messages.mo file and this is what PHP looks into for the translations.

And it would greatly reduce the number of files you have to deal with for language. Especially if you're doing many languages.

Just a thought but I think it might work and it would make a much more flexible solution that would work with the whole gettext centralized localisation for all your other PHP scripts using gettext as well.



.

SUCCESS!

I played around with this idea today and was able to successfully convert OpenCart to using PHP gettext() AND I was able to keep the existing system intact.

So now there would just need to be a switch in the config to determine which localisation system you want to use.

How this works is that when using PHP gettext the only opencart language directories that are used are the 'english' directories.

All the strings in the 'english' language files must be converted to gettext-style strings like _('Order Status') but this does not affect the existing system at all when gettext translation is not being used because PHP natively knows to just use them as straight strings.

For testing, I hard-coded the existing language system to use only 'en' no matter what so as to always drive things to the 'english' files. Later this can be done by switching on the type of localisation set in the config file.

So then using 'xgettext' I extracted all the strings from the 'english' language files into locale/XX/LC_MESSAGES/messages.po where XX is the 2-letter language code for each language I want to support. So for now I put 'en' and 'fr'.
Editing the messages.po file, I then translated some words in the french version and then ran the gettext 'msgfmt' tool to generate the messages.mo binary translation file used by PHP.

Last I put a small header into index.php that set up the 'textdomain' and 'bindtextdomain' for PHP gettext.

And viola! OpenCart picked up the gettext translations when I set HTTP_ACCEPT_LANGUAGE to 'fr'.

Daniel, I guess you're the main developer from what I read. Let me know if you're at all interested in this. If not, I'm still happy. I've got OpenCart running with PHP gettext!


.

Let's get this straight. You went through all the language files and changed them to gettext strings. Then you ran xgettext to create a bunch of other language files, which you then edited. Then you ran gettext msgfmt to convert those language files into a binary form (which is unreadable by humans) for use by php gettext.

And when translating from English to French, does it translate "the" to "le" or "la" or, from English to German, does "the" become "der", "die" or "das"? Or was that part of the editing you did?

Wouldn't it just be easier to translate the existing English language files into the language you want to use? After all, how many languages are likely to be used on a website, one or two?

fido-x wrote:... Wouldn't it just be easier to translate the existing English language files into the language you want to use? ...

No, it would not be easier. I already have a lot of translations available in gettext format being used with other PHP apps. What is the sense of having to retranslate all this stuff for every different PHP app/cart/framework on the planet? That is the purpose of centralising your localisations by using PHP gettext.

And you would only need to know one centralised localisation system, PHP gettext, instead of many different home-grown localisation systems every one of which ends up being different than the next.


.

Using PHP gettext I was able to easily add translations to OpenCart.

Attached are a few screenshots for spanish, french, and german translations of OC done using my PHP gettext conversion.

And the entire translation for one language exists in ONE single file making it easy to do the translations!



Here's an example of what the translation files look like. You can see every file and line number where each word or sentence is being used.


.

How does it work with 3rd party mods? They have separate language files

The mods would have to use the same type of conversion which was actually pretty simple and then they would have the same ability to use PHP gettext. If you used them as is, they might work but would be relying on the old style translations. I haven't had a chance to try mods yet. What would be a simple mod that I could test it with?

UPDATE: I looked at some of the mods and if they are just adding a new text into one of the existing language files then all you do is change the new text into gettext style string (eg: _("new text") ). And the same goes for new language files. All they need is to put the gettext style strings into the 'english' version of the new language files.


.

I was wondering about this when I first looked at the files in oc.
Since, I really don't understand the usage of gettext, I do understand the value of it.

It really allows the translators to translate your files without having to wonder why everything has quotes are them.
However, I don't like how it allows you to _("write full sentences with your code, but it does pretty dang good").

I would like to see a general translation file instead of having to translate "New" in 20 different locations.

The cart is relatively multi-lingual but it really needs to go further since there is a huge range of languages even here in the forums which necessitates a fuller localization syntax in the cart.

localization (dates, writing direction, numbers)
translations

SapporoGuy wrote:I really don't understand the usage of gettext, I do understand the value of it.

It really allows the translators to translate your files without having to wonder why everything has quotes are them.
However, I don't like how it allows you to _("write full sentences with your code, but it does pretty dang good").

The point is, you WANT to be translating full sentences so that certain nuances of language are captured and translated properly. And gettext is perfect because it allows you to translate words, sentences, paragraphs, etc. You cannot just make a dictionary of translated single words and expect any kind of understandable translation to come from that. The best translations come from sentences.

SapporoGuy wrote:I would like to see a general translation file instead of having to translate "New" in 20 different locations.

The exact case for why it is much, much easier to use PHP gettext to do all your translations.

.

my big question is you mention locale/en/LC_MESSAGES/messages.po where does the locale folder start? Is this a place where a person on shared hosting could access it? Not to mention I don't think the vast majority of regular users could figure out all the conversion scripts you said you had to run.

I put my locale folder directly under the installation directory. So it is in ...store/locale.

Shared hosting has nothing to do with it. You do all the translations before you upload your site. Then it just becomes a part of a regular ftp site refresh/upload.

The 2 gettext utilities are really simple to use.

And once you've got one gettext site working, then you know exactly what to do for any other gettext site/app/cart/framework.

greno wrote:The steps are:

1. Put gettext strings in your files wherever you want something translated.

2. Run 'xgettext' pointed at your site tree to generate a 'messages.po' text file that contains all the strings from your files.

3. Using your favorite editor, edit this ONE 'messages.po' file and put in translations for all your strings.

4 Run 'msgfmt' on your 'locale' directory to generate the highly efficient binary translation file needed by webserver.

You're done.

Now just upload/ftp/scp/sneaker-net your site up to your host as you normally do.

And if you're looking for 'gettext' for Windows then go here: http://sourceforge.net/projects/gettext/files/


.

you do realize that getting most of the regular users to upload anything via ftp is almost impossible much less trying to walk them through how to get shell access so they can run a script.

Xsecrets wrote:you do realize that getting most of the regular users to upload anything via ftp is almost impossible much less trying to walk them through how to get shell access so they can run a script.

Repeat. They don't run ANYTHING on their webhost. They don't need shell access.

The translating is done locally and after you are satisfied with the translations THEN you upload them all finished to your webhost.

I'm sure users would rather be editing ONE translation file than several dozen different files which is what is in OC right now, making the same entries over, and over, and over, and over, ...


.

You still haven't answered this:

fido-x wrote:And when translating from English to French, does it translate "the" to "le" or "la" or, from English to German, does "the" become "der", "die" or "das"? Or was that part of the editing you did?

How does any automated translation tool know about gender specificity?
Example: the police officer.
Is that a male police officer or a female police officer? Is it le gendarme or la gendarme?

fido-x wrote:You still haven't answered this:

fido-x wrote:And when translating from English to French, does it translate "the" to "le" or "la" or, from English to German, does "the" become "der", "die" or "das"? Or was that part of the editing you did?

How does any automated translation tool know about gender specificity?

I think you are having some confusion about what these translation tools do.

PHP gettext is not an 'automated translation tool' and neither are any of the translation systems found in any webapps. They do not magically create translations. Translation tools are not 'translators'.

PHP gettext just creates an easy to use system whereby a translator (person) can enter translations (manually) into a SINGLE translation file that PHP then uses to perform the appropriate string substitutions.

If you want automated translation they you have to direct your site users to google translate your page which will give you a pseudo and sometimes very strange translation of your page.

And the problem of gender has nothing to do with tools such as PHP gettext. The proper gender assignment is the problem for the translator (person) doing the translations and entering them into the translation file.


.

In one sentence: your suggestion to use phpGetText will help translators, not will translate anything.

While your idea ist not bad (used the same technique with Mambo many years ago), I do not like to have one single file!

OSWorX wrote:I do not like to have one single file!

And why would that be?

You like doing tons of extra work that PHP gettext frees you from needing to do? You actually like having to make the same entry for the exact same word in dozens of files instead of making it ONE time in just ONE file?

Now I get the impression that people are making up stupid reasons.


.

I've been thinking about this for a bit.

phpGetText seems a bit over kill for a cart.
However, 20+ language and corresponding controller files and is just as much over kill too.

Maybe phpGetText isn't the perfect solution however, it is in a way better than what is available at the moment.
If you want to add "Today's sale price:" to the product page you have to edit the language file for each corresponding language file and also declare it in the controller file.
Even if you are not adding anything new, you still have repetitious language declarations through out many files like the word "Home".

However, I do not like 1 huge file either. Which is why I built into oscommerce a db driven system which I proposed to them along with another person. This may not be the way to go but for a translator, it is just plain easy. I have translated too many scripts in the past and completely hate copying and pasting between quotes. This copying and pasting took me sometimes longer than translating did

greno, do you have any ideas on how translations could be done easier?


ugh, new post added while answering.
oh well, most of the above still applies

A cart is a perfect usage for PHP gettext.

Look at how many different strings need to be translated in the cart. It's hundreds!

And having all the translations in ONE file makes the translators job very easy. I know, I've been doing this for years.

About how to make doing translations easier: find good translators that will help you.

You can also get a start by using the google translate box but some of the results are humorous.

You need a skilled translator to do really good translations.

Using PHP gettext you can also post your messages.po file on launchpad's translator service and there you can have a whole lot of public translators that will fill in your translations. You need to double-check them though, but usually they are very good.

UPDATE: In order to help people appreciate the power and flexibility of PHP gettext, I created a small PHP app that demonstrates how it works. It's in this posting on the next page: http://forum.opencart.com/viewtopic.php ... 20#p112654.

In reference to the next comment about loading all variables at one time. That is NOT how PHP gettext works. It works through a compiled binary library that the webserver utilizes to perform translation string lookups as needed. It is highly-efficient and superfast.


.

greno wrote:

OSWorX wrote:I do not like to have one single file!

And why would that be?

You like doing tons of extra work that PHP gettext frees you from needing to do? You actually like having to make the same entry for the exact same word in dozens of files instead of making it ONE time in just ONE file?

Now I get the impression that people are making up stupid reasons.


.

Maybe a 'stupid reason', but that is my experience in work with Mambo for about 6 years!
Fact.

Why not one 'big' file?
Makes no sence to load all and every variable each time I need one (1) single value (for example, while the usual practice is to load between 5 and 15 variables at once).

I would suggest following:
* making one 'master' language file with common phrases/variables (used by all) - is not practiced now
* leave the rest in several files and load those only when needed

Leaving it in several files is also good for 3PD - how should they provide their language files?

btw: every big(ger) project has several language files, or do you think we now at Joomla use one one?