problem with foreign languages

Quote

Post by loadaverage » Wed Dec 02, 2009 3:10 am

hi there,

there is a problem with opencart when dealing with foreign languages.
try entering "ááááááááááá" as a category name for instance.
it will complain about a too long string.

the problem is basically in system/library/request.php:clean()

Code: Select all

36:         $data = htmlentities($data, ENT_QUOTES, 'UTF-8');
this converts all the accented characters into html entities
taking up 4-6 characters and also defeating the utf8 conversion.

the solution would be to replace htmlentities() with htmlspecialchars().

i also posted this as
http://code.google.com/p/opencart/issues/detail?id=111

thanks for looking into this.

we is experts(tm)

loadaverage
New member
Posts
64
Joined
Fri Nov 20, 2009 2:10 am
Top

Re: problem with foreign languages

Quote

Post by loadaverage » Mon Dec 07, 2009 4:57 am

the htmlentities() is causing another problem: all the mail
that is sent out has these entities encoded into base64.
when the email client shows the message, it is showing
the entitites instead of the rendered characters...

it is either necessary to do a html entities decoding everywhere
when text is touched, or (what i prefer) just replace htmlentitites()
with htmlspecialchars().

Daniel, could you please elaborate on your decision to use
html entities? i see na rational reason for it (especially
with non-english shops).

we is experts(tm)

loadaverage
New member
Posts
64
Joined
Fri Nov 20, 2009 2:10 am
Top

Re: problem with foreign languages

Quote

Post by yaxo » Mon Dec 07, 2009 5:19 am

This is a really big problem in Sweden too, thank you for this.
yaxo
Newbie
Posts
13
Joined
Fri Dec 04, 2009 3:41 am
Top

Re: problem with foreign languages

Quote

Post by axxies » Mon Jan 04, 2010 10:16 pm

I agree (I am from Sweden too). It appears that texts shown on the web pages needs to be separated from those sent out in mails.

I am using 1.3.4.
axxies
New member
Posts
22
Joined
Thu Sep 03, 2009 4:30 pm
Top

Re: problem with foreign languages

Quote

Post by Planck » Fri Jan 15, 2010 10:02 pm

Hello,

Same problem with Greek language. Is there a workaround for this?
Planck
Newbie
Posts
11
Joined
Fri Jan 15, 2010 10:01 pm
Top

Re: problem with foreign languages

Quote

Post by i2Paq » Fri Jan 15, 2010 10:52 pm

Which version?

Norman in 't Veldt
Moderator OpenCart Forums
_________________ READ and Search BEFORE POSTING _________________

Our FREE search: Find your answer FAST!.

[How to] BTW + Verzend + betaal setup.

User avatar
i2Paq
Global Moderator
Posts
12244
Joined
Mon Nov 09, 2009 7:00 pm
Location - Winkel - The Netherlands
Top

Re: problem with foreign languages

Quote

Post by Miguelito » Sat Jan 16, 2010 10:02 pm

I'm using the 1.4 version and outgoing mail shows in web-based email just fine but in Outlook all scandinavian letters (ä, å, ö and so on) are translated to something else. I noticed the same problem as the Swedish guys (I'm from Finland) that when you use scandinavian characters for example in product names and press Save - you get a warning that the product name should be max 32 characters. This is because scandinavian characters (ä, ö, å and so on) take in UTF-8 five to six characters...

The Finnish OpenCart Forum
"Real programmers don't document. If it was hard to write, it should be hard to understand."

Miguelito
Active Member
Posts
796
Joined
Sun Jan 10, 2010 10:11 pm
Top

Re: problem with foreign languages

Quote

Post by Daniel » Sun Jan 17, 2010 6:25 am

the solution is to count the characters after they have been converted back.

OpenCart®
Project Owner & Developer.

User avatar
Daniel
Administrator
Posts
5627
Joined
Fri Nov 03, 2006 6:57 pm
Top

Re: problem with foreign languages

Quote

Post by Miguelito » Sun Jan 17, 2010 6:09 pm

So there is no solution to use ISO-8859-15 rather than UTF-8?

The Finnish OpenCart Forum
"Real programmers don't document. If it was hard to write, it should be hard to understand."

Miguelito
Active Member
Posts
796
Joined
Sun Jan 10, 2010 10:11 pm
Top

Re: problem with foreign languages

Quote

Post by OSWorX » Mon Jan 18, 2010 3:59 pm

Daniel wrote:the solution is to count the characters after they have been converted back.
But does this solve the main problem?

Why not storing values as they are?
Now every output MUST be converted back.
Beside this, every limited field (in the DB) is useless (see title max. 32 chars).
And - the DB is full of characters, but useless.
Or will grow and grow (beside the normal), because each value there will be 2-4 times greater than it could be (if not English is used).

Full Stack Web Developer :: Dedicated OpenCart Development & Support DACH Region
Contact for Custom Work / Fast Support.

User avatar
OSWorX
Guru Member
Posts
7541
Joined
Mon Jan 11, 2010 10:52 pm
Location - Austria
Top

Re: problem with foreign languages

Quote

Post by Daniel » Mon Jan 18, 2010 8:14 pm

the problem is if hackers try to insert soem html or javascript. you want to get rid of things like <script>.

Only certain fields require html such as product description fields.

OpenCart®
Project Owner & Developer.

User avatar
Daniel
Administrator
Posts
5627
Joined
Fri Nov 03, 2006 6:57 pm
Top

Re: problem with foreign languages

Quote

Post by OSWorX » Mon Jan 18, 2010 9:24 pm

Daniel wrote:the problem is if hackers try to insert soem html or javascript. you want to get rid of things like <script>.

Only certain fields require html such as product description fields.
But then there are better ways to solve this 'kiddie' problem ...

Full Stack Web Developer :: Dedicated OpenCart Development & Support DACH Region
Contact for Custom Work / Fast Support.

User avatar
OSWorX
Guru Member
Posts
7541
Joined
Mon Jan 11, 2010 10:52 pm
Location - Austria
Top

Re: problem with foreign languages

Quote

Post by loadaverage » Tue Jan 19, 2010 3:49 am

Daniel wrote:the problem is if hackers try to insert soem html or javascript. you want to get rid of things like <script>.
it has been pointed out to you numerous times that

php.net/htmlspecialchars

is enough for this filtering.

we is experts(tm)

loadaverage
New member
Posts
64
Joined
Fri Nov 20, 2009 2:10 am
Top

Re: problem with foreign languages

Quote

Post by Daniel » Wed Jan 20, 2010 5:48 am

i'm not sure htmlspecialchar can stop hacking attempts.

SO you just want me to do ths:

public function clean($data) {
if (is_array($data)) {
foreach ($data as $key => $value) {
unset($data[$key]);

$data[$this->clean($key)] = $this->clean($value);
}
} else {
$data = htmlspecialchars($data, ENT_QUOTES, 'UTF-8');
}

return $data;
}

OpenCart®
Project Owner & Developer.

User avatar
Daniel
Administrator
Posts
5627
Joined
Fri Nov 03, 2006 6:57 pm
Top

Re: problem with foreign languages

Quote

Post by Miguelito » Wed Jan 20, 2010 5:55 am

Daniel, is your post the fix for this problem? I could go to testing mode if it is...

The Finnish OpenCart Forum
"Real programmers don't document. If it was hard to write, it should be hard to understand."

Miguelito
Active Member
Posts
796
Joined
Sun Jan 10, 2010 10:11 pm
Top

Re: problem with foreign languages

Quote

Post by Daniel » Wed Jan 20, 2010 6:35 am

Yes that is the fix. just find the request class and replcae the clean method with this one.

tell me if the problem is still there.

OpenCart®
Project Owner & Developer.

User avatar
Daniel
Administrator
Posts
5627
Joined
Fri Nov 03, 2006 6:57 pm
Top

Re: problem with foreign languages

Quote

Post by Miguelito » Wed Jan 20, 2010 6:52 am

Daniel, tested with a few product categories and worked perfect :D

Hopefully other users also contribute by testing this fix... so it could be implemented in the next release.

The Finnish OpenCart Forum
"Real programmers don't document. If it was hard to write, it should be hard to understand."

Miguelito
Active Member
Posts
796
Joined
Sun Jan 10, 2010 10:11 pm
Top

Re: problem with foreign languages

Quote

Post by OSWorX » Sat Jan 23, 2010 5:45 am

Daniel wrote:Yes that is the fix. just find the request class and replcae the clean method with this one.

tell me if the problem is still there.
THE problem has gone, but the next is here: now if special characters, accents, umlauts are used, depending of these characters you can save a maximum of 20-26 characters per title!
Defenitely too less!

Full Stack Web Developer :: Dedicated OpenCart Development & Support DACH Region
Contact for Custom Work / Fast Support.

User avatar
OSWorX
Guru Member
Posts
7541
Joined
Mon Jan 11, 2010 10:52 pm
Location - Austria
Top

Re: problem with foreign languages

Quote

Post by loadaverage » Tue Jan 26, 2010 4:38 am

Daniel wrote:i'm not sure htmlspecialchar can stop hacking attempts.
then you don't understand what these functions do...

we is experts(tm)

loadaverage
New member
Posts
64
Joined
Fri Nov 20, 2009 2:10 am
Top

Re: problem with foreign languages

Quote

Post by loadaverage » Wed Jan 27, 2010 7:21 pm

Daniel wrote:Yes that is the fix. just find the request class and replcae the clean method with this one.

tell me if the problem is still there.
it is not the complete fix.
htmlentities() is all over the place.
but it's a good start.

we is experts(tm)

loadaverage
New member
Posts
64
Joined
Fri Nov 20, 2009 2:10 am
Top
Post Reply
Return to “Bug Reports”
Who is online

Users browsing this forum: Bing [Bot] and 157 guests