[1.5.1.3] New 1.5.x search design causes new problems

Back in 1.4.x, the search was limited and only matched exact order of words.
if you typed: "Big Red Truck"
it would not find: "Big Truck"

Now in 1.5.x, the search keywords are exploded and searched individually
if you typed: "Big Red Truck"
it would find all products with: "Big"
it would find all products with: "Red"
it would find all products with: "Truck"

Which is extremely dangerous as you can get results for "Big Car" and "Red Hat" and "Red Apples"... all non-truck related.

So user "loveoscommerce" skyped me with this search problem he was having and I confirm the issue.

To demonstrate the issue, I will use my own site, as it comes into play when you have a lot of products with common title words. In this case we will use my "MIGS" extension.

As you can see here:
http://opencartguru.com/migs

The title of the product is:
MIGS Gateway integration for OpenCart

But if you try to search that exact title on my site (1.5.1.3) you will notice something...
It returns almost a hundred results, and the "MIGS" result doesn't come until page 7!

The problem is because of the new way that keywords are exploded and then searched individually, It is searching:
"integration" and "opencart" and "gateway" which all my products have in the title. But the actual "migs" one is not getting priority.

This is a problem. We need to adjust it to make the less commonly found match be the priority or some other way of filtering.

Perhaps if there is an exact match, then only return that.
Or
If there is no exact match, explode the keywords and search individually until a match is found, then stop. This way the first result would have been for "Migs" ..if that had results, then return just those. If there was no results then continue
to the next word "Integration".. if that had results, just return those, otherwise continue. etc

But as you can see from my "Big Red Truck" example, that wouldn't work

So I think giving priority to the least common match is the necessary solution. Or some way of combining the 3 words first before searching them individually.

e.g.

Big Red Truck should be searched as a phrase initially:
Big Red Truck
Big Truck Red
Red Truck Big
Truck Big Red
Truck Red Big
Big Truck
Big Red
Red Truck
Red Big

Then if no results are found, explode them into
Big, Red, and Truck separately, giving priority at this point, to the least commonly found word.

it is not searching for big AND red it's searching for big OR red. All that needs to change is the OR to an AND. Then it would only pull products that contained all the words, but they don't have to be in that particular order.

IMO!

@Qphoria the default search in OC is less to say useless (no offense guys, but that's the reality).

In fact this was the first thing in OC that i had to fix in order to be usable to some point. What i did was to modify some columns in the database to support full text indexes and then changed the search queries to use the MATCH AGAINST syntax and order the results by their relevance coefficient. This gave me a really nice results without much of speed trade.

With so many possibilities that mysql has to offer today.. using some old school AND/OR statements is not the best thing you can do.

Well no doubt that I personally haven't taken the time to research the best way to handle a search.. There is certainly room for improvement there. I just don't think we've given in the TLC it needs to be done right.

Qphoria wrote:I just don't think we've given in the TLC it needs to be done right.

Yer, Search has always been an unloved child in Open Cart.

justinv of hostjars has done a lot of work on it starting with version 1.4.x
I don't know if it's relevant but he has a mod here:
http://www.opencart.com/index.php?route ... order=DESC

I alredy opened topic like this, almost 3 months ago, but without any response

http://forum.opencart.com/viewtopic.php?f=20&t=43594

So, here it goes again, just to strengthen this one


I know it's not a bug, and the search is designed to search like that, but here is the problem:

If there are products, for example, like those:

Apple MacBook Pro
Apple Cinema
Apple Air

and if I do a search for: apple book

the search result will show all three products. So, how can I make it to show only the products that contain the words apple and book so in this case only Apple MacBook Pro will be shown as a search result.

Well yes that is the bug and that is what we are talking about.

Hi,
We cannot also search by product code. Here is example of searching "product 9" at demo site.

http://demo.opencart.com/index.php?rout ... ption=true

Thanks

Xsecrets wrote:it is not searching for big AND red it's searching for big OR red. All that needs to change is the OR to an AND. Then it would only pull products that contained all the words, but they don't have to be in that particular order.

Does anyone know which lines from model/catalog/product.tpl should i change for the search to return all terms?

I migrated my store from osCommerce to OpenCart a few days ago. I've been loving it until today when I discovered that the search feature is no good for customers. My store has over 5,000 products and almost every search result I've done includes upward of 100 items and the most accurate item is nowhere near the top of the list!

I found this thread while looking for a solution to the problem. Can anyone offer at least a temporary solution for changing the search operator to AND instead of OR? I see there's a module that can take care of this, but I don't want to pay $40 for something that crappy osCommerce did perfectly (and for free) out-of-the-box.

Going to see about a mod to change it back to the 1.4.x way for now

That would be great. I'd appreciate it. I'm sure some others here would too.

Is there any progress to report on this?

My opencart store has been live for 10 days, and today somebody filled out the contact form with a frustrated message to say the search results on my site were "irrelevant" and "pathetic". Sadly, I have to agree.

Considering that my site has been using opencart for only 10 days and there's already been a complaint about the search feature, that says a lot!

Search is a tricky beast... I'd like to have a system like ebay and amazon and google where you can do things like
"phrase match"
+must +match
-ignore
etc.
But still be simple enough for normal searches.

There are 2 obvious implementations:
1. Phrase match directly as a mysql LIKE %% expression, which is what 1.4.x had.
Pros:
- More exact matches ("Big Red Truck" matches "Big Red Truck"
Cons:
- Limited to exact matches ("Big Red Trucks" or "Big Truck" would not match "Big Red Truck")

2. Explode the words first and search each one as a mysql OR expression, which is what 1.5.1.x currently has.
Pros:
- Wider search ("Big Red Truck" matches anything with "Big", "Red", or "Truck" in it)
Cons:
- The search is too wide and you get hits on things like "Red Apples", "Big pants", "Truck wheels".. and things that don't relate to the search

Neither of them are good. So we need something that is smart but that takes some research in best search practices. So research continues.

What about AND? Isn't there an AND expression?

osCommerce and osCMax's default search used AND and it worked perfect.

Big Red
Big Truck
Red Truck
... all of those would match "Big Red Truck"

I'm in the auto parts business so if somebody searches "Toyota Headlight", it's only fitting that they get results that have the words "Toyota" AND "Headlight" such as...

Toyota LED Headlight
Toyota Xenon Headlight
Toyota Halogen Headlight

not...
Toyota Brakes
Toyota Wipers
Subaru Tinted Headlight

Is there any way to look at what osC did and just copy them? In addition to using AND, osC also supported complex searches such as "phrase match" and -negative match. The code that my store ran under was from their 2003 v2.2 release, so considering that we're now in 2012, the current OC search feature is way behind the times.

yes maybe AND is the answer.. I've not yet looked but I'll start there. it might be as simple as that.

Ok .. so AND may be a better choice here. As it does seem to improve results, but needed some more tweaks.

On my site I have a product called "Migs Payment Integration"

With 1.4.x the search for "Migs Integration" found 0 results because it was exact phrase matching. No good.
With 1.5.x "OR", the search for "Migs Integration" found 4 pages of results, and the correct item was bottom of page 2. No good.
With 1.5.x "AND", the search for "Migs Integration" found just the one correct result.
So that is definitely improved.

However, it could still be limiting if you don't have all the necessary terms in the product name. But using tags correctly should resolve that.

If I searched "Payment Integration" it would find all products with "Payment" AND "Integration"
But wouldn't find ones like "Offline Payment System". However, if you use tags and have tags like "Payment, Module, Extension" and leave the TAG search as "OR", then that combination should match everything you need.

So to make it clear, the search logic becomes:
1. Find all products that match "Payment" AND "Integration", which finds all products with those 2 words in the name/description.
2. Additionally, find products that have tags "Payment" OR "Integration" in the tag area, which finds all products that have matching tags, picking up the few that didn't have payment integration in the name/description, but partially matched the tags.

I think we need to add mysql fulltext searching to the database for a more proper solution.... But for now, the quick solution appears to be replacing OR with AND, which I think a lot of people have already figured out in other threads, but hey.. i'm late to the party. But I bring other bug fixes as well. I found that the 1.5.1.3 tag search is actually bugged because it explodes the words, but searches the full phrase, causing tags to be completely broken in 1.5.1.3 without this fix. This AND/OR combination also requires a parenthesis change to work properly.

I've attached the default 1.5.1.3 catalog model file with all changes
Drop it into catalog/model/catalog folder and replace the existing product.php file
OR
To manually make the change and the bug fix yourself:
1. EDIT: catalog/model/catalog/product.php
2. FIND: It should match 4 times.

3. Change ONLY the FIRST and THIRD instances to: This leaves the tag search as "OR" but the name/description search as "AND".

4. FIND (TWICE): 5. REPLACE BOTH WITH: 6. TAG BUG FIX, FIND (TWICE): 7. REPLACE BOTH WITH: 8. Delete all files in your system/cache folder via FTP

9. Now try your searches and see if they've improved.

Again, you may need to use the right combination of tag and title. For my example
If I have a product called "Offline Payment System"
and I want the search for "Payment Integration" to match, I would need to use tags like:

Offline, Payment, System

To guarantee matching more common words that may not all be in the name/desc


This seems to be a great improvement on my site. Let me know how it works on your site.

DO NOT USE THIS FILE UNLESS YOU ARE USING v1.5.1.3
FUTURE VERSIONS WILL ALREADY HAVE THIS FIX AND OTHER CHANGES.
OLDER VERSIONS MAY HAVE DIFFERENT CHANGES AND MAY NOT BE COMPATIBLE WITH THIS FILE.

I have tried the fix on my store but it does not seem to work, lots lots of irrelevant items are still shown.
I have also tried searching"Migs Payment Integration" on your store and obviously the fix does not work.
Link here:
http://opencartguru.com/index.php?route ... ntegration

Gosh, searching really needs improvement!

PS: I just made a vQmod for this fix. You guys can give it a shot and offer feedbacks on this fix:) Report back here whether it works or not!

@Qphoria:

In your eyes the search may not be perfect, but OMFG what an improvement. Nice work! My test search results have gone from literally hundreds of products down to 10-15 (some searches even less). It doesn't matter if you say that ALL the possible products aren't found because 10-15 RELEVANT products is far better than several pages of irrelevant products.

Glad you're making sure that this will be standard in future releases.

I'm surprised with this...

comparing the file attached by Qphoria with the default OC 1.5.1.3 file, I found this twice in default file (lines 80-82 and 444-446):
what is $data['filter_tag'] is doing there? where's $word extracted by foreach()?

in the Qphoria version, I found:
which makes a lot more sense with the $word in the middle of it!

Did I miss a previous fix of this file?!