The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 18, 2019, 11:03:41 AM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: Mapping multiple merchants / products relationships  (Read 4456 times)
DangerMouse
Expert
****
Offline Offline

Posts: 244



View Profile
« on: August 15, 2008, 03:41:48 AM »

Hi all,

Hey all, I'm sure most people here have managed a system like this before, so was hoping you could provide me with a little advice.

I'm developing a fairly simple system thats to use / store data from multiple merchant product feeds - where a feed isnt available I'll be 'borrowing' their content with a little scraping. I'm curious about the best way to map the relationships here in terms of objects and the database.

I'll refrain from going into further details as I think im in need of an original perspective (I'll happily witter on more if needed Smiley) - suffice to say obviously each mechant has different data relating to their products and that I plan to deal with around 100 merchants.

Thoughts?

Steve
Logged
jammaster82
Lifer
*****
Offline Offline

Posts: 666


Thats craigs list for ya


View Profile
« Reply #1 on: August 15, 2008, 04:22:50 AM »

Logged

The watched pot, never boils... But if you walk away from it , the soup burns.  What gives?
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #2 on: August 15, 2008, 07:45:37 AM »

Um, Steve - I believe what you're saying, is that you're going to scrape some sites for data.
And then you'd like to store it.
And do we have any thoughts on how you should do that.
Am I right?

Gonna need more than that man.

What is it you don't understand?
What is the nature of the "objects and database" that are unstandard enough that they require a novel kind of storage?

Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #3 on: August 15, 2008, 07:57:11 AM »

is the problem that you have many different product types, and as a result, the relational data is now a bit perplexing because there is little to no overlap between the product categories?

example would be carrying tee-shirts, which small medium large, and then also having pens which would be red, green, blue, and then also having groceries?

Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
DangerMouse
Expert
****
Offline Offline

Posts: 244



View Profile
« Reply #4 on: August 15, 2008, 08:12:01 AM »

Hey all,

Should have gone into more detail first, but didnt want to bias the responses.

You've partially summed up the issue Nuts, but I'm not even ready to delve into that much depth yet.
 
My system is fortunately far more straight forward than a full eCommerce platform, but the gist is that given a Merchant and Product ID I'll be looking up and displaying the full details on that product. The details will either be provided by looking up the correct product in a database, or by hitting up the merchant website directly (scraping). Imagine the process needed to create a shopping comparision engine out of affiliate feeds and your not far away.

Once this part of the process is complete I'll end up dealing with the issue of "product types" mentioned by Nuts.

Evidently each merchant will have its own method of defining the properties of a product; what I'm wondering is the best strategy for dealing with this. Multiple database tables per merchant? Or a single "Products" table and undertake a full mapping exercise when I import the data - could lead to duplication.

Back to objects rather than databases - there will need to be a method of getting data wehere the data is not stored within the database i.e. a method of constructing a URL and hitting the web for the data.

The problem is coming up with a way of avoiding having a large number of database tables, the same number of classes as merchants (for the finder method), the same number of classes as merchants to reflect the products. - That would be alot of classes!

DM
Logged
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #5 on: August 15, 2008, 11:27:22 AM »

ah ok so the problem is one company lists teeshirts as small medium large, another lists as S M L.

Your only real choice is to have a translation table for those features, if the naming is important for sending the order to the supplier.

so basically you have a table where you define the source value, and you define a display value. That way:
S = Small
M = Medium
Petit = Small
Grande = Large

etc.
There is no real way to automate it, but it will become less and less that you will have to define as you move through products. eventually most product features will have been defined at some point in the past.
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
deregular
Expert
****
Offline Offline

Posts: 172


View Profile
« Reply #6 on: August 15, 2008, 09:30:09 PM »

Back to objects rather than databases - there will need to be a method of getting data wehere the data is not stored within the database i.e. a method of constructing a URL and hitting the web for the data.

The problem is coming up with a way of avoiding having a large number of database tables, the same number of classes as merchants (for the finder method), the same number of classes as merchants to reflect the products. - That would be alot of classes!

Forgive my deficiency to efficiency, as Im still not quite up to par with OO stuff as I could be.
But couldnt you simply use one class. Im presuming that somewhere/somehow you already have some type of unique identifier for each category/merchant, you could have a table row for each merchant (im referring to the scraping part here)..
Re..
ID            MerchantIdentifier             ProductTitleRegex               ProductDescRegex        ProductPriceRegex       
1             shop1
2             shop2
3             shop3 

That sort of thing, then in your class/function loop through the the table (obviously using an extra scraping class/function) scraping each merchant's category for returned matches.
If nothing then show nothing, if something then show something?

To save looping through the table you could always grab it all and throw it into an array as well, to loop through it that way.

Not sure if this helps, im probably way off par here.
Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #7 on: August 16, 2008, 10:05:02 AM »

Your question is a classic, and there's no real 100% answer - every one of them will be a compromise in one direction or another.

If you create a DB that is all about merchant (x) then you'll be able to have everything custom and perfect for that merchant. You'll also have lots of duplication and real trouble when trying to rollup the data into high-level reports.

If you create a DB for all merchants, but different tables for each merchant, you'll have less trouble rolling up, but the logic you'll have to write will be horrible and ugly.

If you create a single DB for everything and all merchant differences are kept in a single table (like a hashed array) then you'll have the slimmest interface and easiest logic, but you may get trouble interpreting differences between vendors and, as NBs says, this part will be all manual at first. In this example, I mean that your "ID" row would be *your* name for an item - then additional columns would be each merchant's version of (that size indicator) - so that when you look up something vertically in that table (show me vendor
  • 's "pico"), you'll instantly be able to go sidewise to translate into your code, or another vendors - "small" or "petit".

Another option is building a larger relational model which can handle a lot more difference yet be pretty normalized. I used this last method for a travel application - I have several vendors, each one calling resort(x) by a different code and each room being called a different code and each one having a different rate, different add ons, etc - so I had to build a pretty wild RDB schema to handle it all, but it does really well. The problem here, is that you have to be a database monster to get that done - so the concession is sleep, eating and relationships while you learn Wink
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
DangerMouse
Expert
****
Offline Offline

Posts: 244



View Profile
« Reply #8 on: August 19, 2008, 03:17:51 PM »

Your question is a classic, and there's no real 100% answer - every one of them will be a compromise in one direction or another.

I suspected as much! What a shame lol. D'oh!

The problem here, is that you have to be a database monster to get that done - so the concession is sleep, eating and relationships while you learn Wink

I think I face this learning curve whichever method I go with lol - I'm glad I only do this stuff for fun rather than for a living  Smiley

But couldnt you simply use one class. Im presuming that somewhere/somehow you already have some type of unique identifier for each category/merchant, you could have a table row for each merchant (im referring to the scraping part here)..
Re..
ID            MerchantIdentifier             ProductTitleRegex               ProductDescRegex        ProductPriceRegex       
1             shop1
2             shop2
3             shop3 

Thats an interesting idea - I never thought of storing my xpath or regex strings in the database - food for thought there thanks.

I had planned to extend a base abstract product object, including the correct way for "finding" a product in each - but the more I think about it the more I feel finding logic should be abstracted away from a "merchant" if possible - it doesnt seem like it's the "Merchant" objects responsibility to get products by <criteria>.

Thanks for the tips, interesting stuff.

DM
Logged
dink
Expert
****
Offline Offline

Posts: 349


View Profile
« Reply #9 on: August 20, 2008, 06:20:03 PM »

I had a similar situation a while back, DM.  My solution was a little draconian, but it still works for me.

I just removed any mention of small, s, pink, green/gold, large, and so on.  I put a small blurb in the
content description that there are many different styles, colors, and options available.  To be sure
that we have what you want, click here.

The click goes to the merchant landing page, where (hopefully) the idjuts have what the shopper needs.
Logged

[quote Nutballs]
the universe has a giant fist, and its got enough whoop ass for everyone.
[/quote]
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #10 on: August 20, 2008, 10:19:16 PM »

Perk's Instant Translater
===============

Pimply Coder Technospeak: "Draconian Solution"
Clear Eyed Adult: "Money Making Efficiency Adjustment"

You go mang. Too many would fret the elegance of the code while eating Top Ramen  ROFLMAO
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
dink
Expert
****
Offline Offline

Posts: 349


View Profile
« Reply #11 on: August 21, 2008, 12:01:11 PM »

Elegance?  Ha!  You've seen some of my code(?).

Actually, the scripts I hacked together to do the merchant database thingy were put together before I even knew what a class was.  I still thought OOP was a misspelling of oops.   ROFLMAO

ps...I love your new translator.
Logged

[quote Nutballs]
the universe has a giant fist, and its got enough whoop ass for everyone.
[/quote]
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!