The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 17, 2019, 06:30:45 PM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: The truth is that the ZODB is faster than your RDBMS.  (Read 5158 times)
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« on: August 10, 2008, 06:29:44 AM »

http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks-revisited
Also the cool part about zodb since you can store entire objects and it is 100% transparent even it is a heck of a lot faster.

for example storing something like this in a regular DB and looking it up would be very costly and a pain in the ass

class Person :
    name

class Car
    passengers -> array of person
Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #1 on: August 10, 2008, 11:28:10 AM »

I've not heard of zodb NOP ... what exactly is it? I just looked at that page and it clearly wants to put itself against postgres, but it's an object storage mechanism?
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« Reply #2 on: August 10, 2008, 01:36:22 PM »

ZODB is kinda useless for u perk, since it only works with python (as u will see why).
(I guess u could make a php bridge but it would be kinda silly Smiley)
It is part of the ZOPE python web package.
But the zodb is the part of the framework that stores everything, as a result zodb itself is very mature, and very reliable since zope powers many huge sites, and has been in existance for over 10 years.
Then someone got the smart idea, seperate zodb from zope so it can be used as a standalone product.
Also it can operate in 2 modes, an imbedded mode, and a remote mode.

Here is an ibm article on zodb http://www.ibm.com/developerworks/aix/library/au-zodb/
The part that the article misses is that if your data contains huge lists/arrays of dictionarys/hashes you should use the zodb provided btrees which on the surface are transparently the same as thier python counterparts but if you have a list/dictionary of like a million entries then the entire list of objects does not need to be loaded into computer memory before u access 1 entry.

Basically in a nutshell all it does is pickle a python object and store it on disk.
But a python object in this case is anything, from a hash/dictionary to an actual object itself.
But the actual "storage" mechanism is written in C for speed, also the part that ensures only one instance on

How it works (and why this method will work with python,perl and javascript but with PHP i have no idea Cheesy ).

In perl for thier class system, any data object can be blessed into a "class", be default most perl programmers bless hashes into classes, but there is no reason why u could not bless an array or any other data type into a class (if u can figure out how to do it, array is difficult but possible) Smiley.

In python and javascript it is a little know fact that all data for an object is stored in a hash.
Object in javascript is actually a hash.
In python there is a protected variable for each object know as __dict__
You can control access to it with __getattribute__ and __setattr__
So when u change an object u can mark it as being "dirty".

Also coupled with this you can add variables to objects at runtime.
And also u can take a object and with tricks throw it into another class definition.

Anyway maybe possible to make same thing for PHP.

zodb is not good solution for all problem, if ur data is "flat" then regular DB way to go.
But if it contains tree like structures, then zodb removes the complexity and will be a heck of a lot quicker






Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #3 on: August 10, 2008, 01:48:15 PM »

Nice overview NOP, thanks a bunch.

In theory, I don't see why it would be that difficult to port to PHP, however I am strained to see a reason why.

In PHP you can serialize and store an object pretty easily (the only thing missing is references to other living objects) so although not as elegant, a reasonable workaround. Serializing hierarchical arrays is just as simple, so when I need to store complicated data I'll just serialize the array. HOWEVER - the data is not really searchable unless you added a FULLTEXT index to the field, and then it would not be able to do things like objects where node=(x) and then child node(y)=(z) ... however you could get somewhat close to that with a REGEXP SQL declaration.

Thanks again,
/p
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« Reply #4 on: August 10, 2008, 05:03:59 PM »

If u are doing ur "normal" applications that as a rule of thumb can be stored in an excel spreadsheet, then traditional db is a way to go.
Biggest problem with an ODBMS is the lookup time of objects, but then again there is an addon, where u can catalog certain key variables, but then u kinda tie u kinda fall into the same problems u have with a traditional DB.

Where ODBMS shine.
Things like triggers etc, in traditional DB have no purpose.
Problem store an dom layout of an html page in an traditional database.
Also it depends greatly on how the language handles classes, with fixed class systems like delphi/C++ etc, you would have to make a base class and inherit a zillion elements off it.
Also if u decide to change the design of ur DB u are fuked, since u have to totally redesign the DB layout etc.
Because of the way python objects are setup, adding an extra field, method etc, is as simple as redeclaring the class.
And then reloading the class data into the new hash. If PHP is not designed like that, (as in objects/class are like thier delphi/C++ it is not possible).

A very simple problem where the wheels fall of a traditional DB.
Markov chain for generation of text (a tree like structure)

u have something like
class Person:
     some data stored about person

class Student(Person):
     xxxxx

class HouseWife(Person):
     xxxxx

You then have a database population which stores all different type of people.
With a traditional DB u would have to try and figure out all the fields of the DB before hand before hand.
Here u can regardless store the person whether they are housewife,student etc serially.
If u decide to come up with new type of person like Doctor, no problem u just shove the object in.
Regular DB u are fuked, as u add more different types of people u have to create/redesign more different tables etc. Also tables do not inherit properly.
If u change Person, u can have a version field, if version 1, lazily upgrade it to version 2 etc Smiley.
So lets say u have a new person doctor.
Here is where u have another database containing schools.

class Doctor(Person)
    medicalSchoolsAttended[] <-- will contain reference to school object this will contain references to the school object.

You could also have a School with a list of graduates, which point to doctors.
No need to worry about circular references. It will sort itself out Wink
If u make a change to any object it is propagated back to the database.

Also regardless of what traditional DB u use, u have to shove the results into some sort of local variable, object etc, this take processor time, adds complexity.
No need because the object in the DB and in memory is the same. (BTrees work on lazy loading, as in object is only loaded when needed, when reference to object is lost it is unloaded).

So even if with massive ammounts of preplanning u are able to create a traditional DB, that can hold different types of people,schools.

Lets say that dipshit who is having u design ur database, wants u to keep track of the vehicals that the person owns.
Back to the drawing board u go. U have to redesign ur entire DB.

ODBMS simple
class Vehicle:
    owner -> field that points to owner

Just add to the person class 1 field vehicles [] (has to be array since 1 person can own many vehicle).
5 minutes work.

if dipshit wants to change vehicles to different types, as in small cars,medium cars, trucks buses what ever.
Just make the appropriate classes and away u go Wink




     




Logged
svakanda
Expert
****
Offline Offline

Posts: 131



View Profile
« Reply #5 on: December 11, 2008, 01:21:47 AM »

this is very interesting nop...i think i am going to go setup some python now.
Logged

a ship is safe in the harbor, but that's not what it's for.
cd1
Rookie
**
Offline Offline

Posts: 16


View Profile
« Reply #6 on: January 16, 2009, 03:07:41 PM »

Hey nop-

I'm looking at doing a multi-user knowledge base, and was thinking about using ZODB as the backend.  I haven't been able to find any good tutorials on ZODB that involve more than "Create an object, put it in,pull it back out.".  So the problem I'm running into is that I need to store structured data, but I don't know all of the types of data I'll be storing at the beginning of the project, so it would be hard to use a SQL database. 

But if ZODB is just a big "object dump", how would I go about applying a security model to data stored there, or define relations between pieces of data?

The other way I'm thinking about doing the project is using a SQL database with two types of tables, a data table that holds the object info in some kind of serialized format (yaml, json, etc.), and a type definition table that stores the required format for different object types. I'd then have an RPC server that the client would talk with to store/retrieve data, and the RPC server would apply the type definition against the submitted serialized data to make sure that it was valid for that type before storing it.

I'm not sure if my rambling made sense... but perhaps you get the idea.

Any other ideas that might work?
Logged

No links in signatures please
arms
Expert
****
Offline Offline

Posts: 235



View Profile
« Reply #7 on: January 26, 2009, 09:08:14 PM »

never heard of this. this is very interesting, thanks.
Logged
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!