The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 23, 2019, 06:12:04 AM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: Access logging, analytics, visitor tracking  (Read 3971 times)
arms
Expert
****
Offline Offline

Posts: 235



View Profile
« on: January 04, 2008, 09:59:14 AM »

so i need a good tracking system, and i don't want to rely on google analytics or pay.
there are two ways i can think of to do this. log info to the db or parse http access logs.
i want to read and write from the db as little as possible so i'm leaning towards access logs since they are being written anyways.
with lighttpd and apache (i think) i can send data in the response from the app to be logged (ex. session id).
then i would just parse the logs, scheduled and/or manually.
i'm just kind of thinking out loud here. i'd love to hear what others here are doing.
this would be for ppc campaigns so i'm going to want as much info with as little overhead as possible, including click streams, hotspots, etc.

also, how would you deal with getting hotspots (i don't know if thats what you call it), meaning the exact spot on the page the user clicked?
i'm thinking javascript that sets a onclick handler onload, wich adds the mouse's x and y position.

thanks
Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #1 on: January 04, 2008, 11:03:12 AM »

I used to track all activity at the page pull, since my sites are all dynamic and flow through a single script.

But like you said, I wanted to minimize DB activity, so I have gone to a single pixel pull to another domain where my tracking is done. The single pixel is something like this: < !mg src="h++p://mytrackingdomain . com/dotclear.php?ref=..." >; and I load all the important stuff into GET parameters that are collected by a little PHP script before it returns a single pixel clear graphic. This takes the load of tracking completely off my primary domains. Note that I use a little routine to make sure there is a session cookie out at the surfer so that I can track visit activity as well (I look for click track and purchase track, as well as try to ascertain the abandon point). This also allows me to scale my tracking for all of my domains pretty easily. Everything goes into a DB at the moment of tracking and then I use a cron job to create my telemetry graphs for activity. My little fun project ATM is to recreate these charts into a perfectly-sized graph for my iPhone.

You can do a click point monitor with what you are describing - an ajax throw is a good way to go (or my XRPC class in the code repository) - shoot off a quick request before you fulfill what the button does and you're good to go. Note that if you fulfill and move to another page before you track then the track notification will be cancelled immediately, especially by IE.

/p
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
arms
Expert
****
Offline Offline

Posts: 235



View Profile
« Reply #2 on: January 04, 2008, 02:22:39 PM »

thanks perk

one domain for central tracking is perfect. i should have thought of that.

@graphs for the iphone, im thinking im getting important enough to get me a blackberry, so i might do the same. Smiley
Logged
kurdt
Lifer
*****
Offline Offline

Posts: 1153


paha arkkitehti


View Profile
« Reply #3 on: May 23, 2009, 06:54:38 AM »

Kinda old thread but wouldn't that one pixel tracking be a really big footprint? In case of bh sites of course... My solution is a central computer that contacts all clusters and fetches logs once per day. Logs are parsed & compressed Apache logs so doesn't take much bandwidth. Maybe when I make my network larger I'll have to do some sort of central server for central servers to have all data at one place for visual tracking.
Logged

I met god and he had nothing to say to me.
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #4 on: May 23, 2009, 07:42:06 AM »

yes it could be a footprint. Though perk doesnt care about that because he is not committing the acts of evil you are Wink
however there are ways to mitigate that as well.

In the JS tracking I now use I copy the JS file locally, and pass the resulting info through on the back end to my mothership.
If an image beacon, same thing. Have a local file to grab the data, and then on the back end send the data to the mothership.

Another thing i have done, is a double pull.
pull a local JS, that is encoded and evals to pull a remote JS from your mothership.
Repeat that re-pull as many times as your paranoid brain needs, but remember it makes your sites slower of course.
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
kurdt
Lifer
*****
Offline Offline

Posts: 1153


paha arkkitehti


View Profile
« Reply #5 on: May 23, 2009, 08:38:28 AM »

yes it could be a footprint. Though perk doesnt care about that because he is not committing the acts of evil you are Wink
however there are ways to mitigate that as well.

In the JS tracking I now use I copy the JS file locally, and pass the resulting info through on the back end to my mothership.
If an image beacon, same thing. Have a local file to grab the data, and then on the back end send the data to the mothership.

Another thing i have done, is a double pull.
pull a local JS, that is encoded and evals to pull a remote JS from your mothership.
Repeat that re-pull as many times as your paranoid brain needs, but remember it makes your sites slower of course.
Yeah, your solutions makes sense. I'll probably JS way but instead of calling another JS, I will send the information with PHP in the background. That way I can get all the benefits from JS tracking and I don't have to worry about footprints. Btw, off topic, what's the best way to encrypt JS? Google can pretty much read JS already so it has to be encrypted with random keys to avoid footprint.
Logged

I met god and he had nothing to say to me.
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #6 on: May 23, 2009, 11:51:42 AM »

yes it could be a footprint. Though perk doesnt care about that because he is not committing the acts of evil you are Wink
however there are ways to mitigate that as well.

True anymore, however the call-out-call-home-with-stats mechanism was my method of choice for the "open source press release article cooperative (  ROFLMAO ) project of mine many years ago. REALLY had to be abstract and no footprints, so Kurdt's description of stat collection is just fine.

Like the way you are doing it now too NBs.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #7 on: May 23, 2009, 01:12:13 PM »

kurdt, yea google can parse, but, i dont think it follows immediately.
so.... here is my pattern for making sure the surfer is a human with a real browser, which you can adapt to suit...

the code is here:  http://www.perkiset.org/forum/php_discussions_help/js_beacon_redirect-t1204.0.html

I set a cookie which has a random codenumber and an expiration time from PHP, and have in page JS that has the same codenumber.
The JS calls a checker page (in my case I use the same exact file, just have a director function that checks the GET strings.
The checks tests the code in the GET against the code in the Cookie (which was sent back again in the headers, because the JS pull is a seperate pull).
if match, its a human, if not, its either a human on a SLOWER THAN HELL connection or its a bot that is not parsing the chain of events in real time.

if you want to get even fancier, put the codes in a DB with expiration times, so they cannot save a "snapshot" and revisit at any future time and be valid.

same principle can be applied to tracker code though, double hit, and only send back the tracking JS if it is a human.

Or your human page in a layer over the spammy stuff.
or your redirect to penis pills.
or or or.

Kind of like a backwards cloaking.
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
vsloathe
vim ftw!
Global Moderator
Lifer
*****
Offline Offline

Posts: 1669



View Profile
« Reply #8 on: May 23, 2009, 01:12:28 PM »

Footprints only exist if you're too lazy to figure out how to nullify them.
Logged

hai
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #9 on: May 23, 2009, 01:27:28 PM »

True'ish

I would agree that most adhere to that rule. But some, like tracking beacons and such, make it a bit harder.

also, ironically, the best way to hid the footprint is to wear the same shoes as everyone else out there and keep with the crowd.

Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #10 on: May 23, 2009, 02:03:15 PM »

I think Nop has hacked VS' account.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!