I do a huge amount of transferring websites from one place to another and changing domain names and all sorts of stuff. One of the things i watch very carefully during the first few months of transfer is how the crawl is going. So basically every time google grabs a page I track them in a log such as..
66.249.73.215
Sunday, 22 July 2012 07:23 pm
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
www.mydomain.comPage Requested: /real_estate_article.asp?gi=1013&ai=432
Page Sent To:via 301-> mls-2956102-26_lorelei_rd_west_orange_twp_nj_07052_1845
Status Reported: 200 (this line not available yet)
But to take the action I want to in a hurry when a lot of fails are occurring.. I need to dynamically get the status of the page that google is looking at. I know that AWstats gets this, but I need it dynamically at the time of the visit.
Is there anyway with PHP, or even Ajax to get this information?
Edit: I suppose I could have the program try to read the file after google leaves, and assume that if I could read it and get the number of bytes that google could probably read it also.. but this seems very inefficient to say the least.