Here is some code to scrape... Now maybe somebody can help me out and figure out how to scrape aspx, asp, and others which this script doesn't want to scrape
I can scrape with a webcontrol without issue, however, I want this to run as a service - so that really isn't a good way to do it.
Public Function ScrapeURL(ByVal MyURL As String) As String
Dim ReturnScrape As String = ""
Dim myUri As New Uri(MyURL)
Dim MyRequest As HttpWebRequest = DirectCast(WebRequest.Create(myUri.AbsoluteUri), HttpWebRequest)
MyRequest.AllowAutoRedirect = True
MyRequest.MaximumAutomaticRedirections = 10
MyRequest.UserAgent = "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
MyRequest.KeepAlive = True
MyRequest.Timeout = 30000
Dim MyResponse As HttpWebResponse = Nothing
Try
MyResponse = DirectCast(MyRequest.GetResponse, HttpWebResponse)
Catch exception1 As WebException
Return ReturnScrape
End Try
Dim MyReader As StreamReader = Nothing
Try
Dim MyEncoding As New UTF8Encoding
MyReader = New StreamReader(MyResponse.GetResponseStream, MyEncoding)
ReturnScrape = MyReader.ReadToEnd
Catch exception As Exception
Console.WriteLine(exception.Message)
End Try
MyReader.Close()
MyResponse.Close()
'Might want to use Regex to Strip out the HTML here
return (ReturnScrape)
End Function
Well, I hope someone can help me figure out how to scrape ALL web pages instead of just some of them. I did find a web control over at example-code.com, the Chilkat spider... However, I would have to do A LOT of processing of the returned text with that thing.
I'd prefer something that I can bring in (just like a web browser would) and then just READ the text off the resultant page. Not sure how easy that would be.
Any help?
