hey all,

i made this little article scraper that basically takes a number of articles and filters out only the body of the article.
now sometimes it returns some really weird characters within the article > –

is there a

perl

  module or function that will filter out only plain text?

thanks

You could have a look at:

HTML::Entities - Encode or decode strings with HTML entities

We used it to get rid of such weird characters.

i was just on my way to edit the post, thats the first thing i used and it removed some but not all of the characters...

i was thinking more of loading any text file and filtering out anything thats not plain text type of thing...

You're probably only interested in

PERL

 , but in

PHP

 , you can also do:

$newStr = htmlspecialchars($inputStr);

... which converts all input weirdness to their HTML encoding values ie., "&" becomes "&" etc. It'll handle all the international stuff as well - but this is all assuming that you're taking an HTML doc and converting it. If you're taking real international input outside of the web world then you'd probably look at htmlentities for

PHP

  just like Dirk has said with

PERL

 .

/p

i will look into that...

thanks,

Instead of eliminating unwanted shit, I only allow alphanurmerics plus a few others like
the underscore, (or whatever).  So, I only allow shit i can see on my keyboard.

Well, I think that's how I do it. 

Bompa

Using a

regex

  you could skip all weird characters and keep only ASCII 0 - 127:

i also do along the lines of what bomps does.

this is an

ASP

  function that does exactly that without using

regex

 . I actually found this to be faster for really long text. i know this is the

PERL

  board, but the concept is the same and doesnt use any functions that wouldnt be available in any language.
have a string of valid characters.
check eat letter in the dirty string against the valid string.
replace the character if its bad.

so for URLs i run it as  stripnonalphanumerics(someURL,"-"
for content i run it as  stripnonalphanumerics(someURL," "

Here's a

PHP

  function to do the same:



/p

thanks for the help guys... this one did the trick    $string =~ s{ ( [^x00-x7E] ) }{}xmsg;

i really suck with

regex

 , i have to practice more!

What does the 'xmsg' do?  Those aren't

regex

  flags are they?

Rather confusing with all the curly braces.  You're making me crosseyed.

I use the 'xms' based on recommendations of

Perl

  Best Practises:

Always use the /x flag (extended formatting).
Always use the /m flag (matching line boudaries).
Always use the /s flag (matching anything).

If you use the brace delimiters {} you don't have to escape the slashes, like http://.

Ah!

Thanks Dirk, being a

PERL

  st00bie, I didn't want to say anything at all... but now I get that those are the

regex

  behavior modifiers. In

PHP

  we do put the modifiers after the closing delimiter ie.,

/^(.*)$/ismg

thanks for clearing that one up, the syntax really had me cross eyed as well

This is the usual syntax which looks more familiar:

dirk, what's with the double curly braces preceeding the xmsg;

{}xmsg;


what do the curlies do?


Bompa

Bompa,

the empty curly braces {} mean that the string in the preceeding curly braces
shall be replaced by nothing. So the special characters will be deleted.

Dirk

ahhh, I finally get it. 

Bompa <--- SLOW

You have an extra curly brace cuz YOU HAVE TO in order to have braces in pairs,
whereas, if we delimit with slashes, we can use jut three.

damn!

Bompa,

here are some more examples:



Normally only three delimiters are required.

But if you use brackets you need 4 (two pairs).