The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 16, 2019, 10:05:20 PM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: A markov in erlang.  (Read 3406 times)
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« on: May 06, 2007, 08:11:52 PM »

Code:
-module(markov).
-export([load/1,make_sentence/1]).

clean_unicode([]) ->
    [];

clean_unicode([H|T]) when H >= 127;H < 32 ->
    clean_unicode(T);

clean_unicode([H|T]) ->
    [H|clean_unicode(T)].

%{key,total,word_list}
load_sentence(MTable,[W1|[]]) ->
    W2 = stop,
    case ets:lookup(MTable,W1) of
[] -> ets:insert(MTable,{W1,1,dict:update_counter(W2,1,dict:new())});
[{_,Total,NextDict}] ->
    ets:insert(MTable,
       {W1,Total+1,
dict:update_counter(W2,1,NextDict)})
    end;

load_sentence(MTable,[W1,W2|Rest]) ->
    case ets:lookup(MTable,W1) of
[] -> ets:insert(MTable,{W1,1,dict:update_counter(W2,1,dict:new())});
[{_,Total,NextDict}] ->
    ets:insert(MTable,
       {W1,Total+1,
dict:update_counter(W2,1,NextDict)})
    end,
    load_sentence(MTable,[W2|Rest]).

load_lines(H,MTable,eof) ->
    file:close(H),
    ok;

load_lines(H,MTable,Sentence) ->
    load_sentence(MTable,[start|string:tokens(clean_unicode(Sentence)," \r\n")]),
    load_lines(H,MTable,io:get_line(H,'')).

load(FileName) ->
    MTable = ets:new(markov,[set]),
    {ok,H} = file:open(FileName,read),
    load_lines(H,MTable,io:get_line(H,'')),
    MTable
    .

walk_next([{Key,Value}|[]],_,_) ->
    Key
    ;

walk_next([{Key,Value}|Rest],Height,Sum) when Height =< Sum ->
     Keymarkov:make_sentence(M).
     ;

walk_next([{Key,Value}|Rest],Height,Sum)->
    %io:format("~s ~w ~n", [Key,Value]),
    walk_next(Rest,Height,Sum+Value)
    .

make_sentence(MTable,Seed) ->
    [{_,Total,NextDict}] = ets:lookup(MTable,Seed),
    Height = random:uniform(Total)-1,
    case walk_next(dict:to_list(NextDict),Height,0) of
stop -> " ";
NextSeed -> NextSeed ++ " " ++ make_sentence(MTable,NextSeed)
    end
    .

make_sentence(MTable) ->
    try
make_sentence(MTable,start)
    catch
error:X -> error
    end.
Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #1 on: May 06, 2007, 08:39:47 PM »

Shittah man... you told me you were still just "looking" at Erlang... that's a helluva view  ROFLMAO Is this essentially serial code just in Erlang as an exercise? I can't read it very well yet, but it doesn't look to be taking advantage of the multithreadedness... ?

Guess I'm, gonna haveta get the Beta of the book you pointed me towards just to read that...

Aside: hows the performance as opposed to straight up Python or PHP?

/p
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nop_90
Global Moderator
Lifer
*****
Offline Offline

Posts: 2203


View Profile
« Reply #2 on: May 06, 2007, 10:01:42 PM »

Yah just an exercise. But way to test something is to push it until it busts  ROFLMAO ROFLMAO ROFLMAO
Salesmen in stores love me Wink

Basically it is an in memory hashtable which points to a hash for each word.
(except this inmemory hashtable is part of the language).
structure looks like this
hashtable[firstword]-> innerhashtable[secondword] = count.
To load a 5MB file of sentences takes over 5 mins.

But once the markov tree is build you can pickle (i am stealing python teminology) and dump it to disk.
That is vey quick 5 secs.
Also to load from disk is very quick.

You can not really compare to php/python since they do not have what erlang calls an ets structure.
http://www.erlang.org/doc/doc-5.5.4/lib/stdlib-1.14.4/doc/html/index.html docs about this ets structure.
This structure from what i read appears to be very fast (that is why i am making markov with it).
I have not done timing on making the sentences.
I am going to make a static site generator and see how that works.
You can not really compare to python/php since they do not have similar structure.

Language is quirky, a wierd cross between prolog,lisp and python Cheesy
Anyway still exploring.

You making me crack the books. The book saids that to make use of multiple CPU u have to use multiple process.
(process in erlang is like a thread i think Huh?).
So Basically once the markov structure is build, it is safe to access from many process at once, so have like 10 process making text, according to the book if u have 10 cpu erlang make it use them properly Huh?.
Have not gotten to that part of book yet about process Smiley.
Logged
KaptainKrayola
Keeper of Pie
Global Moderator
Lifer
*****
Offline Offline

Posts: 994



View Profile WWW
« Reply #3 on: May 07, 2007, 07:39:46 AM »

Shittah man... you told me you were still just "looking" at Erlang... that's a helluva view  ROFLMAO

nop is a high powered mutant of some kind  never even considered for mass production, crazy.
Logged

We can't stop here, this is bat country.
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!