I had this idea and actually tried it out first time back in 2003 (or somewhere around there, can't remember anymore). I had HP's PDA that had WIFI and Skype. The problem still is that 1) it doesn't survive when you leave the area of wifi coverage, 2) it doesn't know how to hop wifi connections while talking, 3) quality is very much depended on the connection quality which in city can change from utter crap to quite good.
But if you carry iPod Touch with you anyway and spend 95% of your time in the city because you live there, well, why the fuck not

The biggest problem is that wifi connection hop.