The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 16, 2019, 09:56:20 PM

Login with username, password and session length


Pages: [1] 2
  Print  
Author Topic: Newsletters and handling bulk email  (Read 5981 times)
deregular
Expert
****
Offline Offline

Posts: 172


View Profile
« on: December 03, 2009, 07:46:37 AM »

Am writing a newsletter script for my CMS, and am at the point now of writing the bulk sending function.

Wondering if anyone has handled something like this before, any suggestions?

Im thinking I'll cron, with some type of limiter so the server doesnt bog down, or the script times out, in the case of hundreds of emails being sent out. I will be pulling member data from my pre-written user module.

Just not sure how to handle the stop/start part of it effectively.

Anyone have any ideas?
Logged
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #1 on: December 03, 2009, 08:56:24 AM »

both perk and I do, and we both have our own custom scripts.

Perk does larger volume to consumers, meaning, spread out a little more organically to different domains.
I do smaller overall, but probably way more to a single domain, because we are mailing to boatloads of sales people within a company like oracle for example.

each has its similar, but different, problems.

I need to be VERY perfect about SPF records and send speed to a single domain. Perk probably also is anal about it, just because thats how he is... lol

I limit my sends to a single domain to 10 per 5 minutes, which is probably overly cautious.

I just have a cron running that hits my sender every 5 mins.
Table that stores the TO info.
An array at runtime that stores the domains and counts.
If a domain passes 10 sends, i add it to another array/string that holds all the "send no more" domains for this run.
My SQL statement that gets the next email to send is a "where NOT domain in ($stringofdomains)" kind of dealio.

Thats really it.

My actual methodology is that the email is an HTML file with tokens in it that get replaced on the fly, just like pretty much any other email package. But I also have some specialized needs that most folks would not have which adds some complexity to my management system.
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #2 on: December 03, 2009, 10:03:31 AM »

I'll weigh in here later
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
deregular
Expert
****
Offline Offline

Posts: 172


View Profile
« Reply #3 on: December 03, 2009, 06:17:02 PM »

Sounds pretty complex already nb.

You just got my gears turning though, so thankyou thankyou thankyou..

At the moment, I have Newsletter Groups, Newsletters, and Newsletter Runs.

A "newsletter run", allows you to choose which newsletter will be sent, and I "WAS"
going to handle the sending from this. Grabbing newsletter, and then everybody who
has an entry in the "newsletter groups" table with the id of the group.

But obviously this would be difficult to handle the stop/starts.

What I am thinking now, is perhaps some type of "Finalize" in my admin.

When finalize is hit, it prepares all necessary data for each send, and stores in a
seperate possibly temporary db table, which the cron can then access for tasks
every  5 minutes. Everytime it runs, the emails it sends, are removed from the
temporary table leaving only the ones that need to be sent still.

Sound feasible?
Logged
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #4 on: December 03, 2009, 08:56:18 PM »

Sure that works. Actually sounds pretty similar to my deal.

More details.
I have a folder which is my campaigns folder.
in there, I have folders for each campaign (in your case newsletters).
In each folder I have 1 HTML file of the email and all the images for it.

My database is a big ass table of emails.
another table which is the send que.

From my management console, I can select groups of emails based on criteria specific to my service, but of course you would have some different criteria to segment your emails.
I also select which campaign to send, which is just a drop down, populated by reading the directories mentioned above.
I also set a send time and date which is to dictate when to send after. (php string to time makes this way easy i discovered Wink )
I submit, and all those emails get qued up in the que table.
date, email, campaign, and a serialized array of shit for replacing out tokens in the email, which again is probably more specific to my service, and would not be neccessary for most.

Thats it. Its simple, fast, and really easy to manage.

Plus I now use google analytics to track opens and clicks, and it works really really well.
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
deregular
Expert
****
Offline Offline

Posts: 172


View Profile
« Reply #5 on: December 03, 2009, 09:27:15 PM »

Thanks for the advice mate, much appreciated.

Being a noob with cron, the question begs to be asked.

Running a cron every 5 mins to access a db table to check for a >0 count,
is this a standard way of doing things?

As there may be a long time between newsletter sends, just wondering if
this would dramatically effect load or bandwidth on the hosting end just
running the check that often..?
Logged
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #6 on: December 03, 2009, 09:38:22 PM »

actually the select is on an empty table, the Que. that literally takes a fraction of a millisecond.
I think its the most efficient way. You could do a file check. like have a file, if it exists, then do DB stuff. But i think that is probably less efficient.
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #7 on: December 03, 2009, 10:54:14 PM »

Sorry for being so late guys. A week of Mondays.

Anyhoo - Nuts is right, we both do stuff like that and very similar, although each is custom coded.

I have a eblast system that combines my user's editing of content with a user-configurable list manager. For every blast job there's a record in a table, for every recipient of each job theres a record in another table. Cron fires of my dispatcher once a minute. If there's anything in the recipient table, then it looks to a system vars table for (this application) and decides how many dispatchlets to fire off. Sometimes we do it as slow as 2 per minute, sometimes 50 per minute. The dispatcher grabs a record id, then spawns a dispatchlet with that id. The dispatchlet is responsible for that one email, and then it dies. The dispatchlet combines the primary job template with its referenced email id - dereferencing merge fields, processing macros like date/time things, creating custom PDFs for attaching to the email or generating custom graphics via GD and such. I have as many as a dozen clients on each server, so it's possible that there's a lot of dispatching going on ... cron fires off a dispatcher for each client application - so I stay away from loading the machine to heavy for all clients, or it starts to get behind. And after that it's bad.

I could throw up some screen shots of the manager as well, if you're interested.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
deregular
Expert
****
Offline Offline

Posts: 172


View Profile
« Reply #8 on: December 03, 2009, 11:39:14 PM »

I think Ive got a good rundown of how Im going to do it now.

- Member table
- Newsletter table
- Newsletter group table
- Newsletter Run table

Each run is a table row... date to send, newsletter id

On hitting "Apply" to a newsletter run, Ill do the manual work, ie.. grabbing user email, newsletter id, date to send etc..
This will then store them as rows in a new table.

Nothing more to do on the admin.

Every 5 minutes, cron checks the new table, for entries to send today, and runs x5 - x10 newsletters at a time.
While this is going, after each send it'll remove the row from the table. Leaving only those that have
not been sent in there.

Obviously.. If there are no table rows for today's date then do nothing.

So all ill have to do is setup the runs and cron will take care of the rest.

cheers fellas
Logged
deregular
Expert
****
Offline Offline

Posts: 172


View Profile
« Reply #9 on: December 04, 2009, 01:17:26 AM »

All done guys, thanks heaps for your input, works a charm. 
Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #10 on: December 04, 2009, 10:51:34 AM »

Well done Dereg.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #11 on: December 04, 2009, 11:07:30 AM »

lol@perk wanting to post a photo of his emailer.

Here's mine.




Do I win? 
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #12 on: December 04, 2009, 01:05:50 PM »

LOL it was only if he wanted more gear spinning Nuts. Do you win? I dunno ... let me post a couple in a little while. I chose a different tact entirely.
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
nutballs
Administrator
Lifer
*****
Offline Offline

Posts: 5627


Back in my day we had 9 planets


View Profile
« Reply #13 on: December 04, 2009, 01:25:09 PM »

no dont post. you will win. And as it stands right now I win.
Yours will be prettier im sure.  ROFLMAO
Logged

I could eat a bowl of Alphabet Soup and shit a better argument than that.
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #14 on: December 04, 2009, 01:27:09 PM »

Oh yeah. I reconsidered and found your post to be a tossing down of the gauntlet.

I'm all over you now, monkeyboy...  Devilish
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
Pages: [1] 2
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!