Solution Overview

You need a pair of scrapers. One of them uses search functionality. It grabs search results, URLs and titles, and saves them to database. Another scraper takes URLs (and titles) from the database and scraps articles.

Why two scrapers instead of one? Because running multiple instances of both scrapers, i.e. scraping different websites in parallel, reduces the risk of getting caught, but, at the same time, allows you to pull out many articles. Directories are happy, because you’re whoring them politely and you’re happy, because you get many articles fast.

Two scrapers per hundreds of directories might be a challenging coding task. I recommend you to write bots so they can get websites related data from the database. Don’t use files and command line arguments too much, because it’s a pain in the ass. Create the design constraint – ‘100% code reuse’ and never violate it. You will reuse this bitch many times over and over again without changing single line of code. It pays off, because when you want to scrap another directory, you simply add one database record and run the scripts. Got it? OK, it’s time to get your hands dirty.

Searching Directories

Don’t type just “article website” or “article directory”. Target a group of websites or platforms, because you need a single solution for multiple websites.

• “Total Articles:” AND “Total Authors:”
• There are * “published articles and” * “registered authors in our article directory.”
• “Powered by WordPress • Using Article Directory plugin”
• “Powered By: Article Friendly”

Select two or three websites. It’s enough for you to start with. Later you will screw them all.

Creating Database

The structure is pretty self-evident, but some fields will be explained later.

articles_websites table:

1. Website Id – auto increment primary key;
2. Website name – domain is OK;
3. Search URL;
4. Next Page regex;
5. Titles URLs regex;
6. Article regex.

search_results table:

1. Article URL – unique index;
2. Article Title;
3. Website name or website Id – foreign key articles_websites.[Website Id].

articles table:

1. Article Id – auto increment primary key;
2. Article Title;
3. Article text;
4. Website name or website Id – foreign key articles_websites.[Website Id].

Whoring Search

Take each selected directory and fill in the articles_websites table:

1. Find out how search URL looks like. Should be something like http://www.website.com/search=keywords. Save it to articles_websites.[Search URL] without ‘keywords’ or replace ‘keywords’ with the %s (depends on the way you build URL).
2. Write regular expression, which extracts “Next page” URL from the search results page. Usually search results are paginated and there is the button Next/NEXT/>/etc. Play with search and keywords if you don’t see the button. Save regex to articles_websites.[Next Page regex].
3. Write regular expression, which extracts all Title+URL pairs (search results) from the search results page. Usually this regex is trickier than previous, but it’s 100% feasible. Save it to articles_websites.[Titles URLs regex].
4. Open any article and write regular expression, which extracts article content. Save it to articles_websites.[Article regex].

Steps 1-4 is everything I need to do to kill another victim. Guess, how much directories you can add in one hour? “Suck boobs and let scrapers suck content” principle in action

Coding Search Results Scraper

Input: website name and keyword(s).

1. Load website’s data from database.
2. For each keyword(s) build URL replacing %s and scrap the search results page.
3. Extract Next Page URL.
4. Extract all Titles+URLs; save them to the search_results table.
5. Timeout.
6. While Next Page URL not empty repeat 2-6.

Output: Titles+URLs saved in the search_results table.

Coding Articles Scraper

Input: website name.

1. Get Titles+URLs from the search_results table where website name is [website name].
2. For each Title+URL scrap URL.
3. If success, then extract article (Article regex), save Title+Article+Website Name (Or Id) to the articles table. Delete record URL+Title from the search_results table.
4. Timeout.

Output: Articles saved in the articles table.

Implementation Tips

1. Rotate/select random proxies while scraping.
2. Always timeout and timeout wisely. The point is to run several instances of the scrapers and scrap different websites in parallel, but not to scrap one website every second.
3. Strip of scripts and tags from article before saving/posting.
4. Find the optimal scraping rate and schedule scrapping.

Happy coding!

Get Google SERP and preg match local business results with

$page = file_get_contents('http://www.google.com/search?q=new+york+pizza');
 
preg_match_all(
    '@<h4[^>]*class=r[^>]*>\s*<a\s*href="(.*)".*class=l.*title="(.*)".*>@siU',
    $page,
    $matches);
 
print_r($matches[1]); // urls
print_r($matches[2]); // titles

Then preg match regular listings (I hope regex works) and go through both $matches, local and regular.

HTH

Scrapers are common. Grab a website or page, extract data, save it. See logical parts? Good. You got the idea.

Building SERP proscraper

Pro: How many search engines do you know?
Noob: About 20.
Pro: How many of them do you scrap?
Noob: Five. Google, Yahoo, Bing, Ask and Cuil.
Pro: And how many scrapers do you have?
Noob: Five. [It's pain in the ass to build them.]
Pro: Five?! I have only one server side scraper for all engines. And I can easily extend the scraper in no time.
Noob: Teach me! Do you have a ebook? [I'll be your pet. I'll suck your balls… Why not?]
Pro: [F*ck off!] Learn design patterns and quit reading gay forums.
Noob: I’ll do what you say. I want to be a blackhat.
Pro: [Hate noobs…] Read carefully. You’ll be a SERP scraper hero soon.

A closer look at SERP scrapers

Haven’t open the Noob’s links yet? Do it now. Can you identify the common parts and the parts that vary? The getPage part is common, but the extraction parts vary. E.g. it’s one-to-one relationship between search engine and parsing-extraction part; and one-to-many relationship between the getPage and search engines. What does it mean? It means – separate the getPage part and the extraction part.

Coding SERP Parser Factory

Factory is the best choice for coding SERP parsers by all means. You are passing the type of search engine to the factory and based on that type the SERP parser factory creates the concrete SERP parser. When you need new search engine scraper, you create concrete SERP parser class and add two lines to the SERP parser factory. Checkmate!

Here are some code snippets for you. Now you have a head start on noobs

iSerpParser interface (i_serp_parser.php)

interface iSerpParser {
    public function parse($serp);
}

Google SERP parser – example of concrete SERP parser (google_serp_parser.php)

include_once('i_serp_parser.php');
 
class GoogleSerpParser implements iSerpParser {
 
    public function parse($serp) {
       // necessary code …
       return $serpData;
    }
 
    // other methods …
}

Yahoo SERP parser – example of concrete SERP parser (yahoo_serp_parser.php)

include_once('i_serp_parser.php');
 
class YahooSerpParser implements iSerpParser {
 
    public function parse($serp) {
       // necessary code …
       return $serpData;
    }
 
    // other methods …
}

And finally, the SerpParserFactory (serp_parser_factory.php)

include_once('google_serp_parser.php');
include_once('yahoo_serp_parser.php');
// include concrete SERP parsers …
 
class SerpParserFactory {
    public static function createSerpParser($searchEngineName) {
        $serpParser = NULL;
 
        if ($searchEngineName == 'google') {
            $serpParser = new GoogleSerpParser();
        } elseif ($searchEngineName == 'yahoo') {
            $serpParser = new YahooSerpParser();
        }
        // create concrete SERP parser and assign to $serpParser …
        return $serpParser;
    }
}

The SERP proscraper coding is almost done. Find out which engine, get page, parse and save (serp_scraper.php)

include_once('serp_parser_factory.php');
 
// get engine name from the command line, database …
$seName = getSearchEngine();
 
// getPage…
$page = getPage(…);
 
// create SERP parser
$serpParser = SerpParserFactory::createSerpParser($seName);
 
// parse SERP
$data = $serpParser->parse($page);
 
// save data …

Aftercoding chat

Pro: Now you’re prepared for SERP scraping battle.
Noob: I can add as many engines as I want. F*king easy! I have to code only SERP parsing class. Thanks Pro! Could you share more blackhat tips? [Teach me and I …]
Pro: Dude, it’s just the beginning. You can scrap thousands articles if you switch your stupid WH/BH mindset to coding mindset. But don’t go out to sea before you’ve learned how to swim. Study coding and design patterns: Five common PHP design patterns, Recommended PHP reading list, Head First Design Patterns (aff).
Noob: Thanks man!

I’m looking back at myself, at that idiot, who thought that making cash online is easy. Huge plans… :} Now I see where I’m and the way I have to go to achieve my goal. It takes longer than I thought. But we all start from Zero. And there is nothing wrong with it.

Cheers!

Rankings (all keywords)

“How does it affect rankings? Can it damage my website?”

Less talk more action.

• One whitehat microsite born in Aug 04
• Very small niche – 2K-40K pages (depends on keyword)
• 9 keyword groups – long-tails only
• 20 backlinks total – pure virgin
• 12 long-tails in top 10 already
• 5-6 visits per day

Rankings

Keyword group 1

Keyword group 2

Keyword group 3

Keyword group 4

Keyword group 5

Keyword group 6

Keyword group 7

Keyword group 8

Keyword group 9

I’m going to throw one dofollow link per day starting from now. I won’t use my websites. Only dofollow blogs.

See updates next Tuesday.

OK, straight to the point, why should you join Luck3 affiliate program – 3Aff?

Commissions

Luck3 guarantees you profit from each player referred by you on a lifetime value of the player as long as you remain network’s partner. Commission is calculated as a percentage from the net revenue from referred players. The initial rate is 25%, but can be extended up to 40% (depends on which Luck3 product, Poker or Casino, you advertise and which resources you use).

No negative carryover

It means that your monthly negative amount will be reseted and you can start the next month without any minuses on your account. The positive amounts are calculated month by month.

Payments

Unlimited earnings. Commissions over $100 paid on a monthly basis at the beginning of month.

Who can play / targeted traffic?

Luck3 Online Casino and Poker accepts players (at least 18 y.o.) from all around the World including US and Canada.

Consider this as exclusive “money making tip” – target cities / areas / countries. Look for small competition. Localization is a key.

Who can join?

Luck3 is open for everyone, pros and amateurs from different countries.

How to get approved?

Pros can skip this part, others read carefully.

Getting approval is not a problem at all.

1. Be able to communicate – be ready to answer questions, including AM related.
2. Be honest with the affiliate network. Never provide fake registration info, etc.
3. Don’t submit spammy websites from completely different niche.
4. Don’t afraid to be rejected only because you reside in India, Indonesia, Philippines, etc. Nobody cares where you live.
5. And last, but not least, use my referral link at the end of the post.

How to promote?

Luck3 offers you banners, texts, articles. Landing pages are OK too.

Questions

Feel free to ask related questions in comments area. Luck3.com affiliate managers will respond. I promise.

Sign up with Luck3.com

When you say never link [splogs] to your affiliate site, do you mean don’t link to your money sites, so these should be linked to buffer sites that are in turn linked to your money sites ?

Right. The proxy layer covers up generated (spam) websites. And there is one reason for this – footprints. High rankings draw everybody’s attention to a website. And it’s better for the website when it looks more natural. But don’t be naive about your “perfect” cover. Competitors (not all) and The Glord know about your websites

1. Ten PR5 links
2. Twenty PR3 links
3. Your choice?

Same price, non-relevant niches.

Forum scraping

1. Buy a bunch of cheap domains – GoDaddy’s $5/domain list or bunch of .info, co.cc. (Hidden or fake registration info).

2. Compile a list of keywords – from ten to twenty is OK.

3. Find 3-5 niche forums and scrap threads with keywords. Remove links from content.

4. Setup wordpress blogs with “Related Blogs” and “All in one SEO pack” plugins. Add scraped content somehow.

5. Build some trackback links with “Related Blogs” plugin. Don’t link to your sites only. Ping, comment spam, etc., etc. Never link to your aff sites.

6. Repeat ∞

P.S.: The best advice of all time – automate

Cheers!