Preface

More and more sites are implementing dynamic updates of their contents. New items are added as the user scrolls down. Twitter is one of these sites. Twitter only displays a certain number of news items initially, loading additional ones on demand. How can sites with this behavior be scraped?

In the previous article we played with Google Chrome extensions to scrape a forum that depends on Javascript and XMLHttpRequest. Here we use the same technique for retrieving a specific number of news items based on a specific search. A list of additional alternatives is available in the Web Scraping Ajax and Javascript Sites article.

Code

Instructions

1. Download the code from github
2. Load the extension in Google Chrome: settings => extensions => check “developer mode” => load unpacked extension
3. An “eye” icon now appears on the Google Chrome bar
4. Go to the Twitter’s search page https://twitter.com/search-home and enter your search keywords
5. Now press the “eye” and then the start button
6. The scraping output is displayed on the console as JSON

Customization

1. To modify the number of news items to be scraped open the file inject.js and change the scrollBottom(100); line by the number of items you would like (e.g: scrollBottom(200);)

Acknowledgments

This source code was written by Matias Palomera from Nektra Advanced Computing.

If you like this article, you might also be interested in

• Running Your Own Anonymous Rotating Proxies
• Scraping for Semi-automatic Market Research
• Automated Discovery of Blog Feeds and Twitter, Facebook, LinkedIn Accounts
• Distributed Scraping With Multiple Tor Circuits
• SQL Server Interception and SQL Injection Attack Prevention

Further Reading

• Some Fresh Twitter Stats (as of July 2012, Dataset Included)
• Reverse Engineering and The Cloud
• Hooking DirectX Applications to Capture Video and Calculate Frames Per Second