A Little R Toolbox Workshop

04/07/2017

University of Maryland

Research design and data collection phase

1. shiny + rdrop for data collection (or use shiny to illustrate results)
2. Web crawling within R using rvest
3. Matching string based spatial data using geocoding

Data analysis phase (exploratory and post hoc)

1. Simple and more advanced use of R graphics with ggplot2
2. Batch regression tabulation using broom and stargazer

What and Why

A Basic Crawler

Extensions

A classic oursourcing versus in-house production problem.

Don't worry, I have a theoretical lens: Transaction Cost Economics.

Three main arguments:

• Crawling tools today have comfortable learning curve (low setup cost for in-house production).
• Many data problems/requirements are uncovered “on the go” (inflexibility of oursourcing).
• Crawling can be easily done within data analysis software such as R (synergy for in-house production).

What and Why

A Basic Crawler

Extensions

Assumption: All we need is copy-paste in batch mode.

Then, there are three key elements:

• (link) URLs of the web pages that contain needed information.
• (template) A bunch of codes to load a page, locate and collect the infomation, and store in a data set.
• (loop) A simple loop function to apply “template” to all “links”.

Where are links coming from? Most likely collect from an index page (example: top rated 250 movies on IMDb)

Method 1: the more visual way.

• Use a browser addon such as “linkKlipper” for Chrome.
• Automatically saved into a .txt file (renamed as linkklipper.txt).
• Load into R.

library(dplyr)
link_klipper<-read.table("linkklipper.txt",header=FALSE)[[1]]
length(link_klipper)

[1] 250

Method 2: use an html page parser such as rvest package.

• Read the source page. What does it look like? Right click, then view page source.

library(rvest)
index_page <- read_html("http://www.imdb.com/chart/top?ref_=nv_ch_250_4")

• Locate the URLs for the top 250 movies. Each stored under this tag: titleColumn. Then, go one step deeper and extract the (partial) link from the href tag.

partial_link <- html_attr(html_node(html_nodes(index_page ,".titleColumn"),"a"),"href")

• Attach the prefix to get full link.

link_rvest <- paste0("http://www.imdb.com/",partial_link)
length(link_rvest)

[1] 250

Chain things together in an easy to read way with magrittr.

Each %>% enters results from previous to the fisrt argument of the next function.

library(magrittr)
link_rvest <- 
  read_html("http://www.imdb.com/chart/top?ref_=nv_ch_250_4") %>% 
  html_nodes(".titleColumn") %>% 
  html_node("a") %>% 
  html_attr("href") %>% 
  paste0("http://www.imdb.com",.)
length(link_rvest)

[1] 250

Time to construct a crawling template to feed into the loop that will iterate through the links.

When extracting links from the index page, we have seen these:

html_attr(html_node(html_nodes(index_page ,".titleColumn"),"a"),"href")

• rvest parses a downloaded web page by uing the html tags.
• Is there an easier way to come up with the “right” tag?
• Check out the CSS Selector Gadget Chrome addon

• Movie title -> CSS Selector -> html tag h1

library(stringi);library(stringr)
title <- link_rvest[14] %>% read_html() %>% 
  html_nodes("h1")
title[[2]] %>% html_text() 

[1] "Inception (2010)            "

• Some text trimming

title <- title[[2]] %>% html_text() %>% 
  stri_trim_both() %>% substr(.,1,nchar(.)-7)
title

[1] "Inception"

Similar work for rating:

rating <- 
  link_rvest[14] %>% 
  read_html() %>%
  html_node(".ratingValue") %>%
  html_text() %>%
  stri_replace_all_fixed("/10","") %>%
  stri_trim_both() %>%
  as.numeric()
rating

[1] 8.8

• A little more work to uniquely locate director.

• Parse out the director text.

director <- link_rvest[14] %>% read_html() %>%
  html_node(".summary_text+ .credit_summary_item") %>%
  html_text() 
director

[1] "\n        Director:\n            \nChristopher Nolan            \n    "

• Movie might two directors (e.g. The Lego Movie).

director %<>% str_split(":|,")
director <- director[[1]][2] %>% str_trim
director

[1] "Christopher Nolan"

• Put title, rating, and director into a data set.

one_movie <- data.frame(title,rating,director)
one_movie

      title rating          director
1 Inception    8.8 Christopher Nolan

Wrap the template into a function for a cleaner loop view.

Test it out with No. 7, Pulp Fiction.

template(link_rvest[7])

         title rating          director
1 Pulp Fiction    8.9 Quentin Tarantino

Everything works fine!

# An empty object to store data
movie_data <- NULL

# Loop through the 250 movie
for (i in 1:250) {
  movie <- template(link_rvest[i])

  # Extend the length of data set after each iteration
  movie_data %<>% rbind(movie)
}

The result is a nice data set ready for analysis!

What and Why

A Basic Crawler

Extensions

Back to the link gathering. What if the links are stored in multiple index pages?

Example: top rated action movies on IMDb

• Open a few pages (usually 3 will be enough), compare addresses to see what's changing.
• Identify moving parts
• Construct URLs for all pages
• Loop through them to collect links (for the movie page)

Basice structure of a web address:

• Page layout defined by http://www.imdb.com/search/title?.

• Everything after that requests what to be filled within the page and in what format.

• Each specific request is separated by &

• Page 1 requests: genres=action& sort=user_rating,desc& title_type=feature& num_votes=25000,& pf_rd_m=A2FGELUUNOQJNL& pf_rd_p=2406822102& pf_rd_r=1HYX8S8DAZ1MCQ1CMYQE& pf_rd_s=right-6& pf_rd_t=15506& pf_rd_i=top& ref_=chttp_gnr_1

• Page 2 requests: genres=action& num_votes=25000,& pf_rd_i=top& pf_rd_m=A2FGELUUNOQJNL& pf_rd_p=2406822102& pf_rd_r=1HYX8S8DAZ1MCQ1CMYQE& pf_rd_s=right-6& pf_rd_t=15506& ref_=chttp_gnr_1& sort=user_rating,desc& start=51& title_type=feature

• Hard to compare; some string parsing is useful here

• Only start= is changing. Construct URLs for the 20 pages.

start_url <- paste0("http://www.imdb.com/search/title?genres=action&sort=user_rating,desc&title_type=feature&num_votes=25000,&pf_rd_m=A2FGELUUNOQJNL&pf_rd_p=2406822102&pf_rd_r=1HYX8S8DAZ1MCQ1CMYQE&pf_rd_s=right-6&pf_rd_t=15506&pf_rd_i=top&ref_=chttp_gnr_1&start=",
                   seq(from=1,to=1051,by=50))
length(start_url)

[1] 22

• Loop through the multiple index pages to collect the movie URLs.

link_rvest <- NULL
for (i in 1:22) {
  links <- read_html(start_url[i]) %>% 
    html_nodes(".lister-item-header a") %>%  
    html_attr("href") %>% 
    paste0("http://www.imdb.com",.)
  link_rvest %<>% c(links)
}
length(link_rvest)

[1] 1083

• At least several pages are accessed every second
• Overflow for the website
• More importantly, this is obviously not human behavior.
• It's one thing to attract attention.
• It's another to attract flooding traffic generated by “headless” crawling.
• Many websites will show captcha box.
• Ops…

• Idea: access once every X seconds.
• Usually it's enough to start with X=5.

movie_data <- NULL

for (i in 1:250) {
  movie <- template(link_rvest[i])
  movie_data %<>% rbind(movie)

  # After each iteration, pause for 5 sec
  Sys.sleep(5)
}

• If the crawler works fine, reduce X and so on.

• Sometimes, low frequency alone does not solve the problem.
• After all, who can do “robotic” clicks every X seconds?
• A robot can…
• So after a while, “captcha” again.
• In this case, add some randomness to the pause, just like a human does.
• If the lowest X is 3, try this: Sys.sleep(runif(1,3,5))

In the IMDb example, we have collected information scattered on a webpage.

Sometimes, we might be better off collecting a whole table.

Possibility 1: information we need is well formated in a table.

Possibility 2: some information available on one page is not available on another page.

An html table starts with the tag table.

Use html_table function to extract table into an R data set.

fb_data <- page_url %>% read_html %>% 
  html_node(".frp") %>% html_node("table") %>% html_table()
fb_data

  X1       X2   X3    X4    X5
1 NA Positive 1626 16725 28407
2 NA  Neutral    1    32    55
3 NA Negative    4    34    49

Some simple text processing stores these feedback ratings as individual variables

fb_data %<>% `[`(3:5) %>% 
  unlist() %>% as.data.frame() %>% 
  t() %>% as.data.frame() %>% tbl_df()
names(fb_data) <- c("p1","m1","n1","p6","m6","n6","p12","m12","n12")
fb_data %>% kable()

Difference in information availability between Amazon and Cyberlink is nontrivial.

Data position also shifts.

Its better to collect all tables and then do text processing.

Many “fancy-looking” pages have Java script generated content. Example: English Premier League Table.

rvest still works fine as a parser, but it won't read the Java content. These content are typically loaded with a delay.

Something more sophisticated needed to load the page. A headless browser such as PhantomJS is great for such task.

“A headless browser is a web browser without a graphical user interface. Headless browsers provide automated control of a web page in an environment similar to popular web browsers” – Wikipedia

• Put PhantomJS binary in the R working directory.
• Write a simple Java script to merely load the page.

var url ='http://www.somepage.com'; # replace this line in R
var page = new WebPage(); var fs = require('fs');
# this opens the page and waits for a while
page.open(url, function (status) {
  just_wait();
});
# the waiting is 2500 miliseconds; adjust as fit
function just_wait() {
  setTimeout(function() {
    fs.write('myfile.html', page.content, 'w');
    phantom.exit();
  }, 2500);
}

• Save the Java script “scrape.js” in the R working directory.
• Call “scrape.js” witin R. The rest are the same as before.

# change the first line of scrape.js scrape
# replace with the URL of the to-be-scraped page
lines <- readLines("scrape.js") 
lines[1] <- paste0("var url ='", url ,"';")
writeLines(lines, "scrape.js")

## let phantomjs download the website
system("phantomjs scrape.js")

• Loaded page is saved as “myfile.html”.
• Then, standard web crawling with rvest.

The more realistic websites to scrape might be:

• OTAs such as Expedia and Travelocity
• Airlines
• Hotels
• Newspapers such as Washington Post and New York Times
• Propriatary company-level data

Google spreadsheet has some basic scraping features.

Subscription-based online services:

• import.io
• scrapinghub

Local one-time price softwares

• web content extractor

What and Why

A Basic Geocoder

Extensions

Within a single data set

• Calculate distance between people, company, etc
• Standardize address to form groups

Between multiple data sets

• Matching observations from one data set to another: e.g., organization data, worker data, who works for which org?

Spatial Econometrics

• Distance between two people/company
• Distance moved from time A to B: e.g., Uber driver

X variants for “100 N Main Ave Tallahassee FL”

• 100 = 101 = 102 …
• N Main = north main = North main
• Ave = avenue
• , between Ave and Tallahassee?
• with or without zip code?
• string matching simply doesn't work well!
• IDEA: no matter which format you tell google map, you always get the lovely red pin on the same spot.

What and Why

Geocoding Packages in R

Extensions

library(ggmap);library(RDSTK)
geocode("florida state university",source="google")

        lon      lat
1 -84.29849 30.44188

geocode("600 W College Ave, Tallahassee, FL 32306",source="google")

        lon      lat
1 -84.29067 30.44096

RDSTK - data science tool kit API

street2coordinates("600 W College Ave, Tallahassee, FL 32306")[c(5,3)]

  longitude latitude
1 -84.28844 30.44073

library(geosphere)

umd <- geocode("university+of+maryland+college+park",source="google")
fsu <- geocode("florida+state+university",source="google")

distCosine(fsu,umd)/1609.34

[1] 723.4794

What and Why

Geocoding Packages in R

Extensions

• One good aspect of Google API is that it simplifies the caculation of distance/time between A and B by travel mode.

• Use my little wrapper function travel.dist.

travel.dist(origin='university+of+maryland+college+park',
destination='florida+state+university')

[1] 879.3297

• Verify it on Google Map

ggplot2 Examples

Evolution of Interaction Effect Plots

• Cheatsheet
• R graphics cookbook: mostly free; hardcopy not expensive
• Graph and code side-by-side catalog: designed in shiny

ggplot2 Examples

Evolution of Interaction Effect Plots

\[ y=\beta_0+\beta_1 x + \beta_2 f + \beta_3 x*f + \epsilon \]

• x is a continuous variable
• f is a two-level factor variable
• Problem: plot the effect of x
• What's the conventional interaction plot look like in R?

\[ y=\beta_0+\beta_1 x + \beta_2 f + \beta_3 x*f + \epsilon \]

• Problem: plot the effect of f
• effect size is this: \( \beta_2 + \beta_3 x \)
• The confidence band plot comes about

\[ y=\beta_0+\beta_1 x + \beta_2 z + \beta_3 m + \beta_4 x*z + \beta_5 x*m + \epsilon \]

• What if there are two “moderators”?
• Plot the effect of x
• Effect size is this: \( \beta_1 + \beta_4 z +\beta_5 m \)
• One way is to fix m and do confidence band plot with z.
• But too many plots!
• Here's a solution with one plot