forked from tidyverse/dplyr
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.Rmd
executable file
·186 lines (133 loc) · 6.71 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
---
output:
md_document:
variant: markdown_github
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, echo = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "README-"
)
```
# dplyr
[](https://travis-ci.org/hadley/dplyr)
[](https://ci.appveyor.com/project/hadley/dplyr)
[](http://cran.r-project.org/package=dplyr)
[](https://codecov.io/github/hadley/dplyr?branch=master)
dplyr is the next iteration of plyr, focussed on tools for working with data frames (hence the `d` in the name). It has three main goals:
* Identify the most important data manipulation tools needed for data analysis
and make them easy to use from R.
* Provide blazing fast performance for in-memory data by writing key pieces
in [C++](http://www.rcpp.org/).
* Use the same interface to work with data no matter where it's stored, whether
in a data frame, a data table or database.
You can install:
* the latest released version from CRAN with
```R
install.packages("dplyr")
````
* the latest development version from github with
```R
if (packageVersion("devtools") < 1.6) {
install.packages("devtools")
}
devtools::install_github("hadley/lazyeval")
devtools::install_github("hadley/dplyr")
```
You'll probably also want to install the data packages used in most examples: `install.packages(c("nycflights13", "Lahman"))`.
If you encounter a clear bug, please file a minimal reproducible example on [github](https://github.com/hadley/dplyr/issues). For questions and other discussion, please use the [manipulatr mailing list](https://groups.google.com/group/manipulatr).
## Learning dplyr
To get started, read the notes below, then read the intro vignette: `vignette("introduction", package = "dplyr")`. To make the most of dplyr, I also recommend that you familiarise yourself with the principles of [tidy data](http://vita.had.co.nz/papers/tidy-data.html): this will help you get your data into a form that works well with dplyr, ggplot2 and R's many modelling functions.
If you need more help, I recommend the following (paid) resources:
* [dplyr](https://www.datacamp.com/courses/dplyr) on datacamp, by Garrett
Grolemund. Learn the basics of dplyr at your own pace in this interactive
online course.
* [Introduction to Data Science with R](http://shop.oreilly.com/product/0636920034834.do):
How to Manipulate, Visualize, and Model Data with the R Language, by Garrett
Grolemund. This O'Reilly video series will teach you the basics needed to be
an effective analyst in R.
## Key data structures
The key object in dplyr is a _tbl_, a representation of a tabular data structure.
Currently `dplyr` supports:
* data frames
* [data tables](https://github.com/Rdatatable/data.table/wiki)
* [SQLite](http://sqlite.org/)
* [PostgreSQL](http://www.postgresql.org/)/[Redshift](http://aws.amazon.com/redshift/)
* [MySQL](http://www.mysql.com/)/[MariaDB](https://mariadb.com/)
* [Bigquery](https://developers.google.com/bigquery/)
* [MonetDB](http://www.monetdb.org/)
* data cubes with arrays (partial implementation)
You can create them as follows:
```{r, message = FALSE}
library(dplyr) # for functions
library(nycflights13) # for data
flights
# Caches data in local SQLite db
flights_db1 <- tbl(nycflights13_sqlite(), "flights")
# Caches data in local postgres db
flights_db2 <- tbl(nycflights13_postgres(host = "localhost"), "flights")
```
Each tbl also comes in a grouped variant which allows you to easily perform operations "by group":
```{r}
carriers_df <- flights %>% group_by(carrier)
carriers_db1 <- flights_db1 %>% group_by(carrier)
carriers_db2 <- flights_db2 %>% group_by(carrier)
```
## Single table verbs
`dplyr` implements the following verbs useful for data manipulation:
* `select()`: focus on a subset of variables
* `filter()`: focus on a subset of rows
* `mutate()`: add new columns
* `summarise()`: reduce each group to a smaller number of summary statistics
* `arrange()`: re-order the rows
They all work as similarly as possible across the range of data sources. The main difference is performance:
```{r}
system.time(carriers_df %>% summarise(delay = mean(arr_delay)))
system.time(carriers_db1 %>% summarise(delay = mean(arr_delay)) %>% collect())
system.time(carriers_db2 %>% summarise(delay = mean(arr_delay)) %>% collect())
```
Data frame methods are much much faster than the plyr equivalent. The database methods are slower, but can work with data that don't fit in memory.
```{r}
system.time(plyr::ddply(flights, "carrier", plyr::summarise,
delay = mean(arr_delay, na.rm = TRUE)))
```
### `do()`
As well as the specialised operations described above, `dplyr` also provides the generic `do()` function which applies any R function to each group of the data.
Let's take the batting database from the built-in Lahman database. We'll group it by year, and then fit a model to explore the relationship between their number of at bats and runs:
```{r}
by_year <- lahman_df() %>%
tbl("Batting") %>%
group_by(yearID)
by_year %>%
do(mod = lm(R ~ AB, data = .))
```
Note that if you are fitting lots of linear models, it's a good idea to use `biglm` because it creates model objects that are considerably smaller:
```{r}
by_year %>%
do(mod = lm(R ~ AB, data = .)) %>%
object.size() %>%
print(unit = "MB")
by_year %>%
do(mod = biglm::biglm(R ~ AB, data = .)) %>%
object.size() %>%
print(unit = "MB")
```
## Multiple table verbs
As well as verbs that work on a single tbl, there are also a set of useful verbs that work with two tbls at a time: joins and set operations.
dplyr implements the four most useful joins from SQL:
* `inner_join(x, y)`: matching x + y
* `left_join(x, y)`: all x + matching y
* `semi_join(x, y)`: all x with match in y
* `anti_join(x, y)`: all x without match in y
And provides methods for:
* `intersect(x, y)`: all rows in both x and y
* `union(x, y)`: rows in either x or y
* `setdiff(x, y)`: rows in x, but not y
## Plyr compatibility
You'll need to be a little careful if you load both plyr and dplyr at the same time. I'd recommend loading plyr first, then dplyr, so that the faster dplyr functions come first in the search path. By and large, any function provided by both dplyr and plyr works in a similar way, although dplyr functions tend to be faster and more general.
## Related approaches
* [Blaze](http://blaze.pydata.org)
* [|Stat](http://oldwww.acm.org/perlman/stat/)
* [Pig](http://dx.doi.org/10.1145/1376616.1376726)