Now back to the problem at hand

I, like many others, am very excited about the early release of Positron and love seeing the community get their hands on it. As a Posit employee, I’ve had the chance to use it for many months. One of its biggest strengths is the Open VSX Registry extension mechanism. It unlocks infinite customizability, just like in VS Code.

Being so attached to the Tomorrow Night Bright theme in RStudio, I quickly noticed that Positron includes Tomorrow Night Blue by default, but not my favorite theme. I tried out Microsoft’s own Tomorrow Night Bright extension, but the colors felt off. It had an excessive amount of syntax highlighting and an especially distracting use of red. Here’s what the same lines of anes_2020.R look like using the Microsoft port of Tomorrow Night Bright in Positron:

The Microsoft port of Tomorrow Night Bright had dated grey/blue workbench elements, reflecting its original design (released 2015, last updated 2017), rather than the style of more modern dark themes. Looking for other ‘bright’ themes, I found GitHub Dark Default, a sleek dark-and-bright theme with 16.3M installs in the VS Code Extension Marketplace and 156K in the Open VSX Registry as of this writing. It is featured in Garrick Aden-Buie‘s Positron Extension Pack, Positron +1e. While very popular, GitHub Dark Default also suffers from excessive (’saturated’) syntax highlighting of R code in Positron:

With help from my brother, I decided to write my own extension to better approximate what I’m used to in RStudio, and I’m pleased to announce that the resulting extension ‘Tomorrow Night Bright (R Classic)’ is now available on both the Open VSX Registry for Positron and the VS Code Extension Marketplace for VS Code!

Here is the final look:

Let’s walk through porting this theme over to Positron.

Creating a VS Code color theme extension

# Install Homebrew (if you don't already have it on macOS)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install Git after Homebrew is installed
brew install –-cask git

# Download and install nvm:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

# Download and install Node.js:
nvm install 22

# Verify the Node.js version:
node -v # Should print "v22.13.1".
nvm current # Should print "v22.13.1".

# Verify npm version:
npm -v # Should print "10.9.2".

npm install --global yo generator-code

cd ~/rrr

yo code

fs::dir_tree("~/rrr/tomorrow-night-bright-r-classic", all = TRUE)
#> ~/rrr/tomorrow-night-bright-r-classic
#> ├── .gitattributes
#> ├── .gitignore
#> ├── .vscode
#> │   ├── launch.json
#> ├── .vscodeignore
#> ├── CHANGELOG.md
#> ├── README.md
#> ├── package.json
#> ├── themes
#> │   └── tomorrow-night-bright-r-classic.json
#> ├── vsc-extension-quickstart.md

Customizing the extension

First, gather all of the color themes that you’ll use for inspiration. In our case, the file tomorrow-night-bright-r-classic.json in our local extension created by Yeoman currently mirrors the Microsoft port of Tomorrow Night Bright. The next steps for us were:

• Inspect the colors in GitHub Dark Default. If you have it installed, the theme file is located at ~/.positron/extensions/github.github-vscode-theme-6.3.5-universal/themes/dark-default.json or similar depending on the version of your copy.
• Change a color setting in tomorrow-night-bright-r-classic.json and review how it impacts your theme in the [Extension Development Host] window.
• Wash, rinse, and repeat until you’re happy with how your new extension looks!

This is the part of extension development that took the most time. It would be difficult to walk you through all of the individual design decisions we made while porting the theme. Briefly, since GitHub Dark Default is a great dark extension, it helped a lot to start with its settings, then to tweak its colors to superimpose editor and workbench colors specific to Tomorrow Night Bright. We kept most of GitHub Dark Default’s workbench colors, except that we changed its workbench green/orange accent colors to blue/orange accent colors — to match the Tomorrow Night Bright theme and aesthetic. In the end, we were able to port the editor colors to behave quite close to how they do in RStudio. We’re quite happy with the result!

Below I’ve reproduced tabsets with editor and workbench colors across the original Tomorrow Night Bright, original GitHub Dark Default, and the new Tomorrow Night Bright (R Classic) for reference. If you’re interested in seeing the R code we used to tame three JSON theme files into the tabsets below, you can find it here.

Editor Colors

Workbench Colors

Fin: Tomorrow Night Bright (R Classic)

Once the tomorrow-night-bright-r-classic.json file was ready, then it was time to publish the theme to the VS Code Extension Marketplace and then the Open VSX Registry. We added some resources on how to do this below. With that, the Tomorrow Night Bright (R Classic) theme is available to the world!

Like any other open-source project, this is just the beginning! We’ve optimized the theme for R scripts and many key workbench elements, but there’s more to do. Our next steps will be to optimize the theme for Python scripts, JSON files, and other files supported by RStudio — so those migrating from RStudio to Positron will feel at home in Tomorrow Night Bright. We welcome you to submit a pull request in the GitHub repository if you’d like to make a contribution or use the files as a reference if you want to create your own Positron theme!

Installing Tomorrow Night Bright (R Classic)

To install the theme in Positron, navigate to the Extensions marketplace by:

• Typing Cmd + Shift + P to access the Command Palette and searching for Extensions: Install Extensions,
• Typing the Extensions shortcut Cmd + Shift + X,
• Or, clicking the button on the left sidebar in Positron.

Search for ‘Tomorrow Night Bright (R Classic)’ and then click Install. Tada 🎉 you now have our theme installed. We hope you enjoy it!

More resources

The process of building and publishing an extension can be quite involved! We found these links to be particularly helpful:

• VS Code: Your First Extension: This is a primer on how to build an extension from first principles.
• VS Code: Color Theme: This is a primer on how to build color theme extensions, and how color theme extensions are structured.
• VS Code: Theme Color: This is comprehensive documentation on all color settings available in VS Code.
• VS Code: Publishing Extensions: This is a tutorial on how to publish an extension on the VS Code Extension Marketplace.
• VS Code: Extension Manifest: This is detailed documentation on the package.json file in the root folder of all extensions, also known as the ‘extension manifest’ or ‘manifest.’ Our extension’s package.json may be helpful for reference. N.B., the Yeoman utility generates only a bare-bones package.json, but to publish your extension online you’ll need to add a publisher field and more. You will also need to add a package-lock.json file!
• Open VSX: Publishing Extensions: This is documentation on how to publish an extension to the Open VSX Registry.
• HaaLeo/publish-vscode-extension: This is a very helpful GitHub Action, recommended by the Open VSX documentation, that automates publishing your extension in the VS Code Extension Marketplace and the Open VSX Registry. We lightly modified this GitHub Action in our extension: publish-extension.yaml.

Thanks to Mickaël Canouil’s Quarto extension, Preview Color, which allowed us to display colors associated with hex codes in this blog post.

Before we begin… Quarto dashboards?!

That’s right! With version 1.4, Quarto now supports dashboards — a new output format for easily creating dashboards from notebooks in R, Python, or Julia. You can include plots, tables, value boxes, and text, and deploy your dashboards as static web pages (no special server needed). For enhanced interactivity, you can integrate a backend Shiny server. You can also customize the appearance by adding a Bootstrap theme or custom Sass elements.

To learn more about creating dashboards, check out the Quarto documentation or watch my R-Ladies Rome Quarto Dashboards talk.

Consumer Price Index (CPI) data

This dashboard uses the Consumer Price Index (CPI) data from the Bureau of Labor Statistics (BLS). The CPI monitors changes in the prices of a basket of goods over time, covering a range of categories, and is updated monthly. The dashboard automatically refreshes each month when the latest BLS data is released.

Setup

To open the project in RStudio, navigate to File > New Project > Version Control and paste the following URL: https://github.com/ivelasq/inflation-explorer.git.

The files in the project include:

├── images
│   └── logo.png            # Logo image to use in the dashboard
├── .gitignore              # Files and folders to ignore when pushing up changes
├── README.md               # Project overview and setup instructions
├── _publish.yml            # File for specifying the publishing destination
├── _quarto.yml             # Quarto project configuration file
├── all_data_report.json    # JSON version of the downloaded BLS data
├── custom.scss             # Custom Sass file
├── index.qmd               # Quarto dashboard document
└── script.R                # R script version of the ETL script

Terminal

Sys.setenv(CONNECT_API_KEY = "connect api key here")

Extract-transform-load process

We use a script to pull raw data from the BLS API, clean it for use in our dashboard, and then save it as a pin. This process, known as Extract-Transform-Load (ETL), is executed in the script.R file:

script.R

#' ---
#' title: "R script file"
#' ---

library(httr2)
library(jsonlite)
library(dplyr)
1library(pins)
library(tidyr)
library(purrr)
library(lubridate)
library(stringr)

2# Environment variables

bls_key <- Sys.getenv("BLS_KEY")
connect_server <- Sys.getenv("CONNECT_SERVER")
connect_api_key <- Sys.getenv("CONNECT_API_KEY")

3# BLS tables

table_ids <- c(
  "CUUR0000SA0",
  "CUUR0000SA0L1E",
  "CUUR0000SAF1",
  "CUUR0000SA0E",
  "CUUR0000SETB01",
  "CUUR0000SAM",
  "CUUR0000SEMC01",
  "CUUR0000SEMD01",
  "CUUR0000SEMF01",
  "CUUR0000SAH1"
)

id_to_label <- c(
  "CUUR0000SA0" = "All groups CPI",
  "CUUR0000SA0L1E" = "All items less food and energy",
  "CUUR0000SAF1" = "Food",
  "CUUR0000SA0E" = "Energy",
  "CUUR0000SETB01" = "Gasoline",
  "CUUR0000SAM" = "Medical care",
  "CUUR0000SEMC01" = "Physicians' services",
  "CUUR0000SEMD01" = "Hospital services",
  "CUUR0000SEMF01" = "Prescription drugs",
  "CUUR0000SAH1" = "Shelter"
)

4# Pull BLS API data

get_bls_data <- function(parameters) {
  response <- request("https://api.bls.gov/publicAPI/v2/timeseries/data/") |>
    req_headers("Content-Type" = "application/json") |>
    req_body_json(parameters, auto_unbox = TRUE) |>
    req_perform()
  
  if (resp_status(response) != 200) {
    stop(paste("API Error:", resp_status(response)))
  }
  
  return(resp_body_json(response))
}

all_data <- list()

for (table_id in table_ids) {
  parameters <- list(
    registrationkey = bls_key,
    seriesid = list(table_id),
    startyear = "2018",
    endyear = "2024",
    calculations = TRUE
  )
  
  bls_data_object <- get_bls_data(parameters)
  all_data[[table_id]] <- bls_data_object
}

file_path <- file.path(getwd(), "all_data_report.json")

write_json(all_data, path = file_path, pretty = TRUE)

dat <- fromJSON(file_path)

5# Clean data

series_dat <- dat |>
  map( ~ .x$Results$series) |>
  map( ~ tibble(seriesID = .x$seriesID, data = .x$data)) |> 
  list_rbind()

combined_dat <- series_dat |>
  unnest(data)

clean_dat <- combined_dat |>
  mutate(
    year_month = ymd(paste(year, str_sub(period, 2, 3), "01", sep = "-")),
    value = as.numeric(value),
    seriesID = as.character(seriesID),
    category_label = recode(seriesID, !!!id_to_label)
  )

january_2018_values <- clean_dat |>
  filter(year_month == ymd("2018-01-01")) |>
  select(seriesID, jan_2018_value = value) |>
  distinct()

joined_dat <- clean_dat |>
  left_join(january_2018_values, by = "seriesID")

final_cpi_dat <- joined_dat |>
  mutate(
    jan_2018_diff = value - jan_2018_value,
    jan_2018_pct_change = (jan_2018_diff / jan_2018_value) * 100
  ) |>
  arrange(year_month) |>
  mutate(percent_change_from_previous_month = (value / lag(value) - 1) * 100,
         .by = category_label)

6# Pin data to Connect

board <- board_connect()
pin_write(board = board, name = "isabella.velasquez/bls-cpi-data", x = final_cpi_dat)

1

The {pins} package allows you to pin data, models, or other objects for easy access or sharing. Pins are ideal for ephemeral, ever-changing data that needs to be referenced but does not require a database.

2

We load the environment variables configured earlier.

3

Next, we define the list of BLS tables containing the data we want to display on the dashboard.

4

A function then pulls this data into R.

5

This is followed by a series of data cleaning and transformation steps.

6

Finally, we configure our Posit Connect server² and write the pin to Connect. Access the pin here.

Posit Connect supports rendering script files (e.g., .py or .R). Once deployed to Connect, these scripts can be scheduled for automatic refresh and their output can be emailed, similar to other content on Connect. For more details, refer to the Scripts documentation.

To publish script.R to Connect, run the following command:

Terminal

rsconnect::deployApp(appFiles = "script.R")

This is the content owner view of the deployed script. You can configure it to update daily at 11 PM PT using the configuration pane.

View the deployed script here.

Creating a Quarto dashboard

Now, we can move on to actually creating a Quarto dashboard!

index.qmd

---
title: "Inflation explorer: showing the change in price for various goods and services"
format:
1  dashboard:
2    logo: images/logo.png
    nav-buttons:
    - icon: github
      href: "https://github.com/ivelasq/inflation-explorer"
    theme:
    - pulse
    - custom.scss
3resource_files:
- custom.scss
- images/logo.png
---

4```{r}
#| label: setup
#| include: false
library(pins)
library(lubridate)
library(dplyr)
library(tidyr)
library(ggplot2)
library(RColorBrewer)
library(gt)
library(plotly)

5board <-
  board_connect(
    auth = "manual",
    server = "https://pub.demo.posit.team/",
    key = Sys.getenv("CONNECT_API_KEY")
  )

df <-
  pin_read(board = board, name = "isabella.velasquez/bls-cpi-data") |>
  mutate(year_month = as.Date(year_month, format = "%Y-%m-%d"))

# Value for last updated date
last_updated_date <- df |>
  summarise(last_updated_date = max(year_month)) |>
  pull(last_updated_date)

# Value for the percentage change from the previous month
cpi_df <- df |> 
  filter(seriesID == "CUUR0000SA0")

latest_cpi_value <- cpi_df |>
  summarise(latest_value = last(value)) |>
  pull(latest_value)

latest_cpi_percent_change <- cpi_df |>
  filter(year_month == last_updated_date) |>
  pull(jan_2018_pct_change)

# Data for table
pivot_df <- df |> 
  select(category_label, year_month, percent_change_from_previous_month) |>
  pivot_wider(names_from = year_month, 
              values_from = percent_change_from_previous_month) %>%
  select(category_label, tail(names(.), 5)) |> 
  arrange(desc(category_label))
```

## Row {height=20%}

```{r} 
#| label: valuebox1
#| content: valuebox
#| title: "Last updated"
#| color: #fff
6last_updated_date
```

```{r} 
#| label: valuebox2
#| content: valuebox
#| title: "Consumer Price Index (CPI)"
#| icon: basket
7list(
  value = round(latest_cpi_value, 2),
  icon = "basket",
  color = "primary"
)
```

```{r}
#| label: valuebox3
#| content: valuebox
#| title: "Increase from previous month"
#| icon: percent
#| color: primary
value = round(latest_cpi_percent_change, 2)
```

## Row {height=80%}

### Column {width=20%}

The **Consumer Price Index (CPI)** is a measure of the average change over time in the prices paid by urban consumers for a market basket of consumer goods and services. 

Indexes are available for the U.S. and various geographic areas. Average price data for select utility, automotive fuel, and food items are also available.

**Source: Bureau of Labor Statistics**

### Column {.tabset width=80%}

```{r}
#| label: pct-change-jan-2018-code
#| include: false
8df <- df %>%
  arrange(desc(category_label), desc(year_month))

p1 <- ggplot(df,             
             aes(x = year_month, 
                 y = jan_2018_pct_change, 
                 color = category_label,
                 group = category_label)) +
  geom_line(aes(group = category_label)) +
  scale_color_manual(values = ifelse(df$category_label == "All groups CPI",
                                     "#765AAF", "#765AAF")) +
  theme_minimal() +
  theme(
    axis.title.x = element_blank(),
    axis.title.y = element_blank(),
    axis.text.x = element_text(size = 8),
    axis.text.y = element_text(size = 9)
    )
```

::: {.card title="Percentage change since Jan 2018"}

Use this inflation data explorer to see which goods and services are getting cheaper or more expensive in the U.S.

```{r}
#| fig-width: 20
#| fig-height: 8
ggplotly(p1)
```
:::

```{r}
#| label: pct-change-monthly-code
#| include: false
latest_month_data <- df %>%
  filter(year_month == max(year_month)) %>%
  arrange(desc(percent_change_from_previous_month))

top_six_categories <- latest_month_data %>%
  slice_head(n = 6)

p2 <- ggplot(
  top_six_categories,
  aes(
    x = reorder(category_label, -percent_change_from_previous_month),
    y = percent_change_from_previous_month,
    fill = category_label
  )
) +
  geom_bar(stat = "identity") +
  scale_fill_manual(values = rev(colorRampPalette(brewer.pal(9, "BuPu"))(6))) +
  labs(x = "", y = "Percent Change") +
  scale_y_continuous(labels = scales::percent_format(accuracy = 0.1)) +
  theme_minimal() +
  theme(
    legend.position = "none",
    axis.text.x = element_text(size = 24),
    axis.title.x = element_text(size = 20),
    axis.text.y = element_text(size = 24)
  ) +
  coord_flip()
```

::: {.card title="Last month's top 6"}
Percentage change in CPI by category for the last month, top six categories

```{r}
#| fig-width: 18
#| fig-height: 8
p2
```
:::

```{r}
#| label: pct-change-monthly-table
#| title: "Monthly changes in prices"
colnames(pivot_df)[1] <- "Category" 
colnames(pivot_df)[-1] <- format(ymd(colnames(pivot_df)[-1]), "%B %Y")

pivot_gt <- pivot_df %>% 
9  gt() %>%
  tab_header(
    title = "Monthly Percent Change by Category",
    subtitle = "Last four months"
  ) %>%
  fmt_number(
    columns = 2:5,
    decimals = 2,
  ) %>%
  tab_style(
    style = cell_text(weight = "bold", align = "center"),
    locations = cells_column_labels(everything())
  ) |> 
  data_color(
    method = "numeric",
    palette = "BuPu"
  )

pivot_gt
```

1

Set the format to dashboard to create a Quarto dashboard.

2

Incorporate additional files, like a custom Sass file for themes and a logo.

3

Include resources for Posit Connect integration.

4

Use Quarto chunk options, as shown in this setup chunk.

5

Connect to Posit Connect and retrieve the recently created pin.

6

Add value boxes to highlight key text or numbers.

7

Use thelist() format as needed.

8

Embed static or interactive plots to enrich your dashboard with text and visuals.

9

Include tables as well.

Publishing scripts and dashboards

There are several ways to publish Quarto dashboards, as detailed in the publishing section of the Quarto website. In the webinar, I focused on Posit Connect. If you’re using RStudio, there are multiple deployment options for Connect, including push-button publishing, Git-backed publishing, and command-line publishing. Posit Connect also allows you to schedule document and script refreshes through a user-friendly configuration window.

7. Null-coalescing operator in R, %||%

OK, this one isn’t in base R – yet! In the upcoming release, R will automatically provide the null-coalescing operator, %||%. Per the release notes:

‘L %||% R’ newly in base is an expressive idiom for the ‘if(!is.null(L)) L else R’ or ‘if(is.null(L)) R else L’ phrases.

Or, in code:

`%||%` <- function(x, y) {
   if (is_null(x)) y else x
}

Essentially, this means: if the first (left-hand) input x is NULL, return y. If x is not NULL, return the input.

It was great to see Jenny Bryan and the R community celebrate the formal inclusion of the null-coalescing operator into the R language on Mastodon. The null-coalescing operator is particularly useful for R package developers, as highlighted by Jenny in her useR! 2018 keynote, used when the tidyverse team needs to assess whether an argument has been supplied, or if the default value which is commonly NULL has been passed, meaning that the default argument has been supplied.

However, the null-coalescing operator can also be useful in interactive use, for functions that take NULL as a valid argument. In this case, if supplied in the argument itself it can yield different interesting behaviors. For example:

There’s more discussion about the utility of the function.

The fun-ctions never stop

Want even more functions (base R or not)? Here are some other resources to check out:

• Maëlle Salmon’s blog posts on useful functions
• Neil Wright’s post on five useful R functions
• Yihui Xie’s list of three useful functions in base R
• Ella Kaye’s Advent of Code walkthroughs
• Ihaddaden M. EL Fodil, Ph.D’s Twitter thread
• TidyX Screencasts, such as this one

Thanks to all community members sharing their code and functions!

Liked this article? I’d love for you to share!

Get a YouTube API key

As is often the case with anything to do with Google, first, you must get credentials. In this example, we were interested in an API key that would work with the YouTube Data API v3.

1. Log into the Google Cloud Platform Console.
2. Create a new project.
3. From your dashboard, click to ‘Go to APIs Overview’ and navigate to ‘Credentials’ in the sidebar.
4. At the top, click ‘Create Credentials’ and choose ‘API key’.
5. Your API key will pop up. Save it for reference later in this walkthrough.
6. In the pop-up window (or in the hamburger menu for the API key), click ‘Edit API key’.
7. Scroll to the bottom and under ‘API restrictions’, select ‘Restrict key’, select ‘YouTube Data API v3’ from the dropdown menu, click ‘OK’ to exit the dropdown menu, then click ‘Save’. This will restrict your API key so that it can only be used with the YouTube Data API v3, which will limit its usefulness should anyone get ahold of your key.

Store your YouTube API key

Now that you have an API key, you will want to save it in your R environment (.Renviron) using usethis::edit_r_environ() if working interactively or in your repository secrets if using GitHub Actions. Storing the API key as an environment variable allows you to access it easily without the need to manually input it every time you run your code. GitHub Actions can interact with the API without manual intervention, and storing it as a repository secret keeps the key separate from your source code so it is not exposed to others.

To do this, follow the instructions below:

1. Install the CRAN version of {httr2} and the GitHub version of {tuber}. {pak} allows for easy installation of packages, whether they are on CRAN, GitHub, pull requests, etc.

# install.packages("pak")      # Install {pak} if not yet installed
pak::pak("httr2", "soodoku/tuber") # CRAN and GitHub syntax side by side!

2. On your local machine, save your YouTube API key in .Renviron. Running the code below will prompt you to save the API key. Paste your API key into the pop-up window:

tuber::yt_set_key()

Now your YouTube API key is saved as YOUTUBE_KEY in .Renviron!

Pull and clean data

Woohoo, you are credentialed! It’s time for the fun part — pulling and cleaning data. I did this in a script called data_processing.R.

In addition to {tuber}, here are other packages that will aid you in the cleaning process:

library(tuber) # GitHub version, installed with pak::pak(soodoku/tuber)
library(readr)
library(dplyr)
library(stringr)
library(DT)

Create a spreadsheet with the channels you are interested in. I created this manually — I just searched on YouTube for all the R-Ladies channels I could find. To make this workflow as programmatic as possible, I only included the chapter name, thumbnail, and channel ID.

dat <-
  read_csv("https://github.com/ivelasq/rladies-video-feed/blob/main/r-ladies_channels.csv")
  
head(dat)

  chapter            image                                              id   
  <chr>              <chr>                                              <chr>
1 R-Ladies Global    https://yt3.ggpht.com/ytc/AKedOLRM4FPiPcPBdFUmYWR… UCDg…
2 R-Ladies Baltimore https://yt3.ggpht.com/RQifC3qp_7AFeTc48-QV1i4WBNM… UC9U…
3 R-Ladies Philly    https://yt3.ggpht.com/ytc/AKedOLShIKBvPUKqbvm_Cpu… UCPq…
4 R-Ladies STL       https://yt3.ggpht.com/ytc/AKedOLSBA7rlz1yvbIJ7TVN… UCQ7…
5 R-Ladies Sydney    https://yt3.ggpht.com/ytc/AKedOLQnaU-dJbb14j2RE6W… UCkr…
6 R-Ladies Vancouver https://yt3.ggpht.com/3yf0Zo8-VKffrG-dRT_Gs85xX_x… UCX5…

Now, create a few more columns with dplyr::mutate() that expand the URLs into HTML format:

dat_urls <-
  dat %>%
  dplyr::mutate(
    feed = paste0("https://www.youtube.com/feeds/videos.xml?channel_id=", id),
    feed_url = paste0("yt:channel:", id),
    channel_image_url = paste0(
      "<img src='",
      image,
      "' alt='Hex Sticker for Chapter' width='40'></img>",
      " <a href='https://www.youtube.com/channel/",
      id,
      "' target='_blank'>",
      chapter,
      "</a><br>"
    ),
  )

The {tuber} documentation describes the many functions available to you. Many of these require OAuth 2.0 authentication. Since I had only the channel IDs, I wanted to use list_channel_videos() to get a complete list of videos from the channels, which thankfully works with an API key!

For example, for the R-Ladies Global channel, you could run list_channel_videos() to get its (currently 165) videos:

tuber::list_channel_videos("UCDgj5-mFohWZ5irWSFMFcng", max_results = 200, auth = "key")

What if you want the results for all the channels we have in our spreadsheet? I used a loop for this.

1dat_videos <- NULL

2for (i in 1:nrow(dat_urls)) {
  tmp <-
3    dat_urls[i, ]["id"] %>%
4    dplyr::pull() %>%
5    tuber::list_channel_videos(
      .,
      part = "snippet",
      config = list("maxResults" = 200),
      auth = "key"
    )
6  dat_videos <- dplyr::bind_rows(dat_videos, tmp)
}

1

Starting with an empty data frame called dat_videos,

2

For each row in dat_urls (which contains the channel IDs),

3

Find the channel ID,

4

Pull the ID,

5

Run it through list_channel_videos(), and

6

Add the information as rows to dat_videos.

The arguments in list_channel_videos() gave me all the columns I am interested in with part = "snippet". Notice that there’s a default limit for the number of videos pulled from the API. I bumped that up a bit with the maxResults argument.

Then, I brought back the URL info:

dat_join <-
  dat_videos %>%
  dplyr::left_join(., dat_urls, by = join_by("snippet.channelId" == "id"))

This results in a lot of information for each video. I cleaned it up a bit so that the data frame contained only the relevant columns in HTML format.

dat_dashboard <-
  dat_join %>%
  dplyr::mutate(
    video_url = paste0(
      "<a href='https://www.youtube.com/watch?v=",
      snippet.resourceId.videoId,
      "' target='_blank'>",
      snippet.title,
      "</a>"
    ),
    channel_url = paste0(
      "<img src='",
      image,
      "' alt='Hex Sticker for Chapter' width='40'></img>",
      "<a href='https://www.youtube.com/channel/",
      snippet.channelId,
      "' target='_blank'>",
      chapter,
      "</a>"
    ),
    date = as.Date(str_sub(snippet.publishedAt, 1, 10))
  ) %>%
  dplyr::arrange(desc(snippet.publishedAt)) %>%
  dplyr::select(date, chapter, channel_url, video_url, channel_image_url)

See the final data processing file on GitHub.

Create a flexdashboard

You have the YouTube data — time to create a pretty dashboard!

My flexdashboard started as R Markdown files often do: with a YAML header. I specified an orientation (columns) and added the link to the GitHub repository in the navigation bar.

---
title: "R-Ladies YouTube Video Feed"
output: 
  flexdashboard::flex_dashboard:
    orientation: columns
    navbar:
      - { icon: "fa-github", href: "https://github.com/ivelasq/rladies-video-feed", align: right }
---

---
title: "R-Ladies YouTube Video Feed"
output: 
  flexdashboard::flex_dashboard:
    orientation: columns
    navbar:
      - { icon: "fa-github", href: "https://github.com/ivelasq/rladies-video-feed", align: right }
    theme:
      version: 4
      bg: "#FDF7F7"
      fg: "#88398A" # purple
      primary: "#88398A" # purple
      base_font:
        google: "Lato"
---

library(flexdashboard)
library(bslib)
source("data-processing.R", local = knitr::knit_global())

Design the flexdashboard layout

This dashboard has a simple layout: just a sidebar and the main section. I recommend checking out the flexdashboard documentation to see all the layout options available to you.

This code builds out the sidebar section. I created a list of each R-Ladies chapter that I was able to find and arranged them by name. With htmltools::HTML(), the dashboard can render the URLs as HTML (the reason for all that manipulation earlier on). We learned that the .noWS = "outside" argument is crucial for deploying the dashboard with GitHub Actions. It omits extra whitespace around the HTML, which ensures that the dashboard is committed only when the number of videos has changed (rather than creating spurious commits each time data is pulled, even if the number of videos is unchanged).

Channels {.sidebar}
-----------------------------------------------------------------------

The purpose of this dashboard is to provide a running feed of R-Ladies videos posted to YouTube. It is refreshed every 24 hours.

Currently, the feed includes these channels:

```{r}
dat_join %>% 
  dplyr::arrange(chapter) %>% 
  dplyr::distinct(channel_image_url) %>% 
  dplyr::pull() %>% 
  htmltools::HTML(.noWS = "outside")
```

For the main body, I used {DT} to create a table for the information in our clean dataset.

Column {data-width=900}
-----------------------------------------------------------------------

### By default, the list is sorted by latest video.

<style>
.dataTables_scrollBody {
    max-height: 100% !important;
}
</style>

```{r}
dat_dashboard %>%
  dplyr::select(-chapter, -channel_image_url) %>%
  DT::datatable(
    colnames = c("Date", "Channel", "Video"),
    filter = "top",
1    escape = FALSE,
2    height = "1000",
    elementId = "dashboard",
3    options = list(columnDefs = list(
      list(className = "dt-middle", targets = "_all")
    ))
  )
```

1

escape = FALSE renders the URLs within the table as HTML.

2

height = "1000" makes it expand to the entire column height.

3

The list within options aligns the text within columns (in this case, to be in the middle of the cell for all columns).¹

And that’s it! See the final dashboard code on GitHub.

Try it out — search “Shiny” to see any video with Shiny in the title, or “Ecuador” to see all the videos from R-Ladies Ecuador!

Create a GitHub Action workflow

Great, we have a completed dashboard! Now, what if we want to keep it updated without manually re-knitting the dashboard? The next step is to create a GitHub Action. GitHub Actions are workflows that you create and configure to automate various tasks within your GitHub repositories. They are event-driven and can be triggered by actions such as code commits, pull requests, issue updates, or scheduled intervals.

on:
  schedule:
1    - cron: '0 0 * * *'

name: rladies-videos-bot

jobs:
  rladies-videos-bot:
    runs-on: ubuntu-latest
    env:
      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
      YOUTUBE_KEY: ${{ secrets.YOUTUBE_KEY }}
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: r-lib/actions/setup-pandoc@v2
      - uses: r-lib/actions/setup-r@v2
        with:
          use-public-rspm: true
2      - uses: r-lib/actions/setup-r-dependencies@v2
        with:
3          extra-packages: soodoku/tuber
      - name: Render and commit dashboard
        run: |
          Rscript -e 'rmarkdown::render("index.Rmd")'
          git config --local user.name "$GITHUB_ACTOR"
          git config --local user.email "$GITHUB_ACTOR@users.noreply.github.com"
          git add index.html
          git commit -m '[Bot] Render and commit dashboard' || echo "No changes to commit"
          git push origin || echo "No changes to commit"

Imports: 
    bslib,
    dplyr,
    DT,
    flexdashboard,
    htmltools,
    httr2,
    jsonlite,
    readr,
    rmarkdown,
    stringr

Host the dashboard

To host the dashboard, I used GitHub Pages. In the GitHub repository for your dashboard, go to ‘Settings’, then ‘Pages’. Choose the branch and folder of your flexdashboard index.html output file, click ‘Save’, and then you will have a URL to showcase your work.

Here is the final link for this dashboard: https://ivelasq.github.io/rladies-video-feed.

Summary

In summary, we have explored the powerful combination of {flexdashboard}, GitHub Actions (via r-lib/actions), and {tuber} to create an automated YouTube dashboard.

• With {flexdashboard}’s customizability and R Markdown support, we can create a visually appealing, interactive dashboard.
• The {tuber} package’s integration with the YouTube Data API and new functions we contributed to permit YouTube API key authentication allow us to easily fetch data and keep track of YouTube videos.
• With the automation capabilities of GitHub Actions, we can schedule dashboard updates and stay up-to-date with new YouTube videos.

It took a while, but we got there! Many thanks to @gvelasq’s support in getting this through the finish line.

Give this tutorial a try and unlock the potential of automated dashboards for your favorite YouTube channels! I’d love to see what you create!

The Problem

Like a new year 🗓️, a new computer 🖥️ (or a fresh reinstall) can bring so many opportunities. A clear desktop! So much memory! And then you remember that you have to install and configure all of your programs again, and that initial excitement gives way to despair and hopelessness. 😱

Okay, that’s a bit of an exaggeration, 🤭 but it can be tedious to reconfigure your setup the way that you like it. However, by using dotfiles, or hidden configuration files (whose names start with a dot/period, which on macOS hides them from view in the Finder), you can store your user preferences on a hosted version control service like GitHub. By dropping dotfiles in the correct directory, you can access and reuse your exact configuration and settings across machines, saving time and hassle. ♻️

In this post, we’ll share some helpful dotfiles for all you RStudio users who are rocking a new macOS 🍎 installation in 2023. This post is geared towards you if:

• You’ve been installing R and RStudio on macOS and setting them up in your own way already, but either forget the many steps (Git, etc.) or are always having to relearn them;
• You do a lot of point-and-click downloading of files for installation, and have been considering jumping into command-line adventures 👩🏻‍💻 with Homebrew;
• You’ve never heard of rig;
• You want a one-stop-shop for setting up your rig next time;
• You have a well-oiled, well-defined workflow but are interested in seeing how others set up their machines, to perhaps learn something new!

Since it is the year of ChatGPT, we did ask its suggested setup. We thought we could do better (at least for now, maybe not in 2024)! 🤗

Let’s fall into a pit of success ^TM (when configuring macOS)!

Installation

We’re going to use Homebrew to facilitate installation steps on macOS. Homebrew styles itself as ‘the missing package manager for macOS.’

1. Open Applications > Utilities > Terminal. Install Homebrew using the terminal command below, also provided on the Homebrew landing page. Paste this code into your terminal and hit enter:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

2. Install Git. This version control system is going to come in handy for future projects on your new installation.

brew install git

3. Configure global Git settings.

Run the code below (replacing the text in the quotation marks with your information) to adjust your Git settings.

git config --global user.name "<full-name>"
git config --global user.email "<email>"

4. Install zsh.

Several years back, macOS switched from using the bash shell to using zsh as the default shell. Installing the latest version of zsh from Homebrew gives you the latest version — otherwise, zsh and Git are both stuck at the version bundled the last time you updated macOS — which makes them open to vulnerabilities and/or don’t have the latest features.

brew install zsh

5. Install Oh My Zsh.

Oh My Zsh (OMZ) is ‘a delightful & open source framework for Zsh.’ Or as ChatGPT succinctly put it:

It comes with a ton of plugins, themes, and functions. For most users, having the great themes available from OMZ are enough to switch. But for us, the real value is all the built-in OMZ plugins — which provide a rich set of aliases for Homebrew, Git, and many other command-line programs. For example, gs is git status, bubo is brew update && brew outdated (very helpful), etc. Here are some you should know:

• The brew plugin and its aliases
• The git plugin
• The macos plugin for easier living through the Terminal
• There’s even a plugin that reminds you what your aliases are! – for maximum discoverability

You can also hand-code your themes into your .zshrc dotfile instead of using the OMZ themes. The aliases not provided by OMZ can be added to the custom.zsh dotfile, such as the ones in the dotfiles linked below.

Install OMZ with:

sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

6. Configure Oh My Zsh.

To configure oh-my-zsh, you can:

1. Edit your settings: Navigate to your home directory ~/ and edit the .zshrc file.
2. Use preconfigured settings: Navigate to your home directory ~/. Replace the default .zshrc file with this file. Then, navigate to the hidden folder ~/.oh-my-zsh/custom and drop this file in the folder.

A custom.zsh file might be helpful because you can add many more commands for OMZ plugins than what are built in. For instance,

alias bog="brew outdated --greedy"

This allows you to use bog to run brew outdated --greedy, instead of having to type the whole command out. That particular command is helpful if you want to find ALL the programs that could potentially be upgraded.

7. Go back to the Terminal and install rig.

rig, the ‘R Installation Manager,’ is an amazing tool. It not only lets you download R, but install, configure, switch between, and remove multiple R versions. It also allows you to manage user-level package libraries (meaning you can delete a version of R without losing all your packages!). Mara Averick wrote a great summary on Day 10 of Tidy Advent 2022.

Install rig by running these commands in the Terminal:

brew tap r-lib/rig
brew install --cask rig

8. Install R.

Install the latest version of R using rig:

rig add release

9. Install RStudio Desktop.

You want the latest and greatest RStudio, no? Use Homebrew to download and install the latest version. Once a new version is released, the community is very quick at updating the cask entry for RStudio in Homebrew.

brew install --cask rstudio

10. Configure .Rprofile.

In RStudio, install the {usethis} package, a tremendous tool for increased productivity with your R projects. In other words, you should use {usethis}. 😀

install.packages("usethis")

Once installed, open your .Rprofile by running:

usethis::edit_r_profile()

Here’s an example of an .Rprofile script. This script [1] sets {devtools} and {usethis} to load with each R session (to write install_github() without a library(devtools)), [2] sets Posit Public Package Manager (P3M) as the default repo for Linux (not relevant for this macOS guide), [3] sets a bunch of {usethis} defaults for package development which need to be overridden with personal details for each person, [4] sets vsc.rstudioapi = TRUE for VS Code (again not relevant for this guide), [5] sets several strict warnings which are helpful when developing packages (e.g., not using partial argument matching).

For your rig configuration, you would create a similar script, edit the .Rprofile, and restart the session.

11. Configure RStudio.

Snippets help quickly insert common short pieces of code. Open RStudio’s snippet config for R scripts:

usethis::edit_rstudio_snippets("r")

Append this script to the bottom of the file for some useful examples.

Try a snippet out! After appending the script to your snippet file, type dml in the Console and press tab:

Now, open RStudio’s preference file by running:

usethis::edit_rstudio_prefs()

Add the content of this file to the rstudio-prefs.json file. This file configures RStudio to look and act the way that you’d like (rainbow parentheses, anybody?). 🌈

Finally, add the contents of rstudio_bindings.json to the local version found in this hidden folder (I don’t think there’s a {usethis} command for this, unfortunately!):

~/.config/rstudio/keybindings/rstudio_bindings.json

Now, anytime you want to make sure you’re using the most current version of RStudio, you just have to run Shift+Cmd+R!

12. Set Git credentials in R.

The {gitcreds} package allows you to set your GitHub token once and use it everywhere (Git, R, and the RStudio IDE). Use it to set up RStudio with a Personal Access Token (PAT) with GitHub and have it work every time.

Install the {gitcreds} package:

install.packages("gitcreds")

Then, run usethis::create_github_token() to open a browser window to the GitHub form to generate a PAT with suggested, pre-selected scopes. This helps you make the right choice of scopes every time.

usethis::create_github_token()

Then, the command will suggest to use gitcreds::gitcreds_set() to save the PAT locally using the macOS Git credential manager (after copying it from GitHub):

gitcreds::gitcreds_set()

13. Install Quarto.

Quarto is a (kinda new) open-source scientific and technical publishing system built on Pandoc. It allows you to create dynamic content with R and Python, like documents, websites (like this blog!), and presentations.

Quarto comes bundled with RStudio v. 2022.07 and later, so the installation in step 9 above should have this covered. But, if you’d like to install Quarto yourself, run this in your Terminal:

brew install --cask quarto

You’ve done it! Congratulations on your configured, custom R data science setup.

Final Thoughts

We hope that this post is helpful in setting up an R data science rig on macOS! We know there were a lot of steps. In summary:

• Dotfiles are hidden configuration files where you can store your user preferences and use them across machines and installations.
• The configuration steps above ensure that you’re using the best tools for the macOS terminal: Homebrew (for package management) and zsh (for execution).
• They also ensure you’re using the best tools for running analyses: Git (for version control), rig (for R versioning), R (the best programming language [in our humble opinion]), RStudio (the GOAT), {usethis} (for project setup and development), {gitcreds} (for managing Git), and Quarto (for content creation and publishing).
• Tools like OMZ and snippets help you use aliases that make running your workflow easier to remember.

We know there are a ton of ways of setting up a data science rig, and we’d love to learn about your dotfile repositories, best practices, or favorite workflow packages on Twitter/Mastodon!

Related Work

• RSwitch was a menubar app that let you switch between different R versions; it is now archived.
• @grrrck’s guide, published in 2020, documents an epic adventure of setting up a Macbook Pro.
• {pak} is an awesome R package and the only reason we didn’t use it is because it doesn’t yet work with renv. Since we use {renv}, we’re holding off on {pak} for now, but we do look forward to using both tools together!

Where we have ChatGPT write our blog post for us

Here’s ChatGPT’s not bad (but still unnerving) response to: ‘Write a blog post, for a technical blog about the R programming language, entitled “Setting up macOS as an R data science rig in 2023.” Include installation and configuration examples for Homebrew, Git, zsh, oh-my-zsh, R, RStudio, and Quarto.’

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

brew install git

brew install zsh

sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

brew install r

install.packages("quarto")

siuba

The siuba package, created by Michael Chow, allows you to use dplyr-like syntax with pandas. Siuba ports over several functions, including select(), filter(), mutate(), summarize(), and arrange(). The package also allows you to use group_by() and a >> pipe.

Let’s check out a few examples using the palmerpenguins dataset (R, Python).

from palmerpenguins import load_penguins
penguins = load_penguins()
from siuba import group_by, summarize, _

(penguins
  >> group_by(_.species)
  >> summarize(n = _.species.count())
)

library(palmerpenguins)
library(dplyr)

(penguins %>% 
    group_by(species) %>%
    summarize(n = n()))

# A tibble: 3 × 2
  species       n
  <fct>     <int>
1 Adelie      152
2 Chinstrap    68
3 Gentoo      124

Thanks to the documentation and interactive tutorials available for siuba, it’s easy to see the parallels and differences with dplyr so that you can craft these data manipulation tasks yourself.

from palmerpenguins import load_penguins
penguins = load_penguins()
from siuba import select

(penguins
  >> select(-_.isalpha(), _.species)
  >> group_by(_.species)
  >> summarize(
      bill_length_mm = _.bill_length_mm.mean(),
      bill_depth_mm = _.bill_depth_mm.mean(),
      flipper_length_mm = _.flipper_length_mm.mean(),
      body_mass_g = _.body_mass_g.mean()
  )
)

(penguins %>%
  group_by(species) %>%
  summarize(across(!where(is.character), mean, na.rm = TRUE)))

# A tibble: 3 × 8
  species   island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
  <fct>      <dbl>          <dbl>         <dbl>             <dbl>       <dbl>
1 Adelie        NA           38.8          18.3              190.       3701.
2 Chinstrap     NA           48.8          18.4              196.       3733.
3 Gentoo        NA           47.5          15.0              217.       5076.
# … with 2 more variables: sex <dbl>, year <dbl>

plotnine

The plotnine package, created by Hassan Kibirige, lets you use a grammar of graphics for Python.

You can use siuba and plotnine together, similar to how you would use dplyr and ggplot2 together.

from siuba import *
from plotnine import *
from palmerpenguins import load_penguins
penguins = load_penguins()

(penguins
 # using siuba pipe
 >> ggplot(aes(x = 'flipper_length_mm', y = 'body_mass_g'))
 # creating plotnine plot
  + geom_point(aes(color = 'species', shape = 'species'),
             size = 3,
             alpha = 0.8)
  + theme_minimal()
  + labs(title = "Penguin size, Palmer Station LTER",
         subtitle = "Flipper length and body mass for Adelie, Chinstrap, and Gentoo Penguins",
        x = "Flipper length (mm)",
        y = "Body mass (g)",
        color = "Penguin species",
        shape = "Penguin species"))

Folks have heuristics to translate ggplot2 code to plotnine. These help understand the nuances between the two.

pyjanitor

OK, this one is cheating a bit because the janitor package by Sam Firke is not part of the tidyverse. One more package that uses ‘tidyverse-like’ verbs is pyjanitor. With pyjanitor, you can clean column names, identify duplicate entries, and more.

from janitor import clean_names
import pandas as pd
import numpy as np

example_df = {
    'Terrible Name 1': ['name1', 'name2', 'name3', 'name4'],
    'PascalCase': [150.0, 200.0, 300.0, 400.0],
    'this_has_punctuation?': [np.nan, np.nan, np.nan, np.nan],
}

pd.DataFrame.from_dict(example_df).clean_names()

Python’s pandas allows you to method chain with pipes. They can be used with pyjanitor, as well.

from janitor import clean_names, remove_empty
import pandas as pd
import numpy as np

example_df = {
    'Terrible Name 1': ['name1', 'name2', 'name3', 'name4'],
    'PascalCase': [150.0, 200.0, 300.0, 400.0],
    'this_has_punctuation?': [np.nan, np.nan, np.nan, np.nan],
}

(pd.DataFrame.from_dict(example_df)
    .pipe(clean_names)
    .pipe(remove_empty)
)

Conclusion

While Python syntax and conventions are still on my “to-learn” list, it is helpful to know there are packages that can bring the familiarity of the tidyverse to Python.

How laziness sparked this post

When I am feeling lazy, I use base R for quick plots:

plot(mtcars$hp, mtcars$mpg)

Because that clearly saves a lot of time compared to the {ggplot2} alternative 😄:

library(ggplot2)
library(magrittr)

mtcars %>%
  ggplot(aes(x = hp, y = mpg)) +
  geom_point()

One day, I was feeling extra lazy, so I tried using the new native R pipe |>:

mtcars |> plot(hp, mpg)
#> Error in get(as.character(FUN), mode = "function", envir = envir) :
#> object 'mpg' of mode 'function' was not found

Oh no! Why an error?

It was a complete misunderstanding of the native R pipe. I thought piping the data with the native R pipe would replace the need for the data$column notation. Turns out, that is not what the native R pipe does at all. So, this attempt to be lazy culminated in an adventuRe to figure out why that doesn’t work and what works instead.

How the {magrittr} pipe %>% works

First, let’s examine the pipe that you might be more familiar with (if you use the {tidyverse} dialect): the %>% forward pipe from the {magrittr} package.

Without any dot syntax (.), the {magrittr} pipe %>% is an infix operator that pipes (moves) what is written on the left-hand side (LHS) of the pipe into the first argument of the function on the right-hand side (RHS) of the pipe. Since R is prone to expressions with many nested parentheses, piping allows one to reason about code from left to right (as when writing in English), instead of right to left with many nested parentheses (see example below).

From the {magrittr} documentation, the pipe’s usage is thus: LHS %>% RHS. It might be easier to think of a ‘pipe’ as one of the famous ‘warp pipes’ from Super Mario Bros. — it warps Mario from the LHS into the first argument of the RHS function!

So:

mtcars2 <-
  mtcars %>%
  dplyr::mutate(hi_mpg = dplyr::if_else(mpg > 25, "high", "low"))

Is equivalent to:

mtcars2 <-
  dplyr::mutate(mtcars, hi_mpg = dplyr::if_else(mpg > 25, "high", "low"))

And,

mtcars %>% plot(hp, mpg)

Is equivalent to:

plot(mtcars, hp, mpg)

# or, more explicitly
plot(x = mtcars, y = hp, type = mpg)

Which does not work and gives us an error message, because the first two arguments of plot() should be objects for the x and y axes (mtcars is an odd x-axis but technically works), and the third argument is for the type of plot (mpg definitely doesn’t work there).

If you want the LHS to be passed somewhere other than the first argument with the {magrittr} pipe, you can use its dot syntax (.): y %>% f(x, .) is equivalent to f(x, y).

mtcars %>%
  lm(mpg ~ hp, data = .)

Is equivalent to:

lm(mpg ~ hp, data = mtcars)

So, how do we get the {magrittr} pipe to work with plot()? We can use the dot syntax as a placeholder for the data frame. However, this also does not work:

mtcars %>% plot(.$hp, .$mpg)
#> Error in match.fun(panel) :
#> '.$mpg' is not a function, character or symbol

Why? The error gives a hint that something is wrong with the .$mpg step.¹ By default, the {magrittr} pipe passes the LHS into the first argument of the RHS, so the call giving the error above is equivalent to writing:

plot(mtcars, mtcars$hp, mtcars$mpg)

# or, more explicitly
plot(x = mtcars, y = mtcars$hp, type = mtcars$mpg)

Which does not work nor is what we intended (we want to have mtcars$hp on the x-axis, and mtcars$mpg on the y-axis). The way to get the {magrittr} pipe to do what we want with plot() is to use its curly brace {} syntax. By wrapping the RHS in curly braces, we can override the rule where the LHS is passed to the first argument:

mtcars %>% {plot(.$hp, .$mpg)}

This works! It’s equivalent to having written:

plot(mtcars$hp, mtcars$mpg)

OK! Now, we can apply what we’ve learned to the native R pipe. Right? …Right?

ACTUALLY I THINK WE NEED A DETOUR TO EXPLAIN ANONYMOUS (LAMBDA) FUNCTIONS BEFORE WE CAN EXPLAIN HOW |> WORKS

Why anonymous? Because they are not named functions from a package or written by you and stored in a function object. Anonymous functions are created on-the-fly, applied immediately, and don’t persist after they’ve been used: function(x) {}.

function(x) {
  x[which.max(x$mpg), ]
}

What does this do? It creates an anonymous function (also known as a lambda function) if it is not saved to an object.

Introduced in R 4.1, the shortcut for anonymous functions \(x) {} is the same as function(x) {}:

# equivalent to the above
\(x) {
  x[which.max(x$mpg), ]
}

What you gain from writing an anonymous function is that you get to direct traffic by explicitly stating the inputs and how they will be used in the function. Going back to our discussion of pipes, you get to direct exactly where the LHS of the pipe goes in the RHS.

How the native R pipe works

Like the {magrittr} pipe %>%, the native R pipe |> pipes the LHS into the first argument of the function on the RHS: LHS |> RHS.

You can write:

mtcars |> sqrt() |> head()

Which is equivalent to:

head(sqrt(mtcars))

So, the native R pipe pipes the LHS into the first argument of the function on the RHS (with an extra requirement of needing a function call on the RHS). But that’s all! If you want to do anything beyond piping the LHS into the first argument of the RHS function, then you need the special anonymous function syntax introduced above.

A gotcha here is that we also needed to write parentheses around the anonymous function, such that a pseudocode version of the above is mtcars |> (anonymous-function-definition)(). The reason for this is so that the second set of () properly points to the complex expression inside the first set of () as the function being called.²³

mtcars |> (\(x) {
   x[which.max(x$mpg), ]
})()

Thinking back to what we learned about the {magrittr} pipe %>%, you might be tempted to use the dot syntax (.). A final important note is that the dot syntax does not work with the native R pipe |> since the dot syntax is a feature of {magrittr} and not of base R. For example, this works:

mtcars %>% plot(.$hp)

But this doesn’t because there is no support for the dot syntax with the native R pipe:

mtcars |> plot(.$hp)
#> Error in pairs.default(data.matrix(x), ...) : object '.' not found

However, if you create an anonymous function, you can decide what the input argument names are, whether ., x, data, anything! So, if you are tied to the dot syntax in {magrittr}, you can ‘bootstrap’ your own dot syntax with \(.) {}.

In sum, the native R pipe does not support the dot syntax unless you explicitly define your own.

Getting to the solution

Finally, we get to the solution: to get the native R pipe to do what we wanted to do with plot(), we need to use an anonymous function and bootstrap the dot syntax (or any other argument name) for ourselves:

# verbosely
mtcars |> (function(.) plot(.$hp, .$mpg))()

# using the anonymous function shortcut, emulating the dot syntax
mtcars |> (\(.) plot(.$hp, .$mpg))()

# or if you prefer x to .
mtcars |> (\(x) plot(x$hp, x$mpg))()

# or if you prefer to be explicit with argument names
mtcars |> (\(data) plot(data$hp, data$mpg))()

That’s it! 🎉 It can be done, but the initial attempt to be lazy ended up taking much more time than originally expected. As the saying goes, ‘a pipe in time saves nine.’

mtcars |> with(plot(hp, mpg))

with(mtcars, plot(hp, mpg))

The true lazy way

This was an exploration of how the %>% and |> pipes work, but we also have another option! The {magrittr} exposition pipe %$% ‘exposes’ the names in the LHS to the RHS expression.

So, the winner for lazy {magrittr} users like myself is:

mtcars %$% plot(hp, mpg)

No dot syntax, no curly braces, no anonymous functions, no terminal function call, just a pipe and column names! It effectively emulates how plot() would work if it were a {tidyverse} function.

Really though, all pipe users are winners here! As shown in the alignment chart tweet, we have many options of doing what we want to do with R.

More resources

• {magrittr} forward pipe operator help documentation (or ?"%>%" in your R console)
• Forward pipe operator help documentation (or ?pipeOp in your R console)
• Functions chapter in Advanced R

Extra credit for those who made it this far

How would you do this with the proposed native R pipe-bind => syntax? Respond to my tweet about this blog post once you have the answer!

Acknowledgements

The author thanks @gvelasq for his comments on the first draft of this blog post, and all who replied to the Twitter thread with insights, alternatives, and feedback.

Install packages

First up is installing the R packages:

```{r}
# Install packages if not already installed
# install.packages(c("tidyverse", "reticulate"))
library(tidyverse)
library(reticulate)
```

In your project, create a folder called “image”. Save the image you would like to convert to a dataset in that folder. To recreate this paper’s dataset, go to this page and download the cartoon gorilla.

The paper mentions the ‘getpixel’ function. With a bit of digging, we find that is from the package pillow (a fork from a package called PIL). Like in R, we need to install the package. In an R Markdown document, I run the functions from {reticulate} below.

```{r}
use_python("/usr/local/bin/python")
# If you haven't installed Python, the line below will prompt you!
py_install("pillow")
```

Since {reticulate} is the package that allows you to call Python, if you do not have Python installed, then you will get a message (which you would reply Y to):

No non-system installation of Python could be found.
Would you like to download and install Miniconda?
Miniconda is an open source environment management system for Python.
See https://docs.conda.io/en/latest/miniconda.html for more details.

Create coordinates

This Stack Overflow thread was very helpful to determine what we need to do with the ‘getpixel’ function in pillow. Again, like in R, we need to call the functions we need but this time, we are doing it in a python chunk.

```{python}
import numpy as np
from PIL import Image
```

Then, we point to the image and create coordinates pixels for the outline of the cartoon. We can set a threshold level for which pixels to keep/discard.

```{python}
# Thanks to Bart Huntley for pointing out a typo previously in this chunk!
im = Image.open("/cloud/project/image/gorilla.jpg")
pixels = np.asarray(im)

# Set threshold level
threshold_level = 50

# Find coordinates of all pixels below threshold
coords = np.column_stack(np.where(pixels < threshold_level))
```

Bring back into R

This results in a NumPy ndarray called coords that contains the coordinates of the pixels of the outline. That’s great but… it’s in Python! How do we bring it back into R?

The {reticulate} Cheat Sheet was very helpful in figuring this out. The section “Python in R Markdown” shows that you can use the py object to access objects created in Python chunks from R chunks.

Knowing that, we can create an R object from coords using py. Next, we follow the steps outlined in the paper for the data preparation.

```{r}
coords <-
  as.data.frame(py$coords) %>%
  sample_n(1768) %>%
  mutate(bmi = V2 * 17 + 15,
         steps = 15000 - V1 * 15000/max(V1)) %>%
  mutate(randvar = rnorm(n(), mean = 0, sd = 10),
         randi = steps * (1 + randvar),
         gender = case_when(randi < median(steps) ~ "Female",
                            TRUE ~ "Male"))
```

If we’d like to see the distribution of data by male/female, we can use count().

```{r}
coords %>%
  count(gender)
```

Create plots

Now for the fun part - visualizing the plots (spoiler: one should always do this before starting an analysis)!

```{r}
coords %>%
  ggplot(aes(x = bmi,
              y = steps)) +
  geom_point() +
  theme_void() +
  xlim(0, 15000)
```

```{r}
coords %>%
  ggplot(aes(x = bmi,
             y = steps,
             color = gender)) +
  geom_point() +
  theme_void() +
  xlim(0, 15000)
```

With that, we’ve seamlessly gone from Python to R and created a dataset leveraging the power of both languages. Thanks to Professors Yanai and Lerche for their publication!

The awesomeness that is the R community

As I started the task of pulling the YAML information, I remembered a blogpost from Garrick Aden-Buie called Find, count and list tags in all blogdown posts (thanks, Garrick!). The post taught me about blogdown:::scan_yaml() which was exactly what I needed to get the information from all the YAML headers in the files.

The members of the R Community, including others at RStudio, are incredibly collaborative and helpful, and I hope to be able to contribute to others’ learning as well.

RStudio tools rock

I remember when my older brother introduced me to the tidyverse in graduate school after I had been struggling to understand how to program in R. I was amazed by its usability and functionality and it enabled me to learn more, learn quicker, and learn better. I’ve been fortunate enough to use RStudio tools for my work, like Shiny and shinyapps.io, and each time appreciated the thoughtful and deep work that RStudio does with its services and products.

How great is it that the blogdown team anticipated the need to look at a blog’s metadata and already created a function that scans the YAML?! And that we can create a blog using R? And doing so is excellently documented?

In addition, below you will see many different tidyverse packages (dplyr, here, tibble, tidyr) being used as well. The seamless integration of R packages, tools, and services makes projects like this one possible and easy.

So much to write about

If you check out the tags from the RStudio blog, you will see so many topics. APIs! BI tools! Package development! Interoperability! RStudio is working on so many things, open source and enterprise, and it is thrilling to be part of an organization that advocates for code-first development, thrives in a diverse and supportive community, and thinks creatively about what challenges to tackle next.

A new chapter in my RStudio journey

I am proud to join an organization that I’ve long admired and hope that my contributions in this realm of RStudio’s business will promote the uptake of amazing RStudio products, and the crucial work RStudio does in free open-source software, data science education, and other spaces as a Public Benefit Corporation. I look forward to working alongside my colleagues at RStudio to achieve the company’s vision.

And with that… I give you, the code!

The code

# Load libraries
library(tidyverse)

# Find the folder with the blog posts
blogdown_content_path <- here::here("content", "post")

# Scan yaml of all posts (make sure the blogdown project is open)
meta_list <-
  blogdown:::scan_yaml(blogdown_content_path)

# Create a data frame that pulls out the title, date, categories, and tags for each entry
meta_df <- tibble(
  title = map_chr(meta_list, "title"),
  date = map(meta_list, "date"),
  categories = map(meta_list, "categories"),
  tags = map(meta_list, "tags"))

meta_df_clean <-
  meta_df %>%
  # Put the fields with multiple options into one column called variable
  pivot_longer(cols = c("categories", "tags"),
               names_to = "variable",
               values_to = "value") %>%
  # Split out the fields with multiple options into individual rows
  unnest(cols = value) %>%
  # For each blog entry, give each category or tag a number
  group_by(title, variable) %>%
  mutate(id = row_number()) %>%
  # Pivot back to wide format
  pivot_wider(names_from = c("id", "variable"),
              names_glue = "{variable}_{id}",
              values_from = value)

Load packages

There are several packages needed for this walkthrough:

# To clean data
library(tidyverse)
library(lubridate)
library(janitor)

# To scrape data
library(rvest)
library(httr)
library(polite)

Scrape table

First, we save the web page with the table that we would like as url:

url <- "https://en.wikipedia.org/wiki/List_of_national_independence_days"

Next, we use polite::bow() to introduce ourselves to the host, Wikipedia. This reads the rules from robots.txt and makes sure we follow them. The object (url_bow in this case) is saved as an object of class polite.

url_bow <- polite::bow(url)
url_bow

Next, we actually ‘scrape’ (pull the content of) the web page using polite::scrape(). This needs an object of class polite (created with bow() from before).

Since we politely scraped the entire web page, we want to use {rvest} to specify what exact content we’d like to pull out. We can do this using html_nodes().

How do we know which node we want? There are probably other ways of doing this. As a Firefox and Mac user, I click Cmd + Shift + C which opens up the Developer Tools so that I can select a specific element from the web page. I hover over the table to determine what the HTML is called, in this case, table.wikitable.

This object is saved as an HTML table which is great, but a data frame would be preferable for analysis. So the final step is to use rvest::html_table() to read this table as something with which we can use tidyverse tools. The parameter fill = TRUE allows you to fill empty rows with NA.

ind_html <-
  polite::scrape(url_bow) %>%  # scrape web page
  rvest::html_nodes("table.wikitable") %>% # pull out specific table
  rvest::html_table(fill = TRUE) 

Flatten table

You will notice that ind_html is saved as a single object (a list) in which each element is a data frame. If we want to convert it to a flat data frame, we can specify that we want the content from only the first element [[1]]. We can then use janitor::clean_names() for nice, standardized column names.

ind_tab <- 
  ind_html[[1]] %>% 
  clean_names()

That’s it! Now we’ve “politely” scraped the Wikipedia table into an analysis-ready data frame.

Conclusion

Additional steps to clean the file can be found in my GitHub repo. After doing so, I submitted to #TidyTuesday by submitting an issue on their page. Then they approved and shared the dataset!

This was a very quick walkthrough. I recommend Ryo Nakagawara’s blog post on politely scraping websites, especially if you would like (1) more in-depth explanations of what {polite} does and (2) more complex scraping examples.

Which Wikipedia table will you analyze next?

Write a parametric equation

First up is figuring out the function you will use to create the plot. I decided to go with a parametric equation for my first #rtistry plot. A parametric equation of a curve expresses the coordinates of points of the curve as functions of a variable. This blog post explains the concept very clearly and also has various examples of wonderful parametric equations.

Why parametric equations? First, even simple equations can create beautiful symmetries. Second, they are easy to modify to find the perfect plot.

The simplest parametric equation uses cosine and sine to make the unit circle:

circleFun <- function(center = c(0, 0), diameter = 1, npoints = 100){
    r = diameter / 2
    tt <- seq(0,2*pi,length.out = npoints)
    xx <- center[1] + r * cos(tt)
    yy <- center[2] + r * sin(tt)
    return(data.frame(x = xx, y = yy))
}

dat <- 
  circleFun(c(1, -1), 2.3, npoints = 100)

ggplot(dat,aes(x, y)) +
  geom_path()

Let’s write a function to create a parametric equation. I based this equation on the aforementioned blog post equations. The parameters are:

• center: the center of the equation
• npoints: the number of points used to evaluate the function.
• c1, c2, c3, c4: the coefficients for the equation

genFun <- function(center = c(0, 0), npoints = 500, c1 = 2.5, c2 = -5, c3 = 4.28, c4 = 2.3){
    t <- seq(0, 2*pi, length.out = npoints)
    xx <- center[1] + c1*(sin(c2*t)*sin(c2*t))*(2^cos(cos(c3*c4*t)))
    yy <- center[2] + c1*sin(sin(c2*t))*(cos(c3*c4*t)*cos(c3*c4*t))
    a <- data.frame(x = xx, y = yy)
    
    return(a)
}

Playing around with the function, we see how the graph gets smoother with more points and how the shape changes with different coefficients.

dat <- 
  genFun(c(1,-1), npoints = 100)

ggplot(dat, aes(x, y)) +
  geom_path()

dat <- 
  genFun(c(1,-1), npoints = 500, c1 = 5, c2 = -3, c3 = 5, c4 = 2)

ggplot(dat, aes(x, y)) +
  geom_path()

Create variation within the graph

Now that we have a basic shape, let’s play around with different aspects of the graph:

• geoms
• Aesthetic specifications
• Repetition

dat <- 
  genFun(c(1,-1), npoints = 5000)

ggplot(dat, aes(x, y)) +
  geom_line()

set.seed(1234)

dat <- 
  genFun(c(1,-1), npoints = 500)

dat %>% 
  ggplot(aes(x, y)) +
  geom_point()

set.seed(1111)

dat <- 
  genFun(c(1,-1), npoints = 5000) %>% 
  mutate(rand_w = sample(n())/3000)

dat %>% 
  ggplot(aes(x, y)) +
  geom_point(size = dat$rand_w) +
  theme_void()

dat %>% 
  ggplot(aes(x, y)) +
  geom_point(size = dat$rand_w,
             shape = 8) +
  theme_void()

set.seed(1234)

dat <- 
  dat %>% 
  mutate(rand_o = sample(n())/5000)

dat %>% 
  ggplot(aes(x, y)) +
  geom_point(size = dat$rand_w,
             alpha = dat$rand_o) +
  theme_void()

set.seed(1234)

dat <- 
  dat %>% 
  mutate(rand_c = sample(n()))

dat %>% 
  ggplot(aes(x, y,  color = rand_c)) +
  geom_point(size = dat$rand_w,
             alpha = dat$rand_o) +
  scale_color_viridis(option = "magma") +
  dark_theme_void() +
  theme(legend.position = "none") # remove legend

Inverted geom defaults of fill and color/colour.
To change them back, use invert_geom_defaults().

genFun <- function(center = c(0, 0), npoints = 500, c1 = 2.5, c2 = -5, c3 = 4.28, c4 = 2.3, size_denom = 1, opacity_denom = 1, color_denom = 1){
    t <- seq(0, 2*pi, length.out = npoints)
    xx <- center[1] + c1*(sin(c2*t)*sin(c2*t))*(2^cos(cos(c3*c4*t)))
    yy <- center[2] + c1*sin(sin(c2*t))*(cos(c3*c4*t)*cos(c3*c4*t))
    rand_w <- sample(0:20, npoints, replace = TRUE)/size_denom
    rand_o <- sample(1:100, npoints, replace = TRUE)/opacity_denom
    rand_c <- sample(1:100, npoints, replace = TRUE)/color_denom
    a <- data.frame(x = xx, y = yy, rand_w = rand_w, rand_o = rand_o, rand_c = rand_c)
    
    return(a)
}

set.seed(1111)

dat <- 
  genFun(c(0, 0), npoints = 5000, c1 = 5, c2 = -3, c3 = 5, c4 = 2, size_denom = 1.5, opacity_denom = 50)

dat %>% 
  ggplot(aes(x, y,  color = rand_c)) +
  geom_point(size = dat$rand_w,
             alpha = dat$rand_o) +
  scale_color_viridis(option = "magma") +
  dark_theme_void() +
  theme(legend.position = "none") # remove legend

dat %>% 
  ggplot() +
  geom_point(aes(x, y,  color = rand_c),
             size = dat$rand_w,
             alpha = dat$rand_o) +
  geom_point(aes(-x, -y,  color = rand_c),
             size = dat$rand_w,
             alpha = dat$rand_o) +
  geom_point(aes(-y, x,  color = rand_c),
             size = dat$rand_w,
             alpha = dat$rand_o) +
  geom_point(aes(-y, -x,  color = rand_c),
             size = dat$rand_w,
             alpha = dat$rand_o) +
  scale_color_viridis(option = "magma") +
  dark_theme_void() +
  theme(legend.position = "none") # remove legend

# ggsave(here::here("public", "img", "rtistry2.png"), dpi = 320, height = 6, width = 8) # this saves the image

Conclusion

This was my attempt to create generative art in R! It was fun to figure out how on earth to even get started and see how the plots change with new parameters on different attempts. I welcome any thoughts and please share your art using the #rtistry hashtag with me!

Setup

To be able to run this code, be sure to have the tidyverse installed. The {wesanderson} package contains beautiful palettes for visualizations.

# Load required packages

library(tidyverse)
library(wesanderson)

Streamgraphs

This post includes three of David Sjöberg’s amazing geoms; he created {ggsankey}, {ggstream}, AND {ggbump}. If you haven’t seen his GitHub, please check it out now.

This first geom, geom_stream(), creates a streamplot (which I’ve also seen called stream graphs). The streamplot is an area graph that usually centers around a central axis and allows us to see large fluctuations over time. More information on streamplot can be found here.

{ggstream} also has other options available to customize the streamgraphs, such as creating an area chart. Check out the repo here.

# remotes::install_github("davidsjoberg/ggstream")
library(ggstream)

ggplot(blockbusters, aes(year, box_office, fill = genre)) +
  geom_stream() +
  scale_fill_manual(values = wes_palette("Darjeeling2")) +
  theme_minimal()

Ridgeline plots

The {ggridges} package by Claus O. Wilke package also has a variety of geoms; check out the repo here. Ridgeline plots show the distribution of a numeric value for different groups and can look like mountain ranges. The R-Ladies Seattle hex sticker was created using ridgelines (very appropriate for the mountainous Washington!).

# install.packages("ggridges")
library(ggridges)
 
ggplot(blockbusters, aes(x = box_office, y = genre, fill = genre)) +
  geom_density_ridges(scale = 4) +
  scale_fill_manual(values = wes_palette("Darjeeling2")) +
  theme_minimal()

Sankey diagrams

Another geom by David Sjöberg is geom_sankey(), repo here. This geom creates Sankey diagrams and alluvial plots, which show flow and transfers in a system or throughout time. These plots are VERY popular on the subreddit dataisbeautiful (check it out on Mondays to see some examples).

# devtools::install_github("davidsjoberg/ggsankey")
library(ggsankey)

example_dat <-
  mtcars %>%
  make_long(cyl, vs, am, gear, carb) # function in ggsankey to format data correctly

ggplot(example_dat,
       aes(x = x, 
               next_x = next_x, 
               node = node, 
               next_node = next_node,
               fill = factor(node))) +
   geom_sankey(flow.alpha = .6) +
  theme_minimal()

Another package for alluvial charts is {ggalluvial} by Jason Cory Brunson, with its repo here. The data can be in more familiar formats than what is required for {ggsankey}.

# install.packages("ggalluvial")
library(ggalluvial)

ggplot(as.data.frame(UCBAdmissions),
       aes(y = Freq, axis1 = Gender, axis2 = Dept)) +
  geom_alluvium(aes(fill = Admit), width = 1/12) +
  scale_fill_manual(values = wes_palette("Darjeeling2")) +
  theme_minimal()

Bump charts

One last one by David Sjöberg is the amazing {ggbump}, repo here. Bump plots help show change in rank over time.

# devtools::install_github("davidsjoberg/ggbump")
library(ggbump)

blockbusters2 <-
  blockbusters %>% 
  filter(genre %in% c("Action", "Comedy", "Drama")) %>% 
  group_by(year) %>% 
  mutate(rank = rank(box_office))

ggplot(blockbusters2, aes(year, rank, color = genre)) +
  geom_point(size = 7) +
  geom_bump() +
  scale_color_manual(values = wes_palette("Darjeeling2")) +
  theme_minimal()

Waffle charts

For waffle charts, which are handy visualizations that show completion or parts of a whole, there is hrbrmstr’s {waffle}. The repo is here. Check out the ability to bring in other {ggplot2} functions, like facet_wrap. {waffle} also allows you to create pictograms using geom_pictogram, which replaces the squares in the ‘waffle’ with pictures.

# install.packages("waffle", repos = "https://cinc.rud.is")
library(waffle)

ggplot(as_tibble(Titanic), aes(fill = Sex, values = n)) +
  geom_waffle(n_rows = 20, color = "white") +
  facet_wrap(~ Survived, ncol = 1)  +
  scale_fill_manual(values = wes_palette("Darjeeling2")) +
  theme_minimal()

Beeswarm charts

Beeswarm charts, similar to jitter plots in {ggplot2}, plot individual points showing distributions without allowing the points to overlap too much. Erik Clarke’s repo for {ggbeeswarm} is here.

# install.packages("ggbeeswarm")
library(ggbeeswarm)

ggplot(blockbusters, aes(x = genre, y = box_office, color = genre)) + 
  geom_quasirandom() +
  theme_minimal() +
  scale_color_manual(values = wes_palette("Darjeeling2")) +
  theme_minimal()

Mosaic charts

Mosaic charts are incredibly helpful when displaying proportions of (multiple) categories. The {ggmosaic} package by Haley Jeppson (repo here) uses geom_mosaic to create these visualizations.

# devtools::install_github("haleyjeppson/ggmosaic")
library(ggmosaic)

ggplot(as.data.frame(UCBAdmissions)) +
  geom_mosaic(aes(x = product(Admit, Dept), fill = Gender, weight = Freq)) +
  scale_fill_manual(values = wes_palette("Darjeeling2")) +
  theme_minimal()

Other geoms

I know there exist a ton of other geoms that work with {ggplot2} out there. Just as I was writing this blogpost, I discovered {gghilbertstrings}! What other gg packages or geoms do you know of? Let me know on Twitter and I’ll list them here!

Learning R Markdown and {bookdown}

The first place to start to learn {bookdown} is Yihui Xie’s bookdown: Authoring Books and Technical Documents with R Markdown. In addition to going into {bookdown}, it provides some information on R Markdown and how to use it as well.

If you are new to R Markdown, Yihui, J. J. Allaire, and Garrett Grolemund wrote a definitive guide, available here.

These books offer great guidance on {bookdown} and R Markdown, but below are some details if you want to get started with a minimal example of a bookdown.

---
title: "Test Book"
author: "Me"
site: bookdown::bookdown_site
output: bookdown::gitbook
---

# Welcome {-}

book_filename: "bookdown-demo"
delete_merged_file: true
new_session: yes

bookdown::gitbook

bookdown::serve_book()

Snooping on others’ GitHub files

When I do not know how to do something in R, I look at people who I want to emulate and snoop on their GitHub repository pages. It is so incredibly handy (and wonderful) that others post their code for all to see.

This is how we learned just how to even start using bookdown (copying people’s files, seeing what happened when we ran them), but also the creative ways that others went about customizing their books.

While there’s an archive of many books using {bookdown}, here are some specific works we used during our bookdown development:

• Forecasting: Principles and Practice by Rob J Hyndman and George Athanasopoulos (book, repo) for customizing CSS on a bookdown with custom fonts, colors, etc.
• Geocomputation with R by Robin Lovelace, Jakub Nowosad, and Jannes Muenchow (book, repo) to learn how to give figures labels.

We highly recommend seeing what others have done in the past and trying things out so your book reflects what you would like it to look like!

.book .book-summary ul.summary li a:hover {
  background-color: #ffffff;
  color: #ffbc49;
}

Caching code chunks

As mentioned in Yihui’s book, you can cache certain code chunks so they do not have to rerun each time you render the book. We found this very handy, as some complex models took some time to run.

Important to note though: if you use set.seed, you will need to include it in every code chunk; setting it in only one code chunk will not mean it is set for all code chunks.

Figuring out your publisher’s requirements

Creating a book using R/R Studio/{bookdown} is great because all of these tools are so versatile and we were able to use each one to meet our publisher requirements, minimize manual work, and reduce redundancies.

For example, here’s how {bookdown} helped us do what we had to do:

• Our manuscript had to be in Word: we were able to render the bookdown using Word.
• Our manuscript needed all tables and figures to be labeled: we were able to automate this using the .yml files.
• Our figures had to be of a certain size and dimension: we were also able to automate this using the .yml files.
• We needed to send all of our figures separately from the manuscript: they are all saved separately due to the rendering function of {bookdown}.
• We needed a bibliography: we were able to create a bibliography of works and packages, all automated with {bookdown}.

Because of R:

• All tables had to be in Word format: at the time, we hadn’t seen this from {gtsummary}, and we spent quite a bit of time trying to use {gt} for our tables before realizing that we wouldn’t be able to output using Word. We then switched to {sjPlot}, which got us exactly what we needed.

• Our bibliography had to be in APA format: thanks to a file that Josh had (saved here), all we had to do was save that file and add biblio-style: "apalike" to our index.Rmd file for everything to show up in the format we needed.

And, thanks to R Studio:

• We were able to spell-check files within the IDE!
• We used its handy search function to create the index.

So, we recommend gathering all of your publisher’s requirements and then mapping them to the different tools at your disposal. This saves a lot of time and headache when it’s finally time to send over the final manuscript.

Determining how to collaborate

Since my co-authors and I all knew GitHub, we decided to use GitHub to write the book. This was great because it was a transparent way of writing the book (non-authors added issues and contributed as they saw fit). We were able to go back in time if needed to fix errors or see what had been changed.

We had to decide how updating the book would work. Ultimately, we decided that one person would be in charge of rendering the book - this helped reduce any code conflicts. However, there may be other ways of working that would be better for you.

Conclusion

In conclusion, we chose to use {bookdown} to create a book manuscript because we wanted to be transparent in our work, leverage R and its many tools, and be as efficient as possible when writing a book across five people. Despite some technical lift (and a lot of trial and error), it eventually resulted in a smooth workflow for our group. Thanks to the many resources and code available out there, we were able to learn {bookdown} as we wrote. Thanks to {bookdown}, we output the final manuscript that resulted in our book!

The problem

Way back in the Before Time, my older brother and I worked together on an R package called {wizehiver}. To collaborate, we used many tools. We had an email thread (of over 35 emails!) that eventually became two email threads. We tried GitHub Issues, but we were in such close communication that the back-and-forth on issues was ineffectual. We also tried pair programming but since we lived in separate cities, we did it while one of us would share our screen on Skype. We wished there was a better, more hands-on, and immediate way to collaborate in a source code editor in real-time, similar to how seamless it is to work on Google Docs.

State of the code editors

The RStudio IDE is in our opinion the best IDE for R out there; however, live collaboration using RStudio’s Project Sharing feature is limited to those with a paid RStudio Server Pro license as of this writing. There are many source code editors out there, and notably Atom and Visual Studio (VS) Code both provide extensions for free, collaborative real-time editing.

We have been following, with increasing interest, the growing community of developers who have been focused on tools for R in VS Code. There are now several R packages, VS Code extensions, and a command-line R console available, as well as several tutorials dedicated to R in VS Code. Here we set out to see if VS Code could fill the particular gap we identified in our previous work together — the need for seamless remote pair programming. A few of our use cases are:

1. We wanted to collaborate on this blog post in a shared .Rmd file which we could edit and knit as we wrote. Ultimately we wrote most of this post on Google Docs before transferring it to a versioned .Rmd on GitHub for finishing touches.

2. We wanted to start collaborating on an open-source R package {mutagen} which will (someday) provide useful extensions to {dplyr}’s mutate():

So far, we only have a hex sticker — #hexdrivendevelopment in action!

Embarrassingly, our last commit to the package was in December 2019. While it would be convenient to blame the pandemic, it was a mix of being busy and other ‘hobby’ projects. However, having a synchronous way of collaborating—where we could see the package being built in action—would enable us to truly get the project underway. Before we begin the tutorial, first we have a public service announcement: if you would like to see Microsoft devote additional resources to support R in VS Code, please upvote R in this vscode-jupyter issue!

Installation

We’re going to use Homebrew to facilitate installation steps on macOS. We only provide instructions for macOS here, but we do provide links for macOS users who prefer to install applications using the point-and-click method, and for Linux and Windows users to find the correct binaries. We also indicate which steps are optional.

1. Open Applications > Utilities > Terminal.

2. Install Homebrew using the terminal command below, also provided on the Homebrew landing page. Paste this code into your terminal:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

1. (Optional) For our favorite free and open-source terminal on macOS, install iTerm2 by pasting this into your terminal:

brew install --cask iterm2

1. If you don’t have them already, you’ll need R and RStudio Desktop. We’re partial to the preview version of RStudio Desktop which has all the latest features. You can install (and update) them easily from your terminal:

brew install --cask r
brew tap homebrew/cask-versions
brew install --cask rstudio-preview

1. Install radian, a ‘21st century R console’ and the recommended R console for VS Code.

Update 2021-02-11: Thanks to the efforts of @jdblischak and @randy3k in this closed issue, radian can now be installed with conda-forge instead of only with pip. Most data science tutorials recommend using Python with conda environments, so we suspect that our readers will be more familiar with using conda than pip, as we are. If you do choose to use pip (or python -m pip), beware that pip should be used carefully inside conda environments.

The steps below first install Miniforge, which we prefer to Miniconda since it sets conda-forge as the default channel, then create an empty conda environment named r-console into which we install radian and jedi. jedi is an optional package that enables Python autocompletion using the {reticulate} R package. Paste this into your terminal:

brew install miniforge
conda create --name r-console
conda activate r-console
conda install radian
conda install jedi # for {reticulate} python autocompletion

1. In RStudio, install the required R packages for VS Code, namely {languageserver} from CRAN and the latest GitHub release of {vscDebugger}:

install.packages("languageserver")
remotes::install_github("ManuelHentschel/vscDebugger", ref = remotes::github_release())

For the tutorial, make sure that you have the following R packages installed, if you don’t already:

install.packages("devtools")
install.packages("dplyr")
install.packages("purrr")
install.packages("tibble")
install.packages("usethis")
install.packages("vctrs")

1. Install VS Code by pasting this into your terminal:

brew install --cask visual-studio-code

1. Open VS Code and install the following extensions from the Extensions gallery (Shift-Command-X):

1. R
2. R LSP Client
3. R Debugger
4. Live Share
5. Live Share Audio

Configuration

1. Make sure the r-console environment is still active in your terminal. If it is, your terminal prompt will look like this, or similar:

(r-console) ~ %

If not, then reactivate the r-console environment:

conda activate r-console

1. Type which radian in your terminal to display the path to the radian executable. Below is what it is on our local machines. Copy the path to your clipboard.

/usr/local/Caskroom/miniforge/base/envs/r-console/bin/radian

1. Determine whether bash or zsh is your default shell. If you don’t know, type this into your terminal:

echo $SHELL

Older Macs may still use bash, while newer Macs or those with older Macs who have created new user accounts since upgrading to macOS Catalina will likely run zsh, the new default macOS shell.

1. Let’s take VS Code for a spin! An useful feature to know about is that typing code <path/file.ext> in your terminal will open a new or existing file in VS Code at the path you specify. Many more details are available in the help page code -h. Depending on whether your default shell is bash or zsh, type code ~/.bashrc or code ~/.zshrc in your terminal to open your shell configuration file, and paste the path you copied in step 2. above to add an alias to radian and bind it to lowercase r:

alias r="/usr/local/Caskroom/miniforge/base/envs/r-console/bin/radian"

You can paste the alias anywhere in the file, but it is probably best to paste it at the very bottom, and avoid inserting it in any preexisting code blocks (e.g., conda initialize blocks) which may be overwritten. Save your configuration file and restart your terminal for the settings to take effect. Now all you need to open radian is to type r in any terminal (Terminal, iTerm2, RStudio, or VS Code)! Useful radian commands to know are:

q() # to exit R, same as with vanilla R and RStudio
;   # to enter shell mode, exit by pressing backspace on an empty line
~   # to enter {reticulate} python mode, exit by pressing backspace on an empty line

1. To configure your .Rprofile, type code ~/.Rprofile in your terminal, or if you are already set up with {usethis} in RStudio, run the R command usethis::edit_r_profile(). At a minimum, you will want to enable {rstudioapi} emulation in VS Code:

options(vsc.rstudioapi = TRUE)

1. To configure radian, type code ~/.radian_profile in your terminal which will open up a new blank editor and create a new file named .radian_profile in the home directory. This is our .radian_profile:

options(
    radian.insert_new_line = FALSE,
    radian.escape_key_map = list(
        list(key = "-", value = " <- "),
        list(key = "m", value = " %>% ")
    )
)

Save ~/.radian_profile and restart your terminal for the settings to take effect.

1. Now we’ll configure VS Code settings. Open VS Code, and navigate to settings.json by using the Command Palette (Shift-Command-P) and navigating to Preferences: Open Settings (JSON). This is our settings.json:

{
    // Liveshare: Prompt when receiving focus requests
    "liveshare.focusBehavior": "prompt",
    // R: Treat`names.like.this` as one word for selection
    "[r]": {
        "editor.wordSeparators": "`~!@#%$^&*()-=+[{]}\\|;:'\",<>/?"
    },
    // R: Use active terminal for all commands
    "r.alwaysUseActiveTerminal": true,
    // R: Use bracketed paste mode when sending code to console (radian)
    "r.bracketedPaste": true,
    // R: R or radian path for macOS
    "r.rterm.mac": "/usr/local/Caskroom/miniforge/base/envs/r-console/bin/radian",
    // R: Enable R session watcher (experimental)
    "r.sessionWatcher": true,
    // R: Remove hidden items when clearing R workspace
    "r.workspaceViewer.removeHiddenItems": true,
    // Telemetry: Disable Microsoft crash reporter
    "telemetry.enableCrashReporter": false,
    // Telemetry: Disable Microsoft telemetry
    "telemetry.enableTelemetry": false,
    // Terminal: Path to integrated shell on macOS
    "terminal.integrated.shell.osx": "/bin/zsh"
}

1. Below are some useful keyboard shortcuts for keybindings.json, found by using the Command Palette (Shift-Command-P) in VS Code and navigating to Preferences: Open Keyboard Shortcuts (JSON). This is our keybindings.json:

[
    {
        "description": "View: Show R Workspace",
        "key": "alt+r",
        "command": "workbench.view.extension.workspaceViewer"
    },
    {
        "description": "R: Create R Terminal",
        "key": "alt+`",
        "command": "r.createRTerm"
    },
    {
        "description": "R: Insert Assignment Operator",
        "key": "alt+-",
        "command": "type",
        "when": "editorLangId == r || editorLangId == rmd && editorTextFocus",
        "args": { "text": " <- " }
    },
    {
        "description": "R: Insert Pipe Operator",
        "key": "shift+cmd+m",
        "command": "type",
        "when": "editorLangId == r || editorLangId == rmd && editorTextFocus",
        "args": { "text": " %>% " }
    },
    {
        "description": "R: Insert Assignment Pipe Operator",
        "key": "shift+cmd+,",
        "command": "type",
        "when": "editorLangId == r || editorLangId == rmd && editorTextFocus",
        "args": { "text": " %<>% " }
    },
    {
        "description": "R: Insert Tee Pipe Operator",
        "key": "shift+cmd+.",
        "command": "type",
        "when": "editorLangId == r || editorLangId == rmd && editorTextFocus",
        "args": { "text": " %T>% " }
    },
    {
        "description": "R: Test Package",
        "key": "shift+cmd+8",
        "command": "r.test"
    },
    {
        "description": "R: Document",
        "key": "shift+cmd+9",
        "command": "r.document"
    },
    {
        "description": "R: Load All",
        "key": "shift+cmd+0",
        "command": "r.loadAll",
    }
]