rHydro2019: Task Automation

<h1> Staying up-to date:<br> 
 automating tasks from downloading data to reporting </h1>

<br>

<h3> Alex Hurley <br> 
<em>University of Birmingham</em> 
</h3>
<br>
<br>
.small[<svg style="height:0.8em;top:.04em;position:relative;fill:lightgrey;" viewBox="0 0 576 512"><path d="M488 312.7V456c0 13.3-10.7 24-24 24H348c-6.6 0-12-5.4-12-12V356c0-6.6-5.4-12-12-12h-72c-6.6 0-12 5.4-12 12v112c0 6.6-5.4 12-12 12H112c-13.3 0-24-10.7-24-24V312.7c0-3.6 1.6-7 4.4-9.3l188-154.8c4.4-3.6 10.8-3.6 15.3 0l188 154.8c2.7 2.3 4.3 5.7 4.3 9.3zm83.6-60.9L488 182.9V44.4c0-6.6-5.4-12-12-12h-56c-6.6 0-12 5.4-12 12V117l-89.5-73.7c-17.7-14.6-43.3-14.6-61 0L4.4 251.8c-5.1 4.2-5.8 11.8-1.6 16.9l25.5 31c4.2 5.1 11.8 5.8 16.9 1.6l235.2-193.7c4.4-3.6 10.8-3.6 15.3 0l235.2 193.7c5.1 4.2 12.7 3.5 16.9-1.6l25.5-31c4.2-5.2 3.4-12.7-1.7-16.9z"/></svg> [aglhurley.rbind.io](https://aglhurley.rbind.io)  
<svg style="height:0.8em;top:.04em;position:relative;fill:lightgrey;" viewBox="0 0 512 512"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg> [aglhurley](https://twitter.com/aglhurley)]

---

# Goals

- <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 320 512"><path d="M295.973 160H180.572L215.19 30.184C219.25 14.956 207.756 0 192 0H56C43.971 0 33.8 8.905 32.211 20.828l-31.996 240C-1.704 275.217 9.504 288 24.004 288h118.701L96.646 482.466C93.05 497.649 104.659 512 119.992 512c8.35 0 16.376-4.374 20.778-11.978l175.973-303.997c9.244-15.967-2.288-36.025-20.77-36.025z"/></svg> Discuss why and what to automate

- <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 512 512"><path d="M184.561 261.903c3.232 13.997-12.123 24.635-24.068 17.168l-40.736-25.455-50.867 81.402C55.606 356.273 70.96 384 96.012 384H148c6.627 0 12 5.373 12 12v40c0 6.627-5.373 12-12 12H96.115c-75.334 0-121.302-83.048-81.408-146.88l50.822-81.388-40.725-25.448c-12.081-7.547-8.966-25.961 4.879-29.158l110.237-25.45c8.611-1.988 17.201 3.381 19.189 11.99l25.452 110.237zm98.561-182.915l41.289 66.076-40.74 25.457c-12.051 7.528-9 25.953 4.879 29.158l110.237 25.45c8.672 1.999 17.215-3.438 19.189-11.99l25.45-110.237c3.197-13.844-11.99-24.719-24.068-17.168l-40.687 25.424-41.263-66.082c-37.521-60.033-125.209-60.171-162.816 0l-17.963 28.766c-3.51 5.62-1.8 13.021 3.82 16.533l33.919 21.195c5.62 3.512 13.024 1.803 16.536-3.817l17.961-28.743c12.712-20.341 41.973-19.676 54.257-.022zM497.288 301.12l-27.515-44.065c-3.511-5.623-10.916-7.334-16.538-3.821l-33.861 21.159c-5.62 3.512-7.33 10.915-3.818 16.536l27.564 44.112c13.257 21.211-2.057 48.96-27.136 48.96H320V336.02c0-14.213-17.242-21.383-27.313-11.313l-80 79.981c-6.249 6.248-6.249 16.379 0 22.627l80 79.989C302.689 517.308 320 510.3 320 495.989V448h95.88c75.274 0 121.335-82.997 81.408-146.88z"/></svg> Introduce task automation concepts

- <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 512 512"><path d="M448 0L320 96v62.06l-83.03 83.03c6.79 4.25 13.27 9.06 19.07 14.87 5.8 5.8 10.62 12.28 14.87 19.07L353.94 192H416l96-128-64-64zM128 278.59L10.92 395.67c-14.55 14.55-14.55 38.15 0 52.71l52.7 52.7c14.56 14.56 38.15 14.56 52.71 0L233.41 384c29.11-29.11 29.11-76.3 0-105.41s-76.3-29.11-105.41 0z"/></svg> Highlight tools for each

- <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 384 512"><path d="M272 428v28c0 10.449-6.68 19.334-16 22.629V488c0 13.255-10.745 24-24 24h-80c-13.255 0-24-10.745-24-24v-9.371c-9.32-3.295-16-12.18-16-22.629v-28c0-6.627 5.373-12 12-12h136c6.627 0 12 5.373 12 12zm-143.107-44c-9.907 0-18.826-6.078-22.376-15.327C67.697 267.541 16 277.731 16 176 16 78.803 94.805 0 192 0s176 78.803 176 176c0 101.731-51.697 91.541-90.516 192.673-3.55 9.249-12.47 15.327-22.376 15.327H128.893zM112 176c0-44.112 35.888-80 80-80 8.837 0 16-7.164 16-16s-7.163-16-16-16c-61.757 0-112 50.243-112 112 0 8.836 7.164 16 16 16s16-7.164 16-16z"/></svg> Use-case: Study Site Explorer with `R` and `TravisCI`
]

### Study Site Explorer?

.center[<img src="./img/report_concept.png" alt="concept of automated report: choose site, download data, generate report" width="450"/>]

---

# What and why?

- <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 640 512"><path d="M629.657 343.598L528.971 444.284c-9.373 9.372-24.568 9.372-33.941 0L394.343 343.598c-9.373-9.373-9.373-24.569 0-33.941l10.823-10.823c9.562-9.562 25.133-9.34 34.419.492L480 342.118V160H292.451a24.005 24.005 0 0 1-16.971-7.029l-16-16C244.361 121.851 255.069 96 276.451 96H520c13.255 0 24 10.745 24 24v222.118l40.416-42.792c9.285-9.831 24.856-10.054 34.419-.492l10.823 10.823c9.372 9.372 9.372 24.569-.001 33.941zm-265.138 15.431A23.999 23.999 0 0 0 347.548 352H160V169.881l40.416 42.792c9.286 9.831 24.856 10.054 34.419.491l10.822-10.822c9.373-9.373 9.373-24.569 0-33.941L144.971 67.716c-9.373-9.373-24.569-9.373-33.941 0L10.343 168.402c-9.373 9.373-9.373 24.569 0 33.941l10.822 10.822c9.562 9.562 25.133 9.34 34.419-.491L96 169.881V392c0 13.255 10.745 24 24 24h243.549c21.382 0 32.09-25.851 16.971-40.971l-16.001-16z"/></svg> recurring tasks: database updates, time series, ancillary data

- <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 384 512"><path d="M360 64c13.255 0 24-10.745 24-24V24c0-13.255-10.745-24-24-24H24C10.745 0 0 10.745 0 24v16c0 13.255 10.745 24 24 24 0 90.965 51.016 167.734 120.842 192C75.016 280.266 24 357.035 24 448c-13.255 0-24 10.745-24 24v16c0 13.255 10.745 24 24 24h336c13.255 0 24-10.745 24-24v-16c0-13.255-10.745-24-24-24 0-90.965-51.016-167.734-120.842-192C308.984 231.734 360 154.965 360 64zM192 208c-57.787 0-104-66.518-104-144h208c0 77.945-46.51 144-104 144z"/></svg> time-consuming tasks: QA + QC, updating/creating reports

.center[<img src="./img/schedule_concept.png" alt="concept of automated report: choose site, download data, generate report" width="450"/>]

---

# Concepts

### Storage

.left-column[
<br>
<br>
**local** <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 576 512"><path d="M528 0H48C21.5 0 0 21.5 0 48v320c0 26.5 21.5 48 48 48h192l-16 48h-72c-13.3 0-24 10.7-24 24s10.7 24 24 24h272c13.3 0 24-10.7 24-24s-10.7-24-24-24h-72l-16-48h192c26.5 0 48-21.5 48-48V48c0-26.5-21.5-48-48-48zm-16 352H64V64h448v288z"/></svg>:
<br>
<br>
<br>
<br>
*vs.*
<br>
<br>
<br>
<br>
**hosted** <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 640 512"><path d="M537.6 226.6c4.1-10.7 6.4-22.4 6.4-34.6 0-53-43-96-96-96-19.7 0-38.1 6-53.3 16.2C367 64.2 315.3 32 256 32c-88.4 0-160 71.6-160 160 0 2.7.1 5.4.2 8.1C40.2 219.8 0 273.2 0 336c0 79.5 64.5 144 144 144h368c70.7 0 128-57.3 128-128 0-61.9-44-113.6-102.4-125.4z"/></svg>:
]
--
.right-column[
- software / triggers, 
- routines (e.g. `R` scripts), 
- outputs  
are **on your computer**
<br>
<br>
<br>
<br>
<br>

- routines and outputs in a repository
- software on a virtual machine
- webservice schedules / triggers  (webhooks)    
**download** or view online]

---
# Concepts

### Execution / Trigger

.center[ <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 640 512"><path d="M629.657 343.598L528.971 444.284c-9.373 9.372-24.568 9.372-33.941 0L394.343 343.598c-9.373-9.373-9.373-24.569 0-33.941l10.823-10.823c9.562-9.562 25.133-9.34 34.419.492L480 342.118V160H292.451a24.005 24.005 0 0 1-16.971-7.029l-16-16C244.361 121.851 255.069 96 276.451 96H520c13.255 0 24 10.745 24 24v222.118l40.416-42.792c9.285-9.831 24.856-10.054 34.419-.492l10.823 10.823c9.372 9.372 9.372 24.569-.001 33.941zm-265.138 15.431A23.999 23.999 0 0 0 347.548 352H160V169.881l40.416 42.792c9.286 9.831 24.856 10.054 34.419.491l10.822-10.822c9.373-9.373 9.373-24.569 0-33.941L144.971 67.716c-9.373-9.373-24.569-9.373-33.941 0L10.343 168.402c-9.373 9.373-9.373 24.569 0 33.941l10.822 10.822c9.562 9.562 25.133 9.34 34.419-.491L96 169.881V392c0 13.255 10.745 24 24 24h243.549c21.382 0 32.09-25.851 16.971-40.971l-16.001-16z"/></svg> recurring + <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 512 512"><path d="M256 8C119 8 8 119 8 256s111 248 248 248 248-111 248-248S393 8 256 8zm57.1 350.1L224.9 294c-3.1-2.3-4.9-5.9-4.9-9.7V116c0-6.6 5.4-12 12-12h48c6.6 0 12 5.4 12 12v137.7l63.5 46.2c5.4 3.9 6.5 11.4 2.6 16.8l-28.2 38.8c-3.9 5.3-11.4 6.5-16.8 2.6z"/></svg> scheduled
<br> 
<br> 
<br> 
*vs.*  
<br>
<br>
<svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 320 512"><path d="M295.973 160H180.572L215.19 30.184C219.25 14.956 207.756 0 192 0H56C43.971 0 33.8 8.905 32.211 20.828l-31.996 240C-1.704 275.217 9.504 288 24.004 288h118.701L96.646 482.466C93.05 497.649 104.659 512 119.992 512c8.35 0 16.376-4.374 20.778-11.978l175.973-303.997c9.244-15.967-2.288-36.025-20.77-36.025z"/></svg> event-based (e.g. on file change)]

---
class: inverse, center, middle

# Tools

---

# Tools

---

### Scheduling (local)

.center[<img src="./img/taskscheduler_app.png" alt="concept of automated report: choose site, download data, generate report" height="410"/>]

---

### Scheduling (local)

- <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 512 512"><path d="M256 8C119 8 8 119 8 256s111 248 248 248 248-111 248-248S393 8 256 8zm57.1 350.1L224.9 294c-3.1-2.3-4.9-5.9-4.9-9.7V116c0-6.6 5.4-12 12-12h48c6.6 0 12 5.4 12 12v137.7l63.5 46.2c5.4 3.9 6.5 11.4 2.6 16.8l-28.2 38.8c-3.9 5.3-11.4 6.5-16.8 2.6z"/></svg> **cron** for recurring tasks 
- <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 448 512"><path d="M12 192h424c6.6 0 12 5.4 12 12v260c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V204c0-6.6 5.4-12 12-12zm436-44v-36c0-26.5-21.5-48-48-48h-48V12c0-6.6-5.4-12-12-12h-40c-6.6 0-12 5.4-12 12v52H160V12c0-6.6-5.4-12-12-12h-40c-6.6 0-12 5.4-12 12v52H48C21.5 64 0 85.5 0 112v36c0 6.6 5.4 12 12 12h424c6.6 0 12-5.4 12-12z"/></svg> **at** for one-of tasks

---

### Event-based

`rOpenSci` <svg style="height:0.8em;top:.04em;position:relative;fill:#FFDD00;" viewBox="0 0 512 512"><path d="M509.5 184.6L458.9 32.8C452.4 13.2 434.1 0 413.4 0H272v192h238.7c-.4-2.5-.4-5-1.2-7.4zM240 0H98.6c-20.7 0-39 13.2-45.5 32.8L2.5 184.6c-.8 2.4-.8 4.9-1.2 7.4H240V0zM0 224v240c0 26.5 21.5 48 48 48h416c26.5 0 48-21.5 48-48V224H0z"/></svg>  [`drake`](https://github.com/ropensci/drake):

- semi-automated workflow manager
- monitors individual units/sections of analyses pipeline
- updates on change,
- but only parts of pipeline that require re-running

---
layout: false
class: inverse, center, middle

# Study Site Explorer

---

# Study Site Explorer

.center[<img src="./img/report_concept.png" alt="concept of automated report: choose site, download data, generate report" width="350"/>]

- R Markdown report with:  
  + map  
  + annual Precipitation + Temperature  
  + 3D view of region
- Hosted online (collaborators can generate report)
- Triggered on file change (add new site)  
<br>

---

---

**Use parameterized report to define meta data!**

.left-column[<img src="./img/report.png" alt="concept of automated report: choose site, download data, generate report" width="150"/>]

```r
---
author: "rHydro Demonstrator"
date: '`r paste("generated at:", Sys.Date())`'
output: html_document
*params:
  location: "Mt St Helens"
  year:
    value: 2000
*title: "`r paste('Overview for:', params$location)`"
---
```
]

---

**Define area of interest and make map:**

```r
# specify area of interest
*aoi <- AOI::getAOI(clip = list(params$location,15 ,15), km = TRUE)

# pull and plot map
map <- OpenStreetMap::openmap(upperLeft = c(aoi@bbox[[4]], aoi@bbox[[1]]),
                       lowerRight = c(aoi@bbox[[2]], aoi@bbox[[3]]),
                       type = "osm",
                       minNumTiles = 12)
```

---

**Download climate data and plot:**

```r
daymet_data <- daymetr::download_daymet(lat = aoi@polygons[[1]]@labpt[2],
                                   lon = aoi@polygons[[1]]@labpt[1],
*                                  start = params$year,
*                                  end = params$year)

daymet_data$data <- dplyr::mutate(daymet_data$data,
                             tmean = (tmax..deg.c. + tmin..deg.c.)/2,
                             date = as.Date(paste(year, yday, sep = "-"),
                                            "%Y-%j"))

library(ggplot2)
ggplot(daymet_data$data,
       aes(x = date,
     y = prcp..mm.day.))  +
    
    geom_col(position = "dodge",
             color = "darkblue") +
    
    labs(x = "Date",
         y = "P (mm/day)",
*        title = paste0(params$location,": ",
*                       params$year,
                        " - Precipitation")) +
    theme_bw()
```

---

**3D-Viz of Site** ([full code available here](https://github.com/the-Hull/02_task_automation/))

```r
*ned_aoi <- aoi %>% HydroData::findNED() # National DEM

# convert to matrix for rayshader
ned <- matrix(raster::extract(ned_aoi$NED,
                              raster::extent(ned_aoi$NED),
                              buffer=1000),
             nrow=ncol(ned_aoi$NED),ncol=nrow(ned_aoi$NED))

overlay <- create_overlay(prcp_raster, ned_aoi$NED)

library(rayshader)
ned %>%
    sphere_shade(texture = "imhof1") %>%
    add_water(detect_water(ned), color="desert") %>%
      add_overlay(overlay, alphacolor = NULL, alphalayer = 0.8) %>%
    add_shadow(ray_shade(ned)) %>%
    add_shadow(ambient_shade(ned)) %>%
    plot_3d(heightmap = ned ,
            zscale = 1,
            # fov = 90,
            lineantialias = TRUE,
            theta = 15,
            phi = 85,
            zoom = 0.3)
render_snapshot()
rgl::rgl.close()
```

---
layout: true

# Study Site Explorer

---

**Generate report**

```r
' Render Location and Climate report
#'
#' @param location Character, location passed to AOI::getAOI()
#' @param year Character, Year in YYYY
#'
#' @return Returns nothing, but writes a file to the reports directory
#' @export
#'
#' @examples
*render_report = function(location, year) {
    # house keeping on names
    location_dir <- stringr::str_replace_all(location, pattern = "[.]", "-")
    location_dir <- stringr::str_replace_all(location_dir, pattern = "[ ]", "_")

*   rmarkdown::render(
*       "./templates/report_template.Rmd", <
*       params = list(
            location = location, 
            year = year 
        ),
       output_dir = "./reports", 
        output_file = paste0("Report-", location_dir, "-", year, ".html")
    )
}
```

---

---

---

---

### Generate report for several sites:

```r
## Script executes report generation for all locations listed in sites.txt
source("R/render_report.R")

sites <- read.csv("sites.csv", header = TRUE, stringsAsFactors = FALSE)
for(site in sites$sites){

render_report(site, year = 2010)
}
```

---

### Generate report for several sites:

```
+-- reports
|   +-- Report-Mt_Baldy-2010.html
|   +-- Report-Mt_St_Helens-2010.html
|   +-- Report-Grand_Teton_Mountain-2010.html
|   \-- Report-El_Capitan_Yosemite-2010.html
```
]

---