Disclaimer: The purpose of the Open Case Studies project is to demonstrate the use of various data science methods, tools, and software in the context of messy, real-world data. A given case study does not cover all aspects of the research process, is not claiming to be the most appropriate way to analyze a given data set, and should not be used in the context of making policy decisions without external consultation from scientific experts.

This work is licensed under the Creative Commons Attribution-NonCommercial 3.0 (CC BY-NC 3.0) United States License.

To cite this case study please use:

Wright, Carrie and Meng, Qier and Jager, Leah and Taub, Margaret and Hicks, Stephanie. (2020). https://github.com//opencasestudies/ocs-bp-air-pollution. Predicting Annual Air Pollution (Version v1.0.0).

To access the GitHub Repository for this case study see here: https://github.com/opencasestudies/ocs-bp-air-pollution.

You may also access and download the data using our OCSdata package. To learn more about this package including examples, see this link. Here is how you would install this package:

install.packages("OCSdata")

This case study is part of a series of public health case studies for the Bloomberg American Health Initiative.

The total reading time for this case study is calculated via koRpus and shown below:

Reading Time	Method
107 minutes	koRpus

Readability Score:

A readability index estimates the reading difficulty level of a particular text. Flesch-Kincaid, FORCAST, and SMOG are three common readability indices that were calculated for this case study via koRpus. These indices provide an estimation of the minimum reading level required to comprehend this case study by grade and age.

Text language: en

index	grade	age
Flesch-Kincaid	10	15
FORCAST	10	15
SMOG	13	18

Please help us by filling out our survey.

Motivation

A variety of different sources contribute different types of pollutants to what we call air pollution.

Some sources are natural while others are anthropogenic (human derived):

[source]

Major types of air pollutants

Gaseous - Carbon Monoxide (CO), Ozone (O₃), Nitrogen Oxides(NO, NO₂), Sulfur Dioxide (SO₂)
Particulate - small liquids and solids suspended in the air (includes lead- can include certain types of dust)
Dust - small solids (larger than particulates) that can be suspended in the air for some time but eventually settle
Biological - pollen, bacteria, viruses, mold spores

See here for more detail on the types of pollutants in the air.

Particulate pollution

Air pollution particulates are generally described by their size.

There are 3 major categories:

Large Coarse Particulate Matter - has diameter of >10 micrometers (10 µm)
Coarse Particulate Matter (called PM_10-2.5) - has diameter of between 2.5 µm and 10 µm
Fine Particulate Matter (called PM_2.5) - has diameter of < 2.5 µm

PM₁₀ includes any particulate matter <10 µm (both coarse and fine particulate matter)

Here you can see how these sizes compare with a human hair:

[source]

The following plot shows the relative sizes of these different pollutants in micrometers (µm):

[source]

This table shows how deeply some of the smaller fine particles can penetrate within the human body:

[source]

Negative impact of particulate exposure on health

Exposure to air pollution is associated with higher rates of mortality in older adults and is known to be a risk factor for many diseases and conditions including but not limited to:

Asthma - fine particle exposure (PM_2.5) was found to be associated with higher rates of asthma in children
Inflammation in type 1 diabetes - fine particle exposure (PM_2.5) from traffic-related air pollution was associated with increased measures of inflammatory markers in youths with Type 1 diabetes
Lung function and emphysema - higher concentrations of ozone (O₃), nitrogen oxides (NO_x), black carbon, and fine particle exposure PM_2.5 , at study baseline were significantly associated with greater increases in percent emphysema per 10 years
Low birthweight - fine particle exposure(PM_2.5) was associated with lower birth weight in full-term live births
Viral Infection - higher rates of infection and increased severity of infection are associated with higher exposures to pollution levels including fine particle exposure (PM_2.5)

See this review article for more information about sources of air pollution and the influence of air pollution on health.

Sparse monitoring is problematic for Public Health

Historically, epidemiological studies would assess the influence of air pollution on health outcomes by relying on a number of monitors located around the country.

However, as can be seen in the following figure, these monitors are relatively sparse in certain regions of the country and are not necessarily located near pollution sources. We will see later when we evaluate the data, that even in certain relatively large cities there is only one monitor!

Furthermore, dramatic differences in pollution rates can be seen even within the same city. In fact, the term micro-environments describes environments within cities or counties which may vary greatly from one block to another.

[source]

This lack of granularity in air pollution monitoring has hindered our ability to discern the full impact of air pollution on health and to identify at-risk locations.

Machine learning offers a solution

An article published in the Environmental Health journal dealt with this issue by using data, including population density and road density, among other features, to model or predict air pollution levels at a more localized scale using machine learning (ML) methods.

[source]

Yanosky, J. D. et al. Spatio-temporal modeling of particulate air pollution in the conterminous United States using geographic and meteorological predictors. Environ Health 13, 63 (2014).

The authors of this article state that:

“Exposure to atmospheric particulate matter (PM) remains an important public health concern, although it remains difficult to quantify accurately across large geographic areas with sufficiently high spatial resolution. Recent epidemiologic analyses have demonstrated the importance of spatially- and temporally-resolved exposure estimates, which show larger PM-mediated health effects as compared to nearest monitor or county-specific ambient concentrations.”

[source]

The article above demonstrates that machine learning methods can be used to predict air pollution levels when traditional monitoring systems are not available in a particular area or when there is not enough spatial granularity with current monitoring systems. We will use similar methods to predict annual air pollution levels spatially within the US.

Main Question

Our main question:

Can we predict annual average air pollution concentrations at the granularity of zip code regional levels using predictors such as data about population density, urbanization, road density, as well as, satellite pollution data and chemical modeling data?

Learning Objectives

In this case study, we will walk you through importing data from CSV files and performing machine learning methods to predict our outcome variable of interest (in this case annual fine particle air pollution estimates).

We will especially focus on using packages and functions from the tidyverse, and more specifically the tidymodels package/ecosystem primarily developed and maintained by Max Kuhn and Davis Vaughan. This package loads more modeling related packages like rsample, recipes, parsnip, yardstick, workflows, and tune packages.

The tidyverse is a library of packages created by RStudio. While some students may be familiar with previous R programming packages, these packages make data science in R especially legible and intuitive.

The skills, methods, and concepts that students will be familiar with by the end of this case study are:

Data Science Learning Objectives:

Familiarity with the tidymodels ecosystem.
Ability to evaluate correlation among predictor variables (corrplot and GGally).
Ability to implement tidymodels packages such as rsample to split the data into training and testing sets as well as cross validation sets.
Ability to use the recipes, parsnip, and workflows to train and test a linear regression model and random forest model.
Demonstrate how to visualize geo-spatial data using ggplot2.

Statistical Learning Objectives:

Basic understanding the utility of machine learning for prediction and classification
Understanding of the need for training and test sets
Understanding of the utility of cross validation
Understanding of random forest
How to interpret root mean squared error (rmse) to assess performance for prediction

We will begin by loading the packages that we will need:

# Load packages for data import and data wrangling
library(here)
library(readr)
library(dplyr)
library(skimr)
library(summarytools)
library(magrittr)
# Load packages for making correlation plots
library(corrplot)
library(RColorBrewer)
library(GGally)
# Load packages for building machine learning algorithm
library(tidymodels)
library(workflows)
library(vip)
library(tune)
library(randomForest)
library(doParallel)
# Load packages for data visualization/creating map
library(ggplot2)
library(stringr)
library(tidyr)
library(lwgeom)
library(proxy)# needed for lwgeom
library(sf)
library(maps)
library(rnaturalearth)
library(rnaturalearthdata) # needed for rnaturalearth
library(rgeos)
library(patchwork)
# Load package for downloading the case study data files
library(OCSdata)

Packages used in this case study:

Package	Use in this case study
here	to easily load and save data
readr	to import CSV files
dplyr	to view/arrange/filter/select/compare specific subsets of data
skimr	to get an overview of data
summarytools	to get an overview of data in a different style
magrittr	to use the `%<>%` pipping operator
corrplot	to make large correlation plots
GGally	to make smaller correlation plots
tidymodels	to load in a set of packages (broom, dials, infer, parsnip, purrr, recipes, rsample, tibble, yardstick)
rsample	to split the data into testing and training sets; to split the training set for cross-validation
recipes	to pre-process data for modeling in a tidy and reproducible way and to extract pre-processed data (major functions are `recipe()`, `prep()` and various transformation `step_*()` functions, as well as `bake` which extracts pre-processed training data (used to require `juice()`) and applies recipe preprocessing steps to testing data). See here for more info.
parsnip	an interface to create models (major functions are `fit()`, `set_engine()`)
yardstick	to evaluate the performance of models
broom	to get tidy output for our model fit and performance
ggplot2	to make visualizations with multiple layers
dials	to specify hyper-parameter tuning
tune	to perform cross validation, tune hyper-parameters, and get performance metrics
workflows	to create modeling workflow to streamline the modeling process
vip	to create variable importance plots
randomForest	to perform the random forest analysis
doParallel	to fit cross validation samples in parallel
stringr	to manipulate the text the map data
tidyr	to separate data within a column into multiple columns
rnaturalearth	to get the geometry data for the earth to plot the US
maps	to get map database data about counties to draw them on our US map
sf	to convert the map data into a data frame
lwgeom	to use the `sf` function to convert map geographical data
rgeos	to use geometry data
patchwork	to allow plots to be combined
OCSdata	to access and download OCS data files

The first time we use a function, we will use the :: to indicate which package we are using. Unless we have overlapping function names, this is not necessary, but we will include it here to be informative about where the functions we will use come from.

Context

The State of Global Air is a report released every year to communicate the impact of air pollution on public health.

The State of Global Air 2019 report which uses data from 2017 stated that:

Air pollution is the fifth leading risk factor for mortality worldwide. It is responsible for more deaths than many better-known risk factors such as malnutrition, alcohol use, and physical inactivity. Each year, more people die from air pollution–related disease than from road traffic injuries or malaria.

[source]

The report also stated that:

In 2017, air pollution is estimated to have contributed to close to 5 million deaths globally — nearly 1 in every 10 deaths.

[source]

The State of Global Air 2018 report using data from 2016 which separated different types of air pollution, found that particulate pollution was particularly associated with mortality.

[source]

The 2019 report shows that the highest levels of fine particulate pollution occur in Africa and Asia and that:

More than 90% of people worldwide live in areas exceeding the World Health Organization (WHO) Guideline for healthy air. More than half live in areas that do not even meet WHO’s least-stringent air quality target.

[source]

Looking at the US specifically, air pollution levels are generally improving, with declining national air pollutant concentration averages as shown from the 2019 Our Nation’s Air report from the US Environmental Protection Agency (EPA):

[source]

However, air pollution continues to contribute to health risk for Americans, in particular in regions with higher than national average rates of pollution that, at times, exceed the WHO’s recommended level. Thus, it is important to obtain high spatial granularity in estimates of air pollution in order to identify locations where populations are experiencing harmful levels of exposure.

You can see that current air quality conditions at this website, and you will notice variation across different cities.

For example, here are the conditions in Topeka Kansas at the time this case study was created:

[source]

It reports particulate values using what is called the Air Quality Index (AQI). This calculator indicates that 114 AQI is equivalent to 40.7 ug/m³ and is considered unhealthy for sensitive individuals. Thus, some areas exceed the WHO annual exposure guideline (10 ug/m³), and this may adversely affect the health of people living in these locations.

Adverse health effects have been associated with populations experiencing higher pollution exposure despite the levels being below suggested guidelines. Also, it appears that the composition of the particulate matter and the influence of other demographic factors may make specific populations more at risk for adverse health effects due to air pollution. For example, see this article for more details.

The monitor data that we will use in this case study come from a system of monitors in which roughly 90% are located within cities. Hence, there is an equity issue in terms of capturing the air pollution levels of more rural areas. To get a better sense of the pollution exposures for the individuals living in these areas, methods like machine learning can be useful to estimate air pollution levels in areas with little to no monitoring. Specifically, these methods can be used to estimate air pollution in these low monitoring areas so that we can make a map like this where we have annual estimates for all of the contiguous US:

[source]

This is what we aim to achieve in this case study.

Limitations

There are some important considerations regarding the data analysis in this case study to keep in mind:

The data do not include information about the composition of particulate matter. Different types of particulates may be more benign or deleterious for health outcomes.
Outdoor pollution levels are not necessarily an indication of individual exposures. People spend differing amounts of time indoors and outdoors and are exposed to different pollution levels indoors. Researchers are now developing personal monitoring systems to track air pollution levels on the personal level.
Our analysis will use annual mean estimates of pollution levels, but these can vary greatly by season, day and even hour. There are data sources that have finer levels of temporal data; however, we are interested in long term exposures, as these appear to be the most influential for health outcomes.

What are the data?

We are going to perform a type of machine learning to try to predict air pollution levels that is called supervised machine learning. This type of machine learning requires that we have an outcome or result of real values (of in our case air pollution) to work with that will guide or supervise our work. This will ultimately allow us to try to predict air pollution values in the future when we don’t have them. For more explanation of what supervised machine learning is (and how it compares to a different kind of machine learning called unsupervised machine learning where we don’t have outcome values) refer to this),

When using supervised machine learning for prediction,there are two main types of data of interest:

A continuous outcome variable that we want to predict
A set of feature(s) (or predictor variables) that we use to predict the outcome variable

The outcome variable is what we are trying to predict. To build (or train) our model, we use both the outcome and features. The goal is to identify informative features that can explain a large amount of variation in our outcome variable. Using this model, we can then predict the outcome from new observations with the same features where have not observed the outcome.

As a simple example, imagine that we have data about the sales and characteristics of cars from last year and we want to predict which cars might sell well this year. We do not have the sales data yet for this year, but we do know the characteristics of our cars for this year. We can build a model of the characteristics that explained sales last year to estimate what cars might sell well this year. In this case, our outcome variable is the sales of cars, while the different characteristics of the cars make up our features.

Start with a question

This is the most commonly missed step when developing a machine learning algorithm. Machine learning can very easily be turned into an engineering problem. Just dump the outcome and the features into a black box algorithm and viola! But this kind of thinking can lead to major problems. In general good machine learning questions:

Have a plausible explanation for why the features predict the outcome.
Consider potential variation in both the features and the outcome over time.
Are consistently re-evaluated on criteria 1 and 2 over time.

In this case study, we want to predict air pollution levels. To build this machine learning algorithm, our outcome variable is average annual fine particulate matter (PM_2.5) captured from air pollution monitors in the contiguous US in 2008. Our features (or predictor variables) include data about population density, road density, urbanization levels, and NASA satellite data.

Our outcome variable

The monitor data that we will be using comes from gravimetric monitors (see picture below) operated by the US Environmental Protection Agency (EPA).

[image curtesy of Kirsten Koehler]

These monitors use a filtration system to specifically capture fine particulate matter.

[source]

The weight of this particulate matter is manually measured daily or weekly. For the EPA standard operating procedure for PM gravimetric analysis in 2008, we refer the reader to here.

For more on Gravimetric analysis, you can expand here

Gravimetric analysis is also used for emission testing. The same idea applies: a fresh filter is applied and the desired amount of time passes, then the filter is removed and weighed.

There are other monitoring systems that can provide hourly measurements, but we will not be using data from these monitors in our analysis. Gravimetric analysis is considered to be among the most accurate methods for measuring particulate matter.

In our data set, the value column indicates the PM_2.5 monitor average for 2008 in mass of fine particles/volume of air for 876 gravimetric monitors. The units are micrograms of fine particulate matter (PM) that is less than 2.5 micrometers in diameter per cubic meter of air - mass concentration (ug/m³). Recall the WHO exposure guideline is < 10 ug/m³ on average annually for PM_2.5.

Our features (predictor variables)

There are 48 features with values for each of the 876 monitors (observations). The data comes from the US Environmental Protection Agency (EPA), the National Aeronautics and Space Administration (NASA), the US Census, and the National Center for Health Statistics (NCHS).

Click here to see a table about the set of features

Variable	Details
id	Monitor number – the county number is indicated before the decimal – the monitor number is indicated after the decimal Example: 1073.0023 is Jefferson county (1073) and .0023 one of 8 monitors
fips	Federal information processing standard number for the county where the monitor is located – 5 digit id code for counties (zero is often the first value and sometimes is not shown) – the first 2 numbers indicate the state – the last three numbers indicate the county Example: Alabama’s state code is 01 because it is first alphabetically (note: Alaska and Hawaii are not included because they are not part of the contiguous US)
Lat	Latitude of the monitor in degrees
Lon	Longitude of the monitor in degrees
state	State where the monitor is located
county	County where the monitor is located
city	City where the monitor is located
CMAQ	Estimated values of air pollution from a computational model called Community Multiscale Air Quality (CMAQ) – A monitoring system that simulates the physics of the atmosphere using chemistry and weather data to predict the air pollution – *Does not use any of the PM_2.5 gravimetric monitoring data.* (There is a version that does use the gravimetric monitoring data, but not this one!) – Data from the EPA
zcta	Zip Code Tabulation Area where the monitor is located – Postal Zip codes are converted into “generalized areal representations” that are non-overlapping – Data from the 2010 Census
zcta_area	Land area of the zip code area in meters squared – Data from the 2010 Census
zcta_pop	Population in the zip code area – Data from the 2010 Census
imp_a500	Impervious surface measure – Within a circle with a radius of 500 meters around the monitor – Impervious surface are roads, concrete, parking lots, buildings – This is a measure of development
imp_a1000	Impervious surface measure – Within a circle with a radius of 1000 meters around the monitor
imp_a5000	Impervious surface measure – Within a circle with a radius of 5000 meters around the monitor
imp_a10000	Impervious surface measure – Within a circle with a radius of 10000 meters around the monitor
imp_a15000	Impervious surface measure – Within a circle with a radius of 15000 meters around the monitor
county_area	Land area of the county of the monitor in meters squared
county_pop	Population of the county of the monitor
Log_dist_to_prisec	Log (Natural log) distance to a primary or secondary road from the monitor – Highway or major road
log_pri_length_5000	Count of primary road length in meters in a circle with a radius of 5000 meters around the monitor (Natural log) – Highways only
log_pri_length_10000	Count of primary road length in meters in a circle with a radius of 10000 meters around the monitor (Natural log) – Highways only
log_pri_length_15000	Count of primary road length in meters in a circle with a radius of 15000 meters around the monitor (Natural log) – Highways only
log_pri_length_25000	Count of primary road length in meters in a circle with a radius of 25000 meters around the monitor (Natural log) – Highways only
log_prisec_length_500	Count of primary and secondary road length in meters in a circle with a radius of 500 meters around the monitor (Natural log) – Highway and secondary roads
log_prisec_length_1000	Count of primary and secondary road length in meters in a circle with a radius of 1000 meters around the monitor (Natural log) – Highway and secondary roads
log_prisec_length_5000	Count of primary and secondary road length in meters in a circle with a radius of 5000 meters around the monitor (Natural log) – Highway and secondary roads
log_prisec_length_10000	Count of primary and secondary road length in meters in a circle with a radius of 10000 meters around the monitor (Natural log) – Highway and secondary roads
log_prisec_length_15000	Count of primary and secondary road length in meters in a circle with a radius of 15000 meters around the monitor (Natural log) – Highway and secondary roads
log_prisec_length_25000	Count of primary and secondary road length in meters in a circle with a radius of 25000 meters around the monitor (Natural log) – Highway and secondary roads
log_nei_2008_pm25_sum_10000	Tons of emissions from major sources data base (annual data) sum of all sources within a circle with a radius of 10000 meters of distance around the monitor (Natural log)
log_nei_2008_pm25_sum_15000	Tons of emissions from major sources data base (annual data) sum of all sources within a circle with a radius of 15000 meters of distance around the monitor (Natural log)
log_nei_2008_pm25_sum_25000	Tons of emissions from major sources data base (annual data) sum of all sources within a circle with a radius of 25000 meters of distance around the monitor (Natural log)
log_nei_2008_pm10_sum_10000	Tons of emissions from major sources data base (annual data) sum of all sources within a circle with a radius of 10000 meters of distance around the monitor (Natural log)
log_nei_2008_pm10_sum_15000	Tons of emissions from major sources data base (annual data) sum of all sources within a circle with a radius of 15000 meters of distance around the monitor (Natural log)
log_nei_2008_pm10_sum_25000	Tons of emissions from major sources data base (annual data) sum of all sources within a circle with a radius of 25000 meters of distance around the monitor (Natural log)
popdens_county	Population density (number of people per kilometer squared area of the county)
popdens_zcta	Population density (number of people per kilometer squared area of zcta)
nohs	Percentage of people in zcta area where the monitor is that do not have a high school degree – Data from the Census
somehs	Percentage of people in zcta area where the monitor whose highest formal educational attainment was some high school education – Data from the Census
hs	Percentage of people in zcta area where the monitor whose highest formal educational attainment was completing a high school degree – Data from the Census
somecollege	Percentage of people in zcta area where the monitor whose highest formal educational attainment was completing some college education – Data from the Census
associate	Percentage of people in zcta area where the monitor whose highest formal educational attainment was completing an associate degree – Data from the Census
bachelor	Percentage of people in zcta area where the monitor whose highest formal educational attainment was a bachelor’s degree – Data from the Census
grad	Percentage of people in zcta area where the monitor whose highest formal educational attainment was a graduate degree – Data from the Census
pov	Percentage of people in zcta area where the monitor is that lived in poverty in 2008 - or would it have been 2007 guidelines??https://aspe.hhs.gov/2007-hhs-poverty-guidelines – Data from the Census
hs_orless	Percentage of people in zcta area where the monitor whose highest formal educational attainment was a high school degree or less (sum of nohs, somehs, and hs)
urc2013	2013 Urban-rural classification of the county where the monitor is located – 6 category variable - 1 is totally urban 6 is completely rural – Data from the National Center for Health Statistics
urc2006	2006 Urban-rural classification of the county where the monitor is located – 6 category variable - 1 is totally urban 6 is completely rural – Data from the National Center for Health Statistics
aod	Aerosol Optical Depth measurement from a NASA satellite – based on the diffraction of a laser – used as a proxy of particulate pollution – unit-less - higher value indicates more pollution – Data from NASA

Many of these features have to do with the circular area around the monitor called the “buffer”. These are illustrated in the following figure:

[source]

Data Import

All of our data was previously collected by a researcher at the Johns Hopkins School of Public Health who studies air pollution and climate change.

We have one CSV file that contains both our single outcome variable and all of our features (or predictor variables). You can download this file using the OCSdata package:

# install.packages("OCSdata")
OCSdata::raw_data("ocs-bp-air-pollution", outpath = getwd())

If you have trouble using the package, you can also find this data on our GitHub repository. Or you can download it more directly by clicking here.

We have created a “raw” subdirectory within a directory called “data” of our working directory of our RStudio project.

Next, we import our data into R now so that we can explore the data further. We will call our data object pm for particulate matter. We import the data using the read_csv() function from the readr package.

We will use the here package to make it easier to find the data file.

Click here to see more about creating new projects in RStudio.

You can create a project by going to the File menu of RStudio like so:

You can also do so by clicking the project button:

See here to learn more about using RStudio projects and here to learn more about the here package.

pm <- readr::read_csv(here("data","raw", "pm25_data.csv"))

We will save this data as an rda file for later in an “imported” subdirectory of the data directory.

save(pm, file = here::here("data", "imported", "imported_pm.rda"))

Data Exploration and Wrangling

If you are following along but stopped, you could start here by first loading the data like so:

load(here::here("data", "imported", "imported_pm.rda"))

If you skipped the data import section click here.

First you need to install the OCSdata package:

install.packages("OCSdata")

Then, you may download and load the imported data .rda file using the following code:

OCSdata::imported_data("ocs-bp-air-pollution", outpath = getwd())
load(here::here("OCSdata", "data", "imported", "imported_pm.rda"))

If the package does not work for you, an RDA file (stands for R data) of the data can be found on our GitHub repository. Or you can download it more directly by clicking here.

To load the downloaded data into your environment, you may double click on the .rda file in Rstudio or using the load() function.

To copy and paste our code below, place the downloaded file in your current working directory within a subdirectory called “imported” within a subdirectory called “data”. We used an RStudio project and the here package to navigate to the file more easily.

load(here::here("data", "imported", "imported_pm.rda"))

Click here to see more about creating new projects in RStudio.

You can create a project by going to the File menu of RStudio like so:

You can also do so by clicking the project button:

See here to learn more about using RStudio projects and here to learn more about the here package.

The first step in performing any data analysis is to explore the data.

For example, we might want to better understand the variables included in the data, as we may learn about important details about the data that we should keep in mind as we try to predict our outcome variable.

First, let’s just get a general sense of our data. We can do that using the glimpse() function of the dplyr package (it is also in the tibble package).

We will also use the %>% pipe, which can be used to define the input for later sequential steps.

This will make more sense when we have multiple sequential steps using the same data object.

To use the pipe notation we need to install and load dplyr as well.

For example, here we start with pm data object and “pipe” the object into as input into the glimpse() function. The output is an overview of what is in the pm object such as the number of rows and columns, all the column names, the data types for each column and the first view values in each column. The output below is scrollable so you can see everything from the glimpse() function.

# Scroll through the output!
pm %>%
  dplyr::glimpse()

Rows: 876
Columns: 50
$ id                          <dbl> 1003.001, 1027.000, 1033.100, 1049.100, 10…
$ value                       <dbl> 9.597647, 10.800000, 11.212174, 11.659091,…
$ fips                        <dbl> 1003, 1027, 1033, 1049, 1055, 1069, 1073, …
$ lat                         <dbl> 30.49800, 33.28126, 34.75878, 34.28763, 33…
$ lon                         <dbl> -87.88141, -85.80218, -87.65056, -85.96830…
$ state                       <chr> "Alabama", "Alabama", "Alabama", "Alabama"…
$ county                      <chr> "Baldwin", "Clay", "Colbert", "DeKalb", "E…
$ city                        <chr> "Fairhope", "Ashland", "Muscle Shoals", "C…
$ CMAQ                        <dbl> 8.098836, 9.766208, 9.402679, 8.534772, 9.…
$ zcta                        <dbl> 36532, 36251, 35660, 35962, 35901, 36303, …
$ zcta_area                   <dbl> 190980522, 374132430, 16716984, 203836235,…
$ zcta_pop                    <dbl> 27829, 5103, 9042, 8300, 20045, 30217, 901…
$ imp_a500                    <dbl> 0.01730104, 1.96972318, 19.17301038, 5.782…
$ imp_a1000                   <dbl> 1.4096021, 0.8531574, 11.1448962, 3.867647…
$ imp_a5000                   <dbl> 3.3360118, 0.9851479, 15.1786154, 1.231141…
$ imp_a10000                  <dbl> 1.9879187, 0.5208189, 9.7253870, 1.0316469…
$ imp_a15000                  <dbl> 1.4386207, 0.3359198, 5.2472094, 0.9730444…
$ county_area                 <dbl> 4117521611, 1564252280, 1534877333, 201266…
$ county_pop                  <dbl> 182265, 13932, 54428, 71109, 104430, 10154…
$ log_dist_to_prisec          <dbl> 4.648181, 7.219907, 5.760131, 3.721489, 5.…
$ log_pri_length_5000         <dbl> 8.517193, 8.517193, 8.517193, 8.517193, 9.…
$ log_pri_length_10000        <dbl> 9.210340, 9.210340, 9.274303, 10.409411, 1…
$ log_pri_length_15000        <dbl> 9.630228, 9.615805, 9.658899, 11.173626, 1…
$ log_pri_length_25000        <dbl> 11.32735, 10.12663, 10.15769, 11.90959, 12…
$ log_prisec_length_500       <dbl> 7.295356, 6.214608, 8.611945, 7.310155, 8.…
$ log_prisec_length_1000      <dbl> 8.195119, 7.600902, 9.735569, 8.585843, 9.…
$ log_prisec_length_5000      <dbl> 10.815042, 10.170878, 11.770407, 10.214200…
$ log_prisec_length_10000     <dbl> 11.88680, 11.40554, 12.84066, 11.50894, 12…
$ log_prisec_length_15000     <dbl> 12.205723, 12.042963, 13.282656, 12.353663…
$ log_prisec_length_25000     <dbl> 13.41395, 12.79980, 13.79973, 13.55979, 13…
$ log_nei_2008_pm25_sum_10000 <dbl> 0.318035438, 3.218632928, 6.573127301, 0.0…
$ log_nei_2008_pm25_sum_15000 <dbl> 1.967358961, 3.218632928, 6.581917457, 3.2…
$ log_nei_2008_pm25_sum_25000 <dbl> 5.067308, 3.218633, 6.875900, 4.887665, 4.…
$ log_nei_2008_pm10_sum_10000 <dbl> 1.35588511, 3.31111648, 6.69187313, 0.0000…
$ log_nei_2008_pm10_sum_15000 <dbl> 2.26783411, 3.31111648, 6.70127741, 3.3500…
$ log_nei_2008_pm10_sum_25000 <dbl> 5.628728, 3.311116, 7.148858, 5.171920, 4.…
$ popdens_county              <dbl> 44.265706, 8.906492, 35.460814, 35.330814,…
$ popdens_zcta                <dbl> 145.716431, 13.639555, 540.887040, 40.7189…
$ nohs                        <dbl> 3.3, 11.6, 7.3, 14.3, 4.3, 5.8, 7.1, 2.7, …
$ somehs                      <dbl> 4.9, 19.1, 15.8, 16.7, 13.3, 11.6, 17.1, 6…
$ hs                          <dbl> 25.1, 33.9, 30.6, 35.0, 27.8, 29.8, 37.2, …
$ somecollege                 <dbl> 19.7, 18.8, 20.9, 14.9, 29.2, 21.4, 23.5, …
$ associate                   <dbl> 8.2, 8.0, 7.6, 5.5, 10.1, 7.9, 7.3, 8.0, 4…
$ bachelor                    <dbl> 25.3, 5.5, 12.7, 7.9, 10.0, 13.7, 5.9, 17.…
$ grad                        <dbl> 13.5, 3.1, 5.1, 5.8, 5.4, 9.8, 2.0, 8.7, 2…
$ pov                         <dbl> 6.1, 19.5, 19.0, 13.8, 8.8, 15.6, 25.5, 7.…
$ hs_orless                   <dbl> 33.3, 64.6, 53.7, 66.0, 45.4, 47.2, 61.4, …
$ urc2013                     <dbl> 4, 6, 4, 6, 4, 4, 1, 1, 1, 1, 1, 1, 1, 2, …
$ urc2006                     <dbl> 5, 6, 4, 5, 4, 4, 1, 1, 1, 1, 1, 1, 1, 2, …
$ aod                         <dbl> 37.36364, 34.81818, 36.00000, 33.08333, 43…

We can see that there are 876 monitors (rows) and that we have 50 total variables (columns) - one of which is the outcome variable. In this case, the outcome variable is called value.

Notice that some of the variables that we would think of as factors (or categorical data) are currently of class character as indicated by the <chr> just to the right of the column names/variable names in the glimpse() output. This means that the variable values are character strings, such as words or phrases.

The other variables are of class <dbl>, which stands for double precision which indicates that they are numeric and that they have decimal values. In contrast, one could have integer values which would not allow for decimal numbers. Here is a link for more information on double precision numeric values.

Another common data class is factor which is abbreviated like this: <fct>. A factor is something that has unique levels but there is no appreciable order to the levels. For example we can have a numeric value that is just an id that we want to be interpreted as just a unique level and not as the number that it would typically indicate. This would be useful for several of our variables:

the monitor ID (id)
the Federal Information Processing Standard number for the county where the monitor was located (fips)
the zip code tabulation area (zcta)

None of the values actually have any real numeric meaning, so we want to make sure that R does not interpret them as if they do.

So let’s convert these variables into factors. We can do this using the across() function of the dplyr package and the as.factor() base function. The across() function has two main arguments: (i) the columns you want to operate on and (ii) the function or list of functions to apply to each column.

In this case, we are also using the magrittr assignment pipe or double pipe that looks like this %<>% of the magrittr package. This allows us use the pm data as input, but also reassigns the output to the same data object name.

# Scroll through the output!
pm %<>%
  dplyr::mutate(across(c(id, fips, zcta), as.factor)) 

glimpse(pm)

Rows: 876
Columns: 50
$ id                          <fct> 1003.001, 1027.0001, 1033.1002, 1049.1003,…
$ value                       <dbl> 9.597647, 10.800000, 11.212174, 11.659091,…
$ fips                        <fct> 1003, 1027, 1033, 1049, 1055, 1069, 1073, …
$ lat                         <dbl> 30.49800, 33.28126, 34.75878, 34.28763, 33…
$ lon                         <dbl> -87.88141, -85.80218, -87.65056, -85.96830…
$ state                       <chr> "Alabama", "Alabama", "Alabama", "Alabama"…
$ county                      <chr> "Baldwin", "Clay", "Colbert", "DeKalb", "E…
$ city                        <chr> "Fairhope", "Ashland", "Muscle Shoals", "C…
$ CMAQ                        <dbl> 8.098836, 9.766208, 9.402679, 8.534772, 9.…
$ zcta                        <fct> 36532, 36251, 35660, 35962, 35901, 36303, …
$ zcta_area                   <dbl> 190980522, 374132430, 16716984, 203836235,…
$ zcta_pop                    <dbl> 27829, 5103, 9042, 8300, 20045, 30217, 901…
$ imp_a500                    <dbl> 0.01730104, 1.96972318, 19.17301038, 5.782…
$ imp_a1000                   <dbl> 1.4096021, 0.8531574, 11.1448962, 3.867647…
$ imp_a5000                   <dbl> 3.3360118, 0.9851479, 15.1786154, 1.231141…
$ imp_a10000                  <dbl> 1.9879187, 0.5208189, 9.7253870, 1.0316469…
$ imp_a15000                  <dbl> 1.4386207, 0.3359198, 5.2472094, 0.9730444…
$ county_area                 <dbl> 4117521611, 1564252280, 1534877333, 201266…
$ county_pop                  <dbl> 182265, 13932, 54428, 71109, 104430, 10154…
$ log_dist_to_prisec          <dbl> 4.648181, 7.219907, 5.760131, 3.721489, 5.…
$ log_pri_length_5000         <dbl> 8.517193, 8.517193, 8.517193, 8.517193, 9.…
$ log_pri_length_10000        <dbl> 9.210340, 9.210340, 9.274303, 10.409411, 1…
$ log_pri_length_15000        <dbl> 9.630228, 9.615805, 9.658899, 11.173626, 1…
$ log_pri_length_25000        <dbl> 11.32735, 10.12663, 10.15769, 11.90959, 12…
$ log_prisec_length_500       <dbl> 7.295356, 6.214608, 8.611945, 7.310155, 8.…
$ log_prisec_length_1000      <dbl> 8.195119, 7.600902, 9.735569, 8.585843, 9.…
$ log_prisec_length_5000      <dbl> 10.815042, 10.170878, 11.770407, 10.214200…
$ log_prisec_length_10000     <dbl> 11.88680, 11.40554, 12.84066, 11.50894, 12…
$ log_prisec_length_15000     <dbl> 12.205723, 12.042963, 13.282656, 12.353663…
$ log_prisec_length_25000     <dbl> 13.41395, 12.79980, 13.79973, 13.55979, 13…
$ log_nei_2008_pm25_sum_10000 <dbl> 0.318035438, 3.218632928, 6.573127301, 0.0…
$ log_nei_2008_pm25_sum_15000 <dbl> 1.967358961, 3.218632928, 6.581917457, 3.2…
$ log_nei_2008_pm25_sum_25000 <dbl> 5.067308, 3.218633, 6.875900, 4.887665, 4.…
$ log_nei_2008_pm10_sum_10000 <dbl> 1.35588511, 3.31111648, 6.69187313, 0.0000…
$ log_nei_2008_pm10_sum_15000 <dbl> 2.26783411, 3.31111648, 6.70127741, 3.3500…
$ log_nei_2008_pm10_sum_25000 <dbl> 5.628728, 3.311116, 7.148858, 5.171920, 4.…
$ popdens_county              <dbl> 44.265706, 8.906492, 35.460814, 35.330814,…
$ popdens_zcta                <dbl> 145.716431, 13.639555, 540.887040, 40.7189…
$ nohs                        <dbl> 3.3, 11.6, 7.3, 14.3, 4.3, 5.8, 7.1, 2.7, …
$ somehs                      <dbl> 4.9, 19.1, 15.8, 16.7, 13.3, 11.6, 17.1, 6…
$ hs                          <dbl> 25.1, 33.9, 30.6, 35.0, 27.8, 29.8, 37.2, …
$ somecollege                 <dbl> 19.7, 18.8, 20.9, 14.9, 29.2, 21.4, 23.5, …
$ associate                   <dbl> 8.2, 8.0, 7.6, 5.5, 10.1, 7.9, 7.3, 8.0, 4…
$ bachelor                    <dbl> 25.3, 5.5, 12.7, 7.9, 10.0, 13.7, 5.9, 17.…
$ grad                        <dbl> 13.5, 3.1, 5.1, 5.8, 5.4, 9.8, 2.0, 8.7, 2…
$ pov                         <dbl> 6.1, 19.5, 19.0, 13.8, 8.8, 15.6, 25.5, 7.…
$ hs_orless                   <dbl> 33.3, 64.6, 53.7, 66.0, 45.4, 47.2, 61.4, …
$ urc2013                     <dbl> 4, 6, 4, 6, 4, 4, 1, 1, 1, 1, 1, 1, 1, 2, …
$ urc2006                     <dbl> 5, 6, 4, 5, 4, 4, 1, 1, 1, 1, 1, 1, 1, 2, …
$ aod                         <dbl> 37.36364, 34.81818, 36.00000, 33.08333, 43…

Great! Now we can see that these variables are now factors as indicated by <fct> after the variable name.

`skim` package

The skim() function of the skimr package is also really helpful for getting a general sense of your data. By design, it provides summary statistics about variables in the data set.

# Scroll through the output!
skimr::skim(pm)

Data summary
Name	pm
Number of rows	876
Number of columns	50
_______________________
Column type frequency:
character	3
factor	3
numeric	44
________________________
Group variables	None

Variable type: character

skim_variable	complete_rate	min	max	n_unique
state	1	4	20	49
county	1	3	20	471
city	1	4	48	607

Variable type: factor

skim_variable	complete_rate	ordered	n_unique	top_counts
id	1	FALSE	876	100: 1, 102: 1, 103: 1, 104: 1
fips	1	FALSE	569	170: 12, 603: 10, 261: 9, 107: 8
zcta	1	FALSE	842	475: 3, 110: 2, 160: 2, 290: 2

Variable type: numeric

skim_variable	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
value	1	10.81	2.580000e+00	3.02	9.27	11.15	12.37	2.316000e+01	▂▆▇▁▁
lat	1	38.48	4.620000e+00	25.47	35.03	39.30	41.66	4.840000e+01	▁▃▅▇▂
lon	1	-91.74	1.496000e+01	-124.18	-99.16	-87.47	-80.69	-6.804000e+01	▃▂▃▇▃
CMAQ	1	8.41	2.970000e+00	1.63	6.53	8.62	10.24	2.313000e+01	▃▇▃▁▁
zcta_area	1	183173481.91	5.425989e+08	15459.00	14204601.75	37653560.50	160041508.25	8.164821e+09	▇▁▁▁▁
zcta_pop	1	24227.58	1.777216e+04	0.00	9797.00	22014.00	35004.75	9.539700e+04	▇▇▃▁▁
imp_a500	1	24.72	1.934000e+01	0.00	3.70	25.12	40.22	6.961000e+01	▇▅▆▃▂
imp_a1000	1	24.26	1.802000e+01	0.00	5.32	24.53	38.59	6.750000e+01	▇▅▆▃▁
imp_a5000	1	19.93	1.472000e+01	0.05	6.79	19.07	30.11	7.460000e+01	▇▆▃▁▁
imp_a10000	1	15.82	1.381000e+01	0.09	4.54	12.36	24.17	7.209000e+01	▇▃▂▁▁
imp_a15000	1	13.43	1.312000e+01	0.11	3.24	9.67	20.55	7.110000e+01	▇▃▁▁▁
county_area	1	3768701992.12	6.212830e+09	33703512.00	1116536297.50	1690826566.50	2878192209.00	5.194723e+10	▇▁▁▁▁
county_pop	1	687298.44	1.293489e+06	783.00	100948.00	280730.50	743159.00	9.818605e+06	▇▁▁▁▁
log_dist_to_prisec	1	6.19	1.410000e+00	-1.46	5.43	6.36	7.15	1.045000e+01	▁▁▃▇▁
log_pri_length_5000	1	9.82	1.080000e+00	8.52	8.52	10.05	10.73	1.205000e+01	▇▂▆▅▂
log_pri_length_10000	1	10.92	1.130000e+00	9.21	9.80	11.17	11.83	1.302000e+01	▇▂▇▇▃
log_pri_length_15000	1	11.50	1.150000e+00	9.62	10.87	11.72	12.40	1.359000e+01	▆▂▇▇▃
log_pri_length_25000	1	12.24	1.100000e+00	10.13	11.69	12.46	13.05	1.436000e+01	▅▃▇▇▃
log_prisec_length_500	1	6.99	9.500000e-01	6.21	6.21	6.21	7.82	9.400000e+00	▇▁▂▂▁
log_prisec_length_1000	1	8.56	7.900000e-01	7.60	7.60	8.66	9.20	1.047000e+01	▇▅▆▃▁
log_prisec_length_5000	1	11.28	7.800000e-01	8.52	10.91	11.42	11.83	1.278000e+01	▁▁▃▇▃
log_prisec_length_10000	1	12.41	7.300000e-01	9.21	11.99	12.53	12.94	1.385000e+01	▁▁▃▇▅
log_prisec_length_15000	1	13.03	7.200000e-01	9.62	12.59	13.13	13.57	1.441000e+01	▁▁▃▇▅
log_prisec_length_25000	1	13.82	7.000000e-01	10.13	13.38	13.92	14.35	1.523000e+01	▁▁▃▇▆
log_nei_2008_pm25_sum_10000	1	3.97	2.350000e+00	0.00	2.15	4.29	5.69	9.120000e+00	▆▅▇▆▂
log_nei_2008_pm25_sum_15000	1	4.72	2.250000e+00	0.00	3.47	5.00	6.35	9.420000e+00	▃▃▇▇▂
log_nei_2008_pm25_sum_25000	1	5.67	2.110000e+00	0.00	4.66	5.91	7.28	9.650000e+00	▂▂▇▇▃
log_nei_2008_pm10_sum_10000	1	4.35	2.320000e+00	0.00	2.69	4.62	6.07	9.340000e+00	▅▅▇▇▂
log_nei_2008_pm10_sum_15000	1	5.10	2.180000e+00	0.00	3.87	5.39	6.72	9.710000e+00	▂▃▇▇▂
log_nei_2008_pm10_sum_25000	1	6.07	2.010000e+00	0.00	5.10	6.37	7.52	9.880000e+00	▁▂▆▇▃
popdens_county	1	551.76	1.711510e+03	0.26	40.77	156.67	510.81	2.682191e+04	▇▁▁▁▁
popdens_zcta	1	1279.66	2.757490e+03	0.00	101.15	610.35	1382.52	3.041884e+04	▇▁▁▁▁
nohs	1	6.99	7.210000e+00	0.00	2.70	5.10	8.80	1.000000e+02	▇▁▁▁▁
somehs	1	10.17	6.200000e+00	0.00	5.90	9.40	13.90	7.220000e+01	▇▂▁▁▁
hs	1	30.32	1.140000e+01	0.00	23.80	30.75	36.10	1.000000e+02	▂▇▂▁▁
somecollege	1	21.58	8.600000e+00	0.00	17.50	21.30	24.70	1.000000e+02	▆▇▁▁▁
associate	1	7.13	4.010000e+00	0.00	4.90	7.10	8.80	7.140000e+01	▇▁▁▁▁
bachelor	1	14.90	9.710000e+00	0.00	8.80	12.95	19.22	1.000000e+02	▇▂▁▁▁
grad	1	8.91	8.650000e+00	0.00	3.90	6.70	11.00	1.000000e+02	▇▁▁▁▁
pov	1	14.95	1.133000e+01	0.00	6.50	12.10	21.22	6.590000e+01	▇▅▂▁▁
hs_orless	1	47.48	1.675000e+01	0.00	37.92	48.65	59.10	1.000000e+02	▁▃▇▃▁
urc2013	1	2.92	1.520000e+00	1.00	2.00	3.00	4.00	6.000000e+00	▇▅▃▂▁
urc2006	1	2.97	1.520000e+00	1.00	2.00	3.00	4.00	6.000000e+00	▇▅▃▂▁
aod	1	43.70	1.956000e+01	5.00	31.66	40.17	49.67	1.430000e+02	▃▇▁▁▁

Notice how there is a column called n_missing about the number of values that are missing.

This is also indicated by the complete_rate variable (or missing/number of observations).

In our data set, it looks like our data do not contain any missing data.

Also notice how the function provides separate tables of summary statistics for each data type: character, factor and numeric.

Next, the n_unique column shows us the number of unique values for each of our columns. We can see that there are 49 states represented in the data.

We can see that for many variables there are many low values as the distribution shows two peaks, one near zero and another with a higher value.

This is true for the imp variables (measures of development), the nei variables (measures of emission sources) and the road density variables.

We can also see that the range of some of the variables is very large, in particular the area and population related variables.

Let’s take a look to see which states are included using the distinct() function of the dplyr package:

pm %>% 
  dplyr::distinct(state)

Scroll through the output:

# A tibble: 49 × 1
   state               
   <chr>               
 1 Alabama             
 2 Arizona             
 3 Arkansas            
 4 California          
 5 Colorado            
 6 Connecticut         
 7 Delaware            
 8 District Of Columbia
 9 Florida             
10 Georgia             
11 Idaho               
12 Illinois            
13 Indiana             
14 Iowa                
15 Kansas              
16 Kentucky            
17 Louisiana           
18 Maine               
19 Maryland            
20 Massachusetts       
21 Michigan            
22 Minnesota           
23 Mississippi         
24 Missouri            
25 Montana             
26 Nebraska            
27 Nevada              
28 New Hampshire       
29 New Jersey          
30 New Mexico          
31 New York            
32 North Carolina      
33 North Dakota        
34 Ohio                
35 Oklahoma            
36 Oregon              
37 Pennsylvania        
38 Rhode Island        
39 South Carolina      
40 South Dakota        
41 Tennessee           
42 Texas               
43 Utah                
44 Vermont             
45 Virginia            
46 Washington          
47 West Virginia       
48 Wisconsin           
49 Wyoming

It looks like “District of Columbia” is being included as a state. We can see that Alaska and Hawaii are not included in the data.

Let’s also take a look to see how many monitors there are in a few cities. We can use the filter() function of the dplyr package to do so. For example, let’s look at Albuquerque, New Mexico.

pm %>% dplyr::filter(city == "Albuquerque")

# A tibble: 2 × 50
  id      value fips    lat   lon state county city   CMAQ zcta  zcta_…¹ zcta_…²
  <fct>   <dbl> <fct> <dbl> <dbl> <chr> <chr>  <chr> <dbl> <fct>   <dbl>   <dbl>
1 35001.…  5.98 35001  35.1 -107. New … Berna… Albu…  10.1 87109  2.62e7   40432
2 35001.…  5.91 35001  35.1 -107. New … Berna… Albu…  10.1 87108  1.52e7   38647
# … with 38 more variables: imp_a500 <dbl>, imp_a1000 <dbl>, imp_a5000 <dbl>,
#   imp_a10000 <dbl>, imp_a15000 <dbl>, county_area <dbl>, county_pop <dbl>,
#   log_dist_to_prisec <dbl>, log_pri_length_5000 <dbl>,
#   log_pri_length_10000 <dbl>, log_pri_length_15000 <dbl>,
#   log_pri_length_25000 <dbl>, log_prisec_length_500 <dbl>,
#   log_prisec_length_1000 <dbl>, log_prisec_length_5000 <dbl>,
#   log_prisec_length_10000 <dbl>, log_prisec_length_15000 <dbl>, …
# ℹ Use `colnames()` to see all variable names

We can see that there were only two monitors in the city of Albuquerque in 2006. Let’s compare this with Baltimore.

pm %>% filter(city == "Baltimore")

# A tibble: 5 × 50
  id      value fips    lat   lon state county city   CMAQ zcta  zcta_…¹ zcta_…²
  <fct>   <dbl> <fct> <dbl> <dbl> <chr> <chr>  <chr> <dbl> <fct>   <dbl>   <dbl>
1 24510.…  12.2 24510  39.3 -76.6 Mary… Balti… Balt…  10.9 21251  4.61e5     934
2 24510.…  12.5 24510  39.3 -76.7 Mary… Balti… Balt…  10.9 21215  1.76e7   60161
3 24510.…  12.8 24510  39.3 -76.5 Mary… Balti… Balt…  10.9 21224  2.45e7   49134
4 24510.…  14.3 24510  39.2 -76.6 Mary… Balti… Balt…  10.9 21226  2.57e7    7561
5 24510.…  13.3 24510  39.3 -76.6 Mary… Balti… Balt…  10.9 21202  4.11e6   22832
# … with 38 more variables: imp_a500 <dbl>, imp_a1000 <dbl>, imp_a5000 <dbl>,
#   imp_a10000 <dbl>, imp_a15000 <dbl>, county_area <dbl>, county_pop <dbl>,
#   log_dist_to_prisec <dbl>, log_pri_length_5000 <dbl>,
#   log_pri_length_10000 <dbl>, log_pri_length_15000 <dbl>,
#   log_pri_length_25000 <dbl>, log_prisec_length_500 <dbl>,
#   log_prisec_length_1000 <dbl>, log_prisec_length_5000 <dbl>,
#   log_prisec_length_10000 <dbl>, log_prisec_length_15000 <dbl>, …
# ℹ Use `colnames()` to see all variable names

There were in contrast five monitors for the city of Baltimore, despite the fact that if we take a look at the land area and population of the counties for Baltimore and Albuquerque, we can see that they had very similar land area and populations.

pm %>% 
  filter(city == "Baltimore") %>% 
  dplyr::select(county_area:county_pop)

# A tibble: 5 × 2
  county_area county_pop
        <dbl>      <dbl>
1   209643241     620961
2   209643241     620961
3   209643241     620961
4   209643241     620961
5   209643241     620961

pm %>% 
  filter(city == "Albuquerque") %>%
  select(county_area:county_pop)

# A tibble: 2 × 2
  county_area county_pop
        <dbl>      <dbl>
1  3006530549     662564
2  3006530549     662564

In fact, the county containing Albuerque had a larger population. Thus the measurements for Albuquerque were not as thorough as they were for Baltimore.

This may be due to the fact that the monitor values were lower in Albuquerque. It is interesting to note here that the CMAQ values are quite similar for both cities.

Evaluate correlation

In prediction analyses, it is also useful to evaluate if any of the variables are correlated. Why should we care about this?

If we are using a linear regression to model our data, then we might run into a problem called multicollinearity which can lead us to misinterpret what is really predictive of our outcome variable. This phenomenon occurs when the predictor variables actually predict one another. See this case study for a deeper explanation about this.

Another reason we should look out for correlation is that we don’t want to include redundant variables. This can add unnecessary noise to our algorithm causing a reduction in prediction accuracy, and it can cause our algorithm to be unnecessarily slower. Finally, it can also make it difficult to interpret what variables are actually predictive.

Let’s first take a look at all of our numeric variables with thecorrplot package: The corrplot package is a great option to look at correlation among possible predictors, and particularly useful if we have many predictors.

First, we calculate the Pearson correlation coefficients between all features pairwise using the cor() function of the stats package (which is loaded automatically). Then we use the corrplot::corrplot() function. The tl.cex = 0.5 argument controls the size of the text label.

PM_cor <- cor(pm %>% dplyr::select_if(is.numeric))
corrplot::corrplot(PM_cor, tl.cex = 0.5)

Nice! Now we can see which variables show a positive (blue) or negative (red) correlation to one another. Variables that show a very little correlation with one another appear as white or lightly colored. We can see that each variable is perfectly correlated with itself, which is it why there is a line of blue squares diagnally across the plot.

We can also plot the absolute value of the Pearson correlation coefficients using the abs() function from base R and change the order of the columns. This can be helplful if we aren’t interested in the direction of the correlation and just want to see which variables have a relationship with one another.

corrplot(abs(PM_cor), order = "hclust", tl.cex = 0.5, cl.lim = c(0, 1))

Nice, this is a bit easier to read now as it isn’t as distracting to look at the different colors - we just want to focus on the intensity. Notice that the legend here now doesn’t inform us of much in the case of the red because we have plotted the absolute values of the correlation values and they will thus all be positive and blue. The darker the blue the more correlation.

There are several options for ordering the variables. See here for more options. Here we will use the “hclust” option for ordering by hierarchical clustering - which will order the variables by how similar they are to one another.

The cl.lim = c(0, 1) argument limits the color label to be between 0 and 1.

We can see that the development variables (imp) variables are correlated with each other as we might expect. We also see that the road density variables seem to be correlated with each other, and the emission variables seem to be correlated with each other.

Also notice that none of the predictors are highly correlated with our outcome variable (value).

We can take also take a closer look using the ggcorr() function and the ggpairs() function of the GGally package.

To select our variables of interest we can use the select() function with the contains() function of the tidyr package.

First let’s look at the imp/development variables. We can change the default color palette (palette = "RdBu") and add on correlation coefficients to the plot (label = TRUE).

select(pm, contains("imp")) %>%
  ggcorr(palette = "RdBu", label = TRUE)

select(pm, contains("imp")) %>%
  ggpairs()

Indeed, we can now see more clearly that imp_a1000 and imp_a500 are highly correlated, as well as imp_a10000, imp_a15000. We also get a sense of how the data points vary across the range of values. Note that in this plot red indicates positive correlation values and blue indicates negative correlation values.This is in contrast to our previous plot.

Next, let’s take a look at the road density data:

select(pm, contains("pri")) %>%
  ggcorr(palette = "RdBu", hjust = .85, size = 3,
       layout.exp=2, label = TRUE)

We can see that many of the road density variables are highly correlated with one another, while others are less so. Again note that in this plot red indicates positive correlation values and blue indicates negative correlation values.

Finally let’s look at the emission variables.

select(pm, contains("nei")) %>%
  ggcorr(palette = "RdBu", hjust = .85, size = 3,
       layout.exp=2, label = TRUE)

select(pm, contains("nei")) %>%
  ggpairs()

We can see some fairly high correlation values as well.

We would also expect the population density data might correlate with some of these variables. Let’s take a look.

pm %>%
select(log_nei_2008_pm25_sum_10000, popdens_county, 
       log_pri_length_10000, imp_a10000) %>%
  ggcorr(palette = "RdBu",  hjust = .85, size = 3,
       layout.exp=2, label = TRUE)

pm %>%
select(log_nei_2008_pm25_sum_10000, popdens_county, 
       log_pri_length_10000, imp_a10000, county_pop) %>%
  ggpairs()

Interesting, so these variables don’t appear to be highly correlated, therefore we might need variables from each of the categories to predict our monitor PM_2.5 pollution values.

Because some variables in our data have extreme values, it might be good to take a log transformation. This can affect our estimates of correlation.

pm %>%
  mutate(log_popdens_county= log(popdens_county)) %>%
select(log_nei_2008_pm25_sum_10000, log_popdens_county, 
       log_pri_length_10000, imp_a10000) %>%
  ggcorr(palette = "RdBu",  hjust = .85, size = 3,
       layout.exp=2, label = TRUE)

pm %>%
  mutate(log_popdens_county= log(popdens_county)) %>%
  mutate(log_pop_county = log(county_pop)) %>%
select(log_nei_2008_pm25_sum_10000, log_popdens_county, 
       log_pri_length_10000, imp_a10000, log_pop_county) %>%
  ggpairs()

Indeed this increased the correlation, but variables from each of these categories may still prove to be useful for prediction.

Now that we have a sense of what our data are, we can get started with building a machine learning model to predict air pollution.

First let’s save our data again because we did wrangle it just a tad. This time we will save it to a subdirectory of the “data” directory called “wrangled”. This is a good practice in general for data analyses to keep your data organized. We will also save a csv version as this is often useful to give to collaborators. To do this we will use the write_csv() function of the readr package.

save(pm, file = here::here("data", "wrangled", "wrangled_pm.rda"))
write_csv(pm, file = here::here("data", "wrangled", "wrangled_pm.csv"))

What is machine learning?

You may have learned about the central dogma of statistics that you sample from a population.

Then you use the sample to try to guess what is happening in the population.

For prediction we have a similar sampling problem

But now we are trying to build a rule that can be used to predict a single observation’s value of some characteristic using characteristics of the other observations.

Let’s make this more concrete.

If you recall from the What are the data? section above, when we are using machine learning for prediction, our data consists of:

An continuous outcome variable that we want to predict
A set of feature(s) (or predictor variables) that we use to predict the outcome variable

We will use $Y$ to denote the outcome variable and $X = (X_1, \dots, X_p)$ to denote $p$ different features (or predictor variables). Because our outcome variable is continuous (as opposed to categorical), we are interested in a particular type of machine learning algorithm.

Our goal is to build a machine learning algorithm that uses the features $X$ as input and predicts an outcome variable (or air pollution levels) in the situation where we do not know the outcome variable.

The way we do this is to use data where we have both the features $(X_1=x_1, \dots X_p=x_p)$ and the actual outcome $Y$ data to train a machine learning algorithm to predict the outcome, which we call $\hat{Y}$.

When we say train a machine learning algorithm we mean that we estimate a function $f$ that uses the predictor variables $X$ as input or $\hat{Y} = f(X)$.

ML as an optimization problem

If we are doing a good job, then our predicted outcome $\hat{Y}$ should closely match our actual outcome $Y$ that we observed.

In this way, we can think of machine learning (ML) as an optimization problem that tries to minimize the distance between $\hat{Y} = f(X)$ and $Y$.

\[d(Y - f(X))\] The choice of distance metric $d(\cdot)$ can be the mean of the absolute or squared difference or something more complicated.

Much of the fields of statistics and computer science are focused on defining $f$ and $d$.

The parts of an ML problem

To set up a machine learning (ML) problem, we need a few components. To solve a (standard) machine learning problem you need:

A data set to train from.
An algorithm or set of algorithms you can use to try values of $f$.
A distance metric $d$ for measuring how close $Y$ is to $\hat{Y}$.
A definition of what a “good” distance is.

While each of these components is a technical problem, there has been a ton of work addressing those technical details. The most pressing open issue in machine learning is realizing that though these are technical steps they are not objective steps. In other words, how you choose the data, algorithm, metric, and definition of “good” says what you value and can dramatically change the results. A couple of cases where this was a big deal are:

Machine learning for recidivism - people built ML models to predict who would re-commit a crime. But these predictions were based on historically biased data which led to biased predictions about who would commit new crimes.
Deciding how self driving cars should act - self driving cars will have to make decisions about how to drive, who they might injure, and how to avoid accidents. Depending on our choices for $f$ and $d$ these might lead to wildly different kinds of self driving cars. Try out the moralmachine to see how this looks in practice.

Now that we know a bit more about machine learning, let’s build a model to predict air pollution levels using the tidymodels framework.

Machine learning with `tidymodels`

The goal is to build a machine learning algorithm that uses the features as input and predicts a outcome variable (or air pollution levels) in the situation where we do not know the outcome variable.

The way we do this is to use data where we have both the input and output data to train a machine learning algorithm.

To train a machine learning algorithm, we will use the tidymodels package ecosystem.

Overview

The `tidymodels` ecosystem

To perform our analysis we will be using the tidymodels suite of packages. You may be familiar with the older packages caret or mlr which are also for machine learning and modeling but are not a part of the tidyverse. Max Kuhn describes tidymodels like this:

“Other packages, such as caret and mlr, help to solve the R model API issue. These packages do a lot of other things too: pre-processing, model tuning, resampling, feature selection, ensembling, and so on. In the tidyverse, we strive to make our packages modular and parsnip is designed only to solve the interface issue. It is not designed to be a drop-in replacement for caret. The tidymodels package collection, which includes parsnip, has other packages for many of these tasks, and they are designed to work together. We are working towards higher-level APIs that can replicate and extend what the current model packages can do.”

There are many R packages in the tidymodels ecosystem, which assist with various steps in the process of building a machine learning algorithm. These are the main packages, but there are others.

This is a schematic of how these packages work together to build a machine learning algorithm:

Here we can see that after exploring and splitting the data, we perform the initial modeling stages of variable assignment and pre-processing steps as shown in the blue box the, while the green box indicates the steps required to train the model by specifying the model, fitting the model and tuning it.

Benefits of `tidymodels`

The two major benefits of tidymodels are:

Standardized workflow/format/notation across different types of machine learning algorithms

Different notations are required for different algorithms as the algorithms have been developed by different people. This would require the painstaking process of reformatting the data to be compatible with each algorithm if multiple algorithms were tested.

Can easily modify pre-processing, algorithm choice, and hyper-parameter tuning making optimization easy

Modifying a piece of the overall process is now easier than before because many of the steps are specified using the tidymodels packages in a convenient manner. Thus the entire process can be rerun after a simple change to pre-processing without much difficulty.

`tidymodels` Steps

Splitting the data

The first step after data exploration in machine learning analysis is to split the data into training and testing data sets.

The training data set will be used to build and tune our model. This is the data that the model “learns” on. The testing data set will be used to evaluate the performance of our model in a more generalizable way. What do we mean by “generalizable”?

Remember that our main goal is to use our model to be able to predict air pollution levels in areas where there are no gravimetric monitors.

Therefore, if our model is really good at predicting air pollution with the data that we use to build it, it might not do the best job for the areas where there are few to no monitors.

This would cause us to have really good prediction accuracy and we might assume that we were going to do a good job estimating air pollution any time we use our model, but in fact this would likely not be the case. This situation is what we call overfitting .

Overfitting happens when we end up modeling not only the major relationships in our data but also the noise within our data.

[source]

If we get good prediction with our testing set, then we know that our model can be applied to other data and will likely perform well. We will discuss this more later.

We will not touch the testing set until we have completed optimizing our model with the training set. This will allow us to have a less biased evaluation of how well our model can do with other data besides the data used in the training set to build the model. Ideally, you would also want a completely independent data set to further test the performance of your model.

To split the data into training and testing, we will use the initial_split() function in the rsample package to specify how we want to split our data.

If you skipped previous sections click here for more information on how to obtain and load the data.

First you need to install the OCSdata package:

install.packages("OCSdata")

Then, you may download and load the wrangled data .rda file using the following code:

OCSdata::wrangled_rda("ocs-bp-air-pollution", outpath = getwd())
load(here::here("OCSdata", "data", "wrangled", "wrangled_pm.rda"))

If the package does not work for you, you may also download this .rda file by clicking this link here.

To load the downloaded data into your environment, you may double click on the .rda file in Rstudio or using the load() function.

To copy and paste our code below, place the downloaded .rda file in your current working directory within a subdirectory called “wrangled” within a subdirectory called “data”. We used an RStudio project and the here package to navigate to the file more easily.

load(here::here("data", "wrangled", "wrangled_pm.rda"))

Click here to see more about creating new projects in RStudio.

You can create a project by going to the File menu of RStudio like so:

You can also do so by clicking the project button:

See here to learn more about using RStudio projects and here to learn more about the here package.

set.seed(1234)
pm_split <- rsample::initial_split(data = pm, prop = 2/3)
pm_split

<Training/Testing/Total>
<584/292/876>

A couple of notes from the code above:

Typically, data are split with the majority of the observations for training and a smaller portion for testing. The default with this function is 3/4 (75%) of the observations for training and 1/4 (25%) for testing. This is the default proportion and does not need to be specified. However, you can change the proportion using the prop argument, which we will do here for illustrative purposes. Often people use 80% (4/5) for training and 20% for testing (1/5) or some other similar proportion like what we used here with 2/3 for training and 1/3 for testing.

Click here to learn more about how people decide what split proportion to use.

Having more training data helps the model to train on a greater variety of observations. However, having more testing data helps to see how generalizable the model is and allows for better comparisons of different models. The need for each may depend on your data and how much variability it has and the size of your data. For smaller datasets, setting aside a larger portion for testing can be beneficial to avoid having a very small testing dataset. Here’s a paper that describes more about this topic.

Since the split is performed randomly, it is a good idea to use the set.seed() function in base R to ensure that if your rerun your code that your split will be the same next time.
We can see the number of monitors in our training, testing, and original data by typing in the name of our split object. The result will look like this: <training data sample number, testing data sample number, original sample number>

Now, you can also specify a variable to stratify by with the strata argument. This is useful if you have imbalanced categorical variables and you would like to intentionally make sure that there are similar number of samples of the rarer categories in both the testing and training sets. Otherwise the split is performed randomly.

According to the documentation for the rsample package:

The strata argument causes the random sampling to be conducted within the stratification variable. This can help ensure that the number of data points in the training data is equivalent to the proportions in the original data set.

In the case with our data set, perhaps we would like our training set to have similar proportions of monitors from each of the states as in the initial data. This might be useful if we want our model to be generalizable across all of the states.

We can see that indeed there are different proportions of monitors in each state by using the count() function of the dplyr package.

count(pm, state)

Scroll through the output:

# A tibble: 49 × 2
   state                    n
   <chr>                <int>
 1 Alabama                 24
 2 Arizona                 17
 3 Arkansas                16
 4 California              85
 5 Colorado                15
 6 Connecticut             14
 7 Delaware                 7
 8 District Of Columbia     3
 9 Florida                 29
10 Georgia                 28
11 Idaho                    7
12 Illinois                38
13 Indiana                 36
14 Iowa                    20
15 Kansas                  10
16 Kentucky                22
17 Louisiana               17
18 Maine                    1
19 Maryland                15
20 Massachusetts           16
21 Michigan                30
22 Minnesota               17
23 Mississippi             12
24 Missouri                13
25 Montana                 16
26 Nebraska                 7
27 Nevada                   4
28 New Hampshire            7
29 New Jersey              23
30 New Mexico              10
31 New York                24
32 North Carolina          35
33 North Dakota             4
34 Ohio                    44
35 Oklahoma                10
36 Oregon                  17
37 Pennsylvania            32
38 Rhode Island             5
39 South Carolina          14
40 South Dakota             9
41 Tennessee                3
42 Texas                   27
43 Utah                    14
44 Vermont                  4
45 Virginia                20
46 Washington               8
47 West Virginia           14
48 Wisconsin               21
49 Wyoming                 12

If our data set were large enough it might be nice then to stratify by state using the strata = "state" argument in initial_split(), but our data is unfortunately not large enough.

Importantly the initial_split() function only determines what rows of our pm data frame should be assigned for training or testing, it does not actually split the data.

To extract the testing and training data we can use the training() and testing() functions also of the rsample package.

train_pm <- rsample::training(pm_split)
test_pm <- rsample::testing(pm_split)
 
# Scroll through the output!
count(train_pm, state)

# A tibble: 49 × 2
   state                    n
   <chr>                <int>
 1 Alabama                 13
 2 Arizona                 12
 3 Arkansas                 8
 4 California              55
 5 Colorado                10
 6 Connecticut             12
 7 Delaware                 3
 8 District Of Columbia     2
 9 Florida                 22
10 Georgia                 20
# … with 39 more rows
# ℹ Use `print(n = ...)` to see more rows

count(test_pm, state)

# A tibble: 47 × 2
   state                    n
   <chr>                <int>
 1 Alabama                 11
 2 Arizona                  5
 3 Arkansas                 8
 4 California              30
 5 Colorado                 5
 6 Connecticut              2
 7 Delaware                 4
 8 District Of Columbia     1
 9 Florida                  7
10 Georgia                  8
# … with 37 more rows
# ℹ Use `print(n = ...)` to see more rows

Preparing for pre-processing the data

After splitting the data, the next step is to process the training and testing data so that the data are are compatible and optimized to be used with the model.

In order to start this, we need to think about what each of the aspects of our data might do in the model. For example, is this particular column of data what we would consider the outcome of interest? Or is these data values possibly helpful for predicting the outcome? This process is described as assigning variables to specific roles within the model.

We will then do what is called pre-processing to prepare the data so it is ready. This involves things like like scaling variables and removing redundant variables.

This process is also called feature engineering.

To do this in tidymodels, we will create what’s called a “recipe” using the recipes package, which is a standardized format for a sequence of steps for pre-processing the data. This can be very useful because it makes testing out different pre-processing steps or different algorithms with the same pre-processing very easy and reproducible. Creating a recipe specifies how a data frame of predictors should be created - it specifies what variables to be used and the pre-processing steps, but it does not execute these steps or create the data frame of predictors.

Step 1: Specify variables roles with `recipe()` function

The first thing to do to create a recipe is to specify which variables we will be using as our outcome and predictors using the recipe() function. In terms of the metaphor of baking, we can think of this as listing our ingredients. Translating this to the recipes package, we use the recipe() function to assign roles to all the variables.

Let’s try the simplest recipe with no pre-processing steps: simply list the outcome and predictor variables.

We can do so in two ways:

Using formula notation
Assigning roles to each variable

Let’s look at the first way using formula notation, which looks like this:

outcome(s) ~ predictor(s)

If in the case of multiple predictors or a multivariate situation with two outcomes, use a plus sign:

outcome1 + outcome2 ~ predictor1 + predictor2

If we want to include all predictors we can use a period like so:

outcome_variable_name ~ .

Now with our data, we will start by making a recipe for our training data. If you recall, the continuous outcome variable is value (the average annual gravimetric monitor PM_2.5 concentration in ug/m³). Our features (or predictor variables) are all the other variables except the monitor ID, which is an id variable.

The reason not to include the id variable is because this variable includes the county number and a number designating which particular monitor the values came from (of the monitors there are in that county). Since this number is arbitrary and the county information is also given in the data, and the fact that each monitor only has one value in the value variable, nothing is gained by including this variable and it may instead introduce noise. However, it is useful to keep this data to take a look at what is happening later. We will show you what to do in this case in just a bit.

To summarize this step, we will use the recipe() function to assign roles to all the variables:

We will describe this step by step and then show all the steps together.

In the simplest case, we might use all predictors like this:

simple_rec <- train_pm %>%
  recipes::recipe(value ~ .)

simple_rec

Recipe

Inputs:

      role #variables
   outcome          1
 predictor         49

We see a recipe has been created with 1 outcome variable and 49 predictor variables (or features). Also, notice how we named the output of recipe(). The naming convention for recipe objects is *_rec or rec.

Now, let’s get back to the id variable. Instead of including it as a predictor variable, we could also use the update_role() function of the recipes package.

simple_rec <- train_pm %>%
  recipes::recipe(value ~ .) %>%
  recipes::update_role(id, new_role = "id variable")

simple_rec

Recipe

Inputs:

        role #variables
 id variable          1
     outcome          1
   predictor         48

Click here to learn more about the working with id variables

This option works well with the newer workflows package, however id variables are often dropped from analyses that do not use this newer package as they can make the process difficult with using the parsnip package alone due to the fact that new levels (or possible values) may be introduced with the testing data.

We could also specify the outcome and predictors in the same way as we just specified the id variable. Please see here for examples of other roles for variables. The role can be actually be any value.

The order is important here, as we first make all variables predictors and then override this role for the outcome and id variable. We will use the everything() function of the dplyr package to start with all of the variables in train_pm.

simple_rec <- recipe(train_pm) %>%
    update_role(everything(), new_role = "predictor") %>%
    update_role(value, new_role = "outcome") %>%
    update_role(id, new_role = "id variable")

simple_rec

Recipe

Inputs:

        role #variables
 id variable          1
     outcome          1
   predictor         48

We can view our recipe in more detail using the base summary() function.

summary(simple_rec)

# A tibble: 50 × 4
   variable type    role        source  
   <chr>    <chr>   <chr>       <chr>   
 1 id       nominal id variable original
 2 value    numeric outcome     original
 3 fips     nominal predictor   original
 4 lat      numeric predictor   original
 5 lon      numeric predictor   original
 6 state    nominal predictor   original
 7 county   nominal predictor   original
 8 city     nominal predictor   original
 9 CMAQ     numeric predictor   original
10 zcta     nominal predictor   original
# … with 40 more rows
# ℹ Use `print(n = ...)` to see more rows

Step 2: Specify the pre-processing steps with `step*()` functions

Next, we use the step*() functions from the recipe package to specify pre-processing steps.

This link and this link show the many options for recipe step functions.

There are step functions for a variety of purposes:

Imputation – filling in missing values based on the existing data
Transformation – changing all values of a variable in the same way, typically to make it more normal or easier to interpret
Discretization – converting continuous values into discrete or nominal values - binning for example to reduce the number of possible levels (However this is generally not advisable!)
Encoding / Creating Dummy Variables – creating a numeric code for categorical variables (More on one-hot and Dummy Variables encoding)
Data type conversions – which means changing from integer to factor or numeric to date etc.
Interaction term addition to the model – which means that we would be modeling for predictors that would influence the capacity of each other to predict the outcome
Normalization – centering and scaling the data to a similar range of values
Dimensionality Reduction/ Signal Extraction – reducing the space of features or predictors to a smaller set of variables that capture the variation or signal in the original variables (ex. Principal Component Analysis and Independent Component Analysis)
Filtering – filtering options for removing variables (ex. remove variables that are highly correlated to others or remove variables with very little variance and therefore likely little predictive capacity)
Row operations – performing functions on the values within the rows (ex. rearranging, filtering, imputing)
Checking functions – Gut checks to look for missing values, to look at the variable classes etc.

All of the step functions look like step_*() with the * replaced with a name, except for the check functions which look like check_*().

There are several ways to select what variables to apply steps to:

Using tidyselect methods: contains(), matches(), starts_with(), ends_with(), everything(), num_range()
Using the type: all_nominal(), all_numeric() , has_type()
Using the role: all_predictors(), all_outcomes(), has_role()
Using the name - use the actual name of the variable/variables of interest

Let’s try adding some steps to our recipe.

We might want to potentially modify some of our categorical variables so that they can be used with certain algorithms, like regression, that require only numeric values.

We can do this with the step_dummy() function and the one_hot = TRUE argument to use a method called One-hot encoding.

One-hot encoding means that we do not simply encode our categorical variables numerically as a simple 1,2,3, as our numeric assignments can be interpreted by algorithms as having a particular rank or order. Instead, new binary variables made up of 1s and 0s are used to arbitrarily assign a numeric value that has no apparent order. Note that while there is a different but similar method to do this referred to as dummy variables, one-hot encoded variables are also sometimes called dummy variables.

For more information about what one-hot encoding is, you can expand here.

For example, say we only had three city values for the city variable that were “Tuscon”, “Denver”, and “New York”.

City
Tuscon
New York
Tuscon
Denver

This would be replaced by three new variables one for if the value was “Tuscon”, one for if the value was “Denver”, and one for if the value was “New York”. Each would be made up of zeros and ones. One for the new Tuscon variable would indicate that indeed “Tuscon” was the value and zero would indicate that it was instead “New York” or “Denver”. Essentially there is one “hot” value possible for each new variable, where the value will be one.

Tuscon	Denver	New York
1	0	0
0	0	1
1	0	0
0	1	0

Here we will create such variables to replace the current, state, country, and city categorical data. Similarly the ZCTA values (which are currently a factor class) are not intended to be interpreted as numeric as they could be thought of like a zip code and thus we would also want to encode this as well.

simple_rec %>%
  step_dummy(state, county, city, zcta, one_hot = TRUE)

Recipe

Inputs:

        role #variables
 id variable          1
     outcome          1
   predictor         48

Operations:

Dummy variables from state, county, city, zcta

Click here to see how the variables would change if our recipe stopped here with simply the one-hot encoding.

To create the data we will use steps that we will demonstrate later, for now we will just show you what the new encoded variables look like.

length(names(train_pm))

[1] 50

length(names(baked_one_hot_rec)) # many more variables!

[1] 1733

train_pm %>% select(zcta, city, state, county) %>% head

# A tibble: 6 × 4
  zcta  city          state      county          
  <fct> <chr>         <chr>      <chr>           
1 46805 Fort Wayne    Indiana    Allen           
2 54520 Not in a city Wisconsin  Forest          
3 92506 Riverside     California Riverside       
4 45711 Not in a city Ohio       Athens          
5 45217 St. Bernard   Ohio       Hamilton        
6 21251 Baltimore     Maryland   Baltimore (City)

# let's look at a few
baked_one_hot_rec %>% select("zcta_X54520", "city_Not.in.a.city", "state_Indiana") %>% head()

# A tibble: 6 × 3
  zcta_X54520 city_Not.in.a.city state_Indiana
        <dbl>              <dbl>         <dbl>
1           0                  0             1
2           1                  1             0
3           0                  0             0
4           0                  1             0
5           0                  0             0
6           0                  0             0

Our fips variable includes a numeric code for state and county - and therefore is essentially a proxy for county. Since we already have county, we will just use it and keep the fips ID as another ID variable.

We can remove the fips variable from the predictors using update_role() to make sure that the role is no longer "predictor". We can make the role anything we want actually, so we will keep it something identifiable.

simple_rec %>%
  update_role("fips", new_role = "county id")

Recipe

Inputs:

        role #variables
   county id          1
 id variable          1
     outcome          1
   predictor         47

We might also want to remove variables that appear to be redundant and are highly correlated with others, as we know from our exploratory data analysis that many of our variables are correlated with one another. We can do this using the step_corr() function.

We don’t want to remove some of our variables, like the CMAQ and aod variables, we can specify this using the - sign before the names of these variables like so:

simple_rec %>%
  step_corr(all_predictors(), - CMAQ, - aod)

Recipe

Inputs:

        role #variables
 id variable          1
     outcome          1
   predictor         48

Operations:

Correlation filter on all_predictors(), -CMAQ, -aod

It is also a good idea to remove variables with near-zero variance, which can be done with the step_nzv() function.

Variables have low variance if all the values are very similar, the values are very sparse, or if they are highly imbalanced. Again we don’t want to remove our CMAQ and aod variables.

simple_rec %>%
  step_nzv(all_predictors(), - CMAQ, - aod)

Recipe

Inputs:

        role #variables
 id variable          1
     outcome          1
   predictor         48

Operations:

Sparse, unbalanced variable filter on all_predictors(), -CMAQ, -aod

Click here to learn about examples where you might have near-zero variance variables

Similar Values - If the population density was nearly the same for every zcta that contained a monitor, then knowing the population density near our monitor would contribute little to our model in assisting us to predict monitor air pollution values.
Sparse Data - If all of the monitors were in locations where the populations did not attend graduate school, then these values would mostly be zero, again this would do very little to help us distinguish our air pollution monitors.When many of the values are zero this is also called sparse data.
Imbalanced Data If nearly all of the monitors were located in one particular state, and all the others only had one monitor each, then the real predictive value would simply be in knowing if a monitor is located in that particular state or not. In this case we don’t want to remove our variable, we just want to simplify it.

See this blog post about why removing near-zero variance variables isn’t always a good idea if we think that a variable might be especially informative.

Let’s put all this together now.

Remember: it is important to add the steps to the recipe in an order that makes sense just like with a cooking recipe.

First, we are going to create numeric values for our categorical variables, then we will look at correlation and near-zero variance. Again, we do not want to remove the CMAQ and aod variables, so we can make sure they are kept in the model by excluding them from those steps. If we specifically wanted to remove a predictor we could use step_rm().

simple_rec %<>%
  update_role("fips", new_role = "county id") %>%
  step_dummy(state, county, city, zcta, one_hot = TRUE) %>%
  step_corr(all_predictors(), - CMAQ, - aod)%>%
  step_nzv(all_predictors(), - CMAQ, - aod)
  
simple_rec

Recipe

Inputs:

        role #variables
   county id          1
 id variable          1
     outcome          1
   predictor         47

Operations:

Dummy variables from state, county, city, zcta
Correlation filter on all_predictors(), -CMAQ, -aod
Sparse, unbalanced variable filter on all_predictors(), -CMAQ, -aod

Running the pre-processing

Step 1: Update the recipe with training data using `prep()`

The next major function of the recipes package is prep(). This function updates the recipe object based on the training data. It estimates parameters (estimating the required quantities and statistics required by the steps for the variables) for pre-processing and updates the variables roles, as some of the predictors may be removed, this allows the recipe to be ready to use on other data sets. It does not necessarily actually execute the pre-processing itself; however, we will specify in argument for it to do this so that we can take a look at the pre-processed data.

There are some important arguments to know about:

training - you must supply a training data set to estimate parameters for pre-processing operations (recipe steps) - this may already be included in your recipe - as is the case for us
fresh - if fresh=TRUE, will retrain and estimate parameters for any previous steps that were already prepped if you add more steps to the recipe (default is FALSE)
verbose - if verbose=TRUE, shows the progress as the steps are evaluated and the size of the pre-processed training set (default is FALSE)
retain - if retain=TRUE, then the pre-processed training set will be saved within the recipe (as template). This is good if you are likely to add more steps and do not want to rerun the prep() on the previous steps. However this can make the recipe size large. This is necessary if you want to actually look at the pre-processed data (default is TRUE)

Let’s try out the prep() function:

prepped_rec <- prep(simple_rec, verbose = TRUE, retain = TRUE )

oper 1 step dummy [training] 
oper 2 step corr [training] 
oper 3 step nzv [training] 
The retained training set is ~ 0.26 Mb  in memory.

names(prepped_rec)

 [1] "var_info"       "term_info"      "steps"          "template"      
 [5] "levels"         "retained"       "requirements"   "tr_info"       
 [9] "orig_lvls"      "last_term_info"

There are also lots of useful things to check out in the output of prep(). You can see:

the steps that were run
the original variable info (var_info)
the updated variable info after pre-processing (term_info)
the new levels of the variables
the original levels of the variables (orig_lvls)
info about the training data set size and completeness (tr_info)

Note: You may see the prep.recipe() function in material that you read about the recipes package. This is referring to the prep() function of the recipes package.

Step 2: Extract pre-processed training data using `bake()`

Since we retained our pre-processed training data (i.e. prep(retain=TRUE)), we can take a look at it by using the bake() function of the recipes package. The bake function allows us to apply our modeling steps (in this case just pre-processing on the training data) and see what it would do the data.

Let’s bake!

Since we don’t have new data (we aren’t looking at the testing data), we need to specify this with new_data = NULL.

# Scroll through the output!
baked_train <- bake(prepped_rec, new_data = NULL)
glimpse(baked_train)

Rows: 584
Columns: 37
$ id                          <fct> 18003.0004, 55041.0007, 6065.1003, 39009.0…
$ value                       <dbl> 11.699065, 6.956780, 13.289744, 10.742000,…
$ fips                        <fct> 18003, 55041, 6065, 39009, 39061, 24510, 6…
$ lat                         <dbl> 41.09497, 45.56300, 33.94603, 39.44217, 39…
$ lon                         <dbl> -85.10182, -88.80880, -117.40063, -81.9088…
$ CMAQ                        <dbl> 10.383231, 3.411247, 11.404085, 7.971165, …
$ zcta_area                   <dbl> 16696709, 370280916, 41957182, 132383592, …
$ zcta_pop                    <dbl> 21306, 4141, 44001, 1115, 6566, 934, 41192…
$ imp_a500                    <dbl> 28.9783737, 0.0000000, 30.3901384, 0.00000…
$ imp_a15000                  <dbl> 13.0547959, 0.3676404, 23.7457506, 0.33079…
$ county_area                 <dbl> 1702419942, 2626421270, 18664696661, 13043…
$ county_pop                  <dbl> 355329, 9304, 2189641, 64757, 802374, 6209…
$ log_dist_to_prisec          <dbl> 6.621891, 8.415468, 7.419762, 6.344681, 5.…
$ log_pri_length_5000         <dbl> 8.517193, 8.517193, 10.150514, 8.517193, 9…
$ log_pri_length_25000        <dbl> 12.77378, 10.16440, 13.14450, 10.12663, 13…
$ log_prisec_length_500       <dbl> 6.214608, 6.214608, 6.214608, 6.214608, 7.…
$ log_prisec_length_1000      <dbl> 9.240294, 7.600902, 7.600902, 8.793450, 8.…
$ log_prisec_length_5000      <dbl> 11.485093, 9.425537, 10.155961, 10.562382,…
$ log_prisec_length_10000     <dbl> 12.75582, 11.44833, 11.59563, 11.69093, 12…
$ log_nei_2008_pm10_sum_10000 <dbl> 4.91110140, 3.86982666, 4.03184660, 0.0000…
$ log_nei_2008_pm10_sum_15000 <dbl> 5.399131, 3.883689, 5.459257, 0.000000, 6.…
$ log_nei_2008_pm10_sum_25000 <dbl> 5.816047, 3.887264, 6.884537, 3.765635, 6.…
$ popdens_county              <dbl> 208.719947, 3.542463, 117.314577, 49.64834…
$ popdens_zcta                <dbl> 1276.059851, 11.183401, 1048.711994, 8.422…
$ nohs                        <dbl> 4.3, 5.1, 3.7, 4.8, 2.1, 0.0, 2.5, 7.7, 0.…
$ somehs                      <dbl> 6.7, 10.4, 5.9, 11.5, 10.5, 0.0, 4.3, 7.5,…
$ hs                          <dbl> 31.7, 40.3, 17.9, 47.3, 30.0, 0.0, 17.8, 2…
$ somecollege                 <dbl> 27.2, 24.1, 26.3, 20.0, 27.1, 0.0, 26.1, 2…
$ associate                   <dbl> 8.2, 7.4, 8.3, 3.1, 8.5, 71.4, 13.2, 7.6, …
$ bachelor                    <dbl> 15.0, 8.6, 20.2, 9.8, 14.2, 0.0, 23.4, 17.…
$ grad                        <dbl> 6.8, 4.2, 17.7, 3.5, 7.6, 28.6, 12.6, 12.3…
$ pov                         <dbl> 13.500, 18.900, 6.700, 14.400, 12.500, 3.5…
$ hs_orless                   <dbl> 42.7, 55.8, 27.5, 63.6, 42.6, 0.0, 24.6, 3…
$ urc2006                     <dbl> 3, 6, 1, 5, 1, 1, 2, 1, 2, 6, 4, 4, 4, 4, …
$ aod                         <dbl> 54.11111, 31.16667, 83.12500, 33.36364, 50…
$ state_California            <dbl> 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, …
$ city_Not.in.a.city          <dbl> 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, …

Note- this process used to require the juice() function.

For easy comparison sake - here is our original data:

# Scroll through the output!
glimpse(pm)

Rows: 876
Columns: 50
$ id                          <fct> 1003.001, 1027.0001, 1033.1002, 1049.1003,…
$ value                       <dbl> 9.597647, 10.800000, 11.212174, 11.659091,…
$ fips                        <fct> 1003, 1027, 1033, 1049, 1055, 1069, 1073, …
$ lat                         <dbl> 30.49800, 33.28126, 34.75878, 34.28763, 33…
$ lon                         <dbl> -87.88141, -85.80218, -87.65056, -85.96830…
$ state                       <chr> "Alabama", "Alabama", "Alabama", "Alabama"…
$ county                      <chr> "Baldwin", "Clay", "Colbert", "DeKalb", "E…
$ city                        <chr> "Fairhope", "Ashland", "Muscle Shoals", "C…
$ CMAQ                        <dbl> 8.098836, 9.766208, 9.402679, 8.534772, 9.…
$ zcta                        <fct> 36532, 36251, 35660, 35962, 35901, 36303, …
$ zcta_area                   <dbl> 190980522, 374132430, 16716984, 203836235,…
$ zcta_pop                    <dbl> 27829, 5103, 9042, 8300, 20045, 30217, 901…
$ imp_a500                    <dbl> 0.01730104, 1.96972318, 19.17301038, 5.782…
$ imp_a1000                   <dbl> 1.4096021, 0.8531574, 11.1448962, 3.867647…
$ imp_a5000                   <dbl> 3.3360118, 0.9851479, 15.1786154, 1.231141…
$ imp_a10000                  <dbl> 1.9879187, 0.5208189, 9.7253870, 1.0316469…
$ imp_a15000                  <dbl> 1.4386207, 0.3359198, 5.2472094, 0.9730444…
$ county_area                 <dbl> 4117521611, 1564252280, 1534877333, 201266…
$ county_pop                  <dbl> 182265, 13932, 54428, 71109, 104430, 10154…
$ log_dist_to_prisec          <dbl> 4.648181, 7.219907, 5.760131, 3.721489, 5.…
$ log_pri_length_5000         <dbl> 8.517193, 8.517193, 8.517193, 8.517193, 9.…
$ log_pri_length_10000        <dbl> 9.210340, 9.210340, 9.274303, 10.409411, 1…
$ log_pri_length_15000        <dbl> 9.630228, 9.615805, 9.658899, 11.173626, 1…
$ log_pri_length_25000        <dbl> 11.32735, 10.12663, 10.15769, 11.90959, 12…
$ log_prisec_length_500       <dbl> 7.295356, 6.214608, 8.611945, 7.310155, 8.…
$ log_prisec_length_1000      <dbl> 8.195119, 7.600902, 9.735569, 8.585843, 9.…
$ log_prisec_length_5000      <dbl> 10.815042, 10.170878, 11.770407, 10.214200…
$ log_prisec_length_10000     <dbl> 11.88680, 11.40554, 12.84066, 11.50894, 12…
$ log_prisec_length_15000     <dbl> 12.205723, 12.042963, 13.282656, 12.353663…
$ log_prisec_length_25000     <dbl> 13.41395, 12.79980, 13.79973, 13.55979, 13…
$ log_nei_2008_pm25_sum_10000 <dbl> 0.318035438, 3.218632928, 6.573127301, 0.0…
$ log_nei_2008_pm25_sum_15000 <dbl> 1.967358961, 3.218632928, 6.581917457, 3.2…
$ log_nei_2008_pm25_sum_25000 <dbl> 5.067308, 3.218633, 6.875900, 4.887665, 4.…
$ log_nei_2008_pm10_sum_10000 <dbl> 1.35588511, 3.31111648, 6.69187313, 0.0000…
$ log_nei_2008_pm10_sum_15000 <dbl> 2.26783411, 3.31111648, 6.70127741, 3.3500…
$ log_nei_2008_pm10_sum_25000 <dbl> 5.628728, 3.311116, 7.148858, 5.171920, 4.…
$ popdens_county              <dbl> 44.265706, 8.906492, 35.460814, 35.330814,…
$ popdens_zcta                <dbl> 145.716431, 13.639555, 540.887040, 40.7189…
$ nohs                        <dbl> 3.3, 11.6, 7.3, 14.3, 4.3, 5.8, 7.1, 2.7, …
$ somehs                      <dbl> 4.9, 19.1, 15.8, 16.7, 13.3, 11.6, 17.1, 6…
$ hs                          <dbl> 25.1, 33.9, 30.6, 35.0, 27.8, 29.8, 37.2, …
$ somecollege                 <dbl> 19.7, 18.8, 20.9, 14.9, 29.2, 21.4, 23.5, …
$ associate                   <dbl> 8.2, 8.0, 7.6, 5.5, 10.1, 7.9, 7.3, 8.0, 4…
$ bachelor                    <dbl> 25.3, 5.5, 12.7, 7.9, 10.0, 13.7, 5.9, 17.…
$ grad                        <dbl> 13.5, 3.1, 5.1, 5.8, 5.4, 9.8, 2.0, 8.7, 2…
$ pov                         <dbl> 6.1, 19.5, 19.0, 13.8, 8.8, 15.6, 25.5, 7.…
$ hs_orless                   <dbl> 33.3, 64.6, 53.7, 66.0, 45.4, 47.2, 61.4, …
$ urc2013                     <dbl> 4, 6, 4, 6, 4, 4, 1, 1, 1, 1, 1, 1, 1, 2, …
$ urc2006                     <dbl> 5, 6, 4, 5, 4, 4, 1, 1, 1, 1, 1, 1, 1, 2, …
$ aod                         <dbl> 37.36364, 34.81818, 36.00000, 33.08333, 43…

Notice how we only have 36 variables now instead of 50! Two of these are our ID variables (fips and the actual monitor ID (id)) and one is our outcome (value). Thus we only have 33 predictors now. We can also see that we no longer have any categorical variables. Variables like state are gone and only state_California remains as it was the only state identity to have nonzero variance.

We can see that California had the largest number of monitors compared to the other states.

pm %>% count(state)

Scroll through the output:

# A tibble: 49 × 2
   state                    n
   <chr>                <int>
 1 Alabama                 24
 2 Arizona                 17
 3 Arkansas                16
 4 California              85
 5 Colorado                15
 6 Connecticut             14
 7 Delaware                 7
 8 District Of Columbia     3
 9 Florida                 29
10 Georgia                 28
11 Idaho                    7
12 Illinois                38
13 Indiana                 36
14 Iowa                    20
15 Kansas                  10
16 Kentucky                22
17 Louisiana               17
18 Maine                    1
19 Maryland                15
20 Massachusetts           16
21 Michigan                30
22 Minnesota               17
23 Mississippi             12
24 Missouri                13
25 Montana                 16
26 Nebraska                 7
27 Nevada                   4
28 New Hampshire            7
29 New Jersey              23
30 New Mexico              10
31 New York                24
32 North Carolina          35
33 North Dakota             4
34 Ohio                    44
35 Oklahoma                10
36 Oregon                  17
37 Pennsylvania            32
38 Rhode Island             5
39 South Carolina          14
40 South Dakota             9
41 Tennessee                3
42 Texas                   27
43 Utah                    14
44 Vermont                  4
45 Virginia                20
46 Washington               8
47 West Virginia           14
48 Wisconsin               21
49 Wyoming                 12

We can also see that there were more monitors listed as "Not in a city" than any city.

pm %>% count(city)

Scroll through the output:

# A tibble: 607 × 2
    city                                                 n
    <chr>                                            <int>
  1 Aberdeen                                             1
  2 Akron                                                2
  3 Albany                                               3
  4 Albuquerque                                          2
  5 Alexandria                                           1
  6 Allen Park                                           1
  7 Altamont                                             1
  8 Alton                                                1
  9 Amarillo                                             1
 10 Anadarko                                             1
 11 Anaheim                                              1
 12 Anderson                                             1
 13 Annandale                                            1
 14 Apache Junction                                      1
 15 Apple Valley                                         1
 16 Appleton                                             1
 17 Arden-Arcade                                         1
 18 Arlington                                            1
 19 Arnold                                               1
 20 Asheville                                            1
 21 Ashland                                              2
 22 Atascadero                                           1
 23 Athens-Clarke County (Remainder)                     1
 24 Atlanta                                              2
 25 Atlantic City                                        1
 26 Augusta-Richmond County (Remainder)                  2
 27 Aurora                                               1
 28 Austin                                               1
 29 Azusa                                                1
 30 Bakersfield                                          3
 31 Baltimore                                            5
 32 Batavia                                              1
 33 Baton Rouge                                          1
 34 Bay City                                             1
 35 Bayport                                              1
 36 Baytown                                              1
 37 Beaver Falls                                         1
 38 Beckley                                              1
 39 Belle Glade                                          1
 40 Bellevue                                             1
 41 Beltsville                                           1
 42 Bend                                                 1
 43 Bennington                                           1
 44 Bensley                                              1
 45 Big Bear City                                        1
 46 Billings                                             1
 47 Birmingham                                           2
 48 Bismarck                                             1
 49 Bladensburg                                          1
 50 Blair                                                1
 51 Blue Ash                                             1
 52 Blue Island                                          1
 53 Boise (corporate name Boise City)                    1
 54 Boone                                                1
 55 Boston                                               4
 56 Boulder                                              1
 57 Boulevard                                            1
 58 Bountiful                                            1
 59 Braidwood                                            1
 60 Brawley                                              1
 61 Bridgeport                                           1
 62 Brigham City                                         1
 63 Bristol                                              2
 64 Brockton                                             1
 65 Brook Park                                           1
 66 Brookings                                            1
 67 Brunswick                                            1
 68 Bryson City (RR name Bryson)                         1
 69 Buffalo                                              1
 70 Burbank                                              1
 71 Burlington                                           2
 72 Burns                                                1
 73 Butte-Silver Bow (Remainder)                         1
 74 Calexico                                             1
 75 Camden                                               1
 76 Candor                                               1
 77 Canton                                               2
 78 Carlisle                                             1
 79 Carlstadt                                            3
 80 Cary                                                 1
 81 Casa Grande                                          1
 82 Cedar Rapids                                         2
 83 Cedarhurst                                           1
 84 Central Point                                        1
 85 Chalmette                                            1
 86 Champaign                                            1
 87 Chapel Hill                                          1
 88 Charleroi                                            1
 89 Charleston                                           2
 90 Charlotte                                            2
 91 Chattanooga                                          1
 92 Chelmsford (Chelmsford Center)                       1
 93 Chester                                              2
 94 Cheyenne                                             1
 95 Chicago                                              5
 96 Chickasaw                                            1
 97 Chicopee                                             1
 98 Childersburg                                         1
 99 Chula Vista                                          1
100 Cicero                                               1
101 Cincinnati                                           3
102 Clairton                                             1
103 Claremont                                            1
104 Clarion                                              1
105 Clarksburg                                           1
106 Clearwater                                           1
107 Cleveland                                            6
108 Clinton                                              2
109 Clive                                                1
110 Clovis                                               1
111 Cockeysville                                         1
112 Cody                                                 1
113 Coloma                                               1
114 Colorado Springs                                     1
115 Columbia                                             1
116 Columbia Falls                                       1
117 Columbus                                             4
118 Columbus (Remainder)                                 3
119 Colusa                                               1
120 Commerce City                                        1
121 Concord                                              1
122 Conway                                               1
123 Corcoran                                             1
124 Cornwall                                             1
125 Corpus Christi                                       2
126 Cottage Grove                                        1
127 Cottonwood West                                      1
128 Council Bluffs                                       1
129 Covington                                            1
130 Crossett                                             1
131 Crossville                                           1
132 Dale                                                 1
133 Dallas                                               3
134 Danbury                                              1
135 Darrington                                           1
136 Davenport                                            3
137 Davie                                                1
138 Dayton                                               1
139 Dearborn                                             1
140 Decatur                                              2
141 Delray Beach                                         1
142 Dentsville (Dents)                                   1
143 Denver                                               3
144 Des Moines                                           1
145 Des Plaines                                          1
146 Detroit                                              5
147 Doraville                                            1
148 Dothan                                               1
149 Douglas                                              1
150 Dover                                                1
151 Duluth                                               2
152 Durham                                               1
153 East Chicago                                         1
154 East Farmingdale                                     1
155 East Hartford                                        2
156 East Highland Park                                   1
157 East Providence                                      1
158 East Ridge                                           1
159 East Saint Louis                                     1
160 East Syracuse                                        1
161 Edgewood                                             1
162 El Cajon                                             1
163 El Centro                                            1
164 El Dorado                                            1
165 El Paso                                              3
166 Elgin                                                1
167 Elizabeth                                            2
168 Elizabethtown                                        1
169 Elkhart                                              1
170 Emmetsburg                                           1
171 Erie                                                 1
172 Escondido                                            1
173 Essex                                                1
174 Eugene                                               2
175 Eureka                                               2
176 Evansville                                           3
177 Fairfield                                            1
178 Fairhope                                             1
179 Fairmont                                             1
180 Fall River                                           1
181 Farmington                                           1
182 Farrell                                              1
183 Fayetteville                                         1
184 Ferry Pass                                           1
185 Flagstaff                                            1
186 Flint                                                1
187 Follansbee                                           1
188 Fontana                                              1
189 Forest Park                                          1
190 Fort Collins                                         1
191 Fort Defiance                                        1
192 Fort Lee                                             1
193 Fort Myers                                           1
194 Fort Pierce                                          1
195 Fort Smith                                           1
196 Fort Wayne                                           1
197 Fort Worth                                           2
198 Frankfort                                            1
199 Franklin                                             1
200 Freemansburg                                         1
201 Fremont                                              1
202 Fresno                                               2
203 Gadsden                                              1
204 Gainesville                                          3
205 Galloway (Township of)                               1
206 Garden City                                          1
207 Gary                                                 3
208 Gastonia                                             1
209 Gilroy                                               1
210 Glen Burnie                                          1
211 Goldsboro                                            1
212 Gordon                                               1
213 Grand Island                                         1
214 Grand Junction                                       1
215 Grand Rapids                                         2
216 Granite City                                         2
217 Grants Pass                                          1
218 Grass Valley                                         1
219 Great Falls                                          1
220 Greater Upper Marlboro                               1
221 Greeley                                              1
222 Green Bay                                            1
223 Greensboro                                           1
224 Greensburg                                           1
225 Greenville                                           3
226 Greenwich (Township of)                              1
227 Grenada                                              1
228 Griffith                                             1
229 Groveton                                             1
230 Gulfport                                             1
231 Hamilton                                             1
232 Hammond                                              3
233 Hampton                                              1
234 Harrison Township                                    1
235 Harrisville                                          1
236 Hattiesburg                                          1
237 Haverhill                                            1
238 Helena                                               2
239 Helena Valley West Central                           1
240 Hernando                                             1
241 Hickory                                              1
242 Highland                                             1
243 Highland Heights                                     1
244 Hillsboro                                            1
245 Hobbs                                                1
246 Holland                                              1
247 Hollister                                            1
248 Hollywood                                            1
249 Homestead                                            1
250 Hoover                                               1
251 Hopewell (Township of)                               1
252 Hot Springs (Hot Springs National Park)              1
253 Houston                                              2
254 Huntington                                           1
255 Huntsville                                           1
256 Indianapolis                                         1
257 Indianapolis (Remainder)                             4
258 Indio                                                1
259 Iowa City                                            1
260 Ironton                                              1
261 Jackson                                              2
262 Jacksonville                                         2
263 Jamesville                                           1
264 Jasper                                               3
265 Jean                                                 1
266 Jeffersonville                                       1
267 Jenison                                              1
268 Jersey City                                          1
269 Jerseyville                                          1
270 Johnstown                                            1
271 Joliet                                               1
272 Kalamazoo                                            1
273 Kalispell                                            1
274 Kansas City                                          3
275 Keeler                                               1
276 Keene                                                1
277 Kenansville                                          1
278 Kenner                                               1
279 Kennesaw                                             1
280 Keokuk                                               1
281 Kinston                                              1
282 Kokomo                                               1
283 La Crosse                                            1
284 La Grande                                            1
285 Lackawanna                                           1
286 Laconia                                              1
287 Ladue                                                1
288 Lafayette                                            3
289 Lake Charles                                         1
290 Lakeland                                             1
291 Lakeport                                             1
292 Lakeview                                             1
293 Lancaster                                            2
294 Lander                                               1
295 Lansing                                              1
296 Las Cruces                                           1
297 Las Vegas                                            1
298 Laurel                                               1
299 Lawrence                                             1
300 Leander                                              1
301 Lebanon                                              2
302 Leeds                                                1
303 Lexington                                            1
304 Lexington-Fayette (corporate name for Lexington)     2
305 Libby                                                1
306 Liberty                                              1
307 Lincoln                                              1
308 Lindon                                               1
309 Little Rock                                          2
310 Littleton                                            1
311 Live Oak                                             1
312 Livermore                                            1
313 Livonia                                              1
314 Logan                                                1
315 Long Beach                                           2
316 Longmont                                             1
317 Los Angeles                                          1
318 Louisville                                           4
319 Luna Pier                                            1
320 Lynchburg                                            1
321 Lynn                                                 1
322 Lynwood                                              1
323 Macon                                                2
324 Madison                                              1
325 Magna                                                1
326 Mamaroneck                                           1
327 Manistee                                             1
328 Marble City Community                                1
329 Maricopa                                             1
330 Marion                                               2
331 Marrero                                              1
332 Martinsburg                                          1
333 Marysville                                           1
334 McAlester                                            1
335 McCook                                               1
336 McDonald                                             1
337 McLean                                               1
338 Medford                                              1
339 Melbourne                                            1
340 Mena                                                 1
341 Merced                                               1
342 Meridian                                             1
343 Mesa                                                 1
344 Miami                                                2
345 Michigan City                                        1
346 Middlesborough (corporate name for Middlesboro)      1
347 Middletown                                           2
348 Midlothian                                           1
349 Milwaukee                                            5
350 Mingo Junction                                       1
351 Minneapolis                                          2
352 Mira Loma                                            1
353 Mission                                              1
354 Mission Viejo                                        1
355 Missoula                                             1
356 Modesto                                              1
357 Mojave                                               1
358 Monroe                                               1
359 Montgomery                                           1
360 Morgantown                                           1
361 Morristown                                           1
362 Moundsville                                          1
363 Muncie                                               1
364 Muscatine                                            1
365 Muscle Shoals                                        1
366 Muskegon                                             1
367 Muskogee                                             1
368 Naperville                                           1
369 Nashua                                               1
370 Natchez                                              1
371 New Albany                                           1
372 New Haven                                            5
373 New Paris                                            1
374 New York                                             9
375 Newark                                               2
376 Newburgh                                             1
377 Newburgh Heights                                     1
378 Newport                                              1
379 Niagara Falls                                        1
380 Nogales                                              1
381 Norfolk                                              1
382 Normal                                               1
383 Norristown                                           1
384 North Braddock                                       1
385 North Brunswick Township                             1
386 North Charleston                                     1
387 North Las Vegas                                      1
388 North Little Rock                                    1
389 Northbrook                                           1
390 Norwalk                                              1
391 Norwich                                              1
392 Norwood                                              1
393 Not in a city                                      103
394 Oak Park                                             1
395 Oakland                                              2
396 Oakridge                                             1
397 Odessa                                               1
398 Ogden                                                1
399 Ogden Dunes (Wickliffe)                              1
400 Oglesby                                              1
401 Oklahoma City                                        2
402 Omaha                                                2
403 Onamia                                               1
404 Ontario                                              1
405 Orlando                                              1
406 Overland Park                                        1
407 Paducah                                              1
408 Painesville                                          1
409 Palm Springs                                         1
410 Palm Springs North                                   1
411 Panama City                                          1
412 Pasadena                                             1
413 Pascagoula                                           1
414 Paterson                                             1
415 Pawtucket                                            1
416 Peach Springs                                        1
417 Pelham                                               1
418 Pendleton                                            1
419 Pennsauken (Pensauken)                               1
420 Peoria                                               1
421 Phenix City                                          1
422 Philadelphia                                         5
423 Phillipsburg                                         1
424 Phoenix                                              3
425 Pico Rivera                                          1
426 Pikeville                                            1
427 Pinedale                                             1
428 Pinehurst (Pine Creek)                               1
429 Pinson                                               1
430 Piru                                                 1
431 Pittsboro                                            1
432 Pittsburgh                                           1
433 Pittsfield                                           1
434 Platteville                                          1
435 Pleasant Prairie                                     1
436 Pompano Beach                                        1
437 Port Arthur                                          1
438 Port Huron                                           1
439 Portland                                             2
440 Portola                                              1
441 Portsmouth                                           2
442 Potosi                                               1
443 Potsdam                                              1
444 Powder Springs                                       1
445 Prescott Valley                                      1
446 Presque Isle                                         1
447 Providence                                           2
448 Provo                                                1
449 Pryor (corporate name Pryor Creek)                   1
450 Pueblo                                               1
451 Quincy                                               2
452 Rahway                                               1
453 Raleigh                                              1
454 Rapid City                                           2
455 Ravenna                                              1
456 Redding                                              1
457 Redwood City                                         1
458 Reno                                                 1
459 Reseda                                               1
460 Richfield                                            1
461 Richmond                                             1
462 Ridge Wood Heights                                   1
463 Ridgecrest                                           1
464 Rio Rancho Estates                                   1
465 Riverside                                            1
466 Roanoke                                              2
467 Rochester                                            2
468 Rock Island Arsenal (U.S. Army)                      1
469 Rock Springs                                         1
470 Rockford                                             1
471 Rockwell                                             1
472 Rocky Mount                                          1
473 Rome                                                 1
474 Roseville                                            1
475 Roswell                                              1
476 Roxborough Park                                      1
477 Royal Palm Beach                                     1
478 Rubidoux                                             1
479 Russellville                                         1
480 Rutland                                              1
481 Sacramento                                           2
482 Saint Petersburg                                     1
483 Salinas                                              1
484 Salt Lake City                                       2
485 San Andreas                                          1
486 San Antonio                                          2
487 San Bernardino                                       1
488 San Diego                                            2
489 San Francisco                                        1
490 San Jose                                             1
491 San Luis Obispo                                      1
492 Sandersville                                         1
493 Sanford                                              1
494 Santa Barbara                                        1
495 Santa Fe                                             1
496 Santa Maria                                          1
497 Santa Rosa                                           1
498 Sault Ste. Marie                                     2
499 Savannah                                             1
500 Schiller Park                                        1
501 Scottsbluff                                          1
502 Scottsdale                                           1
503 Scranton                                             1
504 Seaford                                              1
505 Searcy                                               1
506 Seattle                                              2
507 Seeley Lake                                          1
508 Seven Oaks                                           1
509 Shakopee                                             1
510 Sharonville                                          1
511 Shasta Lake                                          1
512 Sheffield                                            1
513 Shepherdsville                                       1
514 Sheridan                                             2
515 Shreveport                                           1
516 Silver City                                          1
517 Simi Valley                                          1
518 Sioux City                                           1
519 Sioux Falls                                          2
520 Soddy-Daisy                                          1
521 South Bend                                           2
522 South Charleston                                     1
523 South Padre Island                                   1
524 Spanish Fork                                         1
525 Spokane                                              1
526 Springdale                                           1
527 Springfield                                          5
528 Spruce Pine                                          1
529 St. Bernard                                          1
530 St. Cloud                                            1
531 St. Joseph                                           1
532 St. Louis                                            4
533 St. Louis Park                                       1
534 St. Paul                                             3
535 State College                                        1
536 Ste. Genevieve                                       1
537 Steubenville                                         1
538 Stockton                                             1
539 Stuttgart                                            1
540 Summit                                               1
541 Suncook                                              1
542 Swansea                                              1
543 Tacoma                                               1
544 Tallahassee                                          1
545 Tampa                                                1
546 Taylors                                              1
547 Tecumseh                                             1
548 Terre Haute                                          2
549 Texarkana                                            1
550 Theodore                                             1
551 Thomaston                                            1
552 Thompson Falls                                       1
553 Thousand Oaks                                        1
554 Toledo                                               3
555 Toms River                                           1
556 Tooele                                               1
557 Topeka                                               1
558 Trenton                                              1
559 Truckee                                              1
560 Tucson                                               2
561 Tulsa                                                2
562 Tupelo                                               1
563 Tuscaloosa                                           1
564 Ukiah                                                1
565 Underhill (Town of)                                  1
566 Union City                                           1
567 Valdosta                                             1
568 Vallejo                                              1
569 Valrico                                              1
570 Vancouver                                            1
571 Victorville                                          1
572 Vienna                                               1
573 Vinton                                               1
574 Virginia                                             1
575 Virginia Beach                                       1
576 Visalia                                              1
577 Warner Robins                                        1
578 Warren                                               1
579 Washington                                           4
580 Waterbury                                            1
581 Waterloo                                             1
582 Watertown                                            1
583 Waukesha                                             1
584 Waynesville                                          1
585 Weirton                                              2
586 West Orange                                          1
587 West Yellowstone                                     1
588 Westfield                                            1
589 Westport                                             1
590 Wheeling                                             1
591 Whitefish                                            1
592 Wichita                                              3
593 Wilmington                                           1
594 Winston-Salem                                        2
595 Winter Park                                          1
596 Wood River                                           1
597 Woodland                                             1
598 Worcester                                            2
599 Wyandotte                                            1
600 Yakima                                               1
601 Yellow Springs                                       1
602 York                                                 1
603 Youngstown                                           2
604 Ypsilanti                                            1
605 Yuba City                                            1
606 Yuma                                                 1
607 Zion                                                 1

Note: Recall that you must specify retain = TRUE argument of the prep() function to use bake() to see pre-processed training data.

Step 3: Extract pre-processed testing data using `bake()`

According to the tidymodels documentation:

bake() takes a trained recipe and applies the operations to a data set to create a design matrix. For example: it applies the centering to new data sets using these means used to create the recipe.

Therefore, if you wanted to look at the pre-processed testing data you would use the bake() function of the recipes package. (You generally want to leave your testing data alone, but it is good to look for issues like the introduction of NA values).

Let’s bake!

# Scroll through the output!
baked_test_pm <- recipes::bake(prepped_rec, new_data = test_pm)
glimpse(baked_test_pm)

Rows: 292
Columns: 37
$ id                          <fct> 1033.1002, 1055.001, 1069.0003, 1073.0023,…
$ value                       <dbl> 11.212174, 12.375394, 10.508850, 15.591017…
$ fips                        <fct> 1033, 1055, 1069, 1073, 1073, 1073, 1073, …
$ lat                         <dbl> 34.75878, 33.99375, 31.22636, 33.55306, 33…
$ lon                         <dbl> -87.65056, -85.99107, -85.39077, -86.81500…
$ CMAQ                        <dbl> 9.402679, 9.241744, 9.121892, 10.235612, 1…
$ zcta_area                   <dbl> 16716984, 154069359, 162685124, 26929603, …
$ zcta_pop                    <dbl> 9042, 20045, 30217, 9010, 16140, 3699, 137…
$ imp_a500                    <dbl> 19.17301038, 16.49307958, 19.13927336, 41.…
$ imp_a15000                  <dbl> 5.2472094, 5.1612102, 4.7401296, 17.452484…
$ county_area                 <dbl> 1534877333, 1385618994, 1501737720, 287819…
$ county_pop                  <dbl> 54428, 104430, 101547, 658466, 658466, 194…
$ log_dist_to_prisec          <dbl> 5.760131, 5.261457, 7.112373, 6.600958, 6.…
$ log_pri_length_5000         <dbl> 8.517193, 9.066563, 8.517193, 11.156977, 1…
$ log_pri_length_25000        <dbl> 10.15769, 12.01356, 10.12663, 12.98762, 12…
$ log_prisec_length_500       <dbl> 8.611945, 8.740680, 6.214608, 6.214608, 6.…
$ log_prisec_length_1000      <dbl> 9.735569, 9.627898, 7.600902, 9.075921, 8.…
$ log_prisec_length_5000      <dbl> 11.770407, 11.728889, 12.298627, 12.281645…
$ log_prisec_length_10000     <dbl> 12.840663, 12.768279, 12.994141, 13.278416…
$ log_nei_2008_pm10_sum_10000 <dbl> 6.69187313, 4.43719884, 0.92888890, 8.2097…
$ log_nei_2008_pm10_sum_15000 <dbl> 6.70127741, 4.46267932, 3.67473904, 8.6488…
$ log_nei_2008_pm10_sum_25000 <dbl> 7.148858, 4.678311, 3.744629, 8.858019, 8.…
$ popdens_county              <dbl> 35.460814, 75.367038, 67.619664, 228.77763…
$ popdens_zcta                <dbl> 540.8870404, 130.1037411, 185.7391706, 334…
$ nohs                        <dbl> 7.3, 4.3, 5.8, 7.1, 2.7, 11.1, 9.7, 3.0, 8…
$ somehs                      <dbl> 15.8, 13.3, 11.6, 17.1, 6.6, 11.6, 21.6, 1…
$ hs                          <dbl> 30.6, 27.8, 29.8, 37.2, 30.7, 46.0, 39.3, …
$ somecollege                 <dbl> 20.9, 29.2, 21.4, 23.5, 25.7, 17.2, 21.6, …
$ associate                   <dbl> 7.6, 10.1, 7.9, 7.3, 8.0, 4.1, 5.2, 6.6, 4…
$ bachelor                    <dbl> 12.7, 10.0, 13.7, 5.9, 17.6, 7.1, 2.2, 7.8…
$ grad                        <dbl> 5.1, 5.4, 9.8, 2.0, 8.7, 2.9, 0.4, 4.2, 3.…
$ pov                         <dbl> 19.0, 8.8, 15.6, 25.5, 7.3, 8.1, 13.3, 23.…
$ hs_orless                   <dbl> 53.7, 45.4, 47.2, 61.4, 40.0, 68.7, 70.6, …
$ urc2006                     <dbl> 4, 4, 4, 1, 1, 1, 2, 3, 3, 3, 2, 5, 4, 1, …
$ aod                         <dbl> 36.000000, 43.416667, 33.000000, 39.583333…
$ state_California            <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
$ city_Not.in.a.city          <dbl> NA, NA, NA, 0, 1, 1, 1, NA, NA, NA, 0, NA,…

Notice that our city_Not.in.a.city variable seems to be NA values. Why might that be?

Ah! Perhaps it is because some of our levels were not previously seen in the training set!

Let’s take a look using the set operations of the dplyr package. We can take a look at cities that were different between the test and training set.

traincities <- train_pm %>% distinct(city)
testcities <- test_pm %>% distinct(city)

#get the number of cities that were different
dim(dplyr::setdiff(traincities, testcities))

[1] 381   1

#get the number of cities that overlapped
dim(dplyr::intersect(traincities, testcities))

[1] 55  1

Indeed, there are lots of different cities in our test data that are not in our training data!

So, let’s go back to our pm data set and modify the city variable to just be values of in a city or not in a city using the case_when() function of dplyr. This function allows you to vectorize multiple if_else() statements.

pm %>%
  mutate(city = case_when(city == "Not in a city" ~ "Not in a city",
                          city != "Not in a city" ~ "In a city"))

# A tibble: 876 × 50
   id     value fips    lat   lon state county city   CMAQ zcta  zcta_…¹ zcta_…²
   <fct>  <dbl> <fct> <dbl> <dbl> <chr> <chr>  <chr> <dbl> <fct>   <dbl>   <dbl>
 1 1003.…  9.60 1003   30.5 -87.9 Alab… Baldw… In a…  8.10 36532  1.91e8   27829
 2 1027.… 10.8  1027   33.3 -85.8 Alab… Clay   In a…  9.77 36251  3.74e8    5103
 3 1033.… 11.2  1033   34.8 -87.7 Alab… Colbe… In a…  9.40 35660  1.67e7    9042
 4 1049.… 11.7  1049   34.3 -86.0 Alab… DeKalb In a…  8.53 35962  2.04e8    8300
 5 1055.… 12.4  1055   34.0 -86.0 Alab… Etowah In a…  9.24 35901  1.54e8   20045
 6 1069.… 10.5  1069   31.2 -85.4 Alab… Houst… In a…  9.12 36303  1.63e8   30217
 7 1073.… 15.6  1073   33.6 -86.8 Alab… Jeffe… In a… 10.2  35207  2.69e7    9010
 8 1073.… 12.4  1073   33.3 -87.0 Alab… Jeffe… Not … 10.2  35111  1.66e8   16140
 9 1073.… 11.1  1073   33.5 -87.3 Alab… Jeffe… Not …  8.16 35444  3.86e8    3699
10 1073.… 13.1  1073   33.5 -86.5 Alab… Jeffe… In a…  9.30 35094  1.49e8   14212
# … with 866 more rows, 38 more variables: imp_a500 <dbl>, imp_a1000 <dbl>,
#   imp_a5000 <dbl>, imp_a10000 <dbl>, imp_a15000 <dbl>, county_area <dbl>,
#   county_pop <dbl>, log_dist_to_prisec <dbl>, log_pri_length_5000 <dbl>,
#   log_pri_length_10000 <dbl>, log_pri_length_15000 <dbl>,
#   log_pri_length_25000 <dbl>, log_prisec_length_500 <dbl>,
#   log_prisec_length_1000 <dbl>, log_prisec_length_5000 <dbl>,
#   log_prisec_length_10000 <dbl>, log_prisec_length_15000 <dbl>, …
# ℹ Use `print(n = ...)` to see more rows, and `colnames()` to see all variable names

Alternatively you could create a custom step function to do this and add this to your recipe, but that is beyond the scope of this case study.

We will need to repeat all the steps (splitting the data, pre-processing, etc) as the levels of our variables have now changed.

While we are doing this, we might also have this issue for county.

The county variables appears to get dropped due to either correlation or near zero variance.

It is likely due to near zero variance because this is the more granular of these geographic categorical variables and likely sparse.

pm %<>%
  mutate(city = case_when(city == "Not in a city" ~ "Not in a city",
                          city != "Not in a city" ~ "In a city"))

set.seed(1234) # same seed as before
pm_split <-rsample::initial_split(data = pm, prop = 2/3)
pm_split

<Training/Testing/Total>
<584/292/876>

 train_pm <-rsample::training(pm_split)
 test_pm <-rsample::testing(pm_split)

Question Opportunity

See if you can come up with the code for the new recipe.

Click here to reveal the code for the new recipe.

novel_rec <- recipe(train_pm) %>%
    update_role(everything(), new_role = "predictor") %>%
    update_role(value, new_role = "outcome") %>%
    update_role(id, new_role = "id variable") %>%
    update_role("fips", new_role = "county id") %>%
    step_dummy(state, county, city, zcta, one_hot = TRUE) %>%
    step_corr(all_numeric()) %>%
    step_nzv(all_numeric())

novel_rec

Recipe

Inputs:

        role #variables
   county id          1
 id variable          1
     outcome          1
   predictor         47

Operations:

Dummy variables from state, county, city, zcta
Correlation filter on all_numeric()
Sparse, unbalanced variable filter on all_numeric()

Now let’s retrain our training data with the new model recipe and try baking our test data again.

Question Opportunity

Do you recall how to pre-process and extract the pre-processed training data?

Click here to reveal the answer.

prepped_rec <- prep(novel_rec, verbose = TRUE, retain = TRUE)

oper 1 step dummy [training] 
oper 2 step corr [training] 
oper 3 step nzv [training] 
The retained training set is ~ 0.27 Mb  in memory.

baked_train <- bake(prepped_rec, new_data = NULL)

# Scroll through the output!
glimpse(baked_train)

Rows: 584
Columns: 38
$ id                          <fct> 18003.0004, 55041.0007, 6065.1003, 39009.0…
$ value                       <dbl> 11.699065, 6.956780, 13.289744, 10.742000,…
$ fips                        <fct> 18003, 55041, 6065, 39009, 39061, 24510, 6…
$ lat                         <dbl> 41.09497, 45.56300, 33.94603, 39.44217, 39…
$ lon                         <dbl> -85.10182, -88.80880, -117.40063, -81.9088…
$ CMAQ                        <dbl> 10.383231, 3.411247, 11.404085, 7.971165, …
$ zcta_area                   <dbl> 16696709, 370280916, 41957182, 132383592, …
$ zcta_pop                    <dbl> 21306, 4141, 44001, 1115, 6566, 934, 41192…
$ imp_a500                    <dbl> 28.9783737, 0.0000000, 30.3901384, 0.00000…
$ imp_a15000                  <dbl> 13.0547959, 0.3676404, 23.7457506, 0.33079…
$ county_area                 <dbl> 1702419942, 2626421270, 18664696661, 13043…
$ county_pop                  <dbl> 355329, 9304, 2189641, 64757, 802374, 6209…
$ log_dist_to_prisec          <dbl> 6.621891, 8.415468, 7.419762, 6.344681, 5.…
$ log_pri_length_5000         <dbl> 8.517193, 8.517193, 10.150514, 8.517193, 9…
$ log_pri_length_25000        <dbl> 12.77378, 10.16440, 13.14450, 10.12663, 13…
$ log_prisec_length_500       <dbl> 6.214608, 6.214608, 6.214608, 6.214608, 7.…
$ log_prisec_length_1000      <dbl> 9.240294, 7.600902, 7.600902, 8.793450, 8.…
$ log_prisec_length_5000      <dbl> 11.485093, 9.425537, 10.155961, 10.562382,…
$ log_prisec_length_10000     <dbl> 12.75582, 11.44833, 11.59563, 11.69093, 12…
$ log_prisec_length_25000     <dbl> 13.98749, 13.15082, 13.44293, 13.58697, 14…
$ log_nei_2008_pm10_sum_10000 <dbl> 4.91110140, 3.86982666, 4.03184660, 0.0000…
$ log_nei_2008_pm10_sum_15000 <dbl> 5.399131, 3.883689, 5.459257, 0.000000, 6.…
$ log_nei_2008_pm10_sum_25000 <dbl> 5.816047, 3.887264, 6.884537, 3.765635, 6.…
$ popdens_county              <dbl> 208.719947, 3.542463, 117.314577, 49.64834…
$ popdens_zcta                <dbl> 1276.059851, 11.183401, 1048.711994, 8.422…
$ nohs                        <dbl> 4.3, 5.1, 3.7, 4.8, 2.1, 0.0, 2.5, 7.7, 0.…
$ somehs                      <dbl> 6.7, 10.4, 5.9, 11.5, 10.5, 0.0, 4.3, 7.5,…
$ hs                          <dbl> 31.7, 40.3, 17.9, 47.3, 30.0, 0.0, 17.8, 2…
$ somecollege                 <dbl> 27.2, 24.1, 26.3, 20.0, 27.1, 0.0, 26.1, 2…
$ associate                   <dbl> 8.2, 7.4, 8.3, 3.1, 8.5, 71.4, 13.2, 7.6, …
$ bachelor                    <dbl> 15.0, 8.6, 20.2, 9.8, 14.2, 0.0, 23.4, 17.…
$ grad                        <dbl> 6.8, 4.2, 17.7, 3.5, 7.6, 28.6, 12.6, 12.3…
$ pov                         <dbl> 13.500, 18.900, 6.700, 14.400, 12.500, 3.5…
$ hs_orless                   <dbl> 42.7, 55.8, 27.5, 63.6, 42.6, 0.0, 24.6, 3…
$ urc2006                     <dbl> 3, 6, 1, 5, 1, 1, 2, 1, 2, 6, 4, 4, 4, 4, …
$ aod                         <dbl> 54.11111, 31.16667, 83.12500, 33.36364, 50…
$ state_California            <dbl> 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, …
$ city_Not.in.a.city          <dbl> 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, …

And now, let’s try baking our test set to see if we still have NA values.

# Scroll through the output!
baked_test_pm <- bake(prepped_rec, new_data = test_pm)

glimpse(baked_test_pm)

Rows: 292
Columns: 38
$ id                          <fct> 1033.1002, 1055.001, 1069.0003, 1073.0023,…
$ value                       <dbl> 11.212174, 12.375394, 10.508850, 15.591017…
$ fips                        <fct> 1033, 1055, 1069, 1073, 1073, 1073, 1073, …
$ lat                         <dbl> 34.75878, 33.99375, 31.22636, 33.55306, 33…
$ lon                         <dbl> -87.65056, -85.99107, -85.39077, -86.81500…
$ CMAQ                        <dbl> 9.402679, 9.241744, 9.121892, 10.235612, 1…
$ zcta_area                   <dbl> 16716984, 154069359, 162685124, 26929603, …
$ zcta_pop                    <dbl> 9042, 20045, 30217, 9010, 16140, 3699, 137…
$ imp_a500                    <dbl> 19.17301038, 16.49307958, 19.13927336, 41.…
$ imp_a15000                  <dbl> 5.2472094, 5.1612102, 4.7401296, 17.452484…
$ county_area                 <dbl> 1534877333, 1385618994, 1501737720, 287819…
$ county_pop                  <dbl> 54428, 104430, 101547, 658466, 658466, 194…
$ log_dist_to_prisec          <dbl> 5.760131, 5.261457, 7.112373, 6.600958, 6.…
$ log_pri_length_5000         <dbl> 8.517193, 9.066563, 8.517193, 11.156977, 1…
$ log_pri_length_25000        <dbl> 10.15769, 12.01356, 10.12663, 12.98762, 12…
$ log_prisec_length_500       <dbl> 8.611945, 8.740680, 6.214608, 6.214608, 6.…
$ log_prisec_length_1000      <dbl> 9.735569, 9.627898, 7.600902, 9.075921, 8.…
$ log_prisec_length_5000      <dbl> 11.770407, 11.728889, 12.298627, 12.281645…
$ log_prisec_length_10000     <dbl> 12.840663, 12.768279, 12.994141, 13.278416…
$ log_prisec_length_25000     <dbl> 13.79973, 13.70026, 13.85550, 14.45221, 13…
$ log_nei_2008_pm10_sum_10000 <dbl> 6.69187313, 4.43719884, 0.92888890, 8.2097…
$ log_nei_2008_pm10_sum_15000 <dbl> 6.70127741, 4.46267932, 3.67473904, 8.6488…
$ log_nei_2008_pm10_sum_25000 <dbl> 7.148858, 4.678311, 3.744629, 8.858019, 8.…
$ popdens_county              <dbl> 35.460814, 75.367038, 67.619664, 228.77763…
$ popdens_zcta                <dbl> 540.8870404, 130.1037411, 185.7391706, 334…
$ nohs                        <dbl> 7.3, 4.3, 5.8, 7.1, 2.7, 11.1, 9.7, 3.0, 8…
$ somehs                      <dbl> 15.8, 13.3, 11.6, 17.1, 6.6, 11.6, 21.6, 1…
$ hs                          <dbl> 30.6, 27.8, 29.8, 37.2, 30.7, 46.0, 39.3, …
$ somecollege                 <dbl> 20.9, 29.2, 21.4, 23.5, 25.7, 17.2, 21.6, …
$ associate                   <dbl> 7.6, 10.1, 7.9, 7.3, 8.0, 4.1, 5.2, 6.6, 4…
$ bachelor                    <dbl> 12.7, 10.0, 13.7, 5.9, 17.6, 7.1, 2.2, 7.8…
$ grad                        <dbl> 5.1, 5.4, 9.8, 2.0, 8.7, 2.9, 0.4, 4.2, 3.…
$ pov                         <dbl> 19.0, 8.8, 15.6, 25.5, 7.3, 8.1, 13.3, 23.…
$ hs_orless                   <dbl> 53.7, 45.4, 47.2, 61.4, 40.0, 68.7, 70.6, …
$ urc2006                     <dbl> 4, 4, 4, 1, 1, 1, 2, 3, 3, 3, 2, 5, 4, 1, …
$ aod                         <dbl> 36.000000, 43.416667, 33.000000, 39.583333…
$ state_California            <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
$ city_Not.in.a.city          <dbl> 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, …

Great, now we no longer have NA values! :)

Note: if you use the skip option for some of the pre-processing steps, be careful. the juice() function (an older option) will show all of the results ignoring skip = TRUE (as you can still use this function if you prefer it to bake()). The bake() function will not necessarily conduct these steps on the new data.

Specifying the model

So far we have used the packages rsample to split the data and recipes to assign variable types, and to specify and prep our pre-processing (as well as to optionally extract the pre-processed data).

We will now use the parsnip package (which is similar to the previous caret package - and hence why it is named after the vegetable) to specify our model.

There are four things we need to define about our model:

The type of model (using specific functions in parsnip like rand_forest(), logistic_reg() etc.)
The package or engine that we will use to implement the type of model selected (using the set_engine() function)
The mode of learning - classification or regression (using the set_mode() function)
Any arguments necessary for the model/package selected (using the set_args()function - for example the mtry = argument for random forest which is the number of variables to be used as options for splitting at each tree node)

Let’s walk through these steps one by one. For our case, we are going to start our analysis with a linear regression (a very common starting point for modeling) but we will demonstrate how we can try different models. We will also show how to model with the Random Forest method (which is very widely used) later on.

The first step is to define what type of model we would like to use. See here for modeling options in parsnip.

PM_model <- parsnip::linear_reg() # PM was used in the name for particulate matter
PM_model

Linear Regression Model Specification (regression)

Computational engine: lm

OK. So far, all we have defined is that we want to use a linear regression…
Let’s tell parsnip more about what we want.

We would like to use the ordinary least squares method to fit our linear regression. So we will tell parsnip that we want to use the lm package to implement our linear regression (there are many options actually such as rstan glmnet, keras, and sparklyr). See here for a description of the differences and using these different engines with parsnip.

We will do so by using the set_engine() function of the parsnip package.

lm_PM_model <- PM_model  %>%
  parsnip::set_engine("lm")

lm_PM_model

Linear Regression Model Specification (regression)

Computational engine: lm

In some cases some packages can do either classification or prediction, so it is a good idea to specify which mode you intend to perform. Here, we aim to predict the air pollution. You can do this with the set_mode() function of the parsnip package, by using either set_mode("classification") or set_mode("regression").

lm_PM_model <- PM_model  %>%
  parsnip::set_engine("lm") %>%
  set_mode("regression")

lm_PM_model

Linear Regression Model Specification (regression)

Computational engine: lm

Fitting the model

We can use the parsnip package with a newer package called workflows to fit our model.

The workflows package allows us to keep track of both our pre-processing steps and our model specification. It also allows us to implement fancier optimizations in an automated way and it can also handle post-processing operations.

We begin by creating a workflow using the workflow() function in the workflows package.

Next, we use add_recipe() (our pre-processing specifications) and we add our model with the add_model() function – both functions from the workflows package.

Note: We do not need to actually prep() our recipe before using workflows!

If you recall novel_rec is the recipe we previously created with the recipes package and lm_PM_model was created when we specified our model with the parsnip package. Here, we combine everything together into a workflow.

PM_wflow <- workflows::workflow() %>%
            workflows::add_recipe(novel_rec) %>%
            workflows::add_model(lm_PM_model)
PM_wflow

══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: linear_reg()

── Preprocessor ────────────────────────────────────────────────────────────────
3 Recipe Steps

• step_dummy()
• step_corr()
• step_nzv()

── Model ───────────────────────────────────────────────────────────────────────
Linear Regression Model Specification (regression)

Computational engine: lm

Ah, nice. Notice how it tells us about both our pre-processing steps and our model specifications.

Next, we “prepare the recipe” (or estimate the parameters) and fit the model to our training data all at once. Printing the output, we can see the coefficients of the model.

PM_wflow_fit <- parsnip::fit(PM_wflow, data = train_pm)
PM_wflow_fit

══ Workflow [trained] ══════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: linear_reg()

── Preprocessor ────────────────────────────────────────────────────────────────
3 Recipe Steps

• step_dummy()
• step_corr()
• step_nzv()

── Model ───────────────────────────────────────────────────────────────────────

Call:
stats::lm(formula = ..y ~ ., data = data)

Coefficients:
                (Intercept)                          lat  
                  2.936e+02                    3.261e-02  
                        lon                         CMAQ  
                  1.586e-02                    2.463e-01  
                  zcta_area                     zcta_pop  
                 -3.433e-10                    1.013e-05  
                   imp_a500                   imp_a15000  
                  5.064e-03                   -3.066e-03  
                county_area                   county_pop  
                 -2.324e-11                   -7.576e-08  
         log_dist_to_prisec          log_pri_length_5000  
                  6.214e-02                   -2.006e-01  
       log_pri_length_25000        log_prisec_length_500  
                 -5.411e-02                    2.204e-01  
     log_prisec_length_1000       log_prisec_length_5000  
                  1.154e-01                    2.374e-01  
    log_prisec_length_10000      log_prisec_length_25000  
                 -3.436e-02                    5.224e-01  
log_nei_2008_pm10_sum_10000  log_nei_2008_pm10_sum_15000  
                  1.829e-01                   -2.355e-02  
log_nei_2008_pm10_sum_25000               popdens_county  
                  2.403e-02                    2.203e-05  
               popdens_zcta                         nohs  
                 -2.132e-06                   -2.983e+00  
                     somehs                           hs  
                 -2.956e+00                   -2.962e+00  
                somecollege                    associate  
                 -2.967e+00                   -2.999e+00  
                   bachelor                         grad  
                 -2.979e+00                   -2.978e+00  
                        pov                    hs_orless  
                  1.859e-03                           NA  
                    urc2006                          aod  
                  2.577e-01                    1.535e-02  
           state_California           city_Not.in.a.city  
                  3.114e+00                   -4.250e-02

Click here to see the steps that the workflows package performs that used to be required

Previously, the processed training data (baked_train), as opposed to the raw training data, would be required to fit the model.

In this case, we would actually also need to write the model again! Recall that id and fips are ID variables and that values is our outcome of interest (the air pollution measure at each monitor). It is nice that workflows keeps track of this!

baked_train_ready <- baked_train %>% 
  select(-id, -fips)

PM_fit <- lm_PM_model %>% 
  parsnip::fit(value ~., data = baked_train_ready)

Assessing the model fit

After we fit our model, we can use the broom package to look at the output from the fitted model in an easy/tidy way.

The tidy() function returns a tidy data frame with coefficients from the model (one row per coefficient).

Many other broom functions currently only work with parsnip objects, not raw workflows objects.

However, we can use the tidy function if we first use the extract_fit_parsnip() function which is imported as part of the workflows package from the hardhat package (also part of tidymodels).

wflowoutput <- PM_wflow_fit %>% 
  extract_fit_parsnip() %>% 
  broom::tidy()

wflowoutput

# A tibble: 36 × 5
   term         estimate std.error statistic       p.value
   <chr>           <dbl>     <dbl>     <dbl>         <dbl>
 1 (Intercept)  2.94e+ 2  1.18e+ 2     2.49  0.0130       
 2 lat          3.26e- 2  2.28e- 2     1.43  0.153        
 3 lon          1.59e- 2  1.01e- 2     1.58  0.115        
 4 CMAQ         2.46e- 1  3.97e- 2     6.20  0.00000000108
 5 zcta_area   -3.43e-10  1.60e-10    -2.15  0.0320       
 6 zcta_pop     1.01e- 5  5.33e- 6     1.90  0.0578       
 7 imp_a500     5.06e- 3  7.42e- 3     0.683 0.495        
 8 imp_a15000  -3.07e- 3  1.16e- 2    -0.263 0.792        
 9 county_area -2.32e-11  1.97e-11    -1.18  0.238        
10 county_pop  -7.58e- 8  9.29e- 8    -0.815 0.415        
# … with 26 more rows
# ℹ Use `print(n = ...)` to see more rows

We have fit our model on our training data, which means we have created a model to predict values of air pollution based on the predictors that we have included. Yay!

One last thing before we leave this section. We often are interested in getting a sense of which variables are the most important in our model. We can explore the variable importance using the vip() function of the vip package. This function creates a bar plot of variable importance scores for each predictor variable (or feature) in a model. The bar plot is ordered by importance (highest to smallest).

Notice again that we need to use the extract_fit_parsnip() function.

Let’s take a look at the top 10 contributing variables:

PM_wflow_fit %>% 
  extract_fit_parsnip() %>% 
  vip(num_features = 10)

The location of the monitor (being in California versus another state), the CMAQ model, and the aod satellite information appear to be the most important for predicting the air pollution at a given monitor.

Indeed, if we plot monitor values for those in California relative to other states we can see that there are some high values for monitors in California. This may be playing a role in what we are seeing. Here we assume that you have some experience plotting with the ggplot2package. If not, please see this case study.

Click here for an introduction about this package if you are new to using ggplot2

The ggplot2 package is generally intuitive for beginners because it is based on a grammar of graphics or the gg in ggplot2. The idea is that you can construct many sentences by learning just a few nouns, adjectives, and verbs. There are specific “words” that we will need to learn and once we do, you will be able to create (or “write”) hundreds of different plots.

The critical part to making graphics using ggplot2 is the data needs to be in a tidy format. Given that we have just spent time putting our data in tidy format, we are primed to take advantage of all that ggplot2 has to offer!

We will show how it is easy to pipe tidy data (output) as input to other functions that create plots. This all works because we are working within the tidyverse.

What is the ggplot() function? As explained by Hadley Wickham:

The grammar tells us that a statistical graphic is a mapping from data to aesthetic attributes (colour, shape, size) of geometric objects (points, lines, bars). The plot may also contain statistical transformations of the data and is drawn on a specific coordinates system. ggplot2 Terminology:

ggplot - the main function where you specify the dataset and variables to plot (this is where we define the x and y variable names)
geoms - geometric objects
- e.g. geom_point(), geom_bar(), geom_line(), geom_histogram()
aes - aesthetics
- shape, transparency, color, fill, line types
scales - define how your data will be plotted
- continuous, discrete, log, etc

The function aes() is an aesthetic mapping function inside the ggplot() object. We use this function to specify plot attributes (e.g. x and y variable names) that will not change as we add more layers.

Anything that goes in the ggplot() object becomes a global setting. From there, we use the geom objects to add more layers to the base ggplot() object. These will define what we are interested in illustrating using the data.

baked_train_ready %>% 
  mutate(state_California = as.factor(state_California)) %>%
  mutate(state_California = recode(state_California, 
                                   "0" = "Not California", 
                                   "1" = "California")) %>%
  ggplot(aes(x = state_California, y = value)) + 
  geom_boxplot() +
  geom_jitter(width = .05) + 
  xlab("Location of Monitor")

Model performance

In this next section, our goal is to assess the overall model performance. The way we do this is to compare the similarity between the predicted estimates of the outcome variable produced by the model and the true outcome variable values.

If you recall the What is machine learning? section, we showed how to think about machine learning (ML) as an optimization problem that tries to minimize the distance between our predicted outcome $\hat{Y} = f(X)$ and actual outcome $Y$ using our features (or predictor variables) $X$ as input to a function $f$ that we want to estimate.

\[d(Y - \hat{Y})\]

As our goal in this section is to assess overall model performance, we will now talk about different distance metrics that you can use.

First, let’s pull out our predicted outcome values $\hat{Y} = f(X)$ from the models we fit (using different approaches).

wf_fit <- PM_wflow_fit %>% 
  extract_fit_parsnip()

wf_fitted_values <- fitted(wf_fit[["fit"]])
head(wf_fitted_values)

        1         2         3         4         5         6 
12.186782  9.139406 12.646119 10.377628 11.909934  9.520860

Alternatively, we can get the fitted values using the augment() function of the broom package using the output from workflows:

wf_fitted_values <- 
  broom::augment(wf_fit[["fit"]], data = baked_train) %>% 
  select(value, .fitted:.std.resid)

head(wf_fitted_values)

# A tibble: 6 × 6
  value .fitted   .hat .sigma   .cooksd .std.resid
  <dbl>   <dbl>  <dbl>  <dbl>     <dbl>      <dbl>
1 11.7    12.2  0.0370   2.05 0.0000648     -0.243
2  6.96    9.14 0.0496   2.05 0.00179       -1.09 
3 13.3    12.6  0.0484   2.05 0.000151       0.322
4 10.7    10.4  0.0502   2.05 0.0000504      0.183
5 14.5    11.9  0.0243   2.05 0.00113        1.26 
6 12.2     9.52 0.476    2.04 0.0850         1.81

Note that because we use the actual workflow here, we can (and actually need to) use the raw data instead of the pre-processed data.

values_pred_train <- 
  predict(PM_wflow_fit, train_pm) %>% 
  bind_cols(train_pm %>% select(value, fips, county, id)) 

values_pred_train

# A tibble: 584 × 5
   .pred value fips  county           id        
   <dbl> <dbl> <fct> <chr>            <fct>     
 1 12.2  11.7  18003 Allen            18003.0004
 2  9.14  6.96 55041 Forest           55041.0007
 3 12.6  13.3  6065  Riverside        6065.1003 
 4 10.4  10.7  39009 Athens           39009.0003
 5 11.9  14.5  39061 Hamilton         39061.8001
 6  9.52 12.2  24510 Baltimore (City) 24510.0006
 7 12.6  11.2  6061  Placer           6061.0006 
 8 10.3   6.98 6065  Riverside        6065.5001 
 9  8.74  6.61 44003 Kent             44003.0002
10 10.0  11.6  37111 McDowell         37111.0004
# … with 574 more rows
# ℹ Use `print(n = ...)` to see more rows

Visualizing model performance

Now, we can compare the predicted outcome values (or fitted values) $\hat{Y}$ to the actual outcome values $Y$ that we observed:

wf_fitted_values %>% 
  ggplot(aes(x =  value, y = .fitted)) + 
  geom_point() + 
  xlab("actual outcome values") + 
  ylab("predicted outcome values")

OK, so our range of the predicted outcome values appears to be smaller than the real values. We could probably do a bit better.

Quantifying model performance

Next, let’s use different distance functions $d(\cdot)$ to assess how far off our predicted outcome $\hat{Y} = f(X)$ and actual outcome $Y$ values are from each other:

\[d(Y - \hat{Y})\]

As mentioned, there are entire scholarly fields of research dedicated to identifying different distance metrics $d(\cdot)$ for machine learning applications. However, when performing prediction with a continuous outcome $Y$, a few of the mostly commonly used distance metrics are:

mean absolute error (mae)

\[MAE = \frac{\sum_{i=1}^{n}{(|\hat{y_t}- y_t|)}^2}{n}\]

R squared error (rsq) – this is also known as the coefficient of determination which is the squared correlation between truth and estimate

This is calculated and 1 minus the fraction of the residual sum of squares ($SS_res$) by the total sum of squares ($SS_tot$)

\[RSQ = R^2 = 1 - \frac{SSres}{SStot}\]

\[SS_{tot} = \sum_{i=1}^{n}{(y_i- \bar{y})}^2\] The total sum of squares is proportional to the variance of the data. It is calculated as the sum of each true value from the mean true value ($\bar{y}$).

\[SS_{res} = \sum_{i=1}^{n}{(y_i- \hat{y_i})}^2\]

The sum of squares of residuals is calculated as the sum of each predicted value ($\hat{y_i}$ or sometimes $f_i$) from the true value ($y_i$).

root mean squared error (rmse)

\[RMSE = \sqrt{\frac{\sum_{i=1}^{n}{(\hat{y_t}- y_t)}^2}{n}}\]

One way to calculate these metrics within the tidymodels framework is to use the yardstick package using the metrics() function.

Note that you may obtain different results depending on your version of R and the package versions you are using. See Session Info section to learn what we used.

yardstick::metrics(wf_fitted_values, 
                   truth = value, estimate = .fitted)

# A tibble: 3 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 rmse    standard       1.98 
2 rsq     standard       0.392
3 mae     standard       1.47

Alternatively if you only wanted one metric you could use the mae(), rsq(), or rmse() functions, respectively.

yardstick::mae(wf_fitted_values, 
               truth = value, estimate = .fitted)

# A tibble: 1 × 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 mae     standard        1.47

The lower the error values, the better the performance. RMSE and MAE can range from zero to infinity, so we aren’t doing too bad. The MAE value suggests that the average difference between the value predicted and the real value was 1.47 ug/m3. The range of the values was 3-22 in the training data, so this is a relatively small amount. The difference between the RMSE and the MAE can indicate the variance of the errors. Since the RMSE and MAE were similar, this indicates that the errors were quite consistent across the range of values and large errors are unlikely to have occurred, where our prediction would have been really far off. The R squared error value indicates how much variability in the outcome value could be explained by the predictors in the model. This value ranges from 0 to 1 (sometimes -1 depending on the method used for calculation). A value of 1 would indicate the the model perfectly predicted the outcome. Our value indicates that 39% of the variability of the air pollution measures could be explained by the model. So we could maybe do a bit better. See here for more information.

Cross validation

Until now we have used everything in our “training” dataset (and have not touched the “testing” dataset) from the rsample package to build our machine learning (ML) model $\hat{Y} = f(X)$ (or to estimate $f$ using the features or predictor variable $X$).

Here, we take this beyond the simple split into training and testing data sets. We will use the rsample package again in order to further implement what are called cross validation techniques. This is also called re-sampling or repartitioning.

Note: we are not actually getting new samples from the underlying distribution so the term re-sampling is a bit of a misnomer.

Cross validation splits our training data into multiple training data sets to allow for a deeper assessment of the accuracy of the model.

Here is a visualization of the concept for cross validation/resampling/repartitioning from Max Kuhn:

Technically creating our testing and training set out of our original training data is sometimes considered a form of cross validation, called the holdout method. The reason we do this it so we can get a better sense of the accuracy of our model using data that we did not train it on.

However, we can actually do a better job of optimizing our model for accuracy if we also perform another type of cross validation on the newly defined training set that we just created. There are many cross validation methods and most can be easily implemented using rsample package. Here, we will use a very popular method called either k-fold or v-fold cross validation.

This method involves essentially preforming the hold out method iteratively with the training data.

First, the training set is divided into $v$ (or often called called $k$) equally sized smaller pieces.

Next, the model is trained on the model on $v$-1 subsets of the data iteratively (removing a different $v$ until all possible $v$-1 sets have been evaluated) to get a sense of the performance of the model. This is really useful for fine tuning specific aspects of the model in a process called model tuning, which we will learn about in the next section.

Here is a visualization of how the folds are created:

Note: People typically ignore spatial dependence with cross validation of air pollution monitoring data in the air pollution field, so we will do the same. However, it might make sense to leave out blocks of monitors rather than random individual monitors to help account for some spatial dependence.

Creating the $v$-folds using `rsample`

The vfold_cv() function of the rsample package can be used to parse the training data into folds for $v$-fold cross validation.

The v argument specifies the number of folds to create.
The repeats argument specifies if any samples should be repeated across folds - default is FALSE
The strata argument specifies a variable to stratify samples across folds - just like in initial_split().

Again, because these are created at random, we need to use the base set.seed() function in order to obtain the same results each time we knit this document. Generally speaking using 10 folds is good practice, but this depends on the variability within your data. We are going to use 4 for the sake of expediency in this demonstration.

set.seed(1234)
vfold_pm <- rsample::vfold_cv(data = train_pm, v = 4)
vfold_pm

#  4-fold cross-validation 
# A tibble: 4 × 2
  splits            id   
  <list>            <chr>
1 <split [438/146]> Fold1
2 <split [438/146]> Fold2
3 <split [438/146]> Fold3
4 <split [438/146]> Fold4

pull(vfold_pm, splits)

[[1]]
<Analysis/Assess/Total>
<438/146/584>

[[2]]
<Analysis/Assess/Total>
<438/146/584>

[[3]]
<Analysis/Assess/Total>
<438/146/584>

[[4]]
<Analysis/Assess/Total>
<438/146/584>

Now we can see that we have created 4 folds of the data and we can see how many values were set aside for testing (called assessing for cross validation sets) and training (called analysis for cross validation sets) within each fold.

Once the folds are created they can be used to evaluate performance by fitting the model to each of the re-samples that we created:

Assessing model performance on $v$-folds using `tune`

We can fit the model to our cross validation folds using the fit_resamples() function of the tune package, by specifying our workflow object and the cross validation fold object we just created. See here for more information.

resample_fit <- tune::fit_resamples(PM_wflow, vfold_pm)

We can now take a look at various performance metrics based on the fit of our cross validation “resamples”. To do this we will use the show_best() function of the tune package.

tune::show_best(resample_fit, metric = "rmse")

# A tibble: 1 × 6
  .metric .estimator  mean     n std_err .config             
  <chr>   <chr>      <dbl> <int>   <dbl> <chr>               
1 rmse    standard    2.12     4  0.0444 Preprocessor1_Model1

Here we can see the mean RMSE value across all four folds. The function is called show_best() because it is also used for model tuning and it will show the parameter combination with the best performance, we will discuss this more later in the case study when we test different models with different parameters. However, for now this gives us a more nuanced estimate of the RMSE in terms of the performance of this single model with the training data by looking at the subsets of the training data.

Data Analysis

If you have been following along but stopped and are starting here you could load the wrangled data using the following command:

load(here::here( "data", "wrangled", "wrangled_pm.rda"))

If you skipped the previous sections, click here for a step-by-step guide on how to download and load the data.

First you need to install and load the OCSdata package:

install.packages("OCSdata")
library(OCSdata)

Then, you may load the wrangled data .rda file using the following function:

wrangled_rda("ocs-bp-air-pollution", outpath = getwd())
load(here::here("OCSdata", "data", "wrangled", "wrangled_pm.rda"))

If the package does not work for you, you may also download this .rda file by clicking this link here.

To load the downloaded data into your environment, you may double click on the .rda file in Rstudio or using the load() function.

load(here::here("data", "wrangled", "wrangled_pm.rda"))

Click here to see more about creating new projects in RStudio.

You can create a project by going to the File menu of RStudio like so:

You can also do so by clicking the project button:

See here to learn more about using RStudio projects and here to learn more about the here package.

Then you can modify the data to match what we did for the previous model like so to prepare the city variable and to split the data into testing and training sets. (You will see later that this would be required due to the number of levels for the city variable in the original data).

pm %<>%
  mutate(city = case_when(city == "Not in a city" ~ "Not in a city",
                          city != "Not in a city" ~ "In a city"))

set.seed(1234) # same seed as before
pm_split <-rsample::initial_split(data = pm, prop = 2/3)
pm_split

<Training/Testing/Total>
<584/292/876>

 train_pm <-rsample::training(pm_split)
 test_pm <-rsample::testing(pm_split)

We will also split our data into cross validation folds:

set.seed(1234)
vfold_pm <- rsample::vfold_cv(data = train_pm, v = 4)
vfold_pm

#  4-fold cross-validation 
# A tibble: 4 × 2
  splits            id   
  <list>            <chr>
1 <split [438/146]> Fold1
2 <split [438/146]> Fold2
3 <split [438/146]> Fold3
4 <split [438/146]> Fold4

In the previous section, we demonstrated how to build a machine learning model (specifically a linear regression model) to predict air pollution with the tidymodels framework.

In the next few section, we will demonstrate another very different kind of machine learning model. This will allow us to see if this model will have better prediction performance.

Random Forest

Now, to try to see if we can get better prediction performance, we are going to predict our outcome variable (air pollution) using a decision tree method called random forest.

A decision tree is a tool to partition data or anything really, based on a series of sequential (often binary) decisions, where the decisions are chosen based on their ability to optimally split the data.

Here you can see a simple example:

[source]

In the case of random forest, multiple decision trees are created - hence the name forest, and each tree is built using a random subset of the training data (with replacement) - hence the full name random forest. This random aspect helps to keep the algorithm from overfitting the data.

The mean of the predictions from each of the trees is used in the final output.

Overall, a major distinction with our last regression model, is that random forest allows us to use our categorical data largely as is. There is no need to recode these predictors to be numerical. In our case, we are going to use the random forest method of the the randomForest package.

This package is currently not compatible with categorical variables that have more than 53 levels. See here for the documentation about when this was updated from 25 levels. Thus we will remove the zcta and county variables. This is why it is good that we have modified the city variable to two levels, as there were originally nearly 600, so this would not have been compatible with this package.

Note that the step_novel() function is necessary here for the state variable to get all cross validation folds to work, because there will be different levels included in each fold test and training sets. The new levels for some of the test sets would otherwise result in an error.

According to the documentation for the recipes package:

step_novel creates a specification of a recipe step that will assign a previously unseen factor level to a new value.

RF_rec <- recipe(train_pm) %>%
    update_role(everything(), new_role = "predictor")%>%
    update_role(value, new_role = "outcome")%>%
    update_role(id, new_role = "id variable") %>%
    update_role("fips", new_role = "county id") %>%
    step_novel("state") %>%
    step_string2factor("state", "county", "city") %>%
    step_rm("county") %>%
    step_rm("zcta") %>%
    step_corr(all_numeric())%>%
    step_nzv(all_numeric())

The rand_forest() function of the parsnip package has three important arguments that act as an interface for the different possible engines to perform a random forest analysis:

mtry - The number of predictor variables (or features) that will be randomly sampled at each split when creating the tree models. The default number for regression analyses is the number of predictors divided by 3.
min_n - The minimum number of data points in a node that are required for the node to be split further.
trees - the number of trees in the ensemble

We will start by trying an mtry value of 10 and a min_n value of 3. As you might imagine it is a bit difficult to know what to choose. This is where the tuning process that we just started to describe will come in helpful as we can test different models with different values. However, first let’s just start with these values.

Now that we have our recipe (RF_rec), let’s specify the model with rand_forest() from parsnip.

PMtree_model <- parsnip::rand_forest(mtry = 10, min_n = 3)
PMtree_model

Random Forest Model Specification (unknown)

Main Arguments:
  mtry = 10
  min_n = 3

Computational engine: ranger

Next, we set the engine and mode:

Note that you could also use the ranger or spark packages instead of randomForest. If you were to use the ranger package to implement the random forest analysis you would need to specify an importance argument to be able to evaluate predictor importance. The options are impurity or permutation.

These other packages have different advantages and disadvantages- for example ranger and spark are not as limiting for the number of categories for categorical variables. For more information see their documentation: here for ranger, here for spark, and here for randomForest.

See here for more documentation about implementing these engine options with tidymodels. Note that there are also other R packages for implementing random forest algorithms, but these three packages (ranger, spark, and randomForest) are currently compatible with tidymodels.

We also need to specify with the set_mode() function that our outcome variable (air pollution) is continuous.

RF_PM_model <- PMtree_model %>%
  set_engine("randomForest") %>%
  set_mode("regression")

RF_PM_model

Random Forest Model Specification (regression)

Main Arguments:
  mtry = 10
  min_n = 3

Computational engine: randomForest

Then, we put this all together into a workflow:

Question Opportunity

See if you can come up with the code to do this.

Click here to reveal the answer.

RF_wflow <- workflows::workflow() %>%
  workflows::add_recipe(RF_rec) %>%
  workflows::add_model(RF_PM_model)

RF_wflow

══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: rand_forest()

── Preprocessor ────────────────────────────────────────────────────────────────
6 Recipe Steps

• step_novel()
• step_string2factor()
• step_rm()
• step_rm()
• step_corr()
• step_nzv()

── Model ───────────────────────────────────────────────────────────────────────
Random Forest Model Specification (regression)

Main Arguments:
  mtry = 10
  min_n = 3

Computational engine: randomForest

Finally, we fit the data to the model:

Question Opportunity

Do you recall how to do this?

Click here to reveal the answer.

RF_wflow_fit <- parsnip::fit(RF_wflow, data = train_pm)

If you get an error “Can not handle categorical predictors with more than 53 categories.” then you should scroll up a bit and make sure that you removed the categorical variables that have more than 53 categories as this method can’t handle such variables at this time.

RF_wflow_fit

══ Workflow [trained] ══════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: rand_forest()

── Preprocessor ────────────────────────────────────────────────────────────────
6 Recipe Steps

• step_novel()
• step_string2factor()
• step_rm()
• step_rm()
• step_corr()
• step_nzv()

── Model ───────────────────────────────────────────────────────────────────────

Call:
 randomForest(x = maybe_data_frame(x), y = y, mtry = min_cols(~10,      x), nodesize = min_rows(~3, x)) 
               Type of random forest: regression
                     Number of trees: 500
No. of variables tried at each split: 10

          Mean of squared residuals: 2.698043
                    % Var explained: 58.29

Let’s take a look at the top 10 contributing variables:

Question Opportunity

See if you can recall how to do this.

Click here to reveal the answer.

RF_wflow_fit %>% 
  extract_fit_parsnip() %>% 
  vip(num_features = 10)

Interesting! In the previous model the CMAQ values and the state where the monitor was located (being in California or not) were also the top two most important, however predictors about education levels of the communities where the monitor was located was among the top most important. Now we see that population density and proximity to sources of emissions and roads are among the top ten.

Now let’s take a look at model performance by fitting the data using cross validation:

Question Opportunity

See if you can recall how to do this.

Click here to reveal the answer.

set.seed(456)
resample_RF_fit <- tune::fit_resamples(RF_wflow, vfold_pm)
collect_metrics(resample_RF_fit)

# A tibble: 2 × 6
  .metric .estimator  mean     n std_err .config             
  <chr>   <chr>      <dbl> <int>   <dbl> <chr>               
1 rmse    standard   1.67      4  0.101  Preprocessor1_Model1
2 rsq     standard   0.591     4  0.0514 Preprocessor1_Model1

Model Comparison

Now let’s compare the performance of our model with our linear regression model - if you have been following along you could type this to take a look.

# our initial linear regression model:
collect_metrics(resample_fit)

For those starting here we will tell you that our first model had a cross validation mean rmse value of 2.12. It looks like the random forest model had a much lower rmse value of 1.67. This suggest that this model is better at prediction air pollution values. In addition, the R squared value is much higher (it was 31% and is now 59%), suggesting that more of the variance of the air pollution values could be explained by the new Random Forest model.

Question Opportunity

Do you recall how the RMSE is calculated?

Click here to reveal the answer.

\[RMSE = \sqrt{\frac{\sum_{i=1}^{n}{(\hat{y_t}- y_t)}^2}{n}}\]

If we tuned our random forest model based on the number of trees or the value for mtry (which is “The number of predictors that will be randomly sampled at each split when creating the tree models”), we might get a model with even better performance.

However, our cross validated mean rmse value of 1.67 is quite good because our range of true outcome values is much larger: (4.298, 23.161).

Model tuning

Hyperparameters are often things that we need to specify about a model. For example, the number of predictor variables (or features) that will be randomly sampled at each split when creating the tree models called mtry is a hyperparameter. The default number for regression analyses is the number of predictors divided by 3. Instead of arbitrarily specifying this, we can try to determine the best option for model performance by a process called tuning.

Now let’s try some tuning.

Let’s take a closer look at the mtry and min_n hyperparametrs in our Random Forest model.

We aren’t exactly sure what values of mtry and min_n achieve good accuracy yet keep our model generalizable for other data.

This is when our cross validation methods become really handy because now we can test out different values for each of these hyperparameters to assess what values seem to work best for model performance on these resamples of our training set data.

Previously we specified our model like so:

RF_PM_model <- 
  parsnip::rand_forest(mtry = 10, min_n = 3) %>%
  set_engine("randomForest") %>%
  set_mode("regression")

RF_PM_model

Random Forest Model Specification (regression)

Main Arguments:
  mtry = 10
  min_n = 3

Computational engine: randomForest

Now instead of specifying a value for the mtry and min_n arguments, we can use the tune() function of the tune package like so: mtry = tune(). This indicates that these hyperparameters are to be tuned.

tune_RF_model <- rand_forest(mtry = tune(), min_n = tune()) %>%
  set_engine("randomForest") %>%
  set_mode("regression")
    
tune_RF_model

Random Forest Model Specification (regression)

Main Arguments:
  mtry = tune()
  min_n = tune()

Computational engine: randomForest

Again we will add this to a workflow, the only difference here is that we are using a different model specification with tune_RF_model instead of RF_model:

RF_tune_wflow <- workflows::workflow() %>%
  workflows::add_recipe(RF_rec) %>%
  workflows::add_model(tune_RF_model)
RF_tune_wflow

══ Workflow ════════════════════════════════════════════════════════════════════
Preprocessor: Recipe
Model: rand_forest()

── Preprocessor ────────────────────────────────────────────────────────────────
6 Recipe Steps

• step_novel()
• step_string2factor()
• step_rm()
• step_rm()
• step_corr()
• step_nzv()

── Model ───────────────────────────────────────────────────────────────────────
Random Forest Model Specification (regression)

Main Arguments:
  mtry = tune()
  min_n = tune()

Computational engine: randomForest

Now we can use the tune_grid() function of the tune package to evaluate different combinations of values for mtry and min_n using our cross validation samples of our training set (vfold_pm) to see what combination of values performs best.

To use this function we will specify the workflow using the object argument and the samples to use using the resamples argument. The grid argument specifies how many possible options for each argument should be attempted.

By default 10 different values will be attempted for each hyperparameter that is being tuned.

We can use the doParallel package to allow us to fit all these models to our cross validation samples faster. So if you were performing this on a computer with multiple cores or processors, then different models with different hyperparameter values can be fit to the cross validation samples simultaneously across different cores or processors.

You can see how many cores you have access to on your system using the detectCores() function in the parallel package.

parallel::detectCores()

[1] 8

The registerDoParallel() function will use the number for cores specified using the cores= arguement, or it will assign it automatically to one-half of the number of cores detected by the parallel package.

We need to use set.seed() here because the values chosen for mtry and min_n may vary if we preform this evaluation again because they are chosen semi-randomly (meaning that they are within a range of reasonable values but still random).

Note: this step will take some time.

doParallel::registerDoParallel(cores = 2)
set.seed(123)
tune_RF_results <- tune_grid(object = RF_tune_wflow, resamples = vfold_pm, grid = 20)
tune_RF_results

# Tuning results
# 4-fold cross-validation 
# A tibble: 4 × 4
  splits            id    .metrics          .notes          
  <list>            <chr> <list>            <list>          
1 <split [438/146]> Fold1 <tibble [40 × 6]> <tibble [0 × 3]>
2 <split [438/146]> Fold2 <tibble [40 × 6]> <tibble [0 × 3]>
3 <split [438/146]> Fold3 <tibble [40 × 6]> <tibble [1 × 3]>
4 <split [438/146]> Fold4 <tibble [40 × 6]> <tibble [0 × 3]>

There were issues with some computations:

  - Warning(s) x1: 36 columns were requested but there were 35 predictors in the dat...

Run `show_notes(.Last.tune.result)` for more information.

See the tune getting started guide for more information about implementing this in tidymodels.

If you wanted more control over this process you could specify how the different possible options for mtry and min_n in the tune_grid() function using the grid_*() functions of the dials package to create a more specific grid.

By default the values for the hyperparameters being tuned are chosen semi-randomly (meaning that they are within a range of reasonable values but still random)..

Now we can use the collect_metrics() function again to take a look at what happened with our cross validation tests. We can see the different values chosen for mtry and min_n and the mean rmse and rsq values across the cross validation samples.

tune_RF_results %>%
  collect_metrics()

# A tibble: 40 × 8
    mtry min_n .metric .estimator  mean     n std_err .config              
   <int> <int> <chr>   <chr>      <dbl> <int>   <dbl> <chr>                
 1    12    33 rmse    standard   1.72      4  0.0866 Preprocessor1_Model01
 2    12    33 rsq     standard   0.562     4  0.0466 Preprocessor1_Model01
 3    27    35 rmse    standard   1.69      4  0.102  Preprocessor1_Model02
 4    27    35 rsq     standard   0.563     4  0.0511 Preprocessor1_Model02
 5    22    40 rmse    standard   1.71      4  0.106  Preprocessor1_Model03
 6    22    40 rsq     standard   0.556     4  0.0543 Preprocessor1_Model03
 7     1    27 rmse    standard   2.03      4  0.0501 Preprocessor1_Model04
 8     1    27 rsq     standard   0.440     4  0.0245 Preprocessor1_Model04
 9     6    32 rmse    standard   1.77      4  0.0756 Preprocessor1_Model05
10     6    32 rsq     standard   0.552     4  0.0435 Preprocessor1_Model05
# … with 30 more rows
# ℹ Use `print(n = ...)` to see more rows

We can now use the show_best() function (which we used previously to get a better estimate of the RMSE using cross validation fold of the training data) as it was truly intended, to see what values for min_n and mtry resulted in the best performance.

show_best(tune_RF_results, metric = "rmse", n = 1)

# A tibble: 1 × 8
   mtry min_n .metric .estimator  mean     n std_err .config              
  <int> <int> <chr>   <chr>      <dbl> <int>   <dbl> <chr>                
1    32    11 rmse    standard    1.65     4   0.113 Preprocessor1_Model10

There we have it… looks like an mtry of 32 and show_best(tune_RF_results, metric = "rmse", n = 1)$min_n of 4 had the best rmse value. You can verify this in the above output, but it is easier to just pull this row out using this function. We can see that the mean rmse value across the cross validation sets was 1.65. Before tuning it was 1.67 with a fairly similar std_err so the performance may be slightly improved. In other words the model was slightly better at predicting air pollution values.

Final model performance evaluation

Now that we have decided that we have reasonable performance with our training data using the Random Forest method and tuning, we can stop building our model and evaluate performance with our testing data. Note that you might often need to try a variety of methods to optimize your model.

Here, we will use the random forest model that we built to predict values for the monitors in the testing data and we will use the values for mtry and min_n that we just determined based on our tuning analysis to achieve the best performance.

So, first we need to specify these values in a workflow. We can use the select_best() function of the tune package to grab the values that were determined to be best for mtry and min_n.

tuned_RF_values<- select_best(tune_RF_results, "rmse")
tuned_RF_values

# A tibble: 1 × 3
   mtry min_n .config              
  <int> <int> <chr>                
1    32    11 Preprocessor1_Model10

Now we can finalize the model/workflow that we we used for tuning with these values.

RF_tuned_wflow <-RF_tune_wflow %>%
  tune::finalize_workflow(tuned_RF_values)

With the workflows package, we can use the splitting information for our original data pm_split to fit the final model on the full training set and also on the testing data using the last_fit() function of the tune package. No pre-processing steps are required.

The results will show the performance using the testing data.

overallfit <- tune::last_fit(RF_tuned_wflow, pm_split)
 # or
overallfit <- RF_wflow %>%
  tune::last_fit(pm_split)

The overallfit output has a lot of really useful information about the model, the testing and training data split, and the predictions for the testing data.

To see the performance on the test data we can use the collect_metrics() function like we did before.

collect_metrics(overallfit)

# A tibble: 2 × 4
  .metric .estimator .estimate .config             
  <chr>   <chr>          <dbl> <chr>               
1 rmse    standard       1.71  Preprocessor1_Model1
2 rsq     standard       0.612 Preprocessor1_Model1

Awesome! We can see that our rmse of 1.71 is quite similar to that of our testing data cross validation sets (where rmse was 1.65 . We achieved quite good performance, which suggests that we could predict other locations with more sparse monitoring based on our predictors with reasonable accuracy, however some of our predictors involve monitoring itself, so the accuracy would likely be lower.

Now if you wanted to take a look at the predicted values for the test set (the 292 rows with predictions out of the 876 original monitor values) you can use the collect_predictions() function of the tune package:

test_predictions <- collect_predictions(overallfit)

test_predictions

# A tibble: 292 × 5
    id               .pred  .row value .config             
    <chr>            <dbl> <int> <dbl> <chr>               
  1 train/test split 11.9      3 11.2  Preprocessor1_Model1
  2 train/test split 11.8      5 12.4  Preprocessor1_Model1
  3 train/test split 11.1      6 10.5  Preprocessor1_Model1
  4 train/test split 13.1      7 15.6  Preprocessor1_Model1
  5 train/test split 12.0      8 12.4  Preprocessor1_Model1
  6 train/test split 10.5      9 11.1  Preprocessor1_Model1
  7 train/test split 11.2     14 11.8  Preprocessor1_Model1
  8 train/test split 11.2     16 10.0  Preprocessor1_Model1
  9 train/test split 11.6     18 12.0  Preprocessor1_Model1
 10 train/test split 11.9     20 13.2  Preprocessor1_Model1
 11 train/test split 11.2     24 11.7  Preprocessor1_Model1
 12 train/test split  8.02    26  6.91 Preprocessor1_Model1
 13 train/test split  8.90    27  5.93 Preprocessor1_Model1
 14 train/test split  9.52    28 10.5  Preprocessor1_Model1
 15 train/test split  8.11    33  5.53 Preprocessor1_Model1
 16 train/test split 10.7     37  7.47 Preprocessor1_Model1
 17 train/test split 10.9     42 11.2  Preprocessor1_Model1
 18 train/test split 11.1     44 11.7  Preprocessor1_Model1
 19 train/test split 10.9     46 11.2  Preprocessor1_Model1
 20 train/test split 10.9     47 11.2  Preprocessor1_Model1
 21 train/test split 10.9     48 11.1  Preprocessor1_Model1
 22 train/test split 11.2     50 11.3  Preprocessor1_Model1
 23 train/test split 11.5     52 12.1  Preprocessor1_Model1
 24 train/test split 11.1     56 10.7  Preprocessor1_Model1
 25 train/test split 11.0     62 10.2  Preprocessor1_Model1
 26 train/test split 17.4     65 18.7  Preprocessor1_Model1
 27 train/test split 11.9     68  7.6  Preprocessor1_Model1
 28 train/test split 12.4     69  8.22 Preprocessor1_Model1
 29 train/test split 12.5     72  8.08 Preprocessor1_Model1
 30 train/test split  9.24    73  6.70 Preprocessor1_Model1
 31 train/test split 17.7     78 23.2  Preprocessor1_Model1
 32 train/test split 10.6     80  9.03 Preprocessor1_Model1
 33 train/test split 13.0     81 14.2  Preprocessor1_Model1
 34 train/test split 13.5     82 13.9  Preprocessor1_Model1
 35 train/test split 13.6     89 13.7  Preprocessor1_Model1
 36 train/test split 12.6     90  7.12 Preprocessor1_Model1
 37 train/test split 10.5     92 16.7  Preprocessor1_Model1
 38 train/test split 11.9     93  7.17 Preprocessor1_Model1
 39 train/test split 10.8     94 14.3  Preprocessor1_Model1
 40 train/test split 12.6     96 12.8  Preprocessor1_Model1
 41 train/test split 11.3     99 15.3  Preprocessor1_Model1
 42 train/test split 10.3    109  6.94 Preprocessor1_Model1
 43 train/test split 15.0    110 15.8  Preprocessor1_Model1
 44 train/test split 14.0    111  8.64 Preprocessor1_Model1
 45 train/test split 14.7    114 13.3  Preprocessor1_Model1
 46 train/test split 13.5    115 12.3  Preprocessor1_Model1
 47 train/test split 13.9    117 11.3  Preprocessor1_Model1
 48 train/test split 13.6    119 13.7  Preprocessor1_Model1
 49 train/test split 12.0    129 12.3  Preprocessor1_Model1
 50 train/test split 11.1    130  6.77 Preprocessor1_Model1
 51 train/test split 12.0    134  9.08 Preprocessor1_Model1
 52 train/test split 11.7    137 20.6  Preprocessor1_Model1
 53 train/test split  8.99   139  9.69 Preprocessor1_Model1
 54 train/test split 11.0    140 10.7  Preprocessor1_Model1
 55 train/test split  7.89   144  7.18 Preprocessor1_Model1
 56 train/test split  9.10   148  8.22 Preprocessor1_Model1
 57 train/test split  7.37   151  4.32 Preprocessor1_Model1
 58 train/test split  7.17   153  6.68 Preprocessor1_Model1
 59 train/test split  7.72   154  9.11 Preprocessor1_Model1
 60 train/test split 10.9    158 11.9  Preprocessor1_Model1
 61 train/test split 11.3    167 11.3  Preprocessor1_Model1
 62 train/test split 11.4    172 11.4  Preprocessor1_Model1
 63 train/test split 12.2    173 11.3  Preprocessor1_Model1
 64 train/test split 12.4    175 11.5  Preprocessor1_Model1
 65 train/test split 12.8    177 13.5  Preprocessor1_Model1
 66 train/test split 11.8    180 12.4  Preprocessor1_Model1
 67 train/test split  8.02   185  7.56 Preprocessor1_Model1
 68 train/test split  8.79   193  8.55 Preprocessor1_Model1
 69 train/test split  9.06   196 10.6  Preprocessor1_Model1
 70 train/test split  7.94   197  7.97 Preprocessor1_Model1
 71 train/test split  8.21   200  7.55 Preprocessor1_Model1
 72 train/test split  8.86   202  6.20 Preprocessor1_Model1
 73 train/test split  7.98   209  6.93 Preprocessor1_Model1
 74 train/test split 12.0    213 11.9  Preprocessor1_Model1
 75 train/test split 11.9    217 13.7  Preprocessor1_Model1
 76 train/test split 12.4    219 12.7  Preprocessor1_Model1
 77 train/test split 12.1    220 13.1  Preprocessor1_Model1
 78 train/test split 12.5    230 13.5  Preprocessor1_Model1
 79 train/test split 12.4    232 13.2  Preprocessor1_Model1
 80 train/test split 11.6    233 11.9  Preprocessor1_Model1
 81 train/test split 12.3    236 12.5  Preprocessor1_Model1
 82 train/test split  9.09   239  7.91 Preprocessor1_Model1
 83 train/test split  9.34   240  7.72 Preprocessor1_Model1
 84 train/test split  8.69   242  9.05 Preprocessor1_Model1
 85 train/test split  8.11   244  9.72 Preprocessor1_Model1
 86 train/test split  8.39   245 11.7  Preprocessor1_Model1
 87 train/test split 11.1    246  9.18 Preprocessor1_Model1
 88 train/test split 11.2    247 10.5  Preprocessor1_Model1
 89 train/test split 12.4    253 11.9  Preprocessor1_Model1
 90 train/test split 12.7    257 12.0  Preprocessor1_Model1
 91 train/test split 11.4    261 11.3  Preprocessor1_Model1
 92 train/test split 11.3    264 10.8  Preprocessor1_Model1
 93 train/test split 11.4    266  9.36 Preprocessor1_Model1
 94 train/test split 11.1    268 10.1  Preprocessor1_Model1
 95 train/test split 11.9    273 12.4  Preprocessor1_Model1
 96 train/test split 11.1    282 10.4  Preprocessor1_Model1
 97 train/test split 11.6    283 10.7  Preprocessor1_Model1
 98 train/test split 12.2    287 12.2  Preprocessor1_Model1
 99 train/test split 12.3    288 12.5  Preprocessor1_Model1
100 train/test split 11.7    290 12.3  Preprocessor1_Model1
101 train/test split 12.6    300 12.2  Preprocessor1_Model1
102 train/test split 12.5    301 12.9  Preprocessor1_Model1
103 train/test split 11.8    304 12.2  Preprocessor1_Model1
104 train/test split 11.9    311 11.3  Preprocessor1_Model1
105 train/test split 11.9    312 12.4  Preprocessor1_Model1
106 train/test split 12.3    314 11.6  Preprocessor1_Model1
107 train/test split 12.3    315 12.3  Preprocessor1_Model1
108 train/test split 12.2    316 12.5  Preprocessor1_Model1
109 train/test split 12.1    317 12.5  Preprocessor1_Model1
110 train/test split 10.5    320 10.4  Preprocessor1_Model1
111 train/test split 11.4    322 11.0  Preprocessor1_Model1
112 train/test split  9.14   323 10.2  Preprocessor1_Model1
113 train/test split 10.6    325 11.4  Preprocessor1_Model1
114 train/test split 11.0    333 10.3  Preprocessor1_Model1
115 train/test split 11.2    336 13.5  Preprocessor1_Model1
116 train/test split  9.45   337  9.27 Preprocessor1_Model1
117 train/test split  9.12   339  9.15 Preprocessor1_Model1
118 train/test split  9.02   342  9.75 Preprocessor1_Model1
119 train/test split 12.8    351 12.1  Preprocessor1_Model1
120 train/test split 11.3    355 12.0  Preprocessor1_Model1
121 train/test split 12.3    356 12.0  Preprocessor1_Model1
122 train/test split 11.9    359 11.5  Preprocessor1_Model1
123 train/test split 12.1    360 12.2  Preprocessor1_Model1
124 train/test split 12.9    366 12.0  Preprocessor1_Model1
125 train/test split 12.4    368 10.5  Preprocessor1_Model1
126 train/test split 12.1    369 12.0  Preprocessor1_Model1
127 train/test split 11.5    370 10.6  Preprocessor1_Model1
128 train/test split  9.50   373  8.76 Preprocessor1_Model1
129 train/test split 10.6    374  9.25 Preprocessor1_Model1
130 train/test split 10.6    376  9.66 Preprocessor1_Model1
131 train/test split 10.3    377 11.6  Preprocessor1_Model1
132 train/test split 10.3    385 10.6  Preprocessor1_Model1
133 train/test split  9.56   386 10.3  Preprocessor1_Model1
134 train/test split 11.1    391 11.7  Preprocessor1_Model1
135 train/test split 12.9    392 12.6  Preprocessor1_Model1
136 train/test split 11.5    393 13.2  Preprocessor1_Model1
137 train/test split 11.8    396 12.5  Preprocessor1_Model1
138 train/test split 12.1    401 12.5  Preprocessor1_Model1
139 train/test split  9.99   406  9.05 Preprocessor1_Model1
140 train/test split  9.90   407  8.71 Preprocessor1_Model1
141 train/test split  9.78   411 10.8  Preprocessor1_Model1
142 train/test split  9.79   412 10.8  Preprocessor1_Model1
143 train/test split 10.5    415 11.2  Preprocessor1_Model1
144 train/test split 10.5    416 10.7  Preprocessor1_Model1
145 train/test split 11.3    422  8.94 Preprocessor1_Model1
146 train/test split 10.5    423  9.90 Preprocessor1_Model1
147 train/test split 10.8    425 10.6  Preprocessor1_Model1
148 train/test split  9.42   435  6.52 Preprocessor1_Model1
149 train/test split 11.6    436 11.4  Preprocessor1_Model1
150 train/test split 10.6    437  9.59 Preprocessor1_Model1
151 train/test split 11.1    438 10.9  Preprocessor1_Model1
152 train/test split 12.2    443 12.9  Preprocessor1_Model1
153 train/test split 10.9    446 11.0  Preprocessor1_Model1
154 train/test split 11.9    447 13.4  Preprocessor1_Model1
155 train/test split  7.31   451  4.74 Preprocessor1_Model1
156 train/test split  9.88   452  9.49 Preprocessor1_Model1
157 train/test split 10.2    453  9.45 Preprocessor1_Model1
158 train/test split  9.65   458 10.2  Preprocessor1_Model1
159 train/test split  7.84   464  7.81 Preprocessor1_Model1
160 train/test split 10.2    467  8.40 Preprocessor1_Model1
161 train/test split 11.2    472 10.3  Preprocessor1_Model1
162 train/test split 11.2    473  9.80 Preprocessor1_Model1
163 train/test split 11.7    477 12.4  Preprocessor1_Model1
164 train/test split 11.4    483 10.1  Preprocessor1_Model1
165 train/test split 12.7    490 12.9  Preprocessor1_Model1
166 train/test split 12.6    492 13.4  Preprocessor1_Model1
167 train/test split  8.12   493  5.37 Preprocessor1_Model1
168 train/test split  8.64   494  8.89 Preprocessor1_Model1
169 train/test split  8.44   496  7.91 Preprocessor1_Model1
170 train/test split  7.83   503  7.85 Preprocessor1_Model1
171 train/test split  8.45   506  7.19 Preprocessor1_Model1
172 train/test split  8.27   507 10.2  Preprocessor1_Model1
173 train/test split  9.34   509  9.18 Preprocessor1_Model1
174 train/test split  8.46   514  6.78 Preprocessor1_Model1
175 train/test split  8.93   515  8.22 Preprocessor1_Model1
176 train/test split 10.5    516  9.28 Preprocessor1_Model1
177 train/test split  8.90   517  4.52 Preprocessor1_Model1
178 train/test split  9.22   524  8.71 Preprocessor1_Model1
179 train/test split  9.06   525  8.20 Preprocessor1_Model1
180 train/test split 10.4    527 10.2  Preprocessor1_Model1
181 train/test split 11.0    528 10.8  Preprocessor1_Model1
182 train/test split 12.6    532 12.3  Preprocessor1_Model1
183 train/test split 12.6    533 12.7  Preprocessor1_Model1
184 train/test split 12.2    535 13.0  Preprocessor1_Model1
185 train/test split 12.6    536 11.6  Preprocessor1_Model1
186 train/test split 11.2    549 11.2  Preprocessor1_Model1
187 train/test split  8.11   550  5.98 Preprocessor1_Model1
188 train/test split  7.42   553 12.4  Preprocessor1_Model1
189 train/test split  7.67   555  5.07 Preprocessor1_Model1
190 train/test split  6.96   556  6.21 Preprocessor1_Model1
191 train/test split  7.44   558  5.88 Preprocessor1_Model1
192 train/test split  7.85   559  4.51 Preprocessor1_Model1
193 train/test split 10.5    566 10.7  Preprocessor1_Model1
194 train/test split  7.58   568  4.30 Preprocessor1_Model1
195 train/test split  9.42   575  8.15 Preprocessor1_Model1
196 train/test split 11.6    579 10.7  Preprocessor1_Model1
197 train/test split  8.85   581  7.91 Preprocessor1_Model1
198 train/test split 10.4    583 11.0  Preprocessor1_Model1
199 train/test split 11.8    589 12.3  Preprocessor1_Model1
200 train/test split 10.9    603 11.1  Preprocessor1_Model1
201 train/test split 12.2    604 12.4  Preprocessor1_Model1
202 train/test split 11.4    610 11.8  Preprocessor1_Model1
203 train/test split 11.5    611 11.9  Preprocessor1_Model1
204 train/test split 10.3    614 11.2  Preprocessor1_Model1
205 train/test split 11.9    616 11.4  Preprocessor1_Model1
206 train/test split 10.4    617  9.15 Preprocessor1_Model1
207 train/test split  8.65   619  4.57 Preprocessor1_Model1
208 train/test split  8.84   621  8.33 Preprocessor1_Model1
209 train/test split  8.22   622  6.52 Preprocessor1_Model1
210 train/test split 12.6    625 13.8  Preprocessor1_Model1
211 train/test split 12.0    626 12.8  Preprocessor1_Model1
212 train/test split 13.0    628 13.2  Preprocessor1_Model1
213 train/test split 12.9    630 14.0  Preprocessor1_Model1
214 train/test split 13.3    631 13.6  Preprocessor1_Model1
215 train/test split 12.9    633 14.6  Preprocessor1_Model1
216 train/test split 13.0    636 12.4  Preprocessor1_Model1
217 train/test split 13.3    640 15.2  Preprocessor1_Model1
218 train/test split 12.6    650 11.4  Preprocessor1_Model1
219 train/test split 12.6    652 12.1  Preprocessor1_Model1
220 train/test split 12.6    653 12.3  Preprocessor1_Model1
221 train/test split 12.9    655 13.1  Preprocessor1_Model1
222 train/test split 12.7    657 13.4  Preprocessor1_Model1
223 train/test split 13.1    662 12.5  Preprocessor1_Model1
224 train/test split 12.6    664 13.0  Preprocessor1_Model1
225 train/test split 11.3    669 11.6  Preprocessor1_Model1
226 train/test split 10.8    672 10.0  Preprocessor1_Model1
227 train/test split 10.7    674 11.1  Preprocessor1_Model1
228 train/test split  9.67   677  5.22 Preprocessor1_Model1
229 train/test split 10.3    680  6.92 Preprocessor1_Model1
230 train/test split 10.4    684  8.42 Preprocessor1_Model1
231 train/test split 10.4    686  7.31 Preprocessor1_Model1
232 train/test split 11.0    689  8.36 Preprocessor1_Model1
233 train/test split 10.8    690  7.74 Preprocessor1_Model1
234 train/test split  9.42   691  8.28 Preprocessor1_Model1
235 train/test split  9.78   692  6.72 Preprocessor1_Model1
236 train/test split  9.83   693  9.75 Preprocessor1_Model1
237 train/test split 11.6    694 11.4  Preprocessor1_Model1
238 train/test split 13.2    695 13.0  Preprocessor1_Model1
239 train/test split 12.8    698 13.5  Preprocessor1_Model1
240 train/test split 12.8    702 12.4  Preprocessor1_Model1
241 train/test split 13.2    703 12.5  Preprocessor1_Model1
242 train/test split 12.4    704 13.9  Preprocessor1_Model1
243 train/test split 11.7    705 10.8  Preprocessor1_Model1
244 train/test split 13.4    709 13.9  Preprocessor1_Model1
245 train/test split 11.6    710 10.7  Preprocessor1_Model1
246 train/test split 11.9    711 10.1  Preprocessor1_Model1
247 train/test split 12.3    714 11.8  Preprocessor1_Model1
248 train/test split 12.5    715 12.3  Preprocessor1_Model1
249 train/test split 13.4    718 13.3  Preprocessor1_Model1
250 train/test split 13.2    720 13.1  Preprocessor1_Model1
251 train/test split 12.0    722 12.4  Preprocessor1_Model1
252 train/test split 11.6    723 11.0  Preprocessor1_Model1
253 train/test split 13.1    725 13.4  Preprocessor1_Model1
254 train/test split  9.74   728 10.7  Preprocessor1_Model1
255 train/test split  9.56   730  9.03 Preprocessor1_Model1
256 train/test split 11.0    734 12.2  Preprocessor1_Model1
257 train/test split 11.4    735 11.4  Preprocessor1_Model1
258 train/test split 11.7    741 12.4  Preprocessor1_Model1
259 train/test split  7.37   749  5.25 Preprocessor1_Model1
260 train/test split  8.09   752  7.74 Preprocessor1_Model1
261 train/test split  8.17   753  6.72 Preprocessor1_Model1
262 train/test split 11.8    756 12.6  Preprocessor1_Model1
263 train/test split 10.8    759 11.2  Preprocessor1_Model1
264 train/test split 11.2    762 11.5  Preprocessor1_Model1
265 train/test split 10.4    765  9.27 Preprocessor1_Model1
266 train/test split 10.7    768 11.9  Preprocessor1_Model1
267 train/test split 10.6    769 16.2  Preprocessor1_Model1
268 train/test split 11.8    770 11.8  Preprocessor1_Model1
269 train/test split 11.4    771 10.8  Preprocessor1_Model1
270 train/test split  9.36   785  9.70 Preprocessor1_Model1
271 train/test split  9.41   797  8.65 Preprocessor1_Model1
272 train/test split  9.14   800  7.76 Preprocessor1_Model1
273 train/test split 11.8    805 11.3  Preprocessor1_Model1
274 train/test split 11.4    807 11.2  Preprocessor1_Model1
275 train/test split 11.0    809 12.1  Preprocessor1_Model1
276 train/test split 10.8    814 11.5  Preprocessor1_Model1
277 train/test split 10.9    815 10.6  Preprocessor1_Model1
278 train/test split 12.0    818 13.7  Preprocessor1_Model1
279 train/test split 11.2    820 11.7  Preprocessor1_Model1
280 train/test split 12.3    830 14.2  Preprocessor1_Model1
281 train/test split 12.5    837 14.2  Preprocessor1_Model1
282 train/test split 12.1    838 13.7  Preprocessor1_Model1
283 train/test split 12.9    843 13.9  Preprocessor1_Model1
284 train/test split 11.2    845 11.8  Preprocessor1_Model1
285 train/test split 10.6    851 11.8  Preprocessor1_Model1
286 train/test split 12.0    855 12.7  Preprocessor1_Model1
287 train/test split 11.9    856 13.3  Preprocessor1_Model1
288 train/test split 12.0    857 13.1  Preprocessor1_Model1
289 train/test split 10.7    859 11.3  Preprocessor1_Model1
290 train/test split 11.3    864 13.5  Preprocessor1_Model1
291 train/test split  8.23   870  4.5  Preprocessor1_Model1
292 train/test split  6.62   874  6.17 Preprocessor1_Model1

Nice!

Data Visualization

If you have been following along but stopped and are starting here you could load the wrangled data using the following command:

load(here::here("data", "wrangled", "wrangled_pm.rda"))

If you skipped previous sections click here for more information on how to obtain and load the data.

First you need to install the OCSdata package:

install.packages("OCSdata")

Then, you may download and load the wrangled data .rda file using the following code:

OCSdata::wrangled_rda("ocs-bp-air-pollution", outpath = getwd())
load(here::here("OCSdata", "data", "wrangled", "wrangled_pm.rda"))

If the package does not work for you, you may also download this .rda file by clicking this link here.

To load the downloaded data into your environment, you may double click on the .rda file in Rstudio or using the load() function.

load(here::here("data", "wrangled", "wrangled_pm.rda"))

Click here to see more about creating new projects in RStudio.

You can create a project by going to the File menu of RStudio like so:

You can also do so by clicking the project button:

See here to learn more about using RStudio projects and here to learn more about the here package.

Our main question for this case study was:

Can we predict regional annual average air pollution concentrations by zip-code using predictors such as population density, urbanization, road density, as well as, satellite pollution data and chemical modeling data?

Thus far, we have built a machine learning (ML) model to predict fine particulate matter air pollution levels based on our predictor variables (or features).

Now, let’s make a plot of our predicted outcome values ($\hat{Y}$) and actual outcome values $Y$ we observed.

First, let’s start by making a plot of our monitors. To do this, we will use the following packages to create a map of the US:

sf - the simple features package helps to convert geographical coordinates into geometry variables which are useful for making 2D plots
maps - this package contains geographical outlines and plotting functions to create plots with maps
rnaturalearth- this allows for easy interaction with map data from Natural Earth which is a public domain map dataset
rgeos - this package interfaces with the Geometry Engine-Open Source (GEOS) which is also helpful for coordinate conversion

We will start with getting an outline of the US with the ne_countries() function of the rnaturalearth package which will return polygons of the countries in the Natural Earth dataset.

world <- ne_countries(scale = "medium", returnclass = "sf")
glimpse(world)

Rows: 241
Columns: 64
$ scalerank  <int> 3, 1, 1, 1, 1, 3, 3, 1, 1, 1, 3, 1, 5, 3, 1, 1, 1, 1, 1, 1,…
$ featurecla <chr> "Admin-0 country", "Admin-0 country", "Admin-0 country", "A…
$ labelrank  <dbl> 5, 3, 3, 6, 6, 6, 6, 4, 2, 6, 4, 4, 5, 6, 6, 2, 4, 5, 6, 2,…
$ sovereignt <chr> "Netherlands", "Afghanistan", "Angola", "United Kingdom", "…
$ sov_a3     <chr> "NL1", "AFG", "AGO", "GB1", "ALB", "FI1", "AND", "ARE", "AR…
$ adm0_dif   <dbl> 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0,…
$ level      <dbl> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,…
$ type       <chr> "Country", "Sovereign country", "Sovereign country", "Depen…
$ admin      <chr> "Aruba", "Afghanistan", "Angola", "Anguilla", "Albania", "A…
$ adm0_a3    <chr> "ABW", "AFG", "AGO", "AIA", "ALB", "ALD", "AND", "ARE", "AR…
$ geou_dif   <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ geounit    <chr> "Aruba", "Afghanistan", "Angola", "Anguilla", "Albania", "A…
$ gu_a3      <chr> "ABW", "AFG", "AGO", "AIA", "ALB", "ALD", "AND", "ARE", "AR…
$ su_dif     <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ subunit    <chr> "Aruba", "Afghanistan", "Angola", "Anguilla", "Albania", "A…
$ su_a3      <chr> "ABW", "AFG", "AGO", "AIA", "ALB", "ALD", "AND", "ARE", "AR…
$ brk_diff   <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,…
$ name       <chr> "Aruba", "Afghanistan", "Angola", "Anguilla", "Albania", "A…
$ name_long  <chr> "Aruba", "Afghanistan", "Angola", "Anguilla", "Albania", "A…
$ brk_a3     <chr> "ABW", "AFG", "AGO", "AIA", "ALB", "ALD", "AND", "ARE", "AR…
$ brk_name   <chr> "Aruba", "Afghanistan", "Angola", "Anguilla", "Albania", "A…
$ brk_group  <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ abbrev     <chr> "Aruba", "Afg.", "Ang.", "Ang.", "Alb.", "Aland", "And.", "…
$ postal     <chr> "AW", "AF", "AO", "AI", "AL", "AI", "AND", "AE", "AR", "ARM…
$ formal_en  <chr> "Aruba", "Islamic State of Afghanistan", "People's Republic…
$ formal_fr  <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ note_adm0  <chr> "Neth.", NA, NA, "U.K.", NA, "Fin.", NA, NA, NA, NA, "U.S.A…
$ note_brk   <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "Multiple claim…
$ name_sort  <chr> "Aruba", "Afghanistan", "Angola", "Anguilla", "Albania", "A…
$ name_alt   <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ mapcolor7  <dbl> 4, 5, 3, 6, 1, 4, 1, 2, 3, 3, 4, 4, 1, 7, 2, 1, 3, 1, 2, 3,…
$ mapcolor8  <dbl> 2, 6, 2, 6, 4, 1, 4, 1, 1, 1, 5, 5, 2, 5, 2, 2, 1, 6, 2, 2,…
$ mapcolor9  <dbl> 2, 8, 6, 6, 1, 4, 1, 3, 3, 2, 1, 1, 2, 9, 5, 2, 3, 5, 5, 1,…
$ mapcolor13 <dbl> 9, 7, 1, 3, 6, 6, 8, 3, 13, 10, 1, NA, 7, 11, 5, 7, 4, 8, 8…
$ pop_est    <dbl> 103065, 28400000, 12799293, 14436, 3639453, 27153, 83888, 4…
$ gdp_md_est <dbl> 2258.0, 22270.0, 110300.0, 108.9, 21810.0, 1563.0, 3660.0, …
$ pop_year   <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ lastcensus <dbl> 2010, 1979, 1970, NA, 2001, NA, 1989, 2010, 2010, 2001, 201…
$ gdp_year   <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ economy    <chr> "6. Developing region", "7. Least developed region", "7. Le…
$ income_grp <chr> "2. High income: nonOECD", "5. Low income", "3. Upper middl…
$ wikipedia  <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ fips_10    <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ iso_a2     <chr> "AW", "AF", "AO", "AI", "AL", "AX", "AD", "AE", "AR", "AM",…
$ iso_a3     <chr> "ABW", "AFG", "AGO", "AIA", "ALB", "ALA", "AND", "ARE", "AR…
$ iso_n3     <chr> "533", "004", "024", "660", "008", "248", "020", "784", "03…
$ un_a3      <chr> "533", "004", "024", "660", "008", "248", "020", "784", "03…
$ wb_a2      <chr> "AW", "AF", "AO", NA, "AL", NA, "AD", "AE", "AR", "AM", "AS…
$ wb_a3      <chr> "ABW", "AFG", "AGO", NA, "ALB", NA, "ADO", "ARE", "ARG", "A…
$ woe_id     <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ adm0_a3_is <chr> "ABW", "AFG", "AGO", "AIA", "ALB", "ALA", "AND", "ARE", "AR…
$ adm0_a3_us <chr> "ABW", "AFG", "AGO", "AIA", "ALB", "ALD", "AND", "ARE", "AR…
$ adm0_a3_un <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ adm0_a3_wb <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…
$ continent  <chr> "North America", "Asia", "Africa", "North America", "Europe…
$ region_un  <chr> "Americas", "Asia", "Africa", "Americas", "Europe", "Europe…
$ subregion  <chr> "Caribbean", "Southern Asia", "Middle Africa", "Caribbean",…
$ region_wb  <chr> "Latin America & Caribbean", "South Asia", "Sub-Saharan Afr…
$ name_len   <dbl> 5, 11, 6, 8, 7, 5, 7, 20, 9, 7, 14, 10, 23, 22, 17, 9, 7, 1…
$ long_len   <dbl> 5, 11, 6, 8, 7, 13, 7, 20, 9, 7, 14, 10, 27, 35, 19, 9, 7, …
$ abbrev_len <dbl> 5, 4, 4, 4, 4, 5, 4, 6, 4, 4, 9, 4, 7, 10, 6, 4, 5, 4, 4, 5…
$ tiny       <dbl> 4, NA, NA, NA, NA, 5, 5, NA, NA, NA, 3, NA, NA, 2, 4, NA, N…
$ homepart   <dbl> NA, 1, 1, NA, 1, NA, 1, 1, 1, 1, NA, 1, NA, NA, 1, 1, 1, 1,…
$ geometry   <MULTIPOLYGON [°]> MULTIPOLYGON (((-69.89912 1..., MULTIPOLYGON (…

Here you can see the data about the countries in the world. Notice the geometry variable. This is used to create the outlines that we want.

Now we can use the geom_sf() function of the ggplot2 package to create a visual of simple feature (the geometry coordinates found in the geometry variable).

ggplot(data = world) +
    geom_sf()

So now we can see that we have outlines of all the countries in the world.

We want to limit this just to the coordinates for the US. We will do this based on the coordinates we found on Wikipedia. According to this link, these are the latitude and longitude bounds of the continental US:

top = 49.3457868 # north lat
left = -124.7844079 # west long
right = -66.9513812 # east long
bottom = 24.7433195 # south lat

ggplot(data = world) +
    geom_sf() +
    coord_sf(xlim = c(-125, -66), ylim = c(24.5, 50), 
             expand = FALSE)

Now we just have a plot that is mostly limited to the outline of the US.

Now we will use the geom_point() function of the ggplot package to add scatter plot on top of the map. We want to show where the monitors are located based on the latitude and longitude values in the data.

ggplot(data = world) +
    geom_sf() +
    coord_sf(xlim = c(-125, -66), ylim = c(24.5, 50), 
             expand = FALSE)+
    geom_point(data = pm, aes(x = lon, y = lat), size = 2, 
               shape = 23, fill = "darkred")

Nice!

Now let’s add county lines.

County graphical data is available from the maps package. The sf package which again is short for simple features creates a data frame about this graphical data so that we can work with it.

counties <- sf::st_as_sf(maps::map("county", plot = FALSE,
                                   fill = TRUE))

counties

Simple feature collection with 3076 features and 1 field
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -124.6813 ymin: 25.12993 xmax: -67.00742 ymax: 49.38323
Geodetic CRS:  WGS 84
First 10 features:
                 ID                           geom
1   alabama,autauga MULTIPOLYGON (((-86.50517 3...
2   alabama,baldwin MULTIPOLYGON (((-87.93757 3...
3   alabama,barbour MULTIPOLYGON (((-85.42801 3...
4      alabama,bibb MULTIPOLYGON (((-87.02083 3...
5    alabama,blount MULTIPOLYGON (((-86.9578 33...
6   alabama,bullock MULTIPOLYGON (((-85.66866 3...
7    alabama,butler MULTIPOLYGON (((-86.8604 31...
8   alabama,calhoun MULTIPOLYGON (((-85.74313 3...
9  alabama,chambers MULTIPOLYGON (((-85.59416 3...
10 alabama,cherokee MULTIPOLYGON (((-85.46812 3...

Now we will use this data within the geom_sf() function to add this to our plot. We will also add a title using the ggtitle() function, as well as remove axis ticks and titles using the theme() function of the ggplot2 package.

monitors <- ggplot(data = world) +
    geom_sf(data = counties, fill = NA, color = gray(.5))+
      coord_sf(xlim = c(-125, -66), ylim = c(24.5, 50), 
             expand = FALSE) +
    geom_point(data = pm, aes(x = lon, y = lat), size = 2, 
               shape = 23, fill = "darkred") +
    ggtitle("Monitor Locations") +
    theme(axis.title.x=element_blank(),
          axis.text.x = element_blank(),
          axis.ticks.x = element_blank(),
          axis.title.y = element_blank(),
          axis.text.y = element_blank(),
          axis.ticks.y = element_blank())

monitors

Great!

Now, let’s add a fill at the county-level for the true monitor values of air pollution.

First, we need to get the county map data that we just got and our air pollution data to have similarly formatted county names so that we can combine the datasets together.

We can see that in the county data the counties are listed after the state name and a comma. In addition they are all lower case.

head(counties)

Simple feature collection with 6 features and 1 field
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -88.01778 ymin: 30.24071 xmax: -85.06131 ymax: 34.2686
Geodetic CRS:  WGS 84
               ID                           geom
1 alabama,autauga MULTIPOLYGON (((-86.50517 3...
2 alabama,baldwin MULTIPOLYGON (((-87.93757 3...
3 alabama,barbour MULTIPOLYGON (((-85.42801 3...
4    alabama,bibb MULTIPOLYGON (((-87.02083 3...
5  alabama,blount MULTIPOLYGON (((-86.9578 33...
6 alabama,bullock MULTIPOLYGON (((-85.66866 3...

In contrast, our air pollution pm data shows counties as titles with the first letter as upper case.

dplyr::pull(pm, county) %>%
  head()

[1] "Baldwin" "Clay"    "Colbert" "DeKalb"  "Etowah"  "Houston"

We can use the separate() function of the tidyr package to separate the ID variable of our counties data into two variables based on the comma as a separator.

counties %<>% 
  tidyr::separate(ID, into = c("state", "county"), sep = ",")

head(counties)

Simple feature collection with 6 features and 2 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -88.01778 ymin: 30.24071 xmax: -85.06131 ymax: 34.2686
Geodetic CRS:  WGS 84
    state  county                           geom
1 alabama autauga MULTIPOLYGON (((-86.50517 3...
2 alabama baldwin MULTIPOLYGON (((-87.93757 3...
3 alabama barbour MULTIPOLYGON (((-85.42801 3...
4 alabama    bibb MULTIPOLYGON (((-87.02083 3...
5 alabama  blount MULTIPOLYGON (((-86.9578 33...
6 alabama bullock MULTIPOLYGON (((-85.66866 3...

Now we just need to make these names in the new county variable of the counties data to be in title format. We can use the str_to_title() function of the stringr package to do this.

counties[["county"]] <- stringr::str_to_title(counties[["county"]])

Great! Now the county information is the same for the counties and pm data.

We can use the inner_join() function of the dplyr package to join the datasets together based on the county variables in each. This function will keep all rows that are in both datasets.

map_data <- dplyr::inner_join(counties, pm, by = "county")

glimpse(map_data)

Rows: 3,926
Columns: 52
$ state.x                     <chr> "alabama", "alabama", "alabama", "alabama"…
$ county                      <chr> "Baldwin", "Bibb", "Bibb", "Butler", "Butl…
$ id                          <fct> 1003.001, 13021.0007, 13021.0012, 39017.00…
$ value                       <dbl> 9.597647, 12.253134, 12.233673, 13.918079,…
$ fips                        <fct> 1003, 13021, 13021, 39017, 39017, 13059, 1…
$ lat                         <dbl> 30.49800, 32.77746, 32.80541, 39.49380, 39…
$ lon                         <dbl> -87.88141, -83.64110, -83.54352, -84.35430…
$ state.y                     <chr> "Alabama", "Georgia", "Georgia", "Ohio", "…
$ city                        <chr> "Fairhope", "Macon", "Macon", "Middletown"…
$ CMAQ                        <dbl> 8.098836, 11.716801, 11.716801, 11.321991,…
$ zcta                        <fct> 36532, 31206, 31020, 45044, 45015, 30605, …
$ zcta_area                   <dbl> 190980522, 72325015, 276913325, 98746815, …
$ zcta_pop                    <dbl> 27829, 29072, 2541, 52822, 12038, 39952, 5…
$ imp_a500                    <dbl> 0.01730104, 35.64359862, 0.28200692, 27.44…
$ imp_a1000                   <dbl> 1.4096021, 24.4824827, 0.2973616, 28.28871…
$ imp_a5000                   <dbl> 3.3360118, 8.7317283, 2.4691097, 22.697407…
$ imp_a10000                  <dbl> 1.9879187, 9.1999720, 1.9873487, 11.901559…
$ imp_a15000                  <dbl> 1.4386207, 6.4619966, 3.6435089, 8.4052921…
$ county_area                 <dbl> 4117521611, 646879637, 646879637, 12096684…
$ county_pop                  <dbl> 182265, 155547, 155547, 368130, 368130, 11…
$ log_dist_to_prisec          <dbl> 4.648181, 7.635438, 7.576493, 5.959728, 5.…
$ log_pri_length_5000         <dbl> 8.517193, 10.215058, 10.659655, 9.747731, …
$ log_pri_length_10000        <dbl> 9.210340, 12.116408, 11.653566, 10.734382,…
$ log_pri_length_15000        <dbl> 9.630228, 12.591833, 12.313347, 11.172817,…
$ log_pri_length_25000        <dbl> 11.32735, 13.12475, 13.02383, 12.14238, 12…
$ log_prisec_length_500       <dbl> 7.295356, 6.214608, 6.214608, 8.104926, 7.…
$ log_prisec_length_1000      <dbl> 8.195119, 7.600902, 7.600902, 9.180109, 8.…
$ log_prisec_length_5000      <dbl> 10.81504, 11.61283, 11.34202, 11.63677, 11…
$ log_prisec_length_10000     <dbl> 11.88680, 13.23991, 12.47152, 12.62117, 12…
$ log_prisec_length_15000     <dbl> 12.20572, 13.74532, 13.37704, 13.26170, 13…
$ log_prisec_length_25000     <dbl> 13.41395, 14.28483, 14.25295, 14.16745, 14…
$ log_nei_2008_pm25_sum_10000 <dbl> 0.3180354, 5.5380414, 0.1972941, 7.2307845…
$ log_nei_2008_pm25_sum_15000 <dbl> 1.967359, 5.538041, 5.536822, 7.408732, 5.…
$ log_nei_2008_pm25_sum_25000 <dbl> 5.067308, 5.986616, 5.986572, 7.538614, 7.…
$ log_nei_2008_pm10_sum_10000 <dbl> 1.3558851, 5.6035407, 0.9849611, 7.2785763…
$ log_nei_2008_pm10_sum_15000 <dbl> 2.2678341, 5.6035407, 5.5971561, 7.4899282…
$ log_nei_2008_pm10_sum_25000 <dbl> 5.6287284, 6.0302900, 6.0347672, 7.6330042…
$ popdens_county              <dbl> 44.265706, 240.457407, 240.457407, 304.323…
$ popdens_zcta                <dbl> 145.716431, 401.963277, 9.176156, 534.9235…
$ nohs                        <dbl> 3.300, 8.700, 6.800, 1.300, 3.500, 3.800, …
$ somehs                      <dbl> 4.900, 24.700, 17.500, 3.300, 7.700, 6.800…
$ hs                          <dbl> 25.10, 32.60, 46.60, 33.40, 33.40, 17.80, …
$ somecollege                 <dbl> 19.7, 12.9, 18.7, 23.0, 23.2, 16.1, 18.8, …
$ associate                   <dbl> 8.200, 4.800, 5.000, 8.700, 8.100, 5.100, …
$ bachelor                    <dbl> 25.30, 8.90, 4.00, 22.20, 16.20, 25.40, 5.…
$ grad                        <dbl> 13.50, 7.30, 1.30, 8.10, 7.90, 25.00, 3.10…
$ pov                         <dbl> 6.100, 38.800, 26.800, 0.900, 6.900, 12.10…
$ hs_orless                   <dbl> 33.30, 66.00, 70.90, 38.00, 44.60, 28.40, …
$ urc2013                     <dbl> 4, 4, 4, 2, 2, 4, 6, 2, 4, 1, 1, 1, 3, 4, …
$ urc2006                     <dbl> 5, 4, 4, 2, 2, 4, 6, 2, 4, 1, 1, 1, 3, 4, …
$ aod                         <dbl> 37.36364, 36.25000, 30.45455, 48.36364, 51…
$ geom                        <MULTIPOLYGON [°]> MULTIPOLYGON (((-87.93757 3..…

Nice! we can see that we have add a geom variable to the pm data.

Now we can use this to color the counties in our plot based on the value variable of our pm data, which you may recall is the actual monitor data for fine particulate air pollution at each monitor.

WE can do so using the scale_fill_gradientn() function of the ggplot2 package which creates color gradient based on a variable. In this case it is the variable that was specified as the fill in the aes function of the geom_sf() function. We specified that it would be the value variable of the pm data.

This scale_fill_gradientn() function also allows you to specify the colors, what to do about NA values (should they be a specific color or transparent) and the breaks, limits, labels and name/title on the legend for the color gradient.

truth <- ggplot(data = world) +
  coord_sf(xlim = c(-125,-66),
           ylim = c(24.5, 50),
           expand = FALSE) +
  geom_sf(data = map_data, aes(fill = value)) +
  scale_fill_gradientn(colours = topo.colors(7),
                       na.value = "transparent",
                       breaks = c(0, 10, 20),
                       labels = c(0, 10, 20),
                       limits = c(0, 23.5),
                       name = "PM ug/m3") +
  ggtitle("True PM 2.5 levels") +
  theme(axis.title.x = element_blank(),
        axis.text.x = element_blank(),
        axis.ticks.x = element_blank(),
        axis.title.y = element_blank(),
        axis.text.y = element_blank(),
        axis.ticks.y = element_blank())

truth

Nice!

Now let’s do the same with our predicted outcome values.

Let’s grab both the testing and training predicted outcome values so that we have as much data as possible.

First we need to fit our training data with our final model to be able to get the predictions for the monitors included in the training set. We did this using the last_fit() function, but the output of this makes it difficult to grab the predicted values for the training data, and it is also difficult to get the id variables for the testing data.

Thus we will use the parsnip fit() and predict() functions of the parsnip package to do this like so:

Question Opportunity

Why do we not need pre-processed data?

Click here to reveal the answer.

Since we are using a workflow, the data will be pre-processed when it is fit as well.

RF_final_train_fit <- parsnip::fit(RF_tuned_wflow, data = train_pm)
RF_final_test_fit <- parsnip::fit(RF_tuned_wflow, data = test_pm)


values_pred_train <- predict(RF_final_train_fit, train_pm) %>% 
  bind_cols(train_pm %>% select(value, fips, county, id)) 

values_pred_train

# A tibble: 584 × 5
   .pred value fips  county           id        
   <dbl> <dbl> <fct> <chr>            <fct>     
 1 11.9  11.7  18003 Allen            18003.0004
 2  8.14  6.96 55041 Forest           55041.0007
 3 13.8  13.3  6065  Riverside        6065.1003 
 4 11.0  10.7  39009 Athens           39009.0003
 5 13.8  14.5  39061 Hamilton         39061.8001
 6 12.5  12.2  24510 Baltimore (City) 24510.0006
 7 12.0  11.2  6061  Placer           6061.0006 
 8  8.00  6.98 6065  Riverside        6065.5001 
 9  7.64  6.61 44003 Kent             44003.0002
10 11.2  11.6  37111 McDowell         37111.0004
# … with 574 more rows
# ℹ Use `print(n = ...)` to see more rows

values_pred_test <- predict(RF_final_test_fit, test_pm) %>% 
  bind_cols(test_pm %>% select(value, fips, county, id)) 

values_pred_test

# A tibble: 292 × 5
   .pred value fips  county     id       
   <dbl> <dbl> <fct> <chr>      <fct>    
 1  11.6  11.2 1033  Colbert    1033.1002
 2  12.0  12.4 1055  Etowah     1055.001 
 3  11.1  10.5 1069  Houston    1069.0003
 4  14.0  15.6 1073  Jefferson  1073.0023
 5  12.1  12.4 1073  Jefferson  1073.1005
 6  11.3  11.1 1073  Jefferson  1073.1009
 7  11.5  11.8 1073  Jefferson  1073.5003
 8  11.1  10.0 1097  Mobile     1097.0003
 9  11.9  12.0 1101  Montgomery 1101.0007
10  12.9  13.2 1113  Russell    1113.0001
# … with 282 more rows
# ℹ Use `print(n = ...)` to see more rows

Now we can combine this data for the predictions for all monitors using the bind_rows() function of the dplyr package, which will essentially append the second dataset to the first.

all_pred <- bind_rows(values_pred_test, values_pred_train)

all_pred

# A tibble: 876 × 5
   .pred value fips  county     id       
   <dbl> <dbl> <fct> <chr>      <fct>    
 1  11.6  11.2 1033  Colbert    1033.1002
 2  12.0  12.4 1055  Etowah     1055.001 
 3  11.1  10.5 1069  Houston    1069.0003
 4  14.0  15.6 1073  Jefferson  1073.0023
 5  12.1  12.4 1073  Jefferson  1073.1005
 6  11.3  11.1 1073  Jefferson  1073.1009
 7  11.5  11.8 1073  Jefferson  1073.5003
 8  11.1  10.0 1097  Mobile     1097.0003
 9  11.9  12.0 1101  Montgomery 1101.0007
10  12.9  13.2 1113  Russell    1113.0001
# … with 866 more rows
# ℹ Use `print(n = ...)` to see more rows

Great! as we can see there are 876 values like we would expect for all of the monitors. We can use the county variable to combine this with the counties data like we did with the pm data previously so that we can use the value variable as a color scheme for our map.

map_data <- inner_join(counties, all_pred, by = "county")

pred <- ggplot(data = world) +
  coord_sf(xlim = c(-125,-66),
           ylim = c(24.5, 50),
           expand = FALSE) +
  geom_sf(data = map_data, aes(fill = .pred)) +
  scale_fill_gradientn(colours = topo.colors(7),
                       na.value = "transparent",
                       breaks = c(0, 10, 20),
                       labels = c(0, 10, 20),
                       limits = c(0, 23.5),
                       name = "PM ug/m3") +
  ggtitle("Predicted PM 2.5 levels") +
  theme(axis.title.x=element_blank(),
        axis.text.x=element_blank(),
        axis.ticks.x=element_blank(),
        axis.title.y=element_blank(),
        axis.text.y=element_blank(),
        axis.ticks.y=element_blank())

pred

Now we will use the patchwork package to combine our last two plots. This allows us to combine plots using the + or the / . The + will place plots side by side and the / will place plots top to bottom.

Now let’s just combine the truth plot and the prediction plots together:

truth/pred

We can see that the predicted fine particle air pollution values in (ug/m3) are quite similar to the true values measured by the actual gravimetric monitors. We can also see that southern California has some large counties with worse pollution (as they are yellow and thus have much higher particulate matter levels).

Let’s add some text to our plot to explain it a bit more. We can do so using the plot_annotation() function of the patchwork package. The theme argument of this function takes the same theme information using the theme() function of the ggplot2 package as when creating ggplot2plots.

(truth/pred) + 
  plot_annotation(title = "Machine Learning Methods Allow for Prediction of Air Pollution", subtitle = "A random forest model predicts true monitored levels of fine particulate matter (PM 2.5) air pollution based on\ndata about population density and other predictors reasonably well, thus suggesting that we can use similar methods to predict levels\nof pollution in places with poor monitoring",
                  theme = theme(plot.title = element_text(size =12, face = "bold"), 
                                plot.subtitle = element_text(size = 8)))

Summary

Synopsis

In this case study, we explored gravimetric monitoring data of fine particulate matter air pollution (outcome variable). Our goal was to able to predict air pollution where we only had predictor variables (or features) without having observed a corresponding measurement of air pollution.

Our learning objectives were:

Introduce concepts in machine learning
Demonstrate how to build a machine learning model with tidymodels
Demonstrate how to visualize geo-spatial data using ggplot2

Using the machine learning models built in this case study, we could now extend this model to predict air pollution levels in areas with poor monitoring, to help identify regions where populations maybe especially at risk for the health effects of air pollution.

Analyses like the one in our case study are important for defining which groups could benefit the most from interventions, education, and policy changes when attempting to mitigate public health challenges. You can see in this article that many additional considerations would be involved to adequately understand the data enough to recommend policy changes.

Here are some visual summaries about what we learned about using tidymodels to perform prediction analyses.

First the minimal steps required:

Here is a guide for more advanced analyses involving preprocessing, cross validation, or tuning:

Click here for more on what we learned with tidymodels

Here, we provide an overview of the tidymodels framework.

We performed the major steps of machine learning that we introduced in the beginning of the data analysis:

Data exploration

We used packages like skimr, summarytools, corrplot, and GGally to better understand our data. These packages can tell us how many missing values each variable has (if any), the class of each variable, the distribution of values for each variable, the sparsity of each variable, and the level of correlation between variables.

Data splitting

We used the rsample package to first perform an initial split of our data into two pieces: a training set and a testing set. The training set was used to optimize the model, while the testing set was used only to evaluate the performance of our final model. We also used the rsample package to create cross validation subsets of our training data. This allowed us to better assess the performance of our tested models using our training data.

Variable assignment and pre-processing

We used the recipes package to assign variable roles (such as outcome, predictor, and id variable). We also used this package to create a recipe for pre-processing our training and testing data. This involved steps such as: step_dummy to create dummy numeric encodings of our categorical variables, step_corr to remove highly correlated variables, step_nzv to remove near zero variance variables that would contribute little to our model and potentially add noise. We learned that once our recipe was created and prepped using prep()we could extract the pre-processed training data or our pre-processed testing data using bake(). We also learned that if we used the newer workflows package that we did not need to use the prep() or bake() functions, but that it is still useful to know how to do so if we want to look at our data and how the recipe is influencing it more deeply.

Model specification, fitting, tuning and performance evaluation using the training data

We learned that the model needs to first be fit to the training data. We learned that in both classification and prediction, the model is fit to the training data and the explanatory variables are used to estimate numeric values (in the case of prediction) or categorical values (in the case of classification) of the outcome variable of interest. We learned that we specify the model and its specifications using the parnsip package and that we also use this package to fit the model using the fit() function. We learned that if we just use parsnip to fit the model, then we need to use the pre-processed training data (output from bake()). We learned that we can use the raw training data if we use the workflows package to create a workflow that pre-processes our data for us.

We learned that if the model fits well than the estimated values will be very similar to the true outcome variable values in our training data. We learned that we can assess model performance using the yardstick package with the metrics functio or the tune package and the collect_metrics() function (required if using cross validation or tuning). We also learned that we can use subsets of our training data (which we created with the rsample package) to perform cross validation to get a better estimate about the performance of our model using our training data, as we want our results to be generalizable and to perform well with other data, not just our training data. We used the fit_resamples() function of the tune package to fit our model on our different training data subsets and the collect_metrics() function (also of the tune package) to evaluate model performance using these subsets. We also learned that we can potentially improve model performance by tuning aspects about the model called hyperparameters to determine the best option for model performance. We learned that we can do this using the tune and dials packages and evaluating the performance of our model with the different hyperparameter options and our training data subsets that we used for cross validation. After we tested several different methods to model our data, we compared them to choose the best performing model as our final model.

Overall model performance evaluation

Once we chose our final model, we evaluated the final model performance using the testing data using the last_fit() function of the tune package. This gives us a better estimate about how well the model will predict or classify the outcome variable of interest with new independent data. Ideally one would also perform an evaluation with independent data to provide a sense of how generalizable the model is to other data sources.

We also saw that we can use the collect_predictions() function of the tune package to get the predictions for our test data. We saw that we can get more detailed prediction data using the predict() function of the parsnip package.

Suggested Homework

Students can predict air pollution monitor values using a different algorithm and provide an explanation for how that algorithm works and why it may be a good choice for modeling this data.

Additional Information

Helpful Links

A review of tidymodels
A course on tidymodels by Julia Silge
More examples, explanations, and info about tidymodels development from the developers
A guide for pre-processing with recipes
A guide for using GGally to create correlation plots
A guide for using parsnip to try different algorithms or engines
A list of recipe functions
A great blog post about cross validation
A discussion about evaluating model performance for a deeper explanation about how to evaluate model performance
RStudio cheatsheets
An explanation of supervised vs unsupervised machine learning and bias-variance trade-off.
A thorough explanation of principal component analysis.
If you have access, this is a great discussion about the difference between independence, orthogonality, and lack of correlation.
Great video explanation of PCA.

Terms and concepts covered:

Tidyverse
Imputation
Transformation
Discretization
Dummy Variables
One-Hot Encoding
Data Type Conversions
Interaction
Normalization
Dimensionality Reduction/Signal Extraction
Row Operations
Near Zero Varaince
Parameters and Hyperparameters
Supervised and Unspervised Learning
Principal Component Analysis
Linear Combinations
Decision Tree
Random Forest

Packages used in this case study:

Package	Use in this case study
here	to easily load and save data
readr	to import the CSV file data
dplyr	to view/arrange/filter/select/compare specific subsets of the data
skimr	to get an overview of data
summarytools	to get an overview of data in a different style
magrittr	to use the `%<>%` pipping operator
corrplot	to make large correlation plots
GGally	to make smaller correlation plots
rsample	to split the data into testing and training sets and to split the training set for cross-validation
recipes	to pre-process data for modeling in a tidy and reproducible way and to extract pre-processed data (major functions are `recipe()` , `prep()` and various transformation `step_*()` functions, as well as `bake` which extracts pre-processed training data (used to require `juice()`) and applies recipe preprocessing steps to testing data). See here for more info.
parsnip	an interface to create models (major functions are `fit()`, `set_engine()`)
yardstick	to evaluate the performance of models
broom	to get tidy output for our model fit and performance
ggplot2	to make visualizations with multiple layers
dials	to specify hyper-parameter tuning
tune	to perform cross validation, tune hyper-parameters, and get performance metrics
workflows	to create modeling workflow to streamline the modeling process
vip	to create variable importance plots
randomForest	to perform the random forest analysis
doParallel	to fit cross validation samples in parallel
stringr	to manipulate the text the map data
tidyr	to separate data within a column into multiple columns
rnaturalearth	to get the geometry data for the earth to plot the US
maps	to get map database data about counties to draw them on our US map
sf	to convert the map data into a data frame
lwgeom	to use the `sf` function to convert the map geographical data
rgeos	to use geometry data
patchwork	to allow plots to be combined

Session info

sessionInfo()

R version 4.2.0 (2022-04-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur/Monterey 10.16

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] OCSdata_1.0.2             patchwork_1.1.1          
 [3] rgeos_0.5-9               sp_1.5-0                 
 [5] rnaturalearthdata_0.1.0   rnaturalearth_0.1.0      
 [7] maps_3.4.0                sf_1.0-7                 
 [9] proxy_0.4-27              lwgeom_0.2-8             
[11] stringr_1.4.0             doParallel_1.0.17        
[13] iterators_1.0.14          foreach_1.5.2            
[15] randomForest_4.7-1.1      vip_0.3.2                
[17] yardstick_1.0.0           workflowsets_1.0.0       
[19] workflows_1.0.0           tune_1.0.0               
[21] tidyr_1.2.0               tibble_3.1.8             
[23] rsample_1.0.0             recipes_1.0.1            
[25] purrr_0.3.4               parsnip_1.0.0            
[27] modeldata_1.0.0           infer_1.0.2              
[29] dials_1.0.0               scales_1.2.0             
[31] broom_1.0.0               tidymodels_1.0.0         
[33] GGally_2.1.2              ggplot2_3.3.6            
[35] RColorBrewer_1.1-3        corrplot_0.92            
[37] summarytools_1.0.1        skimr_2.1.4              
[39] dplyr_1.0.9               readr_2.1.2              
[41] koRpus.lang.en_0.1-4      koRpus_0.13-8            
[43] sylly_0.1-6               read.so_0.1.1            
[45] wordcountaddin_0.3.0.9000 magrittr_2.0.3           
[47] here_1.0.1                knitr_1.39               

loaded via a namespace (and not attached):
 [1] backports_1.4.1    plyr_1.8.7         repr_1.1.4         splines_4.2.0     
 [5] listenv_0.8.0      usethis_2.1.6      pryr_0.1.5         digest_0.6.29     
 [9] htmltools_0.5.3    magick_2.7.3       fansi_1.0.3        checkmate_2.1.0   
[13] tzdb_0.3.0         remotes_2.4.2      globals_0.15.1     gower_1.0.0       
[17] matrixStats_0.62.0 vroom_1.5.7        hardhat_1.2.0      colorspace_2.0-3  
[21] xfun_0.31          tcltk_4.2.0        crayon_1.5.1       jsonlite_1.8.0    
[25] survival_3.3-1     glue_1.6.2         gtable_0.3.0       ipred_0.9-13      
[29] future.apply_1.9.0 rapportools_1.1    DBI_1.1.3          Rcpp_1.0.9        
[33] units_0.8-0        bit_4.0.4          GPfit_1.0-8        lava_1.6.10       
[37] prodlim_2019.11.13 httr_1.4.3         wk_0.6.0           ellipsis_0.3.2    
[41] farver_2.1.1       pkgconfig_2.0.3    reshape_0.8.9      nnet_7.3-17       
[45] sass_0.4.2         utf8_1.2.2         labeling_0.4.2     tidyselect_1.1.2  
[49] rlang_1.0.4        DiceDesign_1.9     reshape2_1.4.4     munsell_0.5.0     
[53] tools_4.2.0        cachem_1.0.6       cli_3.3.0          generics_0.1.3    
[57] evaluate_0.15      fastmap_1.1.0      yaml_2.3.5         bit64_4.0.5       
[61] fs_1.5.2           pander_0.6.5       s2_1.1.0           future_1.27.0     
[65] compiler_4.2.0     rstudioapi_0.13    curl_4.3.2         e1071_1.7-11      
[69] lhs_1.1.5          bslib_0.4.0        stringi_1.7.8      highr_0.9         
[73] lattice_0.20-45    Matrix_1.4-1       classInt_0.4-7     vctrs_0.4.1       
[77] pillar_1.8.0       lifecycle_1.0.1    furrr_0.3.0        jquerylib_0.1.4   
[81] data.table_1.14.2  sylly.en_0.1-3     R6_2.5.1           KernSmooth_2.23-20
[85] gridExtra_2.3      parallelly_1.32.1  codetools_0.2-18   MASS_7.3-58       
[89] assertthat_0.2.1   rprojroot_2.0.3    withr_2.5.0        hms_1.1.1         
[93] grid_4.2.0         rpart_4.1.16       timeDate_4021.104  class_7.3-20      
[97] rmarkdown_2.14     lubridate_1.8.0    base64enc_0.1-3

Estimate of RMarkdown Compilation Time:

About 208 - 218 seconds

This compilation time was measured on a PC machine operating on Windows 10. This range should only be used as an estimate as compilation time will vary with different machines and operating systems.

Acknowledgments

We would like to acknowledge Roger Peng, Megan Latshaw, and Kirsten Koehler for assisting in framing the major direction of the case study.

We would like to acknowledge Michael Breshock for his contributions to this case study and developing the OCSdata package.

We would also like to acknowledge the Bloomberg American Health Initiative for funding this work.

LS0tCnRpdGxlOiAiT3BlbiBDYXNlIFN0dWRpZXM6IFByZWRpY3RpbmcgQW5udWFsIEFpciBQb2xsdXRpb24iCmNzczogc3R5bGUuY3NzCm91dHB1dDoKICBodG1sX2RvY3VtZW50OgogICAgaW5jbHVkZXM6CiAgICAgICBpbl9oZWFkZXI6IEdBX1NjcmlwdC5SaHRtbAogICAgc2VsZl9jb250YWluZWQ6IHllcwogICAgY29kZV9kb3dubG9hZDogeWVzCiAgICBoaWdobGlnaHQ6IHRhbmdvCiAgICBudW1iZXJfc2VjdGlvbnM6IG5vCiAgICB0aGVtZTogY29zbW8KICAgIHRvYzogeWVzCiAgICB0b2NfZmxvYXQ6IHllcwogIHBkZl9kb2N1bWVudDoKICAgIHRvYzogeWVzCiAgd29yZF9kb2N1bWVudDoKICAgIHRvYzogeWVzCgotLS0KPHN0eWxlPgojVE9DIHsKICBiYWNrZ3JvdW5kOiB1cmwoImh0dHBzOi8vb3BlbmNhc2VzdHVkaWVzLmdpdGh1Yi5pby9pbWcvaWNvbi1iYWhpLnBuZyIpOwogIGJhY2tncm91bmQtc2l6ZTogY29udGFpbjsKICBwYWRkaW5nLXRvcDogMjQwcHggIWltcG9ydGFudDsKICBiYWNrZ3JvdW5kLXJlcGVhdDogbm8tcmVwZWF0Owp9Cjwvc3R5bGU+Cgo8IS0tIE9wZW4gYWxsIGxpbmtzIGluIG5ldyB0YWItLT4gIAo8YmFzZSB0YXJnZXQ9Il9ibGFuayIvPiAgCjxkaXYgaWQ9Imdvb2dsZV90cmFuc2xhdGVfZWxlbWVudCI+PC9kaXY+Cgo8c2NyaXB0IHR5cGU9InRleHQvamF2YXNjcmlwdCIgc3JjPScvL3RyYW5zbGF0ZS5nb29nbGUuY29tL3RyYW5zbGF0ZV9hL2VsZW1lbnQuanM/Y2I9Z29vZ2xlVHJhbnNsYXRlRWxlbWVudEluaXQnPjwvc2NyaXB0PgoKPHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiPgpmdW5jdGlvbiBnb29nbGVUcmFuc2xhdGVFbGVtZW50SW5pdCgpIHsKICBuZXcgZ29vZ2xlLnRyYW5zbGF0ZS5UcmFuc2xhdGVFbGVtZW50KHtwYWdlTGFuZ3VhZ2U6ICdlbid9LCAnZ29vZ2xlX3RyYW5zbGF0ZV9lbGVtZW50Jyk7Cn0KPC9zY3JpcHQ+CgoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmxpYnJhcnkoa25pdHIpCmxpYnJhcnkoaGVyZSkKa25pdHI6Om9wdHNfY2h1bmskc2V0KGluY2x1ZGUgPSBUUlVFLCBjb21tZW50ID0gTkEsIGVjaG8gPSBUUlVFLAogICAgICAgICAgICAgICAgICAgICAgbWVzc2FnZSA9IEZBTFNFLCB3YXJuaW5nID0gRkFMU0UsIGNhY2hlID0gRkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICBmaWcuYWxpZ24gPSAiY2VudGVyIiwgb3V0LndpZHRoID0gJzkwJScpCmxpYnJhcnkobWFncml0dHIpCnJlbW90ZXM6Omluc3RhbGxfZ2l0aHViKCJiZW5tYXJ3aWNrL3dvcmRjb3VudGFkZGluIiwgdHlwZSA9ICJzb3VyY2UiLCBkZXBlbmRlbmNpZXMgPSBUUlVFKQpyZW1vdGVzOjppbnN0YWxsX2dpdGh1YigiYWxpc3RhaXJlNDcvcmVhZC5zbyIpCmxpYnJhcnkod29yZGNvdW50YWRkaW4pCmxpYnJhcnkocmVhZC5zbykKCnJtYXJrZG93bjo6OnBlcmZfdGltZXJfcmVzZXRfYWxsKCkKcm1hcmtkb3duOjo6cGVyZl90aW1lcl9zdGFydCgicmVuZGVyIikKYGBgCgoKIyMjIyB7Lm91dGxpbmUgfQpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiODAwIHB4In0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwgIm1haW5fcGxvdF9tYXBzLnBuZyIpKQpgYGAKCiMjIyMKCiMjIyMgey5kaXNjbGFpbWVyX2Jsb2NrfQoKKipEaXNjbGFpbWVyKio6IFRoZSBwdXJwb3NlIG9mIHRoZSBbT3BlbiBDYXNlIFN0dWRpZXNdKGh0dHBzOi8vb3BlbmNhc2VzdHVkaWVzLmdpdGh1Yi5pbyl7dGFyZ2V0PSJfYmxhbmsifSBwcm9qZWN0IGlzICoqdG8gZGVtb25zdHJhdGUgdGhlIHVzZSBvZiB2YXJpb3VzIGRhdGEgc2NpZW5jZSBtZXRob2RzLCB0b29scywgYW5kIHNvZnR3YXJlIGluIHRoZSBjb250ZXh0IG9mIG1lc3N5LCByZWFsLXdvcmxkIGRhdGEqKi4gQSBnaXZlbiBjYXNlIHN0dWR5IGRvZXMgbm90IGNvdmVyIGFsbCBhc3BlY3RzIG9mIHRoZSByZXNlYXJjaCBwcm9jZXNzLCBpcyBub3QgY2xhaW1pbmcgdG8gYmUgdGhlIG1vc3QgYXBwcm9wcmlhdGUgd2F5IHRvIGFuYWx5emUgYSBnaXZlbiBkYXRhIHNldCwgYW5kIHNob3VsZCBub3QgYmUgdXNlZCBpbiB0aGUgY29udGV4dCBvZiBtYWtpbmcgcG9saWN5IGRlY2lzaW9ucyB3aXRob3V0IGV4dGVybmFsIGNvbnN1bHRhdGlvbiBmcm9tIHNjaWVudGlmaWMgZXhwZXJ0cy4gCgojIyMjCgojIyMjIHsubGljZW5zZV9ibG9ja30KClRoaXMgd29yayBpcyBsaWNlbnNlZCB1bmRlciB0aGUgQ3JlYXRpdmUgQ29tbW9ucyBBdHRyaWJ1dGlvbi1Ob25Db21tZXJjaWFsIDMuMCBbKENDIEJZLU5DIDMuMCldKGh0dHBzOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS1uYy8zLjAvdXMvKXt0YXJnZXQ9Il9ibGFuayJ9IFVuaXRlZCBTdGF0ZXMgTGljZW5zZS4KCiMjIyMKCiMjIyMgey5yZWZlcmVuY2VfYmxvY2t9CgpUbyBjaXRlIHRoaXMgY2FzZSBzdHVkeSBwbGVhc2UgdXNlOgoKV3JpZ2h0LCBDYXJyaWUgYW5kIE1lbmcsIFFpZXIgYW5kIEphZ2VyLCBMZWFoIGFuZCBUYXViLCBNYXJnYXJldCBhbmQgSGlja3MsIFN0ZXBoYW5pZS4gKDIwMjApLiBbaHR0cHM6Ly9naXRodWIuY29tLy9vcGVuY2FzZXN0dWRpZXMvb2NzLWJwLWFpci1wb2xsdXRpb25dKGh0dHBzOi8vZ2l0aHViLmNvbS8vb3BlbmNhc2VzdHVkaWVzL29jcy1icC1haXItcG9sbHV0aW9uLykuIFByZWRpY3RpbmcgQW5udWFsIEFpciBQb2xsdXRpb24gKFZlcnNpb24gdjEuMC4wKS4KCiMjIyMKClRvIGFjY2VzcyB0aGUgR2l0SHViIFJlcG9zaXRvcnkgZm9yIHRoaXMgY2FzZSBzdHVkeSBzZWUgaGVyZTogaHR0cHM6Ly9naXRodWIuY29tL29wZW5jYXNlc3R1ZGllcy9vY3MtYnAtYWlyLXBvbGx1dGlvbi4KCllvdSBtYXkgYWxzbyBhY2Nlc3MgYW5kIGRvd25sb2FkIHRoZSBkYXRhIHVzaW5nIG91ciBgT0NTZGF0YWAgcGFja2FnZS4gVG8gbGVhcm4gbW9yZSBhYm91dCB0aGlzIHBhY2thZ2UgaW5jbHVkaW5nIGV4YW1wbGVzLCBzZWUgdGhpcyBbbGlua10oaHR0cHM6Ly9naXRodWIuY29tL29wZW5jYXNlc3R1ZGllcy9PQ1NkYXRhKS4gSGVyZSBpcyBob3cgeW91IHdvdWxkIGluc3RhbGwgdGhpcyBwYWNrYWdlOgoKYGBge3IsIGV2YWw9RkFMU0V9Cmluc3RhbGwucGFja2FnZXMoIk9DU2RhdGEiKQpgYGAKClRoaXMgY2FzZSBzdHVkeSBpcyBwYXJ0IG9mIGEgc2VyaWVzIG9mIHB1YmxpYyBoZWFsdGggY2FzZSBzdHVkaWVzIGZvciB0aGUgW0Jsb29tYmVyZyBBbWVyaWNhbiBIZWFsdGggIEluaXRpYXRpdmVdKGh0dHBzOi8vYW1lcmljYW5oZWFsdGguamh1LmVkdS9vcGVuLWNhc2Utc3R1ZGllcykuCgoqKioKClRoZSB0b3RhbCByZWFkaW5nIHRpbWUgZm9yIHRoaXMgY2FzZSBzdHVkeSBpcyBjYWxjdWxhdGVkIHZpYSBba29ScHVzXShodHRwczovL2dpdGh1Yi5jb20vdW5Eb2NVTWVhbnRJdC9rb1JwdXMpIGFuZCBzaG93biBiZWxvdzogCgpgYGB7ciwgZWNobz1GQUxTRX0KcmVhZHRhYmxlID0gdGV4dF9zdGF0cygiaW5kZXguUm1kIikgIyBwcm9kdWNpbmcgcmVhZGluZyB0aW1lIG1hcmtkb3duIHRhYmxlCnJlYWR0aW1lID0gcmVhZC5zbzo6cmVhZC5tZChyZWFkdGFibGUpICU+JSBkcGx5cjo6c2VsZWN0KE1ldGhvZCwga29ScHVzKSAlPiUgIyByZWFkaW5nIHRhYmxlIGludG8gZGF0YWZyYW1lLCBzZWxlY3RpbmcgcmVsZXZhbnQgZmFjdG9ycwogIGRwbHlyOjpmaWx0ZXIoTWV0aG9kID09ICJSZWFkaW5nIHRpbWUiKSAlPiUgIyBkcm9wcGluZyB1bm5lY2Vzc2FyeSByb3dzCiAgZHBseXI6Om11dGF0ZShrb1JwdXMgPSBwYXN0ZShyb3VuZChhcy5udW1lcmljKHN0cmluZ3I6OnN0cl9zcGxpdChrb1JwdXMsICIgIilbWzFdXVsxXSkpLCAibWludXRlcyIpKSAlPiUgIyByb3VuZGluZyByZWFkaW5nIHRpbWUgZXN0aW1hdGUKICBkcGx5cjo6bXV0YXRlKE1ldGhvZCA9ICJrb1JwdXMiKSAlPiUgZHBseXI6OnJlbG9jYXRlKGtvUnB1cywgLmJlZm9yZSA9IE1ldGhvZCkgJT4lIGRwbHlyOjpyZW5hbWUoYFJlYWRpbmcgVGltZWAgPSBrb1JwdXMpICMgcmVvcmdhbml6aW5nIHRhYmxlCmtuaXRyOjprYWJsZShyZWFkdGltZSwgZm9ybWF0PSJtYXJrZG93biIpCmBgYAoKKioqCgoqKlJlYWRhYmlsaXR5IFNjb3JlOiAqKgoKQSByZWFkYWJpbGl0eSBpbmRleCBlc3RpbWF0ZXMgdGhlIHJlYWRpbmcgZGlmZmljdWx0eSBsZXZlbCBvZiBhIHBhcnRpY3VsYXIgdGV4dC4gRmxlc2NoLUtpbmNhaWQsIEZPUkNBU1QsIGFuZCBTTU9HIGFyZSB0aHJlZSBjb21tb24gcmVhZGFiaWxpdHkgaW5kaWNlcyB0aGF0IHdlcmUgY2FsY3VsYXRlZCBmb3IgdGhpcyBjYXNlIHN0dWR5IHZpYSBba29ScHVzXShodHRwczovL2dpdGh1Yi5jb20vdW5Eb2NVTWVhbnRJdC9rb1JwdXMpLiBUaGVzZSBpbmRpY2VzIHByb3ZpZGUgYW4gZXN0aW1hdGlvbiBvZiB0aGUgbWluaW11bSByZWFkaW5nIGxldmVsIHJlcXVpcmVkIHRvIGNvbXByZWhlbmQgdGhpcyBjYXNlIHN0dWR5IGJ5IGdyYWRlIGFuZCBhZ2UuIAoKYGBge3IsIGVjaG89RkFMU0V9CnJ0ID0gd29yZGNvdW50YWRkaW46OnJlYWRhYmlsaXR5KCJpbmRleC5SbWQiLCBxdWlldD1UUlVFKSAjIHByb2R1Y2luZyByZWFkYWJpbGl0eSBtYXJrZG93biB0YWJsZQpkZiA9IHJlYWQuc286OnJlYWQubWQocnQpICU+JSBkcGx5cjo6c2VsZWN0KGluZGV4LCBncmFkZSwgYWdlKSAlPiUgICMgcmVhZGluZyB0YWJsZSBpbnRvIGRhdGFmcmFtZSwgc2VsZWN0aW5nIHJlbGV2YW50IGZhY3RvcnMKICB0aWR5cjo6ZHJvcF9uYSgpICU+JSBkcGx5cjo6bXV0YXRlKGdyYWRlID0gcm91bmQoYXMubnVtZXJpYyhncmFkZSkpLCAjIGRyb3BwaW5nIHJvd3Mgd2l0aCBtaXNzaW5nIHZhbHVlcywgcm91bmRpbmcgYWdlIGFuZCBncmFkZSBjb2x1bW5zCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhZ2UgPSByb3VuZChhcy5udW1lcmljKGFnZSkpCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICApCmtuaXRyOjprYWJsZShkZiwgZm9ybWF0PSJtYXJrZG93biIpCmBgYAoKKioqCgpQbGVhc2UgaGVscCB1cyBieSBmaWxsaW5nIG91dCBvdXIgc3VydmV5LgoKCjxkaXYgc3R5bGU9ImRpc3BsYXk6IGZsZXg7IGp1c3RpZnktY29udGVudDogY2VudGVyOyI+PGlmcmFtZSBzcmM9Imh0dHBzOi8vZG9jcy5nb29nbGUuY29tL2Zvcm1zL2QvZS8xRkFJcFFMU2ZwTjRGTjNLRUxxQk5FZ2YyQXRwaTdXeTdOcXkyYmVTa0ZRSU5MN1k1c0FNVjVfdy92aWV3Zm9ybT9lbWJlZGRlZD10cnVlIiB3aWR0aD0iMTIwMCIgaGVpZ2h0PSI3MDAiIGZyYW1lYm9yZGVyPSIwIiBtYXJnaW5oZWlnaHQ9IjAiIG1hcmdpbndpZHRoPSIwIj5Mb2FkaW5n4oCmPC9pZnJhbWU+PC9kaXY+CgoKIyAqKk1vdGl2YXRpb24qKgoqKioKQSB2YXJpZXR5IG9mIGRpZmZlcmVudCBzb3VyY2VzIGNvbnRyaWJ1dGUgZGlmZmVyZW50IHR5cGVzIG9mIHBvbGx1dGFudHMgdG8gd2hhdCB3ZSBjYWxsIGFpciBwb2xsdXRpb24uIAoKU29tZSBzb3VyY2VzIGFyZSBuYXR1cmFsIHdoaWxlIG90aGVycyBhcmUgYW50aHJvcG9nZW5pYyAoaHVtYW4gZGVyaXZlZCk6Cgo8cCBhbGlnbj0iY2VudGVyIj4KPGltZyB3aWR0aD0iNjAwIiBzcmM9Imh0dHBzOi8vd3d3Lm5wcy5nb3Yvc3ViamVjdHMvYWlyL2ltYWdlcy9Tb3VyY2VzX0dyYXBoaWNfSHVnZS5qcGc/bWF4d2lkdGg9MTIwMCZtYXhoZWlnaHQ9MTIwMCZhdXRvcm90YXRlPWZhbHNlIj4KPC9wPgoKIyMjIyMgW1tzb3VyY2VdXShodHRwczovL3d3dy5nb29nbGUuY29tL3VybD9zYT1pJnVybD1odHRwcyUzQSUyRiUyRnd3dy5ucHMuZ292JTJGc3ViamVjdHMlMkZhaXIlMkZzb3VyY2VzLmh0bSZwc2lnPUFPdlZhdzJ2N0FWeFNGOFpTQVBFaE51ZFZ0Yk4mdXN0PTE1ODU3NzA5NjYyMTcwMDAmc291cmNlPWltYWdlcyZjZD12ZmUmdmVkPTBDQUlRalJ4cUZ3b1RDUERONjZxX3hlZ0NGUUFBQUFBZEFBQUFBQkFEKXt0YXJnZXQ9Il9ibGFuayJ9CgojIyMgTWFqb3IgdHlwZXMgb2YgYWlyIHBvbGx1dGFudHMKCjEpICoqR2FzZW91cyoqIC0gQ2FyYm9uIE1vbm94aWRlIChDTyksIE96b25lIChPfjN+KSwgTml0cm9nZW4gT3hpZGVzKE5PLCBOT34yfiksIFN1bGZ1ciBEaW94aWRlIChTT34yfikKMikgKipQYXJ0aWN1bGF0ZSoqIC0gc21hbGwgbGlxdWlkcyBhbmQgc29saWRzIHN1c3BlbmRlZCBpbiB0aGUgYWlyIChpbmNsdWRlcyBsZWFkLSBjYW4gaW5jbHVkZSBjZXJ0YWluIHR5cGVzIG9mIGR1c3QpCjMpICoqRHVzdCoqIC0gc21hbGwgc29saWRzIChsYXJnZXIgdGhhbiBwYXJ0aWN1bGF0ZXMpIHRoYXQgY2FuIGJlIHN1c3BlbmRlZCBpbiB0aGUgYWlyIGZvciBzb21lIHRpbWUgYnV0IGV2ZW50dWFsbHkgc2V0dGxlCjQpICoqQmlvbG9naWNhbCoqIC0gcG9sbGVuLCBiYWN0ZXJpYSwgdmlydXNlcywgbW9sZCBzcG9yZXMKClNlZSBbaGVyZV0oaHR0cDovL3d3dy5yZWRsb2dlbnYuY29tL3dvcmtlci1zYWZldHkvcGFydC0xLWR1c3QtYW5kLXBhcnRpY3VsYXRlLW1hdHRlcikgZm9yIG1vcmUgZGV0YWlsIG9uIHRoZSB0eXBlcyBvZiBwb2xsdXRhbnRzIGluIHRoZSBhaXIuCgoKIyMjIFBhcnRpY3VsYXRlIHBvbGx1dGlvbiAKCkFpciBwb2xsdXRpb24gcGFydGljdWxhdGVzIGFyZSBnZW5lcmFsbHkgZGVzY3JpYmVkIGJ5IHRoZWlyICoqc2l6ZSoqLgoKVGhlcmUgYXJlIDMgbWFqb3IgY2F0ZWdvcmllczoKCjEpICoqTGFyZ2UgQ29hcnNlKiogUGFydGljdWxhdGUgTWF0dGVyIC0gaGFzIGRpYW1ldGVyIG9mID4xMCBtaWNyb21ldGVycyAoMTAgwrVtKSAKCjIpICoqQ29hcnNlKiogUGFydGljdWxhdGUgTWF0dGVyIChjYWxsZWQgKipQTX4xMC0yLjV+KiopIC0gaGFzIGRpYW1ldGVyIG9mIGJldHdlZW4gMi41IMK1bSBhbmQgMTAgwrVtCgozKSAqKkZpbmUqKiBQYXJ0aWN1bGF0ZSBNYXR0ZXIgKGNhbGxlZCAqKlBNfjIuNX4qKikgLSBoYXMgZGlhbWV0ZXIgb2YgPCAyLjUgwrVtIAoKKipQTX4xMH4qKiBpbmNsdWRlcyBhbnkgcGFydGljdWxhdGUgbWF0dGVyIDwxMCDCtW0gKGJvdGggY29hcnNlIGFuZCBmaW5lIHBhcnRpY3VsYXRlIG1hdHRlcikKCkhlcmUgeW91IGNhbiBzZWUgaG93IHRoZXNlIHNpemVzIGNvbXBhcmUgd2l0aCBhIGh1bWFuIGhhaXI6CgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGg9ICI2MDAgcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCAicG0yLjVfc2NhbGVfZ3JhcGhpYy1jb2xvcl8yLmpwZyIpKQpgYGAKCiMjIyMjIFtbc291cmNlXV0oaHR0cHM6Ly93d3cuZXBhLmdvdi9wbS1wb2xsdXRpb24vcGFydGljdWxhdGUtbWF0dGVyLXBtLWJhc2ljcyl7dGFyZ2V0PSJfYmxhbmsifQoKPCEtLSA8cCBhbGlnbj0iY2VudGVyIj4gLS0+CjwhLS0gICA8aW1nIHdpZHRoPSI1MDAiIHNyYz0iaHR0cHM6Ly93d3cuc2Vuc2lyaW9uLmNvbS9pbWFnZXMvc2Vuc2lyaW9uLXNwZWNpYWxpc3QtYXJ0aWNsZS1maWd1cmUtMS1jZGQ3MC5qcGciPiAtLT4KPCEtLSA8L3A+IC0tPgoKCjx1PlRoZSBmb2xsb3dpbmcgcGxvdCBzaG93cyB0aGUgcmVsYXRpdmUgc2l6ZXMgb2YgdGhlc2UgZGlmZmVyZW50IHBvbGx1dGFudHMgaW4gbWljcm9tZXRlcnMgKMK1bSk6PC91PgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoPSAiODAwIHB4In0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwgInBhcnRpY3VsYXRlLXNpemUtY2hhcnQucG5nIikpCmBgYAoKIyMjIyMgW1tzb3VyY2VdXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9QYXJ0aWN1bGF0ZXMpe3RhcmdldD0iX2JsYW5rIn0KCgo8dT5UaGlzIHRhYmxlIHNob3dzIGhvdyBkZWVwbHkgc29tZSBvZiB0aGUgc21hbGxlciBmaW5lIHBhcnRpY2xlcyBjYW4gcGVuZXRyYXRlIHdpdGhpbiB0aGUgaHVtYW4gYm9keTo8L3U+CgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGg9ICI4MDAgcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCAic2l6ZXMuanBnIikpCmBgYAoKIyMjIyMgW1tzb3VyY2VdXShodHRwczovL3d3dy5mcm9udGllcnNpbi5vcmcvYXJ0aWNsZXMvMTAuMzM4OS9mcHViaC4yMDIwLjAwMDE0L2Z1bGwpe3RhcmdldD0iX2JsYW5rIn0KCgojIyMgTmVnYXRpdmUgaW1wYWN0IG9mIHBhcnRpY3VsYXRlIGV4cG9zdXJlIG9uIGhlYWx0aCAKCkV4cG9zdXJlIHRvIGFpciBwb2xsdXRpb24gaXMgYXNzb2NpYXRlZCB3aXRoIGhpZ2hlciByYXRlcyBvZiBbbW9ydGFsaXR5XShodHRwczovL3d3dy5uY2JpLm5sbS5uaWguZ292L3BtYy9hcnRpY2xlcy9QTUM1NzgzMTg2Lyl7dGFyZ2V0PSJfYmxhbmsifSBpbiBvbGRlciBhZHVsdHMgYW5kIGlzIGtub3duIHRvIGJlIGEgcmlzayBmYWN0b3IgZm9yIG1hbnkgZGlzZWFzZXMgYW5kIGNvbmRpdGlvbnMgaW5jbHVkaW5nIGJ1dCBub3QgbGltaXRlZCB0bzoKCjEpIFtBc3RobWFdKGh0dHBzOi8vd3d3Lm5jYmkubmxtLm5paC5nb3YvcHVibWVkLzI5MjQzOTM3KXt0YXJnZXQ9Il9ibGFuayJ9IC0gZmluZSBwYXJ0aWNsZSBleHBvc3VyZSAoKipQTX4yLjV+KiopIHdhcyBmb3VuZCB0byBiZSBhc3NvY2lhdGVkIHdpdGggaGlnaGVyIHJhdGVzIG9mIGFzdGhtYSBpbiBjaGlsZHJlbgoyKSBbSW5mbGFtbWF0aW9uIGluIHR5cGUgMSBkaWFiZXRlc10oaHR0cHM6Ly93d3cubmNiaS5ubG0ubmloLmdvdi9wdWJtZWQvMzE0MTk3NjUpe3RhcmdldD0iX2JsYW5rIn0gLSBmaW5lIHBhcnRpY2xlIGV4cG9zdXJlICgqKlBNfjIuNX4qKikgZnJvbSB0cmFmZmljLXJlbGF0ZWQgYWlyIHBvbGx1dGlvbiB3YXMgYXNzb2NpYXRlZCB3aXRoIGluY3JlYXNlZCBtZWFzdXJlcyBvZiBpbmZsYW1tYXRvcnkgbWFya2VycyBpbiB5b3V0aHMgd2l0aCBUeXBlIDEgZGlhYmV0ZXMKMykgW0x1bmcgZnVuY3Rpb24gYW5kIGVtcGh5c2VtYV0oaHR0cHM6Ly93d3cubmNiaS5ubG0ubmloLmdvdi9wdWJtZWQvMzE0MDgxMzUpe3RhcmdldD0iX2JsYW5rIn0gLSBoaWdoZXIgY29uY2VudHJhdGlvbnMgb2Ygb3pvbmUgKE9+M34pLCBuaXRyb2dlbiBveGlkZXMgKE5Pfnh+KSwgYmxhY2sgY2FyYm9uLCBhbmQgZmluZSBwYXJ0aWNsZSBleHBvc3VyZSAqKlBNfjIuNX4qKiAsIGF0IHN0dWR5IGJhc2VsaW5lIHdlcmUgc2lnbmlmaWNhbnRseSBhc3NvY2lhdGVkIHdpdGggZ3JlYXRlciBpbmNyZWFzZXMgaW4gcGVyY2VudCBlbXBoeXNlbWEgcGVyIDEwIHllYXJzIAo0KSBbTG93IGJpcnRod2VpZ2h0XShodHRwczovL3d3dy5uY2JpLm5sbS5uaWguZ292L3B1Ym1lZC8zMTM4NjY0Myl7dGFyZ2V0PSJfYmxhbmsifSAtIGZpbmUgcGFydGljbGUgZXhwb3N1cmUoKipQTX4yLjV+KiopIHdhcyBhc3NvY2lhdGVkIHdpdGggbG93ZXIgYmlydGggd2VpZ2h0IGluIGZ1bGwtdGVybSBsaXZlIGJpcnRocwo1KSBbVmlyYWwgSW5mZWN0aW9uXShodHRwczovL3d3dy50YW5kZm9ubGluZS5jb20vZG9pL2Z1bGwvMTAuMTA4MC8wODk1ODM3MDcwMTY2NTQzNCl7dGFyZ2V0PSJfYmxhbmsifSAtIGhpZ2hlciByYXRlcyBvZiBpbmZlY3Rpb24gYW5kIGluY3JlYXNlZCBzZXZlcml0eSBvZiBpbmZlY3Rpb24gYXJlIGFzc29jaWF0ZWQgd2l0aCBoaWdoZXIgZXhwb3N1cmVzIHRvIHBvbGx1dGlvbiBsZXZlbHMgaW5jbHVkaW5nIGZpbmUgcGFydGljbGUgZXhwb3N1cmUgKCoqUE1+Mi41fioqKQoKU2VlIHRoaXMgW3JldmlldyBhcnRpY2xlXShodHRwczovL3d3dy5mcm9udGllcnNpbi5vcmcvYXJ0aWNsZXMvMTAuMzM4OS9mcHViaC4yMDIwLjAwMDE0L2Z1bGwpe3RhcmdldD0iX2JsYW5rIn0gZm9yIG1vcmUgaW5mb3JtYXRpb24gYWJvdXQgc291cmNlcyBvZiBhaXIgcG9sbHV0aW9uIGFuZCB0aGUgaW5mbHVlbmNlIG9mIGFpciBwb2xsdXRpb24gb24gaGVhbHRoLgoKIyMjIFNwYXJzZSBtb25pdG9yaW5nIGlzIHByb2JsZW1hdGljIGZvciBQdWJsaWMgSGVhbHRoCgpIaXN0b3JpY2FsbHksIGVwaWRlbWlvbG9naWNhbCBzdHVkaWVzIHdvdWxkIGFzc2VzcyB0aGUgaW5mbHVlbmNlIG9mIGFpciBwb2xsdXRpb24gb24gaGVhbHRoIG91dGNvbWVzIGJ5IHJlbHlpbmcgb24gYSBudW1iZXIgb2YgbW9uaXRvcnMgbG9jYXRlZCBhcm91bmQgdGhlIGNvdW50cnkuIAoKSG93ZXZlciwgYXMgY2FuIGJlIHNlZW4gaW4gdGhlIGZvbGxvd2luZyBmaWd1cmUsIHRoZXNlIG1vbml0b3JzIGFyZSByZWxhdGl2ZWx5IHNwYXJzZSBpbiBjZXJ0YWluIHJlZ2lvbnMgb2YgdGhlIGNvdW50cnkgYW5kIGFyZSBub3QgbmVjZXNzYXJpbHkgbG9jYXRlZCBuZWFyIHBvbGx1dGlvbiBzb3VyY2VzLiBXZSB3aWxsIHNlZSBsYXRlciB3aGVuIHdlIGV2YWx1YXRlIHRoZSBkYXRhLCB0aGF0IGV2ZW4gaW4gY2VydGFpbiByZWxhdGl2ZWx5IGxhcmdlIGNpdGllcyB0aGVyZSBpcyBvbmx5ICBvbmUgbW9uaXRvciEKCkZ1cnRoZXJtb3JlLCBkcmFtYXRpYyBkaWZmZXJlbmNlcyBpbiBwb2xsdXRpb24gcmF0ZXMgY2FuIGJlIHNlZW4gZXZlbiB3aXRoaW4gdGhlIHNhbWUgY2l0eS4gSW4gZmFjdCwgdGhlIHRlcm0gbWljcm8tZW52aXJvbm1lbnRzIGRlc2NyaWJlcyBlbnZpcm9ubWVudHMgd2l0aGluIGNpdGllcyBvciBjb3VudGllcyB3aGljaCBtYXkgdmFyeSBncmVhdGx5IGZyb20gb25lIGJsb2NrIHRvIGFub3RoZXIuCgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGg9ICI4MDAgcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCAibWFwX29mX21vbml0b3JzLmpwZyIpKQpgYGAKCiMjIyMjIFtbc291cmNlXV0oaHR0cHM6Ly9laGpvdXJuYWwuYmlvbWVkY2VudHJhbC5jb20vYXJ0aWNsZXMvMTAuMTE4Ni8xNDc2LTA2OVgtMTMtNjMpe3RhcmdldD0iX2JsYW5rIn0KClRoaXMgbGFjayBvZiBncmFudWxhcml0eSBpbiBhaXIgcG9sbHV0aW9uIG1vbml0b3JpbmcgaGFzIGhpbmRlcmVkIG91ciBhYmlsaXR5IHRvIGRpc2Nlcm4gdGhlIGZ1bGwgaW1wYWN0IG9mIGFpciBwb2xsdXRpb24gb24gaGVhbHRoIGFuZCB0byBpZGVudGlmeSBhdC1yaXNrIGxvY2F0aW9ucy4gCgoKIyMjIE1hY2hpbmUgbGVhcm5pbmcgb2ZmZXJzIGEgc29sdXRpb24KCkFuIFthcnRpY2xlXShodHRwczovL2Voam91cm5hbC5iaW9tZWRjZW50cmFsLmNvbS9hcnRpY2xlcy8xMC4xMTg2LzE0NzYtMDY5WC0xMy02Myl7dGFyZ2V0PSJfYmxhbmsifSBwdWJsaXNoZWQgaW4gdGhlICpFbnZpcm9ubWVudGFsIEhlYWx0aCogam91cm5hbCBkZWFsdCB3aXRoIHRoaXMgaXNzdWUgYnkgdXNpbmcgZGF0YSwgaW5jbHVkaW5nIHBvcHVsYXRpb24gZGVuc2l0eSBhbmQgcm9hZCBkZW5zaXR5LCBhbW9uZyBvdGhlciBmZWF0dXJlcywgdG8gbW9kZWwgb3IgcHJlZGljdCBhaXIgcG9sbHV0aW9uIGxldmVscyBhdCBhIG1vcmUgbG9jYWxpemVkIHNjYWxlIHVzaW5nIG1hY2hpbmUgbGVhcm5pbmcgKE1MKSBtZXRob2RzLiAKCmBgYHtyLCBlY2hvID0gRkFMU0UsIG91dC53aWR0aD0gIjgwMCBweCJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsICJ0aGVwYXBlci5wbmciKSkKYGBgCgojIyMjIyBbW3NvdXJjZV1dKGh0dHBzOi8vZWhqb3VybmFsLmJpb21lZGNlbnRyYWwuY29tL2FydGljbGVzLzEwLjExODYvMTQ3Ni0wNjlYLTEzLTYzKXt0YXJnZXQ9Il9ibGFuayJ9CgojIyMjIHsucmVmZXJlbmNlX2Jsb2NrfQpZYW5vc2t5LCBKLiBELiBldCBhbC4gU3BhdGlvLXRlbXBvcmFsIG1vZGVsaW5nIG9mIHBhcnRpY3VsYXRlIGFpciBwb2xsdXRpb24gaW4gdGhlIGNvbnRlcm1pbm91cyBVbml0ZWQgU3RhdGVzIHVzaW5nIGdlb2dyYXBoaWMgYW5kIG1ldGVvcm9sb2dpY2FsIHByZWRpY3RvcnMuICpFbnZpcm9uIEhlYWx0aCogMTMsIDYzICgyMDE0KS4KCiMjIyMKClRoZSBhdXRob3JzIG9mIHRoaXMgYXJ0aWNsZSBzdGF0ZSB0aGF0OgoKPiAiRXhwb3N1cmUgdG8gYXRtb3NwaGVyaWMgcGFydGljdWxhdGUgbWF0dGVyIChQTSkgcmVtYWlucyBhbiBpbXBvcnRhbnQgcHVibGljIGhlYWx0aCBjb25jZXJuLCBhbHRob3VnaCBpdCByZW1haW5zIGRpZmZpY3VsdCB0byBxdWFudGlmeSBhY2N1cmF0ZWx5IGFjcm9zcyBsYXJnZSBnZW9ncmFwaGljIGFyZWFzIHdpdGggc3VmZmljaWVudGx5IGhpZ2ggc3BhdGlhbCByZXNvbHV0aW9uLiBSZWNlbnQgZXBpZGVtaW9sb2dpYyBhbmFseXNlcyBoYXZlIGRlbW9uc3RyYXRlZCB0aGUgaW1wb3J0YW5jZSBvZiBzcGF0aWFsbHktIGFuZCB0ZW1wb3JhbGx5LXJlc29sdmVkIGV4cG9zdXJlIGVzdGltYXRlcywgd2hpY2ggc2hvdyBsYXJnZXIgUE0tbWVkaWF0ZWQgaGVhbHRoIGVmZmVjdHMgYXMgY29tcGFyZWQgdG8gbmVhcmVzdCBtb25pdG9yIG9yIGNvdW50eS1zcGVjaWZpYyBhbWJpZW50IGNvbmNlbnRyYXRpb25zLiIgCgojIyMjIyBbW3NvdXJjZV1dKGh0dHBzOi8vZWhqb3VybmFsLmJpb21lZGNlbnRyYWwuY29tL2FydGljbGVzLzEwLjExODYvMTQ3Ni0wNjlYLTEzLTYzKXt0YXJnZXQ9Il9ibGFuayJ9CgpUaGUgYXJ0aWNsZSBhYm92ZSBkZW1vbnN0cmF0ZXMgdGhhdCBtYWNoaW5lIGxlYXJuaW5nIG1ldGhvZHMgY2FuIGJlIHVzZWQgdG8gcHJlZGljdCBhaXIgcG9sbHV0aW9uIGxldmVscyB3aGVuIHRyYWRpdGlvbmFsIG1vbml0b3Jpbmcgc3lzdGVtcyBhcmUgbm90IGF2YWlsYWJsZSBpbiBhIHBhcnRpY3VsYXIgYXJlYSBvciB3aGVuIHRoZXJlIGlzIG5vdCBlbm91Z2ggc3BhdGlhbCBncmFudWxhcml0eSB3aXRoIGN1cnJlbnQgbW9uaXRvcmluZyBzeXN0ZW1zLiAKV2Ugd2lsbCB1c2Ugc2ltaWxhciBtZXRob2RzIHRvIHByZWRpY3QgYW5udWFsIGFpciBwb2xsdXRpb24gbGV2ZWxzIHNwYXRpYWxseSB3aXRoaW4gdGhlIFVTLgoKCiMgKipNYWluIFF1ZXN0aW9uKioKKioqCgojIyMjIHsubWFpbl9xdWVzdGlvbl9ibG9ja30KPGI+PHU+IE91ciBtYWluIHF1ZXN0aW9uOiA8L3U+PC9iPgoKMSkgQ2FuIHdlIHByZWRpY3QgYW5udWFsIGF2ZXJhZ2UgYWlyIHBvbGx1dGlvbiBjb25jZW50cmF0aW9ucyBhdCB0aGUgZ3JhbnVsYXJpdHkgb2YgemlwIGNvZGUgcmVnaW9uYWwgbGV2ZWxzIHVzaW5nIHByZWRpY3RvcnMgc3VjaCBhcyBkYXRhIGFib3V0IHBvcHVsYXRpb24gZGVuc2l0eSwgdXJiYW5pemF0aW9uLCByb2FkIGRlbnNpdHksIGFzIHdlbGwgYXMsIHNhdGVsbGl0ZSBwb2xsdXRpb24gZGF0YSBhbmQgY2hlbWljYWwgbW9kZWxpbmcgZGF0YT8KCiMjIyMKCiMgKipMZWFybmluZyBPYmplY3RpdmVzKioKKioqCgpJbiB0aGlzIGNhc2Ugc3R1ZHksIHdlIHdpbGwgd2FsayB5b3UgdGhyb3VnaCBpbXBvcnRpbmcgZGF0YSBmcm9tIENTViBmaWxlcyBhbmQgcGVyZm9ybWluZyBtYWNoaW5lIGxlYXJuaW5nIG1ldGhvZHMgdG8gcHJlZGljdCBvdXIgb3V0Y29tZSB2YXJpYWJsZSBvZiBpbnRlcmVzdCAoaW4gdGhpcyBjYXNlIGFubnVhbCBmaW5lIHBhcnRpY2xlIGFpciBwb2xsdXRpb24gZXN0aW1hdGVzKS4gCgpXZSB3aWxsIGVzcGVjaWFsbHkgZm9jdXMgb24gdXNpbmcgcGFja2FnZXMgYW5kIGZ1bmN0aW9ucyBmcm9tIHRoZSBbYHRpZHl2ZXJzZWBdKGh0dHBzOi8vd3d3LnRpZHl2ZXJzZS5vcmcvKXt0YXJnZXQ9Il9ibGFuayJ9LCBhbmQgbW9yZSBzcGVjaWZpY2FsbHkgdGhlIFtgdGlkeW1vZGVsc2BdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy90aWR5bW9kZWxzL3RpZHltb2RlbHMucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IHBhY2thZ2UvZWNvc3lzdGVtIHByaW1hcmlseSBkZXZlbG9wZWQgYW5kIG1haW50YWluZWQgYnkgW01heCBLdWhuXShodHRwczovL3Jlc291cmNlcy5yc3R1ZGlvLmNvbS9hdXRob3JzL21heC1rdWhuKXt0YXJnZXQ9Il9ibGFuayJ9IGFuZCBbRGF2aXMgVmF1Z2hhbl0oaHR0cHM6Ly9yZXNvdXJjZXMucnN0dWRpby5jb20vYXV0aG9ycy9kYXZpcy12YXVnaGFuKXt0YXJnZXQ9Il9ibGFuayJ9LiAKVGhpcyBwYWNrYWdlIGxvYWRzIG1vcmUgbW9kZWxpbmcgcmVsYXRlZCBwYWNrYWdlcyBsaWtlIGByc2FtcGxlYCwgYHJlY2lwZXNgLCBgcGFyc25pcGAsIGB5YXJkc3RpY2tgLCBgd29ya2Zsb3dzYCwgYW5kIGB0dW5lYCBwYWNrYWdlcy4gCgpUaGUgdGlkeXZlcnNlIGlzIGEgbGlicmFyeSBvZiBwYWNrYWdlcyBjcmVhdGVkIGJ5IFJTdHVkaW8uIApXaGlsZSBzb21lIHN0dWRlbnRzIG1heSBiZSBmYW1pbGlhciB3aXRoIHByZXZpb3VzIFIgcHJvZ3JhbW1pbmcgcGFja2FnZXMsIHRoZXNlIHBhY2thZ2VzIG1ha2UgZGF0YSBzY2llbmNlIGluIFIgZXNwZWNpYWxseSBsZWdpYmxlIGFuZCBpbnR1aXRpdmUuCgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgZmlnLnNob3cgPSAiaG9sZCIsIG91dC53aWR0aCA9ICIyMCUiLCBmaWcuYWxpZ24gPSAiZGVmYXVsdCJ9CmluY2x1ZGVfZ3JhcGhpY3MoImh0dHBzOi8vdGlkeXZlcnNlLnRpZHl2ZXJzZS5vcmcvbG9nby5wbmciKQppbmNsdWRlX2dyYXBoaWNzKCJodHRwczovL3Bicy50d2ltZy5jb20vbWVkaWEvRGtCRnBTc1c0QUl5eUlOLnBuZyIpCmBgYAoKVGhlIHNraWxscywgbWV0aG9kcywgYW5kIGNvbmNlcHRzIHRoYXQgc3R1ZGVudHMgd2lsbCBiZSBmYW1pbGlhciB3aXRoIGJ5IHRoZSBlbmQgb2YgdGhpcyBjYXNlIHN0dWR5IGFyZToKCgo8dT4qKkRhdGEgU2NpZW5jZSBMZWFybmluZyBPYmplY3RpdmVzOioqPC91PiAKICAKMS4gRmFtaWxpYXJpdHkgd2l0aCB0aGUgdGlkeW1vZGVscyBlY29zeXN0ZW0uCjIuIEFiaWxpdHkgdG8gZXZhbHVhdGUgY29ycmVsYXRpb24gYW1vbmcgcHJlZGljdG9yIHZhcmlhYmxlcyAoYGNvcnJwbG90YCBhbmQgYEdHYWxseWApLgozLiBBYmlsaXR5IHRvIGltcGxlbWVudCB0aWR5bW9kZWxzIHBhY2thZ2VzIHN1Y2ggYXMgYHJzYW1wbGVgIHRvIHNwbGl0IHRoZSBkYXRhIGludG8gdHJhaW5pbmcgYW5kIHRlc3Rpbmcgc2V0cyBhcyB3ZWxsIGFzIGNyb3NzIHZhbGlkYXRpb24gc2V0cy4KNC4gQWJpbGl0eSB0byB1c2UgdGhlIGByZWNpcGVzYCwgYHBhcnNuaXBgLCBhbmQgYHdvcmtmbG93c2AgdG8gdHJhaW4gYW5kIHRlc3QgYSBsaW5lYXIgcmVncmVzc2lvbiBtb2RlbCBhbmQgcmFuZG9tIGZvcmVzdCBtb2RlbC4KNS4gRGVtb25zdHJhdGUgaG93IHRvIHZpc3VhbGl6ZSBnZW8tc3BhdGlhbCBkYXRhIHVzaW5nIGBnZ3Bsb3QyYC4KCjx1PioqU3RhdGlzdGljYWwgTGVhcm5pbmcgT2JqZWN0aXZlczoqKjwvdT4gIAogIAoxLiBCYXNpYyB1bmRlcnN0YW5kaW5nIHRoZSB1dGlsaXR5IG9mIG1hY2hpbmUgbGVhcm5pbmcgZm9yIHByZWRpY3Rpb24gYW5kIGNsYXNzaWZpY2F0aW9uCjIuIFVuZGVyc3RhbmRpbmcgb2YgdGhlIG5lZWQgZm9yIHRyYWluaW5nIGFuZCB0ZXN0IHNldHMKMy4gVW5kZXJzdGFuZGluZyBvZiB0aGUgdXRpbGl0eSBvZiBjcm9zcyB2YWxpZGF0aW9uCjQuIFVuZGVyc3RhbmRpbmcgb2YgcmFuZG9tIGZvcmVzdAo1LiBIb3cgdG8gaW50ZXJwcmV0IHJvb3QgbWVhbiBzcXVhcmVkIGVycm9yIChybXNlKSB0byBhc3Nlc3MgcGVyZm9ybWFuY2UgZm9yIHByZWRpY3Rpb24KCldlIHdpbGwgYmVnaW4gYnkgbG9hZGluZyB0aGUgcGFja2FnZXMgdGhhdCB3ZSB3aWxsIG5lZWQ6CgpgYGB7cn0KIyBMb2FkIHBhY2thZ2VzIGZvciBkYXRhIGltcG9ydCBhbmQgZGF0YSB3cmFuZ2xpbmcKbGlicmFyeShoZXJlKQpsaWJyYXJ5KHJlYWRyKQpsaWJyYXJ5KGRwbHlyKQpsaWJyYXJ5KHNraW1yKQpsaWJyYXJ5KHN1bW1hcnl0b29scykKbGlicmFyeShtYWdyaXR0cikKIyBMb2FkIHBhY2thZ2VzIGZvciBtYWtpbmcgY29ycmVsYXRpb24gcGxvdHMKbGlicmFyeShjb3JycGxvdCkKbGlicmFyeShSQ29sb3JCcmV3ZXIpCmxpYnJhcnkoR0dhbGx5KQojIExvYWQgcGFja2FnZXMgZm9yIGJ1aWxkaW5nIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtCmxpYnJhcnkodGlkeW1vZGVscykKbGlicmFyeSh3b3JrZmxvd3MpCmxpYnJhcnkodmlwKQpsaWJyYXJ5KHR1bmUpCmxpYnJhcnkocmFuZG9tRm9yZXN0KQpsaWJyYXJ5KGRvUGFyYWxsZWwpCiMgTG9hZCBwYWNrYWdlcyBmb3IgZGF0YSB2aXN1YWxpemF0aW9uL2NyZWF0aW5nIG1hcApsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkoc3RyaW5ncikKbGlicmFyeSh0aWR5cikKbGlicmFyeShsd2dlb20pCmxpYnJhcnkocHJveHkpIyBuZWVkZWQgZm9yIGx3Z2VvbQpsaWJyYXJ5KHNmKQpsaWJyYXJ5KG1hcHMpCmxpYnJhcnkocm5hdHVyYWxlYXJ0aCkKbGlicmFyeShybmF0dXJhbGVhcnRoZGF0YSkgIyBuZWVkZWQgZm9yIHJuYXR1cmFsZWFydGgKbGlicmFyeShyZ2VvcykKbGlicmFyeShwYXRjaHdvcmspCiMgTG9hZCBwYWNrYWdlIGZvciBkb3dubG9hZGluZyB0aGUgY2FzZSBzdHVkeSBkYXRhIGZpbGVzCmxpYnJhcnkoT0NTZGF0YSkKYGBgCgoKIDx1PioqUGFja2FnZXMgdXNlZCBpbiB0aGlzIGNhc2Ugc3R1ZHk6KiogPC91PgoKUGFja2FnZSAgIHwgVXNlIGluIHRoaXMgY2FzZSBzdHVkeSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKLS0tLS0tLS0tLSB8LS0tLS0tLS0tLS0tLQpbaGVyZV0oaHR0cHM6Ly9naXRodWIuY29tL2plbm55YmMvaGVyZV9oZXJlKXt0YXJnZXQ9Il9ibGFuayJ9ICAgICAgIHwgdG8gZWFzaWx5IGxvYWQgYW5kIHNhdmUgZGF0YQpbcmVhZHJdKGh0dHBzOi8vcmVhZHIudGlkeXZlcnNlLm9yZy8pe3RhcmdldD0iX2JsYW5rIn0gICAgICB8IHRvIGltcG9ydCBDU1YgZmlsZXMKW2RwbHlyXShodHRwczovL2RwbHlyLnRpZHl2ZXJzZS5vcmcvKXt0YXJnZXQ9Il9ibGFuayJ9ICAgICAgfCB0byB2aWV3L2FycmFuZ2UvZmlsdGVyL3NlbGVjdC9jb21wYXJlIHNwZWNpZmljIHN1YnNldHMgb2YgZGF0YSAKW3NraW1yXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvc2tpbXIvaW5kZXguaHRtbCl7dGFyZ2V0PSJfYmxhbmsifSAgICAgIHwgdG8gZ2V0IGFuIG92ZXJ2aWV3IG9mIGRhdGEKW3N1bW1hcnl0b29sc10oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL3NraW1yL2luZGV4Lmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gICAgICB8IHRvIGdldCBhbiBvdmVydmlldyBvZiBkYXRhIGluIGEgZGlmZmVyZW50IHN0eWxlClttYWdyaXR0cl0oaHR0cHM6Ly9tYWdyaXR0ci50aWR5dmVyc2Uub3JnL2FydGljbGVzL21hZ3JpdHRyLmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gICB8IHRvIHVzZSB0aGUgYCU8PiVgIHBpcHBpbmcgb3BlcmF0b3IgCltjb3JycGxvdF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL2NvcnJwbG90L3ZpZ25ldHRlcy9jb3JycGxvdC1pbnRyby5odG1sKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gbWFrZSBsYXJnZSBjb3JyZWxhdGlvbiBwbG90cwpbR0dhbGx5XShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvR0dhbGx5L0dHYWxseS5wZGYpe3RhcmdldD0iX2JsYW5rIn0gfCB0byBtYWtlIHNtYWxsZXIgY29ycmVsYXRpb24gcGxvdHMgIApbdGlkeW1vZGVsc10oaHR0cHM6Ly93d3cudGlkeW1vZGVscy5vcmcpe3RhcmdldD0iX2JsYW5rIn0gfCB0byBsb2FkIGluIGEgc2V0IG9mIHBhY2thZ2VzIChicm9vbSwgZGlhbHMsIGluZmVyLCBwYXJzbmlwLCBwdXJyciwgcmVjaXBlcywgcnNhbXBsZSwgdGliYmxlLCB5YXJkc3RpY2spCltyc2FtcGxlXShodHRwczovL3RpZHltb2RlbHMuZ2l0aHViLmlvL3JzYW1wbGUvYXJ0aWNsZXMvQmFzaWNzLmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gICB8IHRvIHNwbGl0IHRoZSBkYXRhIGludG8gdGVzdGluZyBhbmQgdHJhaW5pbmcgc2V0czsgdG8gc3BsaXQgdGhlIHRyYWluaW5nIHNldCBmb3IgY3Jvc3MtdmFsaWRhdGlvbiAgCltyZWNpcGVzXShodHRwczovL3RpZHltb2RlbHMuZ2l0aHViLmlvL3JlY2lwZXMvKXt0YXJnZXQ9Il9ibGFuayJ9ICAgfCB0byBwcmUtcHJvY2VzcyBkYXRhIGZvciBtb2RlbGluZyBpbiBhIHRpZHkgYW5kIHJlcHJvZHVjaWJsZSB3YXkgYW5kIHRvIGV4dHJhY3QgcHJlLXByb2Nlc3NlZCBkYXRhIChtYWpvciBmdW5jdGlvbnMgYXJlIGByZWNpcGUoKWAsIGBwcmVwKClgIGFuZCB2YXJpb3VzIHRyYW5zZm9ybWF0aW9uIGBzdGVwXyooKWAgZnVuY3Rpb25zLCBhcyB3ZWxsIGFzIGBiYWtlYCB3aGljaCBleHRyYWN0cyBwcmUtcHJvY2Vzc2VkIHRyYWluaW5nIGRhdGEgKHVzZWQgdG8gcmVxdWlyZSBganVpY2UoKWApIGFuZCBhcHBsaWVzIHJlY2lwZSBwcmVwcm9jZXNzaW5nIHN0ZXBzIHRvIHRlc3RpbmcgZGF0YSkuIFNlZSBbaGVyZV0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL3JlY2lwZXMvcmVjaXBlcy5wZGYpe3RhcmdldD0iX2JsYW5rIn0gIGZvciBtb3JlIGluZm8uCltwYXJzbmlwXShodHRwczovL3RpZHltb2RlbHMuZ2l0aHViLmlvL3BhcnNuaXAvKXt0YXJnZXQ9Il9ibGFuayJ9ICAgfCBhbiBpbnRlcmZhY2UgdG8gY3JlYXRlIG1vZGVscyAobWFqb3IgZnVuY3Rpb25zIGFyZSBgZml0KClgLCBgc2V0X2VuZ2luZSgpYCkKW3lhcmRzdGlja10oaHR0cHM6Ly90aWR5bW9kZWxzLmdpdGh1Yi5pby95YXJkc3RpY2svKXt0YXJnZXQ9Il9ibGFuayJ9ICAgfCB0byBldmFsdWF0ZSB0aGUgcGVyZm9ybWFuY2Ugb2YgbW9kZWxzClticm9vbV0oaHR0cHM6Ly93d3cudGlkeXZlcnNlLm9yZy9ibG9nLzIwMTgvMDcvYnJvb20tMC01LTAvKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gZ2V0IHRpZHkgb3V0cHV0IGZvciBvdXIgbW9kZWwgZml0IGFuZCBwZXJmb3JtYW5jZQpbZ2dwbG90Ml0oaHR0cHM6Ly9nZ3Bsb3QyLnRpZHl2ZXJzZS5vcmcvKXt0YXJnZXQ9Il9ibGFuayJ9ICAgIHwgdG8gbWFrZSB2aXN1YWxpemF0aW9ucyB3aXRoIG11bHRpcGxlIGxheWVycwpbZGlhbHNdKGh0dHBzOi8vd3d3LnRpZHl2ZXJzZS5vcmcvYmxvZy8yMDE5LzEwL2RpYWxzLTAtMC0zLyl7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIHNwZWNpZnkgaHlwZXItcGFyYW1ldGVyIHR1bmluZwpbdHVuZV0oaHR0cHM6Ly90dW5lLnRpZHltb2RlbHMub3JnLyl7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIHBlcmZvcm0gY3Jvc3MgdmFsaWRhdGlvbiwgdHVuZSBoeXBlci1wYXJhbWV0ZXJzLCBhbmQgZ2V0IHBlcmZvcm1hbmNlIG1ldHJpY3MKW3dvcmtmbG93c10oaHR0cHM6Ly93d3cucmRvY3VtZW50YXRpb24ub3JnL3BhY2thZ2VzL3dvcmtmbG93cy92ZXJzaW9ucy8wLjEuMSl7dGFyZ2V0PSJfYmxhbmsifXwgdG8gY3JlYXRlIG1vZGVsaW5nIHdvcmtmbG93IHRvIHN0cmVhbWxpbmUgdGhlIG1vZGVsaW5nIHByb2Nlc3MKW3ZpcF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL3ZpcC92aXAucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gY3JlYXRlIHZhcmlhYmxlIGltcG9ydGFuY2UgcGxvdHMKW3JhbmRvbUZvcmVzdF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL3JhbmRvbUZvcmVzdC9yYW5kb21Gb3Jlc3QucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gcGVyZm9ybSB0aGUgcmFuZG9tIGZvcmVzdCBhbmFseXNpcwpbZG9QYXJhbGxlbF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL2RvUGFyYWxsZWwvZG9QYXJhbGxlbC5wZGYpIHwgdG8gZml0IGNyb3NzIHZhbGlkYXRpb24gc2FtcGxlcyBpbiBwYXJhbGxlbCAKW3N0cmluZ3JdKGh0dHBzOi8vc3RyaW5nci50aWR5dmVyc2Uub3JnL2FydGljbGVzL3N0cmluZ3IuaHRtbCl7dGFyZ2V0PSJfYmxhbmsifSAgICB8IHRvIG1hbmlwdWxhdGUgdGhlIHRleHQgdGhlIG1hcCBkYXRhClt0aWR5cl0oaHR0cHM6Ly90aWR5ci50aWR5dmVyc2Uub3JnLyl7dGFyZ2V0PSJfYmxhbmsifSAgICAgIHwgdG8gc2VwYXJhdGUgZGF0YSB3aXRoaW4gYSBjb2x1bW4gaW50byBtdWx0aXBsZSBjb2x1bW5zCltybmF0dXJhbGVhcnRoXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvcm5hdHVyYWxlYXJ0aC9SRUFETUUuaHRtbCl7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIGdldCB0aGUgZ2VvbWV0cnkgZGF0YSBmb3IgdGhlIGVhcnRoIHRvIHBsb3QgdGhlIFVTClttYXBzXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvbWFwcy9tYXBzLnBkZil7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIGdldCBtYXAgZGF0YWJhc2UgZGF0YSBhYm91dCBjb3VudGllcyB0byBkcmF3IHRoZW0gb24gb3VyIFVTIG1hcApbc2ZdKGh0dHBzOi8vci1zcGF0aWFsLmdpdGh1Yi5pby9zZi8pe3RhcmdldD0iX2JsYW5rIn0gfCB0byBjb252ZXJ0IHRoZSBtYXAgZGF0YSBpbnRvIGEgZGF0YSBmcmFtZQpbbHdnZW9tXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvbHdnZW9tL2x3Z2VvbS5wZGYpe3RhcmdldD0iX2JsYW5rIn0gfCB0byB1c2UgdGhlIGBzZmAgZnVuY3Rpb24gdG8gY29udmVydCBtYXAgZ2VvZ3JhcGhpY2FsIGRhdGEKW3JnZW9zXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvcmdlb3Mvcmdlb3MucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gdXNlIGdlb21ldHJ5IGRhdGEKW3BhdGNod29ya10oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL3BhdGNod29yay9wYXRjaHdvcmsucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gYWxsb3cgcGxvdHMgdG8gYmUgY29tYmluZWQKW09DU2RhdGFdKGh0dHBzOi8vZ2l0aHViLmNvbS9vcGVuY2FzZXN0dWRpZXMvT0NTZGF0YSl7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIGFjY2VzcyBhbmQgZG93bmxvYWQgT0NTIGRhdGEgZmlsZXMKX19fCgoKVGhlIGZpcnN0IHRpbWUgd2UgdXNlIGEgZnVuY3Rpb24sIHdlIHdpbGwgdXNlIHRoZSBgOjpgIHRvIGluZGljYXRlIHdoaWNoIHBhY2thZ2Ugd2UgYXJlIHVzaW5nLiAKVW5sZXNzIHdlIGhhdmUgb3ZlcmxhcHBpbmcgZnVuY3Rpb24gbmFtZXMsIHRoaXMgaXMgbm90IG5lY2Vzc2FyeSwgYnV0IHdlIHdpbGwgaW5jbHVkZSBpdCBoZXJlIHRvIGJlIGluZm9ybWF0aXZlIGFib3V0IHdoZXJlIHRoZSBmdW5jdGlvbnMgd2Ugd2lsbCB1c2UgY29tZSBmcm9tLgoKCiMgKipDb250ZXh0KioKKioqCgpUaGUgW1N0YXRlIG9mIEdsb2JhbCBBaXJdKGh0dHBzOi8vd3d3LnN0YXRlb2ZnbG9iYWxhaXIub3JnLyl7dGFyZ2V0PSJfYmxhbmsifSBpcyBhIHJlcG9ydCByZWxlYXNlZCBldmVyeSB5ZWFyIHRvIGNvbW11bmljYXRlIHRoZSBpbXBhY3Qgb2YgYWlyIHBvbGx1dGlvbiBvbiBwdWJsaWMgaGVhbHRoLiAKClRoZSBbU3RhdGUgb2YgR2xvYmFsIEFpciAyMDE5IHJlcG9ydF0oaHR0cHM6Ly93d3cuc3RhdGVvZmdsb2JhbGFpci5vcmcvc2l0ZXMvZGVmYXVsdC9maWxlcy9zb2dhXzIwMTlfcmVwb3J0LnBkZil7dGFyZ2V0PSJfYmxhbmsifQp3aGljaCB1c2VzIGRhdGEgZnJvbSAyMDE3IHN0YXRlZCB0aGF0OgoKPiBBaXIgcG9sbHV0aW9uIGlzIHRoZSAqKmZpZnRoKiogbGVhZGluZyByaXNrIGZhY3RvciBmb3IgbW9ydGFsaXR5IHdvcmxkd2lkZS4gSXQgaXMgcmVzcG9uc2libGUgZm9yIG1vcmUKZGVhdGhzIHRoYW4gbWFueSBiZXR0ZXIta25vd24gcmlzayBmYWN0b3JzIHN1Y2ggYXMgbWFsbnV0cml0aW9uLCBhbGNvaG9sIHVzZSwgYW5kIHBoeXNpY2FsIGluYWN0aXZpdHkuCkVhY2ggeWVhciwgKiptb3JlKiogcGVvcGxlIGRpZSBmcm9tIGFpciBwb2xsdXRpb27igJNyZWxhdGVkIGRpc2Vhc2UgdGhhbiBmcm9tIHJvYWQgKip0cmFmZmljIGluanVyaWVzKiogb3IgKiptYWxhcmlhKiouCgo8cCBhbGlnbj0iY2VudGVyIj4KPGltZyB3aWR0aD0iNjAwIiBzcmM9Imh0dHBzOi8vd3d3LmhlYWx0aGVmZmVjdHMub3JnL3NpdGVzL2RlZmF1bHQvZmlsZXMvU29HQS1GaWd1cmVzLTAxLmpwZyI+CjwvcD4KCiMjIyMjIFtbc291cmNlXV0oaHR0cHM6Ly93d3cuc3RhdGVvZmdsb2JhbGFpci5vcmcvc2l0ZXMvZGVmYXVsdC9maWxlcy9zb2dhXzIwMTlfcmVwb3J0LnBkZil7dGFyZ2V0PSJfYmxhbmsifQoKVGhlIHJlcG9ydCBhbHNvIHN0YXRlZCB0aGF0OgoKPiBJbiAyMDE3LCBhaXIgcG9sbHV0aW9uIGlzIGVzdGltYXRlZCB0byBoYXZlIGNvbnRyaWJ1dGVkIHRvIGNsb3NlIHRvIDUgbWlsbGlvbgpkZWF0aHMgZ2xvYmFsbHkg4oCUIG5lYXJseSAqKjEgaW4gZXZlcnkgMTAgZGVhdGhzKiouCgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGg9IjgwMHB4In0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwiMjAxN2RlYXRocy5wbmciKSkKYGBgCgojIyMjIyBbW3NvdXJjZV1dKGh0dHBzOi8vd3d3LnN0YXRlb2ZnbG9iYWxhaXIub3JnL3NpdGVzL2RlZmF1bHQvZmlsZXMvc29nYV8yMDE5X2ZhY3Rfc2hlZXQucGRmKXt0YXJnZXQ9Il9ibGFuayJ9CgpUaGUgW1N0YXRlIG9mIEdsb2JhbCBBaXIgMjAxOCByZXBvcnRdKGh0dHBzOi8vd3d3LnN0YXRlb2ZnbG9iYWxhaXIub3JnL3NpdGVzL2RlZmF1bHQvZmlsZXMvc29nYS0yMDE4LXJlcG9ydC5wZGYpe3RhcmdldD0iX2JsYW5rIn0gdXNpbmcgZGF0YSBmcm9tIDIwMTYgd2hpY2ggc2VwYXJhdGVkIGRpZmZlcmVudCB0eXBlcyBvZiBhaXIgcG9sbHV0aW9uLCBmb3VuZCB0aGF0ICoqcGFydGljdWxhdGUgcG9sbHV0aW9uIHdhcyBwYXJ0aWN1bGFybHkgYXNzb2NpYXRlZCB3aXRoIG1vcnRhbGl0eSoqLgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoPSI4MDBweCJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsIjIwMTdtb3J0YWxpdHkucG5nIikpCmBgYAoKIyMjIyMgW1tzb3VyY2VdXShodHRwczovL3d3dy5zdGF0ZW9mZ2xvYmFsYWlyLm9yZy9zaXRlcy9kZWZhdWx0L2ZpbGVzL3NvZ2EtMjAxOC1yZXBvcnQucGRmKXt0YXJnZXQ9Il9ibGFuayJ9CgpUaGUgMjAxOSByZXBvcnQgc2hvd3MgdGhhdCB0aGUgaGlnaGVzdCBsZXZlbHMgb2YgZmluZSBwYXJ0aWN1bGF0ZSBwb2xsdXRpb24gb2NjdXIgaW4gQWZyaWNhIGFuZCBBc2lhIGFuZCB0aGF0OgoKPiBNb3JlIHRoYW4gKio5MCUqKiBvZiBwZW9wbGUgd29ybGR3aWRlIGxpdmUgaW4gYXJlYXMgKipleGNlZWRpbmcqKiB0aGUgV29ybGQgSGVhbHRoIE9yZ2FuaXphdGlvbiAoV0hPKSAqKkd1aWRlbGluZSoqIGZvciBoZWFsdGh5IGFpci4gTW9yZSB0aGFuIGhhbGYgbGl2ZSBpbiBhcmVhcyB0aGF0IGRvIG5vdCBldmVuIG1lZXQgV0hPJ3MgbGVhc3Qtc3RyaW5nZW50IGFpciBxdWFsaXR5IHRhcmdldC4KCmBgYHtyLCBlY2hvID0gRkFMU0UsIG91dC53aWR0aD0iODAwcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCJQTXdvcmxkLnBuZyIpKQpgYGAKCiMjIyMjIFtbc291cmNlXV0oaHR0cHM6Ly93d3cuc3RhdGVvZmdsb2JhbGFpci5vcmcvc2l0ZXMvZGVmYXVsdC9maWxlcy9zb2dhXzIwMTlfZmFjdF9zaGVldC5wZGYpe3RhcmdldD0iX2JsYW5rIn0KCkxvb2tpbmcgYXQgdGhlIFVTIHNwZWNpZmljYWxseSwgYWlyIHBvbGx1dGlvbiBsZXZlbHMgYXJlIGdlbmVyYWxseSBpbXByb3ZpbmcsIHdpdGggZGVjbGluaW5nIG5hdGlvbmFsIGFpciBwb2xsdXRhbnQgY29uY2VudHJhdGlvbiBhdmVyYWdlcyBhcyBzaG93biBmcm9tIHRoZSAyMDE5IFsqT3VyIE5hdGlvbidzIEFpcipdKGh0dHBzOi8vZ2lzcHViLmVwYS5nb3YvYWlyL3RyZW5kc3JlcG9ydC8yMDE5LyNob21lKXt0YXJnZXQ9Il9ibGFuayJ9IHJlcG9ydCBmcm9tIHRoZSBVUyBFbnZpcm9ubWVudGFsIFByb3RlY3Rpb24gQWdlbmN5IChFUEEpOiAKCmBgYHtyLCBlY2hvID0gRkFMU0V9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsICJVUy5wbmciKSkKYGBgCgojIyMjIyBbW3NvdXJjZV1dKGh0dHBzOi8vZ2lzcHViLmVwYS5nb3YvYWlyL3RyZW5kc3JlcG9ydC8yMDE5L2RvY3VtZW50YXRpb24vQWlyVHJlbmRzX0ZseWVyLnBkZil7dGFyZ2V0PSJfYmxhbmsifQoKSG93ZXZlciwgYWlyIHBvbGx1dGlvbiAqKmNvbnRpbnVlcyB0byBjb250cmlidXRlIHRvIGhlYWx0aCByaXNrIGZvciBBbWVyaWNhbnMqKiwgaW4gcGFydGljdWxhciBpbiAqKnJlZ2lvbnMgd2l0aCBoaWdoZXIgdGhhbiBuYXRpb25hbCBhdmVyYWdlIHJhdGVzKiogb2YgcG9sbHV0aW9uIHRoYXQsIGF0IHRpbWVzLCBleGNlZWQgdGhlIFdITydzIHJlY29tbWVuZGVkIGxldmVsLiAKVGh1cywgaXQgaXMgaW1wb3J0YW50IHRvIG9idGFpbiBoaWdoIHNwYXRpYWwgZ3JhbnVsYXJpdHkgaW4gZXN0aW1hdGVzIG9mIGFpciBwb2xsdXRpb24gaW4gb3JkZXIgdG8gaWRlbnRpZnkgbG9jYXRpb25zIHdoZXJlIHBvcHVsYXRpb25zIGFyZSBleHBlcmllbmNpbmcgaGFybWZ1bCBsZXZlbHMgb2YgZXhwb3N1cmUuCgpZb3UgY2FuIHNlZSB0aGF0IGN1cnJlbnQgYWlyIHF1YWxpdHkgY29uZGl0aW9ucyBhdCB0aGlzIFt3ZWJzaXRlXShodHRwczovL2FxaWNuLm9yZy9jaXR5L3VzYS8pe3RhcmdldD0iX2JsYW5rIn0sIGFuZCB5b3Ugd2lsbCBub3RpY2UgdmFyaWF0aW9uIGFjcm9zcyBkaWZmZXJlbnQgY2l0aWVzLgoKRm9yIGV4YW1wbGUsIGhlcmUgYXJlIHRoZSBjb25kaXRpb25zIGluIFRvcGVrYSBLYW5zYXMgYXQgdGhlIHRpbWUgdGhpcyBjYXNlIHN0dWR5IHdhcyBjcmVhdGVkOgoKYGBge3IsIGVjaG8gPSBGQUxTRX0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwgIkthbnNhcy5wbmciKSkKYGBgCgojIyMjIyBbW3NvdXJjZV1dKGh0dHBzOi8vYXFpY24ub3JnL2NpdHkvdXNhLyl7dGFyZ2V0PSJfYmxhbmsifQoKSXQgcmVwb3J0cyBwYXJ0aWN1bGF0ZSB2YWx1ZXMgdXNpbmcgd2hhdCBpcyBjYWxsZWQgdGhlIFtBaXIgUXVhbGl0eSBJbmRleF0oaHR0cHM6Ly93d3cuYWlybm93Lmdvdi9pbmRleC5jZm0/YWN0aW9uPWFxaWJhc2ljcy5hcWkpe3RhcmdldD0iX2JsYW5rIn0gKEFRSSkuClRoaXMgW2NhbGN1bGF0b3JdKGh0dHBzOi8vYWlybm93Lmdvdi9pbmRleC5jZm0/YWN0aW9uPWFpcm5vdy5jYWxjdWxhdG9yKXt0YXJnZXQ9Il9ibGFuayJ9IGluZGljYXRlcyB0aGF0IDExNCBBUUkgaXMgZXF1aXZhbGVudCB0byA0MC43IHVnL21eM14gYW5kIGlzIGNvbnNpZGVyZWQgdW5oZWFsdGh5IGZvciBzZW5zaXRpdmUgaW5kaXZpZHVhbHMuClRodXMsIHNvbWUgYXJlYXMgZXhjZWVkIHRoZSBXSE8gYW5udWFsIGV4cG9zdXJlIGd1aWRlbGluZSAoMTAgdWcvbV4zXiksIGFuZCB0aGlzIG1heSBhZHZlcnNlbHkgYWZmZWN0IHRoZSBoZWFsdGggb2YgcGVvcGxlIGxpdmluZyBpbiB0aGVzZSBsb2NhdGlvbnMuCgpBZHZlcnNlIGhlYWx0aCBlZmZlY3RzIGhhdmUgYmVlbiBhc3NvY2lhdGVkIHdpdGggcG9wdWxhdGlvbnMgZXhwZXJpZW5jaW5nIGhpZ2hlciBwb2xsdXRpb24gZXhwb3N1cmUgZGVzcGl0ZSB0aGUgbGV2ZWxzIGJlaW5nIGJlbG93IHN1Z2dlc3RlZCBndWlkZWxpbmVzLiAKQWxzbywgaXQgYXBwZWFycyB0aGF0IHRoZSBjb21wb3NpdGlvbiBvZiB0aGUgcGFydGljdWxhdGUgbWF0dGVyIGFuZCB0aGUgaW5mbHVlbmNlIG9mIG90aGVyIGRlbW9ncmFwaGljIGZhY3RvcnMgbWF5IG1ha2Ugc3BlY2lmaWMgcG9wdWxhdGlvbnMgbW9yZSBhdCByaXNrIGZvciBhZHZlcnNlIGhlYWx0aCBlZmZlY3RzIGR1ZSB0byBhaXIgcG9sbHV0aW9uLiAKRm9yIGV4YW1wbGUsIHNlZSB0aGlzIFthcnRpY2xlXShodHRwczovL3d3dy5uZWptLm9yZy9kb2kvZnVsbC8xMC4xMDU2L05FSk1vYTE3MDI3NDcpe3RhcmdldD0iX2JsYW5rIn0gZm9yIG1vcmUgZGV0YWlscy4KClRoZSBtb25pdG9yIGRhdGEgdGhhdCB3ZSB3aWxsIHVzZSBpbiB0aGlzIGNhc2Ugc3R1ZHkgY29tZSBmcm9tIGEgc3lzdGVtIG9mIG1vbml0b3JzIGluIHdoaWNoIHJvdWdobHkgOTAlIGFyZSBsb2NhdGVkIHdpdGhpbiBjaXRpZXMuIApIZW5jZSwgdGhlcmUgaXMgYW4gKiplcXVpdHkgaXNzdWUqKiBpbiB0ZXJtcyBvZiBjYXB0dXJpbmcgdGhlIGFpciBwb2xsdXRpb24gbGV2ZWxzIG9mIG1vcmUgcnVyYWwgYXJlYXMuIApUbyBnZXQgYSBiZXR0ZXIgc2Vuc2Ugb2YgdGhlIHBvbGx1dGlvbiBleHBvc3VyZXMgZm9yIHRoZSBpbmRpdmlkdWFscyBsaXZpbmcgaW4gdGhlc2UgYXJlYXMsIG1ldGhvZHMgbGlrZSBtYWNoaW5lIGxlYXJuaW5nIGNhbiBiZSB1c2VmdWwgdG8gZXN0aW1hdGUgYWlyIHBvbGx1dGlvbiBsZXZlbHMgaW4gKiphcmVhcyB3aXRoIGxpdHRsZSB0byBubyBtb25pdG9yaW5nKiouIApTcGVjaWZpY2FsbHksIHRoZXNlIG1ldGhvZHMgY2FuIGJlIHVzZWQgdG8gZXN0aW1hdGUgYWlyIHBvbGx1dGlvbiBpbiB0aGVzZSBsb3cgbW9uaXRvcmluZyBhcmVhcyBzbyB0aGF0IHdlIGNhbiBtYWtlIGEgbWFwIGxpa2UgdGhpcyB3aGVyZSB3ZSBoYXZlIGFubnVhbCBlc3RpbWF0ZXMgZm9yIGFsbCBvZiB0aGUgY29udGlndW91cyBVUzoKCjxwIGFsaWduPSJjZW50ZXIiPgogIDxpbWcgd2lkdGg9IjYwMCIgc3JjPSJodHRwczovL2FyYy1hbmdsZXJmaXNoLXdhc2hwb3N0LXByb2Qtd2FzaHBvc3QuczMuYW1hem9uYXdzLmNvbS9wdWJsaWMvU0FXT0VHQlhNVkdRN0FTNVBaNlVVT1g2RlkucG5nIj4KPC9wPgoKIyMjIyMgW1tzb3VyY2VdXShodHRwczovL3d3dy5nb29nbGUuY29tL3VybD9zYT1pJnVybD1odHRwcyUzQSUyRiUyRnd3dy53YXNoaW5ndG9ucG9zdC5jb20lMkZidXNpbmVzcyUyRjIwMTklMkYxMCUyRjIzJTJGYWlyLXBvbGx1dGlvbi1pcy1nZXR0aW5nLXdvcnNlLWRhdGEtc2hvdy1tb3JlLXBlb3BsZS1hcmUtZHlpbmclMkYmcHNpZz1BT3ZWYXczdi1aRFRCUG5MUDJNWXRLZjNVbmRqJnVzdD0xNTg1Nzg0NDc5MDY4MDAwJnNvdXJjZT1pbWFnZXMmY2Q9dmZlJnZlZD0wQ0FJUWpSeHFGd29UQ1BDeW45Znh4ZWdDRlFBQUFBQWRBQUFBQUJBZCl7dGFyZ2V0PSJfYmxhbmsifQoKVGhpcyBpcyB3aGF0IHdlIGFpbSB0byBhY2hpZXZlIGluIHRoaXMgY2FzZSBzdHVkeS4KCiMgKipMaW1pdGF0aW9ucyoqCioqKgoKVGhlcmUgYXJlIHNvbWUgaW1wb3J0YW50IGNvbnNpZGVyYXRpb25zIHJlZ2FyZGluZyB0aGUgZGF0YSBhbmFseXNpcyBpbiB0aGlzIGNhc2Ugc3R1ZHkgdG8ga2VlcCBpbiBtaW5kOiAKCjEuIFRoZSBkYXRhIGRvIG5vdCBpbmNsdWRlIGluZm9ybWF0aW9uIGFib3V0IHRoZSBjb21wb3NpdGlvbiBvZiBwYXJ0aWN1bGF0ZSBtYXR0ZXIuIERpZmZlcmVudCB0eXBlcyBvZiBwYXJ0aWN1bGF0ZXMgbWF5IGJlIG1vcmUgYmVuaWduIG9yIGRlbGV0ZXJpb3VzIGZvciBoZWFsdGggb3V0Y29tZXMuCgoyLiBPdXRkb29yIHBvbGx1dGlvbiBsZXZlbHMgYXJlIG5vdCBuZWNlc3NhcmlseSBhbiBpbmRpY2F0aW9uIG9mIGluZGl2aWR1YWwgZXhwb3N1cmVzLiBQZW9wbGUgc3BlbmQgZGlmZmVyaW5nIGFtb3VudHMgb2YgdGltZSBpbmRvb3JzIGFuZCBvdXRkb29ycyBhbmQgYXJlIGV4cG9zZWQgdG8gZGlmZmVyZW50IHBvbGx1dGlvbiBsZXZlbHMgaW5kb29ycy4gUmVzZWFyY2hlcnMgYXJlIG5vdyBkZXZlbG9waW5nIHBlcnNvbmFsIG1vbml0b3Jpbmcgc3lzdGVtcyB0byB0cmFjayBhaXIgcG9sbHV0aW9uIGxldmVscyBvbiB0aGUgcGVyc29uYWwgbGV2ZWwuCgozLiBPdXIgYW5hbHlzaXMgd2lsbCB1c2UgYW5udWFsIG1lYW4gZXN0aW1hdGVzIG9mIHBvbGx1dGlvbiBsZXZlbHMsIGJ1dCB0aGVzZSBjYW4gdmFyeSBncmVhdGx5IGJ5IHNlYXNvbiwgZGF5IGFuZCBldmVuIGhvdXIuIFRoZXJlIGFyZSBkYXRhIHNvdXJjZXMgdGhhdCBoYXZlIGZpbmVyIGxldmVscyBvZiB0ZW1wb3JhbCBkYXRhOyBob3dldmVyLCB3ZSBhcmUgaW50ZXJlc3RlZCBpbiBsb25nIHRlcm0gZXhwb3N1cmVzLCBhcyB0aGVzZSBhcHBlYXIgdG8gYmUgdGhlIG1vc3QgaW5mbHVlbnRpYWwgZm9yIGhlYWx0aCBvdXRjb21lcy4KCgojICoqV2hhdCBhcmUgdGhlIGRhdGE/KiogeyN3aGF0YXJldGhlZGF0YX0KKioqCgpXZSBhcmUgZ29pbmcgdG8gcGVyZm9ybSBhIHR5cGUgb2YgbWFjaGluZSBsZWFybmluZyB0byB0cnkgdG8gcHJlZGljdCBhaXIgcG9sbHV0aW9uIGxldmVscyB0aGF0IGlzIGNhbGxlZCBzdXBlcnZpc2VkIG1hY2hpbmUgbGVhcm5pbmcuIFRoaXMgdHlwZSBvZiBtYWNoaW5lIGxlYXJuaW5nIHJlcXVpcmVzIHRoYXQgd2UgaGF2ZSBhbiBvdXRjb21lIG9yIHJlc3VsdCBvZiByZWFsIHZhbHVlcyAob2YgaW4gb3VyIGNhc2UgYWlyIHBvbGx1dGlvbikgdG8gd29yayB3aXRoIHRoYXQgd2lsbCBndWlkZSBvciBzdXBlcnZpc2Ugb3VyIHdvcmsuIFRoaXMgd2lsbCB1bHRpbWF0ZWx5IGFsbG93IHVzIHRvIHRyeSB0byBwcmVkaWN0IGFpciBwb2xsdXRpb24gdmFsdWVzIGluIHRoZSBmdXR1cmUgd2hlbiB3ZSBkb24ndCBoYXZlIHRoZW0uIApGb3IgbW9yZSBleHBsYW5hdGlvbiBvZiB3aGF0IHN1cGVydmlzZWQgbWFjaGluZSBsZWFybmluZyBpcyAoYW5kIGhvdyBpdCBjb21wYXJlcyB0byBhIGRpZmZlcmVudCBraW5kIG9mIG1hY2hpbmUgbGVhcm5pbmcgY2FsbGVkIHVuc3VwZXJ2aXNlZCBtYWNoaW5lIGxlYXJuaW5nIHdoZXJlIHdlIGRvbid0IGhhdmUgb3V0Y29tZSB2YWx1ZXMpIHJlZmVyIHRvIFt0aGlzXShodHRwczovL3Rvd2FyZHNkYXRhc2NpZW5jZS5jb20vc3VwZXJ2aXNlZC12cy11bnN1cGVydmlzZWQtbGVhcm5pbmctMTRmNjhlMzJlYThkKXt0YXJnZXQ9Il9ibGFuayJ9KSwgCiAKV2hlbiB1c2luZyBzdXBlcnZpc2VkIG1hY2hpbmUgbGVhcm5pbmcgZm9yIHByZWRpY3Rpb24sdGhlcmUgYXJlIHR3byBtYWluIHR5cGVzIG9mIGRhdGEgb2YgaW50ZXJlc3Q6CgoxLiBBICoqY29udGludW91cyoqIG91dGNvbWUgdmFyaWFibGUgdGhhdCB3ZSB3YW50IHRvIHByZWRpY3QgCjIuIEEgc2V0IG9mIGZlYXR1cmUocykgKG9yIHByZWRpY3RvciB2YXJpYWJsZXMpIHRoYXQgd2UgdXNlIHRvIHByZWRpY3QgdGhlIG91dGNvbWUgdmFyaWFibGUKClRoZSAqKm91dGNvbWUgdmFyaWFibGUqKiBpcyB3aGF0IHdlIGFyZSB0cnlpbmcgdG8gKipwcmVkaWN0KiouIApUbyBidWlsZCAob3IgdHJhaW4pIG91ciBtb2RlbCwgd2UgdXNlIGJvdGggdGhlIG91dGNvbWUgYW5kIGZlYXR1cmVzLgpUaGUgZ29hbCBpcyB0byBpZGVudGlmeSBpbmZvcm1hdGl2ZSBmZWF0dXJlcyB0aGF0IGNhbiBleHBsYWluIGEgbGFyZ2UgYW1vdW50IG9mIHZhcmlhdGlvbiBpbiBvdXIgb3V0Y29tZSB2YXJpYWJsZS4gClVzaW5nIHRoaXMgbW9kZWwsIHdlIGNhbiB0aGVuIHByZWRpY3QgdGhlIG91dGNvbWUgZnJvbSBuZXcgb2JzZXJ2YXRpb25zIHdpdGggdGhlIHNhbWUgZmVhdHVyZXMgd2hlcmUgaGF2ZSBub3Qgb2JzZXJ2ZWQgdGhlIG91dGNvbWUuIAoKQXMgYSBzaW1wbGUgZXhhbXBsZSwgaW1hZ2luZSB0aGF0IHdlIGhhdmUgZGF0YSBhYm91dCB0aGUgc2FsZXMgYW5kIGNoYXJhY3RlcmlzdGljcyBvZiBjYXJzIGZyb20gbGFzdCB5ZWFyIGFuZCB3ZSB3YW50IHRvIHByZWRpY3Qgd2hpY2ggY2FycyBtaWdodCBzZWxsIHdlbGwgdGhpcyB5ZWFyLiAKV2UgZG8gbm90IGhhdmUgdGhlIHNhbGVzIGRhdGEgeWV0IGZvciB0aGlzIHllYXIsIGJ1dCB3ZSBkbyBrbm93IHRoZSBjaGFyYWN0ZXJpc3RpY3Mgb2Ygb3VyIGNhcnMgZm9yIHRoaXMgeWVhci4gCldlIGNhbiBidWlsZCBhIG1vZGVsIG9mIHRoZSBjaGFyYWN0ZXJpc3RpY3MgdGhhdCBleHBsYWluZWQgc2FsZXMgbGFzdCB5ZWFyIHRvIGVzdGltYXRlIHdoYXQgY2FycyBtaWdodCBzZWxsIHdlbGwgdGhpcyB5ZWFyLiAKSW4gdGhpcyBjYXNlLCBvdXIgb3V0Y29tZSB2YXJpYWJsZSBpcyB0aGUgc2FsZXMgb2YgY2Fycywgd2hpbGUgdGhlIGRpZmZlcmVudCBjaGFyYWN0ZXJpc3RpY3Mgb2YgdGhlIGNhcnMgbWFrZSB1cCBvdXIgZmVhdHVyZXMuCgojIyMgKipTdGFydCB3aXRoIGEgcXVlc3Rpb24qKgoqKioKClRoaXMgaXMgdGhlIG1vc3QgY29tbW9ubHkgbWlzc2VkIHN0ZXAgd2hlbiBkZXZlbG9waW5nIGEgbWFjaGluZSBsZWFybmluZyBhbGdvcml0aG0uIApNYWNoaW5lIGxlYXJuaW5nIGNhbiB2ZXJ5IGVhc2lseSBiZSB0dXJuZWQgaW50byBhbiBlbmdpbmVlcmluZyBwcm9ibGVtLiAKSnVzdCBkdW1wIHRoZSBvdXRjb21lIGFuZCB0aGUgZmVhdHVyZXMgaW50byBhIFtibGFjayBib3ggYWxnb3JpdGhtXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9CbGFja19ib3gpIGFuZCB2aW9sYSEgCkJ1dCB0aGlzIGtpbmQgb2YgdGhpbmtpbmcgY2FuIGxlYWQgdG8gbWFqb3IgcHJvYmxlbXMuIEluIGdlbmVyYWwgZ29vZCBtYWNoaW5lIGxlYXJuaW5nIHF1ZXN0aW9uczoKCjEuIEhhdmUgYSBwbGF1c2libGUgZXhwbGFuYXRpb24gZm9yIHdoeSB0aGUgZmVhdHVyZXMgcHJlZGljdCB0aGUgb3V0Y29tZS4gCjIuIENvbnNpZGVyIHBvdGVudGlhbCB2YXJpYXRpb24gaW4gYm90aCB0aGUgZmVhdHVyZXMgYW5kIHRoZSBvdXRjb21lIG92ZXIgdGltZS4KMy4gQXJlIGNvbnNpc3RlbnRseSByZS1ldmFsdWF0ZWQgb24gY3JpdGVyaWEgMSBhbmQgMiBvdmVyIHRpbWUuIAoKSW4gdGhpcyBjYXNlIHN0dWR5LCB3ZSB3YW50IHRvICoqcHJlZGljdCoqIGFpciBwb2xsdXRpb24gbGV2ZWxzLiAKVG8gYnVpbGQgdGhpcyBtYWNoaW5lIGxlYXJuaW5nIGFsZ29yaXRobSwgb3VyICoqb3V0Y29tZSB2YXJpYWJsZSoqIGlzIGF2ZXJhZ2UgYW5udWFsIGZpbmUgcGFydGljdWxhdGUgbWF0dGVyIChQTX4yLjV+KSBjYXB0dXJlZCBmcm9tIGFpciBwb2xsdXRpb24gbW9uaXRvcnMgaW4gdGhlIGNvbnRpZ3VvdXMgVVMgaW4gMjAwOC4gCk91ciAqKmZlYXR1cmVzKiogKG9yIHByZWRpY3RvciB2YXJpYWJsZXMpIGluY2x1ZGUgZGF0YSBhYm91dCBwb3B1bGF0aW9uIGRlbnNpdHksIHJvYWQgZGVuc2l0eSwgdXJiYW5pemF0aW9uIGxldmVscywgYW5kIE5BU0Egc2F0ZWxsaXRlIGRhdGEuIAoKCgojIyMgKipPdXIgb3V0Y29tZSB2YXJpYWJsZSoqCioqKgoKVGhlIG1vbml0b3IgZGF0YSB0aGF0IHdlIHdpbGwgYmUgdXNpbmcgY29tZXMgZnJvbSAqKltncmF2aW1ldHJpYyBtb25pdG9yc10oaHR0cHM6Ly9wdWJsaWNsYWIub3JnL3dpa2kvZmlsdGVyLXBtKXt0YXJnZXQ9Il9ibGFuayJ9KiogKHNlZSBwaWN0dXJlIGJlbG93KSBvcGVyYXRlZCBieSB0aGUgVVMgW0Vudmlyb25tZW50YWwgUHJvdGVjdGlvbiBBZ2VuY3kgKEVQQSldKGh0dHBzOi8vd3d3LmVwYS5nb3YvKXt0YXJnZXQ9Il9ibGFuayJ9LgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoPSIxMDBweCJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsIm1vbml0b3IucG5nIikpCmBgYAoKIyMjIyMgW2ltYWdlIGN1cnRlc3kgb2YgW0tpcnN0ZW4gS29laGxlcl0oaHR0cHM6Ly93d3cuamhzcGguZWR1L2ZhY3VsdHkvZGlyZWN0b3J5L3Byb2ZpbGUvMjkyOC9raXJzdGVuLWtvZWhsZXIpXQoKVGhlc2UgbW9uaXRvcnMgdXNlIGEgZmlsdHJhdGlvbiBzeXN0ZW0gdG8gc3BlY2lmaWNhbGx5IGNhcHR1cmUgZmluZSBwYXJ0aWN1bGF0ZSBtYXR0ZXIuIAoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoPSIxNTBweCJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsImZpbHRlci5wbmciKSkKYGBgCgojIyMjIyBbW3NvdXJjZV1dKGh0dHBzOi8vcHVibGljbGFiLm9yZy93aWtpL2ZpbHRlci1wbSl7dGFyZ2V0PSJfYmxhbmsifQoKVGhlIHdlaWdodCBvZiB0aGlzIHBhcnRpY3VsYXRlIG1hdHRlciBpcyBtYW51YWxseSBtZWFzdXJlZCBkYWlseSBvciB3ZWVrbHkuIApGb3IgdGhlIEVQQSBzdGFuZGFyZCBvcGVyYXRpbmcgcHJvY2VkdXJlIGZvciBQTSBncmF2aW1ldHJpYyBhbmFseXNpcyBpbiAyMDA4LCB3ZSByZWZlciB0aGUgcmVhZGVyIHRvIFtoZXJlXShodHRwczovL3d3dzMuZXBhLmdvdi90dG5hbXRpMS9maWxlcy9hbWJpZW50L3BtMjUvc3BlYy9SVElHcmF2TWFzc1NPUEZJTkFMLnBkZil7dGFyZ2V0PSJfYmxhbmsifS4KCiMjIyMgey5jbGlja190b19leHBhbmRfYmxvY2t9Cgo8ZGV0YWlscz48c3VtbWFyeT5Gb3IgbW9yZSBvbiBHcmF2aW1ldHJpYyBhbmFseXNpcywgeW91IGNhbiBleHBhbmQgaGVyZSA8L3N1bW1hcnk+CgpHcmF2aW1ldHJpYyBhbmFseXNpcyBpcyBhbHNvIHVzZWQgZm9yIFtlbWlzc2lvbiB0ZXN0aW5nXShodHRwczovL3d3dy5tdC5jb20vdXMvZW4vaG9tZS9hcHBsaWNhdGlvbnMvTGFib3JhdG9yeV93ZWlnaGluZy9lbWlzc2lvbnMtdGVzdGluZy1wYXJ0aWN1bGF0ZS1tYXR0ZXIuaHRtbCl7dGFyZ2V0PSJfYmxhbmsifS4gClRoZSBzYW1lIGlkZWEgYXBwbGllczogYSBmcmVzaCBmaWx0ZXIgaXMgYXBwbGllZCBhbmQgdGhlIGRlc2lyZWQgYW1vdW50IG9mIHRpbWUgcGFzc2VzLCB0aGVuIHRoZSBmaWx0ZXIgaXMgcmVtb3ZlZCBhbmQgd2VpZ2hlZC4gCgpUaGVyZSBhcmUgW290aGVyIG1vbml0b3Jpbmcgc3lzdGVtc10oaHR0cHM6Ly93d3cuc2Vuc2lyaW9uLmNvbS9lbi9hYm91dC11cy9uZXdzcm9vbS9zZW5zaXJpb24tc3BlY2lhbGlzdC1hcnRpY2xlcy9wYXJ0aWN1bGF0ZS1tYXR0ZXItc2Vuc2luZy1mb3ItYWlyLXF1YWxpdHktbWVhc3VyZW1lbnRzLyl7dGFyZ2V0PSJfYmxhbmsifSB0aGF0IGNhbiBwcm92aWRlIGhvdXJseSBtZWFzdXJlbWVudHMsIGJ1dCB3ZSB3aWxsIG5vdCBiZSB1c2luZyBkYXRhIGZyb20gdGhlc2UgbW9uaXRvcnMgaW4gb3VyIGFuYWx5c2lzLiAKR3JhdmltZXRyaWMgYW5hbHlzaXMgaXMgY29uc2lkZXJlZCB0byBiZSBhbW9uZyB0aGUgbW9zdCBhY2N1cmF0ZSBtZXRob2RzIGZvciBtZWFzdXJpbmcgcGFydGljdWxhdGUgbWF0dGVyLgoKPC9kZXRhaWxzPgoKIyMjIwoKSW4gb3VyIGRhdGEgc2V0LCB0aGUgYHZhbHVlYCBjb2x1bW4gaW5kaWNhdGVzIHRoZSBQTX4yLjV+IG1vbml0b3IgYXZlcmFnZSBmb3IgMjAwOCBpbiBtYXNzIG9mIGZpbmUgcGFydGljbGVzL3ZvbHVtZSBvZiBhaXIgZm9yIDg3NiBncmF2aW1ldHJpYyBtb25pdG9ycy4gClRoZSB1bml0cyBhcmUgbWljcm9ncmFtcyBvZiBmaW5lIHBhcnRpY3VsYXRlIG1hdHRlciAoUE0pIHRoYXQgaXMgbGVzcyB0aGFuIDIuNSBtaWNyb21ldGVycyBpbiBkaWFtZXRlciBwZXIgY3ViaWMgbWV0ZXIgb2YgYWlyIC0gbWFzcyBjb25jZW50cmF0aW9uICh1Zy9tXjNeKS4KUmVjYWxsIHRoZSBXSE8gZXhwb3N1cmUgZ3VpZGVsaW5lIGlzIDwgMTAgdWcvbV4zXiBvbiBhdmVyYWdlIGFubnVhbGx5IGZvciBQTX4yLjV+LgoKIyMjICoqT3VyIGZlYXR1cmVzIChwcmVkaWN0b3IgdmFyaWFibGVzKSoqCioqKgoKVGhlcmUgYXJlIDQ4IGZlYXR1cmVzIHdpdGggdmFsdWVzIGZvciBlYWNoIG9mIHRoZSA4NzYgbW9uaXRvcnMgKG9ic2VydmF0aW9ucykuIApUaGUgZGF0YSBjb21lcyBmcm9tIHRoZSBVUyBbRW52aXJvbm1lbnRhbCBQcm90ZWN0aW9uIEFnZW5jeSAoRVBBKV0oaHR0cHM6Ly93d3cuZXBhLmdvdi8pe3RhcmdldD0iX2JsYW5rIn0sIHRoZSBbTmF0aW9uYWwgQWVyb25hdXRpY3MgYW5kIFNwYWNlIEFkbWluaXN0cmF0aW9uIChOQVNBKV0oaHR0cHM6Ly93d3cubmFzYS5nb3YvKXt0YXJnZXQ9Il9ibGFuayJ9LCB0aGUgVVMgW0NlbnN1c10oaHR0cHM6Ly93d3cuY2Vuc3VzLmdvdi9hYm91dC93aGF0L2NlbnN1cy1hdC1hLWdsYW5jZS5odG1sKXt0YXJnZXQ9Il9ibGFuayJ9LCBhbmQgdGhlIFtOYXRpb25hbCBDZW50ZXIgZm9yIEhlYWx0aCBTdGF0aXN0aWNzIChOQ0hTKV0oaHR0cHM6Ly93d3cuY2RjLmdvdi9uY2hzL2Fib3V0L2luZGV4Lmh0bSl7dGFyZ2V0PSJfYmxhbmsifS4KCiMjIyMgey5jbGlja190b19leHBhbmRfYmxvY2t9Cgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byBzZWUgYSB0YWJsZSBhYm91dCB0aGUgc2V0IG9mIGZlYXR1cmVzIDwvc3VtbWFyeT4KClZhcmlhYmxlICAgfCBEZXRhaWxzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCi0tLS0tLS0tLS0gfC0tLS0tLS0tLS0tLS0KKippZCoqICB8IE1vbml0b3IgbnVtYmVyICA8YnI+IC0tIHRoZSBjb3VudHkgbnVtYmVyIGlzIGluZGljYXRlZCBiZWZvcmUgdGhlIGRlY2ltYWwgPGJyPiAtLSB0aGUgbW9uaXRvciBudW1iZXIgaXMgaW5kaWNhdGVkIGFmdGVyIHRoZSBkZWNpbWFsIDxicj4gICoqRXhhbXBsZSoqOiAxMDczLjAwMjMgIGlzIEplZmZlcnNvbiBjb3VudHkgKDEwNzMpIGFuZCAuMDAyMyBvbmUgb2YgOCBtb25pdG9ycyAKKipmaXBzKiogfCBGZWRlcmFsIGluZm9ybWF0aW9uIHByb2Nlc3Npbmcgc3RhbmRhcmQgbnVtYmVyIGZvciB0aGUgY291bnR5IHdoZXJlIHRoZSBtb25pdG9yIGlzIGxvY2F0ZWQgPGJyPiAtLSA1IGRpZ2l0IGlkIGNvZGUgZm9yIGNvdW50aWVzICh6ZXJvIGlzIG9mdGVuIHRoZSBmaXJzdCB2YWx1ZSBhbmQgc29tZXRpbWVzIGlzIG5vdCBzaG93bikgPGJyPiAtLSB0aGUgZmlyc3QgMiBudW1iZXJzIGluZGljYXRlIHRoZSBzdGF0ZSA8YnI+IC0tIHRoZSBsYXN0IHRocmVlIG51bWJlcnMgaW5kaWNhdGUgdGhlIGNvdW50eSA8YnI+ICAqKkV4YW1wbGUqKjogQWxhYmFtYSdzIHN0YXRlIGNvZGUgaXMgMDEgYmVjYXVzZSBpdCBpcyBmaXJzdCBhbHBoYWJldGljYWxseSA8YnI+IChub3RlOiBBbGFza2EgYW5kIEhhd2FpaSBhcmUgbm90IGluY2x1ZGVkIGJlY2F1c2UgdGhleSBhcmUgbm90IHBhcnQgb2YgdGhlIGNvbnRpZ3VvdXMgVVMpICAKKipMYXQqKiB8IExhdGl0dWRlIG9mIHRoZSBtb25pdG9yIGluIGRlZ3JlZXMgIAoqKkxvbioqIHwgTG9uZ2l0dWRlIG9mIHRoZSBtb25pdG9yIGluIGRlZ3JlZXMgIAoqKnN0YXRlKiogfCBTdGF0ZSB3aGVyZSB0aGUgbW9uaXRvciBpcyBsb2NhdGVkCioqY291bnR5KiogfCBDb3VudHkgd2hlcmUgdGhlIG1vbml0b3IgaXMgbG9jYXRlZAoqKmNpdHkqKiB8IENpdHkgd2hlcmUgdGhlIG1vbml0b3IgaXMgbG9jYXRlZAoqKkNNQVEqKiAgfCBFc3RpbWF0ZWQgdmFsdWVzIG9mIGFpciBwb2xsdXRpb24gZnJvbSBhIGNvbXB1dGF0aW9uYWwgbW9kZWwgY2FsbGVkIFsqKkNvbW11bml0eSBNdWx0aXNjYWxlIEFpciBRdWFsaXR5IChDTUFRKSoqXShodHRwczovL3d3dy5lcGEuZ292L2NtYXEpe3RhcmdldD0iX2JsYW5rIn0gPGJyPiAtLSAgQSBtb25pdG9yaW5nIHN5c3RlbSB0aGF0IHNpbXVsYXRlcyB0aGUgcGh5c2ljcyBvZiB0aGUgYXRtb3NwaGVyZSB1c2luZyBjaGVtaXN0cnkgYW5kIHdlYXRoZXIgZGF0YSB0byBwcmVkaWN0IHRoZSBhaXIgcG9sbHV0aW9uIDxicj4gLS0gKioqRG9lcyBub3QgdXNlIGFueSBvZiB0aGUgUE1+Mi41fiBncmF2aW1ldHJpYyBtb25pdG9yaW5nIGRhdGEuKioqIChUaGVyZSBpcyBhIHZlcnNpb24gdGhhdCBkb2VzIHVzZSB0aGUgZ3JhdmltZXRyaWMgbW9uaXRvcmluZyBkYXRhLCBidXQgbm90IHRoaXMgb25lISkgPGJyPiAtLSBEYXRhIGZyb20gdGhlIEVQQQoqKnpjdGEqKiB8IFtaaXAgQ29kZSBUYWJ1bGF0aW9uIEFyZWFdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL1pJUF9Db2RlX1RhYnVsYXRpb25fQXJlYSl7dGFyZ2V0PSJfYmxhbmsifSB3aGVyZSB0aGUgbW9uaXRvciBpcyBsb2NhdGVkIDxicj4gLS0gUG9zdGFsIFppcCBjb2RlcyBhcmUgY29udmVydGVkIGludG8gImdlbmVyYWxpemVkIGFyZWFsIHJlcHJlc2VudGF0aW9ucyIgdGhhdCBhcmUgbm9uLW92ZXJsYXBwaW5nICA8YnI+IC0tIERhdGEgZnJvbSB0aGUgMjAxMCBDZW5zdXMgIAoqKnpjdGFfYXJlYSoqIHwgTGFuZCBhcmVhIG9mIHRoZSB6aXAgY29kZSBhcmVhIGluIG1ldGVycyBzcXVhcmVkICA8YnI+IC0tIERhdGEgZnJvbSB0aGUgMjAxMCBDZW5zdXMgIAoqKnpjdGFfcG9wKiogfCBQb3B1bGF0aW9uIGluIHRoZSB6aXAgY29kZSBhcmVhICA8YnI+IC0tIERhdGEgZnJvbSB0aGUgMjAxMCBDZW5zdXMgIAoqKmltcF9hNTAwKiogfCBJbXBlcnZpb3VzIHN1cmZhY2UgbWVhc3VyZSA8YnI+IC0tIFdpdGhpbiBhIGNpcmNsZSB3aXRoIGEgcmFkaXVzIG9mIDUwMCBtZXRlcnMgYXJvdW5kIHRoZSBtb25pdG9yIDxicj4gLS0gSW1wZXJ2aW91cyBzdXJmYWNlIGFyZSByb2FkcywgY29uY3JldGUsIHBhcmtpbmcgbG90cywgYnVpbGRpbmdzIDxicj4gLS0gVGhpcyBpcyBhIG1lYXN1cmUgb2YgZGV2ZWxvcG1lbnQgCioqaW1wX2ExMDAwKiogfCBJbXBlcnZpb3VzIHN1cmZhY2UgbWVhc3VyZSA8YnI+IC0tICBXaXRoaW4gYSBjaXJjbGUgd2l0aCBhIHJhZGl1cyBvZiAxMDAwIG1ldGVycyBhcm91bmQgdGhlIG1vbml0b3IKKippbXBfYTUwMDAqKiB8IEltcGVydmlvdXMgc3VyZmFjZSBtZWFzdXJlIDxicj4gLS0gIFdpdGhpbiBhIGNpcmNsZSB3aXRoIGEgcmFkaXVzIG9mIDUwMDAgbWV0ZXJzIGFyb3VuZCB0aGUgbW9uaXRvciAgCioqaW1wX2ExMDAwMCoqIHwgSW1wZXJ2aW91cyBzdXJmYWNlIG1lYXN1cmUgPGJyPiAtLSAgV2l0aGluIGEgY2lyY2xlIHdpdGggYSByYWRpdXMgb2YgMTAwMDAgbWV0ZXJzIGFyb3VuZCB0aGUgbW9uaXRvciAgIAoqKmltcF9hMTUwMDAqKiB8IEltcGVydmlvdXMgc3VyZmFjZSBtZWFzdXJlIDxicj4gLS0gIFdpdGhpbiBhIGNpcmNsZSB3aXRoIGEgcmFkaXVzIG9mIDE1MDAwIG1ldGVycyBhcm91bmQgdGhlIG1vbml0b3IgIAoqKmNvdW50eV9hcmVhKiogfCBMYW5kIGFyZWEgb2YgdGhlIGNvdW50eSBvZiB0aGUgbW9uaXRvciBpbiBtZXRlcnMgc3F1YXJlZCAgCioqY291bnR5X3BvcCoqIHwgUG9wdWxhdGlvbiBvZiB0aGUgY291bnR5IG9mIHRoZSBtb25pdG9yICAKKipMb2dfZGlzdF90b19wcmlzZWMqKiB8IExvZyAoTmF0dXJhbCBsb2cpIGRpc3RhbmNlIHRvIGEgcHJpbWFyeSBvciBzZWNvbmRhcnkgcm9hZCBmcm9tIHRoZSBtb25pdG9yIDxicj4gLS0gSGlnaHdheSBvciBtYWpvciByb2FkICAKKipsb2dfcHJpX2xlbmd0aF81MDAwKiogfCBDb3VudCBvZiBwcmltYXJ5IHJvYWQgbGVuZ3RoIGluIG1ldGVycyBpbiBhIGNpcmNsZSB3aXRoIGEgcmFkaXVzIG9mIDUwMDAgbWV0ZXJzIGFyb3VuZCB0aGUgbW9uaXRvciAoTmF0dXJhbCBsb2cpIDxicj4gLS0gSGlnaHdheXMgb25seSAgCioqbG9nX3ByaV9sZW5ndGhfMTAwMDAqKiB8IENvdW50IG9mIHByaW1hcnkgcm9hZCBsZW5ndGggaW4gbWV0ZXJzIGluIGEgY2lyY2xlIHdpdGggYSByYWRpdXMgb2YgMTAwMDAgbWV0ZXJzIGFyb3VuZCB0aGUgbW9uaXRvciAoTmF0dXJhbCBsb2cpIDxicj4gLS0gSGlnaHdheXMgb25seSAgCioqbG9nX3ByaV9sZW5ndGhfMTUwMDAqKiB8IENvdW50IG9mIHByaW1hcnkgcm9hZCBsZW5ndGggaW4gbWV0ZXJzIGluIGEgY2lyY2xlIHdpdGggYSByYWRpdXMgb2YgMTUwMDAgbWV0ZXJzIGFyb3VuZCB0aGUgbW9uaXRvciAoTmF0dXJhbCBsb2cpIDxicj4gLS0gSGlnaHdheXMgb25seSAgCioqbG9nX3ByaV9sZW5ndGhfMjUwMDAqKiB8IENvdW50IG9mIHByaW1hcnkgcm9hZCBsZW5ndGggaW4gbWV0ZXJzIGluIGEgY2lyY2xlIHdpdGggYSByYWRpdXMgb2YgMjUwMDAgbWV0ZXJzIGFyb3VuZCB0aGUgbW9uaXRvciAoTmF0dXJhbCBsb2cpIDxicj4gLS0gSGlnaHdheXMgb25seSAgCioqbG9nX3ByaXNlY19sZW5ndGhfNTAwKiogfCBDb3VudCBvZiBwcmltYXJ5IGFuZCBzZWNvbmRhcnkgcm9hZCBsZW5ndGggaW4gbWV0ZXJzIGluIGEgY2lyY2xlIHdpdGggYSByYWRpdXMgb2YgNTAwIG1ldGVycyBhcm91bmQgdGhlIG1vbml0b3IgKE5hdHVyYWwgbG9nKSAgPGJyPiAtLSBIaWdod2F5IGFuZCBzZWNvbmRhcnkgcm9hZHMgIAoqKmxvZ19wcmlzZWNfbGVuZ3RoXzEwMDAqKiB8IENvdW50IG9mIHByaW1hcnkgYW5kIHNlY29uZGFyeSByb2FkIGxlbmd0aCBpbiBtZXRlcnMgaW4gYSBjaXJjbGUgd2l0aCBhIHJhZGl1cyBvZiAxMDAwIG1ldGVycyBhcm91bmQgdGhlIG1vbml0b3IgKE5hdHVyYWwgbG9nKSAgPGJyPiAtLSBIaWdod2F5IGFuZCBzZWNvbmRhcnkgcm9hZHMgIAoqKmxvZ19wcmlzZWNfbGVuZ3RoXzUwMDAqKiB8IENvdW50IG9mIHByaW1hcnkgYW5kIHNlY29uZGFyeSByb2FkIGxlbmd0aCBpbiBtZXRlcnMgaW4gYSBjaXJjbGUgd2l0aCBhIHJhZGl1cyBvZiA1MDAwIG1ldGVycyBhcm91bmQgdGhlIG1vbml0b3IgKE5hdHVyYWwgbG9nKSAgPGJyPiAtLSBIaWdod2F5IGFuZCBzZWNvbmRhcnkgcm9hZHMgIAoqKmxvZ19wcmlzZWNfbGVuZ3RoXzEwMDAwKiogfCBDb3VudCBvZiBwcmltYXJ5IGFuZCBzZWNvbmRhcnkgcm9hZCBsZW5ndGggaW4gbWV0ZXJzIGluIGEgY2lyY2xlIHdpdGggYSByYWRpdXMgb2YgMTAwMDAgbWV0ZXJzIGFyb3VuZCB0aGUgbW9uaXRvciAoTmF0dXJhbCBsb2cpICA8YnI+IC0tIEhpZ2h3YXkgYW5kIHNlY29uZGFyeSByb2FkcyAgCioqbG9nX3ByaXNlY19sZW5ndGhfMTUwMDAqKiB8IENvdW50IG9mIHByaW1hcnkgYW5kIHNlY29uZGFyeSByb2FkIGxlbmd0aCBpbiBtZXRlcnMgaW4gYSBjaXJjbGUgd2l0aCBhIHJhZGl1cyBvZiAxNTAwMCBtZXRlcnMgYXJvdW5kIHRoZSBtb25pdG9yIChOYXR1cmFsIGxvZykgIDxicj4gLS0gSGlnaHdheSBhbmQgc2Vjb25kYXJ5IHJvYWRzICAKKipsb2dfcHJpc2VjX2xlbmd0aF8yNTAwMCoqIHwgQ291bnQgb2YgcHJpbWFyeSBhbmQgc2Vjb25kYXJ5IHJvYWQgbGVuZ3RoIGluIG1ldGVycyBpbiBhIGNpcmNsZSB3aXRoIGEgcmFkaXVzIG9mIDI1MDAwIG1ldGVycyBhcm91bmQgdGhlIG1vbml0b3IgKE5hdHVyYWwgbG9nKSAgPGJyPiAtLSBIaWdod2F5IGFuZCBzZWNvbmRhcnkgcm9hZHMgICAgICAKKipsb2dfbmVpXzIwMDhfcG0yNV9zdW1fMTAwMDAqKiB8IFRvbnMgb2YgZW1pc3Npb25zIGZyb20gbWFqb3Igc291cmNlcyBkYXRhIGJhc2UgKGFubnVhbCBkYXRhKSBzdW0gb2YgYWxsIHNvdXJjZXMgd2l0aGluIGEgY2lyY2xlIHdpdGggYSByYWRpdXMgb2YgMTAwMDAgbWV0ZXJzIG9mIGRpc3RhbmNlIGFyb3VuZCB0aGUgbW9uaXRvciAoTmF0dXJhbCBsb2cpICAgIAoqKmxvZ19uZWlfMjAwOF9wbTI1X3N1bV8xNTAwMCoqIHwgVG9ucyBvZiBlbWlzc2lvbnMgZnJvbSBtYWpvciBzb3VyY2VzIGRhdGEgYmFzZSAoYW5udWFsIGRhdGEpIHN1bSBvZiBhbGwgc291cmNlcyB3aXRoaW4gYSBjaXJjbGUgd2l0aCBhIHJhZGl1cyBvZiAxNTAwMCBtZXRlcnMgb2YgZGlzdGFuY2UgYXJvdW5kIHRoZSBtb25pdG9yIChOYXR1cmFsIGxvZykgICAgIAoqKmxvZ19uZWlfMjAwOF9wbTI1X3N1bV8yNTAwMCoqIHwgVG9ucyBvZiBlbWlzc2lvbnMgZnJvbSBtYWpvciBzb3VyY2VzIGRhdGEgYmFzZSAoYW5udWFsIGRhdGEpIHN1bSBvZiBhbGwgc291cmNlcyB3aXRoaW4gYSBjaXJjbGUgd2l0aCBhIHJhZGl1cyBvZiAyNTAwMCBtZXRlcnMgb2YgZGlzdGFuY2UgYXJvdW5kIHRoZSBtb25pdG9yIChOYXR1cmFsIGxvZykgICAgIAoqKmxvZ19uZWlfMjAwOF9wbTEwX3N1bV8xMDAwMCoqIHwgVG9ucyBvZiBlbWlzc2lvbnMgZnJvbSBtYWpvciBzb3VyY2VzIGRhdGEgYmFzZSAoYW5udWFsIGRhdGEpIHN1bSBvZiBhbGwgc291cmNlcyB3aXRoaW4gYSBjaXJjbGUgd2l0aCBhIHJhZGl1cyBvZiAxMDAwMCBtZXRlcnMgb2YgZGlzdGFuY2UgYXJvdW5kIHRoZSBtb25pdG9yIChOYXR1cmFsIGxvZykgICAgICAKKipsb2dfbmVpXzIwMDhfcG0xMF9zdW1fMTUwMDAqKnwgVG9ucyBvZiBlbWlzc2lvbnMgZnJvbSBtYWpvciBzb3VyY2VzIGRhdGEgYmFzZSAoYW5udWFsIGRhdGEpIHN1bSBvZiBhbGwgc291cmNlcyB3aXRoaW4gYSBjaXJjbGUgd2l0aCBhIHJhZGl1cyBvZiAxNTAwMCBtZXRlcnMgb2YgZGlzdGFuY2UgYXJvdW5kIHRoZSBtb25pdG9yIChOYXR1cmFsIGxvZykgICAgICAKKipsb2dfbmVpXzIwMDhfcG0xMF9zdW1fMjUwMDAqKiB8IFRvbnMgb2YgZW1pc3Npb25zIGZyb20gbWFqb3Igc291cmNlcyBkYXRhIGJhc2UgKGFubnVhbCBkYXRhKSBzdW0gb2YgYWxsIHNvdXJjZXMgd2l0aGluIGEgY2lyY2xlIHdpdGggYSByYWRpdXMgb2YgMjUwMDAgbWV0ZXJzIG9mIGRpc3RhbmNlIGFyb3VuZCB0aGUgbW9uaXRvciAoTmF0dXJhbCBsb2cpICAgICAgCioqcG9wZGVuc19jb3VudHkqKiB8IFBvcHVsYXRpb24gZGVuc2l0eSAobnVtYmVyIG9mIHBlb3BsZSBwZXIga2lsb21ldGVyIHNxdWFyZWQgYXJlYSBvZiB0aGUgY291bnR5KQoqKnBvcGRlbnNfemN0YSoqIHwgUG9wdWxhdGlvbiBkZW5zaXR5IChudW1iZXIgb2YgcGVvcGxlIHBlciBraWxvbWV0ZXIgc3F1YXJlZCBhcmVhIG9mIHpjdGEpCioqbm9ocyoqIHwgUGVyY2VudGFnZSBvZiBwZW9wbGUgaW4gemN0YSBhcmVhIHdoZXJlIHRoZSBtb25pdG9yIGlzIHRoYXQgKipkbyBub3QgaGF2ZSBhIGhpZ2ggc2Nob29sIGRlZ3JlZSoqIDxicj4gLS0gRGF0YSBmcm9tIHRoZSBDZW5zdXMKKipzb21laHMqKiB8IFBlcmNlbnRhZ2Ugb2YgcGVvcGxlIGluIHpjdGEgYXJlYSB3aGVyZSB0aGUgbW9uaXRvciB3aG9zZSBoaWdoZXN0IGZvcm1hbCBlZHVjYXRpb25hbCBhdHRhaW5tZW50IHdhcyAqKnNvbWUgaGlnaCBzY2hvb2wgZWR1Y2F0aW9uKiogPGJyPiAtLSBEYXRhIGZyb20gdGhlIENlbnN1cwoqKmhzKiogfCBQZXJjZW50YWdlIG9mIHBlb3BsZSBpbiB6Y3RhIGFyZWEgd2hlcmUgdGhlIG1vbml0b3Igd2hvc2UgaGlnaGVzdCBmb3JtYWwgZWR1Y2F0aW9uYWwgYXR0YWlubWVudCB3YXMgY29tcGxldGluZyBhICoqaGlnaCBzY2hvb2wgZGVncmVlKiogPGJyPiAtLSBEYXRhIGZyb20gdGhlIENlbnN1cyAgCioqc29tZWNvbGxlZ2UqKiB8IFBlcmNlbnRhZ2Ugb2YgcGVvcGxlIGluIHpjdGEgYXJlYSB3aGVyZSB0aGUgbW9uaXRvciB3aG9zZSBoaWdoZXN0IGZvcm1hbCBlZHVjYXRpb25hbCBhdHRhaW5tZW50IHdhcyBjb21wbGV0aW5nICoqc29tZSBjb2xsZWdlIGVkdWNhdGlvbioqIDxicj4gLS0gRGF0YSBmcm9tIHRoZSBDZW5zdXMgCioqYXNzb2NpYXRlKiogfCBQZXJjZW50YWdlIG9mIHBlb3BsZSBpbiB6Y3RhIGFyZWEgd2hlcmUgdGhlIG1vbml0b3Igd2hvc2UgaGlnaGVzdCBmb3JtYWwgZWR1Y2F0aW9uYWwgYXR0YWlubWVudCB3YXMgY29tcGxldGluZyBhbiAqKmFzc29jaWF0ZSBkZWdyZWUqKiA8YnI+IC0tIERhdGEgZnJvbSB0aGUgQ2Vuc3VzIAoqKmJhY2hlbG9yKiogfCBQZXJjZW50YWdlIG9mIHBlb3BsZSBpbiB6Y3RhIGFyZWEgd2hlcmUgdGhlIG1vbml0b3Igd2hvc2UgaGlnaGVzdCBmb3JtYWwgZWR1Y2F0aW9uYWwgYXR0YWlubWVudCB3YXMgYSAqKmJhY2hlbG9yJ3MgZGVncmVlKiogPGJyPiAtLSBEYXRhIGZyb20gdGhlIENlbnN1cyAKKipncmFkKiogfCBQZXJjZW50YWdlIG9mIHBlb3BsZSBpbiB6Y3RhIGFyZWEgd2hlcmUgdGhlIG1vbml0b3Igd2hvc2UgaGlnaGVzdCBmb3JtYWwgZWR1Y2F0aW9uYWwgYXR0YWlubWVudCB3YXMgYSAqKmdyYWR1YXRlIGRlZ3JlZSoqIDxicj4gLS0gRGF0YSBmcm9tIHRoZSBDZW5zdXMgCioqcG92KiogfCBQZXJjZW50YWdlIG9mIHBlb3BsZSBpbiB6Y3RhIGFyZWEgd2hlcmUgdGhlIG1vbml0b3IgaXMgdGhhdCBsaXZlZCBpbiBbKipwb3ZlcnR5KipdKGh0dHBzOi8vYXNwZS5oaHMuZ292LzIwMDgtaGhzLXBvdmVydHktZ3VpZGVsaW5lcykgaW4gMjAwOCAtIG9yIHdvdWxkIGl0IGhhdmUgYmVlbiAyMDA3IGd1aWRlbGluZXM/P2h0dHBzOi8vYXNwZS5oaHMuZ292LzIwMDctaGhzLXBvdmVydHktZ3VpZGVsaW5lcyA8YnI+IC0tIERhdGEgZnJvbSB0aGUgQ2Vuc3VzICAKKipoc19vcmxlc3MqKiB8ICBQZXJjZW50YWdlIG9mIHBlb3BsZSBpbiB6Y3RhIGFyZWEgd2hlcmUgdGhlIG1vbml0b3Igd2hvc2UgaGlnaGVzdCBmb3JtYWwgZWR1Y2F0aW9uYWwgYXR0YWlubWVudCB3YXMgYSAqKmhpZ2ggc2Nob29sIGRlZ3JlZSBvciBsZXNzKiogKHN1bSBvZiBub2hzLCBzb21laHMsIGFuZCBocykgIAoqKnVyYzIwMTMqKiB8IFsyMDEzIFVyYmFuLXJ1cmFsIGNsYXNzaWZpY2F0aW9uXShodHRwczovL3d3dy5jZGMuZ292L25jaHMvZGF0YS9zZXJpZXMvc3JfMDIvc3IwMl8xNjYucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IG9mIHRoZSBjb3VudHkgd2hlcmUgdGhlIG1vbml0b3IgaXMgbG9jYXRlZCA8YnI+IC0tIDYgY2F0ZWdvcnkgdmFyaWFibGUgLSAxIGlzIHRvdGFsbHkgdXJiYW4gNiBpcyBjb21wbGV0ZWx5IHJ1cmFsIDxicj4gIC0tIERhdGEgZnJvbSB0aGUgW05hdGlvbmFsIENlbnRlciBmb3IgSGVhbHRoIFN0YXRpc3RpY3NdKGh0dHBzOi8vd3d3LmNkYy5nb3YvbmNocy9pbmRleC5odG0pe3RhcmdldD0iX2JsYW5rIn0gICAgIAoqKnVyYzIwMDYqKiB8IFsyMDA2IFVyYmFuLXJ1cmFsIGNsYXNzaWZpY2F0aW9uXShodHRwczovL3d3dy5jZGMuZ292L25jaHMvZGF0YS9zZXJpZXMvc3JfMDIvc3IwMl8xNTQucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IG9mIHRoZSBjb3VudHkgd2hlcmUgdGhlIG1vbml0b3IgaXMgbG9jYXRlZCA8YnI+IC0tIDYgY2F0ZWdvcnkgdmFyaWFibGUgLSAxIGlzIHRvdGFsbHkgdXJiYW4gNiBpcyBjb21wbGV0ZWx5IHJ1cmFsIDxicj4gLS0gRGF0YSBmcm9tIHRoZSBbTmF0aW9uYWwgQ2VudGVyIGZvciBIZWFsdGggU3RhdGlzdGljc10oaHR0cHM6Ly93d3cuY2RjLmdvdi9uY2hzL2luZGV4Lmh0bSl7dGFyZ2V0PSJfYmxhbmsifSAgICAgCioqYW9kKiogfCBBZXJvc29sIE9wdGljYWwgRGVwdGggbWVhc3VyZW1lbnQgZnJvbSBhIE5BU0Egc2F0ZWxsaXRlIDxicj4gLS0gYmFzZWQgb24gdGhlIGRpZmZyYWN0aW9uIG9mIGEgbGFzZXIgPGJyPiAtLSB1c2VkIGFzIGEgcHJveHkgb2YgcGFydGljdWxhdGUgcG9sbHV0aW9uIDxicj4gLS0gdW5pdC1sZXNzIC0gaGlnaGVyIHZhbHVlIGluZGljYXRlcyBtb3JlIHBvbGx1dGlvbiA8YnI+IC0tIERhdGEgZnJvbSBOQVNBICAKCjwvZGV0YWlscz4KCiMjIyMKCk1hbnkgb2YgdGhlc2UgZmVhdHVyZXMgaGF2ZSB0byBkbyB3aXRoIHRoZSBjaXJjdWxhciBhcmVhIGFyb3VuZCB0aGUgbW9uaXRvciBjYWxsZWQgdGhlICJidWZmZXIiLiBUaGVzZSBhcmUgaWxsdXN0cmF0ZWQgaW4gdGhlIGZvbGxvd2luZyBmaWd1cmU6CgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiODAwcHgiLH0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwgInJlZ3Jlc3Npb24ucG5nIikpCmBgYAoKIyMjIyMgW1tzb3VyY2VdXShodHRwczovL3d3dy5uY2JpLm5sbS5uaWguZ292L3B1Ym1lZC8xNTI5MjkwNil7dGFyZ2V0PSJfYmxhbmsifQoKCgojICoqRGF0YSBJbXBvcnQqKgoqKioKCkFsbCBvZiBvdXIgZGF0YSB3YXMgcHJldmlvdXNseSBjb2xsZWN0ZWQgYnkgYSBbcmVzZWFyY2hlcl0oaHR0cDovL3d3dy5iaW9zdGF0Lmpoc3BoLmVkdS9+cnBlbmcvKSBhdCB0aGUgW0pvaG5zIEhvcGtpbnMgU2Nob29sIG9mIFB1YmxpYyBIZWFsdGhdKGh0dHBzOi8vd3d3Lmpoc3BoLmVkdS8pIHdobyBzdHVkaWVzIGFpciBwb2xsdXRpb24gYW5kIGNsaW1hdGUgY2hhbmdlLiAgCgpXZSBoYXZlIG9uZSBDU1YgZmlsZSB0aGF0IGNvbnRhaW5zIGJvdGggb3VyIHNpbmdsZSAqKm91dGNvbWUgdmFyaWFibGUqKiBhbmQgYWxsIG9mIG91ciAqKmZlYXR1cmVzKiogKG9yIHByZWRpY3RvciB2YXJpYWJsZXMpLiBZb3UgY2FuIGRvd25sb2FkIHRoaXMgZmlsZSB1c2luZyB0aGUgYE9DU2RhdGFgIHBhY2thZ2U6CgpgYGB7ciwgZXZhbD1GQUxTRX0KIyBpbnN0YWxsLnBhY2thZ2VzKCJPQ1NkYXRhIikKT0NTZGF0YTo6cmF3X2RhdGEoIm9jcy1icC1haXItcG9sbHV0aW9uIiwgb3V0cGF0aCA9IGdldHdkKCkpCmBgYAoKSWYgeW91IGhhdmUgdHJvdWJsZSB1c2luZyB0aGUgcGFja2FnZSwgeW91IGNhbiBhbHNvIGZpbmQgdGhpcyBkYXRhIG9uIG91ciBbR2l0SHViIHJlcG9zaXRvcnldKGh0dHBzOi8vZ2l0aHViLmNvbS9vcGVuY2FzZXN0dWRpZXMvb2NzLWJwLWFpci1wb2xsdXRpb24vdHJlZS9tYXN0ZXIvZGF0YS9yYXcpLiBPciB5b3UgY2FuIGRvd25sb2FkIGl0IG1vcmUgZGlyZWN0bHkgYnkgY2xpY2tpbmcgW2hlcmVdKGh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9vcGVuY2FzZXN0dWRpZXMvb2NzLWJwLWFpci1wb2xsdXRpb24vbWFzdGVyL2RhdGEvcmF3L3BtMjVfZGF0YS5jc3YpLgoKV2UgaGF2ZSBjcmVhdGVkIGEgInJhdyIgc3ViZGlyZWN0b3J5IHdpdGhpbiBhIGRpcmVjdG9yeSBjYWxsZWQgImRhdGEiIG9mIG91ciB3b3JraW5nIGRpcmVjdG9yeSBvZiBvdXIgUlN0dWRpbyBwcm9qZWN0LgoKTmV4dCwgd2UgaW1wb3J0IG91ciBkYXRhIGludG8gUiBub3cgc28gdGhhdCB3ZSBjYW4gZXhwbG9yZSB0aGUgZGF0YSBmdXJ0aGVyLiAKV2Ugd2lsbCBjYWxsIG91ciBkYXRhIG9iamVjdCBgcG1gIGZvciBwYXJ0aWN1bGF0ZSBtYXR0ZXIuIApXZSBpbXBvcnQgdGhlIGRhdGEgdXNpbmcgdGhlIGByZWFkX2NzdigpYCBmdW5jdGlvbiBmcm9tIHRoZSBgcmVhZHJgIHBhY2thZ2UuIAoKV2Ugd2lsbCB1c2UgdGhlIGBoZXJlYCBwYWNrYWdlIHRvIG1ha2UgaXQgZWFzaWVyIHRvIGZpbmQgdGhlIGRhdGEgZmlsZS4KCiMjIyMgey5jbGlja190b19leHBhbmRfYmxvY2t9Cgo8ZGV0YWlscz4gPHN1bW1hcnk+IENsaWNrIGhlcmUgdG8gc2VlIG1vcmUgYWJvdXQgY3JlYXRpbmcgbmV3IHByb2plY3RzIGluIFJTdHVkaW8uIDwvc3VtbWFyeT4KCllvdSBjYW4gY3JlYXRlIGEgcHJvamVjdCBieSBnb2luZyB0byB0aGUgRmlsZSBtZW51IG9mIFJTdHVkaW8gbGlrZSBzbzoKCgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGg9IjYwJSJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsICJOZXdfcHJvamVjdC5wbmciKSkKYGBgCgpZb3UgY2FuIGFsc28gZG8gc28gYnkgY2xpY2tpbmcgdGhlIHByb2plY3QgYnV0dG9uOgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoPSI2MCUifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCAicHJvamVjdF9idXR0b24ucG5nIikpCmBgYAoKU2VlIFtoZXJlXShodHRwczovL3N1cHBvcnQucnN0dWRpby5jb20vaGMvZW4tdXMvYXJ0aWNsZXMvMjAwNTI2MjA3LVVzaW5nLVByb2plY3RzKSB0byBsZWFybiBtb3JlIGFib3V0IHVzaW5nIFJTdHVkaW8gcHJvamVjdHMgYW5kIFtoZXJlXShodHRwczovL2dpdGh1Yi5jb20vamVubnliYy9oZXJlX2hlcmUpIHRvIGxlYXJuIG1vcmUgYWJvdXQgdGhlIGBoZXJlYCBwYWNrYWdlLgoKPC9kZXRhaWxzPgoKIyMjIwoKCmBgYHtyfQpwbSA8LSByZWFkcjo6cmVhZF9jc3YoaGVyZSgiZGF0YSIsInJhdyIsICJwbTI1X2RhdGEuY3N2IikpCmBgYAoKV2Ugd2lsbCBzYXZlIHRoaXMgZGF0YSBhcyBhbiByZGEgZmlsZSBmb3IgbGF0ZXIgaW4gYW4gImltcG9ydGVkIiBzdWJkaXJlY3Rvcnkgb2YgdGhlIGRhdGEgZGlyZWN0b3J5LgoKYGBge3J9CnNhdmUocG0sIGZpbGUgPSBoZXJlOjpoZXJlKCJkYXRhIiwgImltcG9ydGVkIiwgImltcG9ydGVkX3BtLnJkYSIpKQpgYGAKCgojICoqRGF0YSBFeHBsb3JhdGlvbiBhbmQgV3JhbmdsaW5nKioKKioqCklmIHlvdSBhcmUgZm9sbG93aW5nIGFsb25nIGJ1dCBzdG9wcGVkLCB5b3UgY291bGQgc3RhcnQgaGVyZSBieSBmaXJzdCBsb2FkaW5nIHRoZSBkYXRhIGxpa2Ugc286CgpgYGB7cn0KbG9hZChoZXJlOjpoZXJlKCJkYXRhIiwgImltcG9ydGVkIiwgImltcG9ydGVkX3BtLnJkYSIpKQpgYGAKCiMjIyMgey5jbGlja190b19leHBhbmRfYmxvY2t9Cgo8ZGV0YWlscz4gPHN1bW1hcnk+IElmIHlvdSBza2lwcGVkIHRoZSBkYXRhIGltcG9ydCBzZWN0aW9uIGNsaWNrIGhlcmUuIDwvc3VtbWFyeT4KCkZpcnN0IHlvdSBuZWVkIHRvIGluc3RhbGwgdGhlIGBPQ1NkYXRhYCBwYWNrYWdlOgoKYGBge3IsIGV2YWw9RkFMU0V9Cmluc3RhbGwucGFja2FnZXMoIk9DU2RhdGEiKQpgYGAKClRoZW4sIHlvdSBtYXkgZG93bmxvYWQgYW5kIGxvYWQgdGhlIGltcG9ydGVkIGRhdGEgYC5yZGFgIGZpbGUgdXNpbmcgdGhlIGZvbGxvd2luZyBjb2RlOgoKYGBge3IsIGV2YWw9RkFMU0V9Ck9DU2RhdGE6OmltcG9ydGVkX2RhdGEoIm9jcy1icC1haXItcG9sbHV0aW9uIiwgb3V0cGF0aCA9IGdldHdkKCkpCmxvYWQoaGVyZTo6aGVyZSgiT0NTZGF0YSIsICJkYXRhIiwgImltcG9ydGVkIiwgImltcG9ydGVkX3BtLnJkYSIpKQpgYGAKCklmIHRoZSBwYWNrYWdlIGRvZXMgbm90IHdvcmsgZm9yIHlvdSwgYW4gUkRBIGZpbGUgKHN0YW5kcyBmb3IgUiBkYXRhKSBvZiB0aGUgZGF0YSBjYW4gYmUgZm91bmQgb24gb3VyIFtHaXRIdWIgcmVwb3NpdG9yeV0oaHR0cHM6Ly9naXRodWIuY29tL29wZW5jYXNlc3R1ZGllcy9vY3MtYnAtYWlyLXBvbGx1dGlvbi90cmVlL21hc3Rlci9kYXRhL2ltcG9ydGVkKS4gT3IgeW91IGNhbiBkb3dubG9hZCBpdCBtb3JlIGRpcmVjdGx5IGJ5IGNsaWNraW5nIFtoZXJlXShodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vb3BlbmNhc2VzdHVkaWVzL29jcy1icC1haXItcG9sbHV0aW9uL21hc3Rlci9kYXRhL2ltcG9ydGVkL2ltcG9ydGVkX3BtLnJkYSkuCgpUbyBsb2FkIHRoZSBkb3dubG9hZGVkIGRhdGEgaW50byB5b3VyIGVudmlyb25tZW50LCB5b3UgbWF5IGRvdWJsZSBjbGljayBvbiB0aGUgYC5yZGFgIGZpbGUgaW4gUnN0dWRpbyBvciB1c2luZyB0aGUgYGxvYWQoKWAgZnVuY3Rpb24uCgpUbyBjb3B5IGFuZCBwYXN0ZSBvdXIgY29kZSBiZWxvdywgcGxhY2UgdGhlIGRvd25sb2FkZWQgZmlsZSBpbiB5b3VyIGN1cnJlbnQgd29ya2luZyBkaXJlY3Rvcnkgd2l0aGluIGEgc3ViZGlyZWN0b3J5IGNhbGxlZCAiaW1wb3J0ZWQiIHdpdGhpbiBhIHN1YmRpcmVjdG9yeSBjYWxsZWQgImRhdGEiLiBXZSB1c2VkIGFuIFJTdHVkaW8gcHJvamVjdCBhbmQgdGhlIFtgaGVyZWAgcGFja2FnZV0oaHR0cHM6Ly9naXRodWIuY29tL2plbm55YmMvaGVyZV9oZXJlKSB0byBuYXZpZ2F0ZSB0byB0aGUgZmlsZSBtb3JlIGVhc2lseS4gCgpgYGB7cn0KbG9hZChoZXJlOjpoZXJlKCJkYXRhIiwgImltcG9ydGVkIiwgImltcG9ydGVkX3BtLnJkYSIpKQpgYGAKCjxociBzdHlsZT0iaGVpZ2h0OjFweDtib3JkZXI6bm9uZTtjb2xvcjojMzMzO2JhY2tncm91bmQtY29sb3I6IzMzMzsiIC8+Cgo8ZGV0YWlscz4gPHN1bW1hcnk+IENsaWNrIGhlcmUgdG8gc2VlIG1vcmUgYWJvdXQgY3JlYXRpbmcgbmV3IHByb2plY3RzIGluIFJTdHVkaW8uIDwvc3VtbWFyeT4KCllvdSBjYW4gY3JlYXRlIGEgcHJvamVjdCBieSBnb2luZyB0byB0aGUgRmlsZSBtZW51IG9mIFJTdHVkaW8gbGlrZSBzbzoKCgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGg9IjYwJSJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsICJOZXdfcHJvamVjdC5wbmciKSkKYGBgCgpZb3UgY2FuIGFsc28gZG8gc28gYnkgY2xpY2tpbmcgdGhlIHByb2plY3QgYnV0dG9uOgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoPSI2MCUifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCAicHJvamVjdF9idXR0b24ucG5nIikpCmBgYAoKU2VlIFtoZXJlXShodHRwczovL3N1cHBvcnQucnN0dWRpby5jb20vaGMvZW4tdXMvYXJ0aWNsZXMvMjAwNTI2MjA3LVVzaW5nLVByb2plY3RzKSB0byBsZWFybiBtb3JlIGFib3V0IHVzaW5nIFJTdHVkaW8gcHJvamVjdHMgYW5kIFtoZXJlXShodHRwczovL2dpdGh1Yi5jb20vamVubnliYy9oZXJlX2hlcmUpIHRvIGxlYXJuIG1vcmUgYWJvdXQgdGhlIGBoZXJlYCBwYWNrYWdlLgoKPC9kZXRhaWxzPgoKPGhyIHN0eWxlPSJoZWlnaHQ6MXB4O2JvcmRlcjpub25lO2NvbG9yOiMzMzM7YmFja2dyb3VuZC1jb2xvcjojMzMzOyIgLz4KCjwvZGV0YWlscz4KCiMjIyMKClRoZSBmaXJzdCBzdGVwIGluIHBlcmZvcm1pbmcgYW55IGRhdGEgYW5hbHlzaXMgaXMgdG8gZXhwbG9yZSB0aGUgZGF0YS4gCgpGb3IgZXhhbXBsZSwgd2UgbWlnaHQgd2FudCB0byBiZXR0ZXIgdW5kZXJzdGFuZCB0aGUgdmFyaWFibGVzIGluY2x1ZGVkIGluIHRoZSBkYXRhLCBhcyB3ZSBtYXkgbGVhcm4gYWJvdXQgaW1wb3J0YW50IGRldGFpbHMgYWJvdXQgdGhlIGRhdGEgdGhhdCB3ZSBzaG91bGQga2VlcCBpbiBtaW5kIGFzIHdlIHRyeSB0byBwcmVkaWN0IG91ciBvdXRjb21lIHZhcmlhYmxlLgoKRmlyc3QsIGxldCdzIGp1c3QgZ2V0IGEgZ2VuZXJhbCBzZW5zZSBvZiBvdXIgZGF0YS4gCldlIGNhbiBkbyB0aGF0IHVzaW5nIHRoZSBgZ2xpbXBzZSgpYCBmdW5jdGlvbiBvZiB0aGUgYGRwbHlyYCBwYWNrYWdlIChpdCBpcyBhbHNvIGluIHRoZSBgdGliYmxlYCBwYWNrYWdlKS4KCldlIHdpbGwgYWxzbyB1c2UgdGhlIGAlPiVgIHBpcGUsIHdoaWNoIGNhbiBiZSB1c2VkIHRvIGRlZmluZSB0aGUgaW5wdXQgZm9yIGxhdGVyIHNlcXVlbnRpYWwgc3RlcHMuIAoKVGhpcyB3aWxsIG1ha2UgbW9yZSBzZW5zZSB3aGVuIHdlIGhhdmUgbXVsdGlwbGUgc2VxdWVudGlhbCBzdGVwcyB1c2luZyB0aGUgc2FtZSBkYXRhIG9iamVjdC4gCgpUbyB1c2UgdGhlIHBpcGUgbm90YXRpb24gd2UgbmVlZCB0byBpbnN0YWxsIGFuZCBsb2FkIGBkcGx5cmAgYXMgd2VsbC4KCkZvciBleGFtcGxlLCBoZXJlIHdlIHN0YXJ0IHdpdGggYHBtYCBkYXRhIG9iamVjdCBhbmQgInBpcGUiIHRoZSBvYmplY3QgaW50byBhcyBpbnB1dCBpbnRvIHRoZSBgZ2xpbXBzZSgpYCBmdW5jdGlvbi4gClRoZSBvdXRwdXQgaXMgYW4gb3ZlcnZpZXcgb2Ygd2hhdCBpcyBpbiB0aGUgYHBtYCBvYmplY3Qgc3VjaCBhcyB0aGUgbnVtYmVyIG9mIHJvd3MgYW5kIGNvbHVtbnMsIGFsbCB0aGUgY29sdW1uIG5hbWVzLCB0aGUgZGF0YSB0eXBlcyBmb3IgZWFjaCBjb2x1bW4gYW5kIHRoZSBmaXJzdCB2aWV3IHZhbHVlcyBpbiBlYWNoIGNvbHVtbi4gClRoZSBvdXRwdXQgYmVsb3cgaXMgc2Nyb2xsYWJsZSBzbyB5b3UgY2FuIHNlZSBldmVyeXRoaW5nIGZyb20gdGhlIGBnbGltcHNlKClgIGZ1bmN0aW9uLiAKCiMjIyMgey5zY3JvbGxhYmxlIH0KCmBgYHtyfQojIFNjcm9sbCB0aHJvdWdoIHRoZSBvdXRwdXQhCnBtICU+JQogIGRwbHlyOjpnbGltcHNlKCkKYGBgCgojIyMjCgpXZSBjYW4gc2VlIHRoYXQgdGhlcmUgYXJlIDg3NiBtb25pdG9ycyAocm93cykgYW5kIHRoYXQgd2UgaGF2ZSA1MCB0b3RhbCB2YXJpYWJsZXMgKGNvbHVtbnMpIC0gb25lIG9mIHdoaWNoIGlzIHRoZSBvdXRjb21lIHZhcmlhYmxlLiBJbiB0aGlzIGNhc2UsIHRoZSBvdXRjb21lIHZhcmlhYmxlIGlzIGNhbGxlZCBgdmFsdWVgLiAKCk5vdGljZSB0aGF0IHNvbWUgb2YgdGhlIHZhcmlhYmxlcyB0aGF0IHdlIHdvdWxkIHRoaW5rIG9mIGFzIGZhY3RvcnMgKG9yIGNhdGVnb3JpY2FsIGRhdGEpIGFyZSBjdXJyZW50bHkgb2YgY2xhc3MgY2hhcmFjdGVyIGFzIGluZGljYXRlZCBieSB0aGUgYDxjaHI+YCBqdXN0IHRvIHRoZSByaWdodCBvZiB0aGUgY29sdW1uIG5hbWVzL3ZhcmlhYmxlIG5hbWVzIGluIHRoZSBgZ2xpbXBzZSgpYCBvdXRwdXQuIFRoaXMgbWVhbnMgdGhhdCB0aGUgdmFyaWFibGUgdmFsdWVzIGFyZSBjaGFyYWN0ZXIgc3RyaW5ncywgc3VjaCBhcyB3b3JkcyBvciBwaHJhc2VzLiAKClRoZSBvdGhlciB2YXJpYWJsZXMgYXJlIG9mIGNsYXNzIGA8ZGJsPmAsIHdoaWNoIHN0YW5kcyBmb3IgZG91YmxlIHByZWNpc2lvbiB3aGljaCBpbmRpY2F0ZXMgdGhhdCB0aGV5IGFyZSBudW1lcmljIGFuZCB0aGF0IHRoZXkgaGF2ZSBkZWNpbWFsIHZhbHVlcy4gSW4gY29udHJhc3QsIG9uZSBjb3VsZCBoYXZlIGludGVnZXIgdmFsdWVzIHdoaWNoIHdvdWxkIG5vdCBhbGxvdyBmb3IgZGVjaW1hbCBudW1iZXJzLiBIZXJlIGlzIGEgW2xpbmtdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0RvdWJsZS1wcmVjaXNpb25fZmxvYXRpbmctcG9pbnRfZm9ybWF0KXt0YXJnZXQ9Il9ibGFuayJ9IGZvciBtb3JlIGluZm9ybWF0aW9uIG9uIGRvdWJsZSBwcmVjaXNpb24gbnVtZXJpYyB2YWx1ZXMuCgpBbm90aGVyIGNvbW1vbiBkYXRhIGNsYXNzIGlzIGZhY3RvciB3aGljaCBpcyBhYmJyZXZpYXRlZCBsaWtlIHRoaXM6IGA8ZmN0PmAuIEEgZmFjdG9yIGlzIHNvbWV0aGluZyB0aGF0IGhhcyB1bmlxdWUgbGV2ZWxzIGJ1dCB0aGVyZSBpcyBubyBhcHByZWNpYWJsZSBvcmRlciB0byB0aGUgbGV2ZWxzLiBGb3IgZXhhbXBsZSB3ZSBjYW4gaGF2ZSBhIG51bWVyaWMgdmFsdWUgdGhhdCBpcyBqdXN0IGFuIGlkIHRoYXQgd2Ugd2FudCB0byBiZSBpbnRlcnByZXRlZCBhcyBqdXN0IGEgdW5pcXVlIGxldmVsIGFuZCBub3QgYXMgdGhlIG51bWJlciB0aGF0IGl0IHdvdWxkIHR5cGljYWxseSBpbmRpY2F0ZS4gVGhpcyB3b3VsZCBiZSB1c2VmdWwgZm9yIHNldmVyYWwgb2Ygb3VyIHZhcmlhYmxlczoKCjEuIHRoZSBtb25pdG9yIElEIChgaWRgKQoyLiB0aGUgRmVkZXJhbCBJbmZvcm1hdGlvbiBQcm9jZXNzaW5nIFN0YW5kYXJkIG51bWJlciBmb3IgdGhlIGNvdW50eSB3aGVyZSB0aGUgbW9uaXRvciB3YXMgbG9jYXRlZCAoYGZpcHNgKQozLiB0aGUgemlwIGNvZGUgdGFidWxhdGlvbiBhcmVhIChgemN0YWApCgpOb25lIG9mIHRoZSB2YWx1ZXMgYWN0dWFsbHkgaGF2ZSBhbnkgcmVhbCBudW1lcmljIG1lYW5pbmcsIHNvIHdlIHdhbnQgdG8gbWFrZSBzdXJlIHRoYXQgUiBkb2VzIG5vdCBpbnRlcnByZXQgdGhlbSBhcyBpZiB0aGV5IGRvLiAKClNvIGxldCdzIGNvbnZlcnQgdGhlc2UgdmFyaWFibGVzIGludG8gZmFjdG9ycy4gCldlIGNhbiBkbyB0aGlzIHVzaW5nIHRoZSBgYWNyb3NzKClgIGZ1bmN0aW9uIG9mIHRoZSBgZHBseXJgIHBhY2thZ2UgYW5kIHRoZSBgYXMuZmFjdG9yKClgIGJhc2UgZnVuY3Rpb24uIApUaGUgYGFjcm9zcygpYCBmdW5jdGlvbiBoYXMgdHdvIG1haW4gYXJndW1lbnRzOiAoaSkgdGhlIGNvbHVtbnMgeW91IHdhbnQgdG8gb3BlcmF0ZSBvbiBhbmQgKGlpKSB0aGUgZnVuY3Rpb24gb3IgbGlzdCBvZiBmdW5jdGlvbnMgdG8gYXBwbHkgdG8gZWFjaCBjb2x1bW4uIAoKSW4gdGhpcyBjYXNlLCB3ZSBhcmUgYWxzbyB1c2luZyB0aGUgYG1hZ3JpdHRyYCBhc3NpZ25tZW50IHBpcGUgb3IgZG91YmxlIHBpcGUgdGhhdCBsb29rcyBsaWtlIHRoaXMgYCU8PiVgIG9mIHRoZSBgbWFncml0dHJgIHBhY2thZ2UuIApUaGlzIGFsbG93cyB1cyB1c2UgdGhlIGBwbWAgZGF0YSBhcyBpbnB1dCwgYnV0IGFsc28gcmVhc3NpZ25zIHRoZSBvdXRwdXQgdG8gdGhlIHNhbWUgZGF0YSBvYmplY3QgbmFtZS4KCiMjIyMgey5zY3JvbGxhYmxlIH0KCmBgYHtyfQojIFNjcm9sbCB0aHJvdWdoIHRoZSBvdXRwdXQhCnBtICU8PiUKICBkcGx5cjo6bXV0YXRlKGFjcm9zcyhjKGlkLCBmaXBzLCB6Y3RhKSwgYXMuZmFjdG9yKSkgCgpnbGltcHNlKHBtKQpgYGAKCiMjIyMKCkdyZWF0ISBOb3cgd2UgY2FuIHNlZSB0aGF0IHRoZXNlIHZhcmlhYmxlcyBhcmUgbm93IGZhY3RvcnMgYXMgaW5kaWNhdGVkIGJ5IGA8ZmN0PmAgYWZ0ZXIgdGhlIHZhcmlhYmxlIG5hbWUuCgoKCiMjIyAqKmBza2ltYCBwYWNrYWdlKioKKioqCgpUaGUgYHNraW0oKWAgZnVuY3Rpb24gb2YgdGhlIGBza2ltcmAgcGFja2FnZSBpcyBhbHNvIHJlYWxseSBoZWxwZnVsIGZvciBnZXR0aW5nIGEgZ2VuZXJhbCBzZW5zZSBvZiB5b3VyIGRhdGEuCkJ5IGRlc2lnbiwgaXQgcHJvdmlkZXMgc3VtbWFyeSBzdGF0aXN0aWNzIGFib3V0IHZhcmlhYmxlcyBpbiB0aGUgZGF0YSBzZXQuIAoKCiMjIyMgey5zY3JvbGxhYmxlIH0KCmBgYHtyfQojIFNjcm9sbCB0aHJvdWdoIHRoZSBvdXRwdXQhCnNraW1yOjpza2ltKHBtKQpgYGAKCiMjIyMKCk5vdGljZSBob3cgdGhlcmUgaXMgYSBjb2x1bW4gY2FsbGVkIGBuX21pc3NpbmdgIGFib3V0IHRoZSBudW1iZXIgb2YgdmFsdWVzIHRoYXQgYXJlIG1pc3NpbmcuIAoKVGhpcyBpcyBhbHNvIGluZGljYXRlZCBieSB0aGUgYGNvbXBsZXRlX3JhdGVgIHZhcmlhYmxlIChvciBtaXNzaW5nL251bWJlciBvZiBvYnNlcnZhdGlvbnMpLiAKCkluIG91ciBkYXRhIHNldCwgaXQgbG9va3MgbGlrZSBvdXIgZGF0YSBkbyBub3QgY29udGFpbiBhbnkgbWlzc2luZyBkYXRhLiAKCkFsc28gbm90aWNlIGhvdyB0aGUgZnVuY3Rpb24gcHJvdmlkZXMgc2VwYXJhdGUgdGFibGVzIG9mIHN1bW1hcnkgc3RhdGlzdGljcyBmb3IgZWFjaCBkYXRhIHR5cGU6IGNoYXJhY3RlciwgZmFjdG9yIGFuZCBudW1lcmljLiAKCk5leHQsIHRoZSBgbl91bmlxdWVgIGNvbHVtbiBzaG93cyB1cyB0aGUgbnVtYmVyIG9mIHVuaXF1ZSB2YWx1ZXMgZm9yIGVhY2ggb2Ygb3VyIGNvbHVtbnMuIApXZSBjYW4gc2VlIHRoYXQgdGhlcmUgYXJlIDQ5IHN0YXRlcyByZXByZXNlbnRlZCBpbiB0aGUgZGF0YS4KCldlIGNhbiBzZWUgdGhhdCBmb3IgbWFueSB2YXJpYWJsZXMgdGhlcmUgYXJlIG1hbnkgbG93IHZhbHVlcyBhcyB0aGUgZGlzdHJpYnV0aW9uIHNob3dzIHR3byBwZWFrcywgb25lIG5lYXIgemVybyBhbmQgYW5vdGhlciB3aXRoIGEgaGlnaGVyIHZhbHVlLiAKClRoaXMgaXMgdHJ1ZSBmb3IgdGhlIGBpbXBgIHZhcmlhYmxlcyAobWVhc3VyZXMgb2YgZGV2ZWxvcG1lbnQpLCB0aGUgYG5laWAgdmFyaWFibGVzIChtZWFzdXJlcyBvZiBlbWlzc2lvbiBzb3VyY2VzKSBhbmQgdGhlIHJvYWQgZGVuc2l0eSB2YXJpYWJsZXMuIAoKV2UgY2FuIGFsc28gc2VlIHRoYXQgdGhlIHJhbmdlIG9mIHNvbWUgb2YgdGhlIHZhcmlhYmxlcyBpcyB2ZXJ5IGxhcmdlLCBpbiBwYXJ0aWN1bGFyIHRoZSBhcmVhIGFuZCBwb3B1bGF0aW9uIHJlbGF0ZWQgdmFyaWFibGVzLgoKCkxldCdzIHRha2UgYSBsb29rIHRvIHNlZSB3aGljaCBzdGF0ZXMgYXJlIGluY2x1ZGVkIHVzaW5nIHRoZSBgZGlzdGluY3QoKWAgZnVuY3Rpb24gb2YgdGhlIGBkcGx5cmAgcGFja2FnZToKCmBgYHtyLCBldmFsID0gRkFMU0V9IApwbSAlPiUgCiAgZHBseXI6OmRpc3RpbmN0KHN0YXRlKSAKYGBgCgoKU2Nyb2xsIHRocm91Z2ggdGhlIG91dHB1dDoKCiMjIyMgey5zY3JvbGxhYmxlIH0KYGBge3IsIGVjaG8gPSBGQUxTRX0KIyBTY3JvbGwgdGhyb3VnaCB0aGUgb3V0cHV0IQpwbSAlPiUgCiAgZGlzdGluY3Qoc3RhdGUpICU+JQojIHRoaXMgYWxsb3dzIHVzIHRvIHNob3cgdGhlIGZ1bGwgb3V0cHV0IGluIHRoZSByZW5kZXJlZCBybWFya2Rvd24KIHByaW50KG4gPSAxZTMpCmBgYAojIyMjCgpJdCBsb29rcyBsaWtlICJEaXN0cmljdCBvZiBDb2x1bWJpYSIgaXMgYmVpbmcgaW5jbHVkZWQgYXMgYSBzdGF0ZS4gCldlIGNhbiBzZWUgdGhhdCBBbGFza2EgYW5kIEhhd2FpaSBhcmUgbm90IGluY2x1ZGVkIGluIHRoZSBkYXRhLgoKTGV0J3MgYWxzbyB0YWtlIGEgbG9vayB0byBzZWUgaG93IG1hbnkgbW9uaXRvcnMgdGhlcmUgYXJlIGluIGEgZmV3IGNpdGllcy4gV2UgY2FuIHVzZSB0aGUgYGZpbHRlcigpYCBmdW5jdGlvbiBvZiB0aGUgYGRwbHlyYCBwYWNrYWdlIHRvIGRvIHNvLiBGb3IgZXhhbXBsZSwgbGV0J3MgbG9vayBhdCBBbGJ1cXVlcnF1ZSwgTmV3IE1leGljby4gCgpgYGB7cn0KcG0gJT4lIGRwbHlyOjpmaWx0ZXIoY2l0eSA9PSAiQWxidXF1ZXJxdWUiKQoKYGBgCgpXZSBjYW4gc2VlIHRoYXQgdGhlcmUgd2VyZSBvbmx5IHR3byBtb25pdG9ycyBpbiB0aGUgY2l0eSBvZiBBbGJ1cXVlcnF1ZSBpbiAyMDA2LiBMZXQncyBjb21wYXJlIHRoaXMgd2l0aCBCYWx0aW1vcmUuCgpgYGB7cn0KcG0gJT4lIGZpbHRlcihjaXR5ID09ICJCYWx0aW1vcmUiKQoKYGBgCgpUaGVyZSB3ZXJlIGluIGNvbnRyYXN0IGZpdmUgbW9uaXRvcnMgZm9yIHRoZSBjaXR5IG9mIEJhbHRpbW9yZSwgZGVzcGl0ZSB0aGUgZmFjdCB0aGF0IGlmIHdlIHRha2UgYSBsb29rIGF0IHRoZSBsYW5kIGFyZWEgYW5kIHBvcHVsYXRpb24gb2YgdGhlIGNvdW50aWVzIGZvciBCYWx0aW1vcmUgYW5kIEFsYnVxdWVycXVlLCB3ZSBjYW4gc2VlIHRoYXQgdGhleSBoYWQgdmVyeSBzaW1pbGFyIGxhbmQgYXJlYSBhbmQgcG9wdWxhdGlvbnMuCgpgYGB7cn0KcG0gJT4lIAogIGZpbHRlcihjaXR5ID09ICJCYWx0aW1vcmUiKSAlPiUgCiAgZHBseXI6OnNlbGVjdChjb3VudHlfYXJlYTpjb3VudHlfcG9wKQpwbSAlPiUgCiAgZmlsdGVyKGNpdHkgPT0gIkFsYnVxdWVycXVlIikgJT4lCiAgc2VsZWN0KGNvdW50eV9hcmVhOmNvdW50eV9wb3ApCgpgYGAKCkluIGZhY3QsIHRoZSBjb3VudHkgY29udGFpbmluZyBBbGJ1ZXJxdWUgaGFkIGEgbGFyZ2VyIHBvcHVsYXRpb24uIFRodXMgdGhlIG1lYXN1cmVtZW50cyBmb3IgQWxidXF1ZXJxdWUgd2VyZSBub3QgYXMgdGhvcm91Z2ggYXMgdGhleSB3ZXJlIGZvciBCYWx0aW1vcmUuCgpUaGlzIG1heSBiZSBkdWUgdG8gdGhlIGZhY3QgdGhhdCB0aGUgbW9uaXRvciB2YWx1ZXMgd2VyZSBsb3dlciBpbiBBbGJ1cXVlcnF1ZS4gSXQgaXMgaW50ZXJlc3RpbmcgdG8gbm90ZSBoZXJlIHRoYXQgdGhlIENNQVEgdmFsdWVzIGFyZSBxdWl0ZSBzaW1pbGFyIGZvciBib3RoIGNpdGllcy4KCgojIyAqKkV2YWx1YXRlIGNvcnJlbGF0aW9uKioKKioqCgpJbiBwcmVkaWN0aW9uIGFuYWx5c2VzLCBpdCBpcyBhbHNvIHVzZWZ1bCB0byBldmFsdWF0ZSBpZiBhbnkgb2YgdGhlIHZhcmlhYmxlcyBhcmUgY29ycmVsYXRlZC4gV2h5IHNob3VsZCB3ZSBjYXJlIGFib3V0IHRoaXM/CgpJZiB3ZSBhcmUgdXNpbmcgYSBsaW5lYXIgcmVncmVzc2lvbiB0byBtb2RlbCBvdXIgZGF0YSwgdGhlbiB3ZSBtaWdodCBydW4gaW50byBhIHByb2JsZW0gY2FsbGVkIG11bHRpY29sbGluZWFyaXR5IHdoaWNoIGNhbiBsZWFkIHVzIHRvIG1pc2ludGVycHJldCB3aGF0IGlzIHJlYWxseSBwcmVkaWN0aXZlIG9mIG91ciBvdXRjb21lIHZhcmlhYmxlLiBUaGlzIHBoZW5vbWVub24gb2NjdXJzIHdoZW4gdGhlIHByZWRpY3RvciB2YXJpYWJsZXMgYWN0dWFsbHkgcHJlZGljdCBvbmUgYW5vdGhlci4gU2VlIFt0aGlzIGNhc2Ugc3R1ZHldKGh0dHBzOi8vb3BlbmNhc2VzdHVkaWVzLmdpdGh1Yi5pby9vY3MtYnAtUlRDLWFuYWx5c2lzLykgZm9yIGEgZGVlcGVyIGV4cGxhbmF0aW9uIGFib3V0IHRoaXMuIAoKQW5vdGhlciByZWFzb24gd2Ugc2hvdWxkIGxvb2sgb3V0IGZvciBjb3JyZWxhdGlvbiBpcyB0aGF0IHdlIGRvbid0IHdhbnQgdG8gaW5jbHVkZSByZWR1bmRhbnQgdmFyaWFibGVzLiBUaGlzIGNhbiBhZGQgdW5uZWNlc3Nhcnkgbm9pc2UgdG8gb3VyIGFsZ29yaXRobSBjYXVzaW5nIGEgcmVkdWN0aW9uIGluIHByZWRpY3Rpb24gYWNjdXJhY3ksIGFuZCBpdCBjYW4gY2F1c2Ugb3VyIGFsZ29yaXRobSB0byBiZSB1bm5lY2Vzc2FyaWx5IHNsb3dlci4gRmluYWxseSwgaXQgY2FuIGFsc28gbWFrZSBpdCBkaWZmaWN1bHQgdG8gaW50ZXJwcmV0IHdoYXQgdmFyaWFibGVzIGFyZSBhY3R1YWxseSBwcmVkaWN0aXZlLgoKTGV0J3MgZmlyc3QgdGFrZSBhIGxvb2sgYXQgYWxsIG9mIG91ciBudW1lcmljIHZhcmlhYmxlcyB3aXRoIHRoZWBjb3JycGxvdGAgcGFja2FnZToKVGhlIGBjb3JycGxvdGAgcGFja2FnZSBpcyBhIGdyZWF0IG9wdGlvbiB0byBsb29rIGF0IGNvcnJlbGF0aW9uIGFtb25nIHBvc3NpYmxlIHByZWRpY3RvcnMsIGFuZCBwYXJ0aWN1bGFybHkgdXNlZnVsIGlmIHdlIGhhdmUgbWFueSBwcmVkaWN0b3JzLiAKCkZpcnN0LCB3ZSBjYWxjdWxhdGUgdGhlIFBlYXJzb24gY29ycmVsYXRpb24gY29lZmZpY2llbnRzIGJldHdlZW4gYWxsIGZlYXR1cmVzIHBhaXJ3aXNlIHVzaW5nIHRoZSBgY29yKClgIGZ1bmN0aW9uIG9mIHRoZSBgc3RhdHNgIHBhY2thZ2UgKHdoaWNoIGlzIGxvYWRlZCBhdXRvbWF0aWNhbGx5KS4gVGhlbiB3ZSB1c2UgdGhlIGBjb3JycGxvdDo6Y29ycnBsb3QoKWAgZnVuY3Rpb24uIFRoZSBgdGwuY2V4ID0gMC41YCBhcmd1bWVudCBjb250cm9scyB0aGUgc2l6ZSBvZiB0aGUgdGV4dCBsYWJlbC4gCgpgYGB7cn0KUE1fY29yIDwtIGNvcihwbSAlPiUgZHBseXI6OnNlbGVjdF9pZihpcy5udW1lcmljKSkKY29ycnBsb3Q6OmNvcnJwbG90KFBNX2NvciwgdGwuY2V4ID0gMC41KQpgYGAKTmljZSEgTm93IHdlIGNhbiBzZWUgd2hpY2ggdmFyaWFibGVzIHNob3cgYSBwb3NpdGl2ZSAoYmx1ZSkgb3IgbmVnYXRpdmUgKHJlZCkgY29ycmVsYXRpb24gdG8gb25lIGFub3RoZXIuIFZhcmlhYmxlcyB0aGF0IHNob3cgYSB2ZXJ5IGxpdHRsZSBjb3JyZWxhdGlvbiB3aXRoIG9uZSBhbm90aGVyIGFwcGVhciBhcyB3aGl0ZSBvciBsaWdodGx5IGNvbG9yZWQuIFdlIGNhbiBzZWUgdGhhdCBlYWNoIHZhcmlhYmxlIGlzIHBlcmZlY3RseSBjb3JyZWxhdGVkIHdpdGggaXRzZWxmLCB3aGljaCBpcyBpdCB3aHkgdGhlcmUgaXMgYSBsaW5lIG9mIGJsdWUgc3F1YXJlcyBkaWFnbmFsbHkgYWNyb3NzIHRoZSBwbG90LgoKV2UgY2FuIGFsc28gcGxvdCB0aGUgYWJzb2x1dGUgdmFsdWUgb2YgdGhlIFBlYXJzb24gY29ycmVsYXRpb24gY29lZmZpY2llbnRzIHVzaW5nIHRoZSBgYWJzKClgIGZ1bmN0aW9uIGZyb20gYmFzZSBSIGFuZCBjaGFuZ2UgdGhlIG9yZGVyIG9mIHRoZSBjb2x1bW5zLiBUaGlzIGNhbiBiZSBoZWxwbGZ1bCBpZiB3ZSBhcmVuJ3QgaW50ZXJlc3RlZCBpbiB0aGUgZGlyZWN0aW9uIG9mIHRoZSBjb3JyZWxhdGlvbiBhbmQganVzdCB3YW50IHRvIHNlZSB3aGljaCB2YXJpYWJsZXMgaGF2ZSBhIHJlbGF0aW9uc2hpcCB3aXRoIG9uZSBhbm90aGVyLgoKYGBge3J9CmNvcnJwbG90KGFicyhQTV9jb3IpLCBvcmRlciA9ICJoY2x1c3QiLCB0bC5jZXggPSAwLjUsIGNsLmxpbSA9IGMoMCwgMSkpCgpgYGAKTmljZSwgdGhpcyBpcyBhIGJpdCBlYXNpZXIgdG8gcmVhZCBub3cgYXMgaXQgaXNuJ3QgYXMgZGlzdHJhY3RpbmcgdG8gbG9vayBhdCB0aGUgZGlmZmVyZW50IGNvbG9ycyAtIHdlIGp1c3Qgd2FudCB0byBmb2N1cyBvbiB0aGUgaW50ZW5zaXR5LiBOb3RpY2UgdGhhdCB0aGUgbGVnZW5kIGhlcmUgbm93IGRvZXNuJ3QgaW5mb3JtIHVzIG9mIG11Y2ggaW4gdGhlIGNhc2Ugb2YgdGhlIHJlZCBiZWNhdXNlIHdlIGhhdmUgcGxvdHRlZCB0aGUgYWJzb2x1dGUgdmFsdWVzIG9mIHRoZSBjb3JyZWxhdGlvbiB2YWx1ZXMgYW5kIHRoZXkgd2lsbCB0aHVzIGFsbCBiZSBwb3NpdGl2ZSBhbmQgYmx1ZS4gVGhlIGRhcmtlciB0aGUgYmx1ZSB0aGUgbW9yZSBjb3JyZWxhdGlvbi4KClRoZXJlIGFyZSBzZXZlcmFsIG9wdGlvbnMgZm9yIG9yZGVyaW5nIHRoZSB2YXJpYWJsZXMuIFNlZSBbaGVyZV0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL2NvcnJwbG90L3ZpZ25ldHRlcy9jb3JycGxvdC1pbnRyby5odG1sKSBmb3IgbW9yZSBvcHRpb25zLiBIZXJlIHdlIHdpbGwgdXNlIHRoZSAiaGNsdXN0IiBvcHRpb24gZm9yIG9yZGVyaW5nIGJ5IFtoaWVyYXJjaGljYWwgY2x1c3RlcmluZ10oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvSGllcmFyY2hpY2FsX2NsdXN0ZXJpbmcpIC0gd2hpY2ggd2lsbCBvcmRlciB0aGUgdmFyaWFibGVzIGJ5IGhvdyBzaW1pbGFyIHRoZXkgYXJlIHRvIG9uZSBhbm90aGVyLgoKVGhlIGBjbC5saW0gPSBjKDAsIDEpYCBhcmd1bWVudCBsaW1pdHMgdGhlIGNvbG9yIGxhYmVsIHRvIGJlIGJldHdlZW4gMCBhbmQgMS4gCgoKV2UgY2FuIHNlZSB0aGF0IHRoZSBkZXZlbG9wbWVudCB2YXJpYWJsZXMgKGBpbXBgKSB2YXJpYWJsZXMgYXJlIGNvcnJlbGF0ZWQgd2l0aCBlYWNoIG90aGVyIGFzIHdlIG1pZ2h0IGV4cGVjdC4gCldlIGFsc28gc2VlIHRoYXQgdGhlIHJvYWQgZGVuc2l0eSB2YXJpYWJsZXMgc2VlbSB0byBiZSBjb3JyZWxhdGVkIHdpdGggZWFjaCBvdGhlciwgYW5kIHRoZSBlbWlzc2lvbiB2YXJpYWJsZXMgc2VlbSB0byBiZSBjb3JyZWxhdGVkIHdpdGggZWFjaCBvdGhlci4gCgoKQWxzbyBub3RpY2UgdGhhdCBub25lIG9mIHRoZSBwcmVkaWN0b3JzIGFyZSBoaWdobHkgY29ycmVsYXRlZCB3aXRoIG91ciBvdXRjb21lIHZhcmlhYmxlIChgdmFsdWVgKS4KCldlIGNhbiB0YWtlIGFsc28gdGFrZSBhIGNsb3NlciBsb29rICB1c2luZyB0aGUgYGdnY29ycigpYCBmdW5jdGlvbiBhbmQgdGhlIGBnZ3BhaXJzKClgIGZ1bmN0aW9uIG9mIHRoZSBgR0dhbGx5YCBwYWNrYWdlLiAKClRvIHNlbGVjdCBvdXIgdmFyaWFibGVzIG9mIGludGVyZXN0IHdlIGNhbiB1c2UgdGhlIGBzZWxlY3QoKWAgZnVuY3Rpb24gd2l0aCB0aGUgYGNvbnRhaW5zKClgIGZ1bmN0aW9uIG9mIHRoZSBgdGlkeXJgIHBhY2thZ2UuIAoKRmlyc3QgbGV0J3MgbG9vayBhdCB0aGUgYGltcGAvZGV2ZWxvcG1lbnQgdmFyaWFibGVzLiAKV2UgY2FuIGNoYW5nZSB0aGUgZGVmYXVsdCBjb2xvciBwYWxldHRlIChgcGFsZXR0ZSA9ICJSZEJ1ImApIGFuZCBhZGQgb24gCmNvcnJlbGF0aW9uIGNvZWZmaWNpZW50cyB0byB0aGUgcGxvdCAoYGxhYmVsID0gVFJVRWApLgoKYGBge3IsIG91dC53aWR0aCA9ICI0MDBweCJ9CnNlbGVjdChwbSwgY29udGFpbnMoImltcCIpKSAlPiUKICBnZ2NvcnIocGFsZXR0ZSA9ICJSZEJ1IiwgbGFiZWwgPSBUUlVFKQoKc2VsZWN0KHBtLCBjb250YWlucygiaW1wIikpICU+JQogIGdncGFpcnMoKQpgYGAKCgoKSW5kZWVkLCB3ZSBjYW4gbm93IHNlZSBtb3JlIGNsZWFybHkgdGhhdCBgaW1wX2ExMDAwYCBhbmQgYGltcF9hNTAwYCBhcmUgaGlnaGx5IGNvcnJlbGF0ZWQsIGFzIHdlbGwgYXMgYGltcF9hMTAwMDBgLCBgaW1wX2ExNTAwMGAuIFdlIGFsc28gZ2V0IGEgc2Vuc2Ugb2YgaG93IHRoZSBkYXRhIHBvaW50cyB2YXJ5IGFjcm9zcyB0aGUgcmFuZ2Ugb2YgdmFsdWVzLiBOb3RlIHRoYXQgaW4gdGhpcyBwbG90IHJlZCBpbmRpY2F0ZXMgcG9zaXRpdmUgY29ycmVsYXRpb24gdmFsdWVzIGFuZCBibHVlIGluZGljYXRlcyBuZWdhdGl2ZSBjb3JyZWxhdGlvbiB2YWx1ZXMuVGhpcyBpcyBpbiBjb250cmFzdCB0byBvdXIgcHJldmlvdXMgcGxvdC4KCk5leHQsIGxldCdzIHRha2UgYSBsb29rIGF0IHRoZSByb2FkIGRlbnNpdHkgZGF0YToKCmBgYHtyLCBmaWcud2VpZ2h0PTEyfQpzZWxlY3QocG0sIGNvbnRhaW5zKCJwcmkiKSkgJT4lCiAgZ2djb3JyKHBhbGV0dGUgPSAiUmRCdSIsIGhqdXN0ID0gLjg1LCBzaXplID0gMywKICAgICAgIGxheW91dC5leHA9MiwgbGFiZWwgPSBUUlVFKQpgYGAKCldlIGNhbiBzZWUgdGhhdCBtYW55IG9mIHRoZSByb2FkIGRlbnNpdHkgdmFyaWFibGVzIGFyZSBoaWdobHkgY29ycmVsYXRlZCB3aXRoIG9uZSBhbm90aGVyLCB3aGlsZSBvdGhlcnMgYXJlIGxlc3Mgc28uIEFnYWluIG5vdGUgdGhhdCBpbiB0aGlzIHBsb3QgcmVkIGluZGljYXRlcyBwb3NpdGl2ZSBjb3JyZWxhdGlvbiB2YWx1ZXMgYW5kIGJsdWUgaW5kaWNhdGVzIG5lZ2F0aXZlIGNvcnJlbGF0aW9uIHZhbHVlcy4KCkZpbmFsbHkgbGV0J3MgbG9vayBhdCB0aGUgZW1pc3Npb24gdmFyaWFibGVzLgoKYGBge3J9CnNlbGVjdChwbSwgY29udGFpbnMoIm5laSIpKSAlPiUKICBnZ2NvcnIocGFsZXR0ZSA9ICJSZEJ1IiwgaGp1c3QgPSAuODUsIHNpemUgPSAzLAogICAgICAgbGF5b3V0LmV4cD0yLCBsYWJlbCA9IFRSVUUpCgpzZWxlY3QocG0sIGNvbnRhaW5zKCJuZWkiKSkgJT4lCiAgZ2dwYWlycygpCmBgYApXZSBjYW4gc2VlIHNvbWUgZmFpcmx5IGhpZ2ggY29ycmVsYXRpb24gdmFsdWVzIGFzIHdlbGwuCgoKV2Ugd291bGQgYWxzbyBleHBlY3QgdGhlIHBvcHVsYXRpb24gZGVuc2l0eSBkYXRhIG1pZ2h0IGNvcnJlbGF0ZSB3aXRoIHNvbWUgb2YgdGhlc2UgdmFyaWFibGVzLiAKTGV0J3MgdGFrZSBhIGxvb2suCgpgYGB7cn0KcG0gJT4lCnNlbGVjdChsb2dfbmVpXzIwMDhfcG0yNV9zdW1fMTAwMDAsIHBvcGRlbnNfY291bnR5LCAKICAgICAgIGxvZ19wcmlfbGVuZ3RoXzEwMDAwLCBpbXBfYTEwMDAwKSAlPiUKICBnZ2NvcnIocGFsZXR0ZSA9ICJSZEJ1IiwgIGhqdXN0ID0gLjg1LCBzaXplID0gMywKICAgICAgIGxheW91dC5leHA9MiwgbGFiZWwgPSBUUlVFKQoKcG0gJT4lCnNlbGVjdChsb2dfbmVpXzIwMDhfcG0yNV9zdW1fMTAwMDAsIHBvcGRlbnNfY291bnR5LCAKICAgICAgIGxvZ19wcmlfbGVuZ3RoXzEwMDAwLCBpbXBfYTEwMDAwLCBjb3VudHlfcG9wKSAlPiUKICBnZ3BhaXJzKCkKYGBgCgoKSW50ZXJlc3RpbmcsIHNvIHRoZXNlIHZhcmlhYmxlcyBkb24ndCBhcHBlYXIgdG8gYmUgaGlnaGx5IGNvcnJlbGF0ZWQsIHRoZXJlZm9yZSB3ZSBtaWdodCBuZWVkIHZhcmlhYmxlcyBmcm9tIGVhY2ggb2YgdGhlIGNhdGVnb3JpZXMgdG8gcHJlZGljdCBvdXIgbW9uaXRvciBQTX4yLjV+IHBvbGx1dGlvbiB2YWx1ZXMuCgpCZWNhdXNlIHNvbWUgdmFyaWFibGVzIGluIG91ciBkYXRhIGhhdmUgZXh0cmVtZSB2YWx1ZXMsIGl0IG1pZ2h0IGJlIGdvb2QgdG8gdGFrZSBhIGxvZyB0cmFuc2Zvcm1hdGlvbi4gVGhpcyBjYW4gYWZmZWN0IG91ciBlc3RpbWF0ZXMgb2YgY29ycmVsYXRpb24uIAoKYGBge3J9CnBtICU+JQogIG11dGF0ZShsb2dfcG9wZGVuc19jb3VudHk9IGxvZyhwb3BkZW5zX2NvdW50eSkpICU+JQpzZWxlY3QobG9nX25laV8yMDA4X3BtMjVfc3VtXzEwMDAwLCBsb2dfcG9wZGVuc19jb3VudHksIAogICAgICAgbG9nX3ByaV9sZW5ndGhfMTAwMDAsIGltcF9hMTAwMDApICU+JQogIGdnY29ycihwYWxldHRlID0gIlJkQnUiLCAgaGp1c3QgPSAuODUsIHNpemUgPSAzLAogICAgICAgbGF5b3V0LmV4cD0yLCBsYWJlbCA9IFRSVUUpCgpwbSAlPiUKICBtdXRhdGUobG9nX3BvcGRlbnNfY291bnR5PSBsb2cocG9wZGVuc19jb3VudHkpKSAlPiUKICBtdXRhdGUobG9nX3BvcF9jb3VudHkgPSBsb2coY291bnR5X3BvcCkpICU+JQpzZWxlY3QobG9nX25laV8yMDA4X3BtMjVfc3VtXzEwMDAwLCBsb2dfcG9wZGVuc19jb3VudHksIAogICAgICAgbG9nX3ByaV9sZW5ndGhfMTAwMDAsIGltcF9hMTAwMDAsIGxvZ19wb3BfY291bnR5KSAlPiUKICBnZ3BhaXJzKCkKYGBgCgpJbmRlZWQgdGhpcyBpbmNyZWFzZWQgdGhlIGNvcnJlbGF0aW9uLCBidXQgdmFyaWFibGVzIGZyb20gZWFjaCBvZiB0aGVzZSBjYXRlZ29yaWVzIG1heSBzdGlsbCBwcm92ZSB0byBiZSB1c2VmdWwgZm9yIHByZWRpY3Rpb24uCgpOb3cgdGhhdCB3ZSBoYXZlIGEgc2Vuc2Ugb2Ygd2hhdCBvdXIgZGF0YSBhcmUsIHdlIGNhbiBnZXQgc3RhcnRlZCB3aXRoIGJ1aWxkaW5nIGEgbWFjaGluZSBsZWFybmluZyBtb2RlbCB0byBwcmVkaWN0IGFpciBwb2xsdXRpb24uIAoKRmlyc3QgbGV0J3Mgc2F2ZSBvdXIgZGF0YSBhZ2FpbiBiZWNhdXNlIHdlIGRpZCB3cmFuZ2xlIGl0IGp1c3QgYSB0YWQuIFRoaXMgdGltZSB3ZSB3aWxsIHNhdmUgaXQgdG8gYSBzdWJkaXJlY3Rvcnkgb2YgdGhlICJkYXRhIiBkaXJlY3RvcnkgY2FsbGVkICJ3cmFuZ2xlZCIuIFRoaXMgaXMgYSBnb29kIHByYWN0aWNlIGluIGdlbmVyYWwgZm9yIGRhdGEgYW5hbHlzZXMgdG8ga2VlcCB5b3VyIGRhdGEgb3JnYW5pemVkLiBXZSB3aWxsIGFsc28gc2F2ZSBhIGNzdiB2ZXJzaW9uIGFzIHRoaXMgaXMgb2Z0ZW4gdXNlZnVsIHRvIGdpdmUgdG8gY29sbGFib3JhdG9ycy4gVG8gZG8gdGhpcyB3ZSB3aWxsIHVzZSB0aGUgYHdyaXRlX2NzdigpYCBmdW5jdGlvbiBvZiB0aGUgYHJlYWRyYCBwYWNrYWdlLgoKYGBge3IsIGV2YWwgPSBGQUxTRX0Kc2F2ZShwbSwgZmlsZSA9IGhlcmU6OmhlcmUoImRhdGEiLCAid3JhbmdsZWQiLCAid3JhbmdsZWRfcG0ucmRhIikpCndyaXRlX2NzdihwbSwgZmlsZSA9IGhlcmU6OmhlcmUoImRhdGEiLCAid3JhbmdsZWQiLCAid3JhbmdsZWRfcG0uY3N2IikpCmBgYAoKIyAqKldoYXQgaXMgbWFjaGluZSBsZWFybmluZz8qKiAgeyN3aGF0aXNtbH0KKioqCgpZb3UgbWF5IGhhdmUgbGVhcm5lZCBhYm91dCB0aGUgY2VudHJhbCBkb2dtYSBvZiBzdGF0aXN0aWNzIHRoYXQgeW91IHNhbXBsZSBmcm9tIGEgcG9wdWxhdGlvbi4KCiFbXShpbWcvY2RpMS5wbmcpCgpUaGVuIHlvdSB1c2UgdGhlIHNhbXBsZSB0byB0cnkgdG8gZ3Vlc3Mgd2hhdCBpcyBoYXBwZW5pbmcgaW4gdGhlIHBvcHVsYXRpb24uCgohW10oaW1nL2NkaTIucG5nKQoKRm9yIHByZWRpY3Rpb24gd2UgaGF2ZSBhIHNpbWlsYXIgc2FtcGxpbmcgcHJvYmxlbQoKIVtdKGltZy9jZHAxLnBuZykKCkJ1dCBub3cgd2UgYXJlIHRyeWluZyB0byBidWlsZCBhIHJ1bGUgdGhhdCBjYW4gYmUgdXNlZCB0byBwcmVkaWN0IGEgc2luZ2xlIG9ic2VydmF0aW9uJ3MgdmFsdWUgb2Ygc29tZSBjaGFyYWN0ZXJpc3RpYyB1c2luZyBjaGFyYWN0ZXJpc3RpY3Mgb2YgdGhlIG90aGVyIG9ic2VydmF0aW9ucy4gCgohW10oaW1nL2NkcDIucG5nKQoKTGV0J3MgbWFrZSB0aGlzIG1vcmUgY29uY3JldGUuCgpJZiB5b3UgcmVjYWxsIGZyb20gdGhlIFtXaGF0IGFyZSB0aGUgZGF0YT9dKCN3aGF0YXJldGhlZGF0YSkgc2VjdGlvbiBhYm92ZSwgd2hlbiB3ZSBhcmUgdXNpbmcgbWFjaGluZSBsZWFybmluZyBmb3IgcHJlZGljdGlvbiwgb3VyIGRhdGEgY29uc2lzdHMgb2Y6IAoKMS4gQW4gKipjb250aW51b3VzKiogb3V0Y29tZSB2YXJpYWJsZSB0aGF0IHdlIHdhbnQgdG8gcHJlZGljdCAKMi4gQSBzZXQgb2YgZmVhdHVyZShzKSAob3IgcHJlZGljdG9yIHZhcmlhYmxlcykgdGhhdCB3ZSB1c2UgdG8gcHJlZGljdCB0aGUgb3V0Y29tZSB2YXJpYWJsZQoKV2Ugd2lsbCB1c2UgJFkkIHRvIGRlbm90ZSB0aGUgb3V0Y29tZSB2YXJpYWJsZSBhbmQgJFggPSAoWF8xLCBcZG90cywgWF9wKSQgdG8gZGVub3RlICRwJCBkaWZmZXJlbnQgZmVhdHVyZXMgKG9yIHByZWRpY3RvciB2YXJpYWJsZXMpLiAKQmVjYXVzZSBvdXIgb3V0Y29tZSB2YXJpYWJsZSBpcyAqKmNvbnRpbnVvdXMqKiAoYXMgb3Bwb3NlZCB0byBjYXRlZ29yaWNhbCksIHdlIGFyZSBpbnRlcmVzdGVkIGluIGEgcGFydGljdWxhciB0eXBlIG9mIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtLiAKCk91ciBnb2FsIGlzIHRvIGJ1aWxkIGEgbWFjaGluZSBsZWFybmluZyBhbGdvcml0aG0gdGhhdCB1c2VzIHRoZSBmZWF0dXJlcyAkWCQgYXMgaW5wdXQgYW5kIHByZWRpY3RzIGFuIG91dGNvbWUgdmFyaWFibGUgKG9yIGFpciBwb2xsdXRpb24gbGV2ZWxzKSBpbiB0aGUgc2l0dWF0aW9uIHdoZXJlIHdlIGRvIG5vdCBrbm93IHRoZSBvdXRjb21lIHZhcmlhYmxlLiAKClRoZSB3YXkgd2UgZG8gdGhpcyBpcyB0byB1c2UgZGF0YSB3aGVyZSB3ZSBoYXZlIGJvdGggdGhlIGZlYXR1cmVzICQoWF8xPXhfMSwgXGRvdHMgWF9wPXhfcCkkIGFuZCB0aGUgYWN0dWFsIG91dGNvbWUgJFkkIGRhdGEgdG8gX3RyYWluXyBhIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtIHRvIHByZWRpY3QgdGhlIG91dGNvbWUsIHdoaWNoIHdlIGNhbGwgJFxoYXR7WX0kLiAgCgpXaGVuIHdlIHNheSB0cmFpbiBhIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtIHdlIG1lYW4gdGhhdCB3ZSBlc3RpbWF0ZSBhIGZ1bmN0aW9uICRmJCB0aGF0IHVzZXMgdGhlIHByZWRpY3RvciB2YXJpYWJsZXMgJFgkIGFzIGlucHV0IG9yICRcaGF0e1l9ID0gZihYKSQuIAoKIyMgKipNTCBhcyBhbiBvcHRpbWl6YXRpb24gcHJvYmxlbSoqCgpJZiB3ZSBhcmUgZG9pbmcgYSBnb29kIGpvYiwgdGhlbiBvdXIgcHJlZGljdGVkIG91dGNvbWUgJFxoYXR7WX0kIHNob3VsZCBjbG9zZWx5IG1hdGNoIG91ciBhY3R1YWwgb3V0Y29tZSAkWSQgdGhhdCB3ZSBvYnNlcnZlZC4gCgpJbiB0aGlzIHdheSwgd2UgY2FuIHRoaW5rIG9mIG1hY2hpbmUgbGVhcm5pbmcgKE1MKSBhcyBhbiBvcHRpbWl6YXRpb24gcHJvYmxlbSB0aGF0IHRyaWVzIHRvIG1pbmltaXplIHRoZSBkaXN0YW5jZSBiZXR3ZWVuICRcaGF0e1l9ID0gZihYKSQgYW5kICRZJC4gCgokJGQoWSAtIGYoWCkpJCQKVGhlIGNob2ljZSBvZiBkaXN0YW5jZSBtZXRyaWMgJGQoXGNkb3QpJCBjYW4gYmUgdGhlIG1lYW4gb2YgdGhlIGFic29sdXRlIG9yIHNxdWFyZWQgZGlmZmVyZW5jZSBvciBzb21ldGhpbmcgbW9yZSBjb21wbGljYXRlZC4gCgpNdWNoIG9mIHRoZSBmaWVsZHMgb2Ygc3RhdGlzdGljcyBhbmQgY29tcHV0ZXIgc2NpZW5jZSBhcmUgZm9jdXNlZCBvbiBkZWZpbmluZyAkZiQgYW5kICRkJC4KCiMjICoqVGhlIHBhcnRzIG9mIGFuIE1MIHByb2JsZW0qKgoKVG8gc2V0IHVwIGEgbWFjaGluZSBsZWFybmluZyAoTUwpIHByb2JsZW0sIHdlIG5lZWQgYSBmZXcgY29tcG9uZW50cy4KVG8gc29sdmUgYSAoc3RhbmRhcmQpIG1hY2hpbmUgbGVhcm5pbmcgcHJvYmxlbSB5b3UgbmVlZDogCgoxLiBBIGRhdGEgc2V0IHRvIHRyYWluIGZyb20uIAoyLiBBbiBhbGdvcml0aG0gb3Igc2V0IG9mIGFsZ29yaXRobXMgeW91IGNhbiB1c2UgdG8gdHJ5IHZhbHVlcyBvZiAkZiQuCjMuIEEgZGlzdGFuY2UgbWV0cmljICRkJCBmb3IgbWVhc3VyaW5nIGhvdyBjbG9zZSAkWSQgaXMgdG8gJFxoYXR7WX0kLgo0LiBBIGRlZmluaXRpb24gb2Ygd2hhdCBhICJnb29kIiBkaXN0YW5jZSBpcy4KCldoaWxlIGVhY2ggb2YgdGhlc2UgY29tcG9uZW50cyBpcyBhIF90ZWNobmljYWxfIHByb2JsZW0sIHRoZXJlIGhhcyBiZWVuIGEgdG9uIG9mIHdvcmsgYWRkcmVzc2luZyB0aG9zZSB0ZWNobmljYWwgZGV0YWlscy4gVGhlIG1vc3QgcHJlc3Npbmcgb3BlbiBpc3N1ZSBpbiBtYWNoaW5lIGxlYXJuaW5nIGlzIHJlYWxpemluZyB0aGF0IHRob3VnaCB0aGVzZSBhcmUgX3RlY2huaWNhbF8gc3RlcHMgdGhleSBhcmUgbm90IF9vYmplY3RpdmVfIHN0ZXBzLiBJbiBvdGhlciB3b3JkcywgaG93IHlvdSBjaG9vc2UgdGhlIGRhdGEsIGFsZ29yaXRobSwgbWV0cmljLCBhbmQgZGVmaW5pdGlvbiBvZiAiZ29vZCIgc2F5cyB3aGF0IHlvdSB2YWx1ZSBhbmQgY2FuIGRyYW1hdGljYWxseSBjaGFuZ2UgdGhlIHJlc3VsdHMuIEEgY291cGxlIG9mIGNhc2VzIHdoZXJlIHRoaXMgd2FzIGEgYmlnIGRlYWwgYXJlOiAKCjEuIFtNYWNoaW5lIGxlYXJuaW5nIGZvciByZWNpZGl2aXNtXShodHRwczovL3d3dy5wcm9wdWJsaWNhLm9yZy9hcnRpY2xlL21hY2hpbmUtYmlhcy1yaXNrLWFzc2Vzc21lbnRzLWluLWNyaW1pbmFsLXNlbnRlbmNpbmcpIC0gcGVvcGxlIGJ1aWx0IE1MIG1vZGVscyB0byBwcmVkaWN0IHdobyB3b3VsZCByZS1jb21taXQgYSBjcmltZS4gQnV0IHRoZXNlIHByZWRpY3Rpb25zIHdlcmUgYmFzZWQgb24gaGlzdG9yaWNhbGx5IGJpYXNlZCBkYXRhIHdoaWNoIGxlZCB0byBiaWFzZWQgcHJlZGljdGlvbnMgYWJvdXQgd2hvIHdvdWxkIGNvbW1pdCBuZXcgY3JpbWVzLiAKMi4gW0RlY2lkaW5nIGhvdyBzZWxmIGRyaXZpbmcgY2FycyBzaG91bGQgYWN0XShodHRwczovL3d3dy5uYXR1cmUuY29tL2FydGljbGVzL2Q0MTU4Ni0wMTgtMDcxMzUtMCkgLSBzZWxmIGRyaXZpbmcgY2FycyB3aWxsIGhhdmUgdG8gbWFrZSBkZWNpc2lvbnMgYWJvdXQgaG93IHRvIGRyaXZlLCB3aG8gdGhleSBtaWdodCBpbmp1cmUsIGFuZCBob3cgdG8gYXZvaWQgYWNjaWRlbnRzLiBEZXBlbmRpbmcgb24gb3VyIGNob2ljZXMgZm9yICRmJCBhbmQgJGQkIHRoZXNlIG1pZ2h0IGxlYWQgdG8gd2lsZGx5IGRpZmZlcmVudCBraW5kcyBvZiBzZWxmIGRyaXZpbmcgY2Fycy4gVHJ5IG91dCB0aGUgW21vcmFsbWFjaGluZV0oaHR0cDovL21vcmFsbWFjaGluZS5taXQuZWR1LykgdG8gc2VlIGhvdyB0aGlzIGxvb2tzIGluIHByYWN0aWNlLiAKCk5vdyB0aGF0IHdlIGtub3cgYSBiaXQgbW9yZSBhYm91dCBtYWNoaW5lIGxlYXJuaW5nLCBsZXQncyBidWlsZCBhIG1vZGVsIHRvIHByZWRpY3QgYWlyIHBvbGx1dGlvbiBsZXZlbHMgdXNpbmcgdGhlIGB0aWR5bW9kZWxzYCBmcmFtZXdvcmsuIAoKIyAqKk1hY2hpbmUgbGVhcm5pbmcgd2l0aCBgdGlkeW1vZGVsc2AqKgoqKioKVGhlIGdvYWwgaXMgdG8gYnVpbGQgYSBtYWNoaW5lIGxlYXJuaW5nIGFsZ29yaXRobSB0aGF0IHVzZXMgdGhlIGZlYXR1cmVzIGFzIGlucHV0IGFuZCBwcmVkaWN0cyBhIG91dGNvbWUgdmFyaWFibGUgKG9yIGFpciBwb2xsdXRpb24gbGV2ZWxzKSBpbiB0aGUgc2l0dWF0aW9uIHdoZXJlIHdlIGRvIG5vdCBrbm93IHRoZSBvdXRjb21lIHZhcmlhYmxlLiAKClRoZSB3YXkgd2UgZG8gdGhpcyBpcyB0byB1c2UgZGF0YSB3aGVyZSB3ZSBoYXZlIGJvdGggdGhlIGlucHV0IGFuZCBvdXRwdXQgZGF0YSB0byBfdHJhaW5fIGEgbWFjaGluZSBsZWFybmluZyBhbGdvcml0aG0uIAoKVG8gdHJhaW4gYSBtYWNoaW5lIGxlYXJuaW5nIGFsZ29yaXRobSwgd2Ugd2lsbCB1c2UgdGhlIGB0aWR5bW9kZWxzYCBwYWNrYWdlIGVjb3N5c3RlbS4gCgojIyAqKk92ZXJ2aWV3KioKKioqCgojIyMgKipUaGUgYHRpZHltb2RlbHNgIGVjb3N5c3RlbSoqCioqKgoKVG8gcGVyZm9ybSBvdXIgYW5hbHlzaXMgd2Ugd2lsbCBiZSB1c2luZyB0aGUgYHRpZHltb2RlbHNgIHN1aXRlIG9mIHBhY2thZ2VzLiAKWW91IG1heSBiZSBmYW1pbGlhciB3aXRoIHRoZSBvbGRlciBwYWNrYWdlcyBgY2FyZXRgIG9yIGBtbHJgIHdoaWNoIGFyZSBhbHNvIGZvciBtYWNoaW5lIGxlYXJuaW5nIGFuZCBtb2RlbGluZyBidXQgYXJlIG5vdCBhIHBhcnQgb2YgdGhlIGB0aWR5dmVyc2VgLiAKW01heCBLdWhuXShodHRwczovL3Jlc291cmNlcy5yc3R1ZGlvLmNvbS9hdXRob3JzL21heC1rdWhuKXt0YXJnZXQ9Il9ibGFuayJ9IGRlc2NyaWJlcyBgdGlkeW1vZGVsc2AgbGlrZSB0aGlzOgoKPiAiT3RoZXIgcGFja2FnZXMsIHN1Y2ggYXMgY2FyZXQgYW5kIG1sciwgaGVscCB0byBzb2x2ZSB0aGUgUiBtb2RlbCBBUEkgaXNzdWUuIFRoZXNlIHBhY2thZ2VzIGRvIGEgbG90IG9mIG90aGVyIHRoaW5ncyB0b286IHByZS1wcm9jZXNzaW5nLCBtb2RlbCB0dW5pbmcsIHJlc2FtcGxpbmcsIGZlYXR1cmUgc2VsZWN0aW9uLCBlbnNlbWJsaW5nLCBhbmQgc28gb24uIEluIHRoZSB0aWR5dmVyc2UsIHdlIHN0cml2ZSB0byBtYWtlIG91ciBwYWNrYWdlcyBtb2R1bGFyIGFuZCBwYXJzbmlwIGlzIGRlc2lnbmVkIG9ubHkgdG8gc29sdmUgdGhlIGludGVyZmFjZSBpc3N1ZS4gSXQgaXMgbm90IGRlc2lnbmVkIHRvIGJlIGEgZHJvcC1pbiByZXBsYWNlbWVudCBmb3IgY2FyZXQuClRoZSB0aWR5bW9kZWxzIHBhY2thZ2UgY29sbGVjdGlvbiwgd2hpY2ggaW5jbHVkZXMgcGFyc25pcCwgaGFzIG90aGVyIHBhY2thZ2VzIGZvciBtYW55IG9mIHRoZXNlIHRhc2tzLCBhbmQgdGhleSBhcmUgZGVzaWduZWQgdG8gd29yayB0b2dldGhlci4gV2UgYXJlIHdvcmtpbmcgdG93YXJkcyBoaWdoZXItbGV2ZWwgQVBJcyB0aGF0IGNhbiByZXBsaWNhdGUgYW5kIGV4dGVuZCB3aGF0IHRoZSBjdXJyZW50IG1vZGVsIHBhY2thZ2VzIGNhbiBkby4iCgpUaGVyZSBhcmUgbWFueSBSIHBhY2thZ2VzIGluIHRoZSBgdGlkeW1vZGVsc2AgZWNvc3lzdGVtLCB3aGljaCBhc3Npc3Qgd2l0aCB2YXJpb3VzIHN0ZXBzIGluIHRoZSBwcm9jZXNzIG9mIGJ1aWxkaW5nIGEgbWFjaGluZSBsZWFybmluZyBhbGdvcml0aG0uIFRoZXNlIGFyZSB0aGUgbWFpbiBwYWNrYWdlcywgYnV0IHRoZXJlIGFyZSBvdGhlcnMuCgpgYGB7ciwgZWNobz1GQUxTRSwgb3V0LndpZHRoPSI4MDBweCJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsInNpbXBsZXRpZHltb2RlbHMucG5nIikpCmBgYAoKVGhpcyBpcyBhIHNjaGVtYXRpYyBvZiBob3cgdGhlc2UgcGFja2FnZXMgd29yayB0b2dldGhlciB0byBidWlsZCBhIG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtOgoKYGBge3IsIGVjaG89RkFMU0UsIG91dC53aWR0aD0iODAwcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCJNYWNoaW5lTGVhcm5pbmcucG5nIikpCmBgYAoKSGVyZSB3ZSBjYW4gc2VlIHRoYXQgYWZ0ZXIgZXhwbG9yaW5nIGFuZCAgc3BsaXR0aW5nIHRoZSBkYXRhLCB3ZSBwZXJmb3JtIHRoZSBpbml0aWFsIG1vZGVsaW5nIHN0YWdlcyBvZiB2YXJpYWJsZSBhc3NpZ25tZW50IGFuZCBwcmUtcHJvY2Vzc2luZyBzdGVwcyBhcyBzaG93biBpbiB0aGUgYmx1ZSBib3ggdGhlLCB3aGlsZSB0aGUgZ3JlZW4gYm94IGluZGljYXRlcyB0aGUgc3RlcHMgcmVxdWlyZWQgdG8gIHRyYWluIHRoZSBtb2RlbCBieSBzcGVjaWZ5aW5nIHRoZSBtb2RlbCwgZml0dGluZyB0aGUgbW9kZWwgYW5kIHR1bmluZyBpdC4gCgojIyMgKipCZW5lZml0cyBvZiBgdGlkeW1vZGVsc2AqKgoqKioKClRoZSB0d28gbWFqb3IgYmVuZWZpdHMgb2YgYHRpZHltb2RlbHNgIGFyZTogCgoxLiBTdGFuZGFyZGl6ZWQgd29ya2Zsb3cvZm9ybWF0L25vdGF0aW9uIGFjcm9zcyBkaWZmZXJlbnQgdHlwZXMgb2YgbWFjaGluZSBsZWFybmluZyBhbGdvcml0aG1zICAKCkRpZmZlcmVudCBub3RhdGlvbnMgYXJlIHJlcXVpcmVkIGZvciBkaWZmZXJlbnQgYWxnb3JpdGhtcyBhcyB0aGUgYWxnb3JpdGhtcyBoYXZlIGJlZW4gZGV2ZWxvcGVkIGJ5IGRpZmZlcmVudCBwZW9wbGUuIFRoaXMgd291bGQgcmVxdWlyZSB0aGUgcGFpbnN0YWtpbmcgcHJvY2VzcyBvZiByZWZvcm1hdHRpbmcgdGhlIGRhdGEgdG8gYmUgY29tcGF0aWJsZSB3aXRoIGVhY2ggYWxnb3JpdGhtIGlmIG11bHRpcGxlIGFsZ29yaXRobXMgd2VyZSB0ZXN0ZWQuCgoyLiBDYW4gZWFzaWx5IG1vZGlmeSBwcmUtcHJvY2Vzc2luZywgYWxnb3JpdGhtIGNob2ljZSwgYW5kIGh5cGVyLXBhcmFtZXRlciB0dW5pbmcgbWFraW5nIG9wdGltaXphdGlvbiBlYXN5ICAKCk1vZGlmeWluZyBhIHBpZWNlIG9mIHRoZSBvdmVyYWxsIHByb2Nlc3MgaXMgbm93IGVhc2llciB0aGFuIGJlZm9yZSBiZWNhdXNlIG1hbnkgb2YgdGhlIHN0ZXBzIGFyZSBzcGVjaWZpZWQgdXNpbmcgdGhlIGB0aWR5bW9kZWxzYCBwYWNrYWdlcyBpbiBhIGNvbnZlbmllbnQgbWFubmVyLiBUaHVzIHRoZSBlbnRpcmUgcHJvY2VzcyBjYW4gYmUgcmVydW4gYWZ0ZXIgYSBzaW1wbGUgY2hhbmdlIHRvIHByZS1wcm9jZXNzaW5nIHdpdGhvdXQgbXVjaCBkaWZmaWN1bHR5LgoKIyMgKipgdGlkeW1vZGVsc2AgU3RlcHMqKgoqKioKCmBgYHtyLCBlY2hvPUZBTFNFLCBvdXQud2lkdGg9IjgwMHB4In0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwiVXBkYXRlZF90aWR5bW9kZWxzX2Jhc2ljcy5wbmciKSkKYGBgCgojIyAqKlNwbGl0dGluZyB0aGUgZGF0YSoqCioqKgoKVGhlIGZpcnN0IHN0ZXAgYWZ0ZXIgZGF0YSBleHBsb3JhdGlvbiBpbiBtYWNoaW5lIGxlYXJuaW5nIGFuYWx5c2lzIGlzIHRvIFtzcGxpdCB0aGUgZGF0YV0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL3RyYWluLXZhbGlkYXRpb24tYW5kLXRlc3Qtc2V0cy03MmNiNDBjYmE5ZTcpe3RhcmdldD0iX2JsYW5rIn0gaW50byAqKnRyYWluaW5nKiogYW5kICoqdGVzdGluZyoqIGRhdGEgc2V0cy4gCgpUaGUgdHJhaW5pbmcgZGF0YSBzZXQgd2lsbCBiZSB1c2VkIHRvIGJ1aWxkIGFuZCB0dW5lIG91ciBtb2RlbC4gClRoaXMgaXMgdGhlIGRhdGEgdGhhdCB0aGUgbW9kZWwgImxlYXJucyIgb24uIApUaGUgdGVzdGluZyBkYXRhIHNldCB3aWxsIGJlIHVzZWQgdG8gZXZhbHVhdGUgdGhlIHBlcmZvcm1hbmNlIG9mIG91ciBtb2RlbCBpbiBhIG1vcmUgZ2VuZXJhbGl6YWJsZSB3YXkuIFdoYXQgZG8gd2UgbWVhbiBieSAiZ2VuZXJhbGl6YWJsZSI/CgpSZW1lbWJlciB0aGF0IG91ciBtYWluIGdvYWwgaXMgdG8gdXNlIG91ciBtb2RlbCB0byBiZSBhYmxlIHRvIHByZWRpY3QgYWlyIHBvbGx1dGlvbiBsZXZlbHMgaW4gYXJlYXMgd2hlcmUgdGhlcmUgYXJlIG5vIGdyYXZpbWV0cmljIG1vbml0b3JzLiAKClRoZXJlZm9yZSwgaWYgb3VyIG1vZGVsIGlzIHJlYWxseSBnb29kIGF0IHByZWRpY3RpbmcgYWlyIHBvbGx1dGlvbiB3aXRoIHRoZSBkYXRhIHRoYXQgd2UgdXNlIHRvIGJ1aWxkIGl0LCBpdCBtaWdodCBub3QgZG8gdGhlIGJlc3Qgam9iIGZvciB0aGUgYXJlYXMgd2hlcmUgdGhlcmUgYXJlIGZldyB0byBubyBtb25pdG9ycy4gCgpUaGlzIHdvdWxkIGNhdXNlIHVzIHRvIGhhdmUgcmVhbGx5IGdvb2QgcHJlZGljdGlvbiBhY2N1cmFjeSBhbmQgd2UgbWlnaHQgYXNzdW1lIHRoYXQgd2Ugd2VyZSBnb2luZyB0byBkbyBhIGdvb2Qgam9iIGVzdGltYXRpbmcgYWlyIHBvbGx1dGlvbiBhbnkgdGltZSB3ZSB1c2Ugb3VyIG1vZGVsLCBidXQgaW4gZmFjdCB0aGlzIHdvdWxkIGxpa2VseSBub3QgYmUgdGhlIGNhc2UuIApUaGlzIHNpdHVhdGlvbiBpcyB3aGF0IHdlIGNhbGwgKipbb3ZlcmZpdHRpbmddKGh0dHBzOi8vdG93YXJkc2RhdGFzY2llbmNlLmNvbS90cmFpbi10ZXN0LXNwbGl0LWFuZC1jcm9zcy12YWxpZGF0aW9uLWluLXB5dGhvbi04MGI2MWJlY2E0YjYpe3RhcmdldD0iX2JsYW5rIn0gKiouCgpPdmVyZml0dGluZyBoYXBwZW5zIHdoZW4gd2UgZW5kIHVwIG1vZGVsaW5nIG5vdCBvbmx5IHRoZSBtYWpvciByZWxhdGlvbnNoaXBzIGluIG91ciBkYXRhIGJ1dCBhbHNvIHRoZSBub2lzZSB3aXRoaW4gb3VyIGRhdGEuIAoKYGBge3IsIGVjaG89RkFMU0V9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCJodHRwczovL21pcm8ubWVkaXVtLmNvbS9tYXgvMTExMC8xKnRCRXJYWVZ2VHcyalNVWUs3dGhVMkEucG5nIikKYGBgCgojIyMjIyBbW3NvdXJjZV1dKGh0dHBzOi8vbWlyby5tZWRpdW0uY29tL21heC8xMTEwLzEqdEJFclhZVnZUdzJqU1VZSzd0aFUyQS5wbmcpe3RhcmdldD0iX2JsYW5rIn0KCklmIHdlIGdldCBnb29kIHByZWRpY3Rpb24gd2l0aCBvdXIgdGVzdGluZyBzZXQsIHRoZW4gd2Uga25vdyB0aGF0IG91ciBtb2RlbCBjYW4gYmUgYXBwbGllZCB0byBvdGhlciBkYXRhIGFuZCB3aWxsIGxpa2VseSBwZXJmb3JtIHdlbGwuIFdlIHdpbGwgZGlzY3VzcyB0aGlzIG1vcmUgbGF0ZXIuCgpXZSB3aWxsIG5vdCB0b3VjaCB0aGUgdGVzdGluZyBzZXQgdW50aWwgd2UgaGF2ZSBjb21wbGV0ZWQgb3B0aW1pemluZyBvdXIgbW9kZWwgd2l0aCB0aGUgdHJhaW5pbmcgc2V0LiAKVGhpcyB3aWxsIGFsbG93IHVzIHRvIGhhdmUgYSBsZXNzIGJpYXNlZCBldmFsdWF0aW9uIG9mIGhvdyB3ZWxsIG91ciBtb2RlbCBjYW4gZG8gd2l0aCBvdGhlciBkYXRhIGJlc2lkZXMgdGhlIGRhdGEgdXNlZCBpbiB0aGUgdHJhaW5pbmcgc2V0IHRvIGJ1aWxkIHRoZSBtb2RlbC4gCioqSWRlYWxseSwgeW91IHdvdWxkIGFsc28gd2FudCBhIGNvbXBsZXRlbHkgaW5kZXBlbmRlbnQgZGF0YSBzZXQgdG8gZnVydGhlciB0ZXN0IHRoZSBwZXJmb3JtYW5jZSBvZiB5b3VyIG1vZGVsLioqCgpUbyBzcGxpdCB0aGUgZGF0YSBpbnRvIHRyYWluaW5nIGFuZCB0ZXN0aW5nLCB3ZSB3aWxsIHVzZSB0aGUgYGluaXRpYWxfc3BsaXQoKWAgZnVuY3Rpb24gaW4gdGhlIGByc2FtcGxlYCBwYWNrYWdlIHRvIHNwZWNpZnkgaG93IHdlIHdhbnQgdG8gc3BsaXQgb3VyIGRhdGEuCgoKYGBge3IsIGVjaG89RkFMU0V9Cgprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCJzcGxpdC5wbmciKSkKYGBgCgoKIyMjIyB7LmNsaWNrX3RvX2V4cGFuZF9ibG9ja30KCjxkZXRhaWxzPiA8c3VtbWFyeT4gSWYgeW91IHNraXBwZWQgcHJldmlvdXMgc2VjdGlvbnMgY2xpY2sgaGVyZSBmb3IgbW9yZSBpbmZvcm1hdGlvbiBvbiBob3cgdG8gb2J0YWluIGFuZCBsb2FkIHRoZSBkYXRhLiA8L3N1bW1hcnk+CgpGaXJzdCB5b3UgbmVlZCB0byBpbnN0YWxsIHRoZSBgT0NTZGF0YWAgcGFja2FnZToKCmBgYHtyLCBldmFsPUZBTFNFfQppbnN0YWxsLnBhY2thZ2VzKCJPQ1NkYXRhIikKYGBgCgpUaGVuLCB5b3UgbWF5IGRvd25sb2FkIGFuZCBsb2FkIHRoZSB3cmFuZ2xlZCBkYXRhIGAucmRhYCBmaWxlIHVzaW5nIHRoZSBmb2xsb3dpbmcgY29kZToKCmBgYHtyLCBldmFsPUZBTFNFfQpPQ1NkYXRhOjp3cmFuZ2xlZF9yZGEoIm9jcy1icC1haXItcG9sbHV0aW9uIiwgb3V0cGF0aCA9IGdldHdkKCkpCmxvYWQoaGVyZTo6aGVyZSgiT0NTZGF0YSIsICJkYXRhIiwgIndyYW5nbGVkIiwgIndyYW5nbGVkX3BtLnJkYSIpKQpgYGAKCklmIHRoZSBwYWNrYWdlIGRvZXMgbm90IHdvcmsgZm9yIHlvdSwgeW91IG1heSBhbHNvIGRvd25sb2FkIHRoaXMgYC5yZGFgIGZpbGUgYnkgY2xpY2tpbmcgdGhpcyBsaW5rIFtoZXJlXShodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vb3BlbmNhc2VzdHVkaWVzL29jcy1icC1haXItcG9sbHV0aW9uL21hc3Rlci9kYXRhL3dyYW5nbGVkL3dyYW5nbGVkX3BtLnJkYSkuCgpUbyBsb2FkIHRoZSBkb3dubG9hZGVkIGRhdGEgaW50byB5b3VyIGVudmlyb25tZW50LCB5b3UgbWF5IGRvdWJsZSBjbGljayBvbiB0aGUgYC5yZGFgIGZpbGUgaW4gUnN0dWRpbyBvciB1c2luZyB0aGUgYGxvYWQoKWAgZnVuY3Rpb24uCgpUbyBjb3B5IGFuZCBwYXN0ZSBvdXIgY29kZSBiZWxvdywgcGxhY2UgdGhlIGRvd25sb2FkZWQgYC5yZGFgIGZpbGUgaW4geW91ciBjdXJyZW50IHdvcmtpbmcgZGlyZWN0b3J5IHdpdGhpbiBhIHN1YmRpcmVjdG9yeSBjYWxsZWQgIndyYW5nbGVkIiB3aXRoaW4gYSBzdWJkaXJlY3RvcnkgY2FsbGVkICJkYXRhIi4gV2UgdXNlZCBhbiBSU3R1ZGlvIHByb2plY3QgYW5kIHRoZSBbYGhlcmVgIHBhY2thZ2VdKGh0dHBzOi8vZ2l0aHViLmNvbS9qZW5ueWJjL2hlcmVfaGVyZSkgdG8gbmF2aWdhdGUgdG8gdGhlIGZpbGUgbW9yZSBlYXNpbHkuIAoKYGBge3J9CmxvYWQoaGVyZTo6aGVyZSgiZGF0YSIsICJ3cmFuZ2xlZCIsICJ3cmFuZ2xlZF9wbS5yZGEiKSkKYGBgCgo8aHIgc3R5bGU9ImhlaWdodDoxcHg7Ym9yZGVyOm5vbmU7Y29sb3I6IzMzMztiYWNrZ3JvdW5kLWNvbG9yOiMzMzM7IiAvPgoKPGRldGFpbHM+IDxzdW1tYXJ5PiBDbGljayBoZXJlIHRvIHNlZSBtb3JlIGFib3V0IGNyZWF0aW5nIG5ldyBwcm9qZWN0cyBpbiBSU3R1ZGlvLiA8L3N1bW1hcnk+CgpZb3UgY2FuIGNyZWF0ZSBhIHByb2plY3QgYnkgZ29pbmcgdG8gdGhlIEZpbGUgbWVudSBvZiBSU3R1ZGlvIGxpa2Ugc286CgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoPSI2MCUifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCAiTmV3X3Byb2plY3QucG5nIikpCmBgYAoKWW91IGNhbiBhbHNvIGRvIHNvIGJ5IGNsaWNraW5nIHRoZSBwcm9qZWN0IGJ1dHRvbjoKCmBgYHtyLCBlY2hvID0gRkFMU0UsIG91dC53aWR0aD0iNjAlIn0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwgInByb2plY3RfYnV0dG9uLnBuZyIpKQpgYGAKClNlZSBbaGVyZV0oaHR0cHM6Ly9zdXBwb3J0LnJzdHVkaW8uY29tL2hjL2VuLXVzL2FydGljbGVzLzIwMDUyNjIwNy1Vc2luZy1Qcm9qZWN0cykgdG8gbGVhcm4gbW9yZSBhYm91dCB1c2luZyBSU3R1ZGlvIHByb2plY3RzIGFuZCBbaGVyZV0oaHR0cHM6Ly9naXRodWIuY29tL2plbm55YmMvaGVyZV9oZXJlKSB0byBsZWFybiBtb3JlIGFib3V0IHRoZSBgaGVyZWAgcGFja2FnZS4KCjwvZGV0YWlscz4KCjxociBzdHlsZT0iaGVpZ2h0OjFweDtib3JkZXI6bm9uZTtjb2xvcjojMzMzO2JhY2tncm91bmQtY29sb3I6IzMzMzsiIC8+Cgo8L2RldGFpbHM+CgojIyMjCgpgYGB7cn0Kc2V0LnNlZWQoMTIzNCkKcG1fc3BsaXQgPC0gcnNhbXBsZTo6aW5pdGlhbF9zcGxpdChkYXRhID0gcG0sIHByb3AgPSAyLzMpCnBtX3NwbGl0CmBgYAoKQSBjb3VwbGUgb2Ygbm90ZXMgZnJvbSB0aGUgY29kZSBhYm92ZTogCgotIFR5cGljYWxseSwgZGF0YSBhcmUgc3BsaXQgd2l0aCB0aGUgbWFqb3JpdHkgb2YgdGhlIG9ic2VydmF0aW9ucyBmb3IgdHJhaW5pbmcgYW5kIGEgc21hbGxlciBwb3J0aW9uIGZvciB0ZXN0aW5nLiBUaGUgZGVmYXVsdCB3aXRoIHRoaXMgZnVuY3Rpb24gaXMgMy80ICg3NSUpIG9mIHRoZSBvYnNlcnZhdGlvbnMgZm9yIHRyYWluaW5nIGFuZCAxLzQgKDI1JSkgZm9yIHRlc3RpbmcuIFRoaXMgaXMgdGhlIGRlZmF1bHQgcHJvcG9ydGlvbiBhbmQgZG9lcyBub3QgbmVlZCB0byBiZSBzcGVjaWZpZWQuIEhvd2V2ZXIsIHlvdSBjYW4gY2hhbmdlIHRoZSBwcm9wb3J0aW9uIHVzaW5nIHRoZSBgcHJvcGAgYXJndW1lbnQsIHdoaWNoIHdlIHdpbGwgZG8gaGVyZSBmb3IgaWxsdXN0cmF0aXZlIHB1cnBvc2VzLiBPZnRlbiBwZW9wbGUgdXNlIDgwJSAoNC81KSBmb3IgdHJhaW5pbmcgYW5kIDIwJSBmb3IgdGVzdGluZyAoMS81KSBvciBzb21lIG90aGVyIHNpbWlsYXIgcHJvcG9ydGlvbiBsaWtlIHdoYXQgd2UgdXNlZCBoZXJlIHdpdGggMi8zIGZvciB0cmFpbmluZyBhbmQgMS8zIGZvciB0ZXN0aW5nLiAKCiMjIyMgey5jbGlja190b19leHBhbmRfYmxvY2t9Cgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byBsZWFybiBtb3JlIGFib3V0IGhvdyBwZW9wbGUgZGVjaWRlIHdoYXQgc3BsaXQgcHJvcG9ydGlvbiB0byB1c2UuIDwvc3VtbWFyeT4KCkhhdmluZyBtb3JlIHRyYWluaW5nIGRhdGEgaGVscHMgdGhlIG1vZGVsIHRvIHRyYWluIG9uIGEgZ3JlYXRlciB2YXJpZXR5IG9mIG9ic2VydmF0aW9ucy4gSG93ZXZlciwgaGF2aW5nIG1vcmUgdGVzdGluZyBkYXRhIGhlbHBzIHRvIHNlZSBob3cgZ2VuZXJhbGl6YWJsZSB0aGUgbW9kZWwgaXMgYW5kIGFsbG93cyBmb3IgYmV0dGVyIGNvbXBhcmlzb25zIG9mIGRpZmZlcmVudCBtb2RlbHMuIFRoZSBuZWVkIGZvciBlYWNoIG1heSBkZXBlbmQgb24geW91ciBkYXRhIGFuZCBob3cgbXVjaCB2YXJpYWJpbGl0eSBpdCBoYXMgYW5kIHRoZSBzaXplIG9mIHlvdXIgZGF0YS4gRm9yIHNtYWxsZXIgZGF0YXNldHMsIHNldHRpbmcgYXNpZGUgYSBsYXJnZXIgcG9ydGlvbiBmb3IgdGVzdGluZyBjYW4gYmUgYmVuZWZpY2lhbCB0byBhdm9pZCBoYXZpbmcgYSB2ZXJ5IHNtYWxsIHRlc3RpbmcgZGF0YXNldC4gSGVyZSdzIGEgW3BhcGVyXShodHRwczovL29ubGluZWxpYnJhcnkud2lsZXkuY29tL2RvaS9mdWxsLzEwLjEwMDIvc2FtLjExNTgzKSB0aGF0IGRlc2NyaWJlcyBtb3JlIGFib3V0IHRoaXMgdG9waWMuCgo8L2RldGFpbHM+CgojIyMjCgotIFNpbmNlIHRoZSBzcGxpdCBpcyBwZXJmb3JtZWQgcmFuZG9tbHksIGl0IGlzIGEgZ29vZCBpZGVhIHRvIHVzZSB0aGUgYHNldC5zZWVkKClgIGZ1bmN0aW9uIGluIGJhc2UgUiB0byBlbnN1cmUgdGhhdCBpZiB5b3VyIHJlcnVuIHlvdXIgY29kZSB0aGF0IHlvdXIgc3BsaXQgd2lsbCBiZSB0aGUgc2FtZSBuZXh0IHRpbWUuCi0gV2UgY2FuIHNlZSB0aGUgbnVtYmVyIG9mIG1vbml0b3JzIGluIG91ciB0cmFpbmluZywgdGVzdGluZywgYW5kIG9yaWdpbmFsIGRhdGEgYnkgdHlwaW5nIGluIHRoZSBuYW1lIG9mIG91ciBzcGxpdCBvYmplY3QuIFRoZSByZXN1bHQgd2lsbCBsb29rIGxpa2UgdGhpczoKPHRyYWluaW5nIGRhdGEgc2FtcGxlIG51bWJlciwgdGVzdGluZyBkYXRhIHNhbXBsZSBudW1iZXIsIG9yaWdpbmFsIHNhbXBsZSBudW1iZXI+IAoKTm93LCB5b3UgY2FuIGFsc28gc3BlY2lmeSBhIHZhcmlhYmxlIHRvIHN0cmF0aWZ5IGJ5IHdpdGggdGhlIGBzdHJhdGFgIGFyZ3VtZW50LiAKVGhpcyBpcyB1c2VmdWwgaWYgeW91IGhhdmUgaW1iYWxhbmNlZCBjYXRlZ29yaWNhbCB2YXJpYWJsZXMgYW5kIHlvdSB3b3VsZCBsaWtlIHRvIGludGVudGlvbmFsbHkgbWFrZSBzdXJlIHRoYXQgdGhlcmUgYXJlIHNpbWlsYXIgbnVtYmVyIG9mIHNhbXBsZXMgb2YgdGhlIHJhcmVyIGNhdGVnb3JpZXMgaW4gYm90aCB0aGUgdGVzdGluZyBhbmQgdHJhaW5pbmcgc2V0cy4gCk90aGVyd2lzZSB0aGUgc3BsaXQgaXMgcGVyZm9ybWVkIHJhbmRvbWx5LiAKCkFjY29yZGluZyB0byB0aGUgW2RvY3VtZW50YXRpb25dKGh0dHBzOi8vd3d3LnJkb2N1bWVudGF0aW9uLm9yZy9wYWNrYWdlcy9yc2FtcGxlL3ZlcnNpb25zLzAuMC41L3RvcGljcy9pbml0aWFsX3NwbGl0KSBmb3IgdGhlIGByc2FtcGxlYCBwYWNrYWdlOgoKPiBUaGUgc3RyYXRhIGFyZ3VtZW50IGNhdXNlcyB0aGUgcmFuZG9tIHNhbXBsaW5nIHRvIGJlIGNvbmR1Y3RlZCB3aXRoaW4gdGhlIHN0cmF0aWZpY2F0aW9uIHZhcmlhYmxlLiBUaGlzIGNhbiBoZWxwIGVuc3VyZSB0aGF0IHRoZSBudW1iZXIgb2YgZGF0YSBwb2ludHMgaW4gdGhlIHRyYWluaW5nIGRhdGEgaXMgZXF1aXZhbGVudCB0byB0aGUgcHJvcG9ydGlvbnMgaW4gdGhlIG9yaWdpbmFsIGRhdGEgc2V0LgoKSW4gdGhlIGNhc2Ugd2l0aCBvdXIgZGF0YSBzZXQsIHBlcmhhcHMgd2Ugd291bGQgbGlrZSBvdXIgdHJhaW5pbmcgc2V0IHRvIGhhdmUgc2ltaWxhciBwcm9wb3J0aW9ucyBvZiBtb25pdG9ycyBmcm9tIGVhY2ggb2YgdGhlIHN0YXRlcyBhcyBpbiB0aGUgaW5pdGlhbCBkYXRhLiAKVGhpcyBtaWdodCBiZSB1c2VmdWwgaWYgd2Ugd2FudCBvdXIgbW9kZWwgdG8gYmUgZ2VuZXJhbGl6YWJsZSBhY3Jvc3MgYWxsIG9mIHRoZSBzdGF0ZXMuCgpXZSBjYW4gc2VlIHRoYXQgaW5kZWVkIHRoZXJlIGFyZSBkaWZmZXJlbnQgcHJvcG9ydGlvbnMgb2YgbW9uaXRvcnMgaW4gZWFjaCBzdGF0ZSBieSB1c2luZyB0aGUgYGNvdW50KClgIGZ1bmN0aW9uIG9mIHRoZSBgZHBseXJgIHBhY2thZ2UuIAoKYGBge3IsIGV2YWwgPSBGQUxTRX0KY291bnQocG0sIHN0YXRlKQpgYGAKClNjcm9sbCB0aHJvdWdoIHRoZSBvdXRwdXQ6CgojIyMjIHsuc2Nyb2xsYWJsZSB9CgpgYGB7ciwgZWNobz1GQUxTRX0KIyBTY3JvbGwgdGhyb3VnaCB0aGUgb3V0cHV0IQpjb3VudChwbSwgc3RhdGUpICU+JQogIHByaW50KG4gPSAxZTMpCmBgYAojIyMjCgpJZiBvdXIgZGF0YSBzZXQgd2VyZSBsYXJnZSBlbm91Z2ggaXQgbWlnaHQgYmUgbmljZSB0aGVuIHRvIHN0cmF0aWZ5IGJ5IHN0YXRlIHVzaW5nIHRoZSBgc3RyYXRhID0gInN0YXRlImAgYXJndW1lbnQgaW4gYGluaXRpYWxfc3BsaXQoKWAsIGJ1dCBvdXIgZGF0YSBpcyB1bmZvcnR1bmF0ZWx5IG5vdCBsYXJnZSBlbm91Z2guIAoKSW1wb3J0YW50bHkgdGhlIGBpbml0aWFsX3NwbGl0KClgIGZ1bmN0aW9uIG9ubHkgZGV0ZXJtaW5lcyB3aGF0IHJvd3Mgb2Ygb3VyIGBwbWAgZGF0YSBmcmFtZSBzaG91bGQgYmUgYXNzaWduZWQgZm9yIHRyYWluaW5nIG9yIHRlc3RpbmcsIGl0IGRvZXMgbm90IGFjdHVhbGx5IHNwbGl0IHRoZSBkYXRhLiAKClRvIGV4dHJhY3QgdGhlIHRlc3RpbmcgYW5kIHRyYWluaW5nIGRhdGEgd2UgY2FuIHVzZSB0aGUgYHRyYWluaW5nKClgIGFuZCBgdGVzdGluZygpYCBmdW5jdGlvbnMgYWxzbyBvZiB0aGUgYHJzYW1wbGVgIHBhY2thZ2UuCgojIyMjIHsuc2Nyb2xsYWJsZSB9CmBgYHtyfQp0cmFpbl9wbSA8LSByc2FtcGxlOjp0cmFpbmluZyhwbV9zcGxpdCkKdGVzdF9wbSA8LSByc2FtcGxlOjp0ZXN0aW5nKHBtX3NwbGl0KQogCiMgU2Nyb2xsIHRocm91Z2ggdGhlIG91dHB1dCEKY291bnQodHJhaW5fcG0sIHN0YXRlKQpjb3VudCh0ZXN0X3BtLCBzdGF0ZSkKYGBgCiMjIyMKCgoKIyMgKipQcmVwYXJpbmcgZm9yIHByZS1wcm9jZXNzaW5nIHRoZSBkYXRhKioKKioqCgpBZnRlciBzcGxpdHRpbmcgdGhlIGRhdGEsIHRoZSBuZXh0IHN0ZXAgaXMgdG8gcHJvY2VzcyB0aGUgdHJhaW5pbmcgYW5kIHRlc3RpbmcgZGF0YSBzbyB0aGF0IHRoZSBkYXRhIGFyZSBhcmUgY29tcGF0aWJsZSBhbmQgb3B0aW1pemVkIHRvIGJlIHVzZWQgd2l0aCB0aGUgbW9kZWwuIAoKSW4gb3JkZXIgdG8gc3RhcnQgdGhpcywgd2UgbmVlZCB0byB0aGluayBhYm91dCB3aGF0IGVhY2ggb2YgdGhlIGFzcGVjdHMgb2Ygb3VyIGRhdGEgbWlnaHQgZG8gaW4gdGhlIG1vZGVsLiBGb3IgZXhhbXBsZSwgaXMgdGhpcyBwYXJ0aWN1bGFyIGNvbHVtbiBvZiBkYXRhIHdoYXQgd2Ugd291bGQgY29uc2lkZXIgdGhlIG91dGNvbWUgb2YgaW50ZXJlc3Q/IE9yIGlzIHRoZXNlIGRhdGEgdmFsdWVzIHBvc3NpYmx5IGhlbHBmdWwgZm9yIHByZWRpY3RpbmcgdGhlIG91dGNvbWU/IFRoaXMgcHJvY2VzcyBpcyBkZXNjcmliZWQgYXMgYXNzaWduaW5nIHZhcmlhYmxlcyB0byBzcGVjaWZpYyByb2xlcyB3aXRoaW4gdGhlIG1vZGVsLgoKV2Ugd2lsbCB0aGVuIGRvIHdoYXQgaXMgY2FsbGVkIHByZS1wcm9jZXNzaW5nIHRvIHByZXBhcmUgdGhlIGRhdGEgc28gaXQgaXMgcmVhZHkuIFRoaXMgaW52b2x2ZXMgdGhpbmdzIGxpa2UgbGlrZSBzY2FsaW5nIHZhcmlhYmxlcyBhbmQgcmVtb3ZpbmcgcmVkdW5kYW50IHZhcmlhYmxlcy4gCgpUaGlzIHByb2Nlc3MgaXMgYWxzbyBjYWxsZWQgZmVhdHVyZSBlbmdpbmVlcmluZy4KClRvIGRvIHRoaXMgaW4gYHRpZHltb2RlbHNgLCB3ZSB3aWxsIGNyZWF0ZSB3aGF0J3MgY2FsbGVkIGEgInJlY2lwZSIgdXNpbmcgdGhlIGByZWNpcGVzYCBwYWNrYWdlLCB3aGljaCBpcyBhIHN0YW5kYXJkaXplZCBmb3JtYXQgZm9yIGEgc2VxdWVuY2Ugb2Ygc3RlcHMgZm9yIHByZS1wcm9jZXNzaW5nIHRoZSBkYXRhLgpUaGlzIGNhbiBiZSB2ZXJ5IHVzZWZ1bCBiZWNhdXNlIGl0IG1ha2VzIHRlc3Rpbmcgb3V0IGRpZmZlcmVudCBwcmUtcHJvY2Vzc2luZyBzdGVwcyBvciBkaWZmZXJlbnQgYWxnb3JpdGhtcyB3aXRoIHRoZSBzYW1lIHByZS1wcm9jZXNzaW5nIHZlcnkgZWFzeSBhbmQgcmVwcm9kdWNpYmxlLgpDcmVhdGluZyBhIHJlY2lwZSBzcGVjaWZpZXMgKipob3cgYSBkYXRhIGZyYW1lIG9mIHByZWRpY3RvcnMgc2hvdWxkIGJlIGNyZWF0ZWQqKiAtIGl0IHNwZWNpZmllcyB3aGF0IHZhcmlhYmxlcyB0byBiZSB1c2VkIGFuZCB0aGUgcHJlLXByb2Nlc3Npbmcgc3RlcHMsIGJ1dCBpdCAqKmRvZXMgbm90IGV4ZWN1dGUgdGhlc2Ugc3RlcHMqKiBvciBjcmVhdGUgdGhlIGRhdGEgZnJhbWUgb2YgcHJlZGljdG9ycy4KCiMjIyBTdGVwIDE6IFNwZWNpZnkgdmFyaWFibGVzIHJvbGVzIHdpdGggYHJlY2lwZSgpYCBmdW5jdGlvbgoKVGhlIGZpcnN0IHRoaW5nIHRvIGRvIHRvIGNyZWF0ZSBhIHJlY2lwZSBpcyB0byBzcGVjaWZ5IHdoaWNoIHZhcmlhYmxlcyB3ZSB3aWxsIGJlIHVzaW5nIGFzIG91ciBvdXRjb21lIGFuZCBwcmVkaWN0b3JzIHVzaW5nIHRoZSBgcmVjaXBlKClgIGZ1bmN0aW9uLiAKSW4gdGVybXMgb2YgdGhlIG1ldGFwaG9yIG9mIGJha2luZywgd2UgY2FuIHRoaW5rIG9mIHRoaXMgYXMgbGlzdGluZyBvdXIgaW5ncmVkaWVudHMuIApUcmFuc2xhdGluZyB0aGlzIHRvIHRoZSBgcmVjaXBlc2AgcGFja2FnZSwgd2UgdXNlIHRoZSBgcmVjaXBlKClgIGZ1bmN0aW9uIHRvIGFzc2lnbiByb2xlcyB0byBhbGwgdGhlIHZhcmlhYmxlcy4gCgpMZXQncyB0cnkgdGhlIHNpbXBsZXN0IHJlY2lwZSB3aXRoIG5vIHByZS1wcm9jZXNzaW5nIHN0ZXBzOiBzaW1wbHkgbGlzdCB0aGUgb3V0Y29tZSBhbmQgcHJlZGljdG9yIHZhcmlhYmxlcy4KCldlIGNhbiBkbyBzbyBpbiB0d28gd2F5czogIAoKMSkgVXNpbmcgZm9ybXVsYSBub3RhdGlvbiAgCjIpIEFzc2lnbmluZyByb2xlcyB0byBlYWNoIHZhcmlhYmxlICAKCkxldCdzIGxvb2sgYXQgdGhlIGZpcnN0IHdheSB1c2luZyBmb3JtdWxhIG5vdGF0aW9uLCB3aGljaCBsb29rcyBsaWtlIHRoaXM6ICAKCm91dGNvbWUocykgfiBwcmVkaWN0b3IocykgIAoKSWYgaW4gdGhlIGNhc2Ugb2YgbXVsdGlwbGUgcHJlZGljdG9ycyBvciBhIG11bHRpdmFyaWF0ZSBzaXR1YXRpb24gd2l0aCB0d28gb3V0Y29tZXMsIHVzZSBhIHBsdXMgc2lnbjogIAoKb3V0Y29tZTEgKyBvdXRjb21lMiB+IHByZWRpY3RvcjEgKyBwcmVkaWN0b3IyICAKCklmIHdlIHdhbnQgdG8gaW5jbHVkZSBhbGwgcHJlZGljdG9ycyB3ZSBjYW4gdXNlIGEgcGVyaW9kIGxpa2Ugc286ICAKCm91dGNvbWVfdmFyaWFibGVfbmFtZSB+IC4gIAoKTm93IHdpdGggb3VyIGRhdGEsIHdlIHdpbGwgc3RhcnQgYnkgbWFraW5nIGEgcmVjaXBlIGZvciBvdXIgdHJhaW5pbmcgZGF0YS4KSWYgeW91IHJlY2FsbCwgdGhlIGNvbnRpbnVvdXMgb3V0Y29tZSB2YXJpYWJsZSBpcyBgdmFsdWVgICh0aGUgYXZlcmFnZSBhbm51YWwgZ3JhdmltZXRyaWMgbW9uaXRvciBQTX4yLjV+IGNvbmNlbnRyYXRpb24gaW4gdWcvbV4zXikuIApPdXIgZmVhdHVyZXMgKG9yIHByZWRpY3RvciB2YXJpYWJsZXMpIGFyZSBhbGwgdGhlIG90aGVyIHZhcmlhYmxlcyBleGNlcHQgdGhlIG1vbml0b3IgSUQsIHdoaWNoIGlzIGFuIGBpZGAgdmFyaWFibGUuCgpUaGUgcmVhc29uIG5vdCB0byBpbmNsdWRlIHRoZSBgaWRgIHZhcmlhYmxlIGlzIGJlY2F1c2UgdGhpcyB2YXJpYWJsZSBpbmNsdWRlcyB0aGUgY291bnR5IG51bWJlciBhbmQgYSBudW1iZXIgZGVzaWduYXRpbmcgd2hpY2ggcGFydGljdWxhciBtb25pdG9yIHRoZSB2YWx1ZXMgY2FtZSBmcm9tIChvZiB0aGUgbW9uaXRvcnMgdGhlcmUgYXJlIGluIHRoYXQgY291bnR5KS4gClNpbmNlIHRoaXMgbnVtYmVyIGlzIGFyYml0cmFyeSBhbmQgdGhlIGNvdW50eSBpbmZvcm1hdGlvbiBpcyBhbHNvIGdpdmVuIGluIHRoZSBkYXRhLCBhbmQgdGhlIGZhY3QgdGhhdCBlYWNoIG1vbml0b3Igb25seSBoYXMgb25lIHZhbHVlIGluIHRoZSBgdmFsdWVgIHZhcmlhYmxlLCBub3RoaW5nIGlzIGdhaW5lZCBieSBpbmNsdWRpbmcgdGhpcyB2YXJpYWJsZSBhbmQgaXQgbWF5IGluc3RlYWQgaW50cm9kdWNlIG5vaXNlLiAKSG93ZXZlciwgaXQgaXMgdXNlZnVsIHRvIGtlZXAgdGhpcyBkYXRhIHRvIHRha2UgYSBsb29rIGF0IHdoYXQgaXMgaGFwcGVuaW5nIGxhdGVyLiAKV2Ugd2lsbCBzaG93IHlvdSB3aGF0IHRvIGRvIGluIHRoaXMgY2FzZSBpbiBqdXN0IGEgYml0LgoKVG8gc3VtbWFyaXplIHRoaXMgc3RlcCwgd2Ugd2lsbCB1c2UgdGhlIGByZWNpcGUoKWAgZnVuY3Rpb24gdG8gYXNzaWduIHJvbGVzIHRvIGFsbCB0aGUgdmFyaWFibGVzOiAKCmBgYHtyLCBlY2hvPUZBTFNFLCBvdXQud2lkdGg9IjQwMHB4In0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwiU3RhcnRpbmdfYV9yZWNpcGVfcmVjaXBlczEucG5nIikpCmBgYAoKV2Ugd2lsbCBkZXNjcmliZSB0aGlzIHN0ZXAgYnkgc3RlcCBhbmQgdGhlbiBzaG93IGFsbCB0aGUgc3RlcHMgdG9nZXRoZXIuCgpJbiB0aGUgc2ltcGxlc3QgY2FzZSwgd2UgbWlnaHQgdXNlIGFsbCBwcmVkaWN0b3JzIGxpa2UgdGhpczoKCmBgYHtyfQpzaW1wbGVfcmVjIDwtIHRyYWluX3BtICU+JQogIHJlY2lwZXM6OnJlY2lwZSh2YWx1ZSB+IC4pCgpzaW1wbGVfcmVjCmBgYAoKV2Ugc2VlIGEgcmVjaXBlIGhhcyBiZWVuIGNyZWF0ZWQgd2l0aCAxIG91dGNvbWUgdmFyaWFibGUgYW5kIDQ5IHByZWRpY3RvciB2YXJpYWJsZXMgKG9yIGZlYXR1cmVzKS4gCkFsc28sIG5vdGljZSBob3cgd2UgbmFtZWQgdGhlIG91dHB1dCBvZiBgcmVjaXBlKClgLiAKVGhlIG5hbWluZyBjb252ZW50aW9uIGZvciByZWNpcGUgb2JqZWN0cyBpcyBgKl9yZWNgIG9yIGByZWNgLiAKCk5vdywgbGV0J3MgZ2V0IGJhY2sgdG8gdGhlIGBpZGAgdmFyaWFibGUuIApJbnN0ZWFkIG9mIGluY2x1ZGluZyBpdCBhcyBhIHByZWRpY3RvciB2YXJpYWJsZSwgd2UgY291bGQgYWxzbyB1c2UgdGhlIGB1cGRhdGVfcm9sZSgpYCBmdW5jdGlvbiBvZiB0aGUgYHJlY2lwZXNgIHBhY2thZ2UuCgpgYGB7cn0Kc2ltcGxlX3JlYyA8LSB0cmFpbl9wbSAlPiUKICByZWNpcGVzOjpyZWNpcGUodmFsdWUgfiAuKSAlPiUKICByZWNpcGVzOjp1cGRhdGVfcm9sZShpZCwgbmV3X3JvbGUgPSAiaWQgdmFyaWFibGUiKQoKc2ltcGxlX3JlYwpgYGAKCiMjIyMgey5jbGlja190b19leHBhbmRfYmxvY2t9Cgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byBsZWFybiBtb3JlIGFib3V0IHRoZSB3b3JraW5nIHdpdGggYGlkYCB2YXJpYWJsZXMgPC9zdW1tYXJ5PgoKVGhpcyBvcHRpb24gd29ya3Mgd2VsbCB3aXRoIHRoZSBuZXdlciBgd29ya2Zsb3dzYCBwYWNrYWdlLCBob3dldmVyIGBpZGAgdmFyaWFibGVzIGFyZSBvZnRlbiBkcm9wcGVkIGZyb20gYW5hbHlzZXMgdGhhdCBkbyBub3QgdXNlIHRoaXMgbmV3ZXIgcGFja2FnZSBhcyB0aGV5IGNhbiBtYWtlIHRoZSBwcm9jZXNzIGRpZmZpY3VsdCB3aXRoIHVzaW5nIHRoZSBgcGFyc25pcGAgcGFja2FnZSBhbG9uZSBkdWUgdG8gdGhlIGZhY3QgdGhhdCBuZXcgbGV2ZWxzIChvciBwb3NzaWJsZSB2YWx1ZXMpIG1heSBiZSBpbnRyb2R1Y2VkIHdpdGggdGhlIHRlc3RpbmcgZGF0YS4KCjwvZGV0YWlscz4KCiMjIyMKCldlIGNvdWxkIGFsc28gc3BlY2lmeSB0aGUgb3V0Y29tZSBhbmQgcHJlZGljdG9ycyBpbiB0aGUgc2FtZSB3YXkgYXMgd2UganVzdCBzcGVjaWZpZWQgdGhlIGlkIHZhcmlhYmxlLiAKUGxlYXNlIHNlZSBbaGVyZV0oaHR0cHM6Ly90aWR5bW9kZWxzLmdpdGh1Yi5pby9yZWNpcGVzL3JlZmVyZW5jZS9yZWNpcGUuaHRtbCl7dGFyZ2V0PSJfYmxhbmsifSBmb3IgZXhhbXBsZXMgb2Ygb3RoZXIgcm9sZXMgZm9yIHZhcmlhYmxlcy4gClRoZSByb2xlIGNhbiBiZSBhY3R1YWxseSBiZSBhbnkgdmFsdWUuIAoKVGhlIG9yZGVyIGlzIGltcG9ydGFudCBoZXJlLCBhcyB3ZSBmaXJzdCBtYWtlIGFsbCB2YXJpYWJsZXMgcHJlZGljdG9ycyBhbmQgdGhlbiBvdmVycmlkZSB0aGlzIHJvbGUgZm9yIHRoZSBvdXRjb21lIGFuZCBgaWRgIHZhcmlhYmxlLiAKV2Ugd2lsbCB1c2UgdGhlIGBldmVyeXRoaW5nKClgIGZ1bmN0aW9uIG9mIHRoZSBgZHBseXJgIHBhY2thZ2UgdG8gc3RhcnQgd2l0aCBhbGwgb2YgdGhlIHZhcmlhYmxlcyBpbiBgdHJhaW5fcG1gLgoKYGBge3J9CnNpbXBsZV9yZWMgPC0gcmVjaXBlKHRyYWluX3BtKSAlPiUKICAgIHVwZGF0ZV9yb2xlKGV2ZXJ5dGhpbmcoKSwgbmV3X3JvbGUgPSAicHJlZGljdG9yIikgJT4lCiAgICB1cGRhdGVfcm9sZSh2YWx1ZSwgbmV3X3JvbGUgPSAib3V0Y29tZSIpICU+JQogICAgdXBkYXRlX3JvbGUoaWQsIG5ld19yb2xlID0gImlkIHZhcmlhYmxlIikKCnNpbXBsZV9yZWMKYGBgCgpXZSBjYW4gdmlldyBvdXIgcmVjaXBlIGluIG1vcmUgZGV0YWlsIHVzaW5nIHRoZSBiYXNlIGBzdW1tYXJ5KClgIGZ1bmN0aW9uLgoKYGBge3J9CnN1bW1hcnkoc2ltcGxlX3JlYykKYGBgCgoKCgojIyMgU3RlcCAyOiBTcGVjaWZ5IHRoZSBwcmUtcHJvY2Vzc2luZyBzdGVwcyB3aXRoIGBzdGVwKigpYCBmdW5jdGlvbnMKCk5leHQsIHdlIHVzZSB0aGUgYHN0ZXAqKClgIGZ1bmN0aW9ucyBmcm9tIHRoZSBgcmVjaXBlYCBwYWNrYWdlIHRvIHNwZWNpZnkgcHJlLXByb2Nlc3Npbmcgc3RlcHMuIAoKYGBge3IsIGVjaG89RkFMU0UsIG91dC53aWR0aD0iNDAwcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCJNYWtpbmdfYV9yZWNpcGVfcmVjaXBlczIucG5nIikpCmBgYAoKKipUaGlzIFtsaW5rXShodHRwczovL3RpZHltb2RlbHMuZ2l0aHViLmlvL3JlY2lwZXMvcmVmZXJlbmNlL2luZGV4Lmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gYW5kIHRoaXMgW2xpbmtdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9yZWNpcGVzL3JlY2lwZXMucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IHNob3cgdGhlIG1hbnkgb3B0aW9ucyBmb3IgcmVjaXBlIHN0ZXAgZnVuY3Rpb25zLioqCgo8dT5UaGVyZSBhcmUgc3RlcCBmdW5jdGlvbnMgZm9yIGEgdmFyaWV0eSBvZiBwdXJwb3Nlczo8L3U+CgoxLiBbKipJbXB1dGF0aW9uKipdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0ltcHV0YXRpb25fKHN0YXRpc3RpY3MpKXt0YXJnZXQ9Il9ibGFuayJ9IC0tIGZpbGxpbmcgaW4gbWlzc2luZyB2YWx1ZXMgYmFzZWQgb24gdGhlIGV4aXN0aW5nIGRhdGEgCjIuIFsqKlRyYW5zZm9ybWF0aW9uKipdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0RhdGFfdHJhbnNmb3JtYXRpb25fKHN0YXRpc3RpY3MpKXt0YXJnZXQ9Il9ibGFuayJ9IC0tIGNoYW5naW5nIGFsbCB2YWx1ZXMgb2YgYSB2YXJpYWJsZSBpbiB0aGUgc2FtZSB3YXksIHR5cGljYWxseSB0byBtYWtlIGl0IG1vcmUgbm9ybWFsIG9yIGVhc2llciB0byBpbnRlcnByZXQKMy4gWyoqRGlzY3JldGl6YXRpb24qKl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvRGlzY3JldGl6YXRpb25fb2ZfY29udGludW91c19mZWF0dXJlcyl7dGFyZ2V0PSJfYmxhbmsifSAtLSBjb252ZXJ0aW5nIGNvbnRpbnVvdXMgdmFsdWVzIGludG8gZGlzY3JldGUgb3Igbm9taW5hbCB2YWx1ZXMgLSBiaW5uaW5nIGZvciBleGFtcGxlIHRvIHJlZHVjZSB0aGUgbnVtYmVyIG9mIHBvc3NpYmxlIGxldmVscyAoSG93ZXZlciB0aGlzIGlzIGdlbmVyYWxseSBub3QgYWR2aXNhYmxlISkKNC4gWyoqRW5jb2RpbmcgLyBDcmVhdGluZyBEdW1teSBWYXJpYWJsZXMqKl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvRHVtbXlfdmFyaWFibGVfKHN0YXRpc3RpY3MpKXt0YXJnZXQ9Il9ibGFuayJ9IC0tIGNyZWF0aW5nIGEgbnVtZXJpYyBjb2RlIGZvciBjYXRlZ29yaWNhbCB2YXJpYWJsZXMKKFsqKk1vcmUgb24gb25lLWhvdCBhbmQgRHVtbXkgVmFyaWFibGVzIGVuY29kaW5nKipdKGh0dHBzOi8vbWVkaXVtLmNvbS9wL2I1ODQwYmUzYzQxYS9yZXNwb25zZXMvc2hvdyl7dGFyZ2V0PSJfYmxhbmsifSkKNS4gWyoqRGF0YSB0eXBlIGNvbnZlcnNpb25zKipdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9oYWJsYXIvdmlnbmV0dGVzL2NvbnZlcnQuaHRtbCl7dGFyZ2V0PSJfYmxhbmsifSAgLS0gd2hpY2ggbWVhbnMgY2hhbmdpbmcgZnJvbSBpbnRlZ2VyIHRvIGZhY3RvciBvciBudW1lcmljIHRvIGRhdGUgZXRjLgo2LiBbKipJbnRlcmFjdGlvbioqXShodHRwczovL3N0YXRpc3RpY3NieWppbS5jb20vcmVncmVzc2lvbi9pbnRlcmFjdGlvbi1lZmZlY3RzLyl7dGFyZ2V0PSJfYmxhbmsifSAgdGVybSBhZGRpdGlvbiB0byB0aGUgbW9kZWwgLS0gd2hpY2ggbWVhbnMgdGhhdCB3ZSB3b3VsZCBiZSBtb2RlbGluZyBmb3IgcHJlZGljdG9ycyB0aGF0IHdvdWxkIGluZmx1ZW5jZSB0aGUgY2FwYWNpdHkgb2YgZWFjaCBvdGhlciB0byBwcmVkaWN0IHRoZSBvdXRjb21lCjcuIFsqKk5vcm1hbGl6YXRpb24qKl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvTm9ybWFsaXphdGlvbl8oc3RhdGlzdGljcykpe3RhcmdldD0iX2JsYW5rIn0gLS0gY2VudGVyaW5nIGFuZCBzY2FsaW5nIHRoZSBkYXRhIHRvIGEgc2ltaWxhciByYW5nZSBvZiB2YWx1ZXMKOC4gWyoqRGltZW5zaW9uYWxpdHkgUmVkdWN0aW9uLyBTaWduYWwgRXh0cmFjdGlvbioqXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9EaW1lbnNpb25hbGl0eV9yZWR1Y3Rpb24pe3RhcmdldD0iX2JsYW5rIn0gLS0gcmVkdWNpbmcgdGhlIHNwYWNlIG9mIGZlYXR1cmVzIG9yIHByZWRpY3RvcnMgdG8gYSBzbWFsbGVyIHNldCBvZiB2YXJpYWJsZXMgdGhhdCBjYXB0dXJlIHRoZSB2YXJpYXRpb24gb3Igc2lnbmFsIGluIHRoZSBvcmlnaW5hbCB2YXJpYWJsZXMgKGV4LiBQcmluY2lwYWwgQ29tcG9uZW50IEFuYWx5c2lzIGFuZCBJbmRlcGVuZGVudCBDb21wb25lbnQgQW5hbHlzaXMpCjkuICoqRmlsdGVyaW5nKiogLS0gZmlsdGVyaW5nIG9wdGlvbnMgZm9yIHJlbW92aW5nIHZhcmlhYmxlcyAoZXguIHJlbW92ZSB2YXJpYWJsZXMgdGhhdCBhcmUgaGlnaGx5IGNvcnJlbGF0ZWQgdG8gb3RoZXJzIG9yIHJlbW92ZSB2YXJpYWJsZXMgd2l0aCB2ZXJ5IGxpdHRsZSB2YXJpYW5jZSBhbmQgdGhlcmVmb3JlIGxpa2VseSBsaXR0bGUgcHJlZGljdGl2ZSBjYXBhY2l0eSkKMTAuIFsqKlJvdyBvcGVyYXRpb25zKipdKGh0dHBzOi8vdGFydGFydXMub3JnL2dhcmV0aC9tYXRocy9MaW5lYXJfQWxnZWJyYS9yb3dfb3BlcmF0aW9ucy5wZGYpe3RhcmdldD0iX2JsYW5rIn0gLS0gcGVyZm9ybWluZyBmdW5jdGlvbnMgb24gdGhlIHZhbHVlcyB3aXRoaW4gdGhlIHJvd3MgIChleC4gcmVhcnJhbmdpbmcsIGZpbHRlcmluZywgaW1wdXRpbmcpCjExLiAqKkNoZWNraW5nIGZ1bmN0aW9ucyoqIC0tIEd1dCBjaGVja3MgdG8gbG9vayBmb3IgbWlzc2luZyB2YWx1ZXMsIHRvIGxvb2sgYXQgdGhlIHZhcmlhYmxlIGNsYXNzZXMgZXRjLgoKQWxsIG9mIHRoZSBzdGVwIGZ1bmN0aW9ucyBsb29rIGxpa2UgYHN0ZXBfKigpYCB3aXRoIHRoZSBgKmAgcmVwbGFjZWQgd2l0aCBhIG5hbWUsIGV4Y2VwdCBmb3IgdGhlIGNoZWNrIGZ1bmN0aW9ucyB3aGljaCBsb29rIGxpa2UgYGNoZWNrXyooKWAuCgpUaGVyZSBhcmUgc2V2ZXJhbCB3YXlzIHRvIHNlbGVjdCB3aGF0IHZhcmlhYmxlcyB0byBhcHBseSBzdGVwcyB0bzogIAoKMS4gVXNpbmcgYHRpZHlzZWxlY3RgIG1ldGhvZHM6IGBjb250YWlucygpYCwgYG1hdGNoZXMoKWAsIGBzdGFydHNfd2l0aCgpYCwgYGVuZHNfd2l0aCgpYCwgYGV2ZXJ5dGhpbmcoKWAsIGBudW1fcmFuZ2UoKWAgIAoyLiBVc2luZyB0aGUgdHlwZTogYGFsbF9ub21pbmFsKClgLCBgYWxsX251bWVyaWMoKWAgLCBgaGFzX3R5cGUoKWAgCjMuIFVzaW5nIHRoZSByb2xlOiBgYWxsX3ByZWRpY3RvcnMoKWAsIGBhbGxfb3V0Y29tZXMoKWAsIGBoYXNfcm9sZSgpYAo0LiBVc2luZyB0aGUgbmFtZSAtIHVzZSB0aGUgYWN0dWFsIG5hbWUgb2YgdGhlIHZhcmlhYmxlL3ZhcmlhYmxlcyBvZiBpbnRlcmVzdCAgCgpMZXQncyB0cnkgYWRkaW5nIHNvbWUgc3RlcHMgdG8gb3VyIHJlY2lwZS4KCgpXZSBtaWdodCB3YW50IHRvIHBvdGVudGlhbGx5IG1vZGlmeSBzb21lIG9mIG91ciBjYXRlZ29yaWNhbCB2YXJpYWJsZXMgc28gdGhhdCB0aGV5IGNhbiBiZSB1c2VkIHdpdGggY2VydGFpbiBhbGdvcml0aG1zLCBsaWtlIHJlZ3Jlc3Npb24sIHRoYXQgcmVxdWlyZSBvbmx5IG51bWVyaWMgdmFsdWVzLiAKCldlIGNhbiBkbyB0aGlzIHdpdGggdGhlIGBzdGVwX2R1bW15KClgIGZ1bmN0aW9uIGFuZCB0aGUgYG9uZV9ob3QgPSBUUlVFYCBhcmd1bWVudCB0byB1c2UgYSBtZXRob2QgY2FsbGVkIFtPbmUtaG90IGVuY29kaW5nXShodHRwczovL21hY2hpbmVsZWFybmluZ21hc3RlcnkuY29tL3doeS1vbmUtaG90LWVuY29kZS1kYXRhLWluLW1hY2hpbmUtbGVhcm5pbmcvKXt0YXJnZXQ9Il9ibGFuayJ9LgoKCk9uZS1ob3QgZW5jb2RpbmcgbWVhbnMgdGhhdCB3ZSBkbyBub3Qgc2ltcGx5IGVuY29kZSBvdXIgY2F0ZWdvcmljYWwgdmFyaWFibGVzIG51bWVyaWNhbGx5IGFzIGEgc2ltcGxlIDEsMiwzLCBhcyBvdXIgbnVtZXJpYyBhc3NpZ25tZW50cyBjYW4gYmUgaW50ZXJwcmV0ZWQgYnkgYWxnb3JpdGhtcyBhcyBoYXZpbmcgYSBwYXJ0aWN1bGFyIHJhbmsgb3Igb3JkZXIuIApJbnN0ZWFkLCBuZXcgYmluYXJ5IHZhcmlhYmxlcyBtYWRlIHVwIG9mIDFzIGFuZCAwcyBhcmUgdXNlZCB0byBhcmJpdHJhcmlseSBhc3NpZ24gYSBudW1lcmljIHZhbHVlIHRoYXQgaGFzIG5vIGFwcGFyZW50IG9yZGVyLiBOb3RlIHRoYXQgd2hpbGUgdGhlcmUgaXMgYSBkaWZmZXJlbnQgYnV0IHNpbWlsYXIgbWV0aG9kIHRvIGRvIHRoaXMgcmVmZXJyZWQgdG8gYXMgZHVtbXkgdmFyaWFibGVzLCBvbmUtaG90IGVuY29kZWQgdmFyaWFibGVzIGFyZSBhbHNvIHNvbWV0aW1lcyBjYWxsZWQgZHVtbXkgdmFyaWFibGVzLgoKIyMjIyB7LmNsaWNrX3RvX2V4cGFuZF9ibG9ja30KCjxkZXRhaWxzPjxzdW1tYXJ5PkZvciBtb3JlIGluZm9ybWF0aW9uIGFib3V0IHdoYXQgb25lLWhvdCBlbmNvZGluZyBpcywgeW91IGNhbiBleHBhbmQgaGVyZS4gPC9zdW1tYXJ5PgoKRm9yIGV4YW1wbGUsIHNheSB3ZSBvbmx5IGhhZCB0aHJlZSBjaXR5IHZhbHVlcyBmb3IgdGhlIGNpdHkgdmFyaWFibGUgdGhhdCB3ZXJlICJUdXNjb24iLCAiRGVudmVyIiwgYW5kICJOZXcgWW9yayIuCgp8IENpdHkgCXwKfDotLS06CXwKfCBUdXNjb24gCXwKfCBOZXcgWW9yayAJfAp8IFR1c2NvbiAJfAp8IERlbnZlciAJfAoKVGhpcyB3b3VsZCBiZSByZXBsYWNlZCBieSB0aHJlZSBuZXcgdmFyaWFibGVzIG9uZSBmb3IgaWYgdGhlIHZhbHVlIHdhcyAiVHVzY29uIiwgb25lIGZvciBpZiB0aGUgdmFsdWUgd2FzICJEZW52ZXIiLCBhbmQgb25lIGZvciBpZiB0aGUgdmFsdWUgd2FzICJOZXcgWW9yayIuIEVhY2ggd291bGQgYmUgbWFkZSB1cCBvZiB6ZXJvcyBhbmQgb25lcy4gT25lIGZvciB0aGUgbmV3IFR1c2NvbiB2YXJpYWJsZSB3b3VsZCBpbmRpY2F0ZSB0aGF0IGluZGVlZCAiVHVzY29uIiB3YXMgdGhlIHZhbHVlIGFuZCB6ZXJvIHdvdWxkIGluZGljYXRlIHRoYXQgaXQgd2FzIGluc3RlYWQgIk5ldyBZb3JrIiBvciAiRGVudmVyIi4gRXNzZW50aWFsbHkgdGhlcmUgaXMgb25lICJob3QiIHZhbHVlIHBvc3NpYmxlIGZvciBlYWNoIG5ldyB2YXJpYWJsZSwgd2hlcmUgdGhlIHZhbHVlIHdpbGwgYmUgb25lLgoKCnwgVHVzY29uIAl8IERlbnZlciAJfCBOZXcgWW9yayAJfAp8Oi0tLToJfDotLS06CXw6LS0tOgl8CnwgMSAJfCAwIAl8IDAgCXwKfCAwIAl8IDAgCXwgMSAJfAp8IDEgCXwgMCAJfCAwIAl8CnwgMCAJfCAxIAl8IDAgCXwKCjwvZGV0YWlscz4KCiMjIyMKCkhlcmUgd2Ugd2lsbCBjcmVhdGUgc3VjaCB2YXJpYWJsZXMgdG8gcmVwbGFjZSB0aGUgY3VycmVudCwgc3RhdGUsIGNvdW50cnksIGFuZCBjaXR5IGNhdGVnb3JpY2FsIGRhdGEuIFNpbWlsYXJseSB0aGUgWkNUQSB2YWx1ZXMgKHdoaWNoIGFyZSBjdXJyZW50bHkgYSBmYWN0b3IgY2xhc3MpIGFyZSBub3QgaW50ZW5kZWQgdG8gYmUgaW50ZXJwcmV0ZWQgYXMgbnVtZXJpYyBhcyB0aGV5IGNvdWxkIGJlIHRob3VnaHQgb2YgbGlrZSBhIHppcCBjb2RlIGFuZCB0aHVzIHdlIHdvdWxkIGFsc28gd2FudCB0byBlbmNvZGUgdGhpcyBhcyB3ZWxsLiAKCmBgYHtyfQpzaW1wbGVfcmVjICU+JQogIHN0ZXBfZHVtbXkoc3RhdGUsIGNvdW50eSwgY2l0eSwgemN0YSwgb25lX2hvdCA9IFRSVUUpCmBgYAoKIyMjIyB7LmNsaWNrX3RvX2V4cGFuZF9ibG9ja30KCjxkZXRhaWxzPjxzdW1tYXJ5PiBDbGljayBoZXJlIHRvIHNlZSBob3cgdGhlIHZhcmlhYmxlcyB3b3VsZCBjaGFuZ2UgaWYgb3VyIHJlY2lwZSBzdG9wcGVkIGhlcmUgd2l0aCBzaW1wbHkgdGhlIG9uZS1ob3QgZW5jb2RpbmcuIDwvc3VtbWFyeT4KClRvIGNyZWF0ZSB0aGUgZGF0YSB3ZSB3aWxsIHVzZSBzdGVwcyB0aGF0IHdlIHdpbGwgZGVtb25zdHJhdGUgbGF0ZXIsIGZvciBub3cgd2Ugd2lsbCBqdXN0IHNob3cgeW91IHdoYXQgdGhlIG5ldyBlbmNvZGVkIHZhcmlhYmxlcyBsb29rIGxpa2UuCgpgYGB7ciwgZWNobyA9IEZBTFNFLCBpbmNsdWRlID0gRkFMU0V9Cm9uZV9ob3RfcmVjIDwtIHJlY2lwZSh0cmFpbl9wbSkgJT4lCiAgdXBkYXRlX3JvbGUoZXZlcnl0aGluZygpLCBuZXdfcm9sZSA9ICJwcmVkaWN0b3IiKSAlPiUKICB1cGRhdGVfcm9sZSh2YWx1ZSwgbmV3X3JvbGUgPSAib3V0Y29tZSIpICU+JQogIHVwZGF0ZV9yb2xlKGlkLCBuZXdfcm9sZSA9ICJpZCB2YXJpYWJsZSIpCm9uZV9ob3RfcmVjICU8PiUgc3RlcF9kdW1teShzdGF0ZSwgY291bnR5LCBjaXR5LCB6Y3RhLCBvbmVfaG90ID0gVFJVRSkKcHJlcHBlZF9vbmVfaG90X3JlYyA8LSBwcmVwKG9uZV9ob3RfcmVjICwgdmVyYm9zZSA9IFRSVUUsIHJldGFpbiA9IFRSVUUgKQpiYWtlZF9vbmVfaG90X3JlYyA8LSBiYWtlKHByZXBwZWRfb25lX2hvdF9yZWMsIG5ld19kYXRhID0gTlVMTCkKYGBgCgoKCgoKYGBge3J9Cmxlbmd0aChuYW1lcyh0cmFpbl9wbSkpCmxlbmd0aChuYW1lcyhiYWtlZF9vbmVfaG90X3JlYykpICMgbWFueSBtb3JlIHZhcmlhYmxlcyEKCnRyYWluX3BtICU+JSBzZWxlY3QoemN0YSwgY2l0eSwgc3RhdGUsIGNvdW50eSkgJT4lIGhlYWQKCiMgbGV0J3MgbG9vayBhdCBhIGZldwpiYWtlZF9vbmVfaG90X3JlYyAlPiUgc2VsZWN0KCJ6Y3RhX1g1NDUyMCIsICJjaXR5X05vdC5pbi5hLmNpdHkiLCAic3RhdGVfSW5kaWFuYSIpICU+JSBoZWFkKCkKCmBgYAoKCjwvZGV0YWlscz4KCiMjIyMKCk91ciBgZmlwc2AgdmFyaWFibGUgaW5jbHVkZXMgYSBudW1lcmljIGNvZGUgZm9yIHN0YXRlIGFuZCBjb3VudHkgLSBhbmQgdGhlcmVmb3JlIGlzIGVzc2VudGlhbGx5IGEgcHJveHkgZm9yIGNvdW50eS4KU2luY2Ugd2UgYWxyZWFkeSBoYXZlIGNvdW50eSwgd2Ugd2lsbCBqdXN0IHVzZSBpdCBhbmQga2VlcCB0aGUgYGZpcHNgIElEIGFzIGFub3RoZXIgSUQgdmFyaWFibGUuCgpXZSBjYW4gcmVtb3ZlIHRoZSBgZmlwc2AgdmFyaWFibGUgZnJvbSB0aGUgcHJlZGljdG9ycyB1c2luZyBgdXBkYXRlX3JvbGUoKWAgdG8gbWFrZSBzdXJlIHRoYXQgdGhlIHJvbGUgaXMgbm8gbG9uZ2VyIGAicHJlZGljdG9yImAuIApXZSBjYW4gbWFrZSB0aGUgcm9sZSBhbnl0aGluZyB3ZSB3YW50IGFjdHVhbGx5LCBzbyB3ZSB3aWxsIGtlZXAgaXQgc29tZXRoaW5nIGlkZW50aWZpYWJsZS4KCmBgYHtyfQpzaW1wbGVfcmVjICU+JQogIHVwZGF0ZV9yb2xlKCJmaXBzIiwgbmV3X3JvbGUgPSAiY291bnR5IGlkIikKYGBgCgpXZSBtaWdodCBhbHNvIHdhbnQgdG8gcmVtb3ZlIHZhcmlhYmxlcyB0aGF0IGFwcGVhciB0byBiZSByZWR1bmRhbnQgYW5kIGFyZSBoaWdobHkgY29ycmVsYXRlZCB3aXRoIG90aGVycywgYXMgd2Uga25vdyBmcm9tIG91ciBleHBsb3JhdG9yeSBkYXRhIGFuYWx5c2lzIHRoYXQgbWFueSBvZiBvdXIgdmFyaWFibGVzIGFyZSBjb3JyZWxhdGVkIHdpdGggb25lIGFub3RoZXIuIApXZSBjYW4gZG8gdGhpcyB1c2luZyB0aGUgYHN0ZXBfY29ycigpYCBmdW5jdGlvbi4KCldlIGRvbid0IHdhbnQgdG8gcmVtb3ZlIHNvbWUgb2Ygb3VyIHZhcmlhYmxlcywgbGlrZSB0aGUgYENNQVFgIGFuZCBgYW9kYCB2YXJpYWJsZXMsIHdlIGNhbiBzcGVjaWZ5IHRoaXMgdXNpbmcgdGhlIGAtYCBzaWduIGJlZm9yZSB0aGUgbmFtZXMgb2YgdGhlc2UgdmFyaWFibGVzIGxpa2Ugc286CgpgYGB7cn0Kc2ltcGxlX3JlYyAlPiUKICBzdGVwX2NvcnIoYWxsX3ByZWRpY3RvcnMoKSwgLSBDTUFRLCAtIGFvZCkKYGBgCgoKSXQgaXMgYWxzbyBhIGdvb2QgaWRlYSB0byByZW1vdmUgdmFyaWFibGVzIHdpdGggbmVhci16ZXJvIHZhcmlhbmNlLCB3aGljaCBjYW4gYmUgZG9uZSB3aXRoIHRoZSBgc3RlcF9uenYoKWAgZnVuY3Rpb24uIAoKVmFyaWFibGVzIGhhdmUgbG93IHZhcmlhbmNlIGlmIGFsbCB0aGUgdmFsdWVzIGFyZSB2ZXJ5IHNpbWlsYXIsIHRoZSB2YWx1ZXMgYXJlIHZlcnkgc3BhcnNlLCBvciBpZiB0aGV5IGFyZSBoaWdobHkgaW1iYWxhbmNlZC4gQWdhaW4gd2UgZG9uJ3Qgd2FudCB0byByZW1vdmUgb3VyIGBDTUFRYCBhbmQgYGFvZGAgdmFyaWFibGVzLgoKYGBge3J9CnNpbXBsZV9yZWMgJT4lCiAgc3RlcF9uenYoYWxsX3ByZWRpY3RvcnMoKSwgLSBDTUFRLCAtIGFvZCkKYGBgCgojIyMjIHsuY2xpY2tfdG9fZXhwYW5kX2Jsb2NrfQoKPGRldGFpbHM+PHN1bW1hcnk+IENsaWNrIGhlcmUgdG8gbGVhcm4gYWJvdXQgZXhhbXBsZXMgd2hlcmUgeW91IG1pZ2h0IGhhdmUgbmVhci16ZXJvIHZhcmlhbmNlIHZhcmlhYmxlczwvc3VtbWFyeT4KCjEpICoqU2ltaWxhciBWYWx1ZXMqKiAtIElmIHRoZSBwb3B1bGF0aW9uIGRlbnNpdHkgd2FzIG5lYXJseSB0aGUgc2FtZSBmb3IgZXZlcnkgemN0YSB0aGF0IGNvbnRhaW5lZCBhIG1vbml0b3IsIHRoZW4ga25vd2luZyB0aGUgcG9wdWxhdGlvbiBkZW5zaXR5IG5lYXIgb3VyIG1vbml0b3Igd291bGQgY29udHJpYnV0ZSBsaXR0bGUgdG8gb3VyIG1vZGVsIGluIGFzc2lzdGluZyB1cyB0byBwcmVkaWN0IG1vbml0b3IgYWlyIHBvbGx1dGlvbiB2YWx1ZXMuIAoyKSAqKlNwYXJzZSBEYXRhKiogLSBJZiBhbGwgb2YgdGhlIG1vbml0b3JzIHdlcmUgaW4gbG9jYXRpb25zIHdoZXJlIHRoZSBwb3B1bGF0aW9ucyBkaWQgbm90IGF0dGVuZCBncmFkdWF0ZSBzY2hvb2wsIHRoZW4gdGhlc2UgdmFsdWVzIHdvdWxkIG1vc3RseSBiZSB6ZXJvLCBhZ2FpbiB0aGlzIHdvdWxkIGRvIHZlcnkgbGl0dGxlIHRvIGhlbHAgdXMgZGlzdGluZ3Vpc2ggb3VyIGFpciBwb2xsdXRpb24gbW9uaXRvcnMuV2hlbiBtYW55IG9mIHRoZSB2YWx1ZXMgYXJlIHplcm8gdGhpcyBpcyBhbHNvIGNhbGxlZCBzcGFyc2UgZGF0YS4gIAozKSAqKkltYmFsYW5jZWQgRGF0YSoqIElmIG5lYXJseSBhbGwgb2YgdGhlIG1vbml0b3JzIHdlcmUgbG9jYXRlZCBpbiBvbmUgcGFydGljdWxhciBzdGF0ZSwgYW5kIGFsbCB0aGUgb3RoZXJzIG9ubHkgaGFkIG9uZSBtb25pdG9yIGVhY2gsIHRoZW4gdGhlIHJlYWwgcHJlZGljdGl2ZSB2YWx1ZSB3b3VsZCBzaW1wbHkgYmUgaW4ga25vd2luZyBpZiBhIG1vbml0b3IgaXMgbG9jYXRlZCBpbiB0aGF0IHBhcnRpY3VsYXIgc3RhdGUgb3Igbm90LiBJbiB0aGlzIGNhc2Ugd2UgZG9uJ3Qgd2FudCB0byByZW1vdmUgb3VyIHZhcmlhYmxlLCB3ZSBqdXN0IHdhbnQgdG8gc2ltcGxpZnkgaXQuCgpTZWUgdGhpcyBbYmxvZyBwb3N0XShodHRwczovL3d3dy5yLWJsb2dnZXJzLmNvbS9uZWFyLXplcm8tdmFyaWFuY2UtcHJlZGljdG9ycy1zaG91bGQtd2UtcmVtb3ZlLXRoZW0vKXt0YXJnZXQ9Il9ibGFuayJ9IGFib3V0IHdoeSByZW1vdmluZyBuZWFyLXplcm8gdmFyaWFuY2UgdmFyaWFibGVzIGlzbid0IGFsd2F5cyBhIGdvb2QgaWRlYSBpZiB3ZSB0aGluayB0aGF0IGEgdmFyaWFibGUgbWlnaHQgYmUgZXNwZWNpYWxseSBpbmZvcm1hdGl2ZS4KCjwvZGV0YWlscz4KCiMjIyMKCkxldCdzIHB1dCBhbGwgdGhpcyB0b2dldGhlciBub3cuIAoKKipSZW1lbWJlcjogaXQgaXMgaW1wb3J0YW50IHRvIGFkZCB0aGUgc3RlcHMgdG8gdGhlIHJlY2lwZSBpbiBhbiBvcmRlciB0aGF0IG1ha2VzIHNlbnNlIGp1c3QgbGlrZSB3aXRoIGEgY29va2luZyByZWNpcGUuKioKCkZpcnN0LCB3ZSBhcmUgZ29pbmcgdG8gY3JlYXRlIG51bWVyaWMgdmFsdWVzIGZvciBvdXIgY2F0ZWdvcmljYWwgdmFyaWFibGVzLCB0aGVuIHdlIHdpbGwgbG9vayBhdCBjb3JyZWxhdGlvbiBhbmQgbmVhci16ZXJvIHZhcmlhbmNlLiAKQWdhaW4sIHdlIGRvIG5vdCB3YW50IHRvIHJlbW92ZSB0aGUgYENNQVFgIGFuZCBgYW9kYCB2YXJpYWJsZXMsIHNvIHdlIGNhbiBtYWtlIHN1cmUgdGhleSBhcmUga2VwdCBpbiB0aGUgbW9kZWwgYnkgZXhjbHVkaW5nIHRoZW0gZnJvbSB0aG9zZSBzdGVwcy4gCklmIHdlIHNwZWNpZmljYWxseSB3YW50ZWQgdG8gcmVtb3ZlIGEgcHJlZGljdG9yIHdlIGNvdWxkIHVzZSBgc3RlcF9ybSgpYC4KCmBgYHtyfQpzaW1wbGVfcmVjICU8PiUKICB1cGRhdGVfcm9sZSgiZmlwcyIsIG5ld19yb2xlID0gImNvdW50eSBpZCIpICU+JQogIHN0ZXBfZHVtbXkoc3RhdGUsIGNvdW50eSwgY2l0eSwgemN0YSwgb25lX2hvdCA9IFRSVUUpICU+JQogIHN0ZXBfY29ycihhbGxfcHJlZGljdG9ycygpLCAtIENNQVEsIC0gYW9kKSU+JQogIHN0ZXBfbnp2KGFsbF9wcmVkaWN0b3JzKCksIC0gQ01BUSwgLSBhb2QpCiAgCnNpbXBsZV9yZWMKYGBgCgoKCiMjICoqUnVubmluZyB0aGUgcHJlLXByb2Nlc3NpbmcqKgoqKioKCiMjIyAqKlN0ZXAgMTogVXBkYXRlIHRoZSByZWNpcGUgd2l0aCB0cmFpbmluZyBkYXRhIHVzaW5nIGBwcmVwKClgKioKKioqCgpUaGUgbmV4dCBtYWpvciBmdW5jdGlvbiBvZiB0aGUgYHJlY2lwZXNgIHBhY2thZ2UgaXMgYHByZXAoKWAuClRoaXMgZnVuY3Rpb24gdXBkYXRlcyB0aGUgcmVjaXBlIG9iamVjdCBiYXNlZCBvbiB0aGUgdHJhaW5pbmcgZGF0YS4gCkl0IGVzdGltYXRlcyBwYXJhbWV0ZXJzIChlc3RpbWF0aW5nIHRoZSByZXF1aXJlZCBxdWFudGl0aWVzIGFuZCBzdGF0aXN0aWNzIHJlcXVpcmVkIGJ5IHRoZSBzdGVwcyBmb3IgdGhlIHZhcmlhYmxlcykgZm9yIHByZS1wcm9jZXNzaW5nIGFuZCB1cGRhdGVzIHRoZSB2YXJpYWJsZXMgcm9sZXMsIGFzIHNvbWUgb2YgdGhlIHByZWRpY3RvcnMgbWF5IGJlIHJlbW92ZWQsIHRoaXMgYWxsb3dzIHRoZSByZWNpcGUgdG8gYmUgcmVhZHkgdG8gdXNlIG9uIG90aGVyIGRhdGEgc2V0cy4gCkl0ICoqZG9lcyBub3QgbmVjZXNzYXJpbHkgYWN0dWFsbHkgZXhlY3V0ZSB0aGUgcHJlLXByb2Nlc3NpbmcgaXRzZWxmKio7IGhvd2V2ZXIsIHdlIHdpbGwgc3BlY2lmeSBpbiBhcmd1bWVudCBmb3IgaXQgdG8gZG8gdGhpcyBzbyB0aGF0IHdlIGNhbiB0YWtlIGEgbG9vayBhdCB0aGUgcHJlLXByb2Nlc3NlZCBkYXRhLgoKClRoZXJlIGFyZSBzb21lIGltcG9ydGFudCBhcmd1bWVudHMgdG8ga25vdyBhYm91dDoKCjEuIGB0cmFpbmluZ2AgLSB5b3UgbXVzdCBzdXBwbHkgYSB0cmFpbmluZyBkYXRhIHNldCB0byBlc3RpbWF0ZSBwYXJhbWV0ZXJzIGZvciBwcmUtcHJvY2Vzc2luZyBvcGVyYXRpb25zIChyZWNpcGUgc3RlcHMpIC0gdGhpcyBtYXkgYWxyZWFkeSBiZSBpbmNsdWRlZCBpbiB5b3VyIHJlY2lwZSAtIGFzIGlzIHRoZSBjYXNlIGZvciB1cwoyLiBgZnJlc2hgIC0gaWYgYGZyZXNoPVRSVUVgLCB3aWxsIHJldHJhaW4gYW5kIGVzdGltYXRlIHBhcmFtZXRlcnMgZm9yIGFueSBwcmV2aW91cyBzdGVwcyB0aGF0IHdlcmUgYWxyZWFkeSBwcmVwcGVkIGlmIHlvdSBhZGQgbW9yZSBzdGVwcyB0byB0aGUgcmVjaXBlIChkZWZhdWx0IGlzIGBGQUxTRWApCjMuIGB2ZXJib3NlYCAtIGlmIGB2ZXJib3NlPVRSVUVgLCBzaG93cyB0aGUgcHJvZ3Jlc3MgYXMgdGhlIHN0ZXBzIGFyZSBldmFsdWF0ZWQgYW5kIHRoZSBzaXplIG9mIHRoZSBwcmUtcHJvY2Vzc2VkIHRyYWluaW5nIHNldCAoZGVmYXVsdCBpcyBgRkFMU0VgKQo0LiBgcmV0YWluYCAtIGlmIGByZXRhaW49VFJVRWAsIHRoZW4gdGhlIHByZS1wcm9jZXNzZWQgdHJhaW5pbmcgc2V0IHdpbGwgYmUgc2F2ZWQgd2l0aGluIHRoZSByZWNpcGUgKGFzIHRlbXBsYXRlKS4gVGhpcyBpcyBnb29kIGlmIHlvdSBhcmUgbGlrZWx5IHRvIGFkZCBtb3JlIHN0ZXBzIGFuZCBkbyBub3Qgd2FudCB0byByZXJ1biB0aGUgYHByZXAoKWAgb24gdGhlIHByZXZpb3VzIHN0ZXBzLiBIb3dldmVyIHRoaXMgY2FuIG1ha2UgdGhlIHJlY2lwZSBzaXplIGxhcmdlLiBUaGlzIGlzIG5lY2Vzc2FyeSBpZiB5b3Ugd2FudCB0byBhY3R1YWxseSBsb29rIGF0IHRoZSBwcmUtcHJvY2Vzc2VkIGRhdGEgKGRlZmF1bHQgaXMgYFRSVUVgKQoKTGV0J3MgdHJ5IG91dCB0aGUgYHByZXAoKWAgZnVuY3Rpb246IAoKYGBge3J9CnByZXBwZWRfcmVjIDwtIHByZXAoc2ltcGxlX3JlYywgdmVyYm9zZSA9IFRSVUUsIHJldGFpbiA9IFRSVUUgKQpuYW1lcyhwcmVwcGVkX3JlYykKYGBgCgpUaGVyZSBhcmUgYWxzbyBsb3RzIG9mIHVzZWZ1bCB0aGluZ3MgdG8gY2hlY2sgb3V0IGluIHRoZSBvdXRwdXQgb2YgYHByZXAoKWAuCllvdSBjYW4gc2VlOgoKMS4gdGhlIGBzdGVwc2AgdGhhdCB3ZXJlIHJ1biAgCjIuIHRoZSBvcmlnaW5hbCB2YXJpYWJsZSBpbmZvIChgdmFyX2luZm9gKSAgCjMuIHRoZSB1cGRhdGVkIHZhcmlhYmxlIGluZm8gYWZ0ZXIgcHJlLXByb2Nlc3NpbmcgKGB0ZXJtX2luZm9gKQo0LiB0aGUgbmV3IGBsZXZlbHNgIG9mIHRoZSB2YXJpYWJsZXMgCjUuIHRoZSBvcmlnaW5hbCBsZXZlbHMgb2YgdGhlIHZhcmlhYmxlcyAoYG9yaWdfbHZsc2ApCjYuIGluZm8gYWJvdXQgdGhlIHRyYWluaW5nIGRhdGEgc2V0IHNpemUgYW5kIGNvbXBsZXRlbmVzcyAoYHRyX2luZm9gKQoKKipOb3RlKio6IFlvdSBtYXkgc2VlIHRoZSBgcHJlcC5yZWNpcGUoKWAgZnVuY3Rpb24gaW4gbWF0ZXJpYWwgdGhhdCB5b3UgcmVhZCBhYm91dCB0aGUgYHJlY2lwZXNgIHBhY2thZ2UuIFRoaXMgaXMgcmVmZXJyaW5nIHRvIHRoZSBgcHJlcCgpYCBmdW5jdGlvbiBvZiB0aGUgYHJlY2lwZXNgIHBhY2thZ2UuCgoKIyMjICoqU3RlcCAyOiBFeHRyYWN0IHByZS1wcm9jZXNzZWQgdHJhaW5pbmcgZGF0YSB1c2luZyBgYmFrZSgpYCoqCioqKgoKClNpbmNlIHdlIHJldGFpbmVkIG91ciBwcmUtcHJvY2Vzc2VkIHRyYWluaW5nIGRhdGEgKGkuZS4gYHByZXAocmV0YWluPVRSVUUpYCksIHdlIGNhbiB0YWtlIGEgbG9vayBhdCBpdCBieSB1c2luZyB0aGUgYGJha2UoKWAgZnVuY3Rpb24gb2YgdGhlIGByZWNpcGVzYCBwYWNrYWdlLiBUaGUgYGJha2VgIGZ1bmN0aW9uIGFsbG93cyB1cyB0byBhcHBseSBvdXIgbW9kZWxpbmcgc3RlcHMgKGluIHRoaXMgY2FzZSBqdXN0IHByZS1wcm9jZXNzaW5nIG9uIHRoZSB0cmFpbmluZyBkYXRhKSBhbmQgc2VlIHdoYXQgaXQgd291bGQgZG8gdGhlIGRhdGEuCgpgYGB7ciwgZWNobz1GQUxTRSwgb3V0LndpZHRoPSI0MDBweCJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsInRyYWluaW5nX3ByZXByb2Nlc3NpbmdfcmVjaXBlczMucG5nIikpCmBgYAoKTGV0J3MgYmFrZSEgCgpTaW5jZSB3ZSBkb24ndCBoYXZlIG5ldyBkYXRhICh3ZSBhcmVuJ3QgbG9va2luZyBhdCB0aGUgdGVzdGluZyBkYXRhKSwgd2UgbmVlZCB0byBzcGVjaWZ5IHRoaXMgd2l0aCBgbmV3X2RhdGEgPSBOVUxMYC4KCiMjIyMgey5zY3JvbGxhYmxlIH0KYGBge3J9CiMgU2Nyb2xsIHRocm91Z2ggdGhlIG91dHB1dCEKYmFrZWRfdHJhaW4gPC0gYmFrZShwcmVwcGVkX3JlYywgbmV3X2RhdGEgPSBOVUxMKQpnbGltcHNlKGJha2VkX3RyYWluKQpgYGAKIyMjIwoKKipOb3RlKiotIHRoaXMgcHJvY2VzcyB1c2VkIHRvIHJlcXVpcmUgdGhlIGBqdWljZSgpYCBmdW5jdGlvbi4KCkZvciBlYXN5IGNvbXBhcmlzb24gc2FrZSAtIGhlcmUgaXMgb3VyIG9yaWdpbmFsIGRhdGE6CgojIyMjIHsuc2Nyb2xsYWJsZSB9CgpgYGB7cn0KIyBTY3JvbGwgdGhyb3VnaCB0aGUgb3V0cHV0IQpnbGltcHNlKHBtKQpgYGAKIyMjIwoKTm90aWNlIGhvdyB3ZSBvbmx5IGhhdmUgMzYgdmFyaWFibGVzIG5vdyBpbnN0ZWFkIG9mIDUwISAKVHdvIG9mIHRoZXNlIGFyZSBvdXIgSUQgdmFyaWFibGVzIChgZmlwc2AgYW5kIHRoZSBhY3R1YWwgbW9uaXRvciBJRCAoYGlkYCkpIGFuZCBvbmUgaXMgb3VyIG91dGNvbWUgKGB2YWx1ZWApLiAKVGh1cyB3ZSBvbmx5IGhhdmUgMzMgcHJlZGljdG9ycyBub3cuIApXZSBjYW4gYWxzbyBzZWUgdGhhdCB3ZSBubyBsb25nZXIgaGF2ZSBhbnkgY2F0ZWdvcmljYWwgdmFyaWFibGVzLiAKVmFyaWFibGVzIGxpa2UgYHN0YXRlYCBhcmUgZ29uZSBhbmQgb25seSBgc3RhdGVfQ2FsaWZvcm5pYWAgcmVtYWlucyBhcyBpdCB3YXMgdGhlIG9ubHkgc3RhdGUgaWRlbnRpdHkgdG8gaGF2ZSBub256ZXJvIHZhcmlhbmNlLgoKCldlIGNhbiBzZWUgdGhhdCBDYWxpZm9ybmlhIGhhZCB0aGUgbGFyZ2VzdCBudW1iZXIgb2YgbW9uaXRvcnMgY29tcGFyZWQgdG8gdGhlIG90aGVyIHN0YXRlcy4KCmBgYHtyLCBldmFsID0gRkFMU0V9CnBtICU+JSBjb3VudChzdGF0ZSkgCmBgYAoKClNjcm9sbCB0aHJvdWdoIHRoZSBvdXRwdXQ6CgojIyMjIHsuc2Nyb2xsYWJsZSB9CgpgYGB7ciwgZWNobyA9IEZBTFNFfQpwbSAlPiUgY291bnQoc3RhdGUpICAlPiUKICBwcmludChuID0gMWUzKQpgYGAKCiMjIyMKCgpXZSBjYW4gYWxzbyBzZWUgdGhhdCB0aGVyZSB3ZXJlIG1vcmUgbW9uaXRvcnMgbGlzdGVkIGFzIGAiTm90IGluIGEgY2l0eSJgIHRoYW4gYW55IGNpdHkuIAoKYGBge3IsIGV2YWwgPSBGQUxTRX0KcG0gJT4lIGNvdW50KGNpdHkpCmBgYAoKU2Nyb2xsIHRocm91Z2ggdGhlIG91dHB1dDoKCiMjIyMgey5zY3JvbGxhYmxlIH0KCmBgYHtyLCBlY2hvPUZBTFNFfQpwbSAlPiUgY291bnQoY2l0eSkgJT4lCiAgcHJpbnQobiA9IDFlMykKYGBgCgojIyMjCgoqKk5vdGUqKjogUmVjYWxsIHRoYXQgeW91IG11c3Qgc3BlY2lmeSBgcmV0YWluID0gVFJVRWAgYXJndW1lbnQgb2YgdGhlIGBwcmVwKClgIGZ1bmN0aW9uIHRvIHVzZSBgYmFrZSgpYCB0byBzZWUgcHJlLXByb2Nlc3NlZCB0cmFpbmluZyBkYXRhLgoKIyMjICoqU3RlcCAzOiBFeHRyYWN0IHByZS1wcm9jZXNzZWQgdGVzdGluZyBkYXRhIHVzaW5nIGBiYWtlKClgKioKKioqCgpBY2NvcmRpbmcgdG8gdGhlIGB0aWR5bW9kZWxzYCBkb2N1bWVudGF0aW9uOgoKPiBgYmFrZSgpYCB0YWtlcyBhIHRyYWluZWQgcmVjaXBlIGFuZCBhcHBsaWVzIHRoZSBvcGVyYXRpb25zIHRvIGEgZGF0YSBzZXQgdG8gY3JlYXRlIGEgZGVzaWduIG1hdHJpeC4KIEZvciBleGFtcGxlOiBpdCBhcHBsaWVzIHRoZSBjZW50ZXJpbmcgdG8gbmV3IGRhdGEgc2V0cyB1c2luZyB0aGVzZSBtZWFucyB1c2VkIHRvIGNyZWF0ZSB0aGUgcmVjaXBlLgoKVGhlcmVmb3JlLCBpZiB5b3Ugd2FudGVkIHRvIGxvb2sgYXQgdGhlIHByZS1wcm9jZXNzZWQgdGVzdGluZyBkYXRhIHlvdSB3b3VsZCB1c2UgdGhlIGBiYWtlKClgIGZ1bmN0aW9uIG9mIHRoZSBgcmVjaXBlc2AgcGFja2FnZS4KKFlvdSBnZW5lcmFsbHkgd2FudCB0byBsZWF2ZSB5b3VyIHRlc3RpbmcgZGF0YSBhbG9uZSwgYnV0IGl0IGlzIGdvb2QgdG8gbG9vayBmb3IgaXNzdWVzIGxpa2UgdGhlIGludHJvZHVjdGlvbiBvZiBOQSB2YWx1ZXMpLgoKYGBge3IsIGVjaG89RkFMU0UsIG91dC53aWR0aD0iNDAwcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCJ0ZXN0aW5nX3ByZXByb2Nlc3NpbmdfcmVjaXBlczQucG5nIikpCmBgYAoKTGV0J3MgYmFrZSEgCgojIyMjIHsuc2Nyb2xsYWJsZSB9CmBgYHtyLH0KIyBTY3JvbGwgdGhyb3VnaCB0aGUgb3V0cHV0IQpiYWtlZF90ZXN0X3BtIDwtIHJlY2lwZXM6OmJha2UocHJlcHBlZF9yZWMsIG5ld19kYXRhID0gdGVzdF9wbSkKZ2xpbXBzZShiYWtlZF90ZXN0X3BtKQpgYGAKIyMjIwoKCk5vdGljZSB0aGF0IG91ciBgY2l0eV9Ob3QuaW4uYS5jaXR5YCB2YXJpYWJsZSBzZWVtcyB0byBiZSBOQSB2YWx1ZXMuIApXaHkgbWlnaHQgdGhhdCBiZT8KCkFoISBQZXJoYXBzIGl0IGlzIGJlY2F1c2Ugc29tZSBvZiBvdXIgbGV2ZWxzIHdlcmUgbm90IHByZXZpb3VzbHkgc2VlbiBpbiB0aGUgdHJhaW5pbmcgc2V0IQoKTGV0J3MgdGFrZSBhIGxvb2sgdXNpbmcgdGhlIFtzZXQgb3BlcmF0aW9uc10oaHR0cHM6Ly93d3cucHJvYmFiaWxpdHljb3Vyc2UuY29tL2NoYXB0ZXIxLzFfMl8yX3NldF9vcGVyYXRpb25zLnBocCl7dGFyZ2V0PSJfYmxhbmsifSBvZiB0aGUgYGRwbHlyYCBwYWNrYWdlLiAKV2UgY2FuIHRha2UgYSBsb29rIGF0IGNpdGllcyB0aGF0IHdlcmUgZGlmZmVyZW50IGJldHdlZW4gdGhlIHRlc3QgYW5kIHRyYWluaW5nIHNldC4KCmBgYHtyfQp0cmFpbmNpdGllcyA8LSB0cmFpbl9wbSAlPiUgZGlzdGluY3QoY2l0eSkKdGVzdGNpdGllcyA8LSB0ZXN0X3BtICU+JSBkaXN0aW5jdChjaXR5KQoKI2dldCB0aGUgbnVtYmVyIG9mIGNpdGllcyB0aGF0IHdlcmUgZGlmZmVyZW50CmRpbShkcGx5cjo6c2V0ZGlmZih0cmFpbmNpdGllcywgdGVzdGNpdGllcykpCgojZ2V0IHRoZSBudW1iZXIgb2YgY2l0aWVzIHRoYXQgb3ZlcmxhcHBlZApkaW0oZHBseXI6OmludGVyc2VjdCh0cmFpbmNpdGllcywgdGVzdGNpdGllcykpCmBgYAoKSW5kZWVkLCB0aGVyZSBhcmUgbG90cyBvZiBkaWZmZXJlbnQgY2l0aWVzIGluIG91ciB0ZXN0IGRhdGEgdGhhdCBhcmUgbm90IGluIG91ciB0cmFpbmluZyBkYXRhIQoKClNvLCBsZXQncyBnbyBiYWNrIHRvIG91ciBgcG1gIGRhdGEgc2V0IGFuZCBtb2RpZnkgdGhlIGBjaXR5YCB2YXJpYWJsZSB0byBqdXN0IGJlIHZhbHVlcyBvZiBgaW4gYSBjaXR5YCBvciBgbm90IGluIGEgY2l0eWAgdXNpbmcgdGhlIGBjYXNlX3doZW4oKWAgZnVuY3Rpb24gb2YgYGRwbHlyYC4KVGhpcyBmdW5jdGlvbiBhbGxvd3MgeW91IHRvIHZlY3Rvcml6ZSBtdWx0aXBsZSBgaWZfZWxzZSgpYCBzdGF0ZW1lbnRzLgoKYGBge3J9CnBtICU+JQogIG11dGF0ZShjaXR5ID0gY2FzZV93aGVuKGNpdHkgPT0gIk5vdCBpbiBhIGNpdHkiIH4gIk5vdCBpbiBhIGNpdHkiLAogICAgICAgICAgICAgICAgICAgICAgICAgIGNpdHkgIT0gIk5vdCBpbiBhIGNpdHkiIH4gIkluIGEgY2l0eSIpKQpgYGAKCkFsdGVybmF0aXZlbHkgeW91IGNvdWxkIGNyZWF0ZSBhIFtjdXN0b20gc3RlcCBmdW5jdGlvbl0oaHR0cHM6Ly9yZWNpcGVzLnRpZHltb2RlbHMub3JnL2FydGljbGVzL0N1c3RvbV9TdGVwcy5odG1sKXt0YXJnZXQ9Il9ibGFuayJ9IHRvIGRvIHRoaXMgYW5kIGFkZCB0aGlzIHRvIHlvdXIgcmVjaXBlLCBidXQgdGhhdCBpcyBiZXlvbmQgdGhlIHNjb3BlIG9mIHRoaXMgY2FzZSBzdHVkeS4gCgpXZSB3aWxsIG5lZWQgdG8gcmVwZWF0IGFsbCB0aGUgc3RlcHMgKHNwbGl0dGluZyB0aGUgZGF0YSwgcHJlLXByb2Nlc3NpbmcsIGV0YykgYXMgdGhlIGxldmVscyBvZiBvdXIgdmFyaWFibGVzIGhhdmUgbm93IGNoYW5nZWQuIAoKV2hpbGUgd2UgYXJlIGRvaW5nIHRoaXMsIHdlIG1pZ2h0IGFsc28gaGF2ZSB0aGlzIGlzc3VlIGZvciBgY291bnR5YC4gCgpUaGUgYGNvdW50eWAgdmFyaWFibGVzIGFwcGVhcnMgdG8gZ2V0IGRyb3BwZWQgZHVlIHRvIGVpdGhlciBjb3JyZWxhdGlvbiBvciBuZWFyIHplcm8gdmFyaWFuY2UuIAoKSXQgaXMgbGlrZWx5IGR1ZSB0byBuZWFyIHplcm8gdmFyaWFuY2UgYmVjYXVzZSB0aGlzIGlzIHRoZSBtb3JlIGdyYW51bGFyIG9mIHRoZXNlIGdlb2dyYXBoaWMgY2F0ZWdvcmljYWwgdmFyaWFibGVzIGFuZCBsaWtlbHkgc3BhcnNlLgoKYGBge3J9CnBtICU8PiUKICBtdXRhdGUoY2l0eSA9IGNhc2Vfd2hlbihjaXR5ID09ICJOb3QgaW4gYSBjaXR5IiB+ICJOb3QgaW4gYSBjaXR5IiwKICAgICAgICAgICAgICAgICAgICAgICAgICBjaXR5ICE9ICJOb3QgaW4gYSBjaXR5IiB+ICJJbiBhIGNpdHkiKSkKCnNldC5zZWVkKDEyMzQpICMgc2FtZSBzZWVkIGFzIGJlZm9yZQpwbV9zcGxpdCA8LXJzYW1wbGU6OmluaXRpYWxfc3BsaXQoZGF0YSA9IHBtLCBwcm9wID0gMi8zKQpwbV9zcGxpdAogdHJhaW5fcG0gPC1yc2FtcGxlOjp0cmFpbmluZyhwbV9zcGxpdCkKIHRlc3RfcG0gPC1yc2FtcGxlOjp0ZXN0aW5nKHBtX3NwbGl0KQpgYGAKCgojIyMjIHsucmVjYWxsX2NvZGVfcXVlc3Rpb25fYmxvY2t9CjxiPjx1PiBRdWVzdGlvbiBPcHBvcnR1bml0eSA8L3U+PC9iPgoKU2VlIGlmIHlvdSBjYW4gY29tZSB1cCB3aXRoIHRoZSBjb2RlIGZvciB0aGUgbmV3IHJlY2lwZS4KCioqKgoKPGRldGFpbHM+IDxzdW1tYXJ5PiBDbGljayBoZXJlIHRvIHJldmVhbCB0aGUgY29kZSBmb3IgdGhlIG5ldyByZWNpcGUuIDwvc3VtbWFyeT4KCgpgYGB7cn0Kbm92ZWxfcmVjIDwtIHJlY2lwZSh0cmFpbl9wbSkgJT4lCiAgICB1cGRhdGVfcm9sZShldmVyeXRoaW5nKCksIG5ld19yb2xlID0gInByZWRpY3RvciIpICU+JQogICAgdXBkYXRlX3JvbGUodmFsdWUsIG5ld19yb2xlID0gIm91dGNvbWUiKSAlPiUKICAgIHVwZGF0ZV9yb2xlKGlkLCBuZXdfcm9sZSA9ICJpZCB2YXJpYWJsZSIpICU+JQogICAgdXBkYXRlX3JvbGUoImZpcHMiLCBuZXdfcm9sZSA9ICJjb3VudHkgaWQiKSAlPiUKICAgIHN0ZXBfZHVtbXkoc3RhdGUsIGNvdW50eSwgY2l0eSwgemN0YSwgb25lX2hvdCA9IFRSVUUpICU+JQogICAgc3RlcF9jb3JyKGFsbF9udW1lcmljKCkpICU+JQogICAgc3RlcF9uenYoYWxsX251bWVyaWMoKSkgCmBgYAo8L2RldGFpbHM+CgoqKioKCiMjIyMKCmBgYHtyfQpub3ZlbF9yZWMKYGBgCgoKCk5vdyBsZXQncyByZXRyYWluIG91ciB0cmFpbmluZyBkYXRhIHdpdGggdGhlIG5ldyBtb2RlbCByZWNpcGUgYW5kIHRyeSBiYWtpbmcgb3VyIHRlc3QgZGF0YSBhZ2Fpbi4KCgoKIyMjIyB7LnJlY2FsbF9jb2RlX3F1ZXN0aW9uX2Jsb2NrfQo8Yj48dT4gUXVlc3Rpb24gT3Bwb3J0dW5pdHkgPC91PjwvYj4KCkRvIHlvdSByZWNhbGwgaG93IHRvIHByZS1wcm9jZXNzIGFuZCBleHRyYWN0IHRoZSBwcmUtcHJvY2Vzc2VkIHRyYWluaW5nIGRhdGE/CgoqKioKCjxkZXRhaWxzPiA8c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byByZXZlYWwgdGhlIGFuc3dlci4gPC9zdW1tYXJ5PgoKYGBge3J9CnByZXBwZWRfcmVjIDwtIHByZXAobm92ZWxfcmVjLCB2ZXJib3NlID0gVFJVRSwgcmV0YWluID0gVFJVRSkKYmFrZWRfdHJhaW4gPC0gYmFrZShwcmVwcGVkX3JlYywgbmV3X2RhdGEgPSBOVUxMKQpgYGAKPC9kZXRhaWxzPiAKCioqKgoKIyMjIwoKCiMjIyMgey5zY3JvbGxhYmxlIH0KYGBge3J9CiMgU2Nyb2xsIHRocm91Z2ggdGhlIG91dHB1dCEKZ2xpbXBzZShiYWtlZF90cmFpbikKYGBgCgojIyMjCgpBbmQgbm93LCBsZXQncyB0cnkgYmFraW5nIG91ciB0ZXN0IHNldCB0byBzZWUgaWYgd2Ugc3RpbGwgaGF2ZSBgTkFgIHZhbHVlcy4KCiMjIyMgey5zY3JvbGxhYmxlIH0KCmBgYHtyfQojIFNjcm9sbCB0aHJvdWdoIHRoZSBvdXRwdXQhCmJha2VkX3Rlc3RfcG0gPC0gYmFrZShwcmVwcGVkX3JlYywgbmV3X2RhdGEgPSB0ZXN0X3BtKQoKZ2xpbXBzZShiYWtlZF90ZXN0X3BtKQpgYGAKCiMjIyMKCkdyZWF0LCBub3cgd2Ugbm8gbG9uZ2VyIGhhdmUgYE5BYCB2YWx1ZXMhIDopCgoqKk5vdGUqKjogaWYgeW91IHVzZSB0aGUgc2tpcCBvcHRpb24gZm9yIHNvbWUgb2YgdGhlIHByZS1wcm9jZXNzaW5nIHN0ZXBzLCBiZSBjYXJlZnVsLiAKdGhlIGBqdWljZSgpYCBmdW5jdGlvbiAoYW4gb2xkZXIgb3B0aW9uKSB3aWxsIHNob3cgYWxsIG9mIHRoZSByZXN1bHRzIGlnbm9yaW5nIGBza2lwID0gVFJVRWAgKGFzIHlvdSBjYW4gc3RpbGwgdXNlIHRoaXMgZnVuY3Rpb24gaWYgeW91IHByZWZlciBpdCB0byBgYmFrZSgpYCkuIFRoZQpgYmFrZSgpYCBmdW5jdGlvbiB3aWxsIG5vdCBuZWNlc3NhcmlseSBjb25kdWN0IHRoZXNlIHN0ZXBzIG9uIHRoZSBuZXcgZGF0YS4KCgojIyAqKlNwZWNpZnlpbmcgdGhlIG1vZGVsKioKKioqCgpTbyBmYXIgd2UgaGF2ZSB1c2VkIHRoZSBwYWNrYWdlcyBgcnNhbXBsZWAgdG8gc3BsaXQgdGhlIGRhdGEgYW5kIGByZWNpcGVzYCB0byBhc3NpZ24gdmFyaWFibGUgdHlwZXMsIGFuZCB0byBzcGVjaWZ5IGFuZCBwcmVwIG91ciBwcmUtcHJvY2Vzc2luZyAoYXMgd2VsbCBhcyB0byBvcHRpb25hbGx5IGV4dHJhY3QgdGhlIHByZS1wcm9jZXNzZWQgZGF0YSkuCgpXZSB3aWxsIG5vdyB1c2UgdGhlIGBwYXJzbmlwYCBwYWNrYWdlICh3aGljaCBpcyBzaW1pbGFyIHRvIHRoZSBwcmV2aW91cyBgY2FyZXRgIHBhY2thZ2UgLSBhbmQgaGVuY2Ugd2h5IGl0IGlzIG5hbWVkIGFmdGVyIHRoZSB2ZWdldGFibGUpIHRvIHNwZWNpZnkgb3VyIG1vZGVsLgoKVGhlcmUgYXJlIGZvdXIgdGhpbmdzIHdlIG5lZWQgdG8gZGVmaW5lIGFib3V0IG91ciBtb2RlbDogIAoKMS4gVGhlICoqdHlwZSoqIG9mIG1vZGVsICh1c2luZyBzcGVjaWZpYyBmdW5jdGlvbnMgaW4gcGFyc25pcCBsaWtlIGByYW5kX2ZvcmVzdCgpYCwgYGxvZ2lzdGljX3JlZygpYCBldGMuKSAgCjIuIFRoZSBwYWNrYWdlIG9yICoqZW5naW5lKiogdGhhdCB3ZSB3aWxsIHVzZSB0byBpbXBsZW1lbnQgdGhlIHR5cGUgb2YgbW9kZWwgc2VsZWN0ZWQgKHVzaW5nIHRoZSBgc2V0X2VuZ2luZSgpYCBmdW5jdGlvbikgCjMuIFRoZSAqKm1vZGUqKiBvZiBsZWFybmluZyAtIGNsYXNzaWZpY2F0aW9uIG9yIHJlZ3Jlc3Npb24gKHVzaW5nIHRoZSBgc2V0X21vZGUoKWAgZnVuY3Rpb24pIAo0LiBBbnkgKiphcmd1bWVudHMqKiBuZWNlc3NhcnkgZm9yIHRoZSBtb2RlbC9wYWNrYWdlIHNlbGVjdGVkICh1c2luZyB0aGUgYHNldF9hcmdzKClgZnVuY3Rpb24gLSAgZm9yIGV4YW1wbGUgdGhlIGBtdHJ5ID1gIGFyZ3VtZW50IGZvciByYW5kb20gZm9yZXN0IHdoaWNoIGlzIHRoZSBudW1iZXIgb2YgdmFyaWFibGVzIHRvIGJlIHVzZWQgYXMgb3B0aW9ucyBmb3Igc3BsaXR0aW5nIGF0IGVhY2ggdHJlZSBub2RlKQoKTGV0J3Mgd2FsayB0aHJvdWdoIHRoZXNlIHN0ZXBzIG9uZSBieSBvbmUuIApGb3Igb3VyIGNhc2UsIHdlIGFyZSBnb2luZyB0byBzdGFydCBvdXIgYW5hbHlzaXMgd2l0aCBhIGxpbmVhciByZWdyZXNzaW9uIChhIHZlcnkgY29tbW9uIHN0YXJ0aW5nIHBvaW50IGZvciBtb2RlbGluZykgYnV0IHdlIHdpbGwgZGVtb25zdHJhdGUgaG93IHdlIGNhbiB0cnkgZGlmZmVyZW50IG1vZGVscy4gV2Ugd2lsbCBhbHNvIHNob3cgaG93IHRvIG1vZGVsIHdpdGggdGhlIFJhbmRvbSBGb3Jlc3QgbWV0aG9kICh3aGljaCBpcyB2ZXJ5IHdpZGVseSB1c2VkKSBsYXRlciBvbi4gCgpUaGUgZmlyc3Qgc3RlcCBpcyB0byBkZWZpbmUgd2hhdCB0eXBlIG9mIG1vZGVsIHdlIHdvdWxkIGxpa2UgdG8gdXNlLiAKU2VlIFtoZXJlXShodHRwczovL3d3dy50aWR5bW9kZWxzLm9yZy9maW5kL3BhcnNuaXAvKXt0YXJnZXQ9Il9ibGFuayJ9IGZvciBtb2RlbGluZyBvcHRpb25zIGluIGBwYXJzbmlwYC4KCgpgYGB7cn0KUE1fbW9kZWwgPC0gcGFyc25pcDo6bGluZWFyX3JlZygpICMgUE0gd2FzIHVzZWQgaW4gdGhlIG5hbWUgZm9yIHBhcnRpY3VsYXRlIG1hdHRlcgpQTV9tb2RlbApgYGAKCk9LLiBTbyBmYXIsIGFsbCB3ZSBoYXZlIGRlZmluZWQgaXMgdGhhdCB3ZSB3YW50IHRvIHVzZSBhIGxpbmVhciByZWdyZXNzaW9uLi4uICAKTGV0J3MgdGVsbCBgcGFyc25pcGAgbW9yZSBhYm91dCB3aGF0IHdlIHdhbnQuCgpXZSB3b3VsZCBsaWtlIHRvIHVzZSB0aGUgW29yZGluYXJ5IGxlYXN0IHNxdWFyZXNdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL09yZGluYXJ5X2xlYXN0X3NxdWFyZXMpIG1ldGhvZCB0byBmaXQgb3VyIGxpbmVhciByZWdyZXNzaW9uLiAKU28gd2Ugd2lsbCB0ZWxsIGBwYXJzbmlwYCB0aGF0IHdlIHdhbnQgdG8gdXNlIHRoZSBgbG1gIHBhY2thZ2UgdG8gaW1wbGVtZW50IG91ciBsaW5lYXIgcmVncmVzc2lvbiAodGhlcmUgYXJlIG1hbnkgb3B0aW9ucyBhY3R1YWxseSBzdWNoIGFzIFtgcnN0YW5gXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvcnN0YW4vdmlnbmV0dGVzL3JzdGFuLmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gIFtgZ2xtbmV0YF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL2dsbW5ldC9pbmRleC5odG1sKXt0YXJnZXQ9Il9ibGFuayJ9LCBbYGtlcmFzYF0oaHR0cHM6Ly9rZXJhcy5yc3R1ZGlvLmNvbS8pe3RhcmdldD0iX2JsYW5rIn0sIGFuZCBbYHNwYXJrbHlyYF0oaHR0cHM6Ly90aGVyaW5zcGFyay5jb20vc3RhcnRpbmcuaHRtbCNzdGFydGluZy1zcGFya2x5ci1oZWxsby13b3JsZCl7dGFyZ2V0PSJfYmxhbmsifSkuIFNlZSBbaGVyZV0oaHR0cHM6Ly9wYXJzbmlwLnRpZHltb2RlbHMub3JnL3JlZmVyZW5jZS9saW5lYXJfcmVnLmh0bWwpIGZvciBhIGRlc2NyaXB0aW9uIG9mIHRoZSBkaWZmZXJlbmNlcyBhbmQgdXNpbmcgdGhlc2UgZGlmZmVyZW50IGVuZ2luZXMgd2l0aCBgcGFyc25pcGAuCgpXZSB3aWxsIGRvIHNvIGJ5IHVzaW5nIHRoZSBgc2V0X2VuZ2luZSgpYCBmdW5jdGlvbiBvZiB0aGUgYHBhcnNuaXBgIHBhY2thZ2UuCgpgYGB7cn0KbG1fUE1fbW9kZWwgPC0gUE1fbW9kZWwgICU+JQogIHBhcnNuaXA6OnNldF9lbmdpbmUoImxtIikKCmxtX1BNX21vZGVsCmBgYAoKSW4gc29tZSBjYXNlcyBzb21lIHBhY2thZ2VzIGNhbiBkbyBlaXRoZXIgY2xhc3NpZmljYXRpb24gb3IgcHJlZGljdGlvbiwgc28gaXQgaXMgYSBnb29kIGlkZWEgdG8gc3BlY2lmeSB3aGljaCBtb2RlIHlvdSBpbnRlbmQgdG8gcGVyZm9ybS4gCkhlcmUsIHdlIGFpbSB0byBwcmVkaWN0IHRoZSBhaXIgcG9sbHV0aW9uLiAKWW91IGNhbiBkbyB0aGlzIHdpdGggdGhlIGBzZXRfbW9kZSgpYCBmdW5jdGlvbiBvZiB0aGUgYHBhcnNuaXBgIHBhY2thZ2UsIGJ5IHVzaW5nIGVpdGhlciBgc2V0X21vZGUoImNsYXNzaWZpY2F0aW9uIilgIG9yIGBzZXRfbW9kZSgicmVncmVzc2lvbiIpYC4KCmBgYHtyfQpsbV9QTV9tb2RlbCA8LSBQTV9tb2RlbCAgJT4lCiAgcGFyc25pcDo6c2V0X2VuZ2luZSgibG0iKSAlPiUKICBzZXRfbW9kZSgicmVncmVzc2lvbiIpCgpsbV9QTV9tb2RlbApgYGAKCiMjICoqRml0dGluZyB0aGUgbW9kZWwqKgoqKioKCldlIGNhbiAgdXNlIHRoZSBgcGFyc25pcGAgcGFja2FnZSB3aXRoIGEgbmV3ZXIgcGFja2FnZSBjYWxsZWQgYHdvcmtmbG93c2AgdG8gZml0IG91ciBtb2RlbC4gCgpUaGUgYHdvcmtmbG93c2AgcGFja2FnZSBhbGxvd3MgdXMgdG8ga2VlcCB0cmFjayBvZiBib3RoIG91ciBwcmUtcHJvY2Vzc2luZyBzdGVwcyBhbmQgb3VyIG1vZGVsIHNwZWNpZmljYXRpb24uIEl0IGFsc28gYWxsb3dzIHVzIHRvIGltcGxlbWVudCBmYW5jaWVyIG9wdGltaXphdGlvbnMgaW4gYW4gYXV0b21hdGVkIHdheSBhbmQgaXQgY2FuIGFsc28gaGFuZGxlIHBvc3QtcHJvY2Vzc2luZyBvcGVyYXRpb25zLiAKCgpXZSBiZWdpbiBieSBjcmVhdGluZyBhIHdvcmtmbG93IHVzaW5nIHRoZSBgd29ya2Zsb3coKWAgZnVuY3Rpb24gaW4gdGhlIGB3b3JrZmxvd3NgIHBhY2thZ2UuIAoKTmV4dCwgd2UgdXNlIGBhZGRfcmVjaXBlKClgIChvdXIgcHJlLXByb2Nlc3Npbmcgc3BlY2lmaWNhdGlvbnMpIGFuZCB3ZSBhZGQgb3VyIG1vZGVsIHdpdGggdGhlIGBhZGRfbW9kZWwoKWAgZnVuY3Rpb24gLS0gYm90aCBmdW5jdGlvbnMgZnJvbSB0aGUgYHdvcmtmbG93c2AgcGFja2FnZS4KCioqTm90ZSoqOiBXZSBkbyBub3QgbmVlZCB0byBhY3R1YWxseSBgcHJlcCgpYCBvdXIgcmVjaXBlIGJlZm9yZSB1c2luZyB3b3JrZmxvd3MhCgpJZiB5b3UgcmVjYWxsIGBub3ZlbF9yZWNgIGlzIHRoZSByZWNpcGUgd2UgcHJldmlvdXNseSBjcmVhdGVkIHdpdGggdGhlIGByZWNpcGVzYCBwYWNrYWdlIGFuZCBgbG1fUE1fbW9kZWxgIHdhcyBjcmVhdGVkIHdoZW4gd2Ugc3BlY2lmaWVkIG91ciBtb2RlbCB3aXRoIHRoZSBgcGFyc25pcGAgcGFja2FnZS4KSGVyZSwgd2UgY29tYmluZSBldmVyeXRoaW5nIHRvZ2V0aGVyIGludG8gYSB3b3JrZmxvdy4gCgpgYGB7cn0KUE1fd2Zsb3cgPC0gd29ya2Zsb3dzOjp3b3JrZmxvdygpICU+JQogICAgICAgICAgICB3b3JrZmxvd3M6OmFkZF9yZWNpcGUobm92ZWxfcmVjKSAlPiUKICAgICAgICAgICAgd29ya2Zsb3dzOjphZGRfbW9kZWwobG1fUE1fbW9kZWwpClBNX3dmbG93CmBgYAoKQWgsIG5pY2UuIApOb3RpY2UgaG93IGl0IHRlbGxzIHVzIGFib3V0IGJvdGggb3VyIHByZS1wcm9jZXNzaW5nIHN0ZXBzIGFuZCBvdXIgbW9kZWwgc3BlY2lmaWNhdGlvbnMuCgpOZXh0LCB3ZSAicHJlcGFyZSB0aGUgcmVjaXBlIiAob3IgZXN0aW1hdGUgdGhlIHBhcmFtZXRlcnMpIGFuZCBmaXQgdGhlIG1vZGVsIHRvIG91ciB0cmFpbmluZyBkYXRhIGFsbCBhdCBvbmNlLiAKUHJpbnRpbmcgdGhlIG91dHB1dCwgd2UgY2FuIHNlZSB0aGUgY29lZmZpY2llbnRzIG9mIHRoZSBtb2RlbC4KCmBgYHtyfQpQTV93Zmxvd19maXQgPC0gcGFyc25pcDo6Zml0KFBNX3dmbG93LCBkYXRhID0gdHJhaW5fcG0pClBNX3dmbG93X2ZpdApgYGAKCiMjIyMgey5jbGlja190b19leHBhbmRfYmxvY2t9Cgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byBzZWUgdGhlIHN0ZXBzIHRoYXQgdGhlIGB3b3JrZmxvd3NgIHBhY2thZ2UgcGVyZm9ybXMgdGhhdCB1c2VkIHRvIGJlIHJlcXVpcmVkIDwvc3VtbWFyeT4KClByZXZpb3VzbHksIHRoZSBwcm9jZXNzZWQgdHJhaW5pbmcgZGF0YSAoYGJha2VkX3RyYWluYCksIGFzIG9wcG9zZWQgdG8gdGhlIHJhdyB0cmFpbmluZyBkYXRhLCB3b3VsZCBiZSByZXF1aXJlZCB0byBmaXQgdGhlIG1vZGVsLgoKSW4gdGhpcyBjYXNlLCB3ZSB3b3VsZCBhY3R1YWxseSBhbHNvIG5lZWQgdG8gd3JpdGUgdGhlIG1vZGVsIGFnYWluISAKUmVjYWxsIHRoYXQgYGlkYCBhbmQgYGZpcHNgIGFyZSBJRCB2YXJpYWJsZXMgYW5kIHRoYXQgYHZhbHVlc2AgaXMgb3VyIG91dGNvbWUgb2YgaW50ZXJlc3QgKHRoZSBhaXIgcG9sbHV0aW9uIG1lYXN1cmUgYXQgZWFjaCBtb25pdG9yKS4gSXQgaXMgbmljZSB0aGF0IGB3b3JrZmxvd3NgIGtlZXBzIHRyYWNrIG9mIHRoaXMhCgpgYGB7cn0KYmFrZWRfdHJhaW5fcmVhZHkgPC0gYmFrZWRfdHJhaW4gJT4lIAogIHNlbGVjdCgtaWQsIC1maXBzKQoKUE1fZml0IDwtIGxtX1BNX21vZGVsICU+JSAKICBwYXJzbmlwOjpmaXQodmFsdWUgfi4sIGRhdGEgPSBiYWtlZF90cmFpbl9yZWFkeSkKYGBgCgo8L2RldGFpbHM+CgojIyMjCgojIyAqKkFzc2Vzc2luZyB0aGUgbW9kZWwgZml0KioKKioqCgpBZnRlciB3ZSBmaXQgb3VyIG1vZGVsLCB3ZSBjYW4gdXNlIHRoZSBgYnJvb21gIHBhY2thZ2UgdG8gbG9vayBhdCB0aGUgb3V0cHV0IGZyb20gdGhlIGZpdHRlZCBtb2RlbCBpbiBhbiBlYXN5L3RpZHkgd2F5LiAgIAoKVGhlIGB0aWR5KClgIGZ1bmN0aW9uIHJldHVybnMgYSB0aWR5IGRhdGEgZnJhbWUgd2l0aCBjb2VmZmljaWVudHMgZnJvbSB0aGUgbW9kZWwgKG9uZSByb3cgcGVyIGNvZWZmaWNpZW50KS4KCk1hbnkgb3RoZXIgYGJyb29tYCBmdW5jdGlvbnMgY3VycmVudGx5IG9ubHkgd29yayB3aXRoIGBwYXJzbmlwYCBvYmplY3RzLCBub3QgcmF3IGB3b3JrZmxvd3NgIG9iamVjdHMuIAoKSG93ZXZlciwgd2UgY2FuIHVzZSB0aGUgYHRpZHlgIGZ1bmN0aW9uIGlmIHdlIGZpcnN0IHVzZSB0aGUgYGV4dHJhY3RfZml0X3BhcnNuaXAoKWAgZnVuY3Rpb24gd2hpY2ggaXMgaW1wb3J0ZWQgYXMgcGFydCBvZiB0aGUgYHdvcmtmbG93c2AgcGFja2FnZSBmcm9tIHRoZSBbYGhhcmRoYXRgIHBhY2thZ2VdKGh0dHBzOi8vaGFyZGhhdC50aWR5bW9kZWxzLm9yZy8pIChhbHNvIHBhcnQgb2YgdGlkeW1vZGVscykuCgpgYGB7cn0Kd2Zsb3dvdXRwdXQgPC0gUE1fd2Zsb3dfZml0ICU+JSAKICBleHRyYWN0X2ZpdF9wYXJzbmlwKCkgJT4lIAogIGJyb29tOjp0aWR5KCkgCmBgYAoKCmBgYHtyfQp3Zmxvd291dHB1dApgYGAKCldlIGhhdmUgZml0IG91ciBtb2RlbCBvbiBvdXIgdHJhaW5pbmcgZGF0YSwgd2hpY2ggbWVhbnMgd2UgaGF2ZSBjcmVhdGVkIGEgbW9kZWwgdG8gcHJlZGljdCB2YWx1ZXMgb2YgYWlyIHBvbGx1dGlvbiBiYXNlZCBvbiB0aGUgcHJlZGljdG9ycyB0aGF0IHdlIGhhdmUgaW5jbHVkZWQuIFlheSEKCk9uZSBsYXN0IHRoaW5nIGJlZm9yZSB3ZSBsZWF2ZSB0aGlzIHNlY3Rpb24uIApXZSBvZnRlbiBhcmUgaW50ZXJlc3RlZCBpbiBnZXR0aW5nIGEgc2Vuc2Ugb2Ygd2hpY2ggdmFyaWFibGVzIGFyZSB0aGUgbW9zdCBpbXBvcnRhbnQgaW4gb3VyIG1vZGVsLiAKV2UgY2FuIGV4cGxvcmUgdGhlIHZhcmlhYmxlIGltcG9ydGFuY2UgdXNpbmcgdGhlIGB2aXAoKWAgZnVuY3Rpb24gb2YgdGhlIGB2aXBgIHBhY2thZ2UuIApUaGlzIGZ1bmN0aW9uIGNyZWF0ZXMgYSBiYXIgcGxvdCBvZiB2YXJpYWJsZSBpbXBvcnRhbmNlIHNjb3JlcyBmb3IgZWFjaCBwcmVkaWN0b3IgdmFyaWFibGUgKG9yIGZlYXR1cmUpIGluIGEgbW9kZWwuIApUaGUgYmFyIHBsb3QgaXMgb3JkZXJlZCBieSBpbXBvcnRhbmNlIChoaWdoZXN0IHRvIHNtYWxsZXN0KS4gCgoKTm90aWNlIGFnYWluIHRoYXQgd2UgbmVlZCB0byB1c2UgdGhlIGBleHRyYWN0X2ZpdF9wYXJzbmlwKClgIGZ1bmN0aW9uLgoKTGV0J3MgdGFrZSBhIGxvb2sgYXQgdGhlIHRvcCAxMCBjb250cmlidXRpbmcgdmFyaWFibGVzOgoKYGBge3J9ClBNX3dmbG93X2ZpdCAlPiUgCiAgZXh0cmFjdF9maXRfcGFyc25pcCgpICU+JSAKICB2aXAobnVtX2ZlYXR1cmVzID0gMTApCmBgYAoKVGhlIGxvY2F0aW9uIG9mIHRoZSBtb25pdG9yIChiZWluZyBpbiBDYWxpZm9ybmlhIHZlcnN1cyBhbm90aGVyIHN0YXRlKSwgdGhlIENNQVEgbW9kZWwsIGFuZCB0aGUgYW9kIHNhdGVsbGl0ZSBpbmZvcm1hdGlvbiBhcHBlYXIgdG8gYmUgdGhlIG1vc3QgaW1wb3J0YW50IGZvciBwcmVkaWN0aW5nIHRoZSBhaXIgcG9sbHV0aW9uIGF0IGEgZ2l2ZW4gbW9uaXRvci4KCkluZGVlZCwgaWYgd2UgcGxvdCBtb25pdG9yIHZhbHVlcyBmb3IgdGhvc2UgaW4gQ2FsaWZvcm5pYSByZWxhdGl2ZSB0byBvdGhlciBzdGF0ZXMgd2UgY2FuIHNlZSB0aGF0IHRoZXJlIGFyZSBzb21lIGhpZ2ggdmFsdWVzIGZvciBtb25pdG9ycyBpbiBDYWxpZm9ybmlhLiBUaGlzIG1heSBiZSBwbGF5aW5nIGEgcm9sZSBpbiB3aGF0IHdlIGFyZSBzZWVpbmcuIEhlcmUgd2UgYXNzdW1lIHRoYXQgeW91IGhhdmUgc29tZSBleHBlcmllbmNlIHBsb3R0aW5nIHdpdGggdGhlIGBnZ3Bsb3QyYHBhY2thZ2UuIElmIG5vdCwgcGxlYXNlIHNlZSB0aGlzIFtjYXNlIHN0dWR5XShodHRwczovL3d3dy5vcGVuY2FzZXN0dWRpZXMub3JnL29jcy1icC1jbzItZW1pc3Npb25zLykuCgojIyMjIHsuY2xpY2tfdG9fZXhwYW5kX2Jsb2NrfQoKPGRldGFpbHM+PHN1bW1hcnk+IENsaWNrIGhlcmUgZm9yIGFuIGludHJvZHVjdGlvbiBhYm91dCB0aGlzIHBhY2thZ2UgaWYgeW91IGFyZSAgbmV3IHRvIHVzaW5nIGBnZ3Bsb3QyYCA8L3N1bW1hcnk+CgoKVGhlIFtnZ3Bsb3QyIHBhY2thZ2VdKGh0dHA6Ly9nZ3Bsb3QyLnRpZHl2ZXJzZS5vcmcpIGlzIGdlbmVyYWxseSBpbnR1aXRpdmUgZm9yIGJlZ2lubmVycyBiZWNhdXNlIGl0IGlzIGJhc2VkIG9uIGEgIFtncmFtbWFyIG9mIGdyYXBoaWNzXShodHRwOi8vdml0YS5oYWQuY28ubnovcGFwZXJzL2xheWVyZWQtZ3JhbW1hci5odG1sKSBvciB0aGUgYGdnYCBpbiBgZ2dwbG90MmAuIApUaGUgaWRlYSBpcyB0aGF0IHlvdSBjYW4gY29uc3RydWN0IG1hbnkgc2VudGVuY2VzIGJ5IGxlYXJuaW5nIGp1c3QgYSBmZXcgbm91bnMsIGFkamVjdGl2ZXMsIGFuZCB2ZXJicy4gVGhlcmUgYXJlIHNwZWNpZmljIOKAnHdvcmRz4oCdIHRoYXQgd2Ugd2lsbCBuZWVkIHRvIGxlYXJuIGFuZCBvbmNlIHdlIGRvLCB5b3Ugd2lsbCBiZSBhYmxlIHRvIGNyZWF0ZSAob3Ig4oCcd3JpdGXigJ0pIGh1bmRyZWRzIG9mIGRpZmZlcmVudCBwbG90cy4KClRoZSBjcml0aWNhbCBwYXJ0IHRvIG1ha2luZyBncmFwaGljcyB1c2luZyBgZ2dwbG90MmAgaXMgdGhlIGRhdGEgbmVlZHMgdG8gYmUgaW4gYSBfdGlkeV8gZm9ybWF0LiAKR2l2ZW4gdGhhdCB3ZSBoYXZlIGp1c3Qgc3BlbnQgdGltZSBwdXR0aW5nIG91ciBkYXRhIGluIF90aWR5XyBmb3JtYXQsIHdlIGFyZSBwcmltZWQgdG8gdGFrZSBhZHZhbnRhZ2Ugb2YgYWxsIHRoYXQgYGdncGxvdDJgIGhhcyB0byBvZmZlciEgCgpXZSB3aWxsIHNob3cgaG93IGl0IGlzIGVhc3kgdG8gcGlwZSBfdGlkeV8gZGF0YSAob3V0cHV0KSBhcyBpbnB1dCB0byBvdGhlciBmdW5jdGlvbnMgdGhhdCBjcmVhdGUgcGxvdHMuIApUaGlzIGFsbCB3b3JrcyBiZWNhdXNlIHdlIGFyZSB3b3JraW5nIAp3aXRoaW4gdGhlIF90aWR5dmVyc2VfLiAKCioqV2hhdCBpcyB0aGUgYGdncGxvdCgpYCBmdW5jdGlvbj8qKiAKQXMgZXhwbGFpbmVkIGJ5IEhhZGxleSBXaWNraGFtOgoKPiBUaGUgZ3JhbW1hciB0ZWxscyB1cyB0aGF0IGEgc3RhdGlzdGljYWwgZ3JhcGhpYyBpcyBhIG1hcHBpbmcgZnJvbSBkYXRhIHRvIGFlc3RoZXRpYyBhdHRyaWJ1dGVzIChjb2xvdXIsIHNoYXBlLCBzaXplKSBvZiBnZW9tZXRyaWMgb2JqZWN0cyAocG9pbnRzLCBsaW5lcywgYmFycykuIFRoZSBwbG90IG1heSBhbHNvIGNvbnRhaW4gc3RhdGlzdGljYWwgdHJhbnNmb3JtYXRpb25zIG9mIHRoZSBkYXRhIGFuZCBpcyBkcmF3biBvbiBhIHNwZWNpZmljIGNvb3JkaW5hdGVzIHN5c3RlbS4KYGdncGxvdDJgIFRlcm1pbm9sb2d5OiAKCi0gKipnZ3Bsb3QqKiAtIHRoZSBtYWluIGZ1bmN0aW9uIHdoZXJlIHlvdSBzcGVjaWZ5IHRoZSBkYXRhc2V0IGFuZCB2YXJpYWJsZXMgdG8gcGxvdCAodGhpcyBpcyB3aGVyZSB3ZSBkZWZpbmUgdGhlIGB4YCBhbmQKYHlgIHZhcmlhYmxlIG5hbWVzKQotICoqZ2VvbXMqKiAtIGdlb21ldHJpYyBvYmplY3RzCiAgICAtIGUuZy4gYGdlb21fcG9pbnQoKWAsIGBnZW9tX2JhcigpYCwgYGdlb21fbGluZSgpYCwgYGdlb21faGlzdG9ncmFtKClgCi0gKiphZXMqKiAtIGFlc3RoZXRpY3MKICAgIC0gc2hhcGUsIHRyYW5zcGFyZW5jeSwgY29sb3IsIGZpbGwsIGxpbmUgdHlwZXMKLSAqKnNjYWxlcyoqIC0gZGVmaW5lIGhvdyB5b3VyIGRhdGEgd2lsbCBiZSBwbG90dGVkCiAgICAtIGNvbnRpbnVvdXMsIGRpc2NyZXRlLCBsb2csIGV0YwoKVGhlIGZ1bmN0aW9uIGBhZXMoKWAgaXMgYW4gYWVzdGhldGljIG1hcHBpbmcgZnVuY3Rpb24gaW5zaWRlIHRoZSBgZ2dwbG90KClgIG9iamVjdC4gCldlIHVzZSB0aGlzIGZ1bmN0aW9uIHRvIHNwZWNpZnkgcGxvdCBhdHRyaWJ1dGVzIChlLmcuIGB4YCBhbmQgYHlgIHZhcmlhYmxlIG5hbWVzKSB0aGF0IHdpbGwgbm90IGNoYW5nZSBhcyB3ZSBhZGQgbW9yZSBsYXllcnMuICAKCkFueXRoaW5nIHRoYXQgZ29lcyBpbiB0aGUgYGdncGxvdCgpYCBvYmplY3QgYmVjb21lcyBhIGdsb2JhbCBzZXR0aW5nLiAKRnJvbSB0aGVyZSwgd2UgdXNlIHRoZSBgZ2VvbWAgb2JqZWN0cyB0byBhZGQgbW9yZSBsYXllcnMgdG8gdGhlIGJhc2UgYGdncGxvdCgpYCBvYmplY3QuIApUaGVzZSB3aWxsIGRlZmluZSB3aGF0IHdlIGFyZSBpbnRlcmVzdGVkIGluIGlsbHVzdHJhdGluZyB1c2luZyB0aGUgZGF0YS4KCjwvZGV0YWlscz4KCiMjIyMKCgpgYGB7cn0KCmJha2VkX3RyYWluX3JlYWR5ICU+JSAKICBtdXRhdGUoc3RhdGVfQ2FsaWZvcm5pYSA9IGFzLmZhY3RvcihzdGF0ZV9DYWxpZm9ybmlhKSkgJT4lCiAgbXV0YXRlKHN0YXRlX0NhbGlmb3JuaWEgPSByZWNvZGUoc3RhdGVfQ2FsaWZvcm5pYSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIjAiID0gIk5vdCBDYWxpZm9ybmlhIiwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIjEiID0gIkNhbGlmb3JuaWEiKSkgJT4lCiAgZ2dwbG90KGFlcyh4ID0gc3RhdGVfQ2FsaWZvcm5pYSwgeSA9IHZhbHVlKSkgKyAKICBnZW9tX2JveHBsb3QoKSArCiAgZ2VvbV9qaXR0ZXIod2lkdGggPSAuMDUpICsgCiAgeGxhYigiTG9jYXRpb24gb2YgTW9uaXRvciIpCgpgYGAKCiMjICoqTW9kZWwgcGVyZm9ybWFuY2UqKgoqKioKCkluIHRoaXMgbmV4dCBzZWN0aW9uLCBvdXIgZ29hbCBpcyB0byBhc3Nlc3MgdGhlIG92ZXJhbGwgbW9kZWwgcGVyZm9ybWFuY2UuIApUaGUgd2F5IHdlIGRvIHRoaXMgaXMgdG8gY29tcGFyZSB0aGUgc2ltaWxhcml0eSBiZXR3ZWVuIHRoZSBwcmVkaWN0ZWQgZXN0aW1hdGVzIG9mIHRoZSBvdXRjb21lIHZhcmlhYmxlIHByb2R1Y2VkIGJ5IHRoZSBtb2RlbCBhbmQgdGhlIHRydWUgb3V0Y29tZSB2YXJpYWJsZSB2YWx1ZXMuIAoKSWYgeW91IHJlY2FsbCB0aGUgW1doYXQgaXMgbWFjaGluZSBsZWFybmluZz9dKCN3aGF0aXNtbCkgc2VjdGlvbiwgd2Ugc2hvd2VkIGhvdyB0byB0aGluayBhYm91dCBtYWNoaW5lIGxlYXJuaW5nIChNTCkgYXMgYW4gb3B0aW1pemF0aW9uIHByb2JsZW0gdGhhdCB0cmllcyB0byBtaW5pbWl6ZSB0aGUgZGlzdGFuY2UgYmV0d2VlbiBvdXIgcHJlZGljdGVkIG91dGNvbWUgJFxoYXR7WX0gPSBmKFgpJCBhbmQgYWN0dWFsIG91dGNvbWUgJFkkIHVzaW5nIG91ciBmZWF0dXJlcyAob3IgcHJlZGljdG9yIHZhcmlhYmxlcykgJFgkIGFzIGlucHV0IHRvIGEgZnVuY3Rpb24gJGYkIHRoYXQgd2Ugd2FudCB0byBlc3RpbWF0ZS4gCgokJGQoWSAtIFxoYXR7WX0pJCQKCkFzIG91ciBnb2FsIGluIHRoaXMgc2VjdGlvbiBpcyB0byBhc3Nlc3Mgb3ZlcmFsbCBtb2RlbCBwZXJmb3JtYW5jZSwgd2Ugd2lsbCBub3cgdGFsayBhYm91dCBkaWZmZXJlbnQgZGlzdGFuY2UgbWV0cmljcyB0aGF0IHlvdSBjYW4gdXNlLiAKCkZpcnN0LCBsZXQncyBwdWxsIG91dCBvdXIgcHJlZGljdGVkIG91dGNvbWUgdmFsdWVzICRcaGF0e1l9ID0gZihYKSQgZnJvbSB0aGUgbW9kZWxzIHdlIGZpdCAodXNpbmcgZGlmZmVyZW50IGFwcHJvYWNoZXMpLiAKCgpgYGB7cn0Kd2ZfZml0IDwtIFBNX3dmbG93X2ZpdCAlPiUgCiAgZXh0cmFjdF9maXRfcGFyc25pcCgpCgp3Zl9maXR0ZWRfdmFsdWVzIDwtIGZpdHRlZCh3Zl9maXRbWyJmaXQiXV0pCmhlYWQod2ZfZml0dGVkX3ZhbHVlcykKYGBgCgpBbHRlcm5hdGl2ZWx5LCB3ZSBjYW4gZ2V0IHRoZSBmaXR0ZWQgdmFsdWVzIHVzaW5nIHRoZSBgYXVnbWVudCgpYCBmdW5jdGlvbiBvZiB0aGUgYGJyb29tYCBwYWNrYWdlIHVzaW5nIHRoZSBvdXRwdXQgZnJvbSBgd29ya2Zsb3dzYDogCgpgYGB7cn0Kd2ZfZml0dGVkX3ZhbHVlcyA8LSAKICBicm9vbTo6YXVnbWVudCh3Zl9maXRbWyJmaXQiXV0sIGRhdGEgPSBiYWtlZF90cmFpbikgJT4lIAogIHNlbGVjdCh2YWx1ZSwgLmZpdHRlZDouc3RkLnJlc2lkKQoKaGVhZCh3Zl9maXR0ZWRfdmFsdWVzKQpgYGAKCk5vdGUgdGhhdCBiZWNhdXNlIHdlIHVzZSB0aGUgYWN0dWFsIHdvcmtmbG93IGhlcmUsIHdlIGNhbiAoYW5kIGFjdHVhbGx5IG5lZWQgdG8pIHVzZSB0aGUgcmF3IGRhdGEgaW5zdGVhZCBvZiB0aGUgcHJlLXByb2Nlc3NlZCBkYXRhLgoKYGBge3J9CnZhbHVlc19wcmVkX3RyYWluIDwtIAogIHByZWRpY3QoUE1fd2Zsb3dfZml0LCB0cmFpbl9wbSkgJT4lIAogIGJpbmRfY29scyh0cmFpbl9wbSAlPiUgc2VsZWN0KHZhbHVlLCBmaXBzLCBjb3VudHksIGlkKSkgCgp2YWx1ZXNfcHJlZF90cmFpbgpgYGAKCiMjIyAqKlZpc3VhbGl6aW5nIG1vZGVsIHBlcmZvcm1hbmNlKioKKioqCgpOb3csIHdlIGNhbiBjb21wYXJlIHRoZSBwcmVkaWN0ZWQgb3V0Y29tZSB2YWx1ZXMgKG9yIGZpdHRlZCB2YWx1ZXMpICRcaGF0e1l9JCB0byB0aGUgYWN0dWFsIG91dGNvbWUgdmFsdWVzICRZJCB0aGF0IHdlIG9ic2VydmVkOiAKCmBgYHtyfQp3Zl9maXR0ZWRfdmFsdWVzICU+JSAKICBnZ3Bsb3QoYWVzKHggPSAgdmFsdWUsIHkgPSAuZml0dGVkKSkgKyAKICBnZW9tX3BvaW50KCkgKyAKICB4bGFiKCJhY3R1YWwgb3V0Y29tZSB2YWx1ZXMiKSArIAogIHlsYWIoInByZWRpY3RlZCBvdXRjb21lIHZhbHVlcyIpCmBgYAoKT0ssIHNvIG91ciByYW5nZSBvZiB0aGUgcHJlZGljdGVkIG91dGNvbWUgdmFsdWVzIGFwcGVhcnMgdG8gYmUgc21hbGxlciB0aGFuIHRoZSByZWFsIHZhbHVlcy4gCldlIGNvdWxkIHByb2JhYmx5IGRvIGEgYml0IGJldHRlci4KCiMjIyAqKlF1YW50aWZ5aW5nIG1vZGVsIHBlcmZvcm1hbmNlKioKKioqCgpOZXh0LCBsZXQncyB1c2UgZGlmZmVyZW50IGRpc3RhbmNlIGZ1bmN0aW9ucyAkZChcY2RvdCkkIHRvIGFzc2VzcyBob3cgZmFyIG9mZiBvdXIgcHJlZGljdGVkIG91dGNvbWUgJFxoYXR7WX0gPSBmKFgpJCBhbmQgYWN0dWFsIG91dGNvbWUgJFkkIHZhbHVlcyBhcmUgZnJvbSBlYWNoIG90aGVyOiAKCiQkZChZIC0gXGhhdHtZfSkkJAoKQXMgbWVudGlvbmVkLCB0aGVyZSBhcmUgZW50aXJlIHNjaG9sYXJseSBmaWVsZHMgb2YgcmVzZWFyY2ggZGVkaWNhdGVkIHRvIGlkZW50aWZ5aW5nIGRpZmZlcmVudCBkaXN0YW5jZSBtZXRyaWNzICRkKFxjZG90KSQgZm9yIG1hY2hpbmUgbGVhcm5pbmcgYXBwbGljYXRpb25zLiAKSG93ZXZlciwgd2hlbiBwZXJmb3JtaW5nIHByZWRpY3Rpb24gd2l0aCBhIGNvbnRpbnVvdXMgb3V0Y29tZSAkWSQsIGEgZmV3IG9mIHRoZSBtb3N0bHkgY29tbW9ubHkgdXNlZCBkaXN0YW5jZSBtZXRyaWNzIGFyZTogCgoxLiBtZWFuIGFic29sdXRlIGVycm9yIChgbWFlYCkgIAoKJCRNQUUgPSBcZnJhY3tcc3VtX3tpPTF9XntufXsofFxoYXR7eV90fS0geV90fCl9XjJ9e259JCQKCgoyLiBSIHNxdWFyZWQgZXJyb3IgKGByc3FgKSAtLSB0aGlzIGlzIGFsc28ga25vd24gYXMgdGhlIGNvZWZmaWNpZW50IG9mIGRldGVybWluYXRpb24gd2hpY2ggaXMgdGhlIHNxdWFyZWQgY29ycmVsYXRpb24gYmV0d2VlbiB0cnV0aCBhbmQgZXN0aW1hdGUKClRoaXMgaXMgY2FsY3VsYXRlZCBhbmQgMSBtaW51cyB0aGUgZnJhY3Rpb24gb2YgdGhlIHJlc2lkdWFsIHN1bSBvZiBzcXVhcmVzICgkU1NfcmVzJCkgYnkgdGhlIHRvdGFsIHN1bSBvZiBzcXVhcmVzICgkU1NfdG90JCkKCgokJFJTUSA9IFJeMiA9IDEgLSBcZnJhY3tTU3Jlc317U1N0b3R9JCQKCiQkU1Nfe3RvdH0gPSBcc3VtX3tpPTF9XntufXsoeV9pLSBcYmFye3l9KX1eMiQkClRoZSB0b3RhbCBzdW0gb2Ygc3F1YXJlcyBpcyBwcm9wb3J0aW9uYWwgdG8gdGhlIHZhcmlhbmNlIG9mIHRoZSBkYXRhLiBJdCBpcyBjYWxjdWxhdGVkIGFzIHRoZSBzdW0gb2YgZWFjaCB0cnVlIHZhbHVlIGZyb20gdGhlIG1lYW4gdHJ1ZSB2YWx1ZSAoJFxiYXJ7eX0kKS4KCiQkU1Nfe3Jlc30gPSBcc3VtX3tpPTF9XntufXsoeV9pLSBcaGF0e3lfaX0pfV4yJCQKClRoZSBzdW0gb2Ygc3F1YXJlcyBvZiByZXNpZHVhbHMgaXMgY2FsY3VsYXRlZCBhcyB0aGUgc3VtIG9mIGVhY2ggcHJlZGljdGVkIHZhbHVlICgkXGhhdHt5X2l9JCBvciBzb21ldGltZXMgJGZfaSQpIGZyb20gdGhlIHRydWUgdmFsdWUgKCR5X2kkKS4gCgoKMy4gW3Jvb3QgbWVhbiBzcXVhcmVkIGVycm9yXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9Sb290LW1lYW4tc3F1YXJlX2RldmlhdGlvbil7dGFyZ2V0PSJfYmxhbmsifSAoYHJtc2VgKSAgIAoKJCRSTVNFID0gXHNxcnR7XGZyYWN7XHN1bV97aT0xfV57bn17KFxoYXR7eV90fS0geV90KX1eMn17bn19JCQKCgoKCk9uZSB3YXkgdG8gY2FsY3VsYXRlIHRoZXNlIG1ldHJpY3Mgd2l0aGluIHRoZSBgdGlkeW1vZGVsc2AgZnJhbWV3b3JrIGlzIHRvIHVzZSB0aGUgYHlhcmRzdGlja2AgcGFja2FnZSB1c2luZyB0aGUgYG1ldHJpY3MoKWAgZnVuY3Rpb24uIAoKTm90ZSB0aGF0IHlvdSBtYXkgb2J0YWluIGRpZmZlcmVudCByZXN1bHRzIGRlcGVuZGluZyBvbiB5b3VyIHZlcnNpb24gb2YgUiBhbmQgdGhlIHBhY2thZ2UgdmVyc2lvbnMgeW91IGFyZSB1c2luZy4gU2VlIFtTZXNzaW9uIEluZm9dKCNzZXNzaW9uaW5mbykgc2VjdGlvbiB0byBsZWFybiB3aGF0IHdlIHVzZWQuCgpgYGB7cn0KeWFyZHN0aWNrOjptZXRyaWNzKHdmX2ZpdHRlZF92YWx1ZXMsIAogICAgICAgICAgICAgICAgICAgdHJ1dGggPSB2YWx1ZSwgZXN0aW1hdGUgPSAuZml0dGVkKQpgYGAKCkFsdGVybmF0aXZlbHkgaWYgeW91IG9ubHkgd2FudGVkIG9uZSBtZXRyaWMgeW91IGNvdWxkIHVzZSB0aGUgYG1hZSgpYCwgYHJzcSgpYCwgb3IgYHJtc2UoKWAgZnVuY3Rpb25zLCByZXNwZWN0aXZlbHkuIAoKYGBge3J9CnlhcmRzdGljazo6bWFlKHdmX2ZpdHRlZF92YWx1ZXMsIAogICAgICAgICAgICAgICB0cnV0aCA9IHZhbHVlLCBlc3RpbWF0ZSA9IC5maXR0ZWQpCmBgYAoKVGhlIGxvd2VyIHRoZSBlcnJvciB2YWx1ZXMsIHRoZSBiZXR0ZXIgdGhlIHBlcmZvcm1hbmNlLiBSTVNFIGFuZCBNQUUgY2FuIHJhbmdlIGZyb20gemVybyB0byBpbmZpbml0eSwgc28gd2UgYXJlbid0IGRvaW5nIHRvbyBiYWQuICBUaGUgTUFFIHZhbHVlIHN1Z2dlc3RzIHRoYXQgdGhlIGF2ZXJhZ2UgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSB2YWx1ZSBwcmVkaWN0ZWQgYW5kIHRoZSByZWFsIHZhbHVlIHdhcyBgciByb3VuZCh5YXJkc3RpY2s6Om1hZSh3Zl9maXR0ZWRfdmFsdWVzLCB0cnV0aCA9IHZhbHVlLCBlc3RpbWF0ZSA9IC5maXR0ZWQpJC5lc3RpbWF0ZSwgMilgIHVnL20zLiBUaGUgcmFuZ2Ugb2YgdGhlIHZhbHVlcyB3YXMgMy0yMiBpbiB0aGUgdHJhaW5pbmcgZGF0YSwgc28gdGhpcyBpcyBhIHJlbGF0aXZlbHkgc21hbGwgYW1vdW50LiBUaGUgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBSTVNFIGFuZCB0aGUgTUFFIGNhbiBpbmRpY2F0ZSB0aGUgdmFyaWFuY2Ugb2YgdGhlIGVycm9ycy4gU2luY2UgdGhlIFJNU0UgYW5kIE1BRSB3ZXJlIHNpbWlsYXIsIHRoaXMgaW5kaWNhdGVzIHRoYXQgdGhlIGVycm9ycyB3ZXJlIHF1aXRlIGNvbnNpc3RlbnQgYWNyb3NzIHRoZSByYW5nZSBvZiB2YWx1ZXMgYW5kIGxhcmdlIGVycm9ycyBhcmUgdW5saWtlbHkgdG8gaGF2ZSBvY2N1cnJlZCwgd2hlcmUgb3VyIHByZWRpY3Rpb24gd291bGQgaGF2ZSBiZWVuIHJlYWxseSBmYXIgb2ZmLiBUaGUgW1Igc3F1YXJlZCBlcnJvcl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvQ29lZmZpY2llbnRfb2ZfZGV0ZXJtaW5hdGlvbikgdmFsdWUgaW5kaWNhdGVzIGhvdyBtdWNoIHZhcmlhYmlsaXR5IGluIHRoZSBvdXRjb21lIHZhbHVlIGNvdWxkIGJlIGV4cGxhaW5lZCBieSB0aGUgcHJlZGljdG9ycyBpbiB0aGUgbW9kZWwuIFRoaXMgdmFsdWUgcmFuZ2VzIGZyb20gMCB0byAxIChzb21ldGltZXMgLTEgZGVwZW5kaW5nIG9uIHRoZSBtZXRob2QgdXNlZCBmb3IgY2FsY3VsYXRpb24pLiBBIHZhbHVlIG9mIDEgd291bGQgaW5kaWNhdGUgdGhlIHRoZSBtb2RlbCBwZXJmZWN0bHkgcHJlZGljdGVkIHRoZSBvdXRjb21lLiBPdXIgdmFsdWUgaW5kaWNhdGVzIHRoYXQgYHIgcm91bmQoeWFyZHN0aWNrOjpyc3Eod2ZfZml0dGVkX3ZhbHVlcywgdHJ1dGggPSB2YWx1ZSwgZXN0aW1hdGUgPSAuZml0dGVkKSQuZXN0aW1hdGUsIDIpKjEwMGAlIG9mIHRoZSB2YXJpYWJpbGl0eSBvZiB0aGUgYWlyIHBvbGx1dGlvbiBtZWFzdXJlcyBjb3VsZCBiZSBleHBsYWluZWQgYnkgdGhlIG1vZGVsLiBTbyB3ZSBjb3VsZCBtYXliZSBkbyBhIGJpdCBiZXR0ZXIuIFNlZSBbaGVyZV0oaHR0cHM6Ly93d3cubmNiaS5ubG0ubmloLmdvdi9wbWMvYXJ0aWNsZXMvUE1DNDY3ODM2NS8pIGZvciBtb3JlIGluZm9ybWF0aW9uLiAKCiMjICoqQ3Jvc3MgdmFsaWRhdGlvbioqCioqKgoKVW50aWwgbm93IHdlIGhhdmUgdXNlZCBldmVyeXRoaW5nIGluIG91ciAidHJhaW5pbmciIGRhdGFzZXQgKGFuZCBoYXZlIG5vdCB0b3VjaGVkIHRoZSAidGVzdGluZyIgZGF0YXNldCkgZnJvbSB0aGUgYHJzYW1wbGVgIHBhY2thZ2UgdG8gYnVpbGQgb3VyIG1hY2hpbmUgbGVhcm5pbmcgKE1MKSBtb2RlbCAkXGhhdHtZfSA9IGYoWCkkIChvciB0byBlc3RpbWF0ZSAkZiQgdXNpbmcgdGhlIGZlYXR1cmVzIG9yIHByZWRpY3RvciB2YXJpYWJsZSAkWCQpLiAKCkhlcmUsIHdlIHRha2UgdGhpcyBiZXlvbmQgdGhlIHNpbXBsZSBzcGxpdCBpbnRvIHRyYWluaW5nIGFuZCB0ZXN0aW5nIGRhdGEgc2V0cy4gCldlIHdpbGwgdXNlIHRoZSBgcnNhbXBsZWAgcGFja2FnZSBhZ2FpbiBpbiBvcmRlciB0byBmdXJ0aGVyIGltcGxlbWVudCB3aGF0IGFyZSBjYWxsZWQgW2Nyb3NzIHZhbGlkYXRpb25dKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0Nyb3NzLXZhbGlkYXRpb25fKHN0YXRpc3RpY3MpKXt0YXJnZXQ9Il9ibGFuayJ9IHRlY2huaXF1ZXMuIFRoaXMgaXMgYWxzbyBjYWxsZWQgKipyZS1zYW1wbGluZyoqIG9yICoqcmVwYXJ0aXRpb25pbmcqKi4gIAoKKipOb3RlKio6IHdlIGFyZSBub3QgYWN0dWFsbHkgZ2V0dGluZyBuZXcgc2FtcGxlcyBmcm9tIHRoZSB1bmRlcmx5aW5nIGRpc3RyaWJ1dGlvbiBzbyB0aGUgdGVybSByZS1zYW1wbGluZyBpcyBhIGJpdCBvZiBhIG1pc25vbWVyLgoKW0Nyb3NzIHZhbGlkYXRpb25dKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0Nyb3NzLXZhbGlkYXRpb25fKHN0YXRpc3RpY3MpKXt0YXJnZXQ9Il9ibGFuayJ9IHNwbGl0cyBvdXIgdHJhaW5pbmcgZGF0YSBpbnRvIG11bHRpcGxlIHRyYWluaW5nIGRhdGEgc2V0cyB0byBhbGxvdyBmb3IgYSBkZWVwZXIgYXNzZXNzbWVudCBvZiB0aGUgYWNjdXJhY3kgb2YgdGhlIG1vZGVsLgoKSGVyZSBpcyBhIHZpc3VhbGl6YXRpb24gb2YgdGhlIGNvbmNlcHQgZm9yIGNyb3NzIHZhbGlkYXRpb24vcmVzYW1wbGluZy9yZXBhcnRpdGlvbmluZyBmcm9tIFtNYXggS3Vobl0oaHR0cHM6Ly9yZXNvdXJjZXMucnN0dWRpby5jb20vYXV0aG9ycy9tYXgta3Vobil7dGFyZ2V0PSJfYmxhbmsifToKCmBgYHtyLCBlY2hvPUZBTFNFLCBvdXQud2lkdGg9IjgwMHB4In0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwicmVzYW1wbGluZy5wbmciKSkKYGBgCgpUZWNobmljYWxseSBjcmVhdGluZyBvdXIgdGVzdGluZyBhbmQgdHJhaW5pbmcgc2V0IG91dCBvZiBvdXIgb3JpZ2luYWwgdHJhaW5pbmcgZGF0YSBpcyBzb21ldGltZXMgY29uc2lkZXJlZCBhIGZvcm0gb2YgW2Nyb3NzIHZhbGlkYXRpb25dKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0Nyb3NzLXZhbGlkYXRpb25fKHN0YXRpc3RpY3MpKXt0YXJnZXQ9Il9ibGFuayJ9LCBjYWxsZWQgdGhlIGhvbGRvdXQgbWV0aG9kLiAKVGhlIHJlYXNvbiB3ZSBkbyB0aGlzIGl0IHNvIHdlIGNhbiBnZXQgYSBiZXR0ZXIgc2Vuc2Ugb2YgdGhlIGFjY3VyYWN5IG9mIG91ciBtb2RlbCB1c2luZyBkYXRhIHRoYXQgd2UgZGlkIG5vdCB0cmFpbiBpdCBvbi4gCgpIb3dldmVyLCB3ZSBjYW4gYWN0dWFsbHkgZG8gYSBiZXR0ZXIgam9iIG9mIG9wdGltaXppbmcgb3VyIG1vZGVsIGZvciBhY2N1cmFjeSBpZiB3ZSBhbHNvIHBlcmZvcm0gYW5vdGhlciB0eXBlIG9mIFtjcm9zcyB2YWxpZGF0aW9uXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9Dcm9zcy12YWxpZGF0aW9uXyhzdGF0aXN0aWNzKSl7dGFyZ2V0PSJfYmxhbmsifSBvbiB0aGUgbmV3bHkgZGVmaW5lZCB0cmFpbmluZyBzZXQgdGhhdCB3ZSBqdXN0IGNyZWF0ZWQuIApUaGVyZSBhcmUgbWFueSBbY3Jvc3MgdmFsaWRhdGlvbl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvQ3Jvc3MtdmFsaWRhdGlvbl8oc3RhdGlzdGljcykpe3RhcmdldD0iX2JsYW5rIn0gbWV0aG9kcyBhbmQgbW9zdCBjYW4gYmUgZWFzaWx5IGltcGxlbWVudGVkIHVzaW5nIGByc2FtcGxlYCBwYWNrYWdlLiAKSGVyZSwgd2Ugd2lsbCB1c2UgYSB2ZXJ5IHBvcHVsYXIgbWV0aG9kIGNhbGxlZCBlaXRoZXIgW2stZm9sZCBvciB2LWZvbGQgY3Jvc3MgdmFsaWRhdGlvbl0oaHR0cHM6Ly9tYWNoaW5lbGVhcm5pbmdtYXN0ZXJ5LmNvbS9rLWZvbGQtY3Jvc3MtdmFsaWRhdGlvbi8pe3RhcmdldD0iX2JsYW5rIn0uIAoKVGhpcyBtZXRob2QgaW52b2x2ZXMgZXNzZW50aWFsbHkgcHJlZm9ybWluZyB0aGUgaG9sZCBvdXQgbWV0aG9kIGl0ZXJhdGl2ZWx5IHdpdGggdGhlIHRyYWluaW5nIGRhdGEuIAoKRmlyc3QsIHRoZSB0cmFpbmluZyBzZXQgaXMgZGl2aWRlZCBpbnRvICR2JCAob3Igb2Z0ZW4gY2FsbGVkIGNhbGxlZCAkayQpIGVxdWFsbHkgc2l6ZWQgc21hbGxlciBwaWVjZXMuIAoKTmV4dCwgdGhlIG1vZGVsIGlzIHRyYWluZWQgb24gdGhlIG1vZGVsIG9uICR2JC0xIHN1YnNldHMgb2YgdGhlIGRhdGEgaXRlcmF0aXZlbHkgKHJlbW92aW5nIGEgZGlmZmVyZW50ICR2JCB1bnRpbCBhbGwgcG9zc2libGUgJHYkLTEgc2V0cyBoYXZlIGJlZW4gZXZhbHVhdGVkKSB0byBnZXQgYSBzZW5zZSBvZiB0aGUgcGVyZm9ybWFuY2Ugb2YgdGhlIG1vZGVsLiAKVGhpcyBpcyByZWFsbHkgdXNlZnVsIGZvciBmaW5lIHR1bmluZyBzcGVjaWZpYyBhc3BlY3RzIG9mIHRoZSBtb2RlbCBpbiBhIHByb2Nlc3MgY2FsbGVkIG1vZGVsIHR1bmluZywgd2hpY2ggd2Ugd2lsbCBsZWFybiBhYm91dCBpbiB0aGUgbmV4dCBzZWN0aW9uLiAKCkhlcmUgaXMgYSB2aXN1YWxpemF0aW9uIG9mIGhvdyB0aGUgZm9sZHMgYXJlIGNyZWF0ZWQ6CgpgYGB7ciwgZWNobz1GQUxTRSwgb3V0LndpZHRoPSI4MDBweCJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsICJ2Zm9sZC5wbmciKSkKYGBgCgoqKk5vdGUqKjogUGVvcGxlIHR5cGljYWxseSBpZ25vcmUgc3BhdGlhbCBkZXBlbmRlbmNlIHdpdGggY3Jvc3MgdmFsaWRhdGlvbiBvZiBhaXIgcG9sbHV0aW9uIG1vbml0b3JpbmcgZGF0YSBpbiB0aGUgYWlyIHBvbGx1dGlvbiBmaWVsZCwgc28gd2Ugd2lsbCBkbyB0aGUgc2FtZS4gSG93ZXZlciwgaXQgbWlnaHQgbWFrZSBzZW5zZSB0byBsZWF2ZSBvdXQgYmxvY2tzIG9mIG1vbml0b3JzIHJhdGhlciB0aGFuIHJhbmRvbSBpbmRpdmlkdWFsIG1vbml0b3JzIHRvIGhlbHAgYWNjb3VudCBmb3Igc29tZSBzcGF0aWFsIGRlcGVuZGVuY2UuCgojIyMgKipDcmVhdGluZyB0aGUgJHYkLWZvbGRzIHVzaW5nIGByc2FtcGxlYCoqCioqKgoKVGhlIFtgdmZvbGRfY3YoKWBdKGh0dHBzOi8vdGlkeW1vZGVscy5naXRodWIuaW8vcnNhbXBsZS9yZWZlcmVuY2UvdmZvbGRfY3YuaHRtbCl7dGFyZ2V0PSJfYmxhbmsifSBmdW5jdGlvbiBvZiB0aGUgYHJzYW1wbGVgIHBhY2thZ2UgY2FuIGJlIHVzZWQgdG8gcGFyc2UgdGhlIHRyYWluaW5nIGRhdGEgaW50byBmb2xkcyBmb3IgJHYkLWZvbGQgY3Jvc3MgdmFsaWRhdGlvbi4KCi0gVGhlIGB2YCBhcmd1bWVudCBzcGVjaWZpZXMgdGhlIG51bWJlciBvZiBmb2xkcyB0byBjcmVhdGUuCi0gVGhlIGByZXBlYXRzYCBhcmd1bWVudCBzcGVjaWZpZXMgaWYgYW55IHNhbXBsZXMgc2hvdWxkIGJlIHJlcGVhdGVkIGFjcm9zcyBmb2xkcyAtIGRlZmF1bHQgaXMgYEZBTFNFYAotIFRoZSBgc3RyYXRhYCBhcmd1bWVudCBzcGVjaWZpZXMgYSB2YXJpYWJsZSB0byBzdHJhdGlmeSBzYW1wbGVzIGFjcm9zcyBmb2xkcyAtIGp1c3QgbGlrZSBpbiBgaW5pdGlhbF9zcGxpdCgpYC4KCkFnYWluLCBiZWNhdXNlIHRoZXNlIGFyZSBjcmVhdGVkIGF0IHJhbmRvbSwgd2UgbmVlZCB0byB1c2UgdGhlIGJhc2UgYHNldC5zZWVkKClgIGZ1bmN0aW9uIGluIG9yZGVyIHRvIG9idGFpbiB0aGUgc2FtZSByZXN1bHRzIGVhY2ggdGltZSB3ZSBrbml0IHRoaXMgZG9jdW1lbnQuIApHZW5lcmFsbHkgc3BlYWtpbmcgdXNpbmcgMTAgZm9sZHMgaXMgZ29vZCBwcmFjdGljZSwgYnV0IHRoaXMgZGVwZW5kcyBvbiB0aGUgdmFyaWFiaWxpdHkgd2l0aGluIHlvdXIgZGF0YS4gCldlIGFyZSBnb2luZyB0byB1c2UgNCBmb3IgdGhlIHNha2Ugb2YgZXhwZWRpZW5jeSBpbiB0aGlzIGRlbW9uc3RyYXRpb24uIAoKYGBge3J9CnNldC5zZWVkKDEyMzQpCnZmb2xkX3BtIDwtIHJzYW1wbGU6OnZmb2xkX2N2KGRhdGEgPSB0cmFpbl9wbSwgdiA9IDQpCnZmb2xkX3BtCnB1bGwodmZvbGRfcG0sIHNwbGl0cykKYGBgCgpOb3cgd2UgY2FuIHNlZSB0aGF0IHdlIGhhdmUgY3JlYXRlZCA0IGZvbGRzIG9mIHRoZSBkYXRhIGFuZCB3ZSBjYW4gc2VlIGhvdyBtYW55IHZhbHVlcyB3ZXJlIHNldCBhc2lkZSBmb3IgdGVzdGluZyAoY2FsbGVkIGFzc2Vzc2luZyBmb3IgY3Jvc3MgdmFsaWRhdGlvbiBzZXRzKSBhbmQgdHJhaW5pbmcgKGNhbGxlZCBhbmFseXNpcyBmb3IgY3Jvc3MgdmFsaWRhdGlvbiBzZXRzKSB3aXRoaW4gZWFjaCBmb2xkLgoKT25jZSB0aGUgZm9sZHMgYXJlIGNyZWF0ZWQgdGhleSBjYW4gYmUgdXNlZCB0byBldmFsdWF0ZSBwZXJmb3JtYW5jZSBieSBmaXR0aW5nIHRoZSBtb2RlbCB0byBlYWNoIG9mIHRoZSByZS1zYW1wbGVzIHRoYXQgd2UgY3JlYXRlZDoKCmBgYHtyLCBlY2hvPUZBTFNFLCBvdXQud2lkdGg9IjgwMHB4In0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwgImNyb3NzX3ZhbGlkYXRpb24ucG5nIikpCmBgYAoKIyMjICoqQXNzZXNzaW5nIG1vZGVsIHBlcmZvcm1hbmNlIG9uICR2JC1mb2xkcyB1c2luZyBgdHVuZWAqKgoqKioKCldlIGNhbiBmaXQgdGhlIG1vZGVsIHRvIG91ciBjcm9zcyB2YWxpZGF0aW9uIGZvbGRzIHVzaW5nIHRoZSBgZml0X3Jlc2FtcGxlcygpYCBmdW5jdGlvbiBvZiB0aGUgYHR1bmVgIHBhY2thZ2UsIGJ5IHNwZWNpZnlpbmcgb3VyIGB3b3JrZmxvd2Agb2JqZWN0IGFuZCB0aGUgY3Jvc3MgdmFsaWRhdGlvbiBmb2xkIG9iamVjdCB3ZSBqdXN0IGNyZWF0ZWQuIApTZWUgW2hlcmVdKGh0dHBzOi8vdGlkeW1vZGVscy5naXRodWIuaW8vdHVuZS9yZWZlcmVuY2UvZml0X3Jlc2FtcGxlcy5odG1sKXt0YXJnZXQ9Il9ibGFuayJ9IGZvciBtb3JlIGluZm9ybWF0aW9uLgoKCmBgYHtyfQpyZXNhbXBsZV9maXQgPC0gdHVuZTo6Zml0X3Jlc2FtcGxlcyhQTV93ZmxvdywgdmZvbGRfcG0pCmBgYAoKCldlIGNhbiBub3cgdGFrZSBhIGxvb2sgYXQgdmFyaW91cyBwZXJmb3JtYW5jZSBtZXRyaWNzIGJhc2VkIG9uIHRoZSBmaXQgb2Ygb3VyIGNyb3NzIHZhbGlkYXRpb24gInJlc2FtcGxlcyIuIApUbyBkbyB0aGlzIHdlIHdpbGwgdXNlIHRoZSBgc2hvd19iZXN0KClgIGZ1bmN0aW9uIG9mIHRoZSBgdHVuZWAgcGFja2FnZS4KCmBgYHtyfQp0dW5lOjpzaG93X2Jlc3QocmVzYW1wbGVfZml0LCBtZXRyaWMgPSAicm1zZSIpCmBgYAoKSGVyZSB3ZSBjYW4gc2VlIHRoZSBtZWFuIGBSTVNFYCB2YWx1ZSBhY3Jvc3MgYWxsIGZvdXIgZm9sZHMuIFRoZSBmdW5jdGlvbiBpcyBjYWxsZWQgYHNob3dfYmVzdCgpYCBiZWNhdXNlIGl0IGlzIGFsc28gdXNlZCBmb3IgbW9kZWwgdHVuaW5nIGFuZCBpdCB3aWxsIHNob3cgdGhlIHBhcmFtZXRlciBjb21iaW5hdGlvbiB3aXRoIHRoZSBiZXN0IHBlcmZvcm1hbmNlLCB3ZSB3aWxsIGRpc2N1c3MgdGhpcyBtb3JlIGxhdGVyIGluIHRoZSBjYXNlIHN0dWR5IHdoZW4gd2UgdGVzdCBkaWZmZXJlbnQgbW9kZWxzIHdpdGggZGlmZmVyZW50IHBhcmFtZXRlcnMuIEhvd2V2ZXIsIGZvciBub3cgdGhpcyBnaXZlcyB1cyBhIG1vcmUgbnVhbmNlZCBlc3RpbWF0ZSBvZiB0aGUgUk1TRSBpbiB0ZXJtcyBvZiB0aGUgcGVyZm9ybWFuY2Ugb2YgdGhpcyBzaW5nbGUgbW9kZWwgd2l0aCB0aGUgdHJhaW5pbmcgZGF0YSBieSBsb29raW5nIGF0IHRoZSBzdWJzZXRzIG9mIHRoZSB0cmFpbmluZyBkYXRhLgoKCiMgKipEYXRhIEFuYWx5c2lzKioKKioqCgpJZiB5b3UgaGF2ZSBiZWVuIGZvbGxvd2luZyBhbG9uZyBidXQgc3RvcHBlZCBhbmQgYXJlIHN0YXJ0aW5nIGhlcmUgeW91IGNvdWxkIGxvYWQgdGhlIHdyYW5nbGVkIGRhdGEgdXNpbmcgdGhlIGZvbGxvd2luZyBjb21tYW5kOgoKYGBge3J9CmxvYWQoaGVyZTo6aGVyZSggImRhdGEiLCAid3JhbmdsZWQiLCAid3JhbmdsZWRfcG0ucmRhIikpCmBgYAoKIyMjIyB7LmNsaWNrX3RvX2V4cGFuZF9ibG9ja30KCjxkZXRhaWxzPiA8c3VtbWFyeT4gSWYgeW91IHNraXBwZWQgdGhlIHByZXZpb3VzIHNlY3Rpb25zLCBjbGljayBoZXJlIGZvciBhIHN0ZXAtYnktc3RlcCBndWlkZSBvbiBob3cgdG8gZG93bmxvYWQgYW5kIGxvYWQgdGhlIGRhdGEuIDwvc3VtbWFyeT4KCkZpcnN0IHlvdSBuZWVkIHRvIGluc3RhbGwgYW5kIGxvYWQgdGhlIGBPQ1NkYXRhYCBwYWNrYWdlOgoKYGBge3IsIGV2YWw9RkFMU0V9Cmluc3RhbGwucGFja2FnZXMoIk9DU2RhdGEiKQpsaWJyYXJ5KE9DU2RhdGEpCmBgYAoKVGhlbiwgeW91IG1heSBsb2FkIHRoZSB3cmFuZ2xlZCBkYXRhIGAucmRhYCBmaWxlIHVzaW5nIHRoZSBmb2xsb3dpbmcgZnVuY3Rpb246CgpgYGB7ciwgZXZhbD1GQUxTRX0Kd3JhbmdsZWRfcmRhKCJvY3MtYnAtYWlyLXBvbGx1dGlvbiIsIG91dHBhdGggPSBnZXR3ZCgpKQpsb2FkKGhlcmU6OmhlcmUoIk9DU2RhdGEiLCAiZGF0YSIsICJ3cmFuZ2xlZCIsICJ3cmFuZ2xlZF9wbS5yZGEiKSkKYGBgCgpJZiB0aGUgcGFja2FnZSBkb2VzIG5vdCB3b3JrIGZvciB5b3UsIHlvdSBtYXkgYWxzbyBkb3dubG9hZCB0aGlzIGAucmRhYCBmaWxlIGJ5IGNsaWNraW5nIHRoaXMgbGluayBbaGVyZV0oaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL29wZW5jYXNlc3R1ZGllcy9vY3MtYnAtYWlyLXBvbGx1dGlvbi9tYXN0ZXIvZGF0YS93cmFuZ2xlZC93cmFuZ2xlZF9wbS5yZGEpLgoKVG8gbG9hZCB0aGUgZG93bmxvYWRlZCBkYXRhIGludG8geW91ciBlbnZpcm9ubWVudCwgeW91IG1heSBkb3VibGUgY2xpY2sgb24gdGhlIGAucmRhYCBmaWxlIGluIFJzdHVkaW8gb3IgdXNpbmcgdGhlIGBsb2FkKClgIGZ1bmN0aW9uLgoKVG8gY29weSBhbmQgcGFzdGUgb3VyIGNvZGUgYmVsb3csIHBsYWNlIHRoZSBkb3dubG9hZGVkIGAucmRhYCBmaWxlIGluIHlvdXIgY3VycmVudCB3b3JraW5nIGRpcmVjdG9yeSB3aXRoaW4gYSBzdWJkaXJlY3RvcnkgY2FsbGVkICJ3cmFuZ2xlZCIgd2l0aGluIGEgc3ViZGlyZWN0b3J5IGNhbGxlZCAiZGF0YSIuIFdlIHVzZWQgYW4gUlN0dWRpbyBwcm9qZWN0IGFuZCB0aGUgW2BoZXJlYCBwYWNrYWdlXShodHRwczovL2dpdGh1Yi5jb20vamVubnliYy9oZXJlX2hlcmUpIHRvIG5hdmlnYXRlIHRvIHRoZSBmaWxlIG1vcmUgZWFzaWx5LiAKCmBgYHtyfQpsb2FkKGhlcmU6OmhlcmUoImRhdGEiLCAid3JhbmdsZWQiLCAid3JhbmdsZWRfcG0ucmRhIikpCmBgYAoKPGhyIHN0eWxlPSJoZWlnaHQ6MXB4O2JvcmRlcjpub25lO2NvbG9yOiMzMzM7YmFja2dyb3VuZC1jb2xvcjojMzMzOyIgLz4KCjxkZXRhaWxzPiA8c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byBzZWUgbW9yZSBhYm91dCBjcmVhdGluZyBuZXcgcHJvamVjdHMgaW4gUlN0dWRpby4gPC9zdW1tYXJ5PgoKWW91IGNhbiBjcmVhdGUgYSBwcm9qZWN0IGJ5IGdvaW5nIHRvIHRoZSBGaWxlIG1lbnUgb2YgUlN0dWRpbyBsaWtlIHNvOgoKCmBgYHtyLCBlY2hvID0gRkFMU0UsIG91dC53aWR0aD0iNjAlIn0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwgIk5ld19wcm9qZWN0LnBuZyIpKQpgYGAKCllvdSBjYW4gYWxzbyBkbyBzbyBieSBjbGlja2luZyB0aGUgcHJvamVjdCBidXR0b246CgpgYGB7ciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGg9IjYwJSJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKGhlcmU6OmhlcmUoImltZyIsICJwcm9qZWN0X2J1dHRvbi5wbmciKSkKYGBgCgpTZWUgW2hlcmVdKGh0dHBzOi8vc3VwcG9ydC5yc3R1ZGlvLmNvbS9oYy9lbi11cy9hcnRpY2xlcy8yMDA1MjYyMDctVXNpbmctUHJvamVjdHMpIHRvIGxlYXJuIG1vcmUgYWJvdXQgdXNpbmcgUlN0dWRpbyBwcm9qZWN0cyBhbmQgW2hlcmVdKGh0dHBzOi8vZ2l0aHViLmNvbS9qZW5ueWJjL2hlcmVfaGVyZSkgdG8gbGVhcm4gbW9yZSBhYm91dCB0aGUgYGhlcmVgIHBhY2thZ2UuCgo8L2RldGFpbHM+Cgo8aHIgc3R5bGU9ImhlaWdodDoxcHg7Ym9yZGVyOm5vbmU7Y29sb3I6IzMzMztiYWNrZ3JvdW5kLWNvbG9yOiMzMzM7IiAvPgoKPC9kZXRhaWxzPgoKIyMjIwoKVGhlbiB5b3UgY2FuIG1vZGlmeSB0aGUgZGF0YSB0byBtYXRjaCB3aGF0IHdlIGRpZCBmb3IgdGhlIHByZXZpb3VzIG1vZGVsIGxpa2Ugc28gdG8gcHJlcGFyZSB0aGUgYGNpdHlgIHZhcmlhYmxlIGFuZCB0byBzcGxpdCB0aGUgZGF0YSBpbnRvIHRlc3RpbmcgYW5kIHRyYWluaW5nIHNldHMuIChZb3Ugd2lsbCBzZWUgbGF0ZXIgdGhhdCB0aGlzIHdvdWxkIGJlIHJlcXVpcmVkIGR1ZSB0byB0aGUgbnVtYmVyIG9mIGxldmVscyBmb3IgdGhlIGNpdHkgdmFyaWFibGUgaW4gdGhlIG9yaWdpbmFsIGRhdGEpLgoKYGBge3J9CnBtICU8PiUKICBtdXRhdGUoY2l0eSA9IGNhc2Vfd2hlbihjaXR5ID09ICJOb3QgaW4gYSBjaXR5IiB+ICJOb3QgaW4gYSBjaXR5IiwKICAgICAgICAgICAgICAgICAgICAgICAgICBjaXR5ICE9ICJOb3QgaW4gYSBjaXR5IiB+ICJJbiBhIGNpdHkiKSkKCnNldC5zZWVkKDEyMzQpICMgc2FtZSBzZWVkIGFzIGJlZm9yZQpwbV9zcGxpdCA8LXJzYW1wbGU6OmluaXRpYWxfc3BsaXQoZGF0YSA9IHBtLCBwcm9wID0gMi8zKQpwbV9zcGxpdAogdHJhaW5fcG0gPC1yc2FtcGxlOjp0cmFpbmluZyhwbV9zcGxpdCkKIHRlc3RfcG0gPC1yc2FtcGxlOjp0ZXN0aW5nKHBtX3NwbGl0KQpgYGAKCldlIHdpbGwgYWxzbyBzcGxpdCBvdXIgZGF0YSBpbnRvIGNyb3NzIHZhbGlkYXRpb24gZm9sZHM6CgpgYGB7cn0Kc2V0LnNlZWQoMTIzNCkKdmZvbGRfcG0gPC0gcnNhbXBsZTo6dmZvbGRfY3YoZGF0YSA9IHRyYWluX3BtLCB2ID0gNCkKdmZvbGRfcG0KYGBgCgpJbiB0aGUgcHJldmlvdXMgc2VjdGlvbiwgd2UgZGVtb25zdHJhdGVkIGhvdyB0byBidWlsZCBhIG1hY2hpbmUgbGVhcm5pbmcgbW9kZWwgKHNwZWNpZmljYWxseSBhIGxpbmVhciByZWdyZXNzaW9uIG1vZGVsKSB0byBwcmVkaWN0IGFpciBwb2xsdXRpb24gd2l0aCB0aGUgYHRpZHltb2RlbHNgIGZyYW1ld29yay4gCgpJbiB0aGUgbmV4dCBmZXcgc2VjdGlvbiwgd2Ugd2lsbCBkZW1vbnN0cmF0ZSBhbm90aGVyIHZlcnkgZGlmZmVyZW50IGtpbmQgb2YgbWFjaGluZSBsZWFybmluZyBtb2RlbC4gVGhpcyB3aWxsIGFsbG93IHVzIHRvIHNlZSBpZiB0aGlzIG1vZGVsIHdpbGwgaGF2ZSBiZXR0ZXIgcHJlZGljdGlvbiBwZXJmb3JtYW5jZS4KCgojIyAqKlJhbmRvbSBGb3Jlc3QqKgoqKioKCk5vdywgdG8gdHJ5IHRvIHNlZSBpZiB3ZSBjYW4gZ2V0IGJldHRlciBwcmVkaWN0aW9uIHBlcmZvcm1hbmNlLCB3ZSBhcmUgZ29pbmcgdG8gcHJlZGljdCBvdXIgb3V0Y29tZSB2YXJpYWJsZSAoYWlyIHBvbGx1dGlvbikgdXNpbmcgYSBkZWNpc2lvbiB0cmVlIG1ldGhvZCBjYWxsZWQgW3JhbmRvbSBmb3Jlc3RdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL1JhbmRvbV9mb3Jlc3Qpe3RhcmdldD0iX2JsYW5rIn0uCgpBIFtkZWNpc2lvbiB0cmVlXShodHRwczovL21lZGl1bS5jb20vZ3JleWF0b20vZGVjaXNpb24tdHJlZXMtYS1zaW1wbGUtd2F5LXRvLXZpc3VhbGl6ZS1hLWRlY2lzaW9uLWRjNTA2YTQwM2FlYil7dGFyZ2V0PSJfYmxhbmsifSBpcyBhIHRvb2wgdG8gcGFydGl0aW9uIGRhdGEgb3IgYW55dGhpbmcgcmVhbGx5LCBiYXNlZCBvbiBhIHNlcmllcyBvZiBzZXF1ZW50aWFsIChvZnRlbiBiaW5hcnkpIGRlY2lzaW9ucywgd2hlcmUgdGhlIGRlY2lzaW9ucyBhcmUgY2hvc2VuIGJhc2VkIG9uIHRoZWlyIGFiaWxpdHkgdG8gb3B0aW1hbGx5IHNwbGl0IHRoZSBkYXRhLgoKSGVyZSB5b3UgY2FuIHNlZSBhIHNpbXBsZSBleGFtcGxlOgoKYGBge3IsIGVjaG8gPSBGQUxTRX0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoImh0dHBzOi8vbWlyby5tZWRpdW0uY29tL21heC8xMDAwLzEqTE1vSm1YQ3NRbGNpR1RFeW9TTjM5Zy5qcGVnIikKYGBgCgojIyMjIyBbW3NvdXJjZV1dKGh0dHBzOi8vdG93YXJkc2RhdGFzY2llbmNlLmNvbS91bmRlcnN0YW5kaW5nLXJhbmRvbS1mb3Jlc3QtNTgzODFlMDYwMmQyKXt0YXJnZXQ9Il9ibGFuayJ9CgpJbiB0aGUgY2FzZSBvZiBbcmFuZG9tIGZvcmVzdF0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL2RlY2lzaW9uLXRyZWUtZW5zZW1ibGVzLWJhZ2dpbmctYW5kLWJvb3N0aW5nLTI2NmE4YmE2MGZkOSl7dGFyZ2V0PSJfYmxhbmsifSwgbXVsdGlwbGUgZGVjaXNpb24gdHJlZXMgYXJlIGNyZWF0ZWQgLSBoZW5jZSB0aGUgbmFtZSBmb3Jlc3QsIGFuZCBlYWNoIHRyZWUgaXMgYnVpbHQgdXNpbmcgYSByYW5kb20gc3Vic2V0IG9mIHRoZSB0cmFpbmluZyBkYXRhICh3aXRoIHJlcGxhY2VtZW50KSAtIGhlbmNlIHRoZSBmdWxsIG5hbWUgcmFuZG9tIGZvcmVzdC4gVGhpcyByYW5kb20gYXNwZWN0IGhlbHBzIHRvIGtlZXAgdGhlIGFsZ29yaXRobSBmcm9tIG92ZXJmaXR0aW5nIHRoZSBkYXRhLgoKVGhlIG1lYW4gb2YgdGhlIHByZWRpY3Rpb25zIGZyb20gZWFjaCBvZiB0aGUgdHJlZXMgaXMgdXNlZCBpbiB0aGUgZmluYWwgb3V0cHV0LgoKYGBge3IsIGVjaG8gPSBGQUxTRX0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoImh0dHBzOi8vbWlyby5tZWRpdW0uY29tL21heC8xNDAwLzAqZl9xUVBGcGRvZldHTFFxYy5wbmciKQpgYGAKCgpPdmVyYWxsLCBhIG1ham9yIGRpc3RpbmN0aW9uIHdpdGggb3VyIGxhc3QgcmVncmVzc2lvbiBtb2RlbCwgaXMgdGhhdCByYW5kb20gZm9yZXN0IGFsbG93cyB1cyB0byB1c2Ugb3VyIGNhdGVnb3JpY2FsIGRhdGEgbGFyZ2VseSBhcyBpcy4gVGhlcmUgaXMgbm8gbmVlZCB0byByZWNvZGUgdGhlc2UgcHJlZGljdG9ycyB0byBiZSBudW1lcmljYWwuIEluIG91ciBjYXNlLCB3ZSBhcmUgZ29pbmcgdG8gdXNlIHRoZSByYW5kb20gZm9yZXN0IG1ldGhvZCBvZiB0aGUgdGhlIGByYW5kb21Gb3Jlc3RgIHBhY2thZ2UuIAoKVGhpcyBwYWNrYWdlIGlzIGN1cnJlbnRseSBub3QgY29tcGF0aWJsZSB3aXRoIGNhdGVnb3JpY2FsIHZhcmlhYmxlcyB0aGF0IGhhdmUgbW9yZSB0aGFuIDUzIGxldmVscy4gU2VlIFtoZXJlXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvcmFuZG9tRm9yZXN0L05FV1MpIGZvciB0aGUgZG9jdW1lbnRhdGlvbiBhYm91dCB3aGVuIHRoaXMgd2FzIHVwZGF0ZWQgZnJvbSAyNSBsZXZlbHMuIFRodXMgd2Ugd2lsbCByZW1vdmUgdGhlIGB6Y3RhYCAgYW5kIGBjb3VudHlgIHZhcmlhYmxlcy4gVGhpcyBpcyB3aHkgaXQgaXMgZ29vZCB0aGF0IHdlIGhhdmUgbW9kaWZpZWQgdGhlIGNpdHkgdmFyaWFibGUgdG8gdHdvIGxldmVscywgYXMgdGhlcmUgd2VyZSBvcmlnaW5hbGx5IG5lYXJseSA2MDAsIHNvIHRoaXMgd291bGQgbm90IGhhdmUgYmVlbiBjb21wYXRpYmxlIHdpdGggdGhpcyBwYWNrYWdlLiAKCk5vdGUgdGhhdCB0aGUgYHN0ZXBfbm92ZWwoKWAgZnVuY3Rpb24gaXMgbmVjZXNzYXJ5IGhlcmUgZm9yIHRoZSBgc3RhdGVgIHZhcmlhYmxlIHRvIGdldCBhbGwgY3Jvc3MgdmFsaWRhdGlvbiBmb2xkcyB0byB3b3JrLCBiZWNhdXNlIHRoZXJlIHdpbGwgYmUgZGlmZmVyZW50IGxldmVscyBpbmNsdWRlZCBpbiBlYWNoIGZvbGQgdGVzdCBhbmQgdHJhaW5pbmcgc2V0cy4gVGhlIG5ldyBsZXZlbHMgZm9yIHNvbWUgb2YgdGhlIHRlc3Qgc2V0cyB3b3VsZCBvdGhlcndpc2UgcmVzdWx0IGluIGFuIGVycm9yLgoKQWNjb3JkaW5nIHRvIHRoZSBbZG9jdW1lbnRhdGlvbl0oaHR0cHM6Ly93d3cucmRvY3VtZW50YXRpb24ub3JnL3BhY2thZ2VzL3JlY2lwZXMvdmVyc2lvbnMvMC4xLjEzL3RvcGljcy9zdGVwX25vdmVsKSBmb3IgdGhlIGByZWNpcGVzYCBwYWNrYWdlOgoKPiBzdGVwX25vdmVsIGNyZWF0ZXMgYSBzcGVjaWZpY2F0aW9uIG9mIGEgcmVjaXBlIHN0ZXAgdGhhdCB3aWxsIGFzc2lnbiBhIHByZXZpb3VzbHkgdW5zZWVuIGZhY3RvciBsZXZlbCB0byBhIG5ldyB2YWx1ZS4KCmBgYHtyfQpSRl9yZWMgPC0gcmVjaXBlKHRyYWluX3BtKSAlPiUKICAgIHVwZGF0ZV9yb2xlKGV2ZXJ5dGhpbmcoKSwgbmV3X3JvbGUgPSAicHJlZGljdG9yIiklPiUKICAgIHVwZGF0ZV9yb2xlKHZhbHVlLCBuZXdfcm9sZSA9ICJvdXRjb21lIiklPiUKICAgIHVwZGF0ZV9yb2xlKGlkLCBuZXdfcm9sZSA9ICJpZCB2YXJpYWJsZSIpICU+JQogICAgdXBkYXRlX3JvbGUoImZpcHMiLCBuZXdfcm9sZSA9ICJjb3VudHkgaWQiKSAlPiUKICAgIHN0ZXBfbm92ZWwoInN0YXRlIikgJT4lCiAgICBzdGVwX3N0cmluZzJmYWN0b3IoInN0YXRlIiwgImNvdW50eSIsICJjaXR5IikgJT4lCiAgICBzdGVwX3JtKCJjb3VudHkiKSAlPiUKICAgIHN0ZXBfcm0oInpjdGEiKSAlPiUKICAgIHN0ZXBfY29ycihhbGxfbnVtZXJpYygpKSU+JQogICAgc3RlcF9uenYoYWxsX251bWVyaWMoKSkKYGBgCgpUaGUgYHJhbmRfZm9yZXN0KClgIGZ1bmN0aW9uIG9mIHRoZSBgcGFyc25pcGAgcGFja2FnZSBoYXMgdGhyZWUgaW1wb3J0YW50IGFyZ3VtZW50cyB0aGF0IGFjdCBhcyBhbiBpbnRlcmZhY2UgZm9yIHRoZSBkaWZmZXJlbnQgcG9zc2libGUgZW5naW5lcyB0byBwZXJmb3JtIGEgcmFuZG9tIGZvcmVzdCBhbmFseXNpczoKCjEuIGBtdHJ5YCAtIFRoZSBudW1iZXIgb2YgcHJlZGljdG9yIHZhcmlhYmxlcyAob3IgZmVhdHVyZXMpIHRoYXQgd2lsbCBiZSByYW5kb21seSBzYW1wbGVkIGF0IGVhY2ggc3BsaXQgd2hlbiBjcmVhdGluZyB0aGUgdHJlZSBtb2RlbHMuIFRoZSBkZWZhdWx0IG51bWJlciBmb3IgcmVncmVzc2lvbiBhbmFseXNlcyBpcyB0aGUgbnVtYmVyIG9mIHByZWRpY3RvcnMgZGl2aWRlZCBieSAzLiAKMi4gYG1pbl9uYCAtIFRoZSBtaW5pbXVtIG51bWJlciBvZiBkYXRhIHBvaW50cyBpbiBhIG5vZGUgdGhhdCBhcmUgcmVxdWlyZWQgZm9yIHRoZSBub2RlIHRvIGJlIHNwbGl0IGZ1cnRoZXIuCjMuIGB0cmVlc2AgLSB0aGUgbnVtYmVyIG9mIHRyZWVzIGluIHRoZSBlbnNlbWJsZQoKV2Ugd2lsbCBzdGFydCBieSB0cnlpbmcgYW4gYG10cnlgIHZhbHVlIG9mIDEwIGFuZCBhIGBtaW5fbmAgdmFsdWUgb2YgMy4gQXMgeW91IG1pZ2h0IGltYWdpbmUgaXQgaXMgYSBiaXQgZGlmZmljdWx0IHRvIGtub3cgd2hhdCB0byBjaG9vc2UuIFRoaXMgaXMgd2hlcmUgdGhlIHR1bmluZyBwcm9jZXNzIHRoYXQgd2UganVzdCBzdGFydGVkIHRvIGRlc2NyaWJlIHdpbGwgY29tZSBpbiBoZWxwZnVsIGFzIHdlIGNhbiB0ZXN0IGRpZmZlcmVudCBtb2RlbHMgd2l0aCBkaWZmZXJlbnQgdmFsdWVzLiBIb3dldmVyLCBmaXJzdCBsZXQncyBqdXN0IHN0YXJ0IHdpdGggdGhlc2UgdmFsdWVzLgoKTm93IHRoYXQgd2UgaGF2ZSBvdXIgcmVjaXBlIChgUkZfcmVjYCksIGxldCdzIHNwZWNpZnkgdGhlIG1vZGVsIHdpdGggYHJhbmRfZm9yZXN0KClgIGZyb20gYHBhcnNuaXBgLgoKYGBge3J9ClBNdHJlZV9tb2RlbCA8LSBwYXJzbmlwOjpyYW5kX2ZvcmVzdChtdHJ5ID0gMTAsIG1pbl9uID0gMykKUE10cmVlX21vZGVsCmBgYAoKTmV4dCwgd2Ugc2V0IHRoZSBlbmdpbmUgYW5kIG1vZGU6CgpOb3RlIHRoYXQgeW91IGNvdWxkIGFsc28gdXNlIHRoZSBgcmFuZ2VyYCBvciBgc3BhcmtgIHBhY2thZ2VzIGluc3RlYWQgb2YgYHJhbmRvbUZvcmVzdGAuCklmIHlvdSB3ZXJlIHRvIHVzZSB0aGUgYHJhbmdlcmAgcGFja2FnZSB0byBpbXBsZW1lbnQgdGhlIHJhbmRvbSBmb3Jlc3QgYW5hbHlzaXMgeW91IHdvdWxkIG5lZWQgdG8gc3BlY2lmeSBhbiBgaW1wb3J0YW5jZWAgYXJndW1lbnQgdG8gYmUgYWJsZSB0byBldmFsdWF0ZSBwcmVkaWN0b3IgaW1wb3J0YW5jZS4gIFRoZSBvcHRpb25zIGFyZSBgaW1wdXJpdHlgIG9yIGBwZXJtdXRhdGlvbmAuCgpUaGVzZSBvdGhlciBwYWNrYWdlcyBoYXZlIGRpZmZlcmVudCBhZHZhbnRhZ2VzIGFuZCBkaXNhZHZhbnRhZ2VzLSBmb3IgZXhhbXBsZSBgcmFuZ2VyYCBhbmQgYHNwYXJrYCBhcmUgbm90IGFzIGxpbWl0aW5nIGZvciB0aGUgbnVtYmVyIG9mIGNhdGVnb3JpZXMgZm9yIGNhdGVnb3JpY2FsIHZhcmlhYmxlcy4gRm9yIG1vcmUgaW5mb3JtYXRpb24gc2VlIHRoZWlyIGRvY3VtZW50YXRpb246IFtoZXJlXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvcmFuZ2VyL3Jhbmdlci5wZGYpIGZvciBgcmFuZ2VyYCwgW2hlcmVdKGh0dHA6Ly9zcGFyay5hcGFjaGUub3JnL2RvY3MvbGF0ZXN0L21sbGliLWVuc2VtYmxlcy5odG1sI3JhbmRvbS1mb3Jlc3RzKSBmb3IgYHNwYXJrYCwgYW5kIFtoZXJlXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvcmFuZG9tRm9yZXN0L3JhbmRvbUZvcmVzdC5wZGYpIGZvciBgcmFuZG9tRm9yZXN0YC4KClNlZSBbaGVyZV0oaHR0cHM6Ly9wYXJzbmlwLnRpZHltb2RlbHMub3JnL3JlZmVyZW5jZS9yYW5kX2ZvcmVzdC5odG1sKSBmb3IgbW9yZSBkb2N1bWVudGF0aW9uIGFib3V0IGltcGxlbWVudGluZyB0aGVzZSBlbmdpbmUgb3B0aW9ucyB3aXRoIHRpZHltb2RlbHMuIE5vdGUgdGhhdCB0aGVyZSBhcmUgYWxzbyBbb3RoZXJdKGh0dHBzOi8vd3d3LmxpbmtlZGluLmNvbS9wdWxzZS9kaWZmZXJlbnQtcmFuZG9tLWZvcmVzdC1wYWNrYWdlcy1yLW1hZGh1ci1tb2RpLykgUiBwYWNrYWdlcyBmb3IgaW1wbGVtZW50aW5nIHJhbmRvbSBmb3Jlc3QgYWxnb3JpdGhtcywgYnV0IHRoZXNlIHRocmVlIHBhY2thZ2VzIChgcmFuZ2VyYCwgYHNwYXJrYCwgYW5kIGByYW5kb21Gb3Jlc3RgKSBhcmUgY3VycmVudGx5IGNvbXBhdGlibGUgd2l0aCBgdGlkeW1vZGVsc2AuCgpXZSBhbHNvIG5lZWQgdG8gc3BlY2lmeSB3aXRoIHRoZSBgc2V0X21vZGUoKWAgZnVuY3Rpb24gdGhhdCBvdXIgb3V0Y29tZSB2YXJpYWJsZSAoYWlyIHBvbGx1dGlvbikgaXMgY29udGludW91cy4gCgpgYGB7cn0KClJGX1BNX21vZGVsIDwtIFBNdHJlZV9tb2RlbCAlPiUKICBzZXRfZW5naW5lKCJyYW5kb21Gb3Jlc3QiKSAlPiUKICBzZXRfbW9kZSgicmVncmVzc2lvbiIpCgpSRl9QTV9tb2RlbApgYGAKClRoZW4sIHdlIHB1dCB0aGlzIGFsbCB0b2dldGhlciBpbnRvIGEgYHdvcmtmbG93YDogCgojIyMjIHsucmVjYWxsX2NvZGVfcXVlc3Rpb25fYmxvY2t9CjxiPjx1PiBRdWVzdGlvbiBPcHBvcnR1bml0eSA8L3U+PC9iPgoKU2VlIGlmIHlvdSBjYW4gY29tZSB1cCB3aXRoIHRoZSBjb2RlIHRvIGRvIHRoaXMuCgoqKioKPGRldGFpbHM+IDxzdW1tYXJ5PiBDbGljayBoZXJlIHRvIHJldmVhbCB0aGUgYW5zd2VyLiA8L3N1bW1hcnk+CgpgYGB7cn0KUkZfd2Zsb3cgPC0gd29ya2Zsb3dzOjp3b3JrZmxvdygpICU+JQogIHdvcmtmbG93czo6YWRkX3JlY2lwZShSRl9yZWMpICU+JQogIHdvcmtmbG93czo6YWRkX21vZGVsKFJGX1BNX21vZGVsKQoKYGBgCjwvZGV0YWlscz4gCgoqKioKCiMjIyMKCmBgYHtyfQpSRl93ZmxvdwpgYGAKCgpGaW5hbGx5LCB3ZSBmaXQgdGhlIGRhdGEgdG8gdGhlIG1vZGVsOgoKIyMjIyB7LnJlY2FsbF9jb2RlX3F1ZXN0aW9uX2Jsb2NrfQo8Yj48dT4gUXVlc3Rpb24gT3Bwb3J0dW5pdHkgPC91PjwvYj4KCkRvIHlvdSByZWNhbGwgaG93IHRvIGRvIHRoaXM/CgoqKioKPGRldGFpbHM+IDxzdW1tYXJ5PiBDbGljayBoZXJlIHRvIHJldmVhbCB0aGUgYW5zd2VyLiA8L3N1bW1hcnk+CgpgYGB7cn0KUkZfd2Zsb3dfZml0IDwtIHBhcnNuaXA6OmZpdChSRl93ZmxvdywgZGF0YSA9IHRyYWluX3BtKQpgYGAKCklmIHlvdSBnZXQgYW4gZXJyb3IgIkNhbiBub3QgaGFuZGxlIGNhdGVnb3JpY2FsIHByZWRpY3RvcnMgd2l0aCBtb3JlIHRoYW4gNTMgY2F0ZWdvcmllcy4iIHRoZW4geW91IHNob3VsZCBzY3JvbGwgdXAgYSBiaXQgYW5kIG1ha2Ugc3VyZSB0aGF0IHlvdSByZW1vdmVkIHRoZSBjYXRlZ29yaWNhbCB2YXJpYWJsZXMgdGhhdCBoYXZlIG1vcmUgdGhhbiA1MyBjYXRlZ29yaWVzIGFzIHRoaXMgbWV0aG9kIGNhbid0IGhhbmRsZSBzdWNoIHZhcmlhYmxlcyBhdCB0aGlzIHRpbWUuCgo8L2RldGFpbHM+IAoKKioqCgojIyMjCgpgYGB7cn0KUkZfd2Zsb3dfZml0CmBgYAoKTGV0J3MgdGFrZSBhIGxvb2sgYXQgdGhlIHRvcCAxMCBjb250cmlidXRpbmcgdmFyaWFibGVzOgoKIyMjIyB7LnJlY2FsbF9jb2RlX3F1ZXN0aW9uX2Jsb2NrfQo8Yj48dT4gUXVlc3Rpb24gT3Bwb3J0dW5pdHkgPC91PjwvYj4KClNlZSBpZiB5b3UgY2FuIHJlY2FsbCBob3cgdG8gZG8gdGhpcy4KCgpgYGB7ciwgZWNobyA9IEZBTFNFfQpSRl93Zmxvd19maXQgJT4lIAogIGV4dHJhY3RfZml0X3BhcnNuaXAoKSAlPiUgCiAgdmlwKG51bV9mZWF0dXJlcyA9IDEwKQoKYGBgCgoKKioqCjxkZXRhaWxzPiA8c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byByZXZlYWwgdGhlIGFuc3dlci4gPC9zdW1tYXJ5PgoKYGBge3J9ClJGX3dmbG93X2ZpdCAlPiUgCiAgZXh0cmFjdF9maXRfcGFyc25pcCgpICU+JSAKICB2aXAobnVtX2ZlYXR1cmVzID0gMTApCmBgYAo8L2RldGFpbHM+CgoqKioKCiMjIyMKCgpJbnRlcmVzdGluZyEgSW4gdGhlIHByZXZpb3VzIG1vZGVsIHRoZSBDTUFRIHZhbHVlcyBhbmQgdGhlIHN0YXRlIHdoZXJlIHRoZSBtb25pdG9yIHdhcyBsb2NhdGVkIChiZWluZyBpbiBDYWxpZm9ybmlhIG9yIG5vdCkgd2VyZSBhbHNvIHRoZSB0b3AgdHdvIG1vc3QgaW1wb3J0YW50LCBob3dldmVyIHByZWRpY3RvcnMgYWJvdXQgZWR1Y2F0aW9uIGxldmVscyBvZiB0aGUgY29tbXVuaXRpZXMgd2hlcmUgdGhlIG1vbml0b3Igd2FzIGxvY2F0ZWQgd2FzIGFtb25nIHRoZSB0b3AgbW9zdCBpbXBvcnRhbnQuIE5vdyB3ZSBzZWUgdGhhdCBwb3B1bGF0aW9uIGRlbnNpdHkgYW5kIHByb3hpbWl0eSB0byBzb3VyY2VzIG9mIGVtaXNzaW9ucyBhbmQgcm9hZHMgYXJlIGFtb25nIHRoZSB0b3AgdGVuLgoKCk5vdyBsZXQncyB0YWtlIGEgbG9vayBhdCBtb2RlbCBwZXJmb3JtYW5jZSBieSBmaXR0aW5nIHRoZSBkYXRhIHVzaW5nIGNyb3NzIHZhbGlkYXRpb246CgojIyMjIHsucmVjYWxsX2NvZGVfcXVlc3Rpb25fYmxvY2t9CjxiPjx1PiBRdWVzdGlvbiBPcHBvcnR1bml0eSA8L3U+PC9iPgoKU2VlIGlmIHlvdSBjYW4gcmVjYWxsIGhvdyB0byBkbyB0aGlzLgoKKioqCjxkZXRhaWxzPiA8c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byByZXZlYWwgdGhlIGFuc3dlci4gPC9zdW1tYXJ5PgoKYGBge3IsIGV2YWwgPSBGQUxTRX0Kc2V0LnNlZWQoNDU2KQpyZXNhbXBsZV9SRl9maXQgPC0gdHVuZTo6Zml0X3Jlc2FtcGxlcyhSRl93ZmxvdywgdmZvbGRfcG0pCmNvbGxlY3RfbWV0cmljcyhyZXNhbXBsZV9SRl9maXQpCmBgYAoKPC9kZXRhaWxzPgoKKioqCgojIyMjCgpgYGB7ciwgZWNobyA9IEZBTFNFfQpzZXQuc2VlZCg0NTYpCnJlc2FtcGxlX1JGX2ZpdCA8LSB0dW5lOjpmaXRfcmVzYW1wbGVzKFJGX3dmbG93LCB2Zm9sZF9wbSkKY29sbGVjdF9tZXRyaWNzKHJlc2FtcGxlX1JGX2ZpdCkKYGBgCgojIyAqKk1vZGVsIENvbXBhcmlzb24qKgoqKioKCk5vdyBsZXQncyBjb21wYXJlIHRoZSBwZXJmb3JtYW5jZSBvZiBvdXIgbW9kZWwgd2l0aCBvdXIgbGluZWFyIHJlZ3Jlc3Npb24gbW9kZWwgLSBpZiB5b3UgaGF2ZSBiZWVuIGZvbGxvd2luZyBhbG9uZyB5b3UgY291bGQgdHlwZSB0aGlzIHRvIHRha2UgYSBsb29rLiAKCmBgYHtyLCBldmFsID0gRkFMU0V9CiMgb3VyIGluaXRpYWwgbGluZWFyIHJlZ3Jlc3Npb24gbW9kZWw6CmNvbGxlY3RfbWV0cmljcyhyZXNhbXBsZV9maXQpCmBgYAoKRm9yIHRob3NlIHN0YXJ0aW5nIGhlcmUgd2Ugd2lsbCB0ZWxsIHlvdSB0aGF0IG91ciBmaXJzdCBtb2RlbCBoYWQgYSBjcm9zcyB2YWxpZGF0aW9uIG1lYW4gYHJtc2VgIHZhbHVlIG9mIGByIHJvdW5kKGNvbGxlY3RfbWV0cmljcyhyZXNhbXBsZV9maXQpJG1lYW5bMV0sIDIpYC4KSXQgbG9va3MgbGlrZSB0aGUgcmFuZG9tIGZvcmVzdCBtb2RlbCBoYWQgIGEgbXVjaCBsb3dlciBgcm1zZWAgdmFsdWUgb2YgYHIgcm91bmQoY29sbGVjdF9tZXRyaWNzKHJlc2FtcGxlX1JGX2ZpdCkkbWVhblsxXSwgMilgLiBUaGlzIHN1Z2dlc3QgdGhhdCB0aGlzIG1vZGVsIGlzIGJldHRlciBhdCBwcmVkaWN0aW9uIGFpciBwb2xsdXRpb24gdmFsdWVzLiBJbiBhZGRpdGlvbiwgdGhlIFIgc3F1YXJlZCB2YWx1ZSBpcyBtdWNoIGhpZ2hlciAoaXQgd2FzIGByIHJvdW5kKGNvbGxlY3RfbWV0cmljcyhyZXNhbXBsZV9maXQpJG1lYW5bMl0sIDIpKjEwMGAlIGFuZCBpcyBub3cgYHIgcm91bmQoY29sbGVjdF9tZXRyaWNzKHJlc2FtcGxlX1JGX2ZpdCkkbWVhblsyXSwgMikqMTAwYCUpLCBzdWdnZXN0aW5nIHRoYXQgbW9yZSBvZiB0aGUgdmFyaWFuY2Ugb2YgdGhlIGFpciBwb2xsdXRpb24gdmFsdWVzIGNvdWxkIGJlIGV4cGxhaW5lZCBieSB0aGUgbmV3IFJhbmRvbSBGb3Jlc3QgbW9kZWwuIAoKIyMjIyB7LnRoaW5rX3F1ZXN0aW9uX2Jsb2NrfQo8Yj48dT4gUXVlc3Rpb24gT3Bwb3J0dW5pdHkgPC91PjwvYj4KCkRvIHlvdSByZWNhbGwgaG93IHRoZSBSTVNFIGlzIGNhbGN1bGF0ZWQ/CgoqKioKCjxkZXRhaWxzPiA8c3VtbWFyeT4gQ2xpY2sgaGVyZSB0byByZXZlYWwgdGhlIGFuc3dlci4gPC9zdW1tYXJ5PgokJFJNU0UgPSBcc3FydHtcZnJhY3tcc3VtX3tpPTF9XntufXsoXGhhdHt5X3R9LSB5X3QpfV4yfXtufX0kJAo8L2RldGFpbHM+CgoqKioKCiMjIyMKCklmIHdlIHR1bmVkIG91ciByYW5kb20gZm9yZXN0IG1vZGVsIGJhc2VkIG9uIHRoZSBudW1iZXIgb2YgdHJlZXMgb3IgdGhlIHZhbHVlIGZvciBgbXRyeWAgKHdoaWNoIGlzICJUaGUgbnVtYmVyIG9mIHByZWRpY3RvcnMgdGhhdCB3aWxsIGJlIHJhbmRvbWx5IHNhbXBsZWQgYXQgZWFjaCBzcGxpdCB3aGVuIGNyZWF0aW5nIHRoZSB0cmVlIG1vZGVscyIpLCB3ZSBtaWdodCBnZXQgYSBtb2RlbCB3aXRoIGV2ZW4gYmV0dGVyIHBlcmZvcm1hbmNlLgoKSG93ZXZlciwgb3VyIGNyb3NzIHZhbGlkYXRlZCBtZWFuIHJtc2UgdmFsdWUgb2YgYHIgcm91bmQoY29sbGVjdF9tZXRyaWNzKHJlc2FtcGxlX1JGX2ZpdCkkbWVhblsxXSwgMilgIGlzIHF1aXRlIGdvb2QgYmVjYXVzZSBvdXIgcmFuZ2Ugb2YgdHJ1ZSBvdXRjb21lIHZhbHVlcyBpcyBtdWNoIGxhcmdlcjogKGByIHJvdW5kKHJhbmdlKHRlc3RfcG0kdmFsdWUpLDMpYCkuCgoKIyMgKipNb2RlbCB0dW5pbmcqKgoqKioKCltIeXBlcnBhcmFtZXRlcnNdKGh0dHBzOi8vbWFjaGluZWxlYXJuaW5nbWFzdGVyeS5jb20vZGlmZmVyZW5jZS1iZXR3ZWVuLWEtcGFyYW1ldGVyLWFuZC1hLWh5cGVycGFyYW1ldGVyLykgYXJlIG9mdGVuIHRoaW5ncyB0aGF0IHdlIG5lZWQgdG8gc3BlY2lmeSBhYm91dCBhIG1vZGVsLiBGb3IgZXhhbXBsZSwgdGhlIG51bWJlciBvZiBwcmVkaWN0b3IgdmFyaWFibGVzIChvciBmZWF0dXJlcykgdGhhdCB3aWxsIGJlIHJhbmRvbWx5IHNhbXBsZWQgYXQgZWFjaCBzcGxpdCB3aGVuIGNyZWF0aW5nIHRoZSB0cmVlIG1vZGVscyBjYWxsZWQgYG10cnlgIGlzIGEgaHlwZXJwYXJhbWV0ZXIuIFRoZSBkZWZhdWx0IG51bWJlciBmb3IgcmVncmVzc2lvbiBhbmFseXNlcyBpcyB0aGUgbnVtYmVyIG9mIHByZWRpY3RvcnMgZGl2aWRlZCBieSAzLiBJbnN0ZWFkIG9mIGFyYml0cmFyaWx5IHNwZWNpZnlpbmcgdGhpcywgd2UgY2FuIHRyeSB0byBkZXRlcm1pbmUgdGhlIGJlc3Qgb3B0aW9uIGZvciBtb2RlbCBwZXJmb3JtYW5jZSBieSBhIHByb2Nlc3MgY2FsbGVkIHR1bmluZy4gCgoKTm93IGxldCdzIHRyeSBzb21lIHR1bmluZy4KCkxldCdzIHRha2UgYSBjbG9zZXIgbG9vayBhdCB0aGUgYG10cnlgIGFuZCBgbWluX25gIGh5cGVycGFyYW1ldHJzIGluIG91ciBSYW5kb20gRm9yZXN0IG1vZGVsLgoKV2UgYXJlbid0IGV4YWN0bHkgc3VyZSB3aGF0IHZhbHVlcyBvZiBgbXRyeWAgYW5kIGBtaW5fbmAgYWNoaWV2ZSBnb29kIGFjY3VyYWN5IHlldCBrZWVwIG91ciBtb2RlbCBnZW5lcmFsaXphYmxlIGZvciBvdGhlciBkYXRhLgoKVGhpcyBpcyB3aGVuIG91ciBjcm9zcyB2YWxpZGF0aW9uIG1ldGhvZHMgYmVjb21lIHJlYWxseSBoYW5keSBiZWNhdXNlIG5vdyB3ZSBjYW4gdGVzdCBvdXQgZGlmZmVyZW50IHZhbHVlcyBmb3IgZWFjaCBvZiB0aGVzZSBoeXBlcnBhcmFtZXRlcnMgdG8gYXNzZXNzIHdoYXQgdmFsdWVzIHNlZW0gdG8gd29yayBiZXN0IGZvciBtb2RlbCBwZXJmb3JtYW5jZSBvbiB0aGVzZSByZXNhbXBsZXMgb2Ygb3VyIHRyYWluaW5nIHNldCBkYXRhLgoKUHJldmlvdXNseSB3ZSBzcGVjaWZpZWQgb3VyIG1vZGVsIGxpa2Ugc286CmBgYHtyfQpSRl9QTV9tb2RlbCA8LSAKICBwYXJzbmlwOjpyYW5kX2ZvcmVzdChtdHJ5ID0gMTAsIG1pbl9uID0gMykgJT4lCiAgc2V0X2VuZ2luZSgicmFuZG9tRm9yZXN0IikgJT4lCiAgc2V0X21vZGUoInJlZ3Jlc3Npb24iKQoKUkZfUE1fbW9kZWwKYGBgCgpOb3cgaW5zdGVhZCBvZiBzcGVjaWZ5aW5nIGEgdmFsdWUgZm9yIHRoZSBgbXRyeWAgYW5kIGBtaW5fbmAgYXJndW1lbnRzLCB3ZSBjYW4gdXNlIHRoZSBgdHVuZSgpYCBmdW5jdGlvbiBvZiB0aGUgYHR1bmVgIHBhY2thZ2UgbGlrZSBzbzogYG10cnkgPSB0dW5lKClgLiBUaGlzIGluZGljYXRlcyB0aGF0IHRoZXNlIGh5cGVycGFyYW1ldGVycyBhcmUgdG8gYmUgdHVuZWQuIAoKYGBge3J9Cgp0dW5lX1JGX21vZGVsIDwtIHJhbmRfZm9yZXN0KG10cnkgPSB0dW5lKCksIG1pbl9uID0gdHVuZSgpKSAlPiUKICBzZXRfZW5naW5lKCJyYW5kb21Gb3Jlc3QiKSAlPiUKICBzZXRfbW9kZSgicmVncmVzc2lvbiIpCiAgICAKdHVuZV9SRl9tb2RlbAoKYGBgCgoKQWdhaW4gd2Ugd2lsbCBhZGQgdGhpcyB0byBhIHdvcmtmbG93LCB0aGUgb25seSBkaWZmZXJlbmNlIGhlcmUgaXMgdGhhdCB3ZSBhcmUgdXNpbmcgYSBkaWZmZXJlbnQgbW9kZWwgc3BlY2lmaWNhdGlvbiB3aXRoIGB0dW5lX1JGX21vZGVsYCBpbnN0ZWFkIG9mIGBSRl9tb2RlbGA6CgpgYGB7cn0KClJGX3R1bmVfd2Zsb3cgPC0gd29ya2Zsb3dzOjp3b3JrZmxvdygpICU+JQogIHdvcmtmbG93czo6YWRkX3JlY2lwZShSRl9yZWMpICU+JQogIHdvcmtmbG93czo6YWRkX21vZGVsKHR1bmVfUkZfbW9kZWwpClJGX3R1bmVfd2Zsb3cKCmBgYAoKCk5vdyB3ZSBjYW4gdXNlIHRoZSBgdHVuZV9ncmlkKClgIGZ1bmN0aW9uIG9mIHRoZSBgdHVuZWAgcGFja2FnZSB0byBldmFsdWF0ZSBkaWZmZXJlbnQgY29tYmluYXRpb25zIG9mIHZhbHVlcyBmb3IgYG10cnlgIGFuZCBgbWluX25gIHVzaW5nIG91ciBjcm9zcyB2YWxpZGF0aW9uIHNhbXBsZXMgb2Ygb3VyIHRyYWluaW5nIHNldCAoYHZmb2xkX3BtYCkgdG8gc2VlIHdoYXQgY29tYmluYXRpb24gb2YgdmFsdWVzIHBlcmZvcm1zIGJlc3QuCgpUbyB1c2UgdGhpcyBmdW5jdGlvbiB3ZSB3aWxsIHNwZWNpZnkgdGhlIHdvcmtmbG93IHVzaW5nIHRoZSBgb2JqZWN0YCBhcmd1bWVudCAgYW5kIHRoZSBzYW1wbGVzIHRvIHVzZSB1c2luZyB0aGUgYHJlc2FtcGxlc2AgYXJndW1lbnQuIFRoZSBgZ3JpZGAgYXJndW1lbnQgc3BlY2lmaWVzIGhvdyBtYW55IHBvc3NpYmxlIG9wdGlvbnMgZm9yIGVhY2ggYXJndW1lbnQgc2hvdWxkIGJlIGF0dGVtcHRlZC4KCkJ5IGRlZmF1bHQgMTAgZGlmZmVyZW50IHZhbHVlcyB3aWxsIGJlIGF0dGVtcHRlZCBmb3IgZWFjaCBoeXBlcnBhcmFtZXRlciB0aGF0IGlzIGJlaW5nIHR1bmVkLgoKV2UgY2FuIHVzZSB0aGUgYGRvUGFyYWxsZWxgIHBhY2thZ2UgdG8gYWxsb3cgdXMgdG8gZml0IGFsbCB0aGVzZSBtb2RlbHMgdG8gb3VyIGNyb3NzIHZhbGlkYXRpb24gc2FtcGxlcyBmYXN0ZXIuIFNvIGlmIHlvdSB3ZXJlIHBlcmZvcm1pbmcgdGhpcyBvbiBhIGNvbXB1dGVyIHdpdGggbXVsdGlwbGUgY29yZXMgb3IgcHJvY2Vzc29ycywgdGhlbiBkaWZmZXJlbnQgbW9kZWxzIHdpdGggZGlmZmVyZW50IGh5cGVycGFyYW1ldGVyIHZhbHVlcyBjYW4gYmUgZml0IHRvIHRoZSBjcm9zcyB2YWxpZGF0aW9uIHNhbXBsZXMgc2ltdWx0YW5lb3VzbHkgYWNyb3NzIGRpZmZlcmVudCBjb3JlcyBvciBwcm9jZXNzb3JzLiAKCllvdSBjYW4gc2VlIGhvdyBtYW55IGNvcmVzIHlvdSBoYXZlIGFjY2VzcyB0byBvbiB5b3VyIHN5c3RlbSB1c2luZyB0aGUgYGRldGVjdENvcmVzKClgIGZ1bmN0aW9uIGluIHRoZSBgcGFyYWxsZWxgIHBhY2thZ2UuIAoKYGBge3J9CnBhcmFsbGVsOjpkZXRlY3RDb3JlcygpCmBgYAoKVGhlIGByZWdpc3RlckRvUGFyYWxsZWwoKWAgZnVuY3Rpb24gd2lsbCB1c2UgdGhlIG51bWJlciBmb3IgY29yZXMgc3BlY2lmaWVkIHVzaW5nIHRoZSBgY29yZXM9YCBhcmd1ZW1lbnQsIG9yIGl0IHdpbGwgYXNzaWduIGl0IGF1dG9tYXRpY2FsbHkgdG8gb25lLWhhbGYgb2YgdGhlIG51bWJlciBvZiBjb3JlcyBkZXRlY3RlZCBieSB0aGUgYHBhcmFsbGVsYCBwYWNrYWdlLiAKCldlIG5lZWQgdG8gdXNlIGBzZXQuc2VlZCgpYCBoZXJlIGJlY2F1c2UgdGhlIHZhbHVlcyBjaG9zZW4gZm9yIGBtdHJ5YCBhbmQgYG1pbl9uYCBtYXkgdmFyeSBpZiB3ZSBwcmVmb3JtIHRoaXMgZXZhbHVhdGlvbiBhZ2FpbiBiZWNhdXNlIHRoZXkgYXJlIGNob3NlbiBzZW1pLXJhbmRvbWx5IChtZWFuaW5nIHRoYXQgdGhleSBhcmUgd2l0aGluIGEgcmFuZ2Ugb2YgcmVhc29uYWJsZSB2YWx1ZXMgYnV0IHN0aWxsIHJhbmRvbSkuCgpOb3RlOiB0aGlzIHN0ZXAgd2lsbCB0YWtlIHNvbWUgdGltZS4KCmBgYHtyfQpkb1BhcmFsbGVsOjpyZWdpc3RlckRvUGFyYWxsZWwoY29yZXMgPSAyKQpzZXQuc2VlZCgxMjMpCnR1bmVfUkZfcmVzdWx0cyA8LSB0dW5lX2dyaWQob2JqZWN0ID0gUkZfdHVuZV93ZmxvdywgcmVzYW1wbGVzID0gdmZvbGRfcG0sIGdyaWQgPSAyMCkKdHVuZV9SRl9yZXN1bHRzCmBgYAoKClNlZSBbdGhlIHR1bmUgZ2V0dGluZyBzdGFydGVkIGd1aWRlIF0oaHR0cHM6Ly90aWR5bW9kZWxzLmdpdGh1Yi5pby90dW5lL2FydGljbGVzL2dldHRpbmdfc3RhcnRlZC5odG1sKXt0YXJnZXQ9Il9ibGFuayJ9IGZvciBtb3JlIGluZm9ybWF0aW9uIGFib3V0IGltcGxlbWVudGluZyB0aGlzIGluIGB0aWR5bW9kZWxzYC4KCklmIHlvdSB3YW50ZWQgbW9yZSBjb250cm9sIG92ZXIgdGhpcyBwcm9jZXNzIHlvdSBjb3VsZCBzcGVjaWZ5IGhvdyB0aGUgZGlmZmVyZW50IHBvc3NpYmxlIG9wdGlvbnMgZm9yIGBtdHJ5YCBhbmQgYG1pbl9uYCBpbiB0aGUgYHR1bmVfZ3JpZCgpYCBmdW5jdGlvbiB1c2luZyB0aGUgYGdyaWRfKigpYCBmdW5jdGlvbnMgb2YgdGhlIGBkaWFsc2AgcGFja2FnZSB0byBjcmVhdGUgYSBtb3JlIHNwZWNpZmljIGdyaWQuCgpCeSBkZWZhdWx0IHRoZSB2YWx1ZXMgZm9yIHRoZSBoeXBlcnBhcmFtZXRlcnMgYmVpbmcgdHVuZWQgYXJlIGNob3NlbiBzZW1pLXJhbmRvbWx5IChtZWFuaW5nIHRoYXQgdGhleSBhcmUgd2l0aGluIGEgcmFuZ2Ugb2YgcmVhc29uYWJsZSB2YWx1ZXMgYnV0IHN0aWxsIHJhbmRvbSkuLgoKCk5vdyB3ZSBjYW4gdXNlIHRoZSBgY29sbGVjdF9tZXRyaWNzKClgIGZ1bmN0aW9uIGFnYWluIHRvIHRha2UgYSBsb29rIGF0IHdoYXQgaGFwcGVuZWQgd2l0aCBvdXIgY3Jvc3MgdmFsaWRhdGlvbiB0ZXN0cy4gV2UgY2FuIHNlZSB0aGUgZGlmZmVyZW50IHZhbHVlcyBjaG9zZW4gZm9yIGBtdHJ5YCBhbmQgYG1pbl9uYCBhbmQgdGhlIG1lYW4gcm1zZSBhbmQgcnNxIHZhbHVlcyBhY3Jvc3MgdGhlIGNyb3NzIHZhbGlkYXRpb24gc2FtcGxlcy4KCmBgYHtyfQp0dW5lX1JGX3Jlc3VsdHMgJT4lCiAgY29sbGVjdF9tZXRyaWNzKCkKYGBgCgpXZSBjYW4gbm93IHVzZSB0aGUgYHNob3dfYmVzdCgpYCBmdW5jdGlvbiAod2hpY2ggd2UgdXNlZCBwcmV2aW91c2x5IHRvIGdldCBhIGJldHRlciBlc3RpbWF0ZSBvZiB0aGUgUk1TRSB1c2luZyBjcm9zcyB2YWxpZGF0aW9uIGZvbGQgb2YgdGhlIHRyYWluaW5nIGRhdGEpIGFzIGl0IHdhcyAqKnRydWx5IGludGVuZGVkKiosIHRvIHNlZSB3aGF0IHZhbHVlcyBmb3IgYG1pbl9uYCBhbmQgYG10cnlgIHJlc3VsdGVkIGluIHRoZSBiZXN0IHBlcmZvcm1hbmNlLgoKYGBge3J9CnNob3dfYmVzdCh0dW5lX1JGX3Jlc3VsdHMsIG1ldHJpYyA9ICJybXNlIiwgbiA9IDEpCmBgYAoKVGhlcmUgd2UgaGF2ZSBpdC4uLiBsb29rcyBsaWtlIGFuIGBtdHJ5YCBvZiBgciBzaG93X2Jlc3QodHVuZV9SRl9yZXN1bHRzLCBtZXRyaWMgPSAicm1zZSIsIG4gPSAxKSRtdHJ5YCBhbmQgYHNob3dfYmVzdCh0dW5lX1JGX3Jlc3VsdHMsIG1ldHJpYyA9ICJybXNlIiwgbiA9IDEpJG1pbl9uYCBvZiA0IGhhZCB0aGUgYmVzdCBgcm1zZWAgdmFsdWUuIFlvdSBjYW4gdmVyaWZ5IHRoaXMgaW4gdGhlIGFib3ZlIG91dHB1dCwgYnV0IGl0IGlzIGVhc2llciB0byBqdXN0IHB1bGwgdGhpcyByb3cgb3V0IHVzaW5nIHRoaXMgZnVuY3Rpb24uIFdlIGNhbiBzZWUgdGhhdCB0aGUgbWVhbiBgcm1zZWAgdmFsdWUgYWNyb3NzIHRoZSBjcm9zcyB2YWxpZGF0aW9uIHNldHMgd2FzIGByIHJvdW5kKHNob3dfYmVzdCh0dW5lX1JGX3Jlc3VsdHMsIG1ldHJpYyA9ICJybXNlIiwgbiA9IDEpJG1lYW4sIDIpYC4gQmVmb3JlIHR1bmluZyBpdCB3YXMgYHIgcm91bmQoY29sbGVjdF9tZXRyaWNzKHJlc2FtcGxlX1JGX2ZpdCkkbWVhblsxXSwgMilgICB3aXRoIGEgZmFpcmx5IHNpbWlsYXIgYHN0ZF9lcnJgIHNvIHRoZSBwZXJmb3JtYW5jZSBtYXkgYmUgc2xpZ2h0bHkgaW1wcm92ZWQuIEluIG90aGVyIHdvcmRzIHRoZSBtb2RlbCB3YXMgc2xpZ2h0bHkgYmV0dGVyIGF0IHByZWRpY3RpbmcgYWlyIHBvbGx1dGlvbiB2YWx1ZXMuCgoKIyMgKipGaW5hbCBtb2RlbCBwZXJmb3JtYW5jZSBldmFsdWF0aW9uKioKKioqCgpOb3cgdGhhdCB3ZSBoYXZlIGRlY2lkZWQgdGhhdCB3ZSBoYXZlIHJlYXNvbmFibGUgcGVyZm9ybWFuY2Ugd2l0aCBvdXIgdHJhaW5pbmcgZGF0YSB1c2luZyB0aGUgUmFuZG9tIEZvcmVzdCBtZXRob2QgYW5kIHR1bmluZywgd2UgY2FuIHN0b3AgYnVpbGRpbmcgb3VyIG1vZGVsIGFuZCBldmFsdWF0ZSBwZXJmb3JtYW5jZSB3aXRoIG91ciB0ZXN0aW5nIGRhdGEuIE5vdGUgdGhhdCB5b3UgbWlnaHQgb2Z0ZW4gbmVlZCB0byB0cnkgYSB2YXJpZXR5IG9mIG1ldGhvZHMgdG8gb3B0aW1pemUgeW91ciBtb2RlbC4KCkhlcmUsIHdlIHdpbGwgdXNlIHRoZSByYW5kb20gZm9yZXN0IG1vZGVsIHRoYXQgd2UgYnVpbHQgdG8gcHJlZGljdCB2YWx1ZXMgZm9yIHRoZSBtb25pdG9ycyBpbiB0aGUgdGVzdGluZyBkYXRhIGFuZCB3ZSB3aWxsIHVzZSB0aGUgdmFsdWVzIGZvciBgbXRyeWAgYW5kIGBtaW5fbmAgdGhhdCB3ZSBqdXN0IGRldGVybWluZWQgYmFzZWQgb24gb3VyIHR1bmluZyBhbmFseXNpcyB0byBhY2hpZXZlIHRoZSBiZXN0IHBlcmZvcm1hbmNlLgoKU28sIGZpcnN0IHdlIG5lZWQgdG8gc3BlY2lmeSB0aGVzZSB2YWx1ZXMgaW4gYSB3b3JrZmxvdy4gV2UgY2FuIHVzZSB0aGUgYHNlbGVjdF9iZXN0KClgIGZ1bmN0aW9uIG9mIHRoZSBgdHVuZWAgcGFja2FnZSB0byBncmFiIHRoZSB2YWx1ZXMgdGhhdCB3ZXJlIGRldGVybWluZWQgdG8gYmUgYmVzdCBmb3IgYG10cnlgIGFuZCBgbWluX25gLgoKCgpgYGB7cn0KCnR1bmVkX1JGX3ZhbHVlczwtIHNlbGVjdF9iZXN0KHR1bmVfUkZfcmVzdWx0cywgInJtc2UiKQp0dW5lZF9SRl92YWx1ZXMKYGBgCgpOb3cgd2UgY2FuIGZpbmFsaXplIHRoZSBtb2RlbC93b3JrZmxvdyB0aGF0IHdlIHdlIHVzZWQgZm9yIHR1bmluZyB3aXRoIHRoZXNlIHZhbHVlcy4KCgpgYGB7cn0KUkZfdHVuZWRfd2Zsb3cgPC1SRl90dW5lX3dmbG93ICU+JQogIHR1bmU6OmZpbmFsaXplX3dvcmtmbG93KHR1bmVkX1JGX3ZhbHVlcykKYGBgCgoKV2l0aCB0aGUgYHdvcmtmbG93c2AgcGFja2FnZSwgd2UgY2FuIHVzZSB0aGUgc3BsaXR0aW5nIGluZm9ybWF0aW9uIGZvciBvdXIgb3JpZ2luYWwgZGF0YSBgcG1fc3BsaXRgIHRvIGZpdCB0aGUgZmluYWwgbW9kZWwgb24gdGhlIGZ1bGwgdHJhaW5pbmcgc2V0IGFuZCBhbHNvIG9uIHRoZSB0ZXN0aW5nIGRhdGEgdXNpbmcgdGhlIGBsYXN0X2ZpdCgpYCBmdW5jdGlvbiBvZiB0aGUgYHR1bmVgIHBhY2thZ2UuIE5vIHByZS1wcm9jZXNzaW5nIHN0ZXBzIGFyZSByZXF1aXJlZC4KClRoZSByZXN1bHRzIHdpbGwgc2hvdyB0aGUgcGVyZm9ybWFuY2UgdXNpbmcgdGhlIHRlc3RpbmcgZGF0YS4KCgpgYGB7cn0Kb3ZlcmFsbGZpdCA8LSB0dW5lOjpsYXN0X2ZpdChSRl90dW5lZF93ZmxvdywgcG1fc3BsaXQpCiAjIG9yCm92ZXJhbGxmaXQgPC0gUkZfd2Zsb3cgJT4lCiAgdHVuZTo6bGFzdF9maXQocG1fc3BsaXQpCmBgYAoKVGhlIGBvdmVyYWxsZml0YCBvdXRwdXQgaGFzIGEgbG90IG9mIHJlYWxseSB1c2VmdWwgaW5mb3JtYXRpb24gYWJvdXQgdGhlIG1vZGVsLCB0aGUgdGVzdGluZyBhbmQgdHJhaW5pbmcgZGF0YSBzcGxpdCwgYW5kIHRoZSBwcmVkaWN0aW9ucyBmb3IgdGhlIHRlc3RpbmcgZGF0YS4KClRvIHNlZSB0aGUgcGVyZm9ybWFuY2Ugb24gdGhlIHRlc3QgZGF0YSB3ZSBjYW4gdXNlIHRoZSBgY29sbGVjdF9tZXRyaWNzKClgIGZ1bmN0aW9uIGxpa2Ugd2UgZGlkIGJlZm9yZS4KCmBgYHtyfQpjb2xsZWN0X21ldHJpY3Mob3ZlcmFsbGZpdCkKYGBgCgpBd2Vzb21lISBXZSBjYW4gc2VlIHRoYXQgb3VyIHJtc2Ugb2YgYHIgcm91bmQoY29sbGVjdF9tZXRyaWNzKG92ZXJhbGxmaXQpJC5lc3RpbWF0ZSwgMilbMV1gIGlzIHF1aXRlIHNpbWlsYXIgdG8gdGhhdCBvZiBvdXIgdGVzdGluZyBkYXRhIGNyb3NzIHZhbGlkYXRpb24gc2V0cyAod2hlcmUgcm1zZSB3YXMgYHIgcm91bmQoc2hvd19iZXN0KHR1bmVfUkZfcmVzdWx0cywgbWV0cmljID0gInJtc2UiLCBuID0gMSkkbWVhbiwgMilgIC4gV2UgYWNoaWV2ZWQgcXVpdGUgZ29vZCBwZXJmb3JtYW5jZSwgd2hpY2ggc3VnZ2VzdHMgdGhhdCB3ZSBjb3VsZCBwcmVkaWN0IG90aGVyIGxvY2F0aW9ucyB3aXRoIG1vcmUgc3BhcnNlIG1vbml0b3JpbmcgYmFzZWQgb24gb3VyIHByZWRpY3RvcnMgd2l0aCByZWFzb25hYmxlIGFjY3VyYWN5LCBob3dldmVyIHNvbWUgb2Ygb3VyIHByZWRpY3RvcnMgaW52b2x2ZSBtb25pdG9yaW5nIGl0c2VsZiwgc28gdGhlIGFjY3VyYWN5IHdvdWxkIGxpa2VseSBiZSBsb3dlci4KCk5vdyBpZiB5b3Ugd2FudGVkIHRvIHRha2UgYSBsb29rIGF0IHRoZSBwcmVkaWN0ZWQgdmFsdWVzIGZvciB0aGUgdGVzdCBzZXQgKHRoZSAyOTIgcm93cyB3aXRoIHByZWRpY3Rpb25zIG91dCBvZiB0aGUgODc2IG9yaWdpbmFsIG1vbml0b3IgdmFsdWVzKSB5b3UgY2FuIHVzZSB0aGUgIGBjb2xsZWN0X3ByZWRpY3Rpb25zKClgIGZ1bmN0aW9uIG9mIHRoZSBgdHVuZWAgcGFja2FnZToKCmBgYHtyfQp0ZXN0X3ByZWRpY3Rpb25zIDwtIGNvbGxlY3RfcHJlZGljdGlvbnMob3ZlcmFsbGZpdCkKYGBgCgpgYGB7ciwgZXZhbCA9IEZBTFNFfQp0ZXN0X3ByZWRpY3Rpb25zCmBgYAoKIyMjIyB7LnNjcm9sbGFibGUgfQpgYGB7ciwgZWNobyA9RkFMU0V9CnRlc3RfcHJlZGljdGlvbnMgJT4lCiAgcHJpbnQobiA9IDFlMykKYGBgCgojIyMjCgpOaWNlIQoKIyAqKkRhdGEgVmlzdWFsaXphdGlvbioqCioqKgoKSWYgeW91IGhhdmUgYmVlbiBmb2xsb3dpbmcgYWxvbmcgYnV0IHN0b3BwZWQgYW5kIGFyZSBzdGFydGluZyBoZXJlIHlvdSBjb3VsZCBsb2FkIHRoZSB3cmFuZ2xlZCBkYXRhIHVzaW5nIHRoZSBmb2xsb3dpbmcgY29tbWFuZDoKCmBgYHtyfQpsb2FkKGhlcmU6OmhlcmUoImRhdGEiLCAid3JhbmdsZWQiLCAid3JhbmdsZWRfcG0ucmRhIikpCmBgYAoKIyMjIyB7LmNsaWNrX3RvX2V4cGFuZF9ibG9ja30KCjxkZXRhaWxzPiA8c3VtbWFyeT4gSWYgeW91IHNraXBwZWQgcHJldmlvdXMgc2VjdGlvbnMgY2xpY2sgaGVyZSBmb3IgbW9yZSBpbmZvcm1hdGlvbiBvbiBob3cgdG8gb2J0YWluIGFuZCBsb2FkIHRoZSBkYXRhLiA8L3N1bW1hcnk+CgpGaXJzdCB5b3UgbmVlZCB0byBpbnN0YWxsIHRoZSBgT0NTZGF0YWAgcGFja2FnZToKCmBgYHtyLCBldmFsPUZBTFNFfQppbnN0YWxsLnBhY2thZ2VzKCJPQ1NkYXRhIikKYGBgCgpUaGVuLCB5b3UgbWF5IGRvd25sb2FkIGFuZCBsb2FkIHRoZSB3cmFuZ2xlZCBkYXRhIGAucmRhYCBmaWxlIHVzaW5nIHRoZSBmb2xsb3dpbmcgY29kZToKCmBgYHtyLCBldmFsPUZBTFNFfQpPQ1NkYXRhOjp3cmFuZ2xlZF9yZGEoIm9jcy1icC1haXItcG9sbHV0aW9uIiwgb3V0cGF0aCA9IGdldHdkKCkpCmxvYWQoaGVyZTo6aGVyZSgiT0NTZGF0YSIsICJkYXRhIiwgIndyYW5nbGVkIiwgIndyYW5nbGVkX3BtLnJkYSIpKQpgYGAKCklmIHRoZSBwYWNrYWdlIGRvZXMgbm90IHdvcmsgZm9yIHlvdSwgeW91IG1heSBhbHNvIGRvd25sb2FkIHRoaXMgYC5yZGFgIGZpbGUgYnkgY2xpY2tpbmcgdGhpcyBsaW5rIFtoZXJlXShodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vb3BlbmNhc2VzdHVkaWVzL29jcy1icC1haXItcG9sbHV0aW9uL21hc3Rlci9kYXRhL3dyYW5nbGVkL3dyYW5nbGVkX3BtLnJkYSkuCgpUbyBsb2FkIHRoZSBkb3dubG9hZGVkIGRhdGEgaW50byB5b3VyIGVudmlyb25tZW50LCB5b3UgbWF5IGRvdWJsZSBjbGljayBvbiB0aGUgYC5yZGFgIGZpbGUgaW4gUnN0dWRpbyBvciB1c2luZyB0aGUgYGxvYWQoKWAgZnVuY3Rpb24uCgpUbyBjb3B5IGFuZCBwYXN0ZSBvdXIgY29kZSBiZWxvdywgcGxhY2UgdGhlIGRvd25sb2FkZWQgYC5yZGFgIGZpbGUgaW4geW91ciBjdXJyZW50IHdvcmtpbmcgZGlyZWN0b3J5IHdpdGhpbiBhIHN1YmRpcmVjdG9yeSBjYWxsZWQgIndyYW5nbGVkIiB3aXRoaW4gYSBzdWJkaXJlY3RvcnkgY2FsbGVkICJkYXRhIi4gV2UgdXNlZCBhbiBSU3R1ZGlvIHByb2plY3QgYW5kIHRoZSBbYGhlcmVgIHBhY2thZ2VdKGh0dHBzOi8vZ2l0aHViLmNvbS9qZW5ueWJjL2hlcmVfaGVyZSkgdG8gbmF2aWdhdGUgdG8gdGhlIGZpbGUgbW9yZSBlYXNpbHkuIAoKYGBge3J9CmxvYWQoaGVyZTo6aGVyZSgiZGF0YSIsICJ3cmFuZ2xlZCIsICJ3cmFuZ2xlZF9wbS5yZGEiKSkKYGBgCgo8aHIgc3R5bGU9ImhlaWdodDoxcHg7Ym9yZGVyOm5vbmU7Y29sb3I6IzMzMztiYWNrZ3JvdW5kLWNvbG9yOiMzMzM7IiAvPgoKPGRldGFpbHM+IDxzdW1tYXJ5PiBDbGljayBoZXJlIHRvIHNlZSBtb3JlIGFib3V0IGNyZWF0aW5nIG5ldyBwcm9qZWN0cyBpbiBSU3R1ZGlvLiA8L3N1bW1hcnk+CgpZb3UgY2FuIGNyZWF0ZSBhIHByb2plY3QgYnkgZ29pbmcgdG8gdGhlIEZpbGUgbWVudSBvZiBSU3R1ZGlvIGxpa2Ugc286CgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoPSI2MCUifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCAiTmV3X3Byb2plY3QucG5nIikpCmBgYAoKWW91IGNhbiBhbHNvIGRvIHNvIGJ5IGNsaWNraW5nIHRoZSBwcm9qZWN0IGJ1dHRvbjoKCmBgYHtyLCBlY2hvID0gRkFMU0UsIG91dC53aWR0aD0iNjAlIn0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwgInByb2plY3RfYnV0dG9uLnBuZyIpKQpgYGAKClNlZSBbaGVyZV0oaHR0cHM6Ly9zdXBwb3J0LnJzdHVkaW8uY29tL2hjL2VuLXVzL2FydGljbGVzLzIwMDUyNjIwNy1Vc2luZy1Qcm9qZWN0cykgdG8gbGVhcm4gbW9yZSBhYm91dCB1c2luZyBSU3R1ZGlvIHByb2plY3RzIGFuZCBbaGVyZV0oaHR0cHM6Ly9naXRodWIuY29tL2plbm55YmMvaGVyZV9oZXJlKSB0byBsZWFybiBtb3JlIGFib3V0IHRoZSBgaGVyZWAgcGFja2FnZS4KCjwvZGV0YWlscz4KCjxociBzdHlsZT0iaGVpZ2h0OjFweDtib3JkZXI6bm9uZTtjb2xvcjojMzMzO2JhY2tncm91bmQtY29sb3I6IzMzMzsiIC8+Cgo8L2RldGFpbHM+CgojIyMjCgoKT3VyIG1haW4gcXVlc3Rpb24gZm9yIHRoaXMgY2FzZSBzdHVkeSB3YXM6ICAKCj4gQ2FuIHdlIHByZWRpY3QgcmVnaW9uYWwgYW5udWFsIGF2ZXJhZ2UgYWlyIHBvbGx1dGlvbiBjb25jZW50cmF0aW9ucyBieSB6aXAtY29kZSB1c2luZyBwcmVkaWN0b3JzIHN1Y2ggYXMgcG9wdWxhdGlvbiBkZW5zaXR5LCB1cmJhbml6YXRpb24sIHJvYWQgZGVuc2l0eSwgYXMgd2VsbCBhcywgc2F0ZWxsaXRlIHBvbGx1dGlvbiBkYXRhIGFuZCBjaGVtaWNhbCBtb2RlbGluZyBkYXRhPwoKVGh1cyBmYXIsIHdlIGhhdmUgYnVpbHQgYSBtYWNoaW5lIGxlYXJuaW5nIChNTCkgbW9kZWwgdG8gcHJlZGljdCBmaW5lIHBhcnRpY3VsYXRlIG1hdHRlciBhaXIgcG9sbHV0aW9uIGxldmVscyBiYXNlZCBvbiBvdXIgcHJlZGljdG9yIHZhcmlhYmxlcyAob3IgZmVhdHVyZXMpLgoKTm93LCBsZXQncyBtYWtlIGEgcGxvdCBvZiBvdXIgcHJlZGljdGVkIG91dGNvbWUgdmFsdWVzICgkXGhhdHtZfSQpIGFuZCBhY3R1YWwgb3V0Y29tZSB2YWx1ZXMgJFkkIHdlIG9ic2VydmVkLiAKCkZpcnN0LCBsZXQncyBzdGFydCBieSBtYWtpbmcgYSBwbG90IG9mIG91ciBtb25pdG9ycy4gClRvIGRvIHRoaXMsIHdlIHdpbGwgdXNlIHRoZSBmb2xsb3dpbmcgcGFja2FnZXMgdG8gY3JlYXRlIGEgbWFwIG9mIHRoZSBVUzoKCjEuIGBzZmAgLSB0aGUgc2ltcGxlIGZlYXR1cmVzIHBhY2thZ2UgaGVscHMgdG8gY29udmVydCBnZW9ncmFwaGljYWwgY29vcmRpbmF0ZXMgaW50byBgZ2VvbWV0cnlgIHZhcmlhYmxlcyB3aGljaCBhcmUgdXNlZnVsIGZvciBtYWtpbmcgMkQgcGxvdHMKMi4gYG1hcHNgIC0gdGhpcyBwYWNrYWdlIGNvbnRhaW5zIGdlb2dyYXBoaWNhbCBvdXRsaW5lcyBhbmQgcGxvdHRpbmcgZnVuY3Rpb25zIHRvIGNyZWF0ZSBwbG90cyB3aXRoIG1hcHMgCjMuIGBybmF0dXJhbGVhcnRoYC0gdGhpcyBhbGxvd3MgZm9yIGVhc3kgaW50ZXJhY3Rpb24gd2l0aCBtYXAgZGF0YSBmcm9tIFtOYXR1cmFsIEVhcnRoXShodHRwOi8vd3d3Lm5hdHVyYWxlYXJ0aGRhdGEuY29tLykgd2hpY2ggaXMgYSBwdWJsaWMgZG9tYWluIG1hcCBkYXRhc2V0CjQuIGByZ2Vvc2AgLSB0aGlzIHBhY2thZ2UgaW50ZXJmYWNlcyB3aXRoIHRoZSBHZW9tZXRyeSBFbmdpbmUtT3BlbiBTb3VyY2UgKGBHRU9TYCkgd2hpY2ggaXMgYWxzbyBoZWxwZnVsIGZvciBjb29yZGluYXRlIGNvbnZlcnNpb24KCldlIHdpbGwgc3RhcnQgd2l0aCBnZXR0aW5nIGFuIG91dGxpbmUgb2YgdGhlIFVTIHdpdGggdGhlIGBuZV9jb3VudHJpZXMoKWAgZnVuY3Rpb24gb2YgdGhlIGBybmF0dXJhbGVhcnRoYCBwYWNrYWdlIHdoaWNoIHdpbGwgcmV0dXJuIHBvbHlnb25zIG9mIHRoZSBjb3VudHJpZXMgaW4gdGhlIFtOYXR1cmFsIEVhcnRoXShodHRwOi8vd3d3Lm5hdHVyYWxlYXJ0aGRhdGEuY29tLykgZGF0YXNldC4KCmBgYHtyfQoKd29ybGQgPC0gbmVfY291bnRyaWVzKHNjYWxlID0gIm1lZGl1bSIsIHJldHVybmNsYXNzID0gInNmIikKZ2xpbXBzZSh3b3JsZCkKCmBgYAoKCkhlcmUgeW91IGNhbiBzZWUgdGhlIGRhdGEgYWJvdXQgdGhlIGNvdW50cmllcyBpbiB0aGUgd29ybGQuIE5vdGljZSB0aGUgYGdlb21ldHJ5YCB2YXJpYWJsZS4gVGhpcyBpcyB1c2VkIHRvIGNyZWF0ZSB0aGUgb3V0bGluZXMgdGhhdCB3ZSB3YW50LiAKCk5vdyB3ZSBjYW4gdXNlIHRoZSBgZ2VvbV9zZigpYCBmdW5jdGlvbiBvZiB0aGUgYGdncGxvdDJgIHBhY2thZ2UgdG8gY3JlYXRlIGEgdmlzdWFsIG9mIHNpbXBsZSBmZWF0dXJlICh0aGUgZ2VvbWV0cnkgY29vcmRpbmF0ZXMgZm91bmQgaW4gdGhlIGBnZW9tZXRyeWAgdmFyaWFibGUpLgoKYGBge3J9CmdncGxvdChkYXRhID0gd29ybGQpICsKICAgIGdlb21fc2YoKSAKCmBgYAoKU28gbm93IHdlIGNhbiBzZWUgdGhhdCB3ZSBoYXZlIG91dGxpbmVzIG9mIGFsbCB0aGUgY291bnRyaWVzIGluIHRoZSB3b3JsZC4KCldlIHdhbnQgdG8gbGltaXQgdGhpcyBqdXN0IHRvIHRoZSBjb29yZGluYXRlcyBmb3IgdGhlIFVTLiBXZSB3aWxsIGRvIHRoaXMgYmFzZWQgb24gdGhlIGNvb3JkaW5hdGVzIHdlIGZvdW5kIG9uIFdpa2lwZWRpYS4gQWNjb3JkaW5nIHRvIHRoaXMgW2xpbmtdKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0xpc3Rfb2ZfZXh0cmVtZV9wb2ludHNfb2ZfdGhlX1VuaXRlZF9TdGF0ZXMjV2VzdGVybm1vc3Qpe3RhcmdldD0iX2JsYW5rIn0sIHRoZXNlIGFyZSB0aGUgbGF0aXR1ZGUgYW5kIGxvbmdpdHVkZSBib3VuZHMgb2YgdGhlIGNvbnRpbmVudGFsIFVTOgoKLSB0b3AgPSA0OS4zNDU3ODY4ICMgbm9ydGggbGF0Ci0gbGVmdCA9IC0xMjQuNzg0NDA3OSAjIHdlc3QgbG9uZwotIHJpZ2h0ID0gLTY2Ljk1MTM4MTIgIyBlYXN0IGxvbmcKLSBib3R0b20gPSAgMjQuNzQzMzE5NSAjIHNvdXRoIGxhdAoKYGBge3J9CgpnZ3Bsb3QoZGF0YSA9IHdvcmxkKSArCiAgICBnZW9tX3NmKCkgKwogICAgY29vcmRfc2YoeGxpbSA9IGMoLTEyNSwgLTY2KSwgeWxpbSA9IGMoMjQuNSwgNTApLCAKICAgICAgICAgICAgIGV4cGFuZCA9IEZBTFNFKQpgYGAKTm93IHdlIGp1c3QgaGF2ZSBhIHBsb3QgdGhhdCBpcyBtb3N0bHkgbGltaXRlZCB0byB0aGUgb3V0bGluZSBvZiB0aGUgVVMuCgpOb3cgd2Ugd2lsbCB1c2UgdGhlIGBnZW9tX3BvaW50KClgIGZ1bmN0aW9uIG9mIHRoZSBgZ2dwbG90YCBwYWNrYWdlIHRvIGFkZCBzY2F0dGVyIHBsb3Qgb24gdG9wIG9mIHRoZSBtYXAuIFdlIHdhbnQgdG8gc2hvdyB3aGVyZSB0aGUgbW9uaXRvcnMgYXJlIGxvY2F0ZWQgYmFzZWQgb24gdGhlIGxhdGl0dWRlIGFuZCBsb25naXR1ZGUgdmFsdWVzIGluIHRoZSBkYXRhLgoKYGBge3J9CmdncGxvdChkYXRhID0gd29ybGQpICsKICAgIGdlb21fc2YoKSArCiAgICBjb29yZF9zZih4bGltID0gYygtMTI1LCAtNjYpLCB5bGltID0gYygyNC41LCA1MCksIAogICAgICAgICAgICAgZXhwYW5kID0gRkFMU0UpKwogICAgZ2VvbV9wb2ludChkYXRhID0gcG0sIGFlcyh4ID0gbG9uLCB5ID0gbGF0KSwgc2l6ZSA9IDIsIAogICAgICAgICAgICAgICBzaGFwZSA9IDIzLCBmaWxsID0gImRhcmtyZWQiKQoKYGBgCgpOaWNlIQoKTm93IGxldCdzIGFkZCBjb3VudHkgbGluZXMuCgpDb3VudHkgZ3JhcGhpY2FsIGRhdGEgaXMgYXZhaWxhYmxlIGZyb20gdGhlIGBtYXBzYCBwYWNrYWdlLiAKVGhlIGBzZmAgcGFja2FnZSB3aGljaCBhZ2FpbiBpcyBzaG9ydCBmb3Igc2ltcGxlIGZlYXR1cmVzIGNyZWF0ZXMgYSBkYXRhIGZyYW1lIGFib3V0IHRoaXMgZ3JhcGhpY2FsIGRhdGEgc28gdGhhdCB3ZSBjYW4gd29yayB3aXRoIGl0LgoKYGBge3J9CmNvdW50aWVzIDwtIHNmOjpzdF9hc19zZihtYXBzOjptYXAoImNvdW50eSIsIHBsb3QgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaWxsID0gVFJVRSkpCgpjb3VudGllcwpgYGAKCk5vdyB3ZSB3aWxsIHVzZSB0aGlzIGRhdGEgd2l0aGluIHRoZSBgZ2VvbV9zZigpYCBmdW5jdGlvbiB0byBhZGQgdGhpcyB0byBvdXIgcGxvdC4gIFdlIHdpbGwgYWxzbyBhZGQgYSB0aXRsZSB1c2luZyB0aGUgYGdndGl0bGUoKWAgZnVuY3Rpb24sIGFzIHdlbGwgYXMgcmVtb3ZlIGF4aXMgdGlja3MgYW5kIHRpdGxlcyB1c2luZyB0aGUgYHRoZW1lKClgIGZ1bmN0aW9uIG9mIHRoZSBgZ2dwbG90MmAgcGFja2FnZS4KCmBgYHtyfQptb25pdG9ycyA8LSBnZ3Bsb3QoZGF0YSA9IHdvcmxkKSArCiAgICBnZW9tX3NmKGRhdGEgPSBjb3VudGllcywgZmlsbCA9IE5BLCBjb2xvciA9IGdyYXkoLjUpKSsKICAgICAgY29vcmRfc2YoeGxpbSA9IGMoLTEyNSwgLTY2KSwgeWxpbSA9IGMoMjQuNSwgNTApLCAKICAgICAgICAgICAgIGV4cGFuZCA9IEZBTFNFKSArCiAgICBnZW9tX3BvaW50KGRhdGEgPSBwbSwgYWVzKHggPSBsb24sIHkgPSBsYXQpLCBzaXplID0gMiwgCiAgICAgICAgICAgICAgIHNoYXBlID0gMjMsIGZpbGwgPSAiZGFya3JlZCIpICsKICAgIGdndGl0bGUoIk1vbml0b3IgTG9jYXRpb25zIikgKwogICAgdGhlbWUoYXhpcy50aXRsZS54PWVsZW1lbnRfYmxhbmsoKSwKICAgICAgICAgIGF4aXMudGV4dC54ID0gZWxlbWVudF9ibGFuaygpLAogICAgICAgICAgYXhpcy50aWNrcy54ID0gZWxlbWVudF9ibGFuaygpLAogICAgICAgICAgYXhpcy50aXRsZS55ID0gZWxlbWVudF9ibGFuaygpLAogICAgICAgICAgYXhpcy50ZXh0LnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICAgICAgICBheGlzLnRpY2tzLnkgPSBlbGVtZW50X2JsYW5rKCkpCgptb25pdG9ycwpgYGAKCkdyZWF0IQoKTm93LCBsZXQncyBhZGQgYSBmaWxsIGF0IHRoZSBjb3VudHktbGV2ZWwgZm9yIHRoZSB0cnVlIG1vbml0b3IgdmFsdWVzIG9mIGFpciBwb2xsdXRpb24uCgpGaXJzdCwgd2UgbmVlZCB0byBnZXQgdGhlIGNvdW50eSBtYXAgZGF0YSB0aGF0IHdlIGp1c3QgZ290IGFuZCBvdXIgYWlyIHBvbGx1dGlvbiBkYXRhIHRvIGhhdmUgc2ltaWxhcmx5IGZvcm1hdHRlZCBjb3VudHkgbmFtZXMgc28gdGhhdCB3ZSBjYW4gY29tYmluZSB0aGUgZGF0YXNldHMgdG9nZXRoZXIuCgpXZSBjYW4gc2VlIHRoYXQgaW4gdGhlIGBjb3VudHlgIGRhdGEgdGhlIGNvdW50aWVzIGFyZSBsaXN0ZWQgYWZ0ZXIgdGhlIHN0YXRlIG5hbWUgYW5kIGEgY29tbWEuIEluIGFkZGl0aW9uIHRoZXkgYXJlIGFsbCBsb3dlciBjYXNlLgoKCmBgYHtyfQpoZWFkKGNvdW50aWVzKQpgYGAKCkluIGNvbnRyYXN0LCBvdXIgYWlyIHBvbGx1dGlvbiBgcG1gIGRhdGEgc2hvd3MgY291bnRpZXMgYXMgdGl0bGVzIHdpdGggdGhlIGZpcnN0IGxldHRlciBhcyB1cHBlciBjYXNlLiAKCmBgYHtyfQpkcGx5cjo6cHVsbChwbSwgY291bnR5KSAlPiUKICBoZWFkKCkKYGBgCgpXZSBjYW4gdXNlIHRoZSBgc2VwYXJhdGUoKWAgZnVuY3Rpb24gb2YgdGhlIGB0aWR5cmAgcGFja2FnZSB0byBzZXBhcmF0ZSB0aGUgYElEYCB2YXJpYWJsZSBvZiBvdXIgYGNvdW50aWVzYCBkYXRhIGludG8gdHdvIGB2YXJpYWJsZXNgIGJhc2VkIG9uIHRoZSBjb21tYSBhcyBhIHNlcGFyYXRvci4KCmBgYHtyfQpjb3VudGllcyAlPD4lIAogIHRpZHlyOjpzZXBhcmF0ZShJRCwgaW50byA9IGMoInN0YXRlIiwgImNvdW50eSIpLCBzZXAgPSAiLCIpCgpoZWFkKGNvdW50aWVzKQpgYGAKCk5vdyB3ZSBqdXN0IG5lZWQgdG8gbWFrZSB0aGVzZSBuYW1lcyBpbiB0aGUgbmV3IGBjb3VudHlgIHZhcmlhYmxlIG9mIHRoZSBgY291bnRpZXNgIGRhdGEgdG8gYmUgaW4gdGl0bGUgZm9ybWF0LiBXZSBjYW4gdXNlIHRoZSBgc3RyX3RvX3RpdGxlKClgIGZ1bmN0aW9uIG9mIHRoZSBgc3RyaW5ncmAgcGFja2FnZSB0byBkbyB0aGlzLiAKCmBgYHtyfQpjb3VudGllc1tbImNvdW50eSJdXSA8LSBzdHJpbmdyOjpzdHJfdG9fdGl0bGUoY291bnRpZXNbWyJjb3VudHkiXV0pCmBgYAoKR3JlYXQhIE5vdyB0aGUgY291bnR5IGluZm9ybWF0aW9uIGlzIHRoZSBzYW1lIGZvciB0aGUgYGNvdW50aWVzYCBhbmQgYHBtYCBkYXRhLgoKV2UgY2FuIHVzZSB0aGUgYGlubmVyX2pvaW4oKWAgZnVuY3Rpb24gb2YgdGhlIGBkcGx5cmAgcGFja2FnZSB0byBqb2luIHRoZSBkYXRhc2V0cyB0b2dldGhlciBiYXNlZCBvbiB0aGUgYGNvdW50eWAgdmFyaWFibGVzIGluIGVhY2guIFRoaXMgZnVuY3Rpb24gd2lsbCBrZWVwIGFsbCByb3dzIHRoYXQgYXJlIGluIGJvdGggZGF0YXNldHMuCgpgYGB7cn0KbWFwX2RhdGEgPC0gZHBseXI6OmlubmVyX2pvaW4oY291bnRpZXMsIHBtLCBieSA9ICJjb3VudHkiKQoKZ2xpbXBzZShtYXBfZGF0YSkKCmBgYAoKTmljZSEgd2UgY2FuIHNlZSB0aGF0IHdlIGhhdmUgYWRkIGEgYGdlb21gIHZhcmlhYmxlIHRvIHRoZSBgcG1gIGRhdGEuCgpOb3cgd2UgY2FuIHVzZSB0aGlzIHRvIGNvbG9yIHRoZSBjb3VudGllcyBpbiBvdXIgcGxvdCBiYXNlZCBvbiB0aGUgYHZhbHVlYCB2YXJpYWJsZSBvZiBvdXIgYHBtYCBkYXRhLCB3aGljaCB5b3UgbWF5IHJlY2FsbCBpcyB0aGUgYWN0dWFsIG1vbml0b3IgZGF0YSBmb3IgZmluZSBwYXJ0aWN1bGF0ZSBhaXIgcG9sbHV0aW9uIGF0IGVhY2ggbW9uaXRvci4gCgpXRSBjYW4gZG8gc28gdXNpbmcgdGhlIGBzY2FsZV9maWxsX2dyYWRpZW50bigpYCBmdW5jdGlvbiBvZiB0aGUgYGdncGxvdDJgIHBhY2thZ2Ugd2hpY2ggY3JlYXRlcyBjb2xvciBncmFkaWVudCBiYXNlZCBvbiBhIHZhcmlhYmxlLiBJbiB0aGlzIGNhc2UgaXQgaXMgdGhlIHZhcmlhYmxlIHRoYXQgd2FzIHNwZWNpZmllZCBhcyB0aGUgYGZpbGxgIGluIHRoZSBgYWVzYCBmdW5jdGlvbiBvZiB0aGUgYGdlb21fc2YoKWAgZnVuY3Rpb24uIFdlIHNwZWNpZmllZCB0aGF0IGl0IHdvdWxkIGJlIHRoZSBgdmFsdWVgIHZhcmlhYmxlIG9mIHRoZSBgcG1gIGRhdGEuCgpUaGlzIGBzY2FsZV9maWxsX2dyYWRpZW50bigpYCBmdW5jdGlvbiAgYWxzbyBhbGxvd3MgeW91IHRvIHNwZWNpZnkgdGhlIGNvbG9ycywgd2hhdCB0byBkbyBhYm91dCBOQSB2YWx1ZXMgKHNob3VsZCB0aGV5IGJlIGEgc3BlY2lmaWMgY29sb3Igb3IgdHJhbnNwYXJlbnQpIGFuZCB0aGUgYnJlYWtzLCBsaW1pdHMsIGxhYmVscyBhbmQgbmFtZS90aXRsZSBvbiB0aGUgbGVnZW5kIGZvciB0aGUgY29sb3IgZ3JhZGllbnQuIAoKYGBge3J9CnRydXRoIDwtIGdncGxvdChkYXRhID0gd29ybGQpICsKICBjb29yZF9zZih4bGltID0gYygtMTI1LC02NiksCiAgICAgICAgICAgeWxpbSA9IGMoMjQuNSwgNTApLAogICAgICAgICAgIGV4cGFuZCA9IEZBTFNFKSArCiAgZ2VvbV9zZihkYXRhID0gbWFwX2RhdGEsIGFlcyhmaWxsID0gdmFsdWUpKSArCiAgc2NhbGVfZmlsbF9ncmFkaWVudG4oY29sb3VycyA9IHRvcG8uY29sb3JzKDcpLAogICAgICAgICAgICAgICAgICAgICAgIG5hLnZhbHVlID0gInRyYW5zcGFyZW50IiwKICAgICAgICAgICAgICAgICAgICAgICBicmVha3MgPSBjKDAsIDEwLCAyMCksCiAgICAgICAgICAgICAgICAgICAgICAgbGFiZWxzID0gYygwLCAxMCwgMjApLAogICAgICAgICAgICAgICAgICAgICAgIGxpbWl0cyA9IGMoMCwgMjMuNSksCiAgICAgICAgICAgICAgICAgICAgICAgbmFtZSA9ICJQTSB1Zy9tMyIpICsKICBnZ3RpdGxlKCJUcnVlIFBNIDIuNSBsZXZlbHMiKSArCiAgdGhlbWUoYXhpcy50aXRsZS54ID0gZWxlbWVudF9ibGFuaygpLAogICAgICAgIGF4aXMudGV4dC54ID0gZWxlbWVudF9ibGFuaygpLAogICAgICAgIGF4aXMudGlja3MueCA9IGVsZW1lbnRfYmxhbmsoKSwKICAgICAgICBheGlzLnRpdGxlLnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICAgICAgYXhpcy50ZXh0LnkgPSBlbGVtZW50X2JsYW5rKCksCiAgICAgICAgYXhpcy50aWNrcy55ID0gZWxlbWVudF9ibGFuaygpKQoKdHJ1dGgKCmBgYAoKTmljZSEKCk5vdyBsZXQncyBkbyB0aGUgc2FtZSB3aXRoIG91ciBwcmVkaWN0ZWQgb3V0Y29tZSB2YWx1ZXMuCgpMZXQncyBncmFiIGJvdGggdGhlIHRlc3RpbmcgYW5kIHRyYWluaW5nIHByZWRpY3RlZCBvdXRjb21lIHZhbHVlcyBzbyB0aGF0IHdlIGhhdmUgYXMgbXVjaCBkYXRhIGFzIHBvc3NpYmxlLiAKCkZpcnN0IHdlIG5lZWQgdG8gZml0IG91ciB0cmFpbmluZyBkYXRhIHdpdGggb3VyIGZpbmFsIG1vZGVsIHRvIGJlIGFibGUgdG8gZ2V0IHRoZSBwcmVkaWN0aW9ucyBmb3IgdGhlIG1vbml0b3JzIGluY2x1ZGVkIGluIHRoZSB0cmFpbmluZyBzZXQuIFdlIGRpZCB0aGlzIHVzaW5nIHRoZSBgbGFzdF9maXQoKWAgZnVuY3Rpb24sIGJ1dCB0aGUgb3V0cHV0IG9mIHRoaXMgbWFrZXMgaXQgZGlmZmljdWx0IHRvIGdyYWIgdGhlIHByZWRpY3RlZCB2YWx1ZXMgZm9yIHRoZSB0cmFpbmluZyBkYXRhLCBhbmQgaXQgaXMgYWxzbyBkaWZmaWN1bHQgdG8gZ2V0IHRoZSBpZCB2YXJpYWJsZXMgZm9yIHRoZSB0ZXN0aW5nIGRhdGEuIAoKClRodXMgd2Ugd2lsbCB1c2UgdGhlIHBhcnNuaXAgYGZpdCgpYCBhbmQgYHByZWRpY3QoKWAgZnVuY3Rpb25zIG9mIHRoZSBgcGFyc25pcGAgcGFja2FnZSB0byBkbyB0aGlzIGxpa2Ugc286CgojIyMjIHsudGhpbmtfcXVlc3Rpb25fYmxvY2t9CjxiPjx1PiBRdWVzdGlvbiBPcHBvcnR1bml0eSA8L3U+PC9iPgoKV2h5IGRvIHdlIG5vdCBuZWVkIHByZS1wcm9jZXNzZWQgZGF0YT8KCioqKgoKPGRldGFpbHM+IDxzdW1tYXJ5PiBDbGljayBoZXJlIHRvIHJldmVhbCB0aGUgYW5zd2VyLiA8L3N1bW1hcnk+CgpTaW5jZSB3ZSBhcmUgdXNpbmcgYSB3b3JrZmxvdywgdGhlIGRhdGEgd2lsbCBiZSBwcmUtcHJvY2Vzc2VkIHdoZW4gaXQgaXMgZml0IGFzIHdlbGwuCgo8L2RldGFpbHM+CgoqKioKCiMjIyMKCgpgYGB7cn0KClJGX2ZpbmFsX3RyYWluX2ZpdCA8LSBwYXJzbmlwOjpmaXQoUkZfdHVuZWRfd2Zsb3csIGRhdGEgPSB0cmFpbl9wbSkKUkZfZmluYWxfdGVzdF9maXQgPC0gcGFyc25pcDo6Zml0KFJGX3R1bmVkX3dmbG93LCBkYXRhID0gdGVzdF9wbSkKCgp2YWx1ZXNfcHJlZF90cmFpbiA8LSBwcmVkaWN0KFJGX2ZpbmFsX3RyYWluX2ZpdCwgdHJhaW5fcG0pICU+JSAKICBiaW5kX2NvbHModHJhaW5fcG0gJT4lIHNlbGVjdCh2YWx1ZSwgZmlwcywgY291bnR5LCBpZCkpIAoKdmFsdWVzX3ByZWRfdHJhaW4KCnZhbHVlc19wcmVkX3Rlc3QgPC0gcHJlZGljdChSRl9maW5hbF90ZXN0X2ZpdCwgdGVzdF9wbSkgJT4lIAogIGJpbmRfY29scyh0ZXN0X3BtICU+JSBzZWxlY3QodmFsdWUsIGZpcHMsIGNvdW50eSwgaWQpKSAKCnZhbHVlc19wcmVkX3Rlc3QKYGBgCgpOb3cgd2UgY2FuIGNvbWJpbmUgdGhpcyBkYXRhIGZvciB0aGUgcHJlZGljdGlvbnMgZm9yIGFsbCBtb25pdG9ycyB1c2luZyB0aGUgYGJpbmRfcm93cygpYCBmdW5jdGlvbiBvZiB0aGUgYGRwbHlyYCBwYWNrYWdlLCB3aGljaCB3aWxsIGVzc2VudGlhbGx5IGFwcGVuZCB0aGUgc2Vjb25kIGRhdGFzZXQgdG8gdGhlIGZpcnN0LgoKYGBge3J9CmFsbF9wcmVkIDwtIGJpbmRfcm93cyh2YWx1ZXNfcHJlZF90ZXN0LCB2YWx1ZXNfcHJlZF90cmFpbikKCmFsbF9wcmVkCmBgYAoKR3JlYXQhIGFzIHdlIGNhbiBzZWUgdGhlcmUgYXJlIDg3NiB2YWx1ZXMgbGlrZSB3ZSB3b3VsZCBleHBlY3QgZm9yIGFsbCBvZiB0aGUgbW9uaXRvcnMuIFdlIGNhbiB1c2UgdGhlIGBjb3VudHlgIHZhcmlhYmxlIHRvIGNvbWJpbmUgdGhpcyB3aXRoIHRoZSBgY291bnRpZXNgIGRhdGEgbGlrZSB3ZSBkaWQgd2l0aCB0aGUgYHBtYCBkYXRhIHByZXZpb3VzbHkgc28gdGhhdCB3ZSBjYW4gdXNlIHRoZSBgdmFsdWVgIHZhcmlhYmxlIGFzIGEgY29sb3Igc2NoZW1lIGZvciBvdXIgbWFwLgoKCmBgYHtyfQptYXBfZGF0YSA8LSBpbm5lcl9qb2luKGNvdW50aWVzLCBhbGxfcHJlZCwgYnkgPSAiY291bnR5IikKCnByZWQgPC0gZ2dwbG90KGRhdGEgPSB3b3JsZCkgKwogIGNvb3JkX3NmKHhsaW0gPSBjKC0xMjUsLTY2KSwKICAgICAgICAgICB5bGltID0gYygyNC41LCA1MCksCiAgICAgICAgICAgZXhwYW5kID0gRkFMU0UpICsKICBnZW9tX3NmKGRhdGEgPSBtYXBfZGF0YSwgYWVzKGZpbGwgPSAucHJlZCkpICsKICBzY2FsZV9maWxsX2dyYWRpZW50bihjb2xvdXJzID0gdG9wby5jb2xvcnMoNyksCiAgICAgICAgICAgICAgICAgICAgICAgbmEudmFsdWUgPSAidHJhbnNwYXJlbnQiLAogICAgICAgICAgICAgICAgICAgICAgIGJyZWFrcyA9IGMoMCwgMTAsIDIwKSwKICAgICAgICAgICAgICAgICAgICAgICBsYWJlbHMgPSBjKDAsIDEwLCAyMCksCiAgICAgICAgICAgICAgICAgICAgICAgbGltaXRzID0gYygwLCAyMy41KSwKICAgICAgICAgICAgICAgICAgICAgICBuYW1lID0gIlBNIHVnL20zIikgKwogIGdndGl0bGUoIlByZWRpY3RlZCBQTSAyLjUgbGV2ZWxzIikgKwogIHRoZW1lKGF4aXMudGl0bGUueD1lbGVtZW50X2JsYW5rKCksCiAgICAgICAgYXhpcy50ZXh0Lng9ZWxlbWVudF9ibGFuaygpLAogICAgICAgIGF4aXMudGlja3MueD1lbGVtZW50X2JsYW5rKCksCiAgICAgICAgYXhpcy50aXRsZS55PWVsZW1lbnRfYmxhbmsoKSwKICAgICAgICBheGlzLnRleHQueT1lbGVtZW50X2JsYW5rKCksCiAgICAgICAgYXhpcy50aWNrcy55PWVsZW1lbnRfYmxhbmsoKSkKCnByZWQKYGBgCgpOb3cgd2Ugd2lsbCB1c2UgdGhlIGBwYXRjaHdvcmtgIHBhY2thZ2UgdG8gY29tYmluZSBvdXIgbGFzdCB0d28gcGxvdHMuIFRoaXMgYWxsb3dzIHVzIHRvIGNvbWJpbmUgcGxvdHMgdXNpbmcgdGhlIGArYCBvciB0aGUgYC9gIC4gVGhlIGArYCB3aWxsIHBsYWNlIHBsb3RzIHNpZGUgYnkgc2lkZSBhbmQgdGhlIGAvYCB3aWxsIHBsYWNlIHBsb3RzIHRvcCB0byBib3R0b20uCgoKTm93IGxldCdzIGp1c3QgY29tYmluZSB0aGUgdHJ1dGggcGxvdCBhbmQgdGhlIHByZWRpY3Rpb24gcGxvdHMgdG9nZXRoZXI6CmBgYHtyfQp0cnV0aC9wcmVkCgpgYGAKCldlIGNhbiBzZWUgdGhhdCB0aGUgcHJlZGljdGVkIGZpbmUgcGFydGljbGUgYWlyIHBvbGx1dGlvbiB2YWx1ZXMgaW4gKHVnL20zKSBhcmUgcXVpdGUgc2ltaWxhciB0byB0aGUgdHJ1ZSB2YWx1ZXMgbWVhc3VyZWQgYnkgdGhlIGFjdHVhbCBncmF2aW1ldHJpYyBtb25pdG9ycy4gV2UgY2FuIGFsc28gc2VlIHRoYXQgc291dGhlcm4gQ2FsaWZvcm5pYSBoYXMgc29tZSBsYXJnZSBjb3VudGllcyB3aXRoIHdvcnNlIHBvbGx1dGlvbiAoYXMgdGhleSBhcmUgeWVsbG93IGFuZCB0aHVzIGhhdmUgbXVjaCBoaWdoZXIgcGFydGljdWxhdGUgbWF0dGVyIGxldmVscykuCgpMZXQncyBhZGQgc29tZSB0ZXh0IHRvIG91ciBwbG90IHRvIGV4cGxhaW4gaXQgYSBiaXQgbW9yZS4gV2UgY2FuIGRvIHNvIHVzaW5nIHRoZSBgcGxvdF9hbm5vdGF0aW9uKClgIGZ1bmN0aW9uIG9mIHRoZSBgcGF0Y2h3b3JrYCBwYWNrYWdlLiBUaGUgYHRoZW1lYCBhcmd1bWVudCBvZiB0aGlzIGZ1bmN0aW9uIHRha2VzIHRoZSBzYW1lIHRoZW1lIGluZm9ybWF0aW9uIHVzaW5nIHRoZSBgdGhlbWUoKWAgZnVuY3Rpb24gb2YgdGhlIGBnZ3Bsb3QyYCBwYWNrYWdlIGFzIHdoZW4gY3JlYXRpbmcgYGdncGxvdDJgcGxvdHMuCgpgYGB7cn0KKHRydXRoL3ByZWQpICsgCiAgcGxvdF9hbm5vdGF0aW9uKHRpdGxlID0gIk1hY2hpbmUgTGVhcm5pbmcgTWV0aG9kcyBBbGxvdyBmb3IgUHJlZGljdGlvbiBvZiBBaXIgUG9sbHV0aW9uIiwgc3VidGl0bGUgPSAiQSByYW5kb20gZm9yZXN0IG1vZGVsIHByZWRpY3RzIHRydWUgbW9uaXRvcmVkIGxldmVscyBvZiBmaW5lIHBhcnRpY3VsYXRlIG1hdHRlciAoUE0gMi41KSBhaXIgcG9sbHV0aW9uIGJhc2VkIG9uXG5kYXRhIGFib3V0IHBvcHVsYXRpb24gZGVuc2l0eSBhbmQgb3RoZXIgcHJlZGljdG9ycyByZWFzb25hYmx5IHdlbGwsIHRodXMgc3VnZ2VzdGluZyB0aGF0IHdlIGNhbiB1c2Ugc2ltaWxhciBtZXRob2RzIHRvIHByZWRpY3QgbGV2ZWxzXG5vZiBwb2xsdXRpb24gaW4gcGxhY2VzIHdpdGggcG9vciBtb25pdG9yaW5nIiwKICAgICAgICAgICAgICAgICAgdGhlbWUgPSB0aGVtZShwbG90LnRpdGxlID0gZWxlbWVudF90ZXh0KHNpemUgPTEyLCBmYWNlID0gImJvbGQiKSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcGxvdC5zdWJ0aXRsZSA9IGVsZW1lbnRfdGV4dChzaXplID0gOCkpKQoKYGBgCgoKYGBge3IsIGVjaG8gPSBGQUxTRSwgbWVzc2FnZT1GQUxTRSwgZXZhbD1GQUxTRSwgaW5jbHVkZSA9IEZBTFNFfQpwbmcoaGVyZTo6aGVyZSgiaW1nIiwgIm1haW5fcGxvdF9tYXBzLnBuZyIpLCAKICAgIGhlaWdodCA9IDE1MDAsIHdpZHRoID0gMjAwMCwgcmVzID0gMzAwKQoodHJ1dGgvcHJlZCkgKyAKICBwbG90X2Fubm90YXRpb24odGl0bGUgPSAiTWFjaGluZSBMZWFybmluZyBNZXRob2RzIEFsbG93IGZvciBQcmVkaWN0aW9uIG9mIEFpciBQb2xsdXRpb24iLCBzdWJ0aXRsZSA9ICJBIHJhbmRvbSBmb3Jlc3QgbW9kZWwgcHJlZGljdHMgdHJ1ZSBtb25pdG9yZWQgbGV2ZWxzIG9mIGZpbmUgcGFydGljdWxhdGUgbWF0dGVyIChQTSAyLjUpIGFpciBwb2xsdXRpb24gYmFzZWQgb25cbmRhdGEgYWJvdXQgcG9wdWxhdGlvbiBkZW5zaXR5IGFuZCBvdGhlciBwcmVkaWN0b3JzIHJlYXNvbmFibHkgd2VsbCwgdGh1cyBzdWdnZXN0aW5nIHRoYXQgd2UgY2FuIHVzZSBzaW1pbGFyIG1ldGhvZHMgcHJlZGljdCBsZXZlbHNcbm9mIHBvbGx1dGlvbiBpbiBwbGFjZXMgd2l0aCBwb29yIG1vbml0b3JpbmciLAogICAgICAgICAgICAgICAgICB0aGVtZSA9IHRoZW1lKHBsb3QudGl0bGUgPSBlbGVtZW50X3RleHQoc2l6ZSA9MTIsIGZhY2UgPSAiYm9sZCIpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHBsb3Quc3VidGl0bGUgPSBlbGVtZW50X3RleHQoc2l6ZSA9IDgpKSkKZGV2Lm9mZigpCmBgYAoKIyAqKlN1bW1hcnkqKgoqKioKCiMjICoqU3lub3BzaXMqKgoqKioKCkluIHRoaXMgY2FzZSBzdHVkeSwgd2UgZXhwbG9yZWQgZ3JhdmltZXRyaWMgbW9uaXRvcmluZyBkYXRhIG9mIGZpbmUgcGFydGljdWxhdGUgbWF0dGVyIGFpciBwb2xsdXRpb24gKG91dGNvbWUgdmFyaWFibGUpLiAKT3VyIGdvYWwgd2FzIHRvIGFibGUgdG8gcHJlZGljdCBhaXIgcG9sbHV0aW9uIHdoZXJlIHdlIG9ubHkgaGFkIHByZWRpY3RvciB2YXJpYWJsZXMgKG9yIGZlYXR1cmVzKSB3aXRob3V0IGhhdmluZyBvYnNlcnZlZCBhIGNvcnJlc3BvbmRpbmcgbWVhc3VyZW1lbnQgb2YgYWlyIHBvbGx1dGlvbi4KCk91ciBsZWFybmluZyBvYmplY3RpdmVzIHdlcmU6IAoKLSBJbnRyb2R1Y2UgY29uY2VwdHMgaW4gbWFjaGluZSBsZWFybmluZwotIERlbW9uc3RyYXRlIGhvdyB0byBidWlsZCBhIG1hY2hpbmUgbGVhcm5pbmcgbW9kZWwgd2l0aCBgdGlkeW1vZGVsc2AKLSBEZW1vbnN0cmF0ZSBob3cgdG8gdmlzdWFsaXplIGdlby1zcGF0aWFsIGRhdGEgdXNpbmcgYGdncGxvdDJgCgpVc2luZyB0aGUgbWFjaGluZSBsZWFybmluZyBtb2RlbHMgYnVpbHQgaW4gdGhpcyBjYXNlIHN0dWR5LCB3ZSBjb3VsZCBub3cgZXh0ZW5kIHRoaXMgbW9kZWwgdG8gcHJlZGljdCBhaXIgcG9sbHV0aW9uIGxldmVscyBpbiBhcmVhcyB3aXRoIHBvb3IgbW9uaXRvcmluZywgdG8gaGVscCBpZGVudGlmeSByZWdpb25zIHdoZXJlIHBvcHVsYXRpb25zIG1heWJlIGVzcGVjaWFsbHkgYXQgcmlzayBmb3IgdGhlIGhlYWx0aCBlZmZlY3RzIG9mIGFpciBwb2xsdXRpb24uICAKCkFuYWx5c2VzIGxpa2UgdGhlIG9uZSBpbiBvdXIgY2FzZSBzdHVkeSBhcmUgaW1wb3J0YW50IGZvciBkZWZpbmluZyB3aGljaCBncm91cHMgY291bGQgYmVuZWZpdCB0aGUgbW9zdCBmcm9tIGludGVydmVudGlvbnMsIGVkdWNhdGlvbiwgYW5kIHBvbGljeSBjaGFuZ2VzIHdoZW4gYXR0ZW1wdGluZyB0byBtaXRpZ2F0ZSBwdWJsaWMgaGVhbHRoIGNoYWxsZW5nZXMuIFlvdSBjYW4gc2VlIGluIHRoaXMgW2FydGljbGVdKGh0dHBzOi8vd3d3Lm5lam0ub3JnL2RvaS9mdWxsLzEwLjEwNTYvTkVKTW9hMTcwMjc0Nyl7dGFyZ2V0PSJfYmxhbmsifSB0aGF0IG1hbnkgYWRkaXRpb25hbCBjb25zaWRlcmF0aW9ucyB3b3VsZCBiZSBpbnZvbHZlZCB0byBhZGVxdWF0ZWx5IHVuZGVyc3RhbmQgdGhlIGRhdGEgZW5vdWdoIHRvIHJlY29tbWVuZCBwb2xpY3kgY2hhbmdlcy4KCkhlcmUgYXJlIHNvbWUgdmlzdWFsIHN1bW1hcmllcyBhYm91dCB3aGF0IHdlIGxlYXJuZWQgYWJvdXQgdXNpbmcgYHRpZHltb2RlbHNgIHRvIHBlcmZvcm0gcHJlZGljdGlvbiBhbmFseXNlcy4KCkZpcnN0IHRoZSBtaW5pbWFsIHN0ZXBzIHJlcXVpcmVkOgoKYGBge3IsIGVjaG89RkFMU0UsIG91dC53aWR0aD0iODAwcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCJVcGRhdGVkX3RpZHltb2RlbHNfYmFzaWNzLnBuZyIpKQpgYGAKCgpIZXJlIGlzIGEgZ3VpZGUgZm9yIG1vcmUgYWR2YW5jZWQgYW5hbHlzZXMgaW52b2x2aW5nIHByZXByb2Nlc3NpbmcsIGNyb3NzIHZhbGlkYXRpb24sIG9yIHR1bmluZzoKCmBgYHtyLCBlY2hvPUZBTFNFLCBvdXQud2lkdGg9IjgwMHB4In0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoaGVyZTo6aGVyZSgiaW1nIiwiZnVsbF90aWR5bW9kZWxzX292ZXJ2aWV3LnBuZyIpKQpgYGAKCgojIyMjIHsuY2xpY2tfdG9fZXhwYW5kX2Jsb2NrfQoKPGRldGFpbHM+PHN1bW1hcnk+IENsaWNrIGhlcmUgZm9yIG1vcmUgb24gd2hhdCB3ZSBsZWFybmVkIHdpdGggYHRpZHltb2RlbHNgIDwvc3VtbWFyeT4KCkhlcmUsIHdlIHByb3ZpZGUgYW4gb3ZlcnZpZXcgb2YgdGhlIGB0aWR5bW9kZWxzYCBmcmFtZXdvcmsuIAoKYGBge3IsIGVjaG89RkFMU0UsIG91dC53aWR0aD0iODAwcHgifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcyhoZXJlOjpoZXJlKCJpbWciLCJlY29zeXN0ZW0ucG5nIikpCmBgYAoKCldlIHBlcmZvcm1lZCB0aGUgbWFqb3Igc3RlcHMgb2YgbWFjaGluZSBsZWFybmluZyB0aGF0IHdlIGludHJvZHVjZWQgaW4gdGhlIGJlZ2lubmluZyBvZiB0aGUgZGF0YSBhbmFseXNpczogIAoKMS4gRGF0YSBleHBsb3JhdGlvbiAgCgpXZSB1c2VkIHBhY2thZ2VzIGxpa2UgYHNraW1yYCwgYHN1bW1hcnl0b29sc2AsIGBjb3JycGxvdGAsIGFuZCBgR0dhbGx5YCB0byBiZXR0ZXIgdW5kZXJzdGFuZCBvdXIgZGF0YS4gVGhlc2UgcGFja2FnZXMgY2FuIHRlbGwgdXMgaG93IG1hbnkgbWlzc2luZyB2YWx1ZXMgZWFjaCB2YXJpYWJsZSBoYXMgKGlmIGFueSksIHRoZSBjbGFzcyBvZiBlYWNoIHZhcmlhYmxlLCB0aGUgZGlzdHJpYnV0aW9uIG9mIHZhbHVlcyBmb3IgZWFjaCB2YXJpYWJsZSwgdGhlIHNwYXJzaXR5IG9mIGVhY2ggdmFyaWFibGUsIGFuZCB0aGUgbGV2ZWwgb2YgY29ycmVsYXRpb24gYmV0d2VlbiB2YXJpYWJsZXMuICAKCjIuIERhdGEgc3BsaXR0aW5nIAoKV2UgdXNlZCB0aGUgYHJzYW1wbGVgIHBhY2thZ2UgdG8gZmlyc3QgcGVyZm9ybSBhbiBpbml0aWFsIHNwbGl0IG9mIG91ciBkYXRhIGludG8gdHdvIHBpZWNlczogYSB0cmFpbmluZyBzZXQgYW5kIGEgdGVzdGluZyBzZXQuIFRoZSB0cmFpbmluZyBzZXQgd2FzIHVzZWQgdG8gb3B0aW1pemUgdGhlIG1vZGVsLCB3aGlsZSB0aGUgdGVzdGluZyBzZXQgd2FzIHVzZWQgb25seSB0byBldmFsdWF0ZSB0aGUgcGVyZm9ybWFuY2Ugb2Ygb3VyIGZpbmFsIG1vZGVsLiBXZSBhbHNvIHVzZWQgdGhlIGByc2FtcGxlYCBwYWNrYWdlIHRvIGNyZWF0ZSBjcm9zcyB2YWxpZGF0aW9uIHN1YnNldHMgb2Ygb3VyIHRyYWluaW5nIGRhdGEuIFRoaXMgYWxsb3dlZCB1cyB0byBiZXR0ZXIgYXNzZXNzIHRoZSBwZXJmb3JtYW5jZSBvZiBvdXIgdGVzdGVkIG1vZGVscyB1c2luZyBvdXIgdHJhaW5pbmcgZGF0YS4gIAoKMy4gVmFyaWFibGUgYXNzaWdubWVudCBhbmQgcHJlLXByb2Nlc3NpbmcgICAKCldlIHVzZWQgdGhlIGByZWNpcGVzYCBwYWNrYWdlIHRvIGFzc2lnbiB2YXJpYWJsZSByb2xlcyAoc3VjaCBhcyBvdXRjb21lLCBwcmVkaWN0b3IsIGFuZCBpZCB2YXJpYWJsZSkuIFdlIGFsc28gdXNlZCB0aGlzIHBhY2thZ2UgdG8gY3JlYXRlIGEgcmVjaXBlIGZvciBwcmUtcHJvY2Vzc2luZyBvdXIgdHJhaW5pbmcgYW5kIHRlc3RpbmcgZGF0YS4gVGhpcyBpbnZvbHZlZCBzdGVwcyBzdWNoIGFzOiBgIHN0ZXBfZHVtbXlgIHRvIGNyZWF0ZSBkdW1teSBudW1lcmljIGVuY29kaW5ncyBvZiBvdXIgY2F0ZWdvcmljYWwgdmFyaWFibGVzLCBgc3RlcF9jb3JyYCB0byByZW1vdmUgaGlnaGx5IGNvcnJlbGF0ZWQgdmFyaWFibGVzLCBgc3RlcF9uenZgIHRvIHJlbW92ZSBuZWFyIHplcm8gdmFyaWFuY2UgdmFyaWFibGVzIHRoYXQgd291bGQgY29udHJpYnV0ZSBsaXR0bGUgdG8gb3VyIG1vZGVsIGFuZCBwb3RlbnRpYWxseSBhZGQgbm9pc2UuICBXZSBsZWFybmVkIHRoYXQgb25jZSBvdXIgcmVjaXBlIHdhcyBjcmVhdGVkIGFuZCBwcmVwcGVkIHVzaW5nIGBwcmVwKClgd2UgY291bGQgZXh0cmFjdCB0aGUgcHJlLXByb2Nlc3NlZCB0cmFpbmluZyBkYXRhIG9yIG91ciBwcmUtcHJvY2Vzc2VkIHRlc3RpbmcgZGF0YSB1c2luZyBgYmFrZSgpYC4gV2UgYWxzbyBsZWFybmVkIHRoYXQgaWYgd2UgdXNlZCB0aGUgbmV3ZXIgd29ya2Zsb3dzIHBhY2thZ2UgdGhhdCB3ZSBkaWQgbm90IG5lZWQgdG8gdXNlIHRoZSBgcHJlcCgpYCBvciBgYmFrZSgpYCBmdW5jdGlvbnMsIGJ1dCB0aGF0IGl0IGlzIHN0aWxsIHVzZWZ1bCB0byBrbm93IGhvdyB0byBkbyBzbyBpZiB3ZSB3YW50IHRvIGxvb2sgYXQgb3VyIGRhdGEgYW5kIGhvdyB0aGUgcmVjaXBlIGlzIGluZmx1ZW5jaW5nIGl0IG1vcmUgZGVlcGx5LiAgCgo0LiBNb2RlbCBzcGVjaWZpY2F0aW9uLCBmaXR0aW5nLCB0dW5pbmcgYW5kIHBlcmZvcm1hbmNlIGV2YWx1YXRpb24gdXNpbmcgdGhlIHRyYWluaW5nIGRhdGEgIAoKV2UgbGVhcm5lZCB0aGF0IHRoZSBtb2RlbCBuZWVkcyB0byBmaXJzdCBiZSBmaXQgdG8gdGhlIHRyYWluaW5nIGRhdGEuIFdlIGxlYXJuZWQgdGhhdCBpbiBib3RoIGNsYXNzaWZpY2F0aW9uIGFuZCBwcmVkaWN0aW9uLCB0aGUgbW9kZWwgaXMgZml0IHRvIHRoZSB0cmFpbmluZyBkYXRhIGFuZCB0aGUgZXhwbGFuYXRvcnkgdmFyaWFibGVzIGFyZSB1c2VkIHRvIGVzdGltYXRlIG51bWVyaWMgdmFsdWVzIChpbiB0aGUgY2FzZSBvZiBwcmVkaWN0aW9uKSBvciBjYXRlZ29yaWNhbCB2YWx1ZXMgKGluIHRoZSBjYXNlIG9mIGNsYXNzaWZpY2F0aW9uKSBvZiB0aGUgb3V0Y29tZSB2YXJpYWJsZSBvZiBpbnRlcmVzdC4gV2UgbGVhcm5lZCB0aGF0IHdlIHNwZWNpZnkgdGhlIG1vZGVsIGFuZCBpdHMgc3BlY2lmaWNhdGlvbnMgdXNpbmcgdGhlIGBwYXJuc2lwYCBwYWNrYWdlIGFuZCB0aGF0IHdlIGFsc28gdXNlIHRoaXMgcGFja2FnZSB0byBmaXQgdGhlIG1vZGVsIHVzaW5nIHRoZSBgZml0KClgIGZ1bmN0aW9uLiBXZSBsZWFybmVkIHRoYXQgaWYgd2UganVzdCB1c2UgYHBhcnNuaXBgIHRvIGZpdCB0aGUgbW9kZWwsIHRoZW4gd2UgbmVlZCB0byB1c2UgdGhlIHByZS1wcm9jZXNzZWQgdHJhaW5pbmcgZGF0YSAob3V0cHV0IGZyb20gYGJha2UoKWApLiBXZSBsZWFybmVkIHRoYXQgd2UgY2FuIHVzZSB0aGUgcmF3IHRyYWluaW5nIGRhdGEgaWYgd2UgdXNlIHRoZSBgd29ya2Zsb3dzYCBwYWNrYWdlIHRvIGNyZWF0ZSBhIHdvcmtmbG93IHRoYXQgcHJlLXByb2Nlc3NlcyBvdXIgZGF0YSBmb3IgdXMuICAgCgpXZSBsZWFybmVkIHRoYXQgaWYgdGhlIG1vZGVsIGZpdHMgd2VsbCB0aGFuIHRoZSBlc3RpbWF0ZWQgdmFsdWVzIHdpbGwgYmUgdmVyeSBzaW1pbGFyIHRvIHRoZSB0cnVlIG91dGNvbWUgdmFyaWFibGUgdmFsdWVzIGluIG91ciB0cmFpbmluZyBkYXRhLiBXZSBsZWFybmVkIHRoYXQgd2UgY2FuIGFzc2VzcyBtb2RlbCBwZXJmb3JtYW5jZSB1c2luZyB0aGUgYHlhcmRzdGlja2AgcGFja2FnZSB3aXRoIHRoZSBgbWV0cmljc2AgZnVuY3RpbyBvciB0aGUgYHR1bmVgIHBhY2thZ2UgYW5kIHRoZSBgY29sbGVjdF9tZXRyaWNzKClgIGZ1bmN0aW9uIChyZXF1aXJlZCBpZiB1c2luZyBjcm9zcyB2YWxpZGF0aW9uIG9yIHR1bmluZykuIFdlIGFsc28gbGVhcm5lZCB0aGF0IHdlIGNhbiB1c2Ugc3Vic2V0cyBvZiBvdXIgdHJhaW5pbmcgZGF0YSAod2hpY2ggd2UgY3JlYXRlZCB3aXRoIHRoZSBgcnNhbXBsZWAgcGFja2FnZSkgdG8gcGVyZm9ybSBjcm9zcyB2YWxpZGF0aW9uIHRvIGdldCBhIGJldHRlciBlc3RpbWF0ZSBhYm91dCB0aGUgcGVyZm9ybWFuY2Ugb2Ygb3VyIG1vZGVsIHVzaW5nIG91ciB0cmFpbmluZyBkYXRhLCBhcyB3ZSB3YW50IG91ciByZXN1bHRzIHRvIGJlIGdlbmVyYWxpemFibGUgYW5kIHRvIHBlcmZvcm0gd2VsbCB3aXRoIG90aGVyIGRhdGEsIG5vdCBqdXN0IG91ciB0cmFpbmluZyBkYXRhLiBXZSB1c2VkIHRoZSBgZml0X3Jlc2FtcGxlcygpYCBmdW5jdGlvbiBvZiB0aGUgdHVuZSBwYWNrYWdlIHRvIGZpdCBvdXIgbW9kZWwgb24gb3VyIGRpZmZlcmVudCB0cmFpbmluZyBkYXRhIHN1YnNldHMgYW5kIHRoZSBgY29sbGVjdF9tZXRyaWNzKClgIGZ1bmN0aW9uIChhbHNvIG9mIHRoZSBgdHVuZWAgcGFja2FnZSkgdG8gZXZhbHVhdGUgbW9kZWwgcGVyZm9ybWFuY2UgdXNpbmcgdGhlc2Ugc3Vic2V0cy4gIFdlIGFsc28gbGVhcm5lZCB0aGF0IHdlIGNhbiBwb3RlbnRpYWxseSBpbXByb3ZlIG1vZGVsIHBlcmZvcm1hbmNlIGJ5IHR1bmluZyBhc3BlY3RzIGFib3V0IHRoZSBtb2RlbCBjYWxsZWQgW2h5cGVycGFyYW1ldGVyc10oaHR0cHM6Ly9tYWNoaW5lbGVhcm5pbmdtYXN0ZXJ5LmNvbS9kaWZmZXJlbmNlLWJldHdlZW4tYS1wYXJhbWV0ZXItYW5kLWEtaHlwZXJwYXJhbWV0ZXIvKXt0YXJnZXQ9Il9ibGFuayJ9IHRvIGRldGVybWluZSB0aGUgYmVzdCBvcHRpb24gZm9yIG1vZGVsIHBlcmZvcm1hbmNlLiBXZSBsZWFybmVkIHRoYXQgd2UgY2FuIGRvIHRoaXMgdXNpbmcgdGhlIGB0dW5lYCBhbmQgYGRpYWxzYCBwYWNrYWdlcyBhbmQgZXZhbHVhdGluZyB0aGUgcGVyZm9ybWFuY2Ugb2Ygb3VyIG1vZGVsIHdpdGggdGhlIGRpZmZlcmVudCBoeXBlcnBhcmFtZXRlciBvcHRpb25zIGFuZCBvdXIgdHJhaW5pbmcgZGF0YSBzdWJzZXRzIHRoYXQgd2UgdXNlZCBmb3IgY3Jvc3MgdmFsaWRhdGlvbi4gQWZ0ZXIgd2UgdGVzdGVkIHNldmVyYWwgZGlmZmVyZW50IG1ldGhvZHMgdG8gbW9kZWwgb3VyIGRhdGEsIHdlIGNvbXBhcmVkIHRoZW0gdG8gY2hvb3NlIHRoZSBiZXN0IHBlcmZvcm1pbmcgbW9kZWwgYXMgb3VyIGZpbmFsIG1vZGVsLiAgCgoKNS4gT3ZlcmFsbCBtb2RlbCBwZXJmb3JtYW5jZSBldmFsdWF0aW9uICAKCk9uY2Ugd2UgY2hvc2Ugb3VyIGZpbmFsIG1vZGVsLCB3ZSBldmFsdWF0ZWQgdGhlIGZpbmFsIG1vZGVsIHBlcmZvcm1hbmNlIHVzaW5nIHRoZSB0ZXN0aW5nIGRhdGEgdXNpbmcgdGhlIGBsYXN0X2ZpdCgpYCBmdW5jdGlvbiBvZiB0aGUgYHR1bmVgIHBhY2thZ2UuIFRoaXMgZ2l2ZXMgdXMgYSBiZXR0ZXIgZXN0aW1hdGUgYWJvdXQgaG93IHdlbGwgdGhlIG1vZGVsIHdpbGwgcHJlZGljdCBvciBjbGFzc2lmeSB0aGUgb3V0Y29tZSB2YXJpYWJsZSBvZiBpbnRlcmVzdCB3aXRoIG5ldyBpbmRlcGVuZGVudCBkYXRhLiAqKklkZWFsbHkgb25lIHdvdWxkIGFsc28gcGVyZm9ybSBhbiBldmFsdWF0aW9uIHdpdGggaW5kZXBlbmRlbnQgZGF0YSB0byBwcm92aWRlIGEgc2Vuc2Ugb2YgaG93IGdlbmVyYWxpemFibGUgdGhlIG1vZGVsIGlzIHRvIG90aGVyIGRhdGEgc291cmNlcy4qKgoKV2UgYWxzbyBzYXcgdGhhdCB3ZSBjYW4gdXNlIHRoZSBgY29sbGVjdF9wcmVkaWN0aW9ucygpYCBmdW5jdGlvbiBvZiB0aGUgYHR1bmVgIHBhY2thZ2UgdG8gZ2V0IHRoZSBwcmVkaWN0aW9ucyBmb3Igb3VyIHRlc3QgZGF0YS4gV2Ugc2F3IHRoYXQgd2UgY2FuIGdldCBtb3JlIGRldGFpbGVkIHByZWRpY3Rpb24gZGF0YSB1c2luZyB0aGUgYHByZWRpY3QoKWAgZnVuY3Rpb24gb2YgdGhlIGBwYXJzbmlwYCBwYWNrYWdlLgoKPC9kZXRhaWxzPgoKIyMjIwoKCiMjICoqU3VnZ2VzdGVkIEhvbWV3b3JrKioKKioqCgpTdHVkZW50cyBjYW4gcHJlZGljdCBhaXIgcG9sbHV0aW9uIG1vbml0b3IgdmFsdWVzIHVzaW5nIGEgZGlmZmVyZW50IGFsZ29yaXRobSBhbmQgcHJvdmlkZSBhbiBleHBsYW5hdGlvbiBmb3IgaG93IHRoYXQgYWxnb3JpdGhtIHdvcmtzIGFuZCB3aHkgaXQgbWF5IGJlIGEgZ29vZCBjaG9pY2UgZm9yIG1vZGVsaW5nIHRoaXMgZGF0YS4KCgojICoqQWRkaXRpb25hbCBJbmZvcm1hdGlvbioqCioqKgoKIyMgKipIZWxwZnVsIExpbmtzKioKKioqCgoxLiBBIHJldmlldyBvZiBbdGlkeW1vZGVsc10oaHR0cHM6Ly9ydmlld3MucnN0dWRpby5jb20vMjAxOS8wNi8xOS9hLWdlbnRsZS1pbnRyby10by10aWR5bW9kZWxzLyl7dGFyZ2V0PSJfYmxhbmsifSAgCjIuIEEgW2NvdXJzZSBvbiB0aWR5bW9kZWxzXShodHRwczovL2p1bGlhc2lsZ2UuY29tL2Jsb2cvdGlkeW1vZGVscy1tbC1jb3Vyc2UvKXt0YXJnZXQ9Il9ibGFuayJ9IGJ5IEp1bGlhIFNpbGdlICAKMy4gW01vcmUgZXhhbXBsZXMsIGV4cGxhbmF0aW9ucywgYW5kIGluZm8gYWJvdXQgdGlkeW1vZGVscyBkZXZlbG9wbWVudF0oaHR0cHM6Ly93d3cudGlkeW1vZGVscy5vcmcvbGVhcm4vKXt0YXJnZXQ9Il9ibGFuayJ9IGZyb20gdGhlIGRldmVsb3BlcnMgIAo0LiBBIGd1aWRlIGZvciBbcHJlLXByb2Nlc3Npbmcgd2l0aCByZWNpcGVzXShodHRwOi8vd3d3LnJlYmVjY2FiYXJ0ZXIuY29tL2Jsb2cvMjAxOS0wNi0wNl9wcmVfcHJvY2Vzc2luZy8pe3RhcmdldD0iX2JsYW5rIn0gIAo1LiBBIFtndWlkZV0oaHR0cHM6Ly9icmlhdHRlLmdpdGh1Yi5pby9nZ2NvcnIvKXt0YXJnZXQ9Il9ibGFuayJ9IGZvciB1c2luZyBHR2FsbHkgdG8gY3JlYXRlIGNvcnJlbGF0aW9uIHBsb3RzICAKNi4gQSBbZ3VpZGVdKGh0dHBzOi8vd3d3LnRpZHl2ZXJzZS5vcmcvYmxvZy8yMDE4LzExL3BhcnNuaXAtMC0wLTEvKXt0YXJnZXQ9Il9ibGFuayJ9IGZvciB1c2luZyBwYXJzbmlwIHRvIHRyeSBkaWZmZXJlbnQgYWxnb3JpdGhtcyBvciBlbmdpbmVzICAKNy4gQSBbbGlzdCBvZiByZWNpcGUgZnVuY3Rpb25zXShodHRwczovL3RpZHltb2RlbHMuZ2l0aHViLmlvL3JlY2lwZXMvcmVmZXJlbmNlL2luZGV4Lmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gIAo4LiBBIGdyZWF0IGJsb2cgcG9zdCBhYm91dCBbY3Jvc3MgdmFsaWRhdGlvbl0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL3RyYWluLXRlc3Qtc3BsaXQtYW5kLWNyb3NzLXZhbGlkYXRpb24taW4tcHl0aG9uLTgwYjYxYmVjYTRiNil7dGFyZ2V0PSJfYmxhbmsifSAgCjkuIEEgZGlzY3Vzc2lvbiBhYm91dCBbZXZhbHVhdGluZyBtb2RlbCBwZXJmb3JtYW5jZV0oaHR0cHM6Ly9tZWRpdW0uY29tL0BsaW1hdmFsbGFudGluL21ldHJpY3MtdG8tbWVhc3VyZS1tYWNoaW5lLWxlYXJuaW5nLW1vZGVsLXBlcmZvcm1hbmNlLWU4Yzk2MzY2NTQ3Nil7dGFyZ2V0PSJfYmxhbmsifSBmb3IgYSBkZWVwZXIgZXhwbGFuYXRpb24gYWJvdXQgaG93IHRvIGV2YWx1YXRlIG1vZGVsIHBlcmZvcm1hbmNlICAKMTAuIFtSU3R1ZGlvIGNoZWF0c2hlZXRzXShodHRwczovL3JzdHVkaW8uY29tL3Jlc291cmNlcy9jaGVhdHNoZWV0cy8pe3RhcmdldD0iX2JsYW5rIn0KMTEuIEFuIFtleHBsYW5hdGlvbl0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL3N1cGVydmlzZWQtdnMtdW5zdXBlcnZpc2VkLWxlYXJuaW5nLTE0ZjY4ZTMyZWE4ZCl7dGFyZ2V0PSJfYmxhbmsifSBvZiBzdXBlcnZpc2VkIHZzIHVuc3VwZXJ2aXNlZCBtYWNoaW5lIGxlYXJuaW5nIGFuZCBiaWFzLXZhcmlhbmNlIHRyYWRlLW9mZi4KMTIuIEEgdGhvcm91Z2ggW2V4cGxhbmF0aW9uXShodHRwczovL3JveWFsc29jaWV0eXB1Ymxpc2hpbmcub3JnL2RvaS8xMC4xMDk4L3JzdGEuMjAxNS4wMjAyIzp+OnRleHQ9UHJpbmNpcGFsJTIwY29tcG9uZW50JTIwYW5hbHlzaXMlMjAoUENBKSUyMGlzLHZhcmlhYmxlcyUyMHRoYXQlMjBzdWNjZXNzaXZlbHklMjBtYXhpbWl6ZSUyMHZhcmlhbmNlLil7dGFyZ2V0PSJfYmxhbmsifSBvZiBwcmluY2lwYWwgY29tcG9uZW50IGFuYWx5c2lzLgoxMy4gSWYgeW91IGhhdmUgYWNjZXNzLCB0aGlzIGlzIGEgZ3JlYXQgW2Rpc2N1c3Npb25dKGh0dHBzOi8vd3d3LnRhbmRmb25saW5lLmNvbS9kb2kvYWJzLzEwLjEwODAvMDAwMzEzMDUuMTk4NC4xMDQ4MzE4Myl7dGFyZ2V0PSJfYmxhbmsifSAgYWJvdXQgdGhlIGRpZmZlcmVuY2UgYmV0d2VlbiBpbmRlcGVuZGVuY2UsIG9ydGhvZ29uYWxpdHksIGFuZCBsYWNrIG9mIGNvcnJlbGF0aW9uLgoxNC4gR3JlYXQgW3ZpZGVvIGV4cGxhbmF0aW9uXShodHRwczovL3lvdXR1LmJlL19VVkhuZUJVQlcwKXt0YXJnZXQ9Il9ibGFuayJ9IG9mIFBDQS4gIAoKPHU+VGVybXMgYW5kIGNvbmNlcHRzIGNvdmVyZWQ6PC91PiAgCgpbVGlkeXZlcnNlXShodHRwczovL3d3dy50aWR5dmVyc2Uub3JnLyl7dGFyZ2V0PSJfYmxhbmsifSAgCltJbXB1dGF0aW9uXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9JbXB1dGF0aW9uXyhzdGF0aXN0aWNzKSl7dGFyZ2V0PSJfYmxhbmsifSAgCltUcmFuc2Zvcm1hdGlvbl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvRGF0YV90cmFuc2Zvcm1hdGlvbl8oc3RhdGlzdGljcykpe3RhcmdldD0iX2JsYW5rIn0gIApbRGlzY3JldGl6YXRpb25dKGh0dHBzOi8vZW4ud2lraXBlZGlhLm9yZy93aWtpL0Rpc2NyZXRpemF0aW9uX29mX2NvbnRpbnVvdXNfZmVhdHVyZXMpe3RhcmdldD0iX2JsYW5rIn0gIApbRHVtbXkgVmFyaWFibGVzXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9EdW1teV92YXJpYWJsZV8oc3RhdGlzdGljcykpe3RhcmdldD0iX2JsYW5rIn0gIApbT25lLUhvdCBFbmNvZGluZ10oaHR0cHM6Ly9tYWNoaW5lbGVhcm5pbmdtYXN0ZXJ5LmNvbS93aHktb25lLWhvdC1lbmNvZGUtZGF0YS1pbi1tYWNoaW5lLWxlYXJuaW5nLyl7dGFyZ2V0PSJfYmxhbmsifSAgCltEYXRhIFR5cGUgQ29udmVyc2lvbnNdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9oYWJsYXIvdmlnbmV0dGVzL2NvbnZlcnQuaHRtbCl7dGFyZ2V0PSJfYmxhbmsifSAgCltJbnRlcmFjdGlvbl0oaHR0cHM6Ly9zdGF0aXN0aWNzYnlqaW0uY29tL3JlZ3Jlc3Npb24vaW50ZXJhY3Rpb24tZWZmZWN0cy8pe3RhcmdldD0iX2JsYW5rIn0gIApbTm9ybWFsaXphdGlvbl0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvTm9ybWFsaXphdGlvbl8oc3RhdGlzdGljcykpe3RhcmdldD0iX2JsYW5rIn0gIApbRGltZW5zaW9uYWxpdHkgUmVkdWN0aW9uL1NpZ25hbCBFeHRyYWN0aW9uXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9EaW1lbnNpb25hbGl0eV9yZWR1Y3Rpb24pe3RhcmdldD0iX2JsYW5rIn0gIApbUm93IE9wZXJhdGlvbnNdKGh0dHBzOi8vdGFydGFydXMub3JnL2dhcmV0aC9tYXRocy9MaW5lYXJfQWxnZWJyYS9yb3dfb3BlcmF0aW9ucy5wZGYpe3RhcmdldD0iX2JsYW5rIn0gIApbTmVhciBaZXJvIFZhcmFpbmNlXShodHRwczovL3d3dy5yLWJsb2dnZXJzLmNvbS9uZWFyLXplcm8tdmFyaWFuY2UtcHJlZGljdG9ycy1zaG91bGQtd2UtcmVtb3ZlLXRoZW0vKXt0YXJnZXQ9Il9ibGFuayJ9ICAKW1BhcmFtZXRlcnMgYW5kIEh5cGVycGFyYW1ldGVyc10oaHR0cHM6Ly9tYWNoaW5lbGVhcm5pbmdtYXN0ZXJ5LmNvbS9kaWZmZXJlbmNlLWJldHdlZW4tYS1wYXJhbWV0ZXItYW5kLWEtaHlwZXJwYXJhbWV0ZXIvKXt0YXJnZXQ9Il9ibGFuayJ9ICAgCltTdXBlcnZpc2VkIGFuZCBVbnNwZXJ2aXNlZCBMZWFybmluZ10oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL3N1cGVydmlzZWQtdnMtdW5zdXBlcnZpc2VkLWxlYXJuaW5nLTE0ZjY4ZTMyZWE4ZCl7dGFyZ2V0PSJfYmxhbmsifSAgCltQcmluY2lwYWwgQ29tcG9uZW50IEFuYWx5c2lzXShodHRwczovL21lZGl1bS5jb20vQHNhdmFzdGFtaXJrby9wY2EtYS1saW5lYXItdHJhbnNmb3JtYXRpb24tZjhhYWNkNGViMDA3KXt0YXJnZXQ9Il9ibGFuayJ9ICAKW0xpbmVhciBDb21iaW5hdGlvbnNdKGh0dHBzOi8vd3d3Lm1hdGhib290Y2FtcHMuY29tL2xpbmVhci1jb21iaW5hdGlvbnMtdmVjdG9ycy8pe3RhcmdldD0iX2JsYW5rIn0gIApbRGVjaXNpb24gVHJlZV0oaHR0cHM6Ly9tZWRpdW0uY29tL2dyZXlhdG9tL2RlY2lzaW9uLXRyZWVzLWEtc2ltcGxlLXdheS10by12aXN1YWxpemUtYS1kZWNpc2lvbi1kYzUwNmE0MDNhZWIpe3RhcmdldD0iX2JsYW5rIn0gIApbUmFuZG9tIEZvcmVzdF0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL2RlY2lzaW9uLXRyZWUtZW5zZW1ibGVzLWJhZ2dpbmctYW5kLWJvb3N0aW5nLTI2NmE4YmE2MGZkOSl7dGFyZ2V0PSJfYmxhbmsifSAgCgoKIDx1PioqUGFja2FnZXMgdXNlZCBpbiB0aGlzIGNhc2Ugc3R1ZHk6KiogPC91PgoKUGFja2FnZSAgIHwgVXNlIGluIHRoaXMgY2FzZSBzdHVkeSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKLS0tLS0tLS0tLSB8LS0tLS0tLS0tLS0tLQpbaGVyZV0oaHR0cHM6Ly9naXRodWIuY29tL2plbm55YmMvaGVyZV9oZXJlKXt0YXJnZXQ9Il9ibGFuayJ9ICAgICAgIHwgdG8gZWFzaWx5IGxvYWQgYW5kIHNhdmUgZGF0YQpbcmVhZHJdKGh0dHBzOi8vcmVhZHIudGlkeXZlcnNlLm9yZy8pe3RhcmdldD0iX2JsYW5rIn0gICAgICB8IHRvIGltcG9ydCB0aGUgQ1NWIGZpbGUgZGF0YQpbZHBseXJdKGh0dHBzOi8vZHBseXIudGlkeXZlcnNlLm9yZy8pe3RhcmdldD0iX2JsYW5rIn0gICAgICB8IHRvIHZpZXcvYXJyYW5nZS9maWx0ZXIvc2VsZWN0L2NvbXBhcmUgc3BlY2lmaWMgc3Vic2V0cyBvZiB0aGUgZGF0YSAKW3NraW1yXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvc2tpbXIvaW5kZXguaHRtbCl7dGFyZ2V0PSJfYmxhbmsifSAgICAgIHwgdG8gZ2V0IGFuIG92ZXJ2aWV3IG9mIGRhdGEKW3N1bW1hcnl0b29sc10oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL3NraW1yL2luZGV4Lmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gICAgICB8IHRvIGdldCBhbiBvdmVydmlldyBvZiBkYXRhIGluIGEgZGlmZmVyZW50IHN0eWxlClttYWdyaXR0cl0oaHR0cHM6Ly9tYWdyaXR0ci50aWR5dmVyc2Uub3JnL2FydGljbGVzL21hZ3JpdHRyLmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gICB8IHRvIHVzZSB0aGUgYCU8PiVgIHBpcHBpbmcgb3BlcmF0b3IgCltjb3JycGxvdF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL2NvcnJwbG90L3ZpZ25ldHRlcy9jb3JycGxvdC1pbnRyby5odG1sKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gbWFrZSBsYXJnZSBjb3JyZWxhdGlvbiBwbG90cwpbR0dhbGx5XShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvR0dhbGx5L0dHYWxseS5wZGYpe3RhcmdldD0iX2JsYW5rIn0gfCB0byBtYWtlIHNtYWxsZXIgY29ycmVsYXRpb24gcGxvdHMgIApbcnNhbXBsZV0oaHR0cHM6Ly90aWR5bW9kZWxzLmdpdGh1Yi5pby9yc2FtcGxlL2FydGljbGVzL0Jhc2ljcy5odG1sKXt0YXJnZXQ9Il9ibGFuayJ9ICAgfCB0byBzcGxpdCB0aGUgZGF0YSBpbnRvIHRlc3RpbmcgYW5kIHRyYWluaW5nIHNldHMgYW5kIHRvIHNwbGl0IHRoZSB0cmFpbmluZyBzZXQgZm9yIGNyb3NzLXZhbGlkYXRpb24gIApbcmVjaXBlc10oaHR0cHM6Ly90aWR5bW9kZWxzLmdpdGh1Yi5pby9yZWNpcGVzLyl7dGFyZ2V0PSJfYmxhbmsifSAgIHwgdG8gcHJlLXByb2Nlc3MgZGF0YSBmb3IgbW9kZWxpbmcgaW4gYSB0aWR5IGFuZCByZXByb2R1Y2libGUgd2F5IGFuZCB0byBleHRyYWN0IHByZS1wcm9jZXNzZWQgZGF0YSAobWFqb3IgZnVuY3Rpb25zIGFyZSBgcmVjaXBlKClgICwgYHByZXAoKWAgYW5kIHZhcmlvdXMgdHJhbnNmb3JtYXRpb24gYHN0ZXBfKigpYCBmdW5jdGlvbnMsIGFzIHdlbGwgYXMgYGJha2VgIHdoaWNoIGV4dHJhY3RzIHByZS1wcm9jZXNzZWQgdHJhaW5pbmcgZGF0YSAodXNlZCB0byByZXF1aXJlIGBqdWljZSgpYCkgYW5kIGFwcGxpZXMgcmVjaXBlIHByZXByb2Nlc3Npbmcgc3RlcHMgdG8gdGVzdGluZyBkYXRhKS4gU2VlIFtoZXJlXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvcmVjaXBlcy9yZWNpcGVzLnBkZil7dGFyZ2V0PSJfYmxhbmsifSAgZm9yIG1vcmUgaW5mby4KW3BhcnNuaXBdKGh0dHBzOi8vdGlkeW1vZGVscy5naXRodWIuaW8vcGFyc25pcC8pe3RhcmdldD0iX2JsYW5rIn0gICB8IGFuIGludGVyZmFjZSB0byBjcmVhdGUgbW9kZWxzIChtYWpvciBmdW5jdGlvbnMgYXJlICBgZml0KClgLCBgc2V0X2VuZ2luZSgpYCkKW3lhcmRzdGlja10oaHR0cHM6Ly90aWR5bW9kZWxzLmdpdGh1Yi5pby95YXJkc3RpY2svKXt0YXJnZXQ9Il9ibGFuayJ9ICAgfCB0byBldmFsdWF0ZSB0aGUgcGVyZm9ybWFuY2Ugb2YgbW9kZWxzClticm9vbV0oaHR0cHM6Ly93d3cudGlkeXZlcnNlLm9yZy9ibG9nLzIwMTgvMDcvYnJvb20tMC01LTAvKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gZ2V0IHRpZHkgb3V0cHV0IGZvciBvdXIgbW9kZWwgZml0IGFuZCBwZXJmb3JtYW5jZQpbZ2dwbG90Ml0oaHR0cHM6Ly9nZ3Bsb3QyLnRpZHl2ZXJzZS5vcmcvKXt0YXJnZXQ9Il9ibGFuayJ9ICAgIHwgdG8gbWFrZSB2aXN1YWxpemF0aW9ucyB3aXRoIG11bHRpcGxlIGxheWVycwpbZGlhbHNdKGh0dHBzOi8vd3d3LnRpZHl2ZXJzZS5vcmcvYmxvZy8yMDE5LzEwL2RpYWxzLTAtMC0zLyl7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIHNwZWNpZnkgaHlwZXItcGFyYW1ldGVyIHR1bmluZwpbdHVuZV0oaHR0cHM6Ly90dW5lLnRpZHltb2RlbHMub3JnLyl7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIHBlcmZvcm0gY3Jvc3MgdmFsaWRhdGlvbiwgdHVuZSBoeXBlci1wYXJhbWV0ZXJzLCBhbmQgZ2V0IHBlcmZvcm1hbmNlIG1ldHJpY3MKW3dvcmtmbG93c10oaHR0cHM6Ly93d3cucmRvY3VtZW50YXRpb24ub3JnL3BhY2thZ2VzL3dvcmtmbG93cy92ZXJzaW9ucy8wLjEuMSl7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIGNyZWF0ZSBtb2RlbGluZyB3b3JrZmxvdyB0byBzdHJlYW1saW5lIHRoZSBtb2RlbGluZyBwcm9jZXNzClt2aXBdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy92aXAvdmlwLnBkZil7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIGNyZWF0ZSB2YXJpYWJsZSBpbXBvcnRhbmNlIHBsb3RzCltyYW5kb21Gb3Jlc3RdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9yYW5kb21Gb3Jlc3QvcmFuZG9tRm9yZXN0LnBkZil7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIHBlcmZvcm0gdGhlIHJhbmRvbSBmb3Jlc3QgYW5hbHlzaXMKW2RvUGFyYWxsZWxdKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9kb1BhcmFsbGVsL2RvUGFyYWxsZWwucGRmKSB8IHRvIGZpdCBjcm9zcyB2YWxpZGF0aW9uIHNhbXBsZXMgaW4gcGFyYWxsZWwgCltzdHJpbmdyXShodHRwczovL3N0cmluZ3IudGlkeXZlcnNlLm9yZy9hcnRpY2xlcy9zdHJpbmdyLmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gICAgfCB0byBtYW5pcHVsYXRlIHRoZSB0ZXh0IHRoZSBtYXAgZGF0YQpbdGlkeXJdKGh0dHBzOi8vdGlkeXIudGlkeXZlcnNlLm9yZy8pe3RhcmdldD0iX2JsYW5rIn0gICAgICB8IHRvIHNlcGFyYXRlIGRhdGEgd2l0aGluIGEgY29sdW1uIGludG8gbXVsdGlwbGUgY29sdW1ucwpbcm5hdHVyYWxlYXJ0aF0oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL3JuYXR1cmFsZWFydGgvUkVBRE1FLmh0bWwpe3RhcmdldD0iX2JsYW5rIn0gfCB0byBnZXQgdGhlIGdlb21ldHJ5IGRhdGEgZm9yIHRoZSBlYXJ0aCB0byBwbG90IHRoZSBVUwpbbWFwc10oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL21hcHMvbWFwcy5wZGYpe3RhcmdldD0iX2JsYW5rIn0gfCB0byBnZXQgbWFwIGRhdGFiYXNlIGRhdGEgYWJvdXQgY291bnRpZXMgdG8gZHJhdyB0aGVtIG9uIG91ciBVUyBtYXAKW3NmXShodHRwczovL3Itc3BhdGlhbC5naXRodWIuaW8vc2YvKXt0YXJnZXQ9Il9ibGFuayJ9ICB8IHRvIGNvbnZlcnQgdGhlIG1hcCBkYXRhIGludG8gYSBkYXRhIGZyYW1lCltsd2dlb21dKGh0dHBzOi8vY3Jhbi5yLXByb2plY3Qub3JnL3dlYi9wYWNrYWdlcy9sd2dlb20vbHdnZW9tLnBkZil7dGFyZ2V0PSJfYmxhbmsifSB8IHRvIHVzZSB0aGUgYHNmYCBmdW5jdGlvbiB0byBjb252ZXJ0IHRoZSBtYXAgZ2VvZ3JhcGhpY2FsIGRhdGEKW3JnZW9zXShodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvcmdlb3Mvcmdlb3MucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gdXNlIGdlb21ldHJ5IGRhdGEKW3BhdGNod29ya10oaHR0cHM6Ly9jcmFuLnItcHJvamVjdC5vcmcvd2ViL3BhY2thZ2VzL3BhdGNod29yay9wYXRjaHdvcmsucGRmKXt0YXJnZXQ9Il9ibGFuayJ9IHwgdG8gYWxsb3cgcGxvdHMgdG8gYmUgY29tYmluZWQKCgojIyAqKlNlc3Npb24gaW5mbyoqCioqKgoKCmBgYHtyfQpzZXNzaW9uSW5mbygpCmBgYAoKKipFc3RpbWF0ZSBvZiBSTWFya2Rvd24gQ29tcGlsYXRpb24gVGltZTogKioKCmBgYHtyLCBlY2hvPUZBTFNFfQpybWFya2Rvd246OjpwZXJmX3RpbWVyX3N0b3AoInJlbmRlciIpCnB0cyA9IHJtYXJrZG93bjo6OnBlcmZfdGltZXJfc3VtbWFyeSgpCmNhdCgiQWJvdXQiLCByb3VuZChwdHMkdGltZVsxXS8xMDAwICsgNSksICItIiwgcm91bmQocHRzJHRpbWVbMV0vMTAwMCArIDE1KSwic2Vjb25kcyIpCmBgYAoKVGhpcyBjb21waWxhdGlvbiB0aW1lIHdhcyBtZWFzdXJlZCBvbiBhIFBDIG1hY2hpbmUgb3BlcmF0aW5nIG9uIFdpbmRvd3MgMTAuIFRoaXMgcmFuZ2Ugc2hvdWxkIG9ubHkgYmUgdXNlZCBhcyBhbiBlc3RpbWF0ZSBhcyBjb21waWxhdGlvbiB0aW1lIHdpbGwgdmFyeSB3aXRoIGRpZmZlcmVudCBtYWNoaW5lcyBhbmQgb3BlcmF0aW5nIHN5c3RlbXMuCgojIyAqKkFja25vd2xlZGdtZW50cyoqCioqKgoKV2Ugd291bGQgbGlrZSB0byBhY2tub3dsZWRnZSBbUm9nZXIgUGVuZ10oaHR0cDovL3d3dy5iaW9zdGF0Lmpoc3BoLmVkdS9+cnBlbmcvKSwgW01lZ2FuIExhdHNoYXddKGh0dHBzOi8vd3d3Lmpoc3BoLmVkdS9mYWN1bHR5L2RpcmVjdG9yeS9wcm9maWxlLzE3MDgvbWVnYW4td2VpbC1sYXRzaGF3KSwgYW5kIFtLaXJzdGVuIEtvZWhsZXJdKGh0dHBzOi8vd3d3Lmpoc3BoLmVkdS9mYWN1bHR5L2RpcmVjdG9yeS9wcm9maWxlLzI5Mjgva2lyc3Rlbi1rb2VobGVyKSBmb3IgYXNzaXN0aW5nIGluIGZyYW1pbmcgdGhlIG1ham9yIGRpcmVjdGlvbiBvZiB0aGUgY2FzZSBzdHVkeS4KCldlIHdvdWxkIGxpa2UgdG8gYWNrbm93bGVkZ2UgW01pY2hhZWwgQnJlc2hvY2tdKGh0dHBzOi8vbWJyZXNob2NrLmdpdGh1Yi5pby8pIGZvciBoaXMgY29udHJpYnV0aW9ucyB0byB0aGlzIGNhc2Ugc3R1ZHkgYW5kIGRldmVsb3BpbmcgdGhlIGBPQ1NkYXRhYCBwYWNrYWdlLgoKV2Ugd291bGQgYWxzbyBsaWtlIHRvIGFja25vd2xlZGdlIHRoZSBbQmxvb21iZXJnIEFtZXJpY2FuIEhlYWx0aCBJbml0aWF0aXZlXShodHRwczovL2FtZXJpY2FuaGVhbHRoLmpodS5lZHUvKSBmb3IgZnVuZGluZyB0aGlzIHdvcmsuIAoKCjxzY3JpcHQgdHlwZT0ndGV4dC9qYXZhc2NyaXB0JyBpZD0nY2x1c3RybWFwcycgc3JjPScvL2Nkbi5jbHVzdHJtYXBzLmNvbS9tYXBfdjIuanM/Y2w9MDgwODA4Jnc9YSZ0PXR0JmQ9N1lULUVER2E0TVV3UVhTUXhxZnY5TmQ5TnQ4NTJiN3BsR2RTNlVRSk8xUSZjbz1mZmZmZmYmY21vPTNhY2MzYSZjbW49ZmY1MzUzJmN0PTgwODA4MCc+PC9zY3JpcHQ+Cg==

Open Case Studies: Predicting Annual Air Pollution

Motivation

Major types of air pollutants

Particulate pollution

Negative impact of particulate exposure on health

Sparse monitoring is problematic for Public Health

Machine learning offers a solution

Main Question

Learning Objectives

Context

Limitations

What are the data?

Start with a question

Our outcome variable

[image curtesy of Kirsten Koehler]

Our features (predictor variables)

Data Import

Data Exploration and Wrangling

skim package

Evaluate correlation

What is machine learning?

ML as an optimization problem

The parts of an ML problem

Machine learning with tidymodels

Overview

The tidymodels ecosystem

Benefits of tidymodels

tidymodels Steps

Splitting the data

Preparing for pre-processing the data

Step 1: Specify variables roles with recipe() function

Step 2: Specify the pre-processing steps with step*() functions

Running the pre-processing

Step 1: Update the recipe with training data using prep()

Step 2: Extract pre-processed training data using bake()

Step 3: Extract pre-processed testing data using bake()

Specifying the model

Fitting the model

Assessing the model fit

Model performance

Visualizing model performance

Quantifying model performance

Cross validation

Creating the \(v\)-folds using rsample

Assessing model performance on \(v\)-folds using tune

Data Analysis

Random Forest

Model Comparison

Model tuning

Final model performance evaluation

Data Visualization

`skim` package

Machine learning with `tidymodels`

The `tidymodels` ecosystem

Benefits of `tidymodels`

`tidymodels` Steps

Step 1: Specify variables roles with `recipe()` function

Step 2: Specify the pre-processing steps with `step*()` functions

Step 1: Update the recipe with training data using `prep()`

Step 2: Extract pre-processed training data using `bake()`

Step 3: Extract pre-processed testing data using `bake()`

Creating the \(v\)-folds using `rsample`

Assessing model performance on \(v\)-folds using `tune`