# 12 Experiment Logging

library(rtemis)
  .:rtemis 0.8.0: Welcome, egenn
[x86_64-apple-darwin17.0 (64-bit): Defaulting to 4/4 available cores]
Documentation & vignettes: https://rtemis.netlify.com

## 12.1 Log to directory

You can save any rtemis supervised learning model to disk by specifying an output directory using the outdir argument:

iris.cart <- s.CART(iris,
outdir = "./Results/iris_CART")

This will save:

• A text .log file with the console output
• A PDF with True vs. Fitted for Regression and a confusion matrix for Classification
• A RDS file with the trained model (i.e. the R6 object iris.cart in the above example)

The RDS files can be shared with others and loaded back into R at any time.

When running a series of experiments it makes sense to use the outdir argument to save models to disk for reference.

## 12.2 Interactive logging

The above method of specifying an outdir is the main way to save models to disk. In practice, we often train a series of models interactively and would like to keep track of what we have tried and how it worked out. rtemis includes rtModLogger to help with that. You first create a new logger object, think of it as a container that will hold model parameters and error metrics - not the model itself. Once the logger is created you can add any models to it:

Some synthetic data:

x <- rnormmat(400, 400, seed = 2019)
w <- rnorm(400)
y <- c(x %*% w + rnorm(400))

dat <- data.frame(x, y)
res <- resample(dat)
[2020-06-23 08:22:13 resample] Input contains more than one columns; will stratify on last
[[ Resampling Parameters ]]
n.resamples: 10
resampler: strat.sub
stratify.var: y
train.p: 0.75
strat.n.bins: 4

[2020-06-23 08:22:13 resample] Created 10 stratified subsamples

dat.train <- dat[res$Subsample_1, ] dat.test <- dat[-res$Subsample_1, ]

Initialize a new logger object:

logger <- rtModLogger$new() logger [[ .:rtemis Supervised Model Logger ]] Contents: no models yet  ### 12.2.1 Train some models and add them to the logger: mod.ridge <- s.GLMNET(dat.train, dat.test, alpha = 0, lambda = .01, verbose = F) logger$add(mod.ridge)
[2020-06-23 08:22:17 logger$add] Added 1 model to logger; 1 total  mod.lasso <- s.GLMNET(dat.train, dat.test, alpha = 1, lambda = .01, verbose = F) logger$add(mod.lasso)
[2020-06-23 08:22:17 logger$add] Added 1 model to logger; 2 total  mod.elnet <- s.GLMNET(dat.train, dat.test, alpha = .5, lambda = .01, verbose = F) logger$add(mod.elnet)

results <- logger$summary() results  Train Rsq Test Rsq GLMNET_1 0.9999773 0.5522008 GLMNET_2 0.9998142 0.7465513 GLMNET_3 0.9999467 0.7420425 attr(,"metric") [1] "Rsq" ### 12.2.4 Write model hyperparameters and performance to a multi-sheet XLSX file: logger$tabulate(filename = "~/Desktop/Results/model_metrics.xlsx")
Warning in file.create(to[okay]): cannot create file '~/Desktop/Results/
model_metrics.xlsx', reason 'No such file or directory'

In this example, the XLSX file will contain 3 sheets, one per model. We can save the output of tabulate to a list as well:

tbl <- logger$tabulate() tbl$GLMNET_1
tbl$GLMNET_2 tbl$GLMNET_3