12 Experiment Logging
.:rtemis 0.8.0: Welcome, egenn [x86_64-apple-darwin17.0 (64-bit): Defaulting to 4/4 available cores] Documentation & vignettes: https://rtemis.netlify.com
12.1 Log to directory
You can save any rtemis supervised learning model to disk by specifying an output directory using the
This will save:
- A text .log file with the console output
- A PDF with True vs. Fitted for Regression and a confusion matrix for Classification
- A RDS file with the trained model (i.e. the R6 object
iris.cartin the above example)
The RDS files can be shared with others and loaded back into R at any time.
When running a series of experiments it makes sense to use the
outdir argument to save models to disk for reference.
12.2 Interactive logging
The above method of specifying an
outdir is the main way to save models to disk. In practice, we often train a series of models interactively and would like to keep track of what we have tried and how it worked out. rtemis includes
rtModLogger to help with that. You first create a new logger object, think of it as a container that will hold model parameters and error metrics - not the model itself. Once the logger is created you can add any models to it:
Some synthetic data:
[2020-06-23 08:22:13 resample] Input contains more than one columns; will stratify on last [[ Resampling Parameters ]] n.resamples: 10 resampler: strat.sub stratify.var: y train.p: 0.75 strat.n.bins: 4 [2020-06-23 08:22:13 resample] Created 10 stratified subsamples
Initialize a new logger object:
[[ .:rtemis Supervised Model Logger ]] Contents: no models yet
12.2.1 Train some models and add them to the logger:
[2020-06-23 08:22:17 logger$add] Added 1 model to logger; 1 total
[2020-06-23 08:22:17 logger$add] Added 1 model to logger; 2 total
[2020-06-23 08:22:18 logger$add] Added 1 model to logger; 3 total
12.2.3 Get a quick summary:
Train Rsq Test Rsq GLMNET_1 0.9999773 0.5522008 GLMNET_2 0.9998142 0.7465513 GLMNET_3 0.9999467 0.7420425 attr(,"metric")  "Rsq"
12.2.4 Write model hyperparameters and performance to a multi-sheet XLSX file:
Warning in file.create(to[okay]): cannot create file '~/Desktop/Results/ model_metrics.xlsx', reason 'No such file or directory'
In this example, the XLSX file will contain 3 sheets, one per model. We can save the output of
tabulate to a list as well: