Machine Learning with dabl¶
Philosophy¶
The idea behind dabl is to jump-start your supervised learning task. dabl has several tools that make it easy to clean and inspect your data, and create strong baseline models.
Building machine learning models is an inherently iterative task with a human in the loop. Big jumps in performance are often achieved by better understanding of the data and task, and more appropriate features. dabl tries to provide as much insight into the data as possible, and enable interactive analysis.
Many analyses start with the same rote tasks of cleaning and basic data visualization, and initial modeling. dabl tries to make these steps as easy as possible, so that you can spend your time thinking about the problem and creating more interesting custom analyses.
There are two main packages that dabl takes inspiration from and that dabl builds upon: scikit-learn and auto-sklearn. The design philosophies and use-cases are quite different, however.
Scikit-learn provides many essential building blocks, but is built on the idea to do exactly what the user asks for. That requires specifying every step of the processing in detail. dabl on the other hand has a best-guess philosophy: it tries to do something sensible, and then provides tools for the user to inspect and evaluate the results to judge them.
auto-sklearn is completely automatic and black-box. It searches a vast space of models and constructs complex ensembles of high accuracy, taking a substantial amount of computation and time in the process. The goal of auto-sklearn is to build the best model possible given the data. dabl, conversely, tries to enable the user to quickly iterate and get a grasp on the properties of the data at hand and the fitted models.
dabl is meant to support you in the following tasks, in order:
Data cleaning¶
>>> import dabl
>>> import pandas as pd
>>> data = pd.read_csv(dabl.datasets.data_path("adult.csv.gz"))
>>> data_clean = dabl.clean(data)[::10]
The first step in any data analysis is data cleaning. dabl tries to detect the
types of your data and apply appropriate conversions. It also tries to detect
potential data quality issues.
The field of data cleaning is impossibly broad, and dabl’s approaches are by no
means sophisticated. The goal of dabl is to get the data “clean enough” to
create useful visualizations and models, and to allow users to perform
custom cleaning operations themselves.
In particular if the detection of semantic types (continuous, categorical,
ordinal, text, etc) fails, the user can provide type_hints:
>>> data_clean = dabl.clean(data, type_hints={"capital-gain": "continuous"})
Exploratory Data analysis¶
>>> dabl.plot(data, target_col="income")
Target looks like classification
Linear Discriminant Analysis training set score: ...
The next step in any task should be exploratory data analysis. dabl provides a high-level interface that summarizes several common high-level plots. For low dimensional datasets, all features are shown; for high dimensional datasets, only the most informative features for the given task are shown. This is clearly not guaranteed to surface all interesting aspects with the data, or to find all data quality issues. However, it will give you a quick insight in to what are the important features, their interactions, and how hard the problem might be. It also allows a good assessment of whether there is any data leakage through spurious representations of the target in the data.
Initial Model Building¶
>>> ec = dabl.SimpleClassifier(random_state=0).fit(data, target_col="income")
DummyClassifier(strategy='prior')
accuracy: 0.759 average_precision: 0.241 recall_macro: 0.500 roc_auc: 0.500
new best (using recall_macro):
accuracy 0.759
average_precision 0.241
recall_macro 0.500
roc_auc 0.500
Name: DummyClassifier(strategy='prior'), dtype: float64
GaussianNB()
accuracy: 0.407 average_precision: 0.288 recall_macro: 0.605 roc_auc: 0.607
new best (using recall_macro):
accuracy 0.407
average_precision 0.288
recall_macro 0.605
roc_auc 0.607
Name: GaussianNB(), dtype: float64
MultinomialNB()
accuracy: 0.831 average_precision: 0.773 recall_macro: 0.815 roc_auc: 0.908
new best (using recall_macro):
accuracy 0.831
average_precision 0.773
recall_macro 0.815
roc_auc 0.908
Name: MultinomialNB(), dtype: float64
DecisionTreeClassifier(class_weight='balanced', max_depth=1)
accuracy: 0.710 average_precision: 0.417 recall_macro: 0.759 roc_auc: 0.759
DecisionTreeClassifier(class_weight='balanced', max_depth=5)
accuracy: 0.784 average_precision: 0.711 recall_macro: 0.811 roc_auc: 0.894
DecisionTreeClassifier(class_weight='balanced', min_impurity_decrease=0.01)
accuracy: 0.718 average_precision: 0.561 recall_macro: 0.779 roc_auc: 0.848
LogisticRegression(C=0.1, class_weight='balanced')
accuracy: 0.819 average_precision: 0.789 recall_macro: 0.832 roc_auc: 0.915
new best (using recall_macro):
accuracy 0.819
average_precision 0.789
recall_macro 0.832
roc_auc 0.915
Name: LogisticRegression(C=0.1, class_weight='balanced'), dtype: float64
Best model:
LogisticRegression(C=0.1, class_weight='balanced')
Best Scores:
accuracy 0.819
average_precision 0.789
recall_macro 0.832
roc_auc 0.915
Name: LogisticRegression(C=0.1, class_weight='balanced'), dtype: float64
The SimpleClassifier first tries several baseline and instantaneous models,
potentially on subsampled data, to get an idea of what a low baseline should be.
This again is a good place to surface data leakage, as well as find the main
discriminative features in the dataset. The SimpleClassifier allows
specifying data in the scikit-learn-style fit(X, y) with a 1d y and
features X, or with X being a dataframe and specifying the target
column inside of X as target_col.
The SimpleClassifier also performs preprocessing such as missing value imputation and one-hot encoding. You can inspect the model using:
>>> dabl.explain(ec)
This can lead to additional insights and guide custom processing and cleaning of the data.
Enhanced Model Building¶
>>> # ac = AnyClassifier().fit(data, target_col="income") not implemented yet
After creating an initial model, it’s interesting to explore more powerful
models such as tree ensembles. AnyClassifier searches over a space of
models that commonly perform well, and identifies promising candidates. If
your goal is prediction, AnyClassifier can provide a strong baseline for
further investigation. Again, we can inspect our model to understand it
better:
>>> # explain(ac)
Explainable Model Building¶
TODO this is not done yet!
Sometimes, explainability of a model can be more important than performance. A complex model can serve as a good benchmark on what is achievable on a certain dataset. After this benchmark is established, it is interesting to see if we can build a model that is interpretable while still providing competitive performance.
>>> # xc = ExplainableClassifier().fit(data, target_col="income")
Searching optimal parameters with successive halving¶
dabl provides the dabl.search.GridSuccessiveHalving and
dabl.search.RandomSuccessiveHalving estimators that can be used to
search a parameter space using successive halving 1 2. Successive
halving is an iterative selection process where all candidates are evaluated
with a small amount of resources at the first iteration. Only a subset of
these candidates are selected for the next iteration, which will be
allocated more resources. What defines a resource is typically the number of
samples to train on, or the number of trees for a gradient boosting /
decision forest estimator.
As illustrated in the figure below, only a small subset of candidates ‘survive’ until the last iteration. These are the candidates that have consistently been part of the best candidates across all iterations.
#FIXME: Put figure from plot_successive_halving_iterations.py here
The amount of resources r_i allocated for each candidate at iteration
i is controlled by the parameters ratio and r_min as follows:
r_i = ratio**i * r_min
r_min is the amount of resources used at the first iteration and
ratio defines the proportions of candidates that will be selected for
the next iteration:
n_candidates_to_keep = n_candidates_at_i // ratio
Note that each r_i is a multiple of both ratio and r_min.
Choosing the budget¶
By default, the budget is defined as the number of samples. That is, each
iteration will use an increasing amount of samples to train on. You can however
manually specify a parameter to use as the budget with the budget_on
parameter. Here is an example where the budget is defined as the number of
iterations of a random forest:
>>> from sklearn.datasets import make_classification
>>> from sklearn.ensemble import RandomForestClassifier
>>> import pandas as pd
>>> from dabl.search import GridSuccessiveHalving
>>>
>>> parameters = {'max_depth': [3, 5, 10],
... 'min_samples_split': [2, 5, 10]}
>>> base_estimator = RandomForestClassifier(random_state=0)
>>> X, y = make_classification(n_samples=1000, random_state=0)
>>> sh = GridSuccessiveHalving(base_estimator, parameters, cv=5,
... ratio=2,
... budget_on='n_estimators',
... max_budget=30,
... random_state=0,
... ).fit(X, y)
>>> sh.best_estimator_
RandomForestClassifier(...)
Note that it is not possible to budget on a parameter that is part of the parameter space.
Exhausting the budget¶
As mentioned above, the first iteration uses r_min resources. If you have
a big budget, this may be a waste of resource:
>>> from sklearn.datasets import make_classification
>>> from sklearn.svm import SVC
>>> import pandas as pd
>>> from dabl.search import GridSuccessiveHalving
>>> parameters = {'kernel': ('linear', 'rbf'),
... 'C': [1, 10, 100]}
>>> base_estimator = SVC(gamma='scale')
>>> X, y = make_classification(n_samples=1000)
>>> sh = GridSuccessiveHalving(base_estimator, parameters, cv=5,
... ratio=2).fit(X, y)
>>> results = pd.DataFrame.from_dict(sh.cv_results_)
>>> results.groupby('iter').r_i.unique()
iter
0 [20]
1 [40]
2 [80]
Name: r_i, dtype: object
The search process will only use 80 resources at most, while our maximum budget
is n_samples=1000. Note in this case that r_min = r_0 = 20. In order
for the last iteration to use as many resources as possible, you can use the
force_exhaust_budget parameter:
>>> sh = GridSuccessiveHalving(base_estimator, parameters, cv=5,
... ratio=2, force_exhaust_budget=True,
... ).fit(X, y)
>>> results = pd.DataFrame.from_dict(sh.cv_results_)
>>> results.groupby('iter').r_i.unique()
iter
0 [250]
1 [500]
2 [1000]
Name: r_i, dtype: object
Since force_exhaust_budget chooses an appropriate r_min to start
with, r_min must be set to ‘auto’.
Aggressive elimination of candidates¶
Ideally, we want the last iteration to evaluate ratio candidates. We then
just have to pick the best one. When the number budget is small with respect to
the number of candidates, the last iteration may have to evaluate more than
ratio candidates:
>>> from sklearn.datasets import make_classification
>>> from sklearn.svm import SVC
>>> import pandas as pd
>>> from dabl.search import GridSuccessiveHalving
>>>
>>>
>>> parameters = {'kernel': ('linear', 'rbf'),
... 'C': [1, 10, 100]}
>>> base_estimator = SVC(gamma='scale')
>>> X, y = make_classification(n_samples=1000)
>>> sh = GridSuccessiveHalving(base_estimator, parameters, cv=5,
... ratio=2,
... max_budget=40,
... aggressive_elimination=False,
... ).fit(X, y)
>>> results = pd.DataFrame.from_dict(sh.cv_results_)
>>> results.groupby('iter').r_i.unique()
iter
0 [20]
1 [40]
Name: r_i, dtype: object
>>> results.groupby('iter').r_i.count() # number of candidates used at each iteration
iter
0 6
1 3
Name: r_i, dtype: int64
Since we cannot use more than max_budget=40 resources, the process has to
stop at the second iteration which evaluates more than ratio=2 candidates.
Using the aggressive_elimination parameter, you can force the search
process to end up with less than ratio candidates at the last
iteration. To do this, the process will eliminate as many candidates as
necessary using r_min resources:
>>> sh = GridSuccessiveHalving(base_estimator, parameters, cv=5,
... ratio=2,
... max_budget=40,
... aggressive_elimination=True,
... ).fit(X, y)
>>> results = pd.DataFrame.from_dict(sh.cv_results_)
>>> results.groupby('iter').r_i.unique()
iter
0 [20]
1 [20]
2 [40]
Name: r_i, dtype: object
>>> results.groupby('iter').r_i.count() # number of candidates used at each iteration
iter
0 6
1 3
2 2
Name: r_i, dtype: int64
Notice that we end with 2 candidates at the last iteration since we have
eliminated enough candidates during the first iterations, using r_i = r_min =
20.
References:
- 1
K. Jamieson, A. Talwalkar, Non-stochastic Best Arm Identification and Hyperparameter Optimization, in proc. of Machine Learning Research, 2016.
- 2
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, .A Talwalkar, Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization, in Machine Learning Research 18, 2018.
Limitations¶
Right now dabl does not deal with text data and time series data. It also does not consider neural network models. Image, audio and video data is considered out of scope. All current implementation are quite rudimentary and rely heavily on heuristics. The goal is to replace these with more principled approaches where this provides a benefit.
Future Goals and Roadmap¶
dabl aims to provide easy-to-use, turn-key solutions for supervised machine learning that strongly encourage iterative and interactive model building. Key ingredients to achieve this are:
Ready-made visualizations
Model diagnostics
Efficient model search
Type detection
Automatic preprocessing
Portfolios of well-performing pipelines
The current version of dabl only provides very simple implementations of these, but the goal is for dabl to contain more advanced solutions while providing a simple user interface and strong anytime performance.