Logistic regression (LR) is often the go-to choice for binary classification. Owing to extreme simplicity, LR models are fast to train and easy to deploy, and readily lend themselves for human interpretation.
The predictive performance of LR models depends on the quality and sophistication of feature engineering. There are two major work areas. First, delineating and generating the intended feature space. LR algorithms operate on the feature space they are given. They are not designed to independently discover non-linearities along individual dimensions, or interactions between multiple dimensions. Second, filtering down the feature space. Specialized LR algorithms can prioritize and eliminate dimensions using regularization. However, the most common ones estimate coefficients for all dimensions of the feature space.
Facebook Research has demonstrated how feature engineering can be automated using a gradient boosted decision tree (GBDT) model: Practical Lessons from Predicting Clicks on Ads at Facebook
The idea is to train a GBDT model on a raw feature space and collect and examine the "decision paths" of its member decision tree models. A decision path which operates on a single feature can be regarded as a non-linear transformation on it (eg. binning a continuous feature to a pseudo-categorical feature). A decision path which operates on multiple features can be regarded as an interaction between them.
GBDT algorithms typically grow shallow decision trees. Shallow trees contain short decision paths, which generally lead to easily interpretable derived features.
A boosting algorithm can be swapped for a bagging algorithm. Random forest (RF) algorithms typically grow much deeper decision trees. Longer decision paths lead to more complex derived features (eg. interactions between multiple non-linearly transformed features), which lose in interpretability but gain in information content. For example, discovering cliffs and other anomalies in the decision space by observing which derived features become associated with extreme node scores.
Scikit-Learn documentation dedicates a separate page to GBDT plus LR ensemble models: Feature transformations with ensembles of trees
While the concept and its implementation are discussed in great detail, there is no reusable GBDT+LR estimator available within Scikit-Learn. Interested parties are either expected to copy-paste the example code, or rely on third-party libraries.
sklearn2pmml package version 0.47.0 introduced
sklearn2pmml.ensemble.GBDTLMRegressor ensemble models to address this deficiency.
The Predictive Model Markup Language (PMML) provides standardized data structures for representing all common data pre- and post-processing operations and model types, including the GBDT model type and the LR model type.
If all the parts of a GBDT+LR model are PMML compatible, then it should follow that the GBDT+LR model itself is PMML compatible too? The answer is a definite yes. Better yet, the PMML representation of a GBDT+LR model is reducible to an ordinary GBDT model, which leads to significant conversion- and run-time savings.
The reduction is based on the realization that GBDT+LR is a mechanism for replacing original GBDT leaf node scores with LR coefficients (and the GBDT base score with the LR intercept).
Scikit-Learn does not provide an API for modifying fitted decision trees.
The workaround is to make individual leaf nodes addressable using the one-hot-encoding approach (the
OneHotEncoder.categories attribute is a list of arrays; the size of the list equals the number of decision trees in the GBDT; the size of each array equals the number of leaf nodes in the corresponding decision tree), and then assigning a new score to each address (the
LogisticRegression.coef_ attribute is an array whose size equals the flat-mapped size of the
The PMML representation does not need such layer of indirection, because it is possible to replace leaf node scores in place.
The JPMML-Model library provides Visitor API for traversing, updating and transforming PMML class model objects. In the current case, the Visitor API is used to transform the GBDT side of the GBDT+LR model to a regression-type boosting model. All leaf nodes are assigned new score values as extracted from the LR side.
The GBDT+LR workflow is much simpler than traditional workflows. Specifically, there is no need to perform dedicated feature engineering work, because the GBDT+LR estimator will do it automatically and in a very thorough manner.
Boilerplate for assembling and fitting a GBDT+LR pipeline using user-specified
The most common configuration is to use
GradientBoostingClassifier as the
The "boosting" behaviour can be promoted by growing a larger number of shallower decision trees.
Conversely, the "bagging" behaviour can be promoted by growing a smaller number of deeper decision trees.
GBDTLRClassifier ensemble model accepts any PMML compatible classifier as the
For example, switching from
GradientBoostingClassifier to alternative classifier classes such as
RandomForestClassifier would reduce the risk of overfitting:
The XGBoost plugin library provides an
xgboost.XGBClassifier model, which can be used as a drop-in replacement for Scikit-Learn classifier classes:
The LightGBM plugin library provides a
One of its major selling points is proper support for categorical features.
If the training dataset contains a significant number of (high-cardinality-) categorical features, then the above
make_fit_gbdtlr utility function should be tailored to maintain this information.
As discussed in a recent blog post, the fit method of LightGBM estimators takes an optional
categorical_feature fit parameter.
The problem is passing this parameter to a
LGBMClassifier object, which is contained in the
GBDTLRClassifier object, which is in turn contained in the
The solution follows Scikit-Learn conventions.
Namely, the fit method of the
GBDTLRClassifier class also takes fit parameters, which are passed on to the correct component based on the prefix.
Boilerplate for assembling and fitting an LightGBM+LR pipeline:
Both XGBoost and LightGBM classifiers support missing values.
When working with sparse datasets, then it is possible to make
make_fit_lgbmlr utility functions missing value-aware by replacing the default
LabelEncoder transformers with
sklearn2pmml.preprocessing.PMMLLabelEncoder transformers, respectively.