mojolearn.GradientBoosting

class mojolearn.GradientBoosting(loss='RMSE', n_estimators=None, max_depth=6, learning_rate=None, l2_leaf_reg=None, border_count=128, random_state=0, loss_alpha=None, loss_q=None, loss_delta=None, loss_variance_power=None, loss_border=None, leaf_estimation_method=None, leaf_estimation_iterations=None, bootstrap_type=None, bagging_temperature=1.0, subsample=None, cat_features=None, one_hot_features=None, od_type=None, od_pvalue=None, od_wait=None, use_best_model=None, best_model_min_trees=1, score_function=None, nan_mode='Min', random_strength=None, use_pointwise_searcher=False, boost_from_average=None, border_build_max_samples=200000, class_weights=None, permutation_count=None, ctr_estimation_permutation_id=None, grow_policy='SymmetricTree', max_leaves=None, min_data_in_leaf=1, min_split_gain=None, min_child_hessian=None, feature_fraction=1.0, feature_border_type='GreedyLogSum', boosting_type=None, fold_len_multiplier=2.0, fold_permutation_block=None)

Bases: NumericModeMixin

Gradient-boosted decision trees with CatBoost’s GPU learner as the reference.

Fits an ensemble of trees with CatBoost’s growth policies (oblivious SymmetricTree, Depthwise, Lossguide), losses and leaf estimators. X is (n_samples, n_features) and is converted to float32.

Under the default grow_policy='SymmetricTree' every default follows CatBoost’s GPU learner (task_type='GPU'), not scikit-learn, and several of these change results: 1000 iterations, an automatically chosen learning rate, depth 6, l2_leaf_reg=3, 128 borders, random_strength=1, Bayesian bootstrap, and Ordered boosting on smaller pools. predict returns raw scores for every loss; use predict_proba or predict_classes for classification output.

Supports the numeric_mode parameter ('identical', 'deterministic', or 'fast' on Apple GPUs).

Parameters:

• loss (str, default ‘RMSE’) – One of LOSSES, in CatBoost’s spellings: MultiClass, MultiClassOneVsAll, RMSE, Logloss, CrossEntropy, Quantile, MAE, LogLinQuantile, MAPE, Poisson, Lq, Expectile, Tweedie, Huber, QueryRMSE, PairLogit, YetiRank, MultiRMSE. Four require a parameter: Lq needs loss_q, Huber needs loss_delta, Tweedie needs loss_variance_power and Expectile needs loss_alpha; a missing one raises ValueError. MultiRMSE is multi-target regression: y is (n_samples, n_targets) with n_targets >= 2 and predict returns (n_samples, n_targets) (Plain SymmetricTree, numeric features, no eval set, boost_from_average unset or False).

• n_estimators (int or None, default None) – CatBoost’s iterations. None is 1000 under SymmetricTree (CatBoost’s default) and 100 under Depthwise and Lossguide.

• max_depth (int, default 6) – CatBoost’s depth. Under SymmetricTree the tree is oblivious, so it has exactly 2 ** max_depth leaves; under Depthwise it bounds the number of levels; under Lossguide it bounds any one leaf’s depth and max_leaves is what stops the tree.

• learning_rate (float or None, default None) – None under SymmetricTree is CatBoost’s GPU auto-selection: when l2_leaf_reg, leaf_estimation_method and leaf_estimation_iterations are also unset and the loss is RMSE, Logloss or MultiClass, the rate is computed from the row count and n_estimators with CatBoost’s GPU coefficients (capped at 0.5, rounded to six decimals); otherwise 0.03. Under Depthwise and Lossguide None is 0.03. The value used is stored in learning_rate_.

• l2_leaf_reg (float or None, default None) – None takes the loss’s CatBoost default: 3.0, and 0 for YetiRank. An explicit value is used as given and, as in CatBoost, turns the learning-rate auto-selection off.

• border_count (int, default 128) – Quantization bins per numeric feature (CatBoost’s GPU default; their CPU default is 254).

• random_state (int, default 0) – Random seed.

• loss_alpha (float, optional) – Quantile level for Quantile and LogLinQuantile (default 0.5), and required for Expectile. Ignored by MAE, whose alpha is fixed at 0.5.

• loss_q, loss_delta, loss_variance_power (float, optional) – Lq’s q, Huber’s delta, Tweedie’s variance power.

• loss_border (float, optional) – Logloss’s target threshold, default 0.5.

• leaf_estimation_method ({‘Newton’, ‘Gradient’, ‘Exact’, ‘Simple’}, optional) – None lets the loss decide, as in CatBoost. 'Newton' is refused for Quantile, MAE, LogLinQuantile, MAPE and Lq with q < 2, whose second derivative is zero.

• leaf_estimation_iterations (int, optional) – None lets the loss decide; under SymmetricTree, fewer than 200 iterations on fewer than 20 features then sets it to 1, as CatBoost does.

• bootstrap_type ({‘Bayesian’, ‘Bernoulli’, ‘Poisson’, ‘No’}, optional) – None under SymmetricTree is 'Bayesian' (CatBoost’s GPU default); under Depthwise and Lossguide it means no row sampling. 'Bernoulli' and 'Poisson' take subsample; 'Bayesian' takes bagging_temperature. Ranking losses apply row weights after computing their grouped gradients.

• bagging_temperature (float, default 1.0) – Read by bootstrap_type='Bayesian' only; a non-default value with any other bootstrap raises ValueError.

• subsample (float, optional) – Read by Bernoulli and Poisson only (default 0.66). Passing it with Bayesian (including the default) or no bootstrap raises ValueError.

• cat_features (sequence of int, optional) – Column indices holding dense category codes 0..k-1. As in CatBoost, each column is one-hot encoded when its cardinality is small and otherwise uses target statistics (CTRs). CTRs are computed over the rows in the order you pass them; CatBoost shuffles the pool first and this does not. Rows sorted by target give a worse estimator, so shuffle before fitting and record the order you used. Without cat_features row order does not matter.

• one_hot_features (sequence of int, optional) – Column indices to one-hot encode. An index out of range raises ValueError.

• od_type ({‘None’, ‘IncToDec’, ‘Iter’}, optional) – The overfitting detector. Unset is not the same as 'None': as in CatBoost, the type follows from what was given (od_wait alone means 'Iter', od_pvalue alone means 'IncToDec', neither means 'None'). Requires eval_set; otherwise it raises.

• od_pvalue (float, optional) – IncToDec’s threshold. The default 0 leaves the detector inactive.

• od_wait (int, optional) – Iterations without a new best before stopping, default 20.

• use_best_model (bool, optional) – Truncate the fitted ensemble to the best held-out iteration. Unset is not False: with an eval_set whose target is not constant it defaults to True, as in CatBoost. True without an eval_set raises.

• best_model_min_trees (int, default 1) – The truncation never keeps fewer than this many trees.

• score_function ({‘Cosine’, ‘L2’, ‘NewtonCosine’, ‘NewtonL2’}, optional) – The split score. None is CatBoost’s default for the policy: Cosine for SymmetricTree and Depthwise, NewtonL2 for Lossguide. Only Cosine scores carry random_strength noise. For RMSE each Newton spelling gives the same model as its plain twin. CatBoost’s SolarL2, LOOL2 and SatL2 are refused by name.

• nan_mode ({‘Min’, ‘Max’, ‘Forbidden’}, default ‘Min’) – Where a NaN sorts against the borders: below every border, above, or 'Forbidden' to raise on a NaN, as CatBoost does.

• random_strength (float or None, default None) – CatBoost’s score noise. None is 1.0 under SymmetricTree and 0.0 under Depthwise and Lossguide. An explicit value above 0 is refused with score_function='L2' or 'NewtonL2', which have no noise term; unset resolves to 0 there.

• use_pointwise_searcher (bool, default False) – Grow with CatBoost’s doc-parallel oblivious searcher instead of the greedy subsets searcher. It returns tree structure only, so it always runs the leaf estimator.

• boost_from_average (bool or None, default None) – Start every row at the loss’s optimal constant, stored as bias_. None is CatBoost’s default: True for RMSE, MAE, Quantile and MAPE, False otherwise. True is accepted for those four and for Logloss and CrossEntropy.

• border_build_max_samples (int, default 200000) – Rows subsampled for the border search only; every row is still fitted. 0 uses every row for the borders too.

• class_weights (sequence of float, optional) – One weight per class, multiplied into each row’s weight (so it composes with sample_weight). The length must match the label set or the fit raises. Refused with loss='RMSE', as in CatBoost.

• permutation_count (int, optional) – CatBoost’s permutation_count. None resolves in fit: 4 for an Ordered fit and for CTR categoricals, 1 for a Plain fit without CTRs. Only the categorical path and Ordered boosting read it; a Plain fit with no cat_features refuses it.

• ctr_estimation_permutation_id (int, optional) – Which permutation estimates the CTRs. None means permutation_count - 1. Only the categorical path reads it; it is refused otherwise.

• grow_policy ({‘SymmetricTree’, ‘Depthwise’, ‘Lossguide’}, default ‘SymmetricTree’) – SymmetricTree grows oblivious trees; Depthwise splits every improving leaf of a level on its own best split; Lossguide splits one leaf per step, the best-scoring one, until max_leaves. The non-symmetric policies refuse, as CatBoost does, losses without a non-symmetric GPU trainer (Lq, MultiClass) and use_pointwise_searcher=True. 'Region' raises NotImplementedError.

• max_leaves (int, optional) – The Lossguide leaf budget (CatBoost’s default 31, cap 65536). Read under Lossguide only; for other policies any value other than 2 ** max_depth is refused, as in CatBoost.

• min_data_in_leaf (int, default 1) – A leaf with at most this many rows is terminal. Used under Depthwise and Lossguide only; under SymmetricTree any value but 1 is refused.

• min_split_gain (float or None, default None) – Strict lower bound on a split’s improvement in the selected score function, Depthwise and Lossguide only. None keeps CatBoost’s growth, including Lossguide’s ability to split without positive gain; 0 requires positive improvement. Units depend on score_function, so it is not interchangeable with XGBoost’s gamma or LightGBM’s min_gain_to_split. Must be finite and nonnegative.

• min_child_hessian (float or None, default None) – Minimum weighted Hessian sum in each candidate child (training and bootstrap weights included; equality allowed). Requires Depthwise or Lossguide, NewtonL2 or NewtonCosine scores, and RMSE, Logloss or CrossEntropy.

• feature_fraction (float, default 1.0) – Per-tree numeric feature subsampling in (0, 1]: a deterministic subset of nonconstant features is chosen per tree. Values below 1 require numeric features only (categorical and one-hot columns raise NotImplementedError).

• feature_border_type (str, default ‘GreedyLogSum’) – How numeric borders are chosen, one of BORDER_TYPES: GreedyLogSum and GreedyMinEntropy (greedy splitting), MaxLogSum and MinEntropy (exact dynamic program), Median (quantiles), Uniform (equal width) and UniformAndQuantiles (half of each). Each matches CatBoost’s borders. CTR columns keep their own grids.

• boosting_type ({‘Plain’, ‘Ordered’} or None, default None) – None under SymmetricTree is CatBoost’s data-dependent GPU default: Ordered when the pool has fewer than 50,000 rows and there are at least 500 iterations, otherwise Plain; always Plain for the multiclass losses, the L2 scores, use_pointwise_searcher and feature_fraction < 1. None under Depthwise and Lossguide is Plain. The value used is stored in boosting_type_.

  Ordered follows CatBoost’s GPU ordered boosting: permutation_count permutations, growing folds with leaves estimated on each fold’s prefix, and the exported model estimated on the last permutation. It supports every pointwise loss, Cosine and NewtonCosine scores, every bootstrap, random_strength, every border type and NaN mode, sample and class weights, one-hot categorical columns, boost_from_average and eval sets. On a CPU-only install, sample weights, class weights and one-hot columns are refused under Ordered. Refused, as in CatBoost: Depthwise and Lossguide, the multiclass losses, L2 and NewtonL2 scores, the Exact leaf estimator. Not implemented, refused by name: CTR categorical columns and the ranking losses. Its random streams are this library’s, so an Ordered model matches CatBoost in behavior, not in bits.

• fold_len_multiplier (float, default 2.0) – Ordered only: the fold growth factor, greater than 1. Refused by a Plain fit.

• fold_permutation_block (int or None, default None) – Ordered only. None or 0 means 64 from 50,000 rows up (halved while block * 128 > rows) and 1 below 50,000 rows.

Variables:

• model_ (str) – The fitted ensemble as text. It is a plain string you can store or pass to another process; floats carry their exact bits, so a round trip does not change predictions.

• loss_curve_ (ndarray) – The training loss after each iteration.

• test_loss_curve_ (ndarray or None) – The held-out loss after each iteration, or None without an eval_set. This is the curve the overfitting detector reads.

• learning_rate_ (float) – The learning rate the fit used.

• boosting_type_ (str) – 'Plain' or 'Ordered', the boosting type the fit used.

• best_iteration_ (int) – Index of the lowest test_loss_curve_ entry, or of the lowest training loss without an eval set. With best_model_min_trees above it, the model holds more than best_iteration_ + 1 trees.

• stopped_early_ (bool) – Whether the detector stopped before n_estimators.

• bias_ (float) – The starting constant (0.0 unless boost_from_average).

• n_features_in_ (int) – Number of features seen in fit.

Examples

>>> import numpy as np, mojolearn
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = (X[:, 0] + X[:, 1] > 1).astype(np.float32)
>>> model = mojolearn.GradientBoosting(loss="Logloss", n_estimators=100).fit(X, y)
>>> proba = model.predict_proba(X)

__init__(loss='RMSE', n_estimators=None, max_depth=6, learning_rate=None, l2_leaf_reg=None, border_count=128, random_state=0, loss_alpha=None, loss_q=None, loss_delta=None, loss_variance_power=None, loss_border=None, leaf_estimation_method=None, leaf_estimation_iterations=None, bootstrap_type=None, bagging_temperature=1.0, subsample=None, cat_features=None, one_hot_features=None, od_type=None, od_pvalue=None, od_wait=None, use_best_model=None, best_model_min_trees=1, score_function=None, nan_mode='Min', random_strength=None, use_pointwise_searcher=False, boost_from_average=None, border_build_max_samples=200000, class_weights=None, permutation_count=None, ctr_estimation_permutation_id=None, grow_policy='SymmetricTree', max_leaves=None, min_data_in_leaf=1, min_split_gain=None, min_child_hessian=None, feature_fraction=1.0, feature_border_type='GreedyLogSum', boosting_type=None, fold_len_multiplier=2.0, fold_permutation_block=None)

Methods

fit(X, y[, sample_weight, eval_set, ...])	Fit the ensemble.
get_leaf_values()	Return the stored leaf values, tree-major then leaf-major.
get_params([deep])
get_tree_leaf_counts()	Return the number of leaves in each tree, as in CatBoost.
load(path)	Load a model written by save.
numeric_mode_used()	The numeric mode this estimator runs in.
predict(X)	Return raw scores (not probabilities) as a float32 mojolearn.Array.
predict_classes(X)	Return the predicted class codes (argmax of predict_proba).
predict_proba(X)	Return class probabilities of shape (n_samples, n_classes).
save(path)	Write the fitted model to path as an .npz file.
set_params(**params)	Set constructor parameters.
vendor_used()	The GPU backend this estimator runs on.

Attributes

learning_rate_	the learning rate the last fit used (CatBoost's auto-selection resolved against that fit's pool), None before a fit
boosting_type_	the boosting type the last fit used ('Plain' or 'Ordered')
approx_dim_	1 for every single-output loss; n_classes - 1 for MultiClass, because the last class's approx is pinned at zero and is not stored.

learning_rate_

the learning rate the last fit used (CatBoost’s auto-selection resolved against that fit’s pool), None before a fit

boosting_type_

the boosting type the last fit used (‘Plain’ or ‘Ordered’)

approx_dim_

1 for every single-output loss; n_classes - 1 for MultiClass, because the last class’s approx is pinned at zero and is not stored. Read from the fitted model, never assumed.

fit(X, y, sample_weight=None, eval_set=None, group_id=None, subgroup_id=None, pairs=None, pairs_weight=None)

Fit the ensemble.

Parameters:

• X (array-like of shape (n_samples, n_features)) – Training data, converted to float32.

• y (array-like of shape (n_samples,) or (n_samples, n_targets)) – Targets; 2-D only for loss='MultiRMSE'.

• sample_weight (array-like of shape (n_samples,), optional) – Per-row weights, None meaning all ones. Multiplied with class_weights where both are given.

• eval_set (tuple (X_eval, y_eval) or list of one such tuple, optional) – Held-out data for test_loss_curve_, the overfitting detector and use_best_model. Unlike CatBoost, only one eval set is accepted; more than one raises. Passing it makes use_best_model default to True, so model_ keeps the trees up to the best held-out iteration; pass use_best_model=False to keep every tree.

• group_id (sequence, optional) – One query id per row (string or integer; integers compare by their decimal spelling, so 7 and “7” are one group; floats are refused). Rows of a group must be consecutive. Read by QueryRMSE, PairLogit and YetiRank (SymmetricTree, no categorical features, no eval set); every other loss refuses it. A QueryRMSE fit with no groups, or one row per group, predicts zero.

• subgroup_id (sequence, optional) – Not read by any loss; refused by name.

• pairs (sequence of [winner, loser], optional) – PairLogit only: integer row-index pairs, each inside one group. Requires group_id. Without it, pairs are generated from the groups and y (every two rows of a query with different grades, the higher grade winning, weighted by the query’s first row weight).

• pairs_weight (sequence of float, optional) – One weight per pair, default 1.0.

Returns:

self – The fitted estimator.

Notes

YetiRank uses CatBoost’s defaults (10 permutations, decay 0.85, l2_leaf_reg 0 unless given, Newton leaves at one iteration, which it refuses to change), refuses a query over 1023 rows, and leaves loss_curve_ at zero.

predict(X)

Return raw scores (not probabilities) as a float32 mojolearn.Array.

For a single-output loss the result has shape (n_samples,). For MultiClass it is (n_samples, n_classes - 1): the last class’s score is fixed at zero and not returned, so use predict_proba for probabilities. MultiClassOneVsAll returns (n_samples, n_classes), one raw score per class. MultiRMSE returns (n_samples, n_targets).

Parameters:

X (array-like of shape (n_samples, n_features))

Returns:

mojolearn.Array

predict_proba(X)

Return class probabilities of shape (n_samples, n_classes).

Defined for Logloss and CrossEntropy (sigmoid), MultiClass (softmax over all classes, columns sum to 1) and MultiClassOneVsAll (independent sigmoids per class; columns do not sum to 1, because each class was fitted as a separate problem). Every other loss raises. This is the same transform as CatBoost’s prediction_type='Probability'. Columns are in class-code order. The result is a float32 mojolearn.Array for the two multiclass losses and float64 for Logloss and CrossEntropy.

Parameters:

X (array-like of shape (n_samples, n_features))

Returns:

mojolearn.Array

predict_classes(X)

Return the predicted class codes (argmax of predict_proba).

A separate method because predict returns raw scores for every loss. Ties go to the first class. Raises ValueError for a model fitted with a non-classification loss.

Parameters:

X (array-like of shape (n_samples, n_features))

Returns:

mojolearn.Array – int64 dense class codes of shape (n_samples,).

get_tree_leaf_counts()

Return the number of leaves in each tree, as in CatBoost.

Returns:

mojolearn.Array – uint32 array with one entry per tree.

get_leaf_values()

Return the stored leaf values, tree-major then leaf-major.

For multiclass models each leaf’s dimensions are consecutive: MultiClass stores n_classes - 1 values per leaf and MultiClassOneVsAll stores n_classes. Values are the stored float32 values converted exactly to float64.

Returns:

mojolearn.Array – float64 array.

save(path)

Write the fitted model to path as an .npz file.

The saved model predicts bit for bit like the original. The numeric mode the model used is saved with it. loss_curve_ and test_loss_curve_ are not restored by load.

Parameters:

path (str or path-like) – Destination file.

classmethod load(path)

Load a model written by save.

The loaded model predicts; it does not carry the training configuration or the loss curves. A file whose stored metadata disagrees with its model is refused as corrupt.

Parameters:

path (str or path-like)

Returns:

GradientBoosting