mojolearn.model_selection#

Cross-validation, splitters and hyperparameter search for mojolearn estimators.

This module follows scikit-learn’s sklearn.model_selection: the splitters (KFold, StratifiedKFold, GroupKFold, ShuffleSplit, …), train_test_split, cross_val_score, cross_validate, cross_val_predict, GridSearchCV, RandomizedSearchCV, validation_curve, learning_curve, permutation_test_score and the scorer helpers. scikit-learn is not required; scikit-learn pipelines, splitters and estimators that follow its protocol also work.

Behavior to know:

Fits run serially, one fold at a time. n_jobs must be 1 (or None where scikit-learn accepts it).
Splitters yield (train, test) pairs of int64 row-index arrays.
Shuffled splits draw from mojolearn’s own counter-based random generator, so a given random_state gives a reproducible split, but not the same rows scikit-learn would pick for that seed.
Fold rows are gathered on the device, and X and y stay resident there across folds and candidates. All learning happens in the estimator.

Functions

`cross_val_score`(estimator, X, y, *[, cv, ...])	Evaluate an estimator by cross-validation and return one score per fold.
`split_descriptor`(X, y, *[, estimator, cv, ...])	Describe a cross-validation split so a run can be reproduced.
`train_test_split`(*arrays[, test_size, ...])	Split arrays into random train and test subsets.
`check_cv`([cv, y, classifier, shuffle, ...])	Turn a `cv` argument into a splitter.
`cross_validate`(estimator, X[, y, groups, ...])	Evaluate one or more scores by cross-validation, with fit and score times.
`cross_val_predict`(estimator, X[, y, groups, ...])	Return each row's prediction from the fold that held it out.
`validation_curve`(estimator, X, y, *, ...[, ...])	Training and test scores for each value of one parameter.
`learning_curve`(estimator, X, y, *[, groups, ...])	Training and test scores for growing training-set sizes.
`permutation_test_score`(estimator, X, y, *[, ...])	Test the significance of a cross-validated score by permuting y.
`get_scorer`(scoring)	Return the scorer for a scoring argument.
`make_scorer`(score_func, *[, ...])	Make a scorer from a metric function.
`get_scorer_names`()	Return the sorted list of scorer names that `get_scorer` accepts.

Classes

`KFold`([n_splits, shuffle, random_state])	K-fold cross-validator.
`StratifiedKFold`([n_splits, shuffle, ...])	Stratified k-fold cross-validator.
`GroupKFold`([n_splits, shuffle, random_state])	K-fold cross-validator that keeps each group in a single fold.
`StratifiedGroupKFold`([n_splits, shuffle, ...])	Stratified k-fold cross-validator that keeps each group in a single fold.
`TimeSeriesSplit`([n_splits, max_train_size, ...])	Time-series cross-validator with expanding training windows.
`ShuffleSplit`([n_splits, test_size, ...])	Random permutation cross-validator.
`StratifiedShuffleSplit`([n_splits, ...])	Stratified random-permutation cross-validator.
`GroupShuffleSplit`([n_splits, test_size, ...])	Shuffle-split cross-validator over groups.
`LeaveOneOut`()	Leave-one-out cross-validator.
`LeavePOut`(p)	Leave-P-out cross-validator.
`LeaveOneGroupOut`()	Leave-one-group-out cross-validator.
`LeavePGroupsOut`(n_groups)	Leave-P-groups-out cross-validator.
`RepeatedKFold`(*[, n_splits, n_repeats, ...])	Repeated k-fold cross-validator.
`RepeatedStratifiedKFold`(*[, n_splits, ...])	Repeated stratified k-fold cross-validator.
`PredefinedSplit`(test_fold)	Cross-validator with user-defined folds.
`ParameterGrid`(param_grid)	Grid of parameter combinations, iterated as dicts.
`ParameterSampler`(param_distributions, n_iter, *)	Random sample of parameter combinations, iterated as dicts.
`GridSearchCV`(estimator, param_grid, *[, ...])	Exhaustive search over a parameter grid by cross-validation.
`RandomizedSearchCV`(estimator, ...[, n_iter, ...])	Randomized search over parameter lists by cross-validation.

mojolearn.model_selection.cross_val_score(estimator, X, y, *, cv=None, scoring=None, groups=None, n_jobs=1, error_score='raise')[source]#

Evaluate an estimator by cross-validation and return one score per fold.

For each fold, a fresh clone of estimator is fit on the training rows and scored on the held-out rows. Folds run one after another. Every fold is checked before the first fit: each side must be nonempty, have no repeated rows, and not overlap the other side. A row may be held out in more than one fold.

The default folds are unshuffled, so the scores depend on the order of the rows in X and y. Record split_descriptor(X, y, estimator=estimator, cv=cv) beside the scores if the run must be reproducible.

Parameters:

estimator (estimator) – An unfitted estimator with get_params, fit and (when scoring is None) score. A scikit-learn pipeline works too; put unfitted transforms inside it so they are fit on each training fold.
X (array-like of shape (n_samples, n_features)) – Dense 2-D training data. Its dtype is kept; the estimator checks the dtypes it supports. Sparse input is not supported.
y (array-like of shape (n_samples,)) – 1-D targets matching X’s rows.
cv (int, splitter or iterable of (train, test), default None) – None means five folds. An integer gives that many unshuffled folds, stratified by class for a classifier with discrete labels, plain contiguous folds otherwise. A splitter (any object with split(X, y, groups)) or an iterable of integer index pairs is used as given.
scoring (str, callable or None, default None) – None calls the fitted estimator’s score. A string is one of get_scorer_names(). A callable is called as scoring(estimator, X_test, y_test) and must return a real scalar. Loss scorers are negated ('neg_mean_squared_error'), so higher is always better.
groups (array-like of shape (n_samples,), default None) – Group labels for a splitter that reads them. Passing groups with an integer or default cv raises ValueError, because the default folds would ignore them.
n_jobs (int, default 1) – Only 1 is supported; other values raise NotImplementedError.
error_score (‘raise’, default ‘raise’) – Only 'raise' is supported: an error in a fit or score propagates.

Returns:

scores (Array of shape (n_splits,), float64) – The score of each fold, in fold order, with no aggregation.

Notes

Fit parameters, sample weights, eval_set, precomputed kernels and refitting are not supported. Set numeric_mode explicitly on every pipeline step and custom metric when the results must be reproducible across devices.

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import cross_val_score
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> scores = cross_val_score(mojolearn.Ridge(alpha=1.0), X, y, cv=5)

mojolearn.model_selection.split_descriptor(X, y, *, estimator=None, cv=None, groups=None)[source]#

Describe a cross-validation split so a run can be reproduced.

The scores from cross_val_score depend on the order of the rows in X and y: the default folds are unshuffled, so the fold a row lands in is decided by its position. mojolearn never reorders your data, so it cannot tell two orderings of the same rows apart. Record this descriptor beside the scores; it hashes the data in arrival order and the fold assignment, so a reader can check that a rerun used the same split.

The descriptor covers the split only. It does not hash the scores, the estimator or the numeric mode.

Parameters:

X (array-like of shape (n_samples, n_features)) – Dense, nonempty 2-D training data, in the order passed to cross_val_score.
y (array-like of shape (n_samples,)) – Targets, in the same order. A buffer array or a list of labels.
estimator (estimator, default None) – The estimator given to cross_val_score. Required when cv is None or an integer, because the default folds are stratified for a classifier and plain KFold otherwise; omitting it then raises ValueError.
cv (int, splitter or iterable of (train, test), default None) – The cv given to cross_val_score (None means five folds).
groups (array-like of shape (n_samples,), default None) – Group labels, when the splitter reads them.

Returns:

descriptor (dict) – A JSON-serializable dict with keys schema, n_rows, n_features, X_sha256, y_sha256, groups_sha256, fold_assignment_sha256, n_folds, fold_sizes (one [n_train, n_test] pair per fold) and sha256, a hash over the canonical encoding of every other field. X_sha256 and y_sha256 change when the rows are reordered, even when the fold indices do not.

Raises:

ValueError – If estimator is missing for an integer or default cv, if X is not a nonempty 2-D array, if y does not match X’s rows, or if cv produces no folds.

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import cross_val_score, split_descriptor
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> model = mojolearn.Ridge(alpha=1.0)
>>> scores = cross_val_score(model, X, y)
>>> record = split_descriptor(X, y, estimator=model)

class mojolearn.model_selection.KFold(n_splits=5, *, shuffle=False, random_state=None)[source]#

Bases: _KFoldBase

K-fold cross-validator.

Splits the rows into n_splits contiguous folds of the (optionally shuffled) row order; each fold is the test set once. The first n_samples % n_splits folds are one row larger. Follows scikit-learn’s KFold; shuffling uses mojolearn’s counter-based random generator, so the shuffled folds differ from scikit-learn’s for the same seed.

Parameters:

n_splits (int, default 5) – Number of folds, at least 2.
shuffle (bool, default False) – Shuffle the rows before splitting into folds.
random_state (int or None, default None) – Seed for the shuffle. Setting it with shuffle=False raises ValueError, as in scikit-learn.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import KFold
>>> X = np.random.default_rng(0).random((1000, 8), dtype=np.float32)
>>> for train, test in KFold(n_splits=5, shuffle=True, random_state=0).split(X):
...     pass

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each fold.

Raises ValueError when n_splits exceeds the number of rows.

class mojolearn.model_selection.StratifiedKFold(n_splits=5, *, shuffle=False, random_state=None)[source]#

Bases: _KFoldBase

Stratified k-fold cross-validator.

Each fold keeps roughly the class proportions of y. Follows scikit-learn’s StratifiedKFold: classes are taken in order of first appearance and allocated round-robin over the class-sorted labels. With shuffle=True each class’s fold assignment is permuted by mojolearn’s counter-based random generator, so shuffled folds differ from scikit-learn’s for the same seed.

Parameters:

n_splits (int, default 5) – Number of folds, at least 2.
shuffle (bool, default False) – Shuffle each class’s rows before assigning folds.
random_state (int or None, default None) – Seed for the shuffle. Setting it with shuffle=False raises ValueError.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import StratifiedKFold
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.integers(0, 3, 1000).astype(np.int32)
>>> folds = list(StratifiedKFold(n_splits=5).split(X, y))

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each fold.

y holds the class labels. Raises ValueError when every class has fewer than n_splits members, and warns when the smallest class does.

class mojolearn.model_selection.GroupKFold(n_splits=5, *, shuffle=False, random_state=None)[source]#

Bases: _KFoldBase

K-fold cross-validator that keeps each group in a single fold.

Follows scikit-learn’s GroupKFold. Unshuffled, groups are assigned largest first, each to the fold with the fewest rows so far (the lowest fold index on a tie). Shuffled, the groups are permuted and split into n_splits runs of nearly equal group count.

Parameters:

n_splits (int, default 5) – Number of folds, at least 2 and at most the number of groups.
shuffle (bool, default False) – Shuffle the groups before splitting.
random_state (int or None, default None) – Seed for the shuffle (mojolearn’s counter-based generator). Setting it with shuffle=False raises ValueError.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import GroupKFold
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> groups = rng.integers(0, 20, 1000)
>>> folds = list(GroupKFold(n_splits=5).split(X, groups=groups))

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each fold.

groups is required. Raises ValueError when n_splits exceeds the number of distinct groups.

class mojolearn.model_selection.StratifiedGroupKFold(n_splits=5, *, shuffle=False, random_state=None)[source]#

Bases: _KFoldBase

Stratified k-fold cross-validator that keeps each group in a single fold.

Follows scikit-learn’s StratifiedGroupKFold: groups are taken in descending order of the standard deviation of their class distribution (or in a shuffled order), and each goes to the fold whose class distribution it changes least.

Parameters:

n_splits (int, default 5) – Number of folds, at least 2.
shuffle (bool, default False) – Shuffle the group order instead of sorting it.
random_state (int or None, default None) – Seed for the shuffle (mojolearn’s counter-based generator). Setting it with shuffle=False raises ValueError.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import StratifiedGroupKFold
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.integers(0, 2, 1000)
>>> groups = rng.integers(0, 50, 1000)
>>> folds = list(StratifiedGroupKFold(n_splits=5).split(X, y, groups))

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each fold.

y and groups are required. Raises ValueError when n_splits is greater than the number of members of a class.

class mojolearn.model_selection.TimeSeriesSplit(n_splits=5, *, max_train_size=None, test_size=None, gap=0)[source]#

Bases: _Splitter

Time-series cross-validator with expanding training windows.

Follows scikit-learn’s TimeSeriesSplit. Split i tests on a block of consecutive rows and trains on the rows before it, so a model never trains on rows that come after its test block. There is no randomness.

Parameters:

n_splits (int, default 5) – Number of splits, at least 2.
max_train_size (int or None, default None) – Largest training window; older rows are dropped beyond it.
test_size (int or None, default None) – Rows per test block. None means n_samples // (n_splits + 1).
gap (int, default 0) – Rows left out between the end of each training window and its test block.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import TimeSeriesSplit
>>> X = np.random.default_rng(0).random((1000, 8), dtype=np.float32)
>>> folds = list(TimeSeriesSplit(n_splits=5, gap=10).split(X))

get_n_splits(X=None, y=None, groups=None)[source]#: Return the number of splits, n_splits.

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each split.

Raises ValueError when there are too few rows for n_splits, test_size and gap.

class mojolearn.model_selection.ShuffleSplit(n_splits=10, *, test_size=None, train_size=None, random_state=None)[source]#

Bases: _Splitter

Random permutation cross-validator.

Each split draws a fresh random permutation of the rows; its first n_test rows are the test set and the next n_train rows the training set. Splits may overlap. Follows scikit-learn’s ShuffleSplit; the permutations come from mojolearn’s counter-based random generator, so the rows differ from scikit-learn’s for the same seed.

Parameters:

n_splits (int, default 10) – Number of splits.
test_size (float, int or None, default None) – A float in (0, 1) is a fraction of the rows (rounded up), an int an absolute count. None means the complement of train_size, or 0.1 when both are None.
train_size (float, int or None, default None) – A float in (0, 1) is a fraction of the rows (rounded down), an int an absolute count. None means the complement of test_size.
random_state (int or None, default None) – Seed for the permutations.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import ShuffleSplit
>>> X = np.random.default_rng(0).random((1000, 8), dtype=np.float32)
>>> folds = list(ShuffleSplit(n_splits=5, test_size=0.2, random_state=0).split(X))

get_n_splits(X=None, y=None, groups=None)[source]#: Return the number of splits, n_splits.

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each split.

Raises ValueError for sizes out of range or summing to more than the number of rows.

class mojolearn.model_selection.StratifiedShuffleSplit(n_splits=10, *, test_size=None, train_size=None, random_state=None)[source]#

Bases: ShuffleSplit

Stratified random-permutation cross-validator.

Each split keeps roughly the class proportions of y in both the training and the test set. Follows scikit-learn’s StratifiedShuffleSplit: each class’s rows are permuted and the per-class train and test counts are scikit-learn’s approximate mode of the class proportions; the training and test rows are then shuffled. The permutations come from mojolearn’s counter-based random generator.

Parameters:

n_splits (int, default 10) – Number of splits.
test_size (float, int or None, default None) – Fraction or number of rows in the test set. None means the complement of train_size, or 0.1 when both are None.
train_size (float, int or None, default None) – Fraction or number of rows in the training set. None means the complement of test_size.
random_state (int or None, default None) – Seed for the permutations.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import StratifiedShuffleSplit
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.integers(0, 3, 1000)
>>> cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=0)
>>> folds = list(cv.split(X, y))

split(X, y, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each split.

y is required. Raises ValueError when a class has a single member, or when the training or test set is smaller than the number of classes.

class mojolearn.model_selection.GroupShuffleSplit(n_splits=10, *, test_size=None, train_size=None, random_state=None)[source]#

Bases: ShuffleSplit

Shuffle-split cross-validator over groups.

Works like ShuffleSplit on the sorted distinct groups, and every row follows its group, so a group is never split between train and test. test_size and train_size count groups, not rows. Follows scikit-learn’s GroupShuffleSplit.

Parameters:

n_splits (int, default 10) – Number of splits.
test_size (float, int or None, default None) – Fraction or number of groups in the test set. None means the complement of train_size, or 0.2 when both are None.
train_size (float, int or None, default None) – Fraction or number of groups in the training set. None means the complement of test_size.
random_state (int or None, default None) – Seed for the permutations (mojolearn’s counter-based generator).

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import GroupShuffleSplit
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> groups = rng.integers(0, 50, 1000)
>>> cv = GroupShuffleSplit(n_splits=5, test_size=0.2, random_state=0)
>>> folds = list(cv.split(X, groups=groups))

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each split.

groups is required.

class mojolearn.model_selection.LeaveOneOut[source]#

Bases: _Splitter

Leave-one-out cross-validator.

Each row is the test set once, with every other row as the training set. Follows scikit-learn’s LeaveOneOut. It takes no parameters.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import LeaveOneOut
>>> X = np.random.default_rng(0).random((20, 4), dtype=np.float32)
>>> folds = list(LeaveOneOut().split(X))

get_n_splits(X=None, y=None, groups=None)[source]#: Return the number of rows of X (X is required).

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays, one per row.

Raises ValueError when X has fewer than two rows.

class mojolearn.model_selection.LeavePOut(p)[source]#

Bases: _Splitter

Leave-P-out cross-validator.

Every combination of p rows is the test set once, in lexicographic order, with the remaining rows as the training set. Follows scikit-learn’s LeavePOut. The number of splits grows as n_samples choose p.

Parameters:: p (int) – Size of each test set; must be less than the number of rows.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import LeavePOut
>>> X = np.random.default_rng(0).random((10, 4), dtype=np.float32)
>>> folds = list(LeavePOut(2).split(X))

get_n_splits(X=None, y=None, groups=None)[source]#: Return n_samples choose p.

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each combination.

Raises ValueError when p is not less than the number of rows.

class mojolearn.model_selection.LeaveOneGroupOut[source]#

Bases: _Splitter

Leave-one-group-out cross-validator.

Each distinct group is the test set once, in sorted group order, with the rows of every other group as the training set. Follows scikit-learn’s LeaveOneGroupOut. It takes no parameters.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import LeaveOneGroupOut
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> groups = rng.integers(0, 5, 1000)
>>> folds = list(LeaveOneGroupOut().split(X, groups=groups))

get_n_splits(X=None, y=None, groups=None)[source]#: Return the number of distinct values in groups.

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays, one per group.

Raises ValueError when groups has fewer than two distinct values.

class mojolearn.model_selection.LeavePGroupsOut(n_groups)[source]#

Bases: _Splitter

Leave-P-groups-out cross-validator.

Every combination of n_groups distinct groups is the test set once, with the rows of the other groups as the training set. Follows scikit-learn’s LeavePGroupsOut.

Parameters:: n_groups (int) – Number of groups in each test set; must be less than the number of distinct groups.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import LeavePGroupsOut
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> groups = rng.integers(0, 5, 1000)
>>> folds = list(LeavePGroupsOut(n_groups=2).split(X, groups=groups))

get_n_splits(X=None, y=None, groups=None)[source]#: Return n_distinct_groups choose n_groups.

split(X, y=None, groups=None)[source]#

Yield (train, test) int64 row-index arrays for each combination.

Raises ValueError when n_groups is not less than the number of distinct groups.

class mojolearn.model_selection.RepeatedKFold(*, n_splits=5, n_repeats=10, random_state=None)[source]#

Bases: _Repeated

Repeated k-fold cross-validator.

Runs a shuffled KFold n_repeats times, each repeat with a different shuffle. Follows scikit-learn’s RepeatedKFold; the shuffles come from mojolearn’s counter-based random generator.

Parameters:

n_splits (int, default 5) – Number of folds per repeat, at least 2.
n_repeats (int, default 10) – Number of repeats, at least 1.
random_state (int or None, default None) – Seed that fixes every repeat’s shuffle.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import RepeatedKFold
>>> X = np.random.default_rng(0).random((1000, 8), dtype=np.float32)
>>> folds = list(RepeatedKFold(n_splits=5, n_repeats=3, random_state=0).split(X))

class mojolearn.model_selection.RepeatedStratifiedKFold(*, n_splits=5, n_repeats=10, random_state=None)[source]#

Bases: _Repeated

Repeated stratified k-fold cross-validator.

Runs a shuffled StratifiedKFold n_repeats times, each repeat with a different shuffle. Follows scikit-learn’s RepeatedStratifiedKFold; the shuffles come from mojolearn’s counter-based random generator.

Parameters:

n_splits (int, default 5) – Number of folds per repeat, at least 2.
n_repeats (int, default 10) – Number of repeats, at least 1.
random_state (int or None, default None) – Seed that fixes every repeat’s shuffle.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import RepeatedStratifiedKFold
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.integers(0, 2, 1000)
>>> cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)
>>> folds = list(cv.split(X, y))

class mojolearn.model_selection.PredefinedSplit(test_fold)[source]#

Bases: _Splitter

Cross-validator with user-defined folds.

test_fold[i] is the fold in which row i is tested; a value of -1 keeps that row in every training set and never tests it. Folds are yielded in sorted order of their values. Follows scikit-learn’s PredefinedSplit.

Parameters:: test_fold (array-like of int, shape (n_samples,)) – The fold of each row, or -1.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import PredefinedSplit
>>> cv = PredefinedSplit(np.array([0, 1, -1, 1, 0]))
>>> folds = list(cv.split())

get_n_splits(X=None, y=None, groups=None)[source]#: Return the number of distinct fold values other than -1.

split(X=None, y=None, groups=None)[source]#: Yield (train, test) int64 row-index arrays for each fold.

mojolearn.model_selection.train_test_split(*arrays, test_size=None, train_size=None, random_state=None, shuffle=True, stratify=None)[source]#

Split arrays into random train and test subsets.

Follows scikit-learn’s train_test_split. With shuffle=True the rows are chosen by one ShuffleSplit draw (or StratifiedShuffleSplit when stratify is given) from mojolearn’s counter-based random generator, so the rows differ from scikit-learn’s for the same seed. With shuffle=False the leading rows are the training set and the rows after them the test set.

Parameters:

*arrays (sequence of array-likes) – Arrays or lists with the same number of rows. Buffer arrays are returned as mojolearn Array objects, lists as lists.
test_size (float, int or None, default None) – Fraction (rounded up) or number of rows in the test set. None means the complement of train_size, or 0.25 when both are None.
train_size (float, int or None, default None) – Fraction (rounded down) or number of rows in the training set. None means the complement of test_size.
random_state (int or None, default None) – Seed for the shuffle.
shuffle (bool, default True) – Whether to shuffle before splitting.
stratify (array-like or None, default None) – Class labels to stratify by. Requires shuffle=True; with shuffle=False it raises ValueError.

Returns:

splitting (list) – Two pieces per input, in scikit-learn’s order: X_train, X_test, y_train, y_test, ....

Raises:

ValueError – If no array is given, the arrays have different numbers of rows, or the sizes are out of range.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import train_test_split
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> X_train, X_test, y_train, y_test = train_test_split(
...     X, y, test_size=0.2, random_state=0)

mojolearn.model_selection.check_cv(cv=5, y=None, *, classifier=False, shuffle=False, random_state=None)[source]#

Turn a cv argument into a splitter.

Follows scikit-learn’s check_cv. An integer (or None, meaning 5) becomes StratifiedKFold when classifier is true and y holds discrete labels (strings, or integer-valued finite numbers), and KFold otherwise. An object with a split method is returned as is, and an iterable of (train, test) index pairs is wrapped in a splitter that yields those pairs.

Parameters:

cv (int, splitter, iterable or None, default 5) – The cross-validation strategy.
y (array-like of shape (n_samples,), default None) – Targets, read only to decide whether to stratify.
classifier (bool, default False) – Whether the estimator is a classifier.
shuffle (bool, default False) – Passed to the KFold or StratifiedKFold built from an integer.
random_state (int or None, default None) – Seed for that splitter, used only when shuffle is true.

Returns:

splitter – An object with split(X, y, groups) and get_n_splits.

Raises:

ValueError – If cv is a string or another value that is not an integer, splitter or iterable.

Examples

>>> import numpy as np
>>> from mojolearn.model_selection import check_cv
>>> y = np.random.default_rng(0).integers(0, 2, 1000)
>>> cv = check_cv(5, y, classifier=True)

mojolearn.model_selection.cross_validate(estimator, X, y=None, *, groups=None, scoring=None, cv=None, n_jobs=None, return_train_score=False, return_estimator=False, return_indices=False, error_score=nan)[source]#

Evaluate one or more scores by cross-validation, with fit and score times.

Follows scikit-learn’s cross_validate. For each fold, a fresh clone of estimator is fit on the training rows and scored on the held-out rows. Folds run one after another.

Parameters:

estimator (estimator) – An unfitted estimator with get_params and fit.
X (array-like of shape (n_samples, n_features)) – Training data.
y (array-like of shape (n_samples,), default None) – Targets; None for unsupervised estimators.
groups (array-like of shape (n_samples,), default None) – Group labels for a splitter that reads them.
scoring (str, callable, list, tuple, set, dict or None, default None) – None uses the estimator’s score. A name or callable gives one score, stored as test_score. A list, tuple or set of names, or a dict of name to scorer, gives several, stored as test_<name>.
cv (int, splitter, iterable or None, default None) – As in check_cv: None means five folds, stratified for a classifier.
n_jobs (int or None, default None) – Only None or 1 is supported; other values raise NotImplementedError.
return_train_score (bool, default False) – Also score each fold’s training rows (train_<name>).
return_estimator (bool, default False) – Also return the fitted estimator of each fold.
return_indices (bool, default False) – Also return each fold’s train and test indices.
error_score (‘raise’ or float, default nan) – 'raise' propagates a fit error; a number is recorded as the fold’s score instead.

Returns:

results (dict) – fit_time, score_time (wall-clock seconds) and the score arrays, each a float64 Array of shape (n_splits,); plus estimator (a list) and indices ({'train': [...], 'test': [...]}) when requested.

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import cross_validate
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> res = cross_validate(mojolearn.Ridge(alpha=1.0), X, y, cv=5,
...                      scoring=['r2', 'neg_mean_squared_error'])

mojolearn.model_selection.cross_val_predict(estimator, X, y=None, *, groups=None, cv=None, n_jobs=None, method='predict')[source]#

Return each row’s prediction from the fold that held it out.

Follows scikit-learn’s cross_val_predict. For each fold, a fresh clone of estimator is fit on the training rows and its method is called on the held-out rows; the predictions are put back in row order. The folds must hold out every row exactly once, otherwise ValueError is raised.

Parameters:

estimator (estimator) – An unfitted estimator with get_params, fit and method.
X (array-like of shape (n_samples, n_features)) – Training data.
y (array-like of shape (n_samples,), default None) – Targets.
groups (array-like of shape (n_samples,), default None) – Group labels for a splitter that reads them.
cv (int, splitter, iterable or None, default None) – As in check_cv: None means five folds, stratified for a classifier.
n_jobs (int or None, default None) – Only None or 1 is supported.
method (str, default ‘predict’) – Estimator method to call, for example 'predict_proba'.

Returns:

predictions (Array or list) – Shape (n_samples,) or (n_samples, …) for numeric predictions: int64 when every fold returns 1-D integer predictions, float64 otherwise. Predictions that are not numbers (string labels) come back as a list.

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import cross_val_predict
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> pred = cross_val_predict(mojolearn.Ridge(alpha=1.0), X, y, cv=5)

class mojolearn.model_selection.ParameterGrid(param_grid)[source]#

Bases: object

Grid of parameter combinations, iterated as dicts.

Follows scikit-learn’s ParameterGrid: every combination of each dict’s values, with keys in sorted order, and the dicts in list order. Supports len() and indexing.

Parameters:: param_grid (dict or list of dicts) – Maps each parameter name to a nonempty list of values. Raises TypeError for a value that is not a list-like, and ValueError for an empty one.

Examples

>>> from mojolearn.model_selection import ParameterGrid
>>> list(ParameterGrid({'alpha': [0.1, 1.0], 'fit_intercept': [True, False]}))[0]
{'alpha': 0.1, 'fit_intercept': True}

class mojolearn.model_selection.ParameterSampler(param_distributions, n_iter, *, random_state=None)[source]#

Bases: object

Random sample of parameter combinations, iterated as dicts.

Follows scikit-learn’s ParameterSampler for lists of values: the combinations are sampled without replacement, as a random permutation of the full grid drawn by mojolearn’s counter-based random generator. When the grid has fewer than n_iter combinations, a warning is issued and every combination is returned.

Parameters:

param_distributions (dict or list of dicts) – Maps each parameter name to a list of values. scipy.stats distributions are not supported and raise NotImplementedError.
n_iter (int) – Number of combinations to sample.
random_state (int or None, default None) – Seed for the sampling.

Examples

>>> from mojolearn.model_selection import ParameterSampler
>>> params = list(ParameterSampler({'alpha': [0.01, 0.1, 1.0, 10.0]},
...                                n_iter=2, random_state=0))

class mojolearn.model_selection.GridSearchCV(estimator, param_grid, *, scoring=None, n_jobs=None, refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, return_train_score=False)[source]#

Bases: _BaseSearch

Exhaustive search over a parameter grid by cross-validation.

Follows scikit-learn’s GridSearchCV. Every combination in ParameterGrid(param_grid) is cross-validated on the same folds, one fit at a time. With refit, the best combination is then fit on all of X and y, and predict, predict_proba, decision_function, transform and score call that fitted estimator.

Parameters:

estimator (estimator) – An unfitted estimator with get_params and fit.
param_grid (dict or list of dicts) – Parameter names mapped to lists of values.
scoring (str, callable, list, tuple, set, dict or None, default None) – As in cross_validate. None uses the estimator’s score.
n_jobs (int or None, default None) – Only None or 1 is supported; other values raise NotImplementedError.
refit (bool, str or callable, default True) – Refit the best candidate on the whole data. With several metrics, the name of the metric that picks the best candidate, or a callable that takes cv_results_ and returns the best index.
cv (int, splitter, iterable or None, default None) – As in check_cv: None means five folds, stratified for a classifier.
verbose (int, default 0) – Accepted for compatibility; nothing is printed.
pre_dispatch (str, default ‘2*n_jobs’) – Accepted for compatibility; not used.
error_score (‘raise’ or float, default nan) – 'raise' propagates a fit error; a number is recorded as the fold’s score instead.
return_train_score (bool, default False) – Also record training scores (mean_train_score).

Variables:

cv_results_ (dict) – params, param_<name>, split<i>_test_score, mean_test_score, std_test_score and rank_test_score (1 is best); with several metrics each score key ends in _<metric>.
best_index_ (int) – Index of the best candidate in cv_results_.
best_score_ (float) – Mean cross-validated score of the best candidate (not set when refit is a callable).
best_params_ (dict) – Parameters of the best candidate.
best_estimator_ (estimator) – The best candidate refit on the whole data (only with refit).
scorer_ (callable or dict) – The scorer(s) used (only with refit).
n_splits_ (int) – Number of cross-validation splits.
multimetric_ (bool) – Whether several metrics were scored.

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import GridSearchCV
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> search = GridSearchCV(mojolearn.Ridge(), {'alpha': [0.1, 1.0, 10.0]}, cv=5)
>>> search = search.fit(X, y)
>>> best = search.best_params_

class mojolearn.model_selection.RandomizedSearchCV(estimator, param_distributions, *, n_iter=10, scoring=None, n_jobs=None, refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', random_state=None, error_score=nan, return_train_score=False)[source]#

Bases: _BaseSearch

Randomized search over parameter lists by cross-validation.

Follows scikit-learn’s RandomizedSearchCV for lists of values: n_iter combinations are sampled without replacement by ParameterSampler and each is cross-validated on the same folds, one fit at a time. scipy.stats distributions are not supported. Fitting, refitting, the result attributes and the prediction methods are as in GridSearchCV.

Parameters:

estimator (estimator) – An unfitted estimator with get_params and fit.
param_distributions (dict or list of dicts) – Parameter names mapped to lists of values.
n_iter (int, default 10) – Number of combinations to try.
scoring (str, callable, list, tuple, set, dict or None, default None) – As in cross_validate.
n_jobs (int or None, default None) – Only None or 1 is supported.
refit (bool, str or callable, default True) – As in GridSearchCV.
cv (int, splitter, iterable or None, default None) – As in check_cv.
verbose (int, default 0) – Accepted for compatibility; nothing is printed.
pre_dispatch (str, default ‘2*n_jobs’) – Accepted for compatibility; not used.
random_state (int or None, default None) – Seed for the sampling (mojolearn’s counter-based generator).
error_score (‘raise’ or float, default nan) – As in GridSearchCV.
return_train_score (bool, default False) – Also record training scores.

Variables:

multimetric_ (cv_results_, best_index_, best_score_, best_params_, best_estimator_, scorer_, n_splits_,) – As in GridSearchCV.

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import RandomizedSearchCV
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> search = RandomizedSearchCV(mojolearn.Ridge(), {'alpha': [0.01, 0.1, 1.0, 10.0]},
...                             n_iter=2, random_state=0).fit(X, y)

mojolearn.model_selection.validation_curve(estimator, X, y, *, param_name, param_range, groups=None, cv=None, scoring=None, n_jobs=None, error_score=nan)[source]#

Training and test scores for each value of one parameter.

Follows scikit-learn’s validation_curve. For each value in param_range, the estimator with that parameter set is cross-validated on the same folds, one fit at a time.

Parameters:

estimator (estimator) – An unfitted estimator with get_params, set_params and fit.
X (array-like of shape (n_samples, n_features)) – Training data.
y (array-like of shape (n_samples,)) – Targets.
param_name (str) – Name of the parameter to vary.
param_range (sequence) – Values of the parameter.
groups (array-like of shape (n_samples,), default None) – Group labels for a splitter that reads them.
cv (int, splitter, iterable or None, default None) – As in check_cv.
scoring (str, callable or None, default None) – As in cross_val_score.
n_jobs (int or None, default None) – Only None or 1 is supported.
error_score (‘raise’ or float, default nan) – 'raise' propagates a fit error; a number is recorded instead.

Returns:

train_scores (Array of shape (len(param_range), n_splits), float64)
test_scores (Array of shape (len(param_range), n_splits), float64)

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import validation_curve
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> train, test = validation_curve(mojolearn.Ridge(), X, y, param_name='alpha',
...                                param_range=[0.1, 1.0, 10.0], cv=5)

mojolearn.model_selection.learning_curve(estimator, X, y, *, groups=None, train_sizes=(0.1, 0.325, 0.55, 0.775, 1.0), cv=None, scoring=None, n_jobs=None, shuffle=False, random_state=None, error_score=nan, return_times=False)[source]#

Training and test scores for growing training-set sizes.

Follows scikit-learn’s learning_curve, without exploit_incremental_learning. For each size and fold, a fresh clone is fit on the first size training rows of that fold and scored on those rows and on the fold’s test rows. Fits run one at a time.

Parameters:

estimator (estimator) – An unfitted estimator with get_params and fit.
X (array-like of shape (n_samples, n_features)) – Training data.
y (array-like of shape (n_samples,)) – Targets.
groups (array-like of shape (n_samples,), default None) – Group labels for a splitter that reads them.
train_sizes (sequence of float or int, default (0.1, 0.325, 0.55, 0.775, 1.0)) – Floats are fractions of the smallest training fold (rounded down), ints are row counts. Each must resolve to between 1 and the smallest training fold’s size, or ValueError is raised. Duplicates are dropped and the sizes sorted.
cv (int, splitter, iterable or None, default None) – As in check_cv.
scoring (str, callable or None, default None) – As in cross_val_score.
n_jobs (int or None, default None) – Accepted for compatibility; fits always run serially.
shuffle (bool, default False) – Shuffle each fold’s training rows once before taking the prefixes.
random_state (int or None, default None) – Seed for the shuffle (mojolearn’s counter-based generator).
error_score (‘raise’ or float, default nan) – Accepted for compatibility; a fit error always propagates.
return_times (bool, default False) – Also return fit and score times.

Returns:

train_sizes_abs (Array of shape (n_sizes,), int64) – The training-set sizes used.
train_scores (Array of shape (n_sizes, n_splits), float64)
test_scores (Array of shape (n_sizes, n_splits), float64)
fit_times (Array of shape (n_sizes, n_splits), float64) – Wall-clock seconds; only with return_times=True.
score_times (Array of shape (n_sizes, n_splits), float64) – Wall-clock seconds; only with return_times=True.

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import learning_curve
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> sizes, train, test = learning_curve(mojolearn.Ridge(), X, y, cv=5)

mojolearn.model_selection.permutation_test_score(estimator, X, y, *, groups=None, cv=None, n_permutations=100, n_jobs=None, random_state=0, verbose=0, scoring=None, fit_params=None, params=None)[source]#

Test the significance of a cross-validated score by permuting y.

Follows scikit-learn’s permutation_test_score. The mean cross-validated score on the real targets is compared with the mean scores after n_permutations random permutations of y (within each group when groups is given). The permutations come from mojolearn’s counter-based random generator.

Parameters:

estimator (estimator) – An unfitted estimator with get_params and fit.
X (array-like of shape (n_samples, n_features)) – Training data.
y (array-like of shape (n_samples,)) – Targets.
groups (array-like of shape (n_samples,), default None) – Permute y only within each group; also passed to the splitter.
cv (int, splitter, iterable or None, default None) – As in check_cv.
n_permutations (int, default 100) – Number of permutations.
n_jobs (int or None, default None) – Accepted for compatibility; fits run serially.
random_state (int or None, default 0) – Seed for the permutations.
verbose (int, default 0) – Accepted for compatibility; nothing is printed.
scoring (str, callable or None, default None) – As in cross_val_score.
fit_params, params (dict or None, default None) – Accepted for compatibility; not passed to fit.

Returns:

score (float) – Mean cross-validated score on the real targets.
permutation_scores (Array of shape (n_permutations,), float64) – Mean score for each permutation.
pvalue (float) – (C + 1) / (n_permutations + 1), where C is the number of permutation scores greater than or equal to score.

Examples

>>> import numpy as np, mojolearn
>>> from mojolearn.model_selection import permutation_test_score
>>> rng = np.random.default_rng(0)
>>> X = rng.random((1000, 8), dtype=np.float32)
>>> y = rng.random(1000, dtype=np.float32)
>>> score, perm_scores, pvalue = permutation_test_score(
...     mojolearn.Ridge(), X, y, cv=5, n_permutations=20)

mojolearn.model_selection.get_scorer(scoring)[source]#

Return the scorer for a scoring argument.

Follows scikit-learn’s get_scorer.

Parameters:: scoring (str, callable or None) – A name from get_scorer_names(), a callable (returned as is), or None (returned as is, meaning the estimator’s own score).
Returns:: scorer (callable or None)
Raises:: ValueError – If scoring is an unknown name or not a str, callable or None.

Examples

>>> from mojolearn.model_selection import get_scorer
>>> scorer = get_scorer('r2')

mojolearn.model_selection.make_scorer(score_func, *, response_method='predict', greater_is_better=True, **kwargs)[source]#

Make a scorer from a metric function.

Follows scikit-learn’s make_scorer. The scorer is called as scorer(estimator, X, y): it gets predictions from the estimator’s response method and returns score_func(y, predictions, **kwargs) as a float, negated when greater_is_better=False.

Parameters:

score_func (callable) – Metric with signature score_func(y_true, y_pred, **kwargs).
response_method (str or tuple of str, default ‘predict’) – Estimator method that produces the predictions. A tuple is tried in order and the first method the estimator has is used.
greater_is_better (bool, default True) – False for a loss; the scorer then returns the negated value.
**kwargs – Extra keyword arguments passed to score_func.

Returns:

scorer (callable)

Examples

>>> from mojolearn import metrics
>>> from mojolearn.model_selection import make_scorer
>>> scorer = make_scorer(metrics.mean_absolute_error, greater_is_better=False)

mojolearn.model_selection.get_scorer_names()[source]#

Return the sorted list of scorer names that get_scorer accepts.

The names are scikit-learn’s, computed with mojolearn.metrics. The multilabel 'samples' averages (for example 'f1_samples') are not available.

Examples

>>> from mojolearn.model_selection import get_scorer_names
>>> 'neg_mean_squared_error' in get_scorer_names()
True