mutual_information_feature_selector

Use this library to select features that carry information about a categorical target. It scores categorical and continuous features independently, discretizing continuous values before scoring. It reuses the typed discretization, sparse contingency arithmetic, stable ranking, selection, model validation, and the export facilities of the feature_selection_protocols library.

API documentation

Open the ../../apis/library_index.html#mutual-information-feature-selector link in a web browser.

Loading

To load this library, load its loader.lgt file:

| ?- logtalk_load(mutual_information_feature_selector(loader)).

Testing

To test this library, load its tester.lgt file:

| ?- logtalk_load(mutual_information_feature_selector(tester)).

The tests cover categorical, mixed, and continuous datasets, independent binary and multiclass numerical references, both binning methods, missing observations, stable ties, options, malformed inputs and models, reflection, diagnostics, printing, and clause and file exports.

Scores

The default score_variant(raw) option computes mutual information in bits: sum(p(value,target) * log2(p(value,target)/(p(value)*p(target)))) over occupied contingency cells. The score_variant(normalized) option computes symmetrical uncertainty, 2 * I(feature;target)/(H(feature)+H(target)). The latter is between zero and one and is not mutual information divided by target entropy alone. The recorded metrics are mutual_information_score and symmetrical_uncertainty_score, respectively.

Empty complete-case samples, constant features, and single-class samples score zero. Numeric target labels are class codes, not regression targets.

Preparation and options

The dataset object must implement feature_dataset_protocol. It declares continuous features with attribute_values(Feature, continuous) and categorical features with attribute_values(Feature, Domain), where Domain is a non-empty list of atomic values. Known continuous values must be numbers; known categorical values must belong to their declared domains; known targets must be atomic.

The learn/3 predicate accepts an options list as its last argument:

  • score_variant(Variant) selects raw (default) or normalized.

  • preparation_mode(Mode) selects per_feature (default) or joint. Joint preparation excludes rows missing the target or any candidate feature before fitting bins, so all features use the same observations.

  • discretization(Specification) selects equal_frequency(Bins), equal_width(Bins), or categorical, applied only to continuous features. The default is equal_frequency(10); bin counts must be positive integers. The categorical value preserves numeric values as category codes.

  • feature_discretization(Feature, Specification) overrides the default for one declared feature. The specification can be categorical, equal_frequency(Bins), or equal_width(Bins).

  • selection_strategy(top_k(K)), default top_k(10), selects up to the positive integer K features.

  • selection_strategy(all) selects every candidate.

  • selection_strategy(threshold(T)) selects scores at least the numeric T.

The learn/2 predicate uses the default option values.

Repeated options and repeated overrides are accepted; the first applicable occurrence wins. All supplied options are validated, including later occurrences. Unknown override feature names are rejected.

By default, preparation is per feature: observations with an unbound or absent feature value or an unbound target are excluded from that feature’s score without imputing values. Binning is fitted to its complete cases only. Categorical features retain their declared categories unless explicitly overridden. Equal-width bins partition the observed range; equal-frequency bins use observed quantile cuts without splitting equal values. Occupied category counts can be smaller than requested bin counts. Applying numeric binning to an atomic non-numeric category is an error, not an implicit category encoding.

Scores are sorted in decreasing order, preserving declaration order on ties. Top-k selection may include zero-scoring features. Thresholds should be chosen for the selected score variant.

Models and diagnostics

The learn/2 and learn/3 predicates return models using the following term representation:

mutual_information_feature_selector(FeatureScores, SelectedFeatures, Diagnostics)

The feature_scores/2 predicate returns sorted Feature-Score pairs. The selected_features/2 predicate returns selected names. The diagnostics/2, diagnostic/2, and selector_options/2 predicates expose metadata and effective options. The Diagnostics argument is a list of diagnostic terms, including model/1, example_count/1, options/1, candidate_count/1, selected_count/1, scoring_metric/1, complete_cases(FeatureCounts), discretization(FeatureSpecifications), occupied_categories(FeatureCounts), and preparation_mode(Mode). Joint mode also records usable_example_count/1 and excluded_example_count/1; every complete-case count equals the usable count. Counts and specifications follow declaration order, not score order.

The check_selector/1 predicate checks ground structure, unique sorted scores, selection consistency, metric identity determined by options, valid options, candidate and selected counts, and complete-case counts. The valid_selector/1 predicate fails for malformed models without binding partial models. These checks are structural; they do not recompute scores from a dataset. The export_to_clauses/4 and export_to_file/4 predicates preserve the complete model. The print_selector/1 predicate prints its template and contents.

Limitations

Univariate scores do not detect interaction-only signals or remove redundant features. Raw information favors high-cardinality features; normalization does not eliminate sampling bias. Per-feature missingness can make samples incomparable; joint mode aligns samples but can discard many rows and does not eliminate missing-data bias. Discretization and sample size affect the scores. Binning boundaries are not stored as a transformation for future examples. No regression scoring, p-values, bias correction, or automatic feature-count selection is provided. Arithmetic uses backend floating-point precision.

References