Optimizers

kxy.api.max_ent_optimizers.copula_entropy_analysis(corr, space='dual')

Analyzes the entropy of the copula of a \(d\)-dimensional continuous random vector \(x=\left(x_1, \dots, x_d \right)\), with copula-uniform representation \(u=\left(u_1, \dots, u_d \right)\).

Recall that, for any permutation \((1), \dots, (d)\) of \(1, \dots, d\), by the tower law,

\[h\left(u_1, \dots, u_d\right) = \sum_{i=2}^d h\left( u_{(i)} \vert u_{(i-1)}, \dots, u_{(1)} \right).\]

This function estimates the copula entropy \(h(u)\) by learning the following permutation:

  • \(x_{(1)}\) and \(x_{(2)}\) are chosen to be the two random variables with smallest copula entropy (or equivalently, the highest mutual information).
  • \(x_{(i)}\) for \(i>1\) is the input with the smallest conditional copula entropy \(h\left(* \vert u_{(i-1)}, \dots, u_{(1)} \right)\) (or equivalently, the highest mutual information \(I\left(*; x_{(1)}, \dots, x_{(i-1)}\right)\)). Note that by the time \((i)\) is selected, \(h\left(u_{(i-1)}, \dots, u_{(1)}\right)\) is already known, so that the maximum entropy conditional entropy is simply derived from the maximum entropy copula distribution of \(\left(x_{(i)}, \dots, x_{(1)}\right)\).

This function returns the learned permutation of inputs, the association conditional entropies, as well as the copula entropy \(h(u)\).

Parameters:
  • corr (np.array) – The Spearman correlation matrix.
  • output_index (int) – The index of the column to use as output.
  • space (str, 'primal' | 'dual') – The space in which the maximum entropy problem is solved. When space='primal', the maximum entropy problem is solved in the original observation space, under Pearson covariance constraints, leading to the Gaussian copula. When space='dual', the maximum entropy problem is solved in the copula-uniform dual space, under Spearman rank correlation constraints.
Returns:

res – Dictionary with keys copula_entropy, selection_order, and conditional_copula_entropies.

Return type:

dict

kxy.api.max_ent_optimizers.information_adjusted_correlation_from_spearman(corr, space='dual')

Determine the Pearson correlation matrix that should be used as plug-in replacement for the standard Pearson correlation estimator, in situations where Gaussianity has to be assumed for simplicity, but the data-generating distribution exhibits non-Gaussian traits (e.g. heavier or lighter tails).

We proceed as follows, for every pairwise Spearman rank correlation, we estmate the smallest mutual information that is consistent with the pairwise Spearman rank correlation. This is also the mutual information of the model which, among all models that have the same Spearman rank correlation, is the most uncertain about everything else.

This mutual information captures the essence of the dependence between the two variables without assuming Gaussianity. To go back to the Gaussian scale, we determine the Pearson correlation which, under the Gaussian assumption would yield the same mutual information as estimated.

Parameters:
  • corr (np.array) – The Spearman correlation matrix.
  • space (str, 'primal' | 'dual') – The space in which the maximum entropy problem is solved. When space='primal', the maximum entropy problem is solved in the original observation space, under Pearson covariance constraints, leading to the Gaussian copula. When space='dual', the maximum entropy problem is solved in the copula-uniform dual space, under Spearman rank correlation constraints.
Returns:

res – The array of equivalent Pearson correlation coefficients.

Return type:

np.array

kxy.api.max_ent_optimizers.mutual_information_analysis(corr, output_indices, space='dual', batch_indices=[])

Analyzes the dependency between \(d\)-dimensional continuous random vector \(x=\left(x_1, \dots, x_d \right)\) and one or more continuous random scalar \(y\).

Recall that, for any permutation \(\pi_1, \dots, \pi_d\) of \(1, \dots, d\), by the tower law,

\[I\left(y; x_1, \dots, x_d\right) = I\left(y; x_{\pi_1}\right) + \sum_{i=2}^d I\left(y; x_{\pi_i} \vert x_{\pi_{i-1}}, \dots, x_{\pi_1} \right).\]

This function estimates the mutual information \(I(y; x)\) by learning the following permutation.

  • \(x_{\pi_1}\) is the input with the largest maximum entropy mutual information with \(y\) under Spearman rank correlation constraints.
  • \(x_{\pi_i}\) for \(i>1\) is the input with the largest maximum entropy conditional mutual information \(I\left(y; * \vert x_{\pi_{i-1}}, \dots, x_{\pi_1}\right)\). Note that by the time \(\pi_i\) is selected, \(I\left(y; x_{\pi_{i-1}}, \dots, x_{\pi_1}\right)\) is already known, so that the maximum entropy conditional mutual information is simply derived from the maximum entropy copula distribution of \(I\left(y; x_{\pi_i}, \dots, x_{\pi_1}\right)\).

This function returns the learned permutation of inputs, the associated conditional mutual informations (a.k.a, the incremental input importance scores), as well as the mutual information \(I\left(y; x_1, \dots, x_d\right)\).

Parameters:
  • corr ((d, d) np.array) – The Spearman correlation matrix.
  • output_indices (list of int) – The index (indices) of the column to use as output.
  • space (str, 'primal' | 'dual') – The space in which the maximum entropy problem is solved. When space='primal', the maximum entropy problem is solved in the original observation space, under Pearson covariance constraints, leading to the Gaussian copula. When space='dual', the maximum entropy problem is solved in the copula-uniform dual space, under Spearman rank correlation constraints.
Returns:

res – Dictionary with keys mutual_information, selection_order, and conditional_mutual_informations.

Return type:

dict

kxy.api.max_ent_optimizers.robust_pearson_corr_from_spearman(corr)

Return the Pearson correlation matrix that is equivalent to the input Spearman rank correlation matrix, assuming inputs are jointly Gaussian.

Parameters:corr (np.array) – The Spearman correlation matrix.
Returns:res – The array of equivalent Pearson correlation coefficients.
Return type:np.array