Entropy¶

kxy.api.core.entropy.
discrete_entropy
(x)¶ Estimates the (Shannon) entropy of a discrete random variable taking up to q distinct values given n i.i.d samples,
\[h(x) =  \sum_{i=1}^q p_i \log p_i,\]using the plugin estimator.
Parameters: x ((n,) np.array) – i.i.d. samples from the distribution of interest. Returns: h – The (differential) entropy of the discrete random variable of interest. Return type: float Raises: AssertionError – If the input has the wrong shape.

kxy.api.core.entropy.
least_structured_continuous_entropy
(x, space='dual', batch_indices=[])¶ Estimates the entropy of a continuous \(d\)dimensional random variable under the least structured assumption for its copula.
When \(d>1\),
\[h(x) = h(u) + \sum_{i=1}^d h(x_i).\] \(h(u)\) is estimated using least_structured_copula_entropy.
 \(h(x_i)\) are estimated using scalar_continuous_entropy.
Parameters:  x ((n, d) np.array) – n i.i.d. draws from the data generating distribution.
 space (str, 'primal'  'dual') – The space in which the maximum entropy problem is solved.
When
space='primal'
, the maximum entropy problem is solved in the original observation space, under Pearson covariance constraints, leading to the Gaussian copula. Whenspace='dual'
, the maximum entropy problem is solved in the copulauniform dual space, under Spearman rank correlation constraints.
Returns: h – The (differential) entropy of the data generating distribution, assuming its copula is maximumentropy in the chosen space. By convention, when \(d=1\), this function is the same as scalar_continuous_entropy, and returns 0 when \(n=1\).
Return type: float

kxy.api.core.entropy.
least_structured_copula_entropy
(x, space='dual', batch_indices=[])¶ Estimates the entropy of the maximumentropy copula in the chosen space.
Note
This also corresponds to least amount of total correlation that is evidenced by our maximumentropy constraints.
Parameters:  x ((n, d) np.array) – n i.i.d. draws from the data generating distribution.
 space (str, 'primal'  'dual') – The space in which the maximum entropy problem is solved.
When
space='primal'
, the maximum entropy problem is solved in the original observation space, under Pearson covariance constraints, leading to the Gaussian copula. Whenspace='dual'
, the maximum entropy problem is solved in the copulauniform dual space, under Spearman rank correlation constraints.
Returns: h – The (differential) entropy of the least structured copula consistent with maximumentropy constraints.
Return type: float

kxy.api.core.entropy.
least_structured_mixed_entropy
(x_c, x_d, space='dual', batch_indices=[])¶ Estimates the joint entropy \(h(x_c, x_d)\), where \(x_c\) is continuous random vector and \(x_d\) is a discrete random vector.
Note
We use the identities
\[ \begin{align}\begin{aligned}h(x, y) &= h(y) + h(xy) \\\ &= h(y) + E \left[ h(x \vert y=.) \right]\end{aligned}\end{align} \]that are true when \(x\) and \(y\) are either both continuous or both discrete to extend the definition of the joint entropy to the case where one is continuous and the other discrete.
Specifically,
\[h(x_c, x_d) = h(x_d) + \sum_{j=1}^q \mathbb{P}(x_d=j) h\left(x_c \vert x_d=j \right).\] \(h(x_d)\) is estimated using discrete_entropy.
 \(\mathbb{P}(x_d=j)\) is estimated using frequencies.
 \(h\left(x_c \vert x_d=j \right)\) is estimated using least_structured_continuous_entropy.
Parameters:  x_c ((n, d) np.array) – n i.i.d. draws from the continuous data generating distribution.
 x_d ((n,) np.array) – n i.i.d. draws from the discrete data generating distribution, jointly sampled with x_c.
 space (str, 'primal'  'dual') – The space in which the maximum entropy problem is solved.
When
space='primal'
, the maximum entropy problem is solved in the original observation space, under Pearson covariance constraints, leading to the Gaussian copula. Whenspace='dual'
, the maximum entropy problem is solved in the copulauniform dual space, under Spearman rank correlation constraints.
Returns: h – The entropy of the least structured distribution as evidenced by average the maximum entropy constraints.
Return type: float

kxy.api.core.entropy.
scalar_continuous_entropy
(x, space='dual', method='gaussiankde')¶ Estimates the (differential) entropy of a continuous scalar random variable.
Multiple methods are supported:
'gaussian'
for Gaussian moment matching: \(h(x) = \frac{1}{2} \log\left(2 \pi e \sigma^2 \right)\).'1spacing'
for the standard 1spacing estimator (see [1] and [2]):
\[h(x) \approx  \gamma(1) + \frac{1}{n1} \sum_{i=1}^{n1} \log \left[ n \left(x_{(i+1)}  x_{(i)} \right) \right],\]where \(x_{(i)}\) is the ith smallest sample, and \(\gamma\) is the digamma function.
'gaussiankde'
(the default) for Gaussian kernel density estimation.
\[h(x) \approx \frac{1}{n} \sum_{i=1}^n \log\left( \hat{p}\left(x_i\right) \right)\]where \(\hat{p}\) is the Gaussian kernel density estimator of the true pdf using
statsmodels.api.nonparametric.KDEUnivariate
.Parameters: x ((n,) np.array) – i.i.d. samples from the distribution of interest. Returns: h – The (differential) entropy of the continuous scalar random variable of interest. Return type: float Raises: AssertionError – If the input has the wrong shape or the method is not supported. References
[1] Kozachenko, L. F., and Nikolai N. Leonenko. “Sample estimate of the entropy of a random vector.” Problemy Peredachi Informatsii 23.2 (1987): 916. [2] Beirlant, J., Dudewicz, E.J., Györfi, L., van der Meulen, E.C. “Nonparametric entropy estimation: an overview.” International Journal of Mathematical and Statistical Sciences. 6 (1): 17–40. (1997) ISSN 10557490.
Entropy Rate¶

kxy.api.core.entropy_rate.
auto_predictability
(sample, p=None, robust=False, p_ic='hqic', space='primal')¶ Estimates the measure of autopredictability of a (vectorvalue) time series:
\[ \begin{align}\begin{aligned}\mathbb{PR}\left(\{x_t \} \right) :&= h\left(x_* \right)  h\left( \{ x_t \} \right) \\\ &= h\left(u_{x_*}\right)  h\left( \{ u_{x_t} \} \right).\end{aligned}\end{align} \]Parameters:  sample ((T, d) np.array) – Array of T sample observations of a \(d\)dimensional process.
 p (int or None) – Number of lags to compute for the autocovariance function. If
p=None
(the default), it is inferred by fitting a VAR model on the sample, using as information criterionp_ic
.  robust (bool) – If True, the Pearson autocovariance function is estimated by first estimating a Spearman rank correlation, and then inferring the equivalent Pearson autocovariance function, under the Gaussian assumption.
 p_ic (str) – The criterion used to learn the optimal value of
p
(by fitting a VAR(p) model) whenp=None
. Should be one of ‘hqic’ (HannanQuinn Information Criterion), ‘aic’ (Akaike Information Criterion), ‘bic’ (Bayes Information Criterion) and ‘tstat’ (based on last lag). Same as the ‘ic’ parameter ofstatsmodels.tsa.api.VAR
.  space (str, 'primal'  'dual') – The space in which the maximum entropy problem is solved.
When
space='primal'
, the maximum entropy problem is solved in the original observation space, under Pearson covariance constraints, leading to the Gaussian copula. Whenspace='dual'
, the maximum entropy problem is solved in the copulauniform dual space, under Spearman rank correlation constraints.
Warning
This function only supports primal inference for now.

kxy.api.core.entropy_rate.
estimate_pearson_autocovariance
(sample, p, robust=False)¶ Estimates the sample autocovariance function of a vectorvalue process \(\{x_t\}\) up to lag p (starting from 0).
Parameters:  sample ((T, d) np.array) – Array of T sample observations of a ddimensional process.
 p (int) – Number of lags to compute for the autocovariance function.
 robust (bool) – If True, the Pearson autocovariance function is estimated by first estimating a Spearman rank correlation, and then inferring the equivalent Pearson autocovariance function, under the Gaussian assumption.
Returns: ac – Sample autocovariance matrix whose ij block of size pxp is the covariance between \(x_{t+i}\) and \(x_{t+j}\).
Return type: (dp, dp) np.array

kxy.api.core.entropy_rate.
gaussian_var_copula_entropy_rate
(sample, p=None, robust=False, p_ic='hqic')¶ Estimates the entropy rate of the copulauniform dual representation of a stationary Gaussian VAR(p) (or AR(p)) process from a sample path.
We recall that the copulauniform representation of a \(\mathbb{R}^d\)valued process \(\{x_t\} := \{(x_{1t}, \dots, x_{dt}) \}\) is, by definition, the process \(\{ u_t \} := \{ \left( F_{1t}\left(x_{1t}\right), \dots, F_{dt}\left(x_{dt}\right) \right) \}\) where \(F_{it}\) is the cummulative density function of \(x_{it}\).
It can be shown that
\[h\left( \{ x_t \}\right) = h\left( \{ u_t \}\right) + \sum_{i=1}^d h\left( x_{i*}\right)\]where \(h\left(x_{i*}\right)\) is the entropy of the ith coordinate process at any time.
Parameters:  sample ((T, d) np.array) – Array of T sample observations of a \(d\)dimensional process.
 p (int or None) – Number of lags to compute for the autocovariance function. If
p=None
(the default), it is inferred by fitting a VAR model on the sample, using as information criterionp_ic
.  robust (bool) – If True, the Pearson autocovariance function is estimated by first estimating a Spearman rank correlation, and then inferring the equivalent Pearson autocovariance function, under the Gaussian assumption.
 p_ic (str) – The criterion used to learn the optimal value of
p
(by fitting a VAR(p) model) whenp=None
. Should be one of ‘hqic’ (HannanQuinn Information Criterion), ‘aic’ (Akaike Information Criterion), ‘bic’ (Bayes Information Criterion) and ‘tstat’ (based on last lag). Same as the ‘ic’ parameter ofstatsmodels.tsa.api.VAR
.
Returns:  h (float) – The entropy rate of the copulauniform dual representation of the input process.
 p (int) – Order of the VAR(p).

kxy.api.core.entropy_rate.
gaussian_var_entropy_rate
(sample, p, robust=False)¶ Estimates the entropy rate of a stationary Gaussian VAR(p) or AR(p), namely
\[h\left( \{x_t\} \right) = \frac{1}{2} \log \left( \frac{K_p}{K_{p1}} \right) + \frac{d}{2} \log \left( 2 \pi e\right)\]where \(K_p\) is the determinant of the lagp autocovariance matrix corresponding to this process, from a sample path of size T.
Parameters:  sample ((T, d) np.array) – Array of T sample observations of a ddimensional process.
 p (int) – Number of lags to compute for the autocovariance function.
 robust (bool) – If True, the Pearson autocovariance function is estimated by first estimating a Spearman rank correlation, and then inferring the equivalent Pearson autocovariance function, under the Gaussian assumption.
See also
Returns: h – The entropy rate of the process. Return type: float

kxy.api.core.entropy_rate.
pearson_acovf
(sample, max_order=10, robust=False)¶ Estimates the sample Pearson autocovariance function of a scalarvalued or vectorvalued discretetime stationary ergodic stochastic process \(\{z_t\}\) from a single sample of size \(T\), \((\hat{z}_1, \dots, \hat{z}_T)\).
\[C(h) := \frac{1}{T} \sum_{t=1+h}^T (\hat{z}_t  \bar{z})(\hat{z}_{th}  \bar{z})^T\]with \(\bar{z} := \frac{1}{T} \sum_{t=1}^T \hat{z}_t\).
Parameters:  sample ((T, d) np.array) – Array of T sample observations of a ddimensional process.
 max_order (int) – Maximum number of lags to compute for the autocovariance function.
 robust (bool) – If True, the Pearson autocovariance function is estimated by first estimating a Spearman rank correlation, and then inferring the equivalent Pearson autocovariance function, under the Gaussian assumption.
Returns: acf – Sample autocovariance function up to order max_order.
Return type: (max_order, d, d) np.array