“Is that to say we are against Free Trade? No, we are for Free Trade, because by Free Trade all economical laws, with their most astounding contradictions, will act upon a larger scale, upon the territory of the whole earth; and because from the uniting of all these contradictions in a single group, where they will stand face to face, will result the struggle which will itself eventuate in the emancipation of the proletariat.”
Karl Heinrich Marx · Marx-Engels Collected Works, Vol. VI, p. 290
You can also find this library at CRAN and download it directly from R and RStudio.
Also, we recommend viewing the mind map summary at the end of the article to better understand the relationship between the functions of the package.
LISTEN TO THIS POST AS A PODCAST:
ESCUCHA ESTA PUBLICACIÓN COMO PODCAST
R Library Review
Meet gdpar
General Dynamic Parameter Models via Reference Anchoring
In the fleeting calculus of a two-second decision—overtaking a car on a narrow road—the human brain performs a remarkable statistical trick. It does not build a model of the approaching driver from scratch. Instead, it retrieves a baseline: the average driver, representing typical reaction times and modal aggression. In a split second, it reads the specific signals of the actual driver—relative speed, vehicle type, micro-movements—and estimates how this specific driver deviates from the baseline. The decision to overtake emerges from that synthesis.
This cognitive recipe—population reference + individual deviation—is the philosophical bedrock of the R package gdpar (General Dynamic Parameter models via Reference Anchoring) by José Mauricio Gómez Julián. The package takes this intuition, formalizes it as a rigorous statistical decomposition, proves the conditions under which it is mathematically identifiable, ships a Stan-based Bayesian engine to estimate it, and layers on causal inference, geometry-adaptive sampling, and dependence-robust inference.
❦
The Anatomy of Deviation
Every layer of gdpar is an elaboration of a single, elegant equation. For each observation $i$ with covariates $x_i$:
Read it as: the parameter of individual $i$ equals a population reference, plus a deviation that is itself a function of the individual’s covariates and of the reference itself.
That final clause is where the architecture pivots from classical statistics. The deviation $\Delta$ does not merely depend on who you are (your covariates $x_i$); it depends on what the reference is. If you transplant the model to a new population, the deviation function behaves differently because $\theta_{\text{ref}}$ is one of its arguments. This structural dependence is the defining feature of “reference anchoring.” It distinguishes gdpar from random-effects or varying-coefficient models, where the deviation is structurally separate from the reference.
So, what is the shape of $\Delta$? The package singles out a specific functional form called the Additive–Multiplicative–Modulated (AMM) decomposition:
Three mechanisms, cleanly separated and independently interpretable:
$a(x)$ — A pure additive shift. Think of this as a traditional fixed-effect driven by covariates.
$b(x)\odot\theta_{\text{ref}}$ — A covariate-dependent scaling of the reference (using the Hadamard/elementwise product). This is where “the deviation depends on the reference” enters multiplicatively.
$W(\theta_{\text{ref}})\,x$ — Covariates are mixed through a matrix $W$ that is, itself, tuned by the reference. This is the explicit, structural reference-dependent channel.
Standard models drop out as special cases. Set $\Delta \equiv 0$ and you have fixed-effects regression. Set $W \equiv 0$ and you have a hierarchical model with multiplicative interaction. Set $b \equiv 0$ and you have a varying-coefficient model. The AMM is the smallest natural family that contains all three and elevates the reference to an active argument of the deviation.
❦
The Three Estimation Engines
gdpar defines three complementary engines for estimating $\Delta$. Crucially, only one is executable in the current release—a deliberate choice to promise a mathematical scope that exceeds the executable surface, and to say so honestly.
Path
Engine
Representation
Status
Path 1
Hierarchical Bayesian (Stan)
Parametric AMM
✅ Operational
Path 2
Varying-coefficient (splines)
Smooth $\beta(z)$
🚧 Conceptual
Path 3
Hypernetwork / Neural Net
Net generates $\theta_i$
🚧 Conceptual
Paths 2 and 3 are documented to “reference grade”—full asymptotic theory (contraction rates, Bernstein–von Mises) is developed in the Wiki—but they abort with gdpar_unsupported_feature_error if invoked. Path 1 places priors on every component ($\theta_{\text{ref}}, a, b, W$) and samples the joint posterior with HMC, yielding native, full-posterior uncertainty.
A Tale of Two Posteriors: EB vs. FB
Within Path 1, gdpar offers two inferential regimes. Full Bayes (FB) via gdpar() samples the joint posterior, remaining most faithful to the cognitive analogy. Empirical Bayes (EB) via gdpar_eb() estimates the hyperparameters by maximizing a marginal likelihood via a Laplace approximation, then samples the remaining parameters conditionally.
The EB vs FB Comparator
Rather than forcing a choice, gdpar treats them as parallel routes. It ships a dedicated comparator, gdpar_compare_eb_fb(), which quantifies agreement on $\theta_{\text{ref}}$ and the reduced parameter vector $\xi$. The Wiki develops the theory to first-class depth: EB and FB lower-level posteriors agree asymptotically (Theorem 7A), while EB intervals under-cover by $O(n^{-1})$ (Proposition 7B). If you have ever wondered if EB is “good enough” for your data, gdpar lets you answer that empirically.
Distributional Regression: Every Parameter is a Slot
gdpar is not constrained to modeling the mean. A probability distribution has multiple parameters—location, scale, shape, tail index, zero-inflation probability—and each one can carry its own AMM decomposition. The package indexes these by $k = 1, \dots, K$:
$$ \theta_i^{(k)} = \theta_{\text{ref}}^{(k)} + \Delta^{(k)}(x_i, \theta_{\text{ref}}^{(k)}), \qquad k = 1, \dots, K $$
The built-in roster covers Gaussian, Poisson, negative binomial, Bernoulli, Beta, Gamma, Student-$t$, Tweedie, ZIP, ZINB, and hurdle families. Zero-inflated and hurdle models receive an especially elegant treatment: both the zero-inflation probability $\pi_i$ and the count parameter $\theta_i$ are anchored to their respective references—a dual deviation design.
❦
The Causal Bridge
Because the AMM form produces individual parameters, individual treatment effects emerge naturally. gdpar_causal_bridge() implements a T-learner: fit the anchored model separately under treatment and control, then read the conditional average treatment effect (CATE) at $x_i$ as the difference of the anchored individual predictions:
A second layer, gdpar_compare_meta_learners(), benchmarks the AMM-based learner against external meta-learners via pluggable adapters: grf::causal_forest on the R side and EconML’s CausalForestDML on the Python side (via reticulate). The framework’s causal claims are benchmarked, not asserted.
Mechanics & Clockwork
Several engineering decisions elevate gdpar from a theoretical exercise to a serious computational environment:
Stan Code Generator: Composes programs from canonical pieces—AMM blocks for $p=1$ and $p \geq 1$, EB marginal/conditional blocks, distributional-$K$ blocks—selected by the resolved $(K, p, \text{family}, W, \text{parametrization}, \text{group})$. The $W$ basis supports B-splines with Stan-side Cox–de Boor evaluation, ensuring differentiability inside HMC.
Identifiability Pre-flight: Before any sampling, gdpar_check_identifiability() runs a Gram-matrix check (Proposition 1C), a per-coordinate cross-component check (C4-bis) for $p > 1$, and a per-group anti-aliasing check (C7). If your design is non-identifiable, you find out before the sampler burns your CPU, accompanied by a structured gdpar_identifiability_error naming the dependent directions.
Data-Driven Reparametrization: Treats the parametrization of $b(x) \odot \theta_{\text{ref}}$ as a pre-fit decision. A short pilot computes an information ratio, dispatching to CP, NCP, or—gdpar‘s root-cause resolution—a linear reparametrization that samples the product $\theta_{\text{ref}} \cdot b$ directly, sidestepping bilinear funnels altogether.
Opt-in Power Tools
Two advanced capabilities are switched off by default, documented as thoroughly as the core path.
1. Geometry-Adaptive Sampling
Hierarchical AMM posteriors can be geometrically hostile—funnels, near-determinism, heavy tails. The opt-in geometry engine climbs a ladder of Riemannian metrics: Euclidean → Fisher/SoftAbs → sub-Riemannian → relativistic/Finsler. A certifying orchestrator diagnoses the pathology, selects a metric, tunes the integrator, and emits a certificate. If full sampling is certified infeasible, a Laplace fallback provides a plug-in posterior with ELPD on par with mgcv-REML or INLA-Laplace.
2. Dependence-Robust Inference
gdpar does not model temporal or spatial dependence in its point structure; instead, it makes the inference robust to dependence (a working-independence + sandwich-variance stance in the spirit of Liang & Zeger, 1986). You receive diagnostics (Durbin–Watson, Ljung–Box, Moran’s $I$) and robust SEs via block bootstrap—moving or circular blocks in time (with the Politis–White flat-top automatic block length), tiled randomized-origin blocks in space. Point estimates remain pristine; only the uncertainty is made honest.
⚠️ Honest Limitations
The Wiki is admirably forthright about scope. Only Path 1 is executable in 0.1.0. Dependence is not modelled—only the inference is made robust. The package’s mathematical scope exceeds its executable surface by design. Read the “Implementation status” notes carefully before relying on a feature.
❦
TL;DR
gdpar takes one of the most natural ideas in human prediction—predict an individual as a deviation from a population reference, where the deviation itself depends on the reference—and transforms it into a fully specified, identifiability-checked, Stan-powered Bayesian regression framework. It is theoretically rigorous, computationally serious, and unusually honest about what it does and does not yet do. If your work involves individual heterogeneity, distributional regression, or causal effect estimation with principled uncertainty, gdpar demands a careful look.
A new research uses probability theory — and sixty years of U.S. economic data — to test one of the most consequential (and most overlooked) assumptions in political economy.
The Assumption Hiding in Plain Sight
If you’ve ever read a Marxist analysis of how profits equalize across industries, you’ve probably encountered something called the average rate of profit. The idea is straightforward: competition between capitalists drives different rates of profit in different sectors toward a common, system-wide average. This is one of the pillars of Marx’s theory of value in Capital, Volume III.
But there’s a quieter assumption underneath this one — so quiet that most discussions never mention it explicitly. To arrive at a uniform profit rate, Marx first assumes a uniform rate of surplus value across all productive sectors. In plainer terms: he assumes that the degree to which workers are exploited — the ratio of unpaid labor to paid labor — is roughly the same everywhere, whether you work in steel manufacturing, food processing, or textiles.
Adam Smith proposed this idea before Marx. Smith argued that if one job were obviously more exploitative (in the sense of yielding far more unpaid surplus per dollar of wages paid), workers and capital would flow toward or away from it until the differences vanished. Marx adopted this observation and, as scholar Jonathan Cogliano notes, elevated it to “the status of a central economic law” within his framework.
Yet the assumption has been challenged from multiple directions — Marxist and non-Marxist alike. Is it actually justified? Or is it a convenient simplification that distorts our understanding of how capitalism works?
José Mauricio Gómez Julián, of the Universidad Latina de Costa Rica, decided to approach the question from an unexpected angle: probability theory. His paper, published in Ciencia Económica (2022), asks whether the mathematical law that would need to hold for this assumption to be valid actually does hold — and then checks the answer against six decades of real-world data from the United States.
The Mathematical Backbone: The Law of Large Numbers
If you’ve taken any statistics course, you’ve likely met the Law of Large Numbers (LLN). It tells us that as you observe more and more instances of something random — coin flips, dice rolls, stock returns — the average of those observations settles down toward the true expected value.
There are two versions:
The Weak Law (WLLN): With enough observations, the sample average is probably close to the expected value.
The Strong Law (SLLN): With enough observations, the sample average is almost certainly equal to the expected value — a much stronger guarantee.
Gómez Julián’s insight is this: if you think of each productive sector of the economy as a random variable representing that sector’s rate of surplus value, then the LLN tells you what happens to the average across sectors as the number of sectors grows large. In mathematical language:
Strong Law: The probability that the average surplus-value rate across sectors equals the global expected value, in the limit, is exactly 1.
Weak Law: The probability that the average deviates from the global expected value by more than any tiny amount shrinks to zero.
If either version holds, you get the result Marx needs: across a sufficiently large number of sectors, the rates of surplus value converge to a common value — uniformity, or at least a powerful tendency toward it.
The Catch: Independence and Identical Distributions
Here’s where things get interesting — and where the classical LLN hits a wall.
The textbook version of the LLN requires two conditions:
Independence: The random variables (sectoral surplus-value rates) must be statistically independent of each other.
Identical distribution: Each variable must follow the same probability distribution.
Neither condition holds for the real economy. And Gómez Julián is admirably upfront about this. Sectors are deeply intertwined — the steel industry depends on mining, manufacturing depends on steel, services depend on consumer spending powered by manufacturing wages. The idea that one sector’s surplus-value rate has no relationship to another’s is economically unrealistic. Furthermore, different industries have different cost structures, different labor intensities, and different technologies. There is no reason their surplus-value rates should follow the same statistical distribution.
So does this kill the argument? Not at all. In fact, it’s the most intellectually interesting part of the paper.
Non-Classical Varieties: When the Rules Relax
Over the past several decades, mathematicians and econometricians have developed non-classical versions of the LLN that weaken or entirely drop the independence and identical-distribution requirements. Gómez Julián surveys several of these:
Li, Rao, and Wang (1995) showed the LLN holds for random variables arranged on a lattice structure under certain conditions — a structure that, as it happens, economic data naturally exhibits.
Adler and Rosalsky (1987) proved the law for weighted sums of independent, identically distributed random variables belonging to a normalized sum, generalizing the classical case.
Chen and Sung (2016) extended those results further: the variables no longer need to be identically distributed. They only need to be “stochastically dominated” by a single random variable, with certain weighting conditions.
Sung (2011) showed that the strong law can hold even when variables are dependent on each other, provided their probability moments (roughly, their averages and variability) satisfy certain finiteness conditions.
The crucial point: these results collectively tell us that the LLN’s convergence conclusion can survive even when the classical assumptions are substantially violated — which is exactly the situation with sectoral surplus-value rates.
Gómez Julián argues that the economic dynamics described by Smith — workers and capital moving between sectors in response to unequal advantages — are precisely the kind of compensatory dependence mechanism that these non-classical versions accommodate. The variables aren’t independent, but their dependence is structured in a way that still drives convergence.
What the Data Actually Shows
The theoretical argument is compelling, but Gómez Julián doesn’t stop there. He turns to sixty years of U.S. data (1960–2020), sourced from the Bureau of Economic Analysis (BEA), to see what the empirical evidence says.
He calculates sectoral surplus-value rates using macroeconomic data on gross operating surplus (representing surplus labor time) and employee compensation (representing necessary labor time), following a standard operationalization of Marx’s categories. After carefully determining which sectors qualify as “productive” in the Marxist sense — a nontrivial task, since the service sector includes activities with very different relationships to surplus-value production — he arrives at 36 productive sectors.
Here’s what the statistical analysis found:
Finding 1: No Identical Distributions
A probability distribution fitting exercise (using the Bayesian Information Criterion) revealed that the 36 sectors’ surplus-value rates follow a patchwork of different distributions — Log-Normal, Cauchy, Uniform, Weibull, and Logistic — with none following a normal distribution. The identical-distribution requirement of the classical LLN is not met.
Finding 2: No Statistical Independence
A Pearson correlation analysis across all 630 possible sector pairs yielded a mean correlation of about 0.08 and a median of about 0.14. While these may look small, a deeper cut reveals that roughly 40% of sector pairs have correlations of 0.3 or above — a level that’s practically meaningful. The sectors are not independent. This makes intuitive sense: industries are connected through supply chains, labor markets, and shared macroeconomic conditions.
Finding 3: Differences Tend Toward Zero
This is the key finding. When Gómez Julián computed the differences between each sector’s surplus-value rate and the global average (both the mean and the median), he found that these differences exhibit a strong tendency toward reciprocal nullification — positive differences roughly cancel out negative ones. The sum of all differences relative to the global mean was essentially zero (on the order of 10⁻¹⁴). The mean of differences relative to the global median was 0.0012 — vanishingly small.
Distributional fitting on these differences revealed they follow a Cauchy distribution (when measured against the global mean) or a uniform distribution (against the global median), with the medians of these distributions sitting very close to zero.
In plain language: sectors deviate from the average in different directions, and those deviations largely cancel each other out.
Why This Matters
Gómez Julián’s paper makes two types of contributions that are worth distinguishing:
For Marxist political economy: If the uniformity assumption holds — even approximately, even as a tendency rather than an iron law — then a large body of research on the long-run behavior of the average rate of profit, both within countries and across the global economy, is on sounder footing than critics have suggested. Researchers studying the tendency of the rate of profit to fall (or not) can continue to work without needing to explicitly model sector-by-sector differences in exploitation rates, at least for aggregate, long-run analyses.
For probability theory and economics: The paper demonstrates a productive intersection between a specific question in political economy and the deep mathematics of convergence theorems. It shows that the non-classical LLN theorems aren’t just abstract curiosities — they have direct relevance to understanding real economic phenomena. The structured dependence between economic sectors isn’t a bug that invalidates the mathematical framework; it’s a feature that the right version of the framework already accounts for.
A Few Honest Caveats — And Why They No Longer Apply
The original 2022 paper was unusually transparent about its limitations, and that transparency is one of its strengths. Rather than forcing the data into inappropriate statistical procedures, it openly acknowledged where the available inferential tools broke down.
At the time, three important caveats remained.
First, formal hypothesis testing had to be abandoned.
The reason was purely statistical rather than economic. Classical inferential procedures—Student’s t tests, Wilcoxon tests, and most conventional non-parametric alternatives—are built on assumptions that the data simply did not satisfy. Sectoral surplus-value rates are neither independent nor identically distributed. They are linked through supply chains, technological change, capital mobility, and macroeconomic shocks. Even bootstrap procedures could not fully solve the problem because ordinary resampling may weaken dependence between resamples while leaving the internal dependence structure fundamentally unchanged. Consequently, the 2022 paper relied primarily on descriptive statistics together with probability-theoretic arguments instead of formal significance testing.
Second, the classification of productive sectors inevitably involved theoretical judgment.
Although the paper carefully justified the inclusion and exclusion of economic activities using Marxian categories and modern national accounting, reasonable scholars could still debate where certain services belong within the circuit of capital.
Third, the empirical evidence came exclusively from the United States.
The descriptive regularities were remarkably strong, but demonstrating that the same convergence mechanism operates under different institutional settings naturally remained an empirical question.
Those were genuine limitations in 2022.
Today, however, the first—and arguably the most important—of them has largely been overcome.
A much more comprehensive methodological paper (Gómez Julián, 2026; SSRN 5172185) develops an entirely new inferential framework specifically designed for exactly the type of data that made the original analysis difficult: dependent, heterogeneous, and unbalanced observations. Instead of trying to force classical statistical tests to work outside the assumptions under which they were derived, the newer paper constructs hypothesis testing from the ground up for this class of problems.
The key innovation is recognizing that the convergence of sectoral surplus-value rates is fundamentally a law-of-large-numbers problem under dependence, not an independent-samples problem. The framework therefore combines three complementary asymptotic structures—triangular arrays (TAC), correlation-weighted sums (WSC), and mixingale processes (MPC)—which respectively model hierarchical dependence, contemporaneous intersectoral dependence, and temporal dependence. Rather than treating these as competing approaches, the paper proves conditions under which they become metrically equivalent and therefore support the same inferential conclusions.
The inferential consequences are substantial.
Instead of abandoning significance testing because dependence invalidates classical procedures, the new framework explicitly extends the Neyman-Pearson paradigm to dependent observations, derives dependence-aware confidence regions, establishes rigorous Type I error control under strong-mixing assumptions, and integrates Bayesian and frequentist inference into a single coherent architecture. Robust procedures—including fixed-b heteroskedasticity-and-autocorrelation-robust inference, block bootstrap techniques that preserve dependence, adaptive conformal inference, composite and Whittle likelihoods, and hierarchical Bayesian estimation—serve as mutually reinforcing validation mechanisms rather than isolated alternatives.
In other words, what had been acknowledged as a methodological limitation in the 2022 paper became the central research question of the later work.
Rather than concluding that inference was impossible under dependence, the subsequent research asks a more fundamental question: what should hypothesis testing look like when dependence is the normal state of the data rather than an exception? The result is a unified inferential framework specifically intended for datasets that violate the assumptions of classical statistics—precisely the situation encountered with sectoral surplus-value rates.
The other caveats remain, although they are considerably less problematic than before. The classification of productive sectors continues to depend on theoretical interpretation, because that issue belongs to political economy rather than statistics. Likewise, expanding the empirical analysis to additional countries remains a desirable avenue for future research. Yet the principal statistical objection—that no valid inferential procedure existed for dependent sectoral data—has now been directly addressed through a purpose-built mathematical framework.
Looking back, the 2022 paper can therefore be read as identifying an important statistical obstacle, while the later work attempts to remove it. Together, the two papers form a coherent research program: first demonstrating that the convergence hypothesis is theoretically plausible and descriptively supported, and then developing the inferential machinery required to test that hypothesis rigorously without relying on unrealistic assumptions of independence or identical distributions.
Gómez Julián, J.M. (2022). Sobre la validez del supuesto de uniformidad en las tasas de plusvalía sectorial desde la teoría de las probabilidades. Ciencia Económica, 11(17).DOI: 10.22201/fe.24484962e.2022.11.17.2
Gómez Julián, J.M. (2026). Hypothesis Testing for Dependent Variables with Unbalanced Data: A Unified Framework: Theory, Robustness, and Software. SSRN Electronic Journal. DOI: 10.2139/ssrn.5172185.
You can also find this libraryat CRAN and download it directly from R and RStudio.
LISTEN TO THIS POST AS A PODCAST:
The Problem Every Quantitative Researcher Knows
Imagine you are staring at a spreadsheet with 50 columns and a few hundred rows of macroeconomic indicators. Somewhere inside this matrix, a handful of variables carry the signal you care about. The rest is noise — or worse, confounding variation that masquerades as signal. You need to answer three questions: Which variables actually matter? What is the latent structure driving them? And how persistent are the components you extract?
If you have worked with panel data for any length of time, you know that these questions are rarely addressed by a single tool. You run a LASSO in one environment, a PCA in another, an ADF test in a third. Each method lives in a different package with different input formats, different assumptions, and different output structures. Stitching the results together is left to you.
SignalY is an R package that was built to solve exactly this fragmentation problem. Developed by José Mauricio Gómez Julián and released under the MIT license, it provides a unified framework for signal extraction from panel data through multivariate time series analysis. Its design rests on three analytical pillars — column selection, series decomposition, and persistence analysis — that can be used independently or chained together through a single orchestrating function.
This post walks through the problem SignalY solves, the methods it implements, and why the combination matters for applied econometrics and data science.
Pillar 1: Which Variables Matter?
The first challenge in any high-dimensional analysis is selection. When you have dozens of potential predictors, you need a principled way to determine which ones carry structural information and which ones are along for the ride.
The Horseshoe Prior
SignalY approaches this problem through Bayesian sparse regression with the Horseshoe prior. The Horseshoe, introduced by Carvalho, Polson, and Scott (2010) and refined for practical variable selection by Piironen and Vehtari (2017), is a global-local shrinkage prior with a distinctive property: it is aggressive around zero (shrinking noise variables strongly toward zero) while maintaining heavy tails (allowing true signals to escape shrinkage). This dual behavior makes it particularly well-suited for sparse problems where you expect only a few variables to matter, but you do not know which ones.
The math behind this is elegant. Each coefficient βⱼ is given a prior with two layers of shrinkage:
A local parameter λⱼ that controls how much each individual coefficient is shrunk.
A global parameter τ that governs overall sparsity.
The result is a shrinkage profile where most coefficients collapse toward zero — the global pull — while a small number of coefficients stand apart, barely affected — the local escape. This is what gives the Horseshoe its name: the prior density looks like the shape of a horseshoe, with a sharp spike at zero and long, flat arms extending outward.
SignalY’s fit_horseshoe() function estimates this model and provides built-in shrinkage profile diagnostics, so you can visually inspect which variables survived shrinkage and by how much.
Four Ways to Select Variables
Fitting a model is one thing; converting the posterior into a concrete variable selection is another. SignalY offers four distinct selection strategies, each with different strengths:
Projection predictive selection (select_by_projection()): This is the most theoretically robust approach. It works by projecting the full posterior onto candidate submodels and selecting the smallest submodel whose predictive distribution is close enough to the full model. The reference is Piironen and Vehtari (2017), and the implementation respects the posterior geometry rather than relying on ad-hoc thresholds.
Credible interval exclusion (select_by_credible_interval()): Selects variables whose posterior credible intervals do not include zero. Intuitive and easy to interpret, though it can be conservative in high-dimensional settings.
Shrinkage-based selection (select_by_shrinkage()): Uses the kappa (shrinkage fraction) parameters to identify variables that escaped shrinkage. This is useful when you want to understand the degree of shrinkage, not just the binary in-or-out question.
Magnitude-based screening (select_by_magnitude()): A straightforward effect-size filter. Useful as a first pass or when you need to combine Bayesian inference with a frequentist-style screening step.
The fact that SignalY provides all four in a coherent pipeline — not as separate, unrelated functions — is the key design decision. You can run all four and cross-validate the results, or choose the one that best matches your inferential philosophy.
Beyond Regression: Factor Discovery
Sometimes the question is not “which of my 50 variables matter?” but rather “what are the few latent factors driving all 50?” SignalY addresses this through two complementary methods:
PCA with block bootstrap (pca_bootstrap()): Standard principal component analysis, but with block bootstrap confidence intervals that account for temporal dependence in time series data. It also includes entropy-based topology analysis, which measures the informational content of each component.
Dynamic Factor Models (estimate_dfm()): Implements the Bai and Ng (2002) information criteria for automatic determination of the number of static factors, then fits a VAR (Vector Autoregression) on the factor dynamics. This captures not just what the latent factors are, but how they evolve over time.
Pillar 2: What Is the Underlying Structure?
Once you know which variables matter (or have constructed a composite signal), the next question is decomposition: what are the trend, cycle, and residual components of your series?
This is where signal processing meets econometrics, and SignalY implements three methodologically distinct approaches, each with its own strengths.
Wavelet Decomposition
filter_wavelet() implements the Maximal Overlap Discrete Wavelet Transform (MODWT) using Daubechies wavelets, following the framework of Percival and Walden (2000).
Unlike a Fourier transform, which decomposes a signal into infinite sinusoids (losing all time information), a wavelet decomposes a signal into localized, finite-length oscillations at different scales. The MODWT variant is particularly useful for time series because it does not decimate the data (no downsampling), meaning the output length matches the input length at every scale.
In practice, the wavelet decomposition separates a series into:
Detail coefficients (D1, D2, D3, …): capturing oscillations at progressively coarser time scales — high-frequency noise in D1-D2, business-cycle frequencies in D3-D4, and longer cycles in higher levels.
Smooth coefficients (S): the low-frequency approximation that captures the trend.
SignalY includes multi-resolution variance analysis, which tells you how much of the total variance is explained at each scale. This is invaluable for understanding whether your series is dominated by high-frequency noise, medium-term fluctuations, or long-run trends.
Empirical Mode Decomposition
filter_emd() implements Empirical Mode Decomposition (Huang et al., 1998), a fundamentally different approach. Where wavelets impose a predetermined basis (Daubechies, Haar, Symmlet, etc.), EMD is fully data-adaptive. It works by iteratively sifting the signal — identifying local extrema, fitting envelopes, and subtracting the mean — until it extracts Intrinsic Mode Functions (IMFs) that satisfy specific oscillatory conditions.
The key advantage of EMD is that it makes no assumptions about stationarity or linearity. The IMFs are defined by the data itself, not by a mathematical basis. This makes EMD particularly powerful for:
Non-stationary signals whose frequency content changes over time.
Non-linear oscillations that cannot be captured by fixed-basis decompositions.
Signals where the “natural” decomposition is not known a priori.
The trade-off is that EMD can be sensitive to end effects and mode mixing, though SignalY’s implementation includes standard mitigations.
HP-GC Bayesian Filter
filter_hpgc() implements the Grant and Chan (2017) unobserved-components Hodrick-Prescott filter, estimated via MCMC. This is a significant upgrade over the traditional HP filter, which requires you to manually set the smoothing parameter λ (the famous λ = 1600 for quarterly data, or λ = 6.25 for annual data, or any of the other arbitrary rules of thumb floating around the literature).
The HP-GC approach formulates the decomposition as a Bayesian unobserved-components model:
A trend component whose second differences are penalized (this is the smoothness prior, equivalent to the HP penalty).
A cycle component modeled as an AR(2) process.
The smoothing parameter λ is estimated from the data via MCMC, not fixed by the user.
This removes one of the most criticized aspects of the classical HP filter — its sensitivity to the arbitrary choice of λ — while preserving its interpretability. The output includes the estimated trend, cycle, and residual, each with full posterior distributions.
filter_all(): Compare All Three
A particularly useful design choice is the filter_all() function, which runs all three decomposition methods on the same series and returns the results in a comparable format. This is not just a convenience function; it is an epistemological statement. No single decomposition method is universally correct. By running all three and comparing, you can identify components that are robust across methods (strong signal) versus components that depend on the specific decomposition assumptions (potentially method artifact).
Pillar 3: How Persistent Is the Signal?
The third question — what is the persistence regime of your series or its components — is critical for downstream modeling. If your extracted trend is a random walk, that has very different implications than if it is a stationary AR process. If your cycle is near-unit-root, standard mean-reversion models will fail.
A Comprehensive Unit Root Battery
test_unit_root() runs four classical tests with complementary null hypotheses:
Test
Null Hypothesis
Key Feature
Augmented Dickey-Fuller (ADF)
Unit root exists
Most widely used; sensitive to lag selection
Phillips-Perron (PP)
Unit root exists
Non-parametric correction for serial correlation
KPSS
Series is stationary
Reversed null; useful as cross-check against ADF
Elliott-Rothenberg-Stock (ERS)
Unit root exists
Point optimal test with higher power near unity
The critical insight is that no single test is definitive. The ADF and PP tests can fail to reject a false unit root (low power near unity). The KPSS test has the opposite null hypothesis, so it can detect stationarity that ADF misses. By running all four and synthesizing the results, SignalY provides a more robust classification than any individual test.
The automated synthesis follows a standard decision logic:
If ADF/PP/ERS reject unit root and KPSS fails to reject stationarity → stationary.
If ADF/PP/ERS fail to reject and KPSS rejects → unit root.
Mixed results → borderline / near-unit-root, flagged for careful interpretation.
This automated synthesis is not a black box; the individual test statistics and p-values are all available for inspection. But the synthesis gives you a quick, defensible classification without manually cross-referencing four separate test outputs.
The Orchestrator: One Call, Full Pipeline
The signal_analysis() function is the centerpiece of SignalY’s design philosophy. A single call can run the complete analysis pipeline:
…and returns a unified result object with print(), summary(), and plot() methods. The plot() method generates interactive plotly dashboards with filter trends, coefficient profiles, PCA loadings, and DFM factor panels.
The formula interface (Y ~ X1 + X2 + X3) makes it feel like a standard R regression call, while the methods argument lets you mix and match analytical layers as needed.
How Well Does It Work?
The Wiki includes recovery benchmarks on synthetic data with known ground truth. These are worth highlighting because they address the most important question: does this actually work?
Task
Method
Recovery Metric
Factor structure (3 latent factors)
PCA / DFM
r > 0.95, exact factor count
Sparse variable selection (5 of 50)
Horseshoe
F1 > 0.85, Precision > 0.90
Logarithmic trend recovery
EMD
r > 0.95 with true trend
Multi-scale cycle extraction
Wavelet (D3+D4)
r > 0.70 with true cycle
Stochastic trend + AR(2) cycle
HP-GC Bayesian
Trend r > 0.90, cycle r > 0.50
Stationarity classification
Unit Root Battery
4/4 correct on synthetic data
A few things stand out:
The Horseshoe achieves over 90% precision in a 5-of-50 sparse selection problem. This means that when it says a variable matters, it is almost always right. The F1 score above 0.85 indicates a good balance between precision and recall.
Factor recovery is near-perfect (r > 0.95), and the DFM correctly identifies the exact number of latent factors.
Wavelet cycle extraction at r > 0.70 and HP-GC cycle extraction at r > 0.50 reflect the inherent difficulty of extracting cyclical components from noisy data. These are realistic numbers, not inflated claims.
Unit root classification achieves 100% accuracy on synthetic data with clear-cut cases. Real-world data is messier, but this validates the synthesis logic.
Who Should Use SignalY?
SignalY is built for three overlapping communities:
Economists and econometricians working with panel or multivariate time series data who need to move from raw data to structural inference — identifying relevant variables, extracting latent factors, decomposing signals, and characterizing persistence — without stitching together five different packages.
Quantitative researchers in finance, macro, or political economy who face high-dimensional predictor sets and need principled Bayesian variable selection rather than stepwise regression or arbitrary LASSO tuning.
Data scientists working on signal processing problems where the signals are non-stationary, non-linear, or embedded in high-dimensional panels, and where the standard Python signal processing toolkit does not provide the statistical rigor needed for publication-quality inference.
Getting Started
Installation is straightforward:
# From GitHub
remotes::install_github("IsadoreNabi/SignalY")
library(SignalY)
# Minimal workflow
data<-data.frame(Y=as.vector(Y), X)
result<-signal_analysis(data=data, y_formula="Y",
methods=c("pca", "wavelet", "unitroot"))
plot(result)
The package is MIT-licensed, actively maintained (current version 1.1.2), and designed to work with standard R data frames.
The Bigger Picture
What makes SignalY interesting is not any single method — the Horseshoe prior, MODWT, EMD, and unit root tests all exist in other packages. The value is in the integration. By placing Bayesian sparse regression, spectral decomposition, and persistence analysis inside a single coherent framework with a unified interface, SignalY enables workflows that are difficult to replicate otherwise:
Run a Horseshoe regression to select variables, then decompose the fitted signal with wavelets, then test the stationarity of the extracted components — all without changing packages, data formats, or mental models.
Compare wavelet, EMD, and HP-GC decompositions of the same series to identify robust components versus method-dependent artifacts.
Use the DFM to discover latent factors, then test each factor’s persistence regime to inform your downstream modeling choices.
In applied econometrics, the quality of your inference depends on the coherence of your pipeline. SignalY makes that coherence a feature rather than a chore.
SignalY is developed by José Mauricio Gómez Julián. The source code, documentation, and wiki are available at github.com/IsadoreNabi/SignalY under the MIT License.
You can also find this libraryat CRAN and download it directly from R and RStudio.
LISTEN TO THIS POST AS A PODCAST:
You have a national Consumer Price Index. It is a single number per year. What you actually want is the price index for each sector of the economy — manufacturing, services, agriculture, construction — because those individual paths are what your model, your policy analysis, or your investment thesis really needs. You know how the sectors combine into the whole: the weights are public, or at least knowable. What you do not know is the sectoral numbers themselves. You only ever see their weighted sum.
This is the disaggregation problem. It sounds like bookkeeping. It is actually a quietly profound statistical question, and the R package BayesianDisaggregation — built by José Mauricio Gómez Julián — tackles it with a degree of intellectual honesty that is rare in software. This post is a deep, plain-language tour of what the package does, why it works the way it does, and what it teaches about building statistical tools that tell the truth.
If you want to follow along with the code, the project’s wiki has installation instructions, function references, and worked examples. This post deliberately stays code-free so the ideas come through.
The problem, precisely
Let’s make it concrete. You observe an aggregate index — call it the CPI — over T years. You also have a matrix of weights W: for each year and each of K sectors, W tells you that sector’s share of the total. The weights sum to one within each year. The aggregate is, up to measurement noise, the weighted sum of the latent sectoral indices:
CPI at year t ≈ the weighted sum of the K sectoral indices, using that year’s weights.
The goal: recover the K sectoral indices — call them φ — from the single aggregate series and the known weights.
Here is the catch, and it is the entire heart of the matter. At every single year, the aggregate pins down one linear combination of the K sectors. The remaining K−1 directions are completely unconstrained by the data. With four sectors, you have one equation and four unknowns per year. The system is, in a precise mathematical sense, under-determined.
This is not a numerical annoyance you can engineer away. It is structural. Any method that hands you a single, sharp sectoral path from an aggregate alone is, whether it admits it or not, smuggling in assumptions to fill the gap — and most methods do not tell you how much of the answer is data and how much is assumption.
The question is not can you disaggregate. You always can. The question is: can you do it honestly, showing your work, and carrying the right amount of uncertainty forward?
The wrong way: a cautionary tale
The first version of the package — the 0.1.x line — advertised “MCMC-free Bayesian disaggregation.” The pitch was appealing: no slow Markov chain Monte Carlo sampling, just clean closed-form math. The implementation used a family of deterministic update rules — weighted, multiplicative, Dirichlet, adaptive — over the prior weight matrix.
It did not work. Not in the sense of crashing. In the much more dangerous sense of appearing to work while silently not doing the one thing it claimed to do.
The package’s own author, in a moment of radical honesty that is worth pausing on, audited the 0.1.x family and catalogued six foundational defects, labeled F1 through F6:
F1 — the aggregate never entered the computation. The “posterior” was derived entirely from the prior weight matrix. The actual observed CPI — the one piece of real evidence — was never used. The method was not conditioning on data; it was rearranging priors.
F2 — the Dirichlet concentration cancelled on renormalization. A parameter that was supposed to control how concentrated the sectoral estimates were simply vanished in the algebra when weights were normalized to sum to one.
F3 — the temporal pattern cancelled too. A component meant to encode smoothness over time also disappeared in the renormalization.
F4 — the “efficiency” term was a fixed constant. It looked like a data-dependent quality score; it was actually invariant.
F5 — there were no recovery tests. No one had ever generated synthetic data with a known truth and checked whether the method got it back.
F6 — a correlation helper cheated. It computed both Pearson and Spearman correlation and reported whichever was larger, a form of silent data-snooping.
The most damning of these is F1. A Bayesian method that does not condition on data is not a Bayesian method. It is a deterministic transformation dressed in Bayesian vocabulary. And the worst part is that it looked reasonable — it returned numbers, they moved in plausible directions, and nothing crashed.
The author’s response is, I think, the most important thing about this whole project. Rather than patching the defects one by one — adding the CPI here, fixing the concentration there — the author recognized that the foundational problem (not using the data) cannot be fixed within a deterministic re-weighting framework. The fix is not a patch. The fix is a fundamentally different model: one in which the aggregate enters as genuine evidence.
So the entire 0.1.x family was deleted. Every function — bayesian_disaggregate(), compute_L_from_P(), spread_likelihood(), the four update rules, the grid search, the save function, the cheating correlation helper — all of it, gone. In its place: two Bayesian engines that actually condition on the data. The package version jumped to 0.2.0, the DESCRIPTION was edited to remove any claim of novelty, and the documentation was rewritten to be explicit about what was removed and why.
This is what intellectual honesty in software looks like. It is not common. We should notice it when it happens.
The right idea: the aggregate as evidence
The conceptual move is simple to state and deep in consequence. Instead of treating the relationship between the aggregate and the sectors as a renormalization identity — a bit of algebra you apply after the fact — treat it as an observation density. The aggregate CPI is data. It is evidence about the latent sectors. The model should condition on it.
In the package’s canonical engine, this means: the latent sectoral indices φ evolve over time as a random walk with drift, and the observed CPI is generated from the weighted sum of the sectors with some observation noise. The aggregate is not a constraint imposed after estimation; it is the likelihood. The sectors come out the other side as a posterior distribution — not a single number, but a full cloud of plausible values with credible intervals.
This is the difference between solving an equation and updating beliefs in light of evidence. The first gives you a point. The second gives you a distribution. And as we will see, the distribution is the whole point.
Two engines, one trade-off
The package offers two ways to do the Bayesian inference, and the choice between them is a clean, well-explained trade-off between richness and exactness.
The state-space MCMC engine
This is the canonical, full-featured model. The sectoral indices live in log space, which guarantees they stay positive — a natural constraint for price indices that a model in raw levels could violate. Each sector’s log-index follows a random walk with its own drift and its own innovation scale (the amount of jitter per period).
Two layers of hierarchical structure make this more than K independent random walks:
Partial pooling on the drift. Each sector has its own drift, but the drifts are drawn from a common distribution. This means sectors share information about their average growth rate without being forced to be identical — the classic shrinkage trade-off.
Partial pooling on the innovation scale. Similarly, each sector’s volatility is drawn from a shared distribution. Sectors borrow strength in estimating how jittery they are.
The initial cross-section — the starting levels of the sectors at the first period — is anchored at the aggregate level with an estimable dispersion parameter. This is a subtle but important point. In the old, broken 0.1.x family, the “concentration” parameter was supposed to control how spread out the sectors were, but it cancelled in the algebra and had no effect. In the new model, the dispersion is a genuine parameter that the data and priors can actually estimate. It does not cancel. It does real work.
Finally, the observation: the CPI is modeled as coming from a Student-t distribution centered at the weighted sum of the sectors, with an estimable scale. The Student-t (rather than a Gaussian) makes the model robust to outliers in the aggregate — a heavy-tailed observation can be accommodated without wrecking the fit. If you prefer, you can switch to a plain Gaussian observation.
Because this model is not conjugate — the log transform, the Student-t, and the hierarchical structure break the neat algebra that would allow a closed-form solution — it is fit by Hamiltonian Monte Carlo via Stan (using either the cmdstanr or rstan backend). HMC is the gold standard for this kind of model: it handles the correlated, high-dimensional parameter space efficiently and comes with reliable diagnostics. The package runs four chains by default, checks the R-hat convergence statistic and the number of divergent transitions, and returns posterior draws of every sectoral index at every period.
The closed-form conjugate engine
The second engine is the linear-Gaussian counterpart. The sectoral indices evolve as a random walk in levels (not logs), and the aggregate is observed with Gaussian noise. This model is conjugate, which means its exact posterior can be computed in closed form — no MCMC, no sampling, no convergence diagnostics. The tool is the Kalman filter combined with the Rauch-Tung-Striebel smoother: the filter passes forward through time, updating beliefs about the sectors given each new CPI observation, and the smoother passes backward, refining those beliefs using future information.
If you want joint posterior draws — not just the smoothed means and variances, but actual correlated samples from the full posterior — the package uses the Durbin-Koopman simulation smoother, a elegant technique that produces draws with the correct cross-time and cross-sector covariance structure. These draws are not marginal approximations; they are genuine samples from the joint posterior.
This engine is the “correct realization of the original MCMC-free posterior idea.” The 0.1.x family wanted a closed-form Bayesian answer; the problem was not that closed-form is un-Bayesian (conjugacy is perfectly Bayesian), but that the old method did not use the data. This engine uses the data — the aggregate enters as the observation equation — and it does so in exact, closed form.
The trade-off is explicit and documented. The closed-form engine buys you speed and mathematical exactness. It costs you three things: positivity (levels can drift slightly negative, which logs prevent), robustness (Gaussian observations are sensitive to outliers, which the Student-t handles), and the cross-sector hierarchy (there is no partial pooling in the linear-Gaussian model). The MCMC engine buys you all three of those, at the cost of sampling.
Both engines return the same thing: an array of posterior draws of the sectoral indices, with dimensions [periods × sectors × draws], plus summary tables of medians and 95% credible bands.
The honesty at the center: weak identification
Here is where the package earns its deepest respect. It would be tempting, after building a real Bayesian model that conditions on the data, to declare victory and hand users sharp sectoral estimates. The package does not do this. Instead, it is explicit about a fact that most disaggregation methods gloss over: the sectoral split is only weakly identified.
Remember the under-determination: at each period, the aggregate pins down one linear combination of the K sectors. The remaining K−1 directions are governed by the prior — the cross-sectional dispersion and the temporal smoothness — not by the data. This means the posterior intervals for individual sectors are wide, and they are influenced by the prior. This is not a bug. It is not a limitation to be engineered away. It is the correct representation of what the data can and cannot tell you.
The package’s own recovery tests — which generate synthetic data from the model’s own data-generating process, where the true sectoral paths are known — confirm this directly. The aggregate is recovered essentially perfectly: the correlation between the fitted aggregate and the true aggregate is above 0.95, often essentially 1.0. The aggregate is strongly identified. But the sectoral coverage — the fraction of times the true sectoral path falls within the 95% credible band — is around 0.84, with the bands being deliberately wide. The test asserts that coverage exceeds 0.70, a conservative threshold, because the package refuses to claim sectoral precision that the data cannot deliver.
This is “rigour by layers”: assert tightly what is identified, assert conservatively what is not. It would be easy to tune the prior to produce narrower, more impressive-looking bands. The package deliberately does not.
Why the full posterior matters: propagating uncertainty
If the sectoral estimates are uncertain — wide bands, prior-influenced — then what good are they? This is where the design of the package reveals its purpose. The sectoral indices are not the final product. They are input to a downstream model. In the author’s research program, they feed a nested Ornstein-Uhlenbeck model of price gravitation. But the principle is general: any time a disaggregated estimate feeds a second-stage analysis, the uncertainty in the first stage should flow into the second.
The package handles this by multiple imputation, following Rubin’s rules. Each posterior draw of the sectoral indices is treated as one imputation — one plausible version of the truth. The downstream model is fit once per imputation, and the results are combined. The effect is that the weak per-sector identification — the wide bands, the prior influence — is carried forward into the downstream uncertainty intervals rather than being discarded. You do not plug in a point estimate and pretend it is the truth. You plug in the whole cloud and let the cloud’s shape propagate.
The package’s documentation is candid about a consequence: because disaggregation is under-determined, the random-walk smoother prior dilutes the reversion signal, biasing the downstream reversion speed toward slowness by a modest, quantified amount (roughly 13–26%). Crucially, the direction is conservative — the true gravitation is at least as fast as reported — and the fraction of missing information that is propagated is about 0.4. This is not hidden. It is measured, reported, and flagged as a property of honest under-determination.
A sharp contrast: an ad-hoc method that simply added noise to a point estimate did not produce proper imputations and gave sub-nominal coverage when routed through Rubin’s rules. The coherent posterior from the conjugate engine — the one that actually conditions on the data — did. The mathematics of multiple imputation demands proper posterior draws; garbage in, garbage out.
How it is validated
The package’s validation strategy is worth studying because it embodies a philosophy: test the identified quantity tightly, test the unidentified quantity conservatively, and test the computation itself exactly.
Three layers, kept deliberately separate:
Smoke tests run always, on every check. They confirm that both engines compile, sample, and return the correct [periods × sectors × draws] array structure on synthetic data. They catch breakage.
Recovery tests are gated behind an environment flag because they require actually compiling and sampling the Stan model, which is slow. They generate data from the model’s own data-generating process — the same random walk with drift, the same partial pooling, the same aggregate observation — so the true sectoral paths are known. Then they check: does the aggregate come back essentially perfectly? (Yes, correlation above 0.95.) Do the sectoral bands cover the truth at a reasonable rate? (Yes, above 0.70, honestly wide.) The recovery test is well-posed because the simulator uses the same process as the model. If the model cannot recover its own data, something is wrong with the sampler or the implementation. If it can, you have a meaningful baseline.
Golden tests run always and are the most stringent. They use Stan’s generate_quantities function — which deterministically recomputes derived quantities from frozen parameter draws, with no random number generation involved — and demand a bit-for-bit match against a frozen reference output. This catches any change to the model’s computed quantities: if someone edits the Stan code and the log-likelihood values shift by even one bit, the test fails. The reference fixture is generated by the same code path, isolating the CSV serialization so the comparison is exact.
This is not the “does it run?” school of testing. It is the “does it compute the right thing, and does it compute exactly the same thing tomorrow?” school.
Where it sits among existing methods
The package is careful — almost unusually careful — about situating itself relative to the existing literature. It makes no claim of being the first or only solution to this problem. The documentation uses the phrase “we did not find” rather than “we are the first,” and the DESCRIPTION file was explicitly edited to remove any “novel/original” claim.
The adjacent traditions, and what each misses:
Biproportional balancing (RAS, IPF) iteratively scales a matrix to match new margins. It is deterministic: no posterior, no credible intervals, no treatment of the aggregate as evidence. It is a useful accounting tool, not an inference method.
Temporal disaggregation (Denton, Chow-Lin, Fernández) distributes a low-frequency aggregate to higher frequency using an indicator series. This is a temporal problem — splitting annual into quarterly — not a cross-sectional one. It assumes you already have the sectoral decomposition and just need finer time resolution.
Forecast reconciliation (MinT and related methods) projects inconsistent hierarchical forecasts onto a coherent subspace. It is forecast-centric and linear-algebraic: it corrects forecasts that do not add up, rather than recovering latent components from an aggregate by Bayesian updating.
Compositional or Dirichlet state-space models evolve simplex weights over time. They model how shares move, not how the components themselves are recovered conditioned on their weighted sum.
Each tradition addresses a real problem. None, as far as the package’s author could find, does exactly this: recover latent cross-sectional components from a single aggregate by conditioning on it as a genuine observation density and returning a posterior that can be propagated downstream. The claim is narrow and checkable, not sweeping.
The data pipeline
The package is not just a model; it is a usable tool. It includes hardened readers for the real inputs. A CPI reader pattern-matches on column headers (in English or Spanish — it recognizes “date,” “fecha,” “year,” “año” for the time column and “cpi,” “indice,” “price” for the value column), parses localized number formats (European-style decimals and thousands separators), collapses duplicate years by averaging, and returns a clean, sorted data frame. A weights reader loads a sector-by-year table, normalizes weights to the simplex within each year, and handles missing entries gracefully.
An alignment function intersects the years covered by the CPI and the weights, ensuring both cover the same periods before either engine runs. A convenience wrapper reads both files and runs the disaggregation in one call. And a simulator generates synthetic data from the model’s own data-generating process — the same random walk with drift, the same partial pooling, the same aggregate observation — so that recovery tests, examples, and exploratory analysis are always well-posed.
One data note worth flagging, because it is a common error: the model works in index levels, not rates of change. Feeding a percent-change series (inflation rate) instead of a level series (the CPI itself) is a category error — the aggregate would not be on the same scale as the weighted sum of the sectors. The CPI must be a level series, re-indexed to the same base as whatever the sectors will be compared against.
The bigger lesson
You could read this package as a technical contribution: a Bayesian state-space model for disaggregation with two engines, honest uncertainty, and a propagation contract. That reading is correct but incomplete.
The deeper lesson is about how to build statistical software that tells the truth. The 0.1.x family did not fail by crashing. It failed by producing plausible-looking output that did not depend on the data. That is the most dangerous failure mode in statistics, because there is no error message. The numbers look reasonable. The plots look smooth. Nothing warns you that the entire computation is a rearrangement of priors.
The author caught it — caught it in their own work, which is harder than catching it in someone else’s — by doing the unglamorous thing: generating data with a known truth and checking whether the method recovered it. When it did not, they did not patch. They deleted and rebuilt. And then they documented the deletion, in public, with the defects labeled and explained, so that anyone reading the history would understand not just what changed but why.
The resulting package has a quality that is hard to name but easy to feel when you read the source: every design choice has a reason, every reason is documented, and the documentation is honest about what the method can and cannot do. The aggregate is strongly identified; the sectors are weakly identified; the uncertainty is wide and prior-influenced; and all of that is surfaced, not hidden, because the whole point is to carry that uncertainty forward rather than fake it away.
In a field where it is tempting to claim sharp results from sparse data, this is a quiet act of integrity. The package does not solve the under-determination. Nothing can. It does something better: it honors it, by returning the wide, honest, propagatable posterior that the data actually supports.
The package, its source code, installation instructions, and full function reference are on GitHub, with extended documentation in the wiki. It is MIT-licensed and written in R, with the MCMC engine powered by Stan.
You can also find this libraryat CRAN and download it directly from R and RStudio.
LISTEN TO THIS POST AS A PODCAST:
When Market Prices Gravitate: A Bayesian Look at an Old Question in Economics
An old question, asked again — properly
There is a question in economics that is older than most of the academic disciplines that border it. Do market prices — the noisy, day-to-day, here-and-now prices at which goods actually change hands — tend to settle toward some underlying center of gravity? And if they do, how fast, how violently, and through what mechanism?
Classical political economy, from Smith and Ricardo through Marx, thought they do. The idea was that behind the churning surface of market prices there sit “prices of production”: long-run, cost-anchored prices toward which actual prices are pulled, the way a spring pulls a weight back toward its rest position. In the Marxian version, there is one more layer underneath: those prices of production themselves are supposed to gravitate around “values,” the labour embodied in commodities. Whether any of this is true is an empirical question, and for a long time the empirical tools to answer it were not really up to the job.
A small R package called bayesianOU, written by José Mauricio Gómez Julián and hosted on GitHub, takes a serious swing at that question. It is not the first attempt to test price gravitation statistically, but it is one of the most technically careful I have seen, and it is built in a way that is instructive far beyond the Marxian debate that motivates it. What follows is a walkthrough of what the package does, why it is interesting, and — just as importantly — where it honestly admits its own limits.
The tool that makes it possible: the Ornstein-Uhlenbeck process
Strip the economics away for a moment and the statistical core of the package is a workhorse object from physics: the Ornstein-Uhlenbeck (OU) process. Imagine a particle moving in a fluid, attached to a spring. Brownian motion jiggles it randomly; the spring pulls it back toward a fixed point. The further it drifts away, the harder the pull. The result is a wiggly series that never settles but always tends to settle — a mean-reverting random walk.
The OU process is exactly the mathematical object you want when you suspect a variable is noisy but anchored. It has a “speed of reversion” (how hard the spring pulls) and an “equilibrium level” (where the spring’s rest point is). Estimate those, and you can say something quantitative about gravitation: not just “yes, prices come back,” but “they come back with a half-life of about nine years.”
That number — the half-life — is the prize. It is the difference between “market prices eventually settle” (which could mean anything) and “market prices settle on a timescale comparable to a business cycle” (which is a falsifiable, interpretable claim).
What the package actually builds
The package fits, by Bayesian inference, a family of models built on the OU process but considerably richer than the textbook version. There are two first-class models, sharing one inference engine.
The single-level model
The first model asks: do market prices revert toward an equilibrium that is a function of the prices of production, and what does that reversion look like once we let it be nonlinear, volatile, heavy-tailed, and structurally heterogeneous across sectors?
Each of those adjectives is doing real work, and each corresponds to a feature that simpler approaches handle poorly or not at all:
Nonlinear drift. A plain OU process pulls back with a force proportional to the deviation. The package allows a cubic correction, so the restoring force can strengthen super-linearly when prices are far from equilibrium. This matters: real markets may behave gently near the center and violently at the extremes, and a linear model cannot represent that.
Stochastic volatility. Financial data, and economic data generally, go through quiet stretches and turbulent ones. The package does not assume a single noise level; it lets the volatility itself wander over time, following its own mean-reverting process on the log-variance. This is the same idea that powers modern stochastic-volatility models in finance, and it is essential for not fooling yourself about the precision of your estimates.
Heavy tails. Economic shocks are not Gaussian. Crashes, booms, and policy shocks produce outliers that a normal distribution would call essentially impossible. The package uses Student-t innovations and estimates the degrees of freedom from the data, so the model can discover for itself just how fat-tailed the world is.
Hierarchical structure across sectors. An economy has dozens of sectors, and each one presumably has its own reversion speed, its own equilibrium, its own noise. Estimating each sector in isolation throws away the information that they are all part of the same economy. Estimating them all with one set of parameters pretends they are identical. The package takes the middle path — hierarchical, or “partial pooling,” priors — where each sector’s parameters are drawn from a shared distribution whose properties the model also estimates. Sectors borrow strength from one another without being forced into lockstep.
A time-varying coupling. This is the most economically loaded feature. The strength with which market prices track prices of production is allowed to depend on the aggregate profit rate (what the package calls TMG). When the general rate of profit is high, the pull of production prices on market prices may be one thing; when it is low, another. Whether that modulation exists, and in which direction, is a hypothesis the model can test rather than assume.
All of this is estimated jointly, with full Bayesian uncertainty, using Stan’s Hamiltonian Monte Carlo sampler. You do not get a point estimate of the reversion speed; you get a posterior distribution, and from it a credible interval and a probability statement like “there is a 95% chance the half-life is between six and eighteen years.”
The nested cascade
The second model is the more ambitious one, and it is where the package earns its “nested” branding. Instead of market prices reverting to a fixed equilibrium, they revert to a latent production price — a hidden, unobserved series that itself evolves over time according to its own OU process, driven by the general profit rate. And, if you turn on the third level, that latent production price in turn gravitates around an observed “value” index built directly from labour-content accounting.
So the full structure is a cascade: market price → latent production price → value. Each arrow is an OU reversion, each with its own speed, and the speeds are constrained so that the outer (market) layer reverts faster than the inner (production) layer — an economically natural separation of timescales, enforced softly so the data can push back.
The reason this matters is that it converts a slogan — “prices of production gravitate around values” — into a literal statistical hierarchy that can be fit to data and compared against alternatives. The headline empirical result, from a fit to 37 US sectors over 1960–2020, is a value-coupling coefficient essentially equal to one, with the posterior probability of it being positive effectively equal to one. In plain terms: in standardized units, prices of production track labour values almost one-for-one. That is a found result, not an assumed one — the prior on the coupling was centred at zero, deliberately neutral.
The inference engine, and why it is not a footnote
It would be easy to glance at the model description, nod, and move on. But how these quantities are estimated is half of what makes the package serious, and it is worth a paragraph for readers who do not think about MCMC every day.
Bayesian inference works by exploring the space of all parameter values consistent with both the data and the prior, and characterizing that space as a probability distribution. For models this complex — with latent volatility paths, hierarchical structure, and hundreds of parameters — you cannot do that with pencil and paper. You use a Markov chain Monte Carlo sampler, specifically Hamiltonian Monte Carlo, which borrows an idea from physics: give the parameter space a “potential energy” (the log-posterior) and a “kinetic energy” (a randomly chosen momentum), and let the system glide around the posterior like a ball rolling over a landscape.
Stan’s NUTS sampler automates this about as well as it can be automated, and the package uses it with within-chain parallelism (via Stan’s reduce_sum) to handle the fact that the likelihood must be summed over many timepoints and sectors. The diagnostics — R-hat for chain agreement, effective sample size, divergence counts — are surfaced through a validate_ou_fit function, and the package is explicit that you should look at them before believing anything.
Model comparison is done with PSIS-LOO, a clever technique that approximates leave-one-out cross-validation without refitting the model dozens of times, by reweighting the posterior draws using importance sampling. It is the modern standard, and the package is appropriately cautious about it: because the model has a latent volatility state at every observation, plain LOO is known to be optimistic, and the documentation says so plainly.
The honesty that makes it credible
Here is where the package surprised me, and here is why I think it deserves a wider audience than the Marxian-economics niche it lives in.
A naïve reading of the results would be triumphant: the value coupling is one-to-one, the reversion exists, the half-life is about nine years. But the package’s own validation section does something rare. It runs the model against legitimate rivals on genuinely held-out data — a full decade, 2011 to 2020 — and reports, without spin, that a random walk beats the OU model at forecasting, that a no-gravitation restriction ties or beats it, and that the value term adds no detectable predictive density.
That sounds like a refutation. The package argues, carefully, that it is nothing of the sort — and the argument is the most intellectually interesting thing here.
The key move is to distinguish two different questions. One is structural: does a reversion mechanism exist, and how fast is it? The other is predictive: can you forecast next year’s price better than a naïve benchmark? These are related but not identical, and for a slow process they come apart in a specific, predictable way.
If gravitation is real but slow — a half-life of nine years on a dataset whose test window is a decade — then over the forecast horizon the process looks, to first order, like a random walk. The reversion is there, but it is too weak to show up in a one-step or few-step prediction. The random walk, which assumes no reversion, will forecast almost as well, because over short horizons a barely-reverting process and a non-reverting one are nearly indistinguishable. So the random walk winning the forecasting horse race is not evidence against gravitation; it is evidence consistent with gravitation being slow.
This is not special pleading. It is a logical point about what different functionals of a model can and cannot tell you. The structural parameters — estimated from the joint likelihood over the whole panel, borrowing strength across 37 sectors and 61 years — use far more information than any single-series forecast. They can pin down a central tendency that a univariate test cannot. And the package shows, through simulation-based calibration and adversarial negative controls, that the estimation pipeline does not manufacture gravitation when none is present: feed it a true random walk and it reports a half-life of about fifty years; feed it a null value-coupling and the posterior honestly covers zero.
The low-kappa trap, and why it matters to everyone
The package names a difficulty it calls the low-kappa trap, and it is worth understanding because it is a trap that catches far more than Marxian price theory.
Kappa is the reversion speed. As kappa shrinks toward zero, the OU process approaches a pure random walk. The trouble is that there is no bright line separating “slow mean reversion” from “no mean reversion.” It is a continuum, and three distinct problems stack up exactly there:
Algebraically, reversion speed and discrete-time persistence are two sides of the same coin; kappa going to zero is the same as the autocorrelation going to one. There is no internal frontier.
Statistically, the power of a unit-root test — the standard tool for asking “is this a random walk?” — collapses exactly as the truth approaches the random walk boundary. With a finite sample and a half-life comparable to the sample length, the test simply cannot tell. This is a well-known result in econometrics, and it is why decades of “is the real exchange rate stationary?” papers argued past one another.
Numerically, if the reversion speed is parameterized to be strictly positive (as it must be, for the sampler to behave), then “the probability that kappa is greater than zero” is trivially one — it tells you nothing. The informative quantity is the half-life, and the probability that the half-life exceeds some sensible horizon.
The package’s response to the trap is instructive. It does not pretend the trap is not there. It states all three layers explicitly, reports the slow tail honestly (one sector has a non-trivial posterior probability of a half-life beyond forty years), and argues that the joint hierarchical posterior — which pools information across the whole panel — is a more powerful discriminator than any univariate test. That is a defensible position, and it is stated with the caveat attached rather than buried in a footnote.
This is the broader lesson. Anyone working with time series that might be slowly mean-reverting — interest rates, real exchange rates, commodity prices, climate variables, pollutant concentrations — runs into exactly this trap. The package’s framing of it, in three layers, is one of the clearest expositions I have read, and it would travel well into any of those domains.
What I appreciate, and what I would watch for
A few things stand out as genuinely good practice, and they are worth naming because they are rarer than they should be.
The separation of economic and sampler convergence. The package is scrupulous about not confusing two senses of “convergence.” Economic convergence — does the price revert? — is a statement about kappa and the half-life. Sampler convergence — did the MCMC chains agree? — is a statement about R-hat and divergences. These share a word and nothing else, and conflating them is a classic source of muddled reasoning. The documentation keeps them lexically distinct throughout.
Neutral priors on the load-bearing hypotheses. The prior on the profit-rate modulation is centred at zero. The prior on the value coupling is centred at zero. The package does not bake the answer into the question. When the posterior then moves clearly away from zero, that means something.
Out-of-sample integrity by construction. A subtle and common error in time-series work is “leakage”: accidentally letting future information contaminate the training procedure, so that out-of-sample results are secretly in-sample. The package offers a fit_window switch that keeps the two designs genuinely separate, and it computes the common-factor loadings from the training window only. This is the kind of plumbing detail that separates trustworthy work from work that just looks trustworthy.
The negative results are reported. Many packages, and most blog posts about them, would quietly omit the fact that a random walk out-forecasts the model. This one leads with it and then reasons about it. That is how a field accumulates reliable knowledge rather than just encouraging headlines.
What should a careful reader watch for? The half-life estimate of about nine years is, by the package’s own account, probably conservatively slow — a controlled study of the disaggregation step suggests the true figure may be closer to seven or eight. The cubic nonlinearity is a minor refinement on this data (its coefficient sits near its prior). The Student-t degrees of freedom and the stochastic-volatility scale are weakly identified when both are present, a known tension the documentation flags but does not resolve. And the headline value-coupling result, while striking, is measured on standardized levels that share a cost-price component by construction; the package defends this with a “wedge” argument — subtracting the shared component and testing the residual — but a sceptical reader should follow that argument itself rather than take it on trust.
None of these caveats undermine the project. They are the project. A statistical framework that cannot articulate its own soft spots is not a framework you should believe.
Why it is worth your time
You do not need to be a Marxian economist, or any kind of economist, to get something out of this package. If you work with time series that exhibit slow, noisy reversion toward a moving target — and a great deal of the physical and social world does — the modelling ideas here are directly portable: the nonlinear OU drift, the stochastic volatility, the hierarchical pooling across groups, the careful separation of structural estimation from forecasting, and the three-layer diagnosis of the low-reversion trap.
And if you are interested in the classical question of whether prices gravitate toward values, this is about as good a statistical treatment as you will find: modern machinery, honest reporting, and a willingness to let the data argue back against the theory that motivated the exercise in the first place.
The repository, the full mathematical specification, the validation blocks, and a frank discussion of every methodological decision live at github.com/IsadoreNabi/bayesianOU, with the wiki carrying the complete technical detail. Read the methodology notes before you quote a number; that is what they are there for.
