Data Science Distinguished Lecture: Agostino Capponi - Industrial Engineering and Operations Research, Columbia University

About

This talk is co-sponsored with the ORIE Colloquium and the Center for Data Science for Enterprise & Society.

TALK: Data-Driven Learning of Latent Dynamics

Abstract: High-dimensional time series often exhibit complex dependencies that can be explained by a few latent factors. Uncovering these factors and modeling their dynamics is a fundamental challenge in data science, with applications ranging from signal processing and forecasting to asset pricing and financial risk management.

We introduce a data-driven framework for modeling and predicting a multivariate response y(t) from a high-dimensional covariate process x(t), allowing for nonlinear dependencies without imposing a parametric model on the covariate dynamics. We leverage diffusion maps to uncover low-dimensional representations of the covariate process that retain explanatory power for the response time series (y(t)). We establish concentration inequalities that provide convergence guarantees for time averages and show that, under mild assumptions, the resulting diffusion map embeddings exhibit approximately linear dynamics. This property enables us to develop a geometric Kalman filter that operates in the learned embedding space to estimate latent states and predict the response variable.

We demonstrate the effectiveness of our framework in financial stress testing. Using historical data spanning three major financial crises, we show that our data-driven approach reduces the mean absolute error of scenario-based portfolio return predictions by up to 52% and 57%, relative to standard scenario analysis and dynamic principal component analysis, respectively.

Beyond financial stress testing, we highlight the broader applicability of our framework to problems such as nowcasting, inverse problems, and data-driven portfolio selection.

Bio: Agostino Capponi is a Professor in the Department of Industrial Engineering and Operations Research at Columbia University, where he also holds a courtesy appointment at Columbia Business School. He is a member of Columbia's Data Science Institute and the founding Director of the Center for Digital Finance and Technologies.

His research lies at the intersection of financial engineering, artificial intelligence, and economics, with particular interests in financial technology, machine learning in finance, market microstructure, and economic networks. His research has received funding from major government agencies, including the NFS, DARPA, the Department of Energy (DOE), as well as leading industry organizations such as IBM, JPMorgan, Ripple, and the Ethereum Foundation. His contributions have been recognized with numerous honors, including the NSF CAREER Award, the JPMorgan AI Research Faculty Award, the University Blockchain Research Innovator Award, and the Presidential Early Career Award for Scientists and Engineers (PECASE). His research has also received media coverage from Bloomberg, the Financial Times, Vox, and Politico.