Complex data analytics using functional and adaptive approach

11 Jan 2017. NUS statisticians have developed adaptive functional time series models that improve the forecast accuracy of complex data such as term structure of interest rates, limit order book liquidity and electricity price curves.

The availability of economic and financial data in modern times provides a huge opportunity to understand, investigate and answer scientifically interesting yet practical issues in related areas. It also presents unique computational and statistical challenges, given the massive sample size, high dimensionality and non-stationary serial cross-dependence of the big data. A research team led by Prof CHEN Ying from the Department of Statistics & Applied Probability, NUS together with her collaborators from Humboldt-Universität zu Berlin, Xiamen University and University of North Carolina have developed adaptive functional time series models to describe the dynamic behaviours of the complex data.

The adaptive approach which they have developed is not only able to automatically detect structure changes in the dataset but also improve the forecast accuracy over time. When this model was used to analyse the term structure of interest rates, the identified dynamics over time were found to coincide with policy changes and the onset of the 2008 financial crisis. Comparing the new approach with models which are used in the financial sector, the team found that their model achieved a much higher prediction accuracy of between 25 and 60 percent when forecasting interest rates change for three months to one year ahead.

The new approach has also been successfully applied to the energy market to predictelectricity price curves due to seasonal variations and in the financial market for limit order book liquidity. Their work proposed a novel predictive model for curves that simultaneously accounts for cross time-dependence and seasonal variations of large dimensional data. This improved the predictability substantially. It leads to a 35 to 104 percent error reduction relative to alternative predictive models in use for California electricity price curves. For 12 stocks which are traded on NASDAQ, it explains 98.2 percent of the out-of-sample forecasts and produces accurate 5-, 25- and 50-minute forecasts with an accuracy of up to 95.5 percent.

The research team plans to further develop adaptive functional data analysis methods and the underlying theories behind them for analysing complex big data. This is expected to have wide-ranging applications in neuroeconomics, economics, financial services and other diverse areas.

Figure shows the 10-step ahead (50-minute forward prediction) forecast of liquidity demand (blue) and supply (red) curves against the actual values (dots) and the smoothed curves for stock counter Apple Inc (AAPL) traded on 24 February 2015 at 1500 hrs.

References

Chen Y; Li B, “An Adaptive Functional Autoregressive Forecast Model to Predict Electricity Price Curves, JOURNAL OF BUSINESS AND ECONOMIC STATISTICS, accepted, DOI:10.1080/07350015.2015.1092976, 2015

Chen Y; Niu LL*, “Adaptive dynamic Nelson-Siegel term structure model with applications” JOURNAL OF ECONOMETRICS Volume: 180 Issue: 1 Pages: 98-115 DOI: 10.1016/j.jeconom.2014.02.009 Published: 2014

Chen Y; Chua W*; Härdle WK, “Forecasting Limit Order Book Liquidity Supply-Demand Curves with Functional AutoRegressive Dynamics”, working paper, 2016

Chen Y; Marron SJ; Zhang J, “Modeling Seasonality and Serial Dependence of Electricity Price Curves with Warping Functional Autoregressive Dynamics”, working paper, 2016