Dataset shift detection in non-stationary environments using EWMA charts

Raza, H; Prasad, G; Li, Y

Abstract

Dataset shift is a major challenge in the non-stationary environments wherein the input data distribution may change over time. Detecting the dataset shift point in the time-series data, where the distribution of time-series changes its properties, is of utmost interest. Dataset shift exists in a broad range of real-world systems. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shift so as to decide about initiating adaptive corrections in a timely manner. This paper presents an algorithm to detect the shift-point in a non-stationary time-series data. The proposed method detects the shift-point based on an exponentially weighted moving average (EWMA) control chart for auto-correlated observations. This algorithm is suitable to be run in real-time and monitors the data to detect the dataset shift. Its performance is evaluated through experiments using synthetic and real-world datasets. Results show that all the dataset-shifts are detected without the delay.

Citation

Raza, H., Prasad, G., & Li, Y. (2013, October). Dataset shift detection in non-stationary environments using EWMA charts. Presented at Institute of Electrical and Electronics Engineers (IEEE) International Conference on Systems, Man, and Cybernetics, Manchester

• Presentation Conference Type: Other
• Conference Name: Institute of Electrical and Electronics Engineers (IEEE) International Conference on Systems, Man, and Cybernetics
• Conference Location: Manchester
• Start Date: Oct 13, 2013
• End Date: Oct 16, 2013
• Publication Date: Oct 13, 2013
• Deposit Date: Jul 27, 2015
• Publisher: Institute of Electrical and Electronics Engineers
• Publisher URL: http://dx.doi.org/10.1109/SMC.2013.537
• Related Public URLs: http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=6689802; http://www.smc2013.org/
• Additional Information: Event Type : Conference