Manufacturing processes undergo continuous changes to meet various requirements, such as process/product changes and variations in tool/workpiece conditions, leading to mixed, heterogenous, or anomalous data. As a result, a quality prediction model trained from previous data may not perform well when new tasks emerge. To achieve in-time and accurate product quality prediction, it is crucial to develop a predictive method that adapts to variations in the manufacturing system, capable of learning from new tasks without forgetting previous ones and detecting unknown tasks. This study proposes a deep learning method integrated with continual learning for in-situ quality prediction that is capable of learning from new tasks without forgetting previous ones. To demonstrate this idea, deep convolutional neural networks (CNNs) are designed to analyze in-process sensor data, which consist of shared layers to capture the common underlying features across all tasks, and task-specific layers that capture specific characteristics of each individual task. To identify the task to which the incoming product belongs, a task prediction approach based on task relevancy using filter subspace distance is proposed. When new data come in, the model first identifies the task, followed by predicting the quality of the current product. The proposed method is demonstrated in two case studies, including quality prediction of the workpiece using acoustic emissions during the laser-induced plasma micromachining process and quality prediction of the product through thermal images during the hot stamping process.