Directed Energy Deposition (DED) has significant potential for rapidly manufacturing complex and multi-material parts. However, it is prone to internal defects, such as lack of fusion porosity and cracks, that may compromise the mechanical and microstructural properties, thereby, impacting the overall performance and reliability of manufactured components. This study focuses on in-situ monitoring and characterization of melt pools closely associated with internal defects like porosity, aiming to enhance defect detection and quality control in DED-printed parts. Traditional machine learning (ML) approaches for defect identification require extensive labeled datasets. However, in real-life manufacturing settings, labeling such large datasets accurately is often challenging and expensive, leading to a scarcity of labeled datasets. To overcome this challenge, our framework utilizes self-supervised learning using large amounts of unlabeled melt pool data on a state-of-the-art Vision Transformer-based Masked Autoencoder (MAE), yielding highly representative embeddings. The fine-tuned model is subsequently leveraged through transfer learning to train classifiers on a limited labeled dataset, effectively identifying melt pool anomalies associated with porosity. In this study, we employ two different classifiers to comprehensively compare and evaluate the effectiveness of our combined framework with the self-supervised model in melt pool characterization. The first classifier model is a Vision Transformer (ViT) classifier using the fine-tuned MAE Encoder’s parameters, while the second model utilizes the fine-tuned MAE Encoder to leverage its learned spatial features, combined with an MLP classifier head to perform the classification task. Our approach achieves overall accuracy ranging from 95.44% to 99.17% and an average F1 score exceeding 80%, with the ViT Classifier outperforming the MAE Encoder Classifier only by a small margin. This demonstrates the potential of our framework as a scalable and cost-effective solution for automated quality control in DED, effectively utilizing minimal labeled data to achieve accurate defect detection.