Fabrics were widely used in clothing, home textiles, industrial manufacturing and other fields. Their surface quality directly affected the product use value, and defect detection thus became a key research direction in textile quality control. However, annotated fabric samples are often difficult to obtain, and fabric products from different production sources usually exhibit significant domain discrepancies. To address the domain shift among samples and improve detection accuracy under few-shot conditions, a multi-source Fisher embedding transfer dictionary learning algorithm (MSFTDL) was proposed. The proposed method was built on dictionary learning and transfer learning. It constructed a shared dictionary from multi-domain samples and introduced a dual Fisher discriminative embedding term during training to enhance intra-class compactness and inter-class separability. A multi-source domain projection and alignment term was further incorporated to reduce the distribution discrepancy between source domains and the target domain, thereby enabling effective cross-domain feature transfer. Experimental results on cross-texture fabric defect detection within the TILDA dataset and cross-dataset fabric defect detection from TILDA to AITEX show that MSFTDL significantly outperforms the comparison methods across multiple evaluation metrics.