Design an approach to adapt your pre-trained CNN feature extractor to selectively encode only the factors that are relevant for each specific task or environment. Explain how this would improve system performance.

Implement a novel semi-supervised learning module that leverages attention mechanisms to dynamically focus on task-relevant features while ignoring irrelevant ones. This can be achieved by encoding task-specific priors and constraints into the network architecture, allowing it to selectively disentangle relevant factors from complex dependencies between object properties.