Silk? Most mitters are traditional cloth or microfiber. Assuming you have one of those materials, the buildup you are seeing is either dirt or algae. If it is dirt, you need more or better soap/detergent lubrication on the car and directed at the mitter. You also need to make sure you are flushing the mitter with enough water to rinse away soil. If it is algae, that can be remedied by a lower pH shampoo/detergent. There is also the possibility that you have a low grade detergent that is leaving deposits on your cloth. What pH is the detergent you are using with the mitter?
Is your mitter completely and continuously flushed with a soap solution while in contact with the vehicle? By a sprayer and wash it down with straight Clorox (Usually highest active ingredient) Let sit for 10 minutes - not longer and rinse. If buildup goes away it was an algae buildup. You may need a periodic application of Bleach. Perhaps a tablet or 2 in the solution tank.