Fear of a White Cat

Why is it that only white cat fur appears on my clothes? I can hang around a black or orange cat all day and my outfit will be unaffected. But if I wear a dark shirt while visiting a white cat, the shirt will be visibly covered with fur in a matter of seconds. 

Beware the fur.