I can almost guarantee you that it isn't mold. It takes a lot for plastic, even textured or matte plastics, to allow mold to grow outside in the elements. I have the same issue on my Smart's door handle and window trim.
I'm 90% sure that it's simply discoloration of the plastics. My Saturn had black plastic on all 4 doors along the B-Pillar, and on all 4 trim pieces, the glossy coating was wearing off, leaving the matte unfinished plastic underneath exposed.
Take a look at cars like the Pontiac Aztec or Honda Element that are covered in black plastic. Even the older Saturns or Jeeps with black bumpers. They either turn all sorts of lighter grey, or they expose swirls and other imperfections in the plastic. Some restorer for black plastic trim is probably all you (we) need.