I too have them both. Really depends on the road surface to a large extent as to which one is noisier. The xB handles some better than the smart and vise versa. Overall I would say it is a toss up. Now my xB is a 2005 so it isn't the most recent heavier, more powerful version. As to which one drives better, if I was running around on the highway mostly, I would probably pick the xB, in the city the smart is hard to beat though. I freely admit I enjoy both of them, but it's fun to show off in the smart.
I guess my use of all my vehicles probably speaks to what I prefer.
Driving to work - I take the smart, wife drives the scion (she would drive the smart if she could)
Driving on weekends around the greater Austin / Round Rock area when we don't have our son with us - smart.
Driving further than an hour or so or with our son - usually our Honda Ridgeline.
If I didn't have the money and had one only have one of them, I guess I would have to take the Scion only because our son wouldn't care to be strapped to the roof.