That's what bumpers are for? Kidding.
I was joking, but it is an interesting idea. Those GPS speed limits are often not correct and less correct (or absent) the further you get from major centres. For example where I am, it states the limit is 90kph everywhere, which pretty fast to be blowing through a 30kph school zone! And once I'm on the highway, it says 0kph in some places and 140kph in others (neither are correct). That's using a TomTom.
But that aside, it's an excellent point. That information certainly could be ported to the cruise (not easily, but possible) and then you'd even have information such as turns. But as you said, it can't rely only on that. So now we need a Lidar or FLR (forward looking radar) system to detect the cars ahead of us.
There is work on this field already. They call it "adaptive cruise control" and some extremely expensive cars already have it. They also control the brakes. I don't think they read speed signs though

That was my idea!
- Steven