I assumed he'd be a backseat driver and tell you not to pass anyoneOh, no!
It’s bad enough if the guy is a fraud, which is most likely: He might whack you in the head, rob you, do unspeakable things to you. That’s bad. At best he’s a prankster, right?
What could be worse is that the guy really is Gandalf! There’s no telling where you’d end up, with whom, or what you’d be doing! And he has the habit of vanishing just when he would seem most useful: in a pinch! “No, really, officer, I just picked up this gentleman on the side of the road back there…” “What gentleman, sir?” “Ulp…!” “Sir, please get out of the car, and stay right here until some nice people arrive to take you to see a nice doctor.” Ooh! Bad.
This reminded me of something, i think the Dark Tree is not a real tree, wait, yeah there are branches and such but within it hides the dark rider of Cuiviénen. I believe this tree was at one point in Tol-in-Gaurhoth, on the actual island but Sauron did not know.