It never hurts to go out there for a visit! If you take the magnet train in Goldenrod, I believe you wind up in Saffron, and Lavender Town isn't too far from there according to the map!