Because of its cultural and spiritual importance to the Elves, the Grey Havens in time became the primary Elven settlement west of the Misty Mountains prior to the establishment of Eregion and, later, Rivendell. Even after the death of Gil-galad and as the Elves dwindled in numbers by the year, the Grey Havens remained a focal point of the history in the northern part of Middle-earth.
After the defeat of Sauron and the coronation of Aragorn II Elessar, Gandalf, Galadriel, and Elrond left for Valinor from the Grey Havens. The Ring-bearers Bilbo Baggins and Frodo Baggins accompanied them to Valinor. Years later, Samwise Gamgee did likewise, himself being a Ring-bearer (although briefly).
The Grey Havens still existed into the Fourth Age under Círdan but its population declined as did the elven presence in Middle-earth. It is unclear just what the Fate of the Elves of Middle-earth was in the early Fourth Age and how long Círdan or his remaining folk dwelled at the Havens and continued to build the great ships that carried the Elves to the Blessed Realm.