I always found the Dreamcast to be notable for being the first console to have polished 3D graphics. I don't consider it part of the fifth generation because I believe those consoles went a generation too early for 3D gaming, at least to the degree their game developers did. The difference between your typical PSX game running at 15 FPS with claustrophobic draw distances and SoulCalibur (or any halfway-decent PC offering of the time) was night and day. You'll hear cynical, lazy narratives about piracy, but that kind of thing was always on the margins in the 90's. It was the rapidly-moving market that would be the problem for Sega in the end, as PS2 and Xbox represented yet another big step forward for nascent 3D technology.
The thing is, despite running up against the best-selling console ever made, the opportunity was still there for the Dreamcast. Sega bungled their Japan release but had a far better than expected showing in North America, led by a strong launch lineup and an untapped market filled by the 2K sports games. The Dreamcast is a great case study in the necessity of agile marketing; immediately pivoting towards a stronger Western footing after the successful 1999 launch would have put Sega in the position to capitalize on future success. The PS2 had supply issues and a thin library in its early years. Sega also had the foresight to put modems on their consoles, and Phantasy Star Online would go on to be one of the best selling games on the system. The US had better Internet infrastructure and adoption than Japan, and the lack of online service was the one weakness the PS2 had. Sega being positioned to compete with Xbox Live would have dramatically altered the market landscape. Instead, Sega only had one major online title in the end, but even that would come too late. When Shenmue flopped (due to major budget overruns), that was that. The Dreamcast library had peaked, and higher-ups at Sega were already moving to pull the plug.