It's a good point to keep in mind and certainly possible.
However, we do know ESPN TV deal renegotiations get triggered if the ACC dips below 15 teams. The B1G will already have 18 teams beginning in 2024, and if FSU and Clemson leave the ACC, they'll either go to the B1G - potentially increasing its membership to 20 - or to the SEC, potentially increasing its membership to 18. These increases could be higher if additional teams (e.g. UNC, UM, Virginia) leave the ACC.
The Big 12 (presently at 16 in 2024) is also rumored to be eyeing Wazzu and OSU, in part to resolve the litigation with departing members, including some who are going to the Big 12 and stand to lose significant basketball and other revenue if the PAC continues to exist and those teams don't join. So the Big 12 could also go to 18 very soon.
Point being, it looks like the B1G, SEC, and Big 12 will soon be getting bigger one way or another (to 18+ members). And as noted the ACC can't risk falling below 15 teams. I think the speculation (expectation?) is that the ACC is actually going to lose more than just FSU, Clemson, and UNC, creating at the same both larger competitor conferences and possibly more spots to backfill within the ACC. But even if the ACC stays at 15, it would seem rather foolish to just stay at that number if your goal is to survive long-term (not that that would stop them from doing it).