Well Wikipedia says:

Tea is an aromatic beverage prepared by pouring hot or boiling water over cured or fresh leaves of Camellia sinensis, an evergreen shrub native to East Asia which probably originated in the borderlands of south-western China and northern Myanmar.[3][4][5] Tea is also made, but rarely, from the leaves of Camellia taliensis.

As in, to be considered "tea", it has to be made from the tea plant.

However, it goes on to say:

The term herbal tea refers to drinks not made from Camellia sinensis. They are the infusions of fruit, leaves, or other plant parts, such as steeps of rosehip, chamomile, or rooibos.

I think you know where I'm going with this. Coffee is made from the seeds of the coffee plant (technically the coffea plant). Seeds are "other plant parts". Coffee might not be "tea" but it very clearly is "herbal tea" 😀