I worked in appliances for about ten years, and not a single washer manufacturer would actually recommend using fabric softener. It horribly gums up the workings of the machine, even when you use the tiny amount you are actually supposed to (which most people use way too much). They are (or were originally) basically just animal fats and emulsifiers with some fragrance thrown in. They smell awful when they are left stuck somewhere for a long time (like the outer walls of the inner tub of your washing machine - seriously, it probably looks furry if you opened it up to see).
I can't speak to what it does to your clothes specifically, but I can imagine several downsides to essentially coating fabric in lavender scented industrial mayonnaise.
I'm happy buying detergent honestly - it last a LONG time when you actually use the correct amount per load. I think the real crime is the "measuring caps" on liquid detergent basically tricking everyone into using WAY too much detergent. Most washers will recommend 1-2 tablespoons of detergent maximum for heavily soiled loads.. Most measuring caps are over that even at the first of several marks, and people rarely think they need the minimum (moar soap moar clean, right?) - so people tend to add 5-10 times the detergent they need.