I worked in appliances for about ten years, and not a single washer manufacturer would actually recommend using fabric softener. It horribly gums up the workings of the machine, even when you use the tiny amount you are actually supposed to (which most people use way too much). They are (or were originally) basically just animal fats and emulsifiers with some fragrance thrown in. They smell awful when they are left stuck somewhere for a long time (like the outer walls of the inner tub of your washing machine - seriously, it probably looks furry if you opened it up to see).

I can’t speak to what it does to your clothes specifically, but I can imagine several downsides to essentially coating fabric in lavender scented industrial mayonnaise.