Don’t get me wrong, if you have a good jam or preserves it’s totally fine on a sandwich, and I wouldn’t turn my nose up at it. However, peanut butter doesn’t really need the help, and the extra sweetness from jelly is totally unnecessary. Toasted bread and peanut butter is all you need for a quality simple sandwich.
I like peanut butter with a bit of agave. Something I think might differ between you and the people objecting to this is the type of peanut butter used. “Natural” peanut butter where you have to mix the peanut butter is better for plain sandwiches because the oil makes it less gunky imo.