Sushi as we know it is actually relatively modern. The Japanese made a few exceptions to vegetarianism mostly out of practicality. For example, birds were also not seen as animals.

Somewhat related to this, there was an emperor who died of beriberi because apparently all he ate was polished white rice.