I agree that mood swings do not amount to personality changes. However, as you change moods, you express yourself differently. Anger makes people speak tersely and forcefully, with less concern about tact, while amusement makes people choose more loopy and elliptical language.
If you are in a particular mood long enough, you may get in the habit of expressing yourself that way. It is not that the music brainwashes you; more like it creates an environment where you practice certain behaviors, both inward and outward. Do this long enough, and eventually, long after you have stopped listening, you retain your instinct to act that way. In my view, these acquired habits can add up to a personality change.
Arac, you said, “I don’t think listening to more grindcore is going to make me more abrasive/aggressive or whatever.” True enough, abrasive music does not necessarily make an abrasive person. The question is what mood grindcore does put you in, and how you act when you are in that mood, and–assuming you listen to grindcore often enough–what habits you pick up in the process.