No matter who you are rooting for, the refs have simply too much influence in the game of handball. The rules are too loose and way too open to subjective interpretation. Just look at this sub and compare the amount of discussion about the (bad) refereeing vs. discussion about the actual games and the sport itself.
I know lots of people who find the sport interesting, but don't follow it because they are frustrated about how much the refs can swing the games. It is probably one of the biggest reasons holding handball back from becoming really popular (worldwide).
This has been a well exposed issue for quite a while now, so why don't the old farts in position do something about it? At the end of the day, increased popularity would also mean increased income for them (not that this is the goal, but still it should motivate them).
P.S. As a Slovenian, please don't think all of us are as biased, short sighted and rude as some of the comments here lately made it seem. We have a well known "bandwagon fans" problem in all sports, not just handball, so it's annoying for us as well.