Genesis 37:11 And his brothers envied him, but his father kept the matter in mind.
Joseph had the special love of his father, largely due to being the son of Jacob's deceased favorite wife. Could his brothers have done anything "right" about it? Later, some of them did excel, and were praised by Jacob on his deathbed.
Hatred was the wrong response; serving God diligently was the right response.