Why did Crooks not want Lennie in his room at first?

In part 4 of Of Mice and Men, while most of the men go out to town to spend money George leaves behind Lennie, and Crooks stay in his room. Lennie tried to enter Crooks's room, but Crooks didn't want him to come in. Lennie tried to give him a little smile, but that didn't seem to change crooks's mind. "Crooks said sharply, 'You got no right to come in my room'" (68). Crooks didn't want Lennie to come in because he thought since Lennie was white and he was black he might be treated rudely like he does from everyone else. The other men excluded Crooks from certain activities like cards, then made up terrible excuses. "'Cause I'm black. They lay cards in there, but I can't play because I'm black. They say I stink'" (68). But after some time Crooks realized Lennie meant no harm. Also, Crooks didn't mind some company because only two people besides himself had ever stepped foot in that room.