At this point in his life, Eminem was experiencing a lot of frustration: His (first) marriage to Kim was failing, he was getting in trouble with the law over an argument with a man she was allegedly cheating on him with, and he was frequently criticized by the media, yet still admired by lots of fans but not getting any privacy.
He's mostly expression that he never thought fame would be like this and that he feels it's ruining his life. He was seriously contemplating leaving his fame (Hollywood) behind to try to give his daughter a normal life. But unfortunately, you can't just shut fame off.