How to deal with what people think about me and my attractive guy friend?

Some people may say that I am feminine and attractive myself while most people think that my guy friend is very masculine and attractive. The funny thing is that I am a feminist and he is kind of like a feminist man. Our friendship is based on our love for poetry, books, movies and we have the same religious background. I think people think that I am just trying to flirt with him when I really just enjoy his intellect and companionship. The ironic thing is that he is more flirty with me than I am with him. It pisses me off (as I am a feminist) that people think I just like him because he is hot and ume that is why I spend so much time with him. I am actually more attracted to artsy waifier guys and he is more muscular. He isn't really physically my type. I have even wanted to stop hanging around him so much, just to prove a point but I don't think this is fair to him and our relationship. How should I handle this?