Feelings for someone I am having casual with :(?

For the first time I am having casual . Very confused about how it works, but I am 29 and still single and to be honest I was getting lonely. I met him online, he invited me to his place, he has very young kids that live with him. He is recently single. We both discussed that night how we didnt want to be in a relationship, cuz I jus had got out of relationship too. I ocionally go over to his house to spend time with him. He has never asked me on a real date. I noticed that he rarely calls and when we talk he doesnt really listen cuz he asks me a question about something I have already told him Inspite of alll that one thing led to another and we began an intimate relationship. But things havent changed he still seldom calls nor text. Last night we had , and this morning when we woke up he got a call and scrambled to pick up the phone. He answered to late but immediatly called her back. I could hear the excitement in his voice and hers. And he told her he would call her on the way to work. I at that point felt a feeling like I have never felt before. I actually thought he liked me by the things he says when i am there with him. To make things worst I didnt say n e thing about how I felt. He made a couple comments aout getting his kids up, and I smiled and pretended to be ok and he rushed me out the front door. We hugged and said GB. I didnt hear from him all day so I text him he text something small back, and then I havent heard from him since. What in the hell am I doing? How did this haappen? And i am certain he will call tomorrow and say he was sooo tired from our big night, but yall I dont think I can do this. Somewhere in the middle of this all I got feelings for him, and I am so hurt he doesnt even respect me enough to not take a call while I am there. What do I do? Do I stop sleeping with him? and If i do do I call and tell him, or just let it go and ignore his calls like he frequently does mine! am I overreacting? How do you just let go of someone you had with? Am I the only one this has happened too?