loving him is all that i want to do

i broke up with my fiance two years ago.  through all the drama that we had 4 years of it(in & out of jail, cheating, etc.)  we still call each other about once a month just to say hello because we were friends first!  we both know that it is over.  he has a child on the way from someone that he is dating.  well, he has a close friend that he has known forever.  i always thought his friend was a nice person, sexy as hell, etc... over the last year me and him have been hanging around some mutual friends of ours...we were drinking and he told me that he always liked me and that his friend was a fool for dogging me!!!  I always thought i should have dated him to begin with, but you know how s_it happens.  anyway 2 months later we ran into each other, exchanged numbers and have been hanging out together and everything.   we were starting to dig each other.  we had sex it was very satisfying, but  we both decided to stop calling each other because we didn't want to start something that we weren't ready to finish and the guilt of the relationship. i had missed him so much during the two months that we were away from each other. we seen each other in passing.   three weeks ago he called me and we have been together every since...late night conversations, making love, romantic times and going dancing together.  today i called my new boo and my old boo answered his cell phone!! my old boo is really hurt!! he moved on!! shouldn't I?   Give me some advice...to handle this situation...some of our family and friends are giving us the blues, but we are holding it down !!!