My boss and I have a thing

I’m gay, he’s straight. He’s married. We’ve been working together for years and have grown our company incredibly well. Millions in revenue to billions, 200 employees to 5000 in 5 years. I value our business relationship so much because of that success. I’m 32 and a pretty normal American guy, he’s 50-something and Middle Eastern. I never thought of him sexually at all. Then something changed. We started doing stuff. And I’m very torn. I like it, obviously, or else I wouldn’t be doing it, but I also feel bad for his wife and kids, and just weird about it in general. It’s hard to explain but when you have someone in your mind in a certain role, which was like mentor and almost father figure, and then that changes, it’s hard to reconcile. We don’t talk when it happens, it’s just sex. And then the next day we’ll be in a meeting and it’s all business. I think that maybe we should stop for the good of our relationship and the company and his family and everything, but because we don’t talk about it at all, I don’t know how to even start to approach that. I need to figure this out. What do you all think?