Is remote work really that great? Afraid of what it might cost me

I hate going into the office. The commute is one thing but I have a kid now and would really like to be home to see her and my wife.

The thing is, my current position is very secure (cleared position in the defense sector), has great company culture, time off policy is good, and my boss is very flexible and honestly great to work for. The company has made some effort to allow remote work but for government enforced reasons basically everything HAS to be done onsite. It's out of their hands. The only real gripes I have is that career progression is slow and I don't feel like the pay is that good (currently making 82k a year).

Am I crazy to consider looking for a new job to not have to go into the office? Will I make more anywhere else or am I delusional?