TOKYO - Prime Minister Shinzo Abe grumbled Wednesday about his ruling party's second-highest lawmaker's controversial remarks calling people who opt not to have children "selfish."