@ Lee - As soon as the illegal substitution has occurred (and every pitch thereafter), a protest may be made. Even if a protest is not made, the error needs to be corrected as soon as it is discovered. However, everyone involved (managers, coaches, umpires, scorekeeper, Tournament Director), should try to prevent an illegal substitution before it occurs, however.
If a pitch has been thrown, and then the error is discovered, that pitch counts. If it's discovered several plays later, all those plays count as well.
To correct it, a third substitute (let's say that's Player C) must take the place of the ineligible substitute (Player B).
That's because the starter (Player A) cannot re-enter until his/her substitute (Player B in your scenario) has completed mandatory play.
In the regular season, since Player B has not completed mandatory play in your scenario, BOTH of those players cannot re-enter for the remainder of the game. In Player B's case, that's because subs cannot re-enter. (Exception: If one of the legal players in the lineup has to leave the lineup for some reason, such as injury, the opposing manager chooses who, among the ineligible players, will re-enter.)
So the "penalty" during the regular season is actually the penalty levied against the manager for failure to meet mandatory play requirements.
In Tournament play, Player B can re-enter once Player C has completed mandatory play (since subs can re-enter in tournament) - but only in the same spot in the batting order. Once Player B has completed mandatory play, Player A (the starter) can go back into the game - in the same spot in the batting order.
Of course, depending on the inning in which the error is discovered, the manager may not be able to meet mandatory play. So the penalty is the manager's penalty for failure to meet mandatory play.
However, if it is discovered that the manager knowingly and willingly violated the rule, a more severe penalty may be levied by the Board of Directors (regular season) or International Tournament Committee (Tournament).