We could view this question from the perspective of one side that considers itself to be in the right: we are only fighting because someone is fighting against us (or doing something else terrible), and we will take what we think are reasonable measures to defend ourselves (or a third party), but we won't fight dirty because we're the good guys. It's possible for both sides to think that they are in the right, but if one side is not trying to maintain that pretense, why would that side agree to any kind of rules? Sometimes they don't, and when they do abide by the rules, perhaps they do so to avoid reprisal once the armed conflict is over. The bad guys might be perfectly willing to commit atrocities in order to win but want to avoid being convicted of war crimes in the event of a loss.
Which raises the question of whether it is easier, or possible, to get away with war crimes for the side that wins the war. In any case, if we have any ability to punish war crimes, couldn't we do the same for any kind of killing? That is, reckon that killing for reasons other than self-defense, or defense of another, is a crime?
In short, what does it even mean to draw a line between war crimes and non-criminal acts of war? I feel I must be missing something.