Pour off some of the oil (not all, or the bottom of the jar gets really dry), stir, then store it upside down, so the oil rises to the 'top'(which will be the bottom, in this case).
And make sure to always read the ingredients of your 'natural' p.b. Some compainies (Skippy? or is it Jif?) are labeling p.b. as natural when it contains sugar and added oils.