Words starting with ‘ph’ are Greek in origin, and they all contained the Greek letter φ (phi). In Classical Greek this was pronounced as an aspirated [pʰ], which the Latins wrote as ph when they borrowed the words from Greek. Later this sound changed into an [f] in both Greek and Latin, and was passed as such into French, and then into English.
Once the idea that ph was pronounced [f] was established, it spread to a few other areas, as well. Borrowings from Hebrew and other Semitic languages sometimes use ph, especially since the Hebrew letter פ can be [p] or [f] depending on context. Vietnamese regularly uses ph for [f], in this case because the modern Vietnamese orthography was designed by the French.