Horse Face is an enemy that guards the entrance to King Yama's lair, alongside Ox Face. He is functionally identical to Ox Face except for appearance and spawn point.
He will spawn shortly after entering 5-4 to the left of the Spelunker, about halfway to the edge of your screen (assuming you haven't moved).
It is highly recommended to kill him as quickly as possible, as he will soon start dropping bombs on the floor to take out the thin layer of blocks keeping you out of the lava. Shooting him thrice with a shotgun or one sticky bomb will kill him, although the latter could destroy the floor below him and cause the Bomb Box he drops to fall into the lava. Stomping Horse Head is futile unless the Spike Shoes are worn.
If kept alive, Horse Head will slowly walk towards the Spelunker and will as mentioned occasionally drop bombs. Furthermore, if the Spelunker is hanging onto a rope above, or if there is a gap in the floor, Horse Head may jump towards the Spelunker. He can jump about the height of a rope.
Trivia
Horse Head, together with Ox Face, comes from Buddhaism and Hindu mythology. However, their names are partially swapped: the two lieutenants of Yama were originally named Horse Face and Ox Head.