Niko Bellic has a fighting style that takes from many different martial arts. The mix is similar to MMA (mixed martial arts) fighting. While we don't see a ground fighting ability in game, Niko has a set of strikes and kicks while standing.
Niko uses boxing strikes for most of his upper body work. Boxing offers powerful strikes with the fist and a good fighting stance. Left hand jabs, right crosses, left hooks, and right uppercuts make up the vast majority of the hand strikes in game.
Sometimes in game you can spot Niko throwing elbow strikes, some kicks are also just knee kicks. Muay Thai is unlike traditional boxing. In traditional boxing you never hit below the belt and never hit with elbows or legs. Thai boxing has elbow strikes that can leave cuts on the face, and allows for some extremely powerful kicks.
Some of Nikos strikes and kicks (especially in finishers) are over the top and fancy. While these are not necessarily impossible, they would most likely be impractical in a street fight (it looks awesome in game though).