Whats the hardest instrument overall?

It depends on the person playing them.

I play saxophone (alto, tenor & saprano), flute, trumpet, guitar, piano, various Tibetan drums, and bassoon (and cymbals in marching band back in the day). Of those, I'd say bassoon presented me with the most challenges. It's difficult to play bassoon without sounding like a goose. The double reed and the double embouchure you need to play the double reed... well, balancing them is like a graceful dance.

Of the ones I tried and had no success with in any small measure, I'd voice, oboe and trombone are the hardest.

Oh, I forgot djembe and conga drums were really hard for me too.

Oooo, violin... anything stringed. I can't play any string instruments. I don't "understand" them.

I agree with SkyDancer in relation to stringed instruments I find them extremely difficult to comprehend.
She has a point though about the person playing the instruments, like me for example I am musically talented naturally. I play the clarinet, piano, the saxophone, trombone and trumpet and also I sing (vocals are categorised as instruments). I am very good at all of these things as I feel I was born with a musical talent, however someone who is not naturally good at music may find it hard with several instruments untill they find the perfect one. I mentioned that I cannot understand guitar well my friend tried to play piano, he tried to sing and play harp but he couldn't and was considering giving up on the idea of playing an instrument until he found guitar!
CONCLUSION: Different for everyone xx

In my view instruments than can make only one tone at a time are much easier to play than instruments capable of making a number of tones at once. Compare the sheet music for a trumpet part to a piano or guitar and you will see the difference. You have a bass clef doing one thing and a trebel clef doing something entirely different with 4-5 notes on each. Try to play that on a trumpet. :-)