Simple solutions - or solutions that appear simple seem to work best.
For at home play, I quite like the original XBox console - able to run an near A500 speeds, can play Megadrive / CD32 / Neo-Geo SNES and a little N64 with very little effort.
On the Go, well today if your tablet cannot do it, then your mobile phone can - however I love the first edition PSP-2000 - once put on to custom firmware its truly an amazing little hand-held.... with some great emulators - in fact I think it has the best version of FUSE (ZX Spectrum) I have ever seen! with flawless rendition of MegaDrive / 32x and Mega CD, SNES is pretty tasty too.... but my biggest love is CPS1, 2 and the Neo Geo..... ahhh its delightful - you can even set up an adhoc network with your buddies and play 2 player CPS1 / 2 and Neo Geo games!!!
SNES is great and theres so much more with PSP and PS1 games on the system.... I now have 2x 32GB Cards!
Of course if you want it ALL IN then there is the humble PC, you don't need much, a cheap re-serviced ex-office computer will suffice - something like a DELL optiplex USFF 745 is perfect when hooked up to a large screen - stick on XBMC as well and you are truly in for a delight that would likely cost you less than a second hand PS4 game =)
Being a PC, there are a lot of adaptors for USB to joystick / pad and any other input device for that matter =D