Keeping horses can be very rewarding, and you'll have your favorite trail buddy right there, ready to go whenever you are. It can also be very expensive, since the general rule is 1 acre per fullsize horse, and you'll need grain/oats, salt licks, hay, and horses love a little alfalfa every now and again - it's like candy to a horse.
And that's just food, you'll also need tack, and depending on what discipline and maker you choose it can be anywhere from kind of expensive, to holy-crap-that's-expensive expensive. Whether you pick the English discipline or the Western discipline, there's basic tack that both require, like halters, lead ropes, lunge lines, fly masks - if you're new to horses and don't know what all this jargon means, have no fear, I'll explain it all in a bit.