There are two ways to go here, new and used. For the beginner, you should just make life easy on yourself and grab a new razor. If you find yourself beginning something of a collection (this is, after-all, a hobby as well as a daily ritual), you'll find yourself trolling eBay at 2am looking for that 1940's brass Schick Injector. Trust me, it's in your best interest to delay that crazy obsession for as long as possible.
I've never tried one personally, but Merkur has an excellent reputation, and it seems like as good a place as any to start.
Their most popular model – the Merkur Heavy Duty – is a little more than thrice the price of a new Fusion, but its stainless steel construction will outlast its cheap plastic counter-part (which is only good until Gillette's next miracle breakthrough anyway), and remember that we're saving money mostly on the blades, not the razor itself. Actual Merkur blades are about $4.50/10 pack, but as always, I'd recommend finding Personnas locally and saving on the shipping costs.
Or you can start scouring the auction sites, by which I mean eBay.
The older Gillettes, the ones made in the 40's and 50's (and to a slightly lesser extent, the 60's), are pretty kick-ass. I've been using a sixties "bell-bottom" as my #1 for about six months now. Many knock it… I think it's awesome.
I can't believe I got it thrown in for free.
I found a Gem Micromatic Single Blade on eBay for about $10, and when I decided I was less than ecstatic with the shave, I boiled the nasty-looking #2 razor that was chucked into the box as an afterthought. It was rusted, caked with 40 years of nasty, but damned if it didn't come out looking shiny and new.
It was like Tim Robbins crawling through 5 football fields of foul-smelling shit to come out clean on the other side. Anyway, it shaves like a dream with the el-cheapo blades. It immdediately made its way into the rotation after 6 shutout innings, and it's the staff ace now.