Baby carrier or sling - this allows you to carry your baby and keep your hands free, making activities such as using public transport easier.
Baby bath - your baby can use the big bath, share with you, or you could use a new washing-up bowl at the very beginning.
Toys - although not really necessary at first, they're nice for you and your baby to enjoy together.
Changing gear - you might want a changing mat or baby dresser with a changing top.

How to cut the cost
Buy second-hand baby gear. Look in your local paper, shop windows, charity shops. Family and friends may offer you hand-me-downs.
Keep in mind that some items are only guaranteed from new for a certain length of time, so there may be safety implications if they're second-hand.
Check equipment for stability and safety, and only buy car seats if they're clearly unused - if they've been involved in an accident you may not be able to tell, but their fixtures could be unsafe and the inside cracked.