My aviation and flying interests are fairly unique among my coworkers, friends and peers, and I don't really have anyone to chat with about such things; oh sure, they all go "Wow, that sounds cool!" But then when I start talking the more down-to-earth side of things, they all tend to look at me funny and say "That sounds overly complicated and expensive. Why would you want to do that?"
You're not alone -- most of us are (or have been ) in the same boat. Few people understand (or much care) about details of aviation beyond touting the virtues (or lack thereof) certain airlines or experiences, having a wrong concept of how airplanes fly, terminology and many other things -- it's a foreign language to them, just like golf is a foreign language to me.

I do have a model railroading addiction/hobby, which can get awfully expensive in a hurry, but I (try to, mostly successfully) keep those expenditures in check.
Me too. Curiously, I find that model railroading, ham radio, motorcycling and a couple of other things are quite common among pilots.

One suggestion, Jason -- one can learn a LOT from hangar flying (being careful to not believe certain tall tales, and to take along your "grains of salt" shaker), and one can often relieve a certain amount of the flying itch with hangar flying (temporarily), so if things work out so that you can, you might spend a Saturday or Sunday morning or afternoon (usually need at least somewhat warm weather, depending on the locale) hanging around the airport just to chat with people, perhaps stroll around the hangars or sit on a bench near the flight school, or some other opportunity dependent on the layout and customs at your particular airport. You'll make friends, learn a lot and maybe even get some opportunities that otherwise would never show up for you.

I specify the weekend for the above mainly because that's when you're the most likely to find others there, but mid-week occasionally has pleasant surprises, too.