I think the biggest issue is cost, and more specifically, that the sport has a large barrier to entry in terms of costs.
Once you've taken up the sport, there's many ways to reduce costs, as well as things that are one-time/rare costs or things you don't need anymore. Buying your own equipment saves money, there's all sorts of programs out there that reduce ticket costs, you don't need lessons, once you HAVE clothing/gear (not including skis/boots) it's not a large expense yearly, etc.
I don't pay all that much for my skiing. Skis + boots last 100+ days. Clothing and other equipment last a long time. Frequent skier programs, pre-purchase programs, or just season passes, mean that most of the time I'm skiing for 50% or much less of the walk-up ticket price. It's quite affordable to me.
But going from non-skier to skier, is a lot of money. Equipment rentals, lessons, walk-up lift ticket rates, goggles, and most people don't have remotely decent ski pants, jackets, gloves, etc. Your first days in this sport are a lot of money. You COULD avoid spending that money with a bunch of work finding ways to make it cheaper, but the reality is most people don't start activities with a burning desire. They go try them out, THEN might get hooked and really interested. If the cost seems high they won't care enough to find a cheaper way, they just won't try it.
I'm not sure how to make that barrier to entry easy either, short of encouraging clubs and such where sharing gear or hand me downs are more accessible.
