Casual gamers only play certain games, like Call of Duty, sports games, Fortnite, etc.
Most casual focused games that sell big will not be on Game Pass or will be free to play (but not really on Xbox.)
Game Pass offers appeal to hardcore gamers who want to play lots of different games.
Why would a casual gamer even pay for games if they can play COD WZ, GTA online, Fortnite, Rocket League, Apex, etc all for free?
Or there are some casuals who just buy FIFA and play that all year until the next game comes out.
Because the casual gamers need to play those free games on a system. Electing for the more accessible system that also comes with a Netflix-style game service would probably be the more desired option.
There are many people I know who don't game, view it as something interesting, but aren't inclined to pay hundreds of dollars upfront to give it a shot. This is something that would sway them, especially since the setup and finding games on Game Pass is so easy. These are the same people who play crappy mobile games so it'd be a massive step up. It's also not even about not being able to afford the cost, it's just not something that's so imminently valued that they feel like it warrants the (current) upfront price tag.
Game Pass definitely appeals to hardcore gamers, I use it and I like it, but I don't think it's primarily for hardcore gamers. It's trying to be an all-in-one gaming service where you only need to engage with Game Pass, which many hardcore gamers frankly won't settle with.
I just pay my Game Pass subscription for 1 month to play anything I'm interested in then don't renew, while most of my gaming comes from paying full price for the games I want on PS4/Switch/PC. I want to play those games on the best possible hardware, not stream them on any device (the future XCloud integration).
So Project XCloud and Game Pass aren't primarily targeting me, I'd think they're primarily targeting the value-focused gamer that just wants to play anything decent for a cheap price wherever they feel like. It's like comparing the typical Spotify music listener vs an enthusiast with Hi-Fi gear and FLAC music.