Can I use Xbox Reward Points to buy physical stuff?

Yes, you can. Xbox gift cards and Windows gift cards all add to your Microsoft account balance which can be used in the Microsoft Store.

If you pre-order from the Microsoft store, it will be digital. You can pre-order a physical copy from participating retailers. However, the Microsoft store is more advantageous in my opinion. You can take advantage the Microsoft rewards program when you
buy from the store. Usually a $60 game, rewards you with 5,000 Microsoft points, that you can use for in game purchases, extending your xbox live service, or even buying a Starbucks gift card.

You sign in with your Microsoft Account on Bing or on the Microsoft Rewards Xbox app and you will receive rewards points for doing searches and activities that can then be used to "buy" various rewards that will be sent to your email in the rewards tab
using those points.

Also with physical some of the loyalty rewards are outstanding compared to Xbox rewards trash. Rakuten often has a 25x points deal on now, which effectively gives you a good £5 in points on each game to use off something else.

There are some physical games that come with a trial, and if its available in your country you can always gain points in Microsoft Rewards/Xbox Live Rewards (not sure if the latter is a thing still) and you can spend those points on codes for free Gold.

Hope that helps, cheers!