I took this in the UK two years ago: https://4z2.de/atm_windows.jpg - I don't know enough about how the various versions of Windows look but maybe this helps.

It also doesn't look very trust-inspiring.

You need to trust the bank's network security more than you trust the ATM itself.

There were some high profile breaches at some retailers in the past couple years that exploited some 0days. How can you defend your POS/ATM against a 0day if the retailer/bank has bad network security practices?

I guess these machines are not connected to the Internet, and have a very limited user interface surface.

Flaws are less important.

In the US, we have private leased lines to connect with some of the major banks. But the banks most likely use something else to connect to the ATMs.

Most attacks are physically breaking into the cash safe but there were some attacks a couple years ago where people were plugging into the USB port and getting money or something.

I'm not GP, but if I had to guess it would be Windows Embedded, I know of a number of ATMs that use it.