Unix has a long history on the IBM mainframe, prior to Linux. It was running under VM/370 at Princeton in 1977 - https://www.bell-labs.com/usr/dmr/www/portpap.pdf page 3 - and in the early 1980s AT&T ran UNIX under TSS/370, which was used in development of the 5ESS switch - see https://www.bell-labs.com/usr/dmr/www/otherports/ibm.pdf. Amdahl UTS came out in 1981 (descended from the 1970s work at Princeton) and IBM/Interactive Systems VM/IX around 1984; then in 1988, the IBM/Locus AIX/370, followed by AIX/390. And then in 1993, MVS acquired a Unix compatibility subsystem (originally called OpenEdition, now called Unix Systems Services), and under its present name of z/OS is officially certified as a Unix, since Unix System Services passes the X/Open tests.
I think the real problem for Unix on the mainframe has been the cost of the mainframe platform. Once you are developing portable applications, it is easy to shift them on to something else (AIX, Solaris, Linux, whatever), and you'll probably save money by doing so. When developing using a classic mainframe technology stack (COBOL, CICS, IMS, etc), replatforming is much more effort/risk, so those apps are more likely to survive on the mainframe in the long-term than Unix ones.
(Disclaimer: I work for a competitor to IBM so I may well be biased.)
I think the real problem for Unix on the mainframe has been the cost of the mainframe platform. Once you are developing portable applications, it is easy to shift them on to something else (AIX, Solaris, Linux, whatever), and you'll probably save money by doing so. When developing using a classic mainframe technology stack (COBOL, CICS, IMS, etc), replatforming is much more effort/risk, so those apps are more likely to survive on the mainframe in the long-term than Unix ones.
(Disclaimer: I work for a competitor to IBM so I may well be biased.)