[XeTeX] \setmathrm with fontspec has no affect on \mathrm{x}?

[Daniel Greenhoe]

If I use \setmathrm{fontx} in the preamble, shouldn't I expect fontx
to be invoked when I use $\mathrm{x}$ in the document body? But that
isn't the case. Why is that and how do I get what I expect without
exiting math mode? Here is an example (see also attachments):

\documentclass[12pt]{book}
\usepackage{fontspec}
\usepackage{unicode-math}
\setmathfont{xits-math.otf}
\setmathrm{texgyreadventor-regular.otf}
\pagestyle{empty}
\begin{document}%
$\mathrm{x}\mathrm{a}$%
\end{document}%

When compiled with xelatex, this produces a pdf that only uses
xits-math, not Adventor. Why aren't "x" and "a" typeset using
Adventor?

The log file reports
. Font family 'texgyreadventor-regular.otf(0)' created for font
. 'texgyreadventor-regular.otf' with options [].

<pdffonts mathrm.pdf> reports
name                                 type              emb sub uni object ID
------------------------------------ ----------------- --- --- --- ---------
JIHIAR+XITSMath-Identity-H           CID Type 0C       yes yes yes      5  0

Many thanks in advance,
Dan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mathrm.log
Type: application/octet-stream
Size: 29194 bytes
Desc: not available
URL: <http://tug.org/pipermail/xetex/attachments/20111217/0a5129f3/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mathrm.pdf
Type: application/pdf
Size: 2419 bytes
Desc: not available
URL: <http://tug.org/pipermail/xetex/attachments/20111217/0a5129f3/attachment-0001.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mathrm.tex
Type: application/x-tex
Size: 427 bytes
Desc: not available
URL: <http://tug.org/pipermail/xetex/attachments/20111217/0a5129f3/attachment-0001.tex>