What I'm attempting to do is one thing on most <br> tags and another
thing on the last <br> tag under a <li> or <p> tag (I've bolded these in
the example below). My problem is the <br> tag can be n deeply nested
under formatting tags such as <strong>, <em>, etc.

You probably need
 match="br[ancestor::p or ancestor::li]"
and
 match="br[ancestor::p or ancestor::li][position()=last()]"
or so.

The second one might need some parenthesis, can't tell without
testing...