I just managed to join a private @matrix room Form # using the # bridge!

I think what hindered me before was that my matrix account which was already joined in that room before was using the same nick as my @xmpp account tried to use when joining. Renaming the xmpp nick did the trick ;)