Thanks to cross-realm zones, you can group with people from different realms. In order to populate zones with plenty of players, people are put into a zone with people of different realms automatically. When you leave the zone, you will not see those people anymore.
Each zone is "hosted" on a particular server for the cross-realm. Some players might be temporarily shifted onto your realm. Other times, you will find yourself temporarily on a different realm when you enter a zone. (You can't really tell unless you take a look at the server time near your mini-map and find it's different than usual.)
The player you encountered may have been on PvP, but it's unlikely he was from a PvP server. Those are not grouped together with PvE or RP servers via CRZ. You could still be invited into a group by a Battletag friend from a different server and potentially shift to a PvP realm or whatever, but this should not normally happen with random people you meet via CRZ.
Even though you can group, you cannot trade with or friend players from other realms. You also cannot join guilds in other realms. (You CAN become Battletag friends, but those are different from regular friends.)
Realm transfers are usually a paid service so you can put your characters on different servers. For example, you may want to join a particular guild on a certain server, or would rather be with a particular community or group of friends on their server.