The first solution comes in mind when it comes to chat is "sockets". The easiest way to do so is using a read made service called "pusher" https://pusher.com/

you'll have a channel for each conversation and all the user who are interested to chat to gether will listen to that channel. You'll have multiple events and all the users will subscribe to those events.

For example: user1, user2 and user 3 wants to chat together. You'll have a channel for example called "chat1"

Then user1, user2 and user3 will need to listen to this channel. Then you'll have for example an event called "new_msg" all the users will subscribe to this event. when any new message is being sent this "new_msg" event will be fired to all the subscribed users will be notified and see that message.

If you are considering Rails 5 then it is lot easier to develop using shiny ActionCable feature. Action Cable seamlessly integrates WebSockets to give real-time data transfer in rails.

Checkout DHH's video https://www.youtube.com/watch?v=n0WUjGkDFS0