Messenger Application From Scratch: Complete guide Part 3

Messenger application from scratch

This part 3 of messenger application from scratch will guide you on how to finish your project if you did not read part 2 and part 1 then click here for Messenger Application From Scratch part 2 and for the first part of this tutorial click here Messenger Application From Scratch part 1

How the Client maintain an open connection with the Chat/Node server?

Coming back to our user X and Node2 Server is constantly heart beating back and forth. Suppose, N2 server dies down who was managing our user X.So, what will happen in that case if N2 server dies, then WebSocket connection will break and user X will know that the other end of the server died.

Ø So, here what it’s going to do is it’ll initiate a new connection with the new host (maybe N3 or N4 server). And N3 is going to start updating the heartbeat information into the Redis cache with the last time this heartbeat information was received by N3. So, when one server dies, it is not a big deal as some other server would start taking its responsibility and will start updating

Ø Another thing is that if user X dies or is inactive, then he/she stops heart beating with N3 then N3 will stop updating the heartbeat information in Redis which means that other users can see this information stored in the cache to see that when this user was online last time. So, you would have noticed on Facebook, it shows that these users are online, and this user was online 1 hour or 30 minutes ago. All that information can be retrieved from the Redis cache where we can say that if a particular user is online or not based on the last updated heartbeat time.

So, we have this constant heartbeat going on and now I am going to explain about how what happens when we actually want to send a chat message to another party/user who is online right now. let's say there is a user Y who is a friend of user X and he /she is also online right now and is constantly heart beating with n4 server for example. So, here N4 will store this user Y entry in Redis. Let's assume user X wants to send a message to user Y and here, User X has his connection open with

For example, user X sends “hi” message to User Y and here in our case, this “hi” message goes all the way to N3. And what N3 will do now is it is going to persist this message in the NoSQL DB(HBase/Cassandra Database/Mongo DB) and then it's going to check that is there any anyone who is managing user Y and because of that N3 will look into this Redis to see who is his user Y and if he/she is online and if he/she's online, then which server is managing user Y. So, from Redis cache, it will find that N4 is managing user Y and he was online very recently.

So, now what N3 will do is it will send this “hi” message to N4 and fact that we need to send this message to user Y but N3 will not initiate a direct connection with user Y because N4 already has this direct WebSocket connection open to user Y then why to create another connection again. N3 will then send this message “hi” to the user Y and user Y will instantly see this message. So, this is the flow of what happens when both user X and user Yare

What happens when a person sends a message to the user while he/she isOnline?

So now, let me explain what happens when user X wants to send a message to user Z and user Z is not online at that point in time. In this case, user X sends a message “hello Z” to the N3 server and then N3 will look into this Redis, and as there is no entry for user Z in Redis cache then this tells that user Zis offline and none of the servers is managing that. In this case, the N3 server will just persist this message(“hello Z”) into the database and also persist the flag that there are some unread messages that also persist in the database.

Now, when user Z does come online and whoever is managing user Z, let's say N1 is managing, then N1 will read from the database and it checks that there are some unread messages in the database and then N1 going to transmit all that information to user Z and at the same time N1 will update its heartbeat entries in Redis with the info. that user Z is So this is the main concept of how to chat communication will happen between two users.

So, next, I am going to mention here about how our database schema will look like and how we're going to handle some of the security.

· User Table

· Communication Table

· UnRead_AuditTable

· Read_Audit Table

These are the three bare minimum database tables mentioned as above which we will be required to run our chat service. First table is the “User” table. In the User table, we will have the column as User 1, User 2, “Chat_ID” and then “auth_key/encryption_key”. So here “auth_key” will be the randomly generated encryption key. It will be encrypted with another master key which Facebook will source somewhere else.“Chat_ID” is the unique generated number. “User 1” and “User 2” are any two users and they are friends on Facebook after accepting each other’s friend requests.

Now when user X logs in, then we can we know that also we know who all his/her friends are. So, we’ll pull all the information about his friends related to his chat into the So, let's assume that user X only has one friend i.e. user Y.So, we are going to pull user X, his/herChat_ID and the decrypted version of the encryption key.

The Node 2 server will pull all that information and then send it across to user X.So, user X now have all the information and now it is all set to chat with user Y.Now, user Y came online, and he/she also pulled in all information from the same User table. Now, when user X wants to send a message to user Y. first thing, he/she does here is he/she encrypts the message on the device itself with the encryption key(stored in a database) then he/she sends the encrypted message to N2 server with the Chat_ID. Here, what N2 does is, it first persists this information into the

So, in our Communication table, the Chat_ID is the foreign key and we have the “message” column which is the encrypted text and the next column is the “Timestamp” which is uniquely generated. So, in our example N2 will search that who is managing the user Y. From Redis table, it will find that N3 is managing the user Y.

So, now N2 sends the encrypted message to N3 that is sent by user X and N3 will send this message to user Y and what user Y is going to do is as he/she has already the decryption key to encrypt the message, so he/she's just going to use that key to decrypt this message and he/she will get the actual message.

Here, one other thing I want to mention was that each of the nodes(N1, N2, N3 and N4 etc.) can manage hundreds or potentially thousands of sessions. So, it's not like N2 is just talking to user X but N2 could potentially be talking to tons of such users.

So, as I explained already, this way when user X sends the message, on the other end-user Y receives the message. And if user Y wants to send a response back to the user Y, then again, the reverse process is going to happen. So, when user Y replies back the message it first gets encrypted with this encryption key. The message is sent to N3 with the conversation ID.

N3 will first persist that information in the “Communication” table and then N3 is going to check who is managing user X. As N2 is managing user A, then N3 is going to send this message to N2 with that encrypted message and now N2 is going to send it back to user X.As user X also has the same encryption key, so it is going to decrypt that message sent by user Y and then he/she is going to receive the message. So, this way a basic flow will happen with the help of our database tables.

Offline messaging:(What happens when a person sends a message to the user while he/she is Offline

Let me write about what happens when you send a message to an offline user. let's say user X wants to send a message to user Z who is offline right now. So, just like before when user X logged in, he can fetch all the information about both user Y and user Z and also fetches the conversation ID and their encryption key.

Now user X wants to send a message to your user Z. First user the message gets encrypted with the user Z’s encryption key and then he's going to send this information to N2 and along with the fact that it's going to check the User table to see the chat ID. N2 will first insert this entry into the Communication table.

So here, data will get inserted with the Chat_ID, time and then encrypted version of message and “from_user” is (the user from whom the message is coming from). As soon as this entry is written into the communication table, we know that the message is saved and at that point N2 can confirm user X that the message is sent.

We confirmed that user Z has still not read the message. We sent a confirmation to user X that the message is sent, and that confirmation is sent by N2 but the message is still not read by user Z. Now N2 server will look into the Redis to see that is there any user Z online right now and is there any host which is managing user Z. Since user Z is offline right now, X cannot send this message to user Z. So what he's going to do is to store this message in our “Unread_Audit”

Now, let's assume that user Z comes online after say ten hours. So as soon as user Z comes online, first it's going to fetch all this information basically Chat_ID and user info from the User table about who sent a message and also the encryption key for user X. and let’s say user Z is heart beating with some other server N1. So remember as this is a bi-directional connection so user Z is talking back and forth with N1 and N1 was the one who fetched all this information from the database for the user Z and sends it.

Also, N1 is now updating the Redis cache indicating that user Z is online because N2 is constantly looking into the radius to see if friends of user X I mean which friends of user X are online. As soon as user Z comes online N1 updates into Redis and then N2 reads that and it knows that the user Z is So, it is going to notify that information to the user X.

As soon as user Z comes online, he/she wants to see if there are any unread messages and N1 is going to do that for him/her. Here N1 is going to look into this “Unread_Audit” table and then it seems that there is a user X and Z and it already knows the chat_ID between user X and user Z. Then it is going to look into the communication table and will fetch this full this information and send this encrypted version of the message to user Z. As the user Z already has the encryption key. so he/she's going to decrypt this message and then to see the actual message from user X.

As soon as user Z gets the message, at that point users, Z is going to conform to N1 that it has read the N1 now knows that the message which was sent from user X has been read so then N1 is going to see that if still, the user X is online and if he is online then X will get notified. For that N1 will see first which hosts is responsible for user X, and here that's N2. So, N2 will tell user X that user Z has read this message and as soon as that happens, we will remove the unread user details entry from the Unread_Audit table.

On the other hand, if user X was offline that time, then N1 will just put an entry to the “Read_Audit” table saying that message from user X to user Z at the time T has been read. And when user X comes online and then N2 will read from the “Read audit” table to see that user Z has read this message and notify User X that his message at time T has been read by user Z.

Sending Images between Users:

So, next, let me explain here how do we handle the case where user X wants to send a picture to user Y. Now user X takes a picture on his device and he wants to send that picture to user Y using messenger. First what the device will do is it already has this information about user Y which is Chat_ID and encryption key from User Table. So, by using the encryption key, it will encrypt that picture and then send this message N2. N2 will receive the picture then in the encrypted format with the conversation ID and the fact that it needs to be sent to user Y.

Then N2 knows that it's a picture, not a text So, it's going to talk to the Blob storage. Every company has blob storage. Here, N2 server will talk to this blob storage and it will store this picture and get a corresponding URL back. So,the picture is stored in the blob storage in an encrypted format which is encrypted with the key from the user table.

So, N2 gets a URL back and then it is going to store this entry into our user table. Then in our communication table, it is going to store the “Chat_ID”, timestamp, and the URL of the picture in the URL column. Then N2 will check if the user Y is online and, in our case, Y is online because it is a heart beating with N3.So N2 will send this URL and a thumbnail of the picture to N3 server and N3 will send them back to user B.

The original image could be 2-10megabytes, but the thumbnail could just be five kilobytes. So N3 will send that thumbnail picture to user Y along with this URL and if user Y is interested, he can click on the picture, and then we will take this URL and download the entire picture. So N3 we'll take this URL and download the entire picture for user Y. Then user Y can decrypt that picture with the help of the encryption

So, this is one of the ways by which you can transfer a picture, or you can send a picture from the user X to user Y and this could also be used to send like some other file or whatever other format files.

Note: You can think about some of the optimizations you can do in persisting the data and also how you could potentially do a basic search on the conversations. Try it yourself and read our all technical blogs for detailed component-wise explanations.

Conclusion:

Here, in this blog, I have explained the very basic fundamentals of any type of Messaging system designing. In this context, the requirement is designing a messenger service which will be capable of sending text messages, images, voice message between two users/clients. All the design steps with the detailed components I have mentioned so that you can be able to implement the requirement with minimal effort.