Subham Sahu

Time.

Sep 6, 2022

Congrats buddy. 🙌 I'm intrigued by the following question:

There is a file of size 1 TB which needs to be transferred to 10k machines. How can this be done efficiently, keeping in mind that there can be network failures, machines going out of disk space, etc.

I initially thought that since there could be network failures, we need something reliable, so the transport layer must be TCP. The simplest thing I can think of is FTP. But since there are 10k machines, we cannot transfer to all of them at once. Because (I think) our server cannot have 2*10k TCP connections simultaneously.

Even if it could, the transfer will be very slow because of the following:

network failure is very likely to occur between some machines and servers.
The operation is also IO-bound.

Maybe peer-to-peer is the way to go. Network failure between a machine and a server is very likely, but network failure between a machine and all machines & servers is very unlikely.

I'm not sure how we'll manage/sync a table which maps fragmentId to machineId(s), though. Kinda like gossip protocol maybe?

I would really appreciate it if you could share your approach and any specific details the interviewer shared on it. Feel free to criticise any particular thing you didn't like in my thinking process above. 🙂

Thank you, bhai. And again, congratulations on this new feat. 💪

I think we must go over some clarifying questions to begin, like:

How are the machines connected to our machine (the one with the file)?
Do they have access to Internet or are they just within the same network?
Do we have any sort of access to the 10k machines?

I do not remember the exact details of the talk we had back then, but it was a very knowledgable conversation. The interviewer was a senior at LinkedIn and had worked in the industry for a long time and he did tell me that we need to clarify these questions first.

Peer-to-peer was one of the approaches that I suggested too. I later found out that there are distributed file systems too and they can be used in such a scenario, though I do not understand how that would work exactly. You can read about them here - unf.edu/~sahuja/cis6302/filesystems.html.

Some of the approaches I think that can be used:

Peer-to-peer transfer, more like how the tor network deals with transfers.
Providing access to the file by uploading it to something like an S3 bucket and running a script that can loop over the machines and issue commands remotely to download the file. This would leave the whole tracking and network failure part to the S3 library. Assumption: Machines are able to connect to AWS S3 infra.
We can divide the file into chunks, and then transfer these chunks to the machines, also appending a checksum for each chunk which can be verified at the client machine and an acknowledgment can be sent back to the server which will keep track of any failures and try sending the chunk some more times. If the machine, is not able to receive the chunks, then that machine can be flagged and tried later or a report can be generated for such machines. This would require establishing a protocol and writing some code.

What are your views?

Saurav Shrivastav;

I think we must go over some clarifying questions to begin

Definitely. That's the way to go.

I later found out that there are distributed file systems too and they can be used in such a scenario

Ah. I assumed that We have a single machine, and need to transfer data to 10k machines. So I ruled out this thing from the very beginning. My bad. 🙈

Peer-to-peer transfer, more like how the tor network deals with transfers.

Yeah. But even in P2P we need to store a mappings from chunkId to machine(s)Id. I'm not sure if we can have a eventual consistent mapping/index in a system with so many network failures.

Providing access to the file by uploading it to something like an S3 bucket and running a script that can loop over the machines and issue commands remotely to download the file. This would leave the whole tracking and network failure part to the S3 library. Assumption: Machines are able to connect to AWS S3 infra.

That will be the best thing. S3 is a distributed object storage system, so it does make sense. (I have a gut feeling that we're missing something. I don't know what. 🥲).

We can divide the file into chunks, and then transfer these chunks to the machines, also appending a checksum for each chunk which can be verified at the client machine and an acknowledgment can be sent back to the server which will keep track of any failures and try sending the chunk some more times. If the machine, is not able to receive the chunks, then that machine can be flagged and tried later or a report can be generated for such machines. This would require establishing a protocol and writing some code.

Damn. That's cool. Although, isn't TCP performs this flow control, retries, in-order transmission, and acknowledgement thing by default? 🤔 By the way I think this manual chunks formation thing is definitely needed in Approach 1.

Anyhow, nice work buddy.

PS: Now my old comment sounds dumber. lol

Search Hashnode