I think we must go over some clarifying questions to begin, like:

1. How are the machines connected to our machine (the one with the file)?
2. Do they have access to Internet or are they just within the same network?
3. Do we have any sort of access to the 10k machines?

I do not remember the exact details of the talk we had back then, but it was a very knowledgable conversation. The interviewer was a senior at LinkedIn and had worked in the industry for a long time and he did tell me that we need to clarify these questions first.

Peer-to-peer was one of the approaches that I suggested too. I later found out that there are distributed file systems too and they can be used in such a scenario, though I do not understand how that would work exactly. You can read about them here - unf.edu/~sahuja/cis6302/filesystems.html.

Some of the approaches I think that can be used:

1. Peer-to-peer transfer, more like how the tor network deals with transfers.
2. Providing access to the file by uploading it to something like an S3 bucket and running a script that can loop over the machines and issue commands remotely to download the file. This would leave the whole tracking and network failure part to the S3 library. Assumption: Machines are able to connect to AWS S3 infra.
3. We can divide the file into chunks, and then transfer these chunks to the machines, also appending a checksum for each chunk which can be verified at the client machine and an acknowledgment can be sent back to the server which will keep track of any failures and try sending the chunk some more times. If the machine, is not able to receive the chunks, then that machine can be flagged and tried later or a report can be generated for such machines. This would require establishing a protocol and writing some code.

What are your views?

Saurav Shrivastav;

I think we must go over some clarifying questions to begin

Definitely. That's the way to go.

I later found out that there are distributed file systems too and they can be used in such a scenario

Ah. I assumed that We have a single machine, and need to transfer data to 10k machines. So I ruled out this thing from the very beginning. My bad. 🙈

1. Peer-to-peer transfer, more like how the tor network deals with transfers.

Yeah. But even in P2P we need to store a mappings from chunkId to machine(s)Id. I'm not sure if we can have a eventual consistent mapping/index in a system with so many network failures.

1. Providing access to the file by uploading it to something like an S3 bucket and running a script that can loop over the machines and issue commands remotely to download the file. This would leave the whole tracking and network failure part to the S3 library. Assumption: Machines are able to connect to AWS S3 infra.

That will be the best thing. S3 is a distributed object storage system, so it does make sense. (I have a gut feeling that we're missing something. I don't know what. 🥲).

1. We can divide the file into chunks, and then transfer these chunks to the machines, also appending a checksum for each chunk which can be verified at the client machine and an acknowledgment can be sent back to the server which will keep track of any failures and try sending the chunk some more times. If the machine, is not able to receive the chunks, then that machine can be flagged and tried later or a report can be generated for such machines. This would require establishing a protocol and writing some code.

Damn. That's cool. Although, isn't TCP performs this flow control, retries, in-order transmission, and acknowledgement thing by default? 🤔 By the way I think this manual chunks formation thing is definitely needed in Approach 1.

Anyhow, nice work buddy.

PS: Now my old comment sounds dumber. lol