I am looking into implementing a video encoding solution for adaptive streaming. The best way at the moment, to have adaptive streaming is to use MPEG Dash streaming standard; as it is the only open standard which will be the future of adaptive streaming, and is supported by major browser vendors and companies.
I am looking into creating a distributed encoding service where in I can use 3-4 nodes to encode a source file into the required formats, and bitrates and can then create an adaptive streaming video out of it. The idea is to divide the source file into smaller durations, split them among the worker nodes who would then do the encoding of those small bits and then we can re-assemble those encoded parts into a complete video.
I plan to use Node on the server and ffmpeg to do video encoding. I know its one hell of a job, doing distributed video encoding (there are literallyy companies running on such products) but I do not plan to have a fully scalable service doing tons of video data encoding per day, just a small service which can suffice my workloads of few hours of video per day.
I wanted to know if anyone has ever tried to implement such a distributed thing using node, and queues such as ZeroMQ. Any tips on how one should be looking at designing such a system, and any learnings or gotchas one may have faced when building one.
Elemental Cloud is doing it, what added value do you offer?
I worked in a company that had its own encoding platform, worked nicely and had very little issues, still moved to elemental when the time came.
The hint it, you can not work only with dash, you need to deliver to all platforms and each platform has its own DRM. Now, If you are not going to use any DRM, then anyone could steal your encoded files and what is worse, anyone could build what you are building.
What I am saying is, think on what problem are you solving and what is the added value. IF there is none, use what is outhere now.
Yep, you have a hell of a job ahead of you.
Sounds like you have the right idea though. Luckily we're in an AWS world and you can implement this kind of system on a reasonable budget in an integrated ecosystem.
Generally you want to gather your mezzanine files in an S3 bucket, determine your encoding profile, and use something like an Elemental Cloud cluster to generate your renditions.
I've never used ZeroMQ, but also take a look at Aspera and SQS to see if they fit in your price range. I'll admit I've seen bottlenecking issues in both, but these were amidst cutting real-time highlights during a full slate of hockey games during our NHL.tv launch.
In terms of considerations:
1) If you're only using DASH, consider your video player, especially for Apple devices. 2) Choose your transcode profile wisely. The less renditions, the less overhead. 3) Troubleshooting DRM is a bitch.
Gotchas:
1) Consider your content metadata workflow. If you're associating things like titles and descriptions with these videos, you'll have to have a separate system in place. The industry standard is thru MRSS feed. 2) S3 is your best friend. If you're setting up any automated workflow, remember that, and trend towards using watch folders and 0 byte delivery.complete files as markers for triggering workflow. 3) Cost will escalate if you're storing mezz files, consider the best archival practice for your situation (S3? Glacier?)
Good luck!