I recently saw the following video about making music with RxJS and was surprised that it was not scheduling notes using the Web Audio API clock. Rather, it was just playing audio directly using the System based timers with the standard schedulers in RxJS: youtube.com/watch
The Web Audio clock has a much higher resolution than the System clock, which is suitable for sample accurate playback (see info on clock differences below). The common setInterval and setTimeout functions typically used are based on the System clock, these functions are known to be imprecise and these happen to be what the current RxJS schedulers all use!
I'm aware that Tone.js can be used for playback but it comes with a large API and it's own idea of a music timeline that is limiting and not well suited for playing back arbitrary note structures. Using RxJS music can be processed in many ways so it makes sense to have the right scheduler as well.
From my research it looks like a custom RxJS scheduler is required that uses the Web Audio clock as a source. This way events scheduled on an Observable become queued actions that get scheduled directly to the Web Audio context (thread). How do I go about creating such a custom scheduler (there does not seem to be much info about this topic)? For most tasks the standard RxJS schedulers would suffice but for accurate music playback I see no other way.
CLOCK INFO
The Web Audio API has an extremely high-resolution and reliable timing system which differs in many ways from the JavaScript timing system of which the maximum resolution is only 1 millisecond, 0.001. The Web Audio API timing system has a resolution of 1/1000 of a picosecond, 0.000000000000001, or 1 femtosecond. This resolution is designed to be able to specify alignment on the sample-level, even with high-resolution audio. The timing system is always relative to the creation of the AudioContext, always starting at 0, and progressing in high-resolution seconds. The value can be accessed at any time by calling the ‘currentTime’ parameter of the AudioContext. This timing system makes it possible to build accurate sequencers, drum-machines, or time-reliant effects.
var audioContext = new AudioContext;
audioContext.currentTime; // example: 507.570793650793631
Music notes or any numerical AudioParam can be automated and scheduled. For instance, you could schedule the cutoff frequency of a filter to change over time, creating a filter sweep, or fadeout a sound over time.
Scheduling accurate audio for any kind of app using the Web Audio API is rather important. Maybe just as important as the RAF scheduler, which RxJS already has.
Is integrating RxJS and Web Audio via a dedicated scheduler more than just a custom scheduler, should it rather be an official scheduler that comes with RxJS out of the box?
Some further information:
The 'arbitrary units of work' that a Web Audio scheduler would run would be the music notes and audio params. To picture this lets say we have a frame/slice of notes pulled out of a score of data using RxJS pipeline operators. Each note in the quantized time frame/slice still has it's own fine durational timing. Each note event then needs to be scheduled as an 'action' in the Web Audio scheduler. The scheduler then pushes out each scheduled note to the Web Audio context/thread at the scheduled time (or maybe it flushes all the notes instead?).
What is key is to switch to the Web Audio scheduler (maybe using observeOn) just as the notes are scheduled and sent to the Web Audio context. The essential thing is to use the Web Audio clock to know when to schedule each note. So in the over all pipeline this vital aspect has to be done outside the standard RxJS schedulers to avoid the jitter and low-resolution that comes from using setInterval and setTimeout, which are based on the System clock not the high-resolution Web Audio clock.
The Web Audio world has had an imperative workaround solution to setInterval and setTimeout for a few years now using a hack. Now in this new functional and reactive age we have a great opportunity to fix this hack. What I and the rest of the musical world would really appreciate is if RxJS experts like yourself would take a good look at the following article and it's demo. Please note also that the demo schedules visuals using RAF and then separately schedules the audio notes using the workaround hack. It is this hack that a new RxJS scheduler for Web Audio would improve upon.
html5rocks.com/en/tutorials/audio/scheduling webaudiodemos.appspot.com/metronome/index.html github.com/cwilso/metronome