softcut feature-request: non-realtime(low-priority) ‘poke’ and ‘peek’ functions

[RajaTheResidentAlien]

Hi, i was mentioning in this thread about a softcut 'poke' and 'peek' function to allow us to access samples in softcut buffers directly and process them offline/non-realtime with our own dsp algorithms. for example, we could have something like so:
softcut.peek(buffer-channel, sample-index)
and the same with poke:
softcut.poke(buffer-channel, sample-index, write-value)…
then in my Lua, i could write something like:

for i=1,2048 do
local fxdsamp = softcut.peek(2, 108+i)  --read source samp from buffer channel-'2' at sample-time-'108+i'
fxdsamp = [...some form of processing to the peeked samp-value...]
softcut.poke(1, 54+i, fxdsamp) --write effected samp back to buffer channel-'1' at sample-time-'54+i'
end

…this would open up norns to the world of non-realtime digital signal processing where we could process the samples in softcut buffers by our own dsp algorithms…
if this could be done at extremely low-priority(i imagine the _copy_stereo and _copy_mono methods are like this), then this could allow other norns processes to run regular audio without being interrupted by this offline processing…
An app could record audio-input, start playing it back in realtime, then effect samples offline in non-realtime, write those samples back to a different part of the softcut-buffer/channels, and then switch to play the effected audio after its ready(i suppose that means the poke function might need a callback function too to indicate when the newly written information is safe to be read).

in the thread linked above, Brian also mentioned for optimization, it could be offered in blocks(i'd recommend some maximum in samples that is close to 10-50ms (like between 512-2048 samps) shorter block-sizes would allow people to upload samples of varying length to their norns, and there would be a minimal need for truncation(or fancy overlap-windowing), on the other hand, larger block-sizes would probably be faster/easier/more-efficient)... just some extra food for thought.

No worries if it's too much of an ask(my main reason for this is that softcut is so fast and efficient, that it would be magical to have some additional custom-dsp options right there on that layer(i want to add my own softclip and other distortions, plus some options open up for offline filtering, etc.), for the data stored in softcut buffers... but i would not want anything changed/added if it was too difficult to keep softcut/norns as efficient and fast as it is now).

Thanks for considering it :)

[catfact]

hey thanks for the suggestion

an experimental branch reworks norns into a single process running both lua interpreter and softcut jack client.

making softcut buffer contents directly accessible from matron/lua, is one of the nice things that the new architecture facilitates. but APIs for such access have not yet been implemented.

in any case, async access could also be done. add functions and job types to BufDiskWorker
and possibly add event types or (re)use SOFTCUT_RENDER

[RajaTheResidentAlien]

that all sounds great!
-the new architecture sounds really cool: i like how responsive and low-latency everything with softcut feels(this is what excites me most about softcut - i notice response-times for control, and latency of recording-input is far superior to anything i've ever experienced from supercollider or max/msp on a laptop), tying everything into one single process seems like it would just solidify that awesomeness all the more 👍
-async access might still be additionally helpful: if something very low priority could happen, and then a callback occur once the samples are written back to buffer and ready to be read, that would be ideal(so maybe adding event types or (re)using softcut_render might be most efficient(?)... i notice with the event_render, there's an 'interval' as the retrieved floats/samples are pulled at that interval to cover the duration requested with only the number of samples requested, and i guess this would require an interval of 1(every consecutive sample), but as i'm still new to the system(i don't know enough about BufDiskWorker to tell if it's a better place for this, for example), mainly just checkin in with my ideas here to show my excitement for whatever y'all decide is best - i like your design/developer instincts and am excited to use it all however you decide working it best 🍶 )

Truly appreciative of your considering this 🙇

[catfact]

yeah, there's a problem with actually passing significant amounts of buffer memory between the jack client and lua. SOFTCUT_RENDER event could be used for the peek function (with 1 sample stride as you say) but it imposes the substantial cost of allocating an additional copy of the buffer RAM. (for poke it would be sort of worse, since we'd have substantial memory allocated in lua and then somehow made accessible to the worker thread, yucky)

BufDiskWorker is a module in the audio client which operates on its own thread (not the audio thread.) so its methods have direct access to the buffer memory; also unlike work done in the lua interpreter, work done in BufDiskWorker shouldn't interfere with main thread timing (sequencing, gesture handling etc.) so that's really the preferred way to perform the kind of non-RT operations you're talking about.

nonetheless, there are many issues with BufDiskWorker, it's not the most robustly designed system @ryleelyman has probably worked with it more than anyone else in the context of using it as non-RT buffer manipulation backend from lua.

off the top of my head, here are a few issues with BufDiskWorker as it is right now:

• softcut processing isn't aware of job progress, so recording can clobber async processing and vice versa.
• there is (IIRC) no facility for signalling lua on buffer job completion.
• thread priority isn't set, because it uses the c++11 std::thread API which doesn't do that. (needs to drop down to pthread.)

---

setting all those issues aside, i think it would be possible / nice to extend BufDiskWorker to both provide a greater range of "DSP primitives" and maybe even to allow custom DSP routines defined in lua. for example:

• weaver pulls lua function from stack, with signature like float (int, float) [accepts sample index and value, returns new value]
• that is somehow wrapped in a function pointer and passed as parameter to new BufDiskWorker job type (JobType::ProcessCustom or w/e)
• worker iterates through requested samples calling lua-defined custom process func

---

i should reiterate that i'm not really expecting to have time to personally make significant code contributions to norns in immediate future. but happy to help / explain / review/ etc

[RajaTheResidentAlien]

all sounds good. also, feel free to mark this feature-request with the lowest priority(if BufDiskWorker is being extended already, this'll be ace, otherwise, no worries from me: i've enough to play with, and still much to learn, on norns as is). Thanks again!

[robbielyman]

consider my curiosity piqued. Lua-scriptable non-RT buffer manipulation sounds rad.
I think the SOFTCUT_RENDER function, whatever it's called in BufDiskWorker, does actually execute a callback, but it would be rad to have that more generally available for NRT processing.

Not sure of the best way to get BufDiskWorker and softcut to play nice with each other... might be a bigger change than I'd want to do by myself.

Anyway, I might sit down and hammer away at this sometime next month; would be a cool addition.

[RajaTheResidentAlien]

Thanks @ryleelyman truly appreciate you're looking into it(and no rush/urgency, as you find the most convenient time; i'm still very new to C++ and also norns, so while i wish i could help you out here, it still all looks foreign/daunting to me... but, i'm working on learning so perhaps at some point). Thanks again!

[robbielyman]

huh, I cannot for the life of me seem to be able to divine the right way to wrap a lua function into a function pointer... I'll keep poking around

[robbielyman]

aha, it's exactly how I maybe thought it would be, how confusing. Could use some help with liblo, but maybe we're to the point where I should create a draft PR and get comments there.