Table of contents
Tutorial 2 - Pattern Interpolation
In this tutorial, we will expand on the previous tutorial and add the ability to interpolate between two patterns.
the source code for this tutorial is available in the tutorials branch of the repository.
This tutorial assumes that you have completed the previous tutorial and have a working plugin. Please refer to the previous tutorial if you have not completed it yet.
Description
We will be using the same model as the previous tutorial. However, instead of having a single button which generates a random pattern, we will have two buttons which generate two random patterns. Then, we will add a slider which interpolates between these two patterns.
Plugin Name and Description
Let’s start by changing the name and description of the plugin.
To do this, we will modify the NeuralMidiFXPlugin/NeuralMidiFXPlugin/CMakeLists.txt file as follows:
project(Tutorial2NMFx VERSION 0.0.1)
set (BaseTargetName Tutorial2NMFx)
....
juce_add_plugin("${BaseTargetName}"
COMPANY_NAME "AIMCTutorials"
...
PLUGIN_CODE AZCL # a unique 4 character code for your plugin
...
PRODUCT_NAME "Tutorial2NMFx") # Replace with your plugin title
Once you re-build the cmake project, and re-build the plugin, you should see the name of the plugin change in the DAW:
Now we are ready to move on to the next step.
Modifying the GUI
As mentioned in the description, we will be adding two buttons and a slider to the GUI. Moreover, we will keep the second tab (for midi mappings) as is without any changes.
Let’s go back to the Configs_GUI.h file and modify it as follows:
namespace Tabs {
const bool show_grid = false;
const bool draw_borders_for_components = false;
const std::vector<tab_tuple> tabList{
tab_tuple
{
"RandomGeneration",
slider_list
{
},
rotary_list
{
rotary_tuple {"Interpolate", 0, 1, 0.0, "Kg", "Pr"},
},
button_list
{
button_tuple{"Random A", false, "El", "Iq"},
button_tuple{"Random B", false, "Rl", "Vq"},
}
},
// Rest as before
ModelThread::deploy()
Before modifying the code, let’s think about what we want to do.
We want to have two patterns, A and B, and a slider which interpolates between these two patterns. To do this, whenever button A is pressed, we will generate a random latent vector and store it somewhere (latent_A). Similarly, whenever button B is pressed, we will generate another random latent vector and store it somewhere (latent_B).
Subsequently, we will calculate the actual latent vector to be used for inference by interpolating between these two:
latent = (1 - slider_value) * latent_A + slider_value * latent_B
We are familiar with process of generating a random latent vector and using it for inference. However, this is the first time that we need to store a custom value that may be used in the future calls of ModelThread::deploy() method. As mentioned in the Data Types, section of the documentation, we can modify the MDLData struct to include our custom data. Additionally, we will store the slider value in the MDLData struct as well to check whether it has changed or since the last call to ModelThread::deploy().
To do so, let’s navigate to CustomStructs.h and modify the MDLData struct as follows:
// MDLData struct in CustomStructs.h
struct MDLData {
torch::Tensor latent_A;
torch::Tensor latent_B;
double interpolate_slider_value{0};
};
When building the plugin, the plugin will automatically instantiate the
MDLDatastruct and pass it to you asmdl_datain theModelThread::deploy()method.
Having done this, we can now modify the ModelThread::deploy() method to generate a random latent vector and store it in mdl_data whenever button A or B is pressed.
Before carrying on, let’s print some messages to ensure everything is setup correctly
// ModelThread::deploy() in ModelThread.cpp
if (!isModelLoaded) {
load("drumLoopVAE.pt");
}
bool should_interpolate = false; // flag to check if we should interpolate
// =================================================================================
// === 1. ACCESSING GUI PARAMETERS
// Refer to:
// https://neuralmidifx.github.io/docs/v1_0_0/datatypes/GuiParams#accessing-the-ui-parameters
// =================================================================================
// check if the buttons have been clicked, if so, update the MDLdata
auto ButtonATriggered = gui_params.wasButtonClicked("Random A");
if (ButtonATriggered) {
should_interpolate = true;
PrintMessage("Button A Clicked");
MDLdata.latent_A = torch::randn({ 1, 128 });
}
auto ButtonBTriggered = gui_params.wasButtonClicked("Random B");
if (ButtonBTriggered) {
should_interpolate = true;
PrintMessage("Button B Clicked");
MDLdata.latent_B = torch::randn({ 1, 128 });
}
// check if the interpolate slider has changed, if so, update the MDLdata
auto sliderValue = gui_params.getValueFor("Interpolate");
bool sliderChanged = (sliderValue != MDLdata.interpolate_slider_value);
if (sliderChanged) {
should_interpolate = true;
PrintMessage("Slider Changed");
MDLdata.interpolate_slider_value = sliderValue;
}
Seeing that the GUI is working as expected, we can now implement the interpolation process
To interpolate we need both states to be randomized, as a result, to simplify the code, we will ensure that on the first call to ModelThread::deploy() both latent_A and latent_B are randomized.
// ModelThread::deploy() in ModelThread.cpp
// ... previous code
// =================================================================================
// === 2. initialize latent vectors on the first call
// =================================================================================
if (MDLdata.latent_A.size(0) == 0) {
MDLdata.latent_A = torch::randn({ 1, 128 });
}
if (MDLdata.latent_B.size(0) == 0) {
MDLdata.latent_B = torch::randn({ 1, 128 });
}
Now, to finish this tutorial, we need to interpolate between latent_A and latent_B and store the result in latent.
// ModelThread::deploy() in ModelThread.cpp
// ... previous code
// =================================================================================
// === Inference
// =================================================================================
bool newPatternGenerated = false;
if (should_interpolate) {
if (isModelLoaded)
{
// calculate interpolated latent vector
auto slider_value = MDLdata.interpolate_slider_value;
auto latent_A = MDLdata.latent_A;
auto latent_B = MDLdata.latent_B;
auto latentVector = (1 - slider_value) * latent_A + slider_value * latent_B;
// ... previous code
