DDSP: Differentiable Digital Signal Processing

栏目: IT技术 · 发布时间: 4年前

内容简介:DDSP is a library of differentiable versions of common DSP functions (such as synthesizers, waveshapers, and filters). This allows these interpretable elements to be used as part of an deep learning model, especially as the output layers for audio generati

DDSP: Differentiable Digital Signal Processing

DDSP: Differentiable Digital Signal Processing

DDSP is a library of differentiable versions of common DSP functions (such as synthesizers, waveshapers, and filters). This allows these interpretable elements to be used as part of an deep learning model, especially as the output layers for audio generation.

Getting Started

First, follow the steps in thesection to install the DDSP package and its dependencies. DDSP modules can be used to generate and manipulate audio from neural network outputs as in this simple example:

import ddsp

# Get synthesizer parameters from a neural network.
outputs = network(inputs)

# Initialize signal processors.
additive = ddsp.synths.Additive()

# Generates audio from additive synthesizer.
audio = additive(outputs['amplitudes'],
                 outputs['harmonic_distribution'],
                 outputs['f0_hz'])

Tutorials

The best place to start is the step-by-step tutorials for all the major library components that can be found in colabs/tutorials .

Modules

The DDSP library code is separated into several modules:

  • Core : All the core differentiable DSP functions.
  • Processors : Base classes for Processor and ProcessorGroup.
  • Synths : Processors that generate audio from network outputs.
  • Effects : Processors that transorm audio according to network outputs.
  • Losses : Loss functions relevant to DDSP applications.
  • Spectral Ops : Helper library of Fourier and related transforms.
  • Pretrained Models : Helper library of models for perceptual loss functions.

Overview

Processor

The Processor is the main object type and preferred API of the DDSP library. It inherits from tfkl.Layer and can be used like any other differentiable module.

Unlike other layers, Processors (such as Synthesizers and Effects) specifically format their inputs into controls that are physically meaningful. For instance, a synthesizer might need to remove frequencies above the Nyquist frquency to avoid aliasing or ensure that its amplitudes are strictly positive. To this end, they have the methods:

  • get_controls() : inputs -> controls.
  • get_signal() : controls -> signal.
  • __call__() : inputs -> signal. (i.e. get_signal(**get_controls()) )

Where:

inputs
controls
signal

For example, here are of some inputs to an Additive() synthesizer:

DDSP: Differentiable Digital Signal Processing

And here are the resulting controls after logarithmically scaling amplitudes, removing harmonics above the Nyquist frequency, and normalizing the remaining harmonic distribution:

DDSP: Differentiable Digital Signal Processing

Notice that only 18 harmonics are nonzero (sample rate 16kHz, Nyquist 8kHz, 18*440=7920Hz) and they sum to 1.0 at all times

ProcessorGroup

Consider the situation where you want to string together a group of Processors. Since Processors are just instances of tfkl.Layer you could use python control flow, as you would with any other differentiable modules.

In the example below, we have an audio autoencoder that uses a differentiable harmonic+noise synthesizer with reverb to generate audio for a multi-scale spectrogram reconstruction loss.

import ddsp

# Get synthesizer parameters from the input audio.
outputs = network(audio_input)

# Initialize signal processors.
additive = ddsp.synths.Additive()
filtered_noise = ddsp.synths.FilteredNoise()
reverb = ddsp.effects.TrainableReverb()
spectral_loss = ddsp.losses.SpectralLoss()

# Generate audio.
audio_additive = additive(outputs['amplitudes'],
                          outputs['harmonic_distribution'],
                          outputs['f0_hz'])
audio_noise = filtered_noise(outputs['magnitudes'])
audio = audio_additive + audio_noise
audio = reverb(audio)

# Multi-scale spectrogram reconstruction loss.
loss = spectral_loss(audio, audio_input)

ProcessorGroup (with a list)

A ProcessorGroup allows specifies a as a Directed Acyclic Graph (DAG) of processors. The main advantage of using a ProcessorGroup is that the entire signal processing chain can be specified in a .gin file, removing the need to write code in python for every different configuration of processors.

You can specify the DAG as a list of tuples dag = [(processor, ['input1', 'input2', ...]), ...] where processor is an Processor instance, and ['input1', 'input2', ...] is a list of strings specifying input arguments. The output signal of each processor can be referenced as an input by the string 'processor_name/signal' where processor_name is the name of the processor at construction. The ProcessorGroup takes a dictionary of inputs, who keys can be referenced in the DAG.

import ddsp
import gin

# Get synthesizer parameters from the input audio.
outputs = network(audio_input)

# Initialize signal processors.
additive = ddsp.synths.Additive()
filtered_noise = ddsp.synths.FilteredNoise()
add = ddsp.processors.Add()
reverb = ddsp.effects.TrainableReverb()
spectral_loss = ddsp.losses.SpectralLoss()

# Processor group DAG
dag = [
  (additive,
   ['amps', 'harmonic_distribution', 'f0_hz']),
  (filtered_noise,
   ['magnitudes']),
  (add,
   ['additive/signal', 'filtered_noise/signal']),
  (reverb,
   ['add/signal'])
]
processor_group = ddsp.processors.ProcessorGroup(dag=dag)

# Generate audio.
audio = processor_group(outputs)

# Multi-scale spectrogram reconstruction loss.
loss = spectral_loss(audio, audio_input)

ProcessorGroup (with gin )

The main advantage of a ProcessorGroup is that it can be defined with a .gin file, allowing flexible configurations without having to write new python code for every new DAG.

In the example below we pretend we have an external file written, which we treat here as a string. Now, after parsing the gin file, the ProcessorGroup will have its arguments configured on construction.

import ddsp
import gin

gin_config = """
import ddsp

processors.ProcessorGroup.dag = [
  (@ddsp.synths.Additive(),
   ['amplitudes', 'harmonic_distribution', 'f0_hz']),
  (@ddsp.synths.FilteredNoise(),
   ['magnitudes']),
  (@ddsp.processors.Add(),
   ['filtered_noise/signal', 'additive/signal']),
  (@ddsp.effects.TrainableReverb(),
   ['add/signal'])
]
"""

with gin.unlock_config():
  gin.parse_config(gin_config)

# Get synthesizer parameters from the input audio.
outputs = network(audio_input)

# Initialize signal processors, arguments are configured by gin.
processor_group = ddsp.processors.ProcessorGroup()

# Generate audio.
audio = processor_group(outputs)

# Multi-scale spectrogram reconstruction loss.
loss = spectral_loss(audio, audio_input)

A word about gin ...

The gin library is a "super power" of dependency injection, and we find it very helpful for our experiments, but with great power comes great responsibility. There are two methods for injecting dependencies with gin.

  • @gin.configurable makes a function globally configurable, such that anywhere the function or object is called, gin sets its default arguments/constructor values. This can lead to a lot of unintended side-effects.

  • @gin.register registers a function or object with gin, and only sets the default argument values when the function or object itself is used as an argument to another function.

To "use gin responsibly", by wrapping most functions with @gin.register so that they can be specified as arguments of more "global" @gin.configurable functions/objects such as ProcessorGroup in the main library and Model , train() , evaluate() , and sample() in ddsp/training .

As you can see in the code, this allows us to flexibly define hyperparameters of most functions without worrying about side-effects. One exception is ddsp.core.cumsum where we configure special optimizations for TPU.

Installation

Requires tensorflow version >= 2.1.0, but runs in either eager or graph mode.

sudo apt-get install libsndfile-dev
pip install --upgrade pip
pip install --upgrade ddsp

Contributing

We're eager to collaborate with you! See CONTRIBUTING.md for a guide on how to contribute.

Citation

If you use this code please cite it as:

@inproceedings{
  engel2020ddsp,
  title={{\{}DDSP{\}}: Differentiable Digital Signal Processing},
  author={Jesse Engel and Lamtharn (Hanoi) Hantrakul and Chenjie Gu and Adam Roberts},
  booktitle={International Conference on Learning Representations},
  year={2020},
  url={https://openreview.net/forum?id=B1x1ma4tDr}
}

Disclaimer

This is not an official Google product.


以上所述就是小编给大家介绍的《DDSP: Differentiable Digital Signal Processing》,希望对大家有所帮助,如果大家有任何疑问请给我留言,小编会及时回复大家的。在此也非常感谢大家对 码农网 的支持!

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

Kafka权威指南

Kafka权威指南

Neha Narkhede、Gwen Shapira、Todd Palino / 薛命灯 / 人民邮电出版社 / 2017-12-26 / 69.00元

每个应用程序都会产生数据,包括日志消息、度量指标、用户活动记录、响应消息等。如何移动数据,几乎变得与数据本身一样重要。如果你是架构师、开发者或者产品工程师,同时也是Apache Kafka新手,那么这本实践指南将会帮助你成为流式平台上处理实时数据的专家。 本书由出身于LinkedIn的Kafka核心作者和一线技术人员共同执笔,详细介绍了如何部署Kafka集群、开发可靠的基于事件驱动的微服务,......一起来看看 《Kafka权威指南》 这本书的介绍吧!

在线进制转换器
在线进制转换器

各进制数互转换器

Base64 编码/解码
Base64 编码/解码

Base64 编码/解码

Markdown 在线编辑器
Markdown 在线编辑器

Markdown 在线编辑器