Sondius Physical Modeling Work at CCRMA 1993-1997
Click on Image below for PDF of the deck
Stanford/CCRMA, Sondius Project (1994-1996)
From 1983-1994, I spent 11 years working as a programmer and manager in the EDA business (CAD software for designing chips). I yearned to return to working with audio. From 1994-1996 I was very fortunate to have an opportunity to work at Stanford/CCRMA as a paid researcher. I was a member of the "Sondius" team that developed technology around Stanford/CCRMA's Physical Modeling Patents (Sondius-XG). This was perhaps the best working experience that I have had to date. I felt that I had come full circle by working on things that had interested me 15 years earlier.
At Stanford/CCRMA, I developed portions of SynthBuilder (which was written primarly by Nick Porcaro). SynthBuilder was written in Objective-C on a NeXT machine. I designed the hierarchy algorthms, the loop ordering algorithm, the pin algorithms, the authorization code system, and worked on many of the inspectors. While at Stanford, I co-authored a patent, and co-authored seven papers (see below) SynthBuilder was awarded the 1997 Grand Prize in the Bourges International Music Software Competition.
NeXT machine(Click to see a larger image)
Bill Putnam and Tim Stilson
(Click to see a larger Image)
SynthBuilder and the Frankenstein
Box. (Click to see a larger Image)
I also worked on developing a number of Physical Models. These models were developed on a NeXT machine using SynthBuilder (screenshot-1 gif, 68kb, screenshot-2 gif, 55kb) and the NeXT MusicKit running on Motorola 56k DSPs. We had 3 DSP platforms. The original NeXT machine had an onboard 25Mhz 56k. Bill Putnam and Tim Stilson also designed a DSP farm known as the Frankenstein box (screenshot-3 gif, 154kb, screenshot-4 gif, 177kb) that had 8 Motorola 56k EVMs over-clocked to 80 MHz connected to a P5 NeXT machine via an ISA interface. We also had a single card EVM integrated onto an ISA card , known as a “Cocktail Frank” (Tim Stilson and I hand built these). Here are some sound samples, and a brief explanation of each model.
This model was based on Perry Cook's Flute model. I added a lookup table to calibrate the waveguide loop filter. I used my tin flute and a spectrum analyzer to find coefficients for the loop filter (as a function of pitch) that matched the real tin flute. This made it possible for the model to sound similar to my tin flute.
This recording was made in August 1996 on the Frankenstein DSP box. In this recording, I’m playing the flute model using a 6-channel guitar MIDI controller and a breath control head set (for overblowing the flute). I really enjoyed playing this model. Overblowing the model was was very expressive.
- Waveguide Guitar Model, Different Pickups (mp3, 386 kb)
- Waveguide Guitar Model, Distortion and Amplifier Feedback (mp3, 631 kb)
- Waveguide Guitar Model, Wah-wah (mp3, 295 kb)
- Waveguide Guitar Model, Jazz Guitar (ES-175) (mp3, 436 kb)
(Click to see a larger image)
(Click to see a larger image)
This model was based on Julius Smith's Guitar models, as well as a distortion model that was done by C. R. Sullivan (CMJ fall 1990). I added many new things to this model.
- I designed it to be a 6 string model running on 6 MIDI channels so that my guitar MIDI controller could control it polyphonically. It responded to polyphonic PitchBend.
- I added the amplifier feedback model with a distance control so that the "sweet spot" for feedback could be controlled by MIDI. I usually mapped this to a foot pedal so that I could change the amp distance while playing the model.
- I added a waw-wah pedal. Tim Stilson and Bill Putnam improved this by collecting some real data from my old Cry Baby waw-wah. They built an improved version of the wah-waw pedal.
- This model had selector switches that allowed you to change the model to sound like several different classic electric guitars. It was in part based on a pick model using data from real guitars. I collected data from my Steinberger, Les Paul and ES175. I found that I could use the first period of the real instruments as the excitation (the pick)! In addition created lookup tables for the filter coefficients so that the decay time of the strings would match the real instruments.
- This model led us to invent the patented DelayAI delay line (patent, paper pdf, 671kb). I found that high notes would decay too quickly when I used a linearly interpolating delay line. This is because the linear interpolation of the delayline itself acts as a filter. The DelayAI delay line combines the lossless properties of an allpass filter, and ability to change the delay length without clicks (as with the interpolating delay line).
This recording was made in August 1996 on the Frankenstein DSP box. The guitar model is being played by me using a 6 channel guitar MIDI controller a MIDI pedal for wah-wah and a MIDI pedal for amplifier distance. This model was eventually porting to the Pentium/SynthCore during the early days of Staccato. I used to LOVE playing this model. It was really like playing a super hot electric guitar. Sometimes when I was working at Staccato late at night, I would fire up this model and play the Neil Young's theme from the movie "Dead Man". Totally Haunting. I believe that it will be years before another guitar player will be able to have access to a model like this.
I developed the harpsichord model from scratch. It had a pair of slightly detuned coupled string models to model the pair of strings found in a real harpsichord. Later Scott Van Duyne contributed the excitation model. This model used Julius Smiths' commuted Synthesis to model the soundboard.
This model is based on Scott Van Duyne's Coupled Mode Synthesis (CMS). It was calibrated using Julius Smith and Scott Van Duyne's Calibration MatLab code. What's really great about CMS is that if you have a real recording of a percussive instrument you can get the model to sound fairly similar. Basically the real recording can be analyzed into partials, each with a unique decay rate (coupling between partials is ignored). The model can be calibrated with a simplified list of partials and their decay rate. Also the excitation "hammer" model responded to MIDI velocity so that you could hit the bells hard or soft. If you hit the bell a second time, the model already contains energy from the initial hit, allowing you to "rearticulate" the model.
The data used to calibrate these models came from a bell that I brought back from Tibet, and from a tenor wind chime in my back yard.
RECORDINGS
All recordings are Synthesized Physical Models played from a Roland guitar controller.
Sondius Recordings (1994-1996)
Here are all the Sondius Recordings.
- 01 Sondius Demo2:49
- 02 Waveguide Flute0:18
- 03 Taiko Percussion0:23
- 04 Percussion Using CMS1:03
- 05 Tibetan Bell Using CMS0:48
- 06 Windchimes Using CMS0:43
- 07 Tubular Bells Using CMS1:13
- 08 Under The Sea, All Physical Model0:27
- 09 Israel - Piano, Bass3:21
- 10 Harpsichord2:28
- 11 Guitar Model3:38
- 12 Acoustic Bass0:58
- 13 Tek_No1:47
- 14 James - Rhodes, Bass, Percussion4:00
We wrote a number of papers during this research period, won an award for SynthBuilder, and I was co-author of one patent.
PATENTS
- 5,742,532: April, 21, 1998. Van Duyne; Scott A. (Stanford, CA), Jaffe; David A. (Berkeley, CA), Scandalis; Gregory P. (Mountain View, CA), Stilson; Timothy S. (Mountain View, CA) "System and Method for Generating Fractional Length Delay Lines in a Digital Signal Processing System."
- 6,959,094: Oct 25, 2005. Cascone; Kim (Pacifica, CA), Petkevich; Daniel T. (San Jose, CA), Scandalis; Gregory P. (Mountain View, CA), Stilson; Timothy S. (Mountain View, CA), Taylor; Kord F. (San Jose, CA), Van Duyne; Scott A. (Palo Alto, CA) "Apparatus and methods for synthesis of internal combustion engine vehicle sounds"
PAPERS
- “SynthBuilder: A Graphical Rapid-Prototyping Tool for the Development of Music Synthesis and Effects Patches on Multiple Platforms” (pdf, 1.5 mb) , Nick Porcaro, David Jaffe, Pat Scandalis, Julius Smith, Tim Stilson, and Scott Van Duyne, Computer Music Journal, Volume 22, Number 2, pp. 35 - 43, MIT Press, 1998.
- “A Lossless, Click-Free, Pitchbend-able Delay Line Loop Interpolation Scheme” (pdf, 671 kb), Scott A. Van Duyne, David A. Jaffe, Gregory Pat Scandalis, Timothy S. Stilson, 1997 International Computer Music Conference, Greece, 1997.
- “SynthBuilder and Frankenstein” (pdf 305 kb), N. Porcaro, W. Putnam, P. Scandalis, T Stilson, D. Jaffe, and J. O. Smith, S. Van Duyne, ICAD 1996.
- “Work in Progress, SynthScript and SynthServer” (pdf 8 kb), P. Scandalis, David Jaffe, CCRMA Affiliates Presentation 1996
- “SynthBuilder: A Rapid-Prototyping Tool for Sound Synthesis and Audio” (pdf 14kb), Nick Porcaro, Pat Scandalis, Julius Smith, 1996 Presented at Berkeley EE seminar.
- “Using SynthBuilder for the Creation of Physical Models” (pdf 283 kb), N. Porcaro, P. Scandalis, D. Jaffe, and J. O. Smith, 1996 International Computer Music Conference, Hong Kong. 1996.
- "SynthBuilder Demonstration, A Graphical Real-Time Synthesis, Processing and Performance System" (pdf 415 kb) Nick Porcaro, Pat Scandalis, Julius Smith, David Jaffe and Tim Stilson, 1995 International Computer Music Conference, Banff. 1995.
Awards
Staccato Systems (1997-2000)
In late 1996, (at Stanford) we had been working on a portable host based implementation of something like the NeXT MusicKit. This was known as SynthServer/SynthScript. SynthServer was a portable Synthesis Engine. SynthScript was an interchange format (inspired by postscript) to describe Synthesis Algorithms (paper, early spec pdf cache 325 kb ). David Jaffe implemented SynthServer and I implemented SynthScript. By Nov. 1996, SynthBuilder was able to send SynthScript Algorithms and real time updates to SynthServer via a network connection.
In early 1997, the Sondius Team from Stanford (David Jaffe, Joe Kopenick, Nick Porcaro, Pat Scandalis, Julius Smith, Tim Stilson, Scott Van Duyne) formed Staccato Systems.
The SynthServer/SynthScript work was brought to Staccato, and evolved into SynthCore. SynthBuilder was ported to windows using Apple Computer’s “Yellowbox” technology. Here is a video of SynthBuilder/SynthCore as it was around 2000. Note that I made this video in March 2012 using my legacy Win98 machine with yellowbox installed. As far as I know, this is the only extant copy of SynthBuilder for Windows.
SynthCore was both a DLS (Down Loadable Sounds)/General Midi Synthesizer and a programmable Algorithmic Synthesizer that could be programmed via SynthScript Down Loadable Algorithms (DLAs). SynthCore was offered as a product in 2 forms.
- Packaged as a “midi driver”, SynthCore could replace the wavetable chip on a sound card for a host based audio solutio (SynthCore-OEM)
- Packaged as a DLL/COM service, SynthCore could be integrated into game titles so that games could make use of interactive audio algorithms (race car, space guns, car crashes) (SynthCore-SDK)
Here is a zip file of a demo of the that Staccato did. This was mostly the work of Tim Stilson and Scott Van Duyne. This is a small program that uses the SynthCore DLL to load a collection of DLAs (found in an encrypted .dla file). Click on the picture to download the zip file.
My role at Staccato was both as a programmer and I managed the SynthCore Team. As a programmer I developed SynthScript (pdf cache 325 kb) and its parser, the DLAManager (Down Loadable Algorithm Manager) ( pdf cache), much of the public API, some of the documentation, and most of the integration work to productize these two products. David Jaffe wrote the core Synthesis Engine and the XG-lite General Midi Synthesizer. David Van Brink wrote the DLS-1 Synthesizer. Danny Petkevich optimized the code for SIMD instruction sets, and developed the reverb and chorus algorithms.
I did very little modeling work at Staccato. However, Danny Petkevich and I developed a technique that we called “Compositing”. In May 1999 I watched a sound designer (Nick Peck) create a complex gun sound for a game. Watching how Nick worked was the inspiration for “Compositing”. He wanted to start from a collection of small sound samples and create a set of 10 or so “composite” gun samples that could be used in a game. He wanted these sounds to be similar, but different so that the gun sound was not repetitive. For example, he had 5 fire samples, 5 recoil samples and 5 hit samples. I saw that in Pro-tools he set up these sounds as 5 tracks in 3 columns. When he rendered a final gun sound he mixed and match samples of each column. He would sometimes trigger the sounds in random and statistical clusters.
Starting in June 1999 Danny implemented these ideas as a set of “Compositing” sound algorithms for race car games. Danny implemented things like tire skids, crashes, thuds, scrapes. The algorithms used SynthCore’s DLS-1 Synthesizer a custom sample set, and an event-processing algorithm to translate game collision events into random and statistical clusters of triggered samples. I believe that this algorithm was used in Electronic Art’s “Nascar 2000”.
As a footnote, SynthCore-OEM is now a part of the Analog Devices "SoundMAX" product. It can be found on about 60% of shipping PC's (See http://www.soundmax.com/OEMPartners/index.html). Examples include the currently shipping HP Pavilion (HP502n), Compaq (1510, 705) and IBM (NetVista A and M Series) systems as well as the Intel® D845PEBT2 Desktop Motherboard. I co-developed SynthCore-OEM, as well as managed the development team. " SynthCore-SKD can be found in a number of games, including Electronic Art's NASCAR-Revolution and NASCAR-2000. I co-developed SynthCore-SDK, as well as managed the development team.