The Ghost Orchestra – Real-Time Spectral Reverberant Accompaniment for Solo Instrumentalists

The Ghost Orchestra is a real-time audio processing system that turns a solo instrumentalist into a spectral ensemble. The performer's audio is analyzed on-the-fly for pitch, timbre, and dynamics. The system then generates multiple layered voices (strings, brass, woodwinds, percussion) that respond to the input, but with a crucial twist: each layer is deliberately imperfect. Pitches waver, timing sways, and the whole texture is drenched in a reverberant, decaying ambience—as if the orchestra is heard from far away, through a fog, or as a faint memory. The goal is not to create a perfect virtual orchestra, but an evocative, eerie, and unpredictable partner that inspires the performer and audiences alike. The entire system runs on a single laptop, using pre-trained spectral models and custom DSP, designed by and for one person—the performer-engineer—who builds it as a tool for their own live shows.

Solo instrumentalists often lack the lush, interactive texture of an ensemble. Existing solutions like loopers, sequencers, or virtual orchestras are either too rigid (quantized loops) or too sterile (perfectly tuned synths). They kill the spontaneity and the emotional give-and-take of live music. What if a soloist could have an orchestra that listens and breathes with them, but with the character of a ghost—haunting, unstable, and deeply atmospheric? This project scratches that itch: a tool that turns solitude into a spectral conversation.

• Build a real-time audio analysis pipeline that extracts pitch, loudness, and spectral centroid from a solo instrument or voice.
• Design a generative engine that creates up to 12 simultaneous voice layers (strings, winds, brass, percussion) using wavetable synthesis and spectral morphing.
• Imbue each layer with controlled imperfections: ±5 cents pitch drift, random timing offsets up to 50ms, and dynamic volume fluctuations.
• Apply a shared, convolution-based reverb that changes subtly over time (e.g., shifting impulse response every 30 seconds) to create a 'haunted hall' effect.
• Create a simple footswitch/gesture interface for the performer to cue changes in orchestral density, mood, and decay rate.
• Ensure total latency <20ms for responsive feel during live performance.
• Release as a standalone application (macOS/Windows) for other solo performers.

• Not a general-purpose DAW plugin or VST—it's a self-contained instrument.
• Not a tool for perfect, note-for-note orchestral transcription—the output is suggestively orchestral, not literal.
• No built-in notation or score following—purely improvisational.
• Not a commercial product—the plan is to share it freely as an open-source project.
• No visual interface beyond a minimal status window—the focus is audio-only interaction.

• Language: C++ (JUCE framework) for low-latency audio I/O and DSP.
• Machine learning: Python (ONNX runtime) for pitch estimation (CREPE) and timbre classification (custom lightweight model trained on 200 hours of solo instrument recordings).
• Spectral processing: FFT-based pitch shifting and spectral morphing via STFT (librosa routines ported to JUCE).
• Reverb: Convolution reverb using custom impulse responses captured from empty churches and stairwells (48kHz, 4s tail).
• Synthesis: Wavetable oscillators with crossfading between orchestral sample slices.
• Hardware: Single laptop (MacBook Pro M1), audio interface (Focusrite Scarlett 2i2), and a MIDI footcontroller (Behringer FCB1010).

The system operates in four stages: 1) Audio input is buffered and analyzed in 1024-sample frames at 48kHz. Pitch is estimated every 10ms using CREPE (run on GPU via CoreML). Timbre classification (bright/warm/percussive) triggers different orchestral templates. 2) The generative engine maintains a 'ghost state'—an array of 12 voice objects, each with its own pitch memory, drift oscillator, and envelope. When a new note is detected, the engine layers multiple voices, transposing the original pitch by intervals (up to 3 octaves) and assigning them to different instrument families based on a Markov chain. 3) Each voice is passed through a spectral shaper that applies subtle formant filtering and a time-varying all-pass filter for 'unstable' timbre. 4) All voices are mixed, sent through the convolution reverb, and summed with a dry/wet ratio that follows the performer's dynamics (louder input = wetter mix, up to 90%). The footcontroller can override overall mix, reverb decay time, and number of active layers.

• Latency from AI pitch estimation (CREPE) could exceed 10ms, adding perceptible delay. Mitigation: use hybrid approach—basic pitch from autocorrelation for speed, CREPE for refinement.
• Timbre classification may fail on unusual playing techniques (pizzicato, harmonics). Mitigation: player can preset 'mode' via footcontroller.
• CPU overload from 12 voices + reverb. Mitigation: limit active voices dynamically, use single-precision floats, and allow user to cap voice count.
• The 'ghostly' imperfections might become annoying rather than evocative. Mitigation: extensive live testing with diverse audiences.

• How to handle chords? The soloist can play double stops, but the system currently treats them as a single pitch. Should it detect polyphony up to 4 notes?
• What is the best way to generate 'orchestral' timbre without sounding like a cheap synth? Should we use physical modeling instead of wavetables?
• Should the system learn from the performer over time (e.g., adapt drift patterns to their style)?
• How to package the ML models so that the application is portable (e.g., use CoreML for macOS, TensorFlow Lite for Windows)?

The project stalled after completing research and initial prototyping of the pitch estimator. The creator realized that training a timbre classifier on solo instrument recordings would require far more data than a single person could collect and label in a reasonable timeframe. Plus, life moved on—they started a band and the need for a ghost orchestra faded. The plan remains a 'what if' that still sparks excitement.

The idea was inspired by a late-night improvisation in a resonant stairwell. The echoes from different floors seemed to create a chorus. The plan includes a detailed notes section on the exact drift algorithms (random walk with per-voice inertia) and the emotional intent behind each orchestral layer.