Project Scriabin
An Exploration of Sound and Color
by Darren Woodland Jr.
This project will map musical pitches with the corresponding visual light spectrum (colors) in an attempt to create a musical composition from the visuals of a photo or work of art. It is an attempt to visualize the music of imagery that can possibly inform on its characteristics or structure.
The user will be able to use a second screen painting application, in this instance, Adobe Photoshop to place color on the screen in a grid and create notes and chords on a simulated musical measure. The finished product (musical composition) would bring an element of surprise in that it is unknown how it will sound. There will also be surprise in the process, from the creation of imagery to match certain chords or how the colors can be mapped to specific tonal values.
The original idea for the project was derived from the medical condition of synesthesia, where some individuals can “see” sound. In particular, from Alexander Scriabin, a Russian composer and pianist who has sound-color synesthesia. The novelty of the project comes from the idea that colors have emotions associated with them, and that tonal values or chords can also be considered warm/cool or happy/sad.
As I studied the life and work of Alexander Scriabin I became increasingly interested in the condition of synesthesia. With an individual who has synesthesia, as their brain processes information from one scene it will randomly trigger another sense. I began to look into how the human brain processes aural and visual information. I wanted to explore how using visual and aural information at different points and at different intensities throughout the process of creation affected the users experience.
Research: I began the process by researching Alexander Scriabin’s life and work. I then moved to understanding how sound and color have been used in relation with one another and how the brain processes the two.
TouchDesigner: The bulk of the work was done in the visual programming application TouchDesigner. I First began the process by using my webcam and cropping it into 5 X 3 grid, so 15 “individual cameras.”
I moved on from this approach to using a remote painting application, Adobe Photoshop. I integrated that with the program and cropped into into the same 15 portions.
.png)
Next, I began to assign the RGB color values seen by Alexander Scriabin to the corresponding tonal pitches. I used a table system with min and max values in order to compensate for any deviation of values made by the software or transferring data from one application to another.
The next step was to link the table with Photoshop camera setup using a series of containers and nodes that read the max and min values of the table into each individual section. I used Python code, if statements to go through each portion of the table and determine which value was on the screen.
Code:
n = op('midiout')
def onOffToOn(channel, sampleIndex, val, prev):
#node5
if op('select55')['r'] != 0:
n.sendNoteOn(1, op('select55')['r'])
if op('select56')['r'] != 0:
n.sendNoteOn(1, op('select56')['r'])
if op('select57')['r'] != 0:
n.sendNoteOn(1, op('select57')['r'])
if op('select58')['r'] != 0:
n.sendNoteOn(1, op('select58')['r'])
if op('select59')['r'] != 0:
n.sendNoteOn(1, op('select59')['r'])
if op('select60')['r'] != 0:
n.sendNoteOn(1, op('select60')['r'])
if op('select61')['r'] != 0:
n.sendNoteOn(1, op('select61')['r'])
if op('select62')['r'] != 0:
n.sendNoteOn(1, op('select62')['r'])
if op('select63')['r'] != 0:
n.sendNoteOn(1, op('select63')['r'])
if op('select64')['r'] != 0:
n.sendNoteOn(1, op('select64')['r'])
if op('select65')['r'] != 0:
n.sendNoteOn(1, op('select65')['r'])
if op('select66')['r'] != 0:
n.sendNoteOn(1, op('select66')['r'])
if op('select67')['r'] != 0:
n.sendNoteOn(1, op('select67')['r'])
return
def whileOn(channel, sampleIndex, val, prev):
return
def onOnToOff(channel, sampleIndex, val, prev):
#node5off
if op('select55')['r'] != 0:
n.sendNoteOff(1, op('select55')['r'])
if op('select56')['r'] != 0:
n.sendNoteOff(1, op('select56')['r'])
if op('select57')['r'] != 0:
n.sendNoteOff(1, op('select57')['r'])
if op('select58')['r'] != 0:
n.sendNoteOff(1, op('select58')['r'])
if op('select59')['r'] != 0:
n.sendNoteOff(1, op('select59')['r'])
if op('select60')['r'] != 0:
n.sendNoteOff(1, op('select60')['r'])
if op('select61')['r'] != 0:
n.sendNoteOff(1, op('select61')['r'])
if op('select62')['r'] != 0:
n.sendNoteOff(1, op('select62')['r'])
if op('select63')['r'] != 0:
n.sendNoteOff(1, op('select63')['r'])
if op('select64')['r'] != 0:
n.sendNoteOff(1, op('select64')['r'])
if op('select65')['r'] != 0:
n.sendNoteOff(1, op('select65')['r'])
if op('select66')['r'] != 0:
n.sendNoteOff(1, op('select66')['r'])
if op('select67')['r'] != 0:
n.sendNoteOff(1, op('select67')['r'])
return
def whileOff(channel, sampleIndex, val, prev):
return
def onValueChange(channel, sampleIndex, val, prev):
return
Ableton Live: The final step was sending out individual MIDI signals from TouchDesigner to Ableton Live. In Ableton I set up vocal voices to sing a simulated C- Major Bach Chorale created using the colors in Photoshop.
.png)
This experiment on sound and color was quite successful. I do want to expand the capabilities of the program to recognize a wider range of color values and fine tune the transfer of data between applications.
Taking It Further: There are a few directions that i would like to explore next when it comes the the general premise of this project.
-
Would like to be able to read a greater range of colors for each musical pitch.
-
Being able to read more than just “color blocks”, scan entire live paintings / images.
-
Interactive installation using a camera and speakers.
-
Would like to go from sound to color.
-
Would be interesting to try and incorporate machine learning to make the process run smoother/do more.