“Walk on the Sun” STEREO Space Mission Science Exhibit

Credits:

Produced and designed by Marty Quinn of Design Rhythmics Sonification Research in collaboration with the Christa McAuliffe Planetarium and the UC Berkeley Space Science Lab and funded by a two-year STSCI NASA Ideas Grant.

'Musica Medica' vibration transducers by KETANO LLC of Nashua, NH USA. Contact ketanollc@gmail.com and visit www.musicamedica.us for more information.

Special thanks to Dr. Jeanne Gerulskis of the the Christa McAuliffe Planetarium and Dr. Laura Peticolas of the UC Berkeley Space Science Lab for their invaluable contributions to this project.

Visual surveillance software used in the exhibit is based on Efi Merdler's Visual Surveillance Toolkit in C#.

Description:

The "Walk on the Sun" Solar TErestrial RElations Observatory (STEREO) Space Mission Science Exhibit makes possible new ways of interacting with and understanding data and imagery from this recent space mission. It is designed to increase accessibility and provide new ways to explore the millions of images and other data recorded by the two spacecraft as they study the Sun, Coronal Mass Ejections and solar winds.

Funded by a two-year Space Telescope Science Institute (stsci.edu) NASA Ideas two year grant, the exhibit features an overhead data projector and web cam both facing downward toward the ground. The images of the Sun are projected onto the floor, with the web cam used to track the movements of visitors within a 5'X5' space in front of the image area. Two controllers are placed on either side of the viewing/moving area. These units represent the two spacecraft and provide the means for visitors to select the data they wish to explore.

Paul Cooper, Education and Human Resources Coordinator for the American Geophysical Union recently described the exhibit after seeing it in action at the AGU and NASA sponsored “Exploration Station” in Ft. Lauderdale, Florida. A video clip of a young (sighted) boy (Figure 2) experiencing the exhibit at that event can be viewed at www.drsrl.com/exhibits.

“Marty Quinn of Design Rhythmics Sonification Research Lab astonished visitors with compellingly interactive apparatus which blended solar imagery with music. From a gantry rig, a projector displayed STEREO mission solar images onto the floor and a web camera, installed next to the projector, tracked movements of anyone standing within its field of view. A computer translated movements that the web camera captured into a cursor traveling across the surface of the projected image. Correlating to the movement of the cursor Quinn's software mapped changes in image colors to sampled instruments and changes in image brightness to alterations in pitch. The result was a music version of a visual image. Quinn uses this translation technique to bring image data to people with visual impairments as well as create unique music from scientific data.”

Each pad controller unit features sixteen 1"x1" MIDI trigger pads. Eight of the pads select one of eight cameras to display. These include four temperature sensitive cameras called EUVI color-coded red, blue yellow, and green. Four additional trigger pads select coronagraph 1 and coronagraph 2, and solar wind imagery of the space between the Sun and the Earth called Hi1 and Hi2. The coronograph images show the Sun's outermost atmosphere of the Sun, the corona and is designed to image the large explosions from the Sun known as Coronal Mass Ejections as they leave the Sun. The Hi1 and Hi2 cameras are meant to watch these explosions as the move across the space between the Sun to Earth but can also capture other interesting solar wind events. Two of the pads allow the starting and stopping of playback of images as a movie. Two pads control the forward or backward play of the images through time. One touch sensitive pad controls the step amount between images during movie playback to allow for a faster presentation of the rotation of the Sun. The harder this pad is struck, the faster the Sun rotates. Another pad allows the 'NEXT' image to be displayed and still another will allow the selection of 3D images, requiring the use of 3D glasses. Each pad is labeled in Braille and English.

When an image is displayed and the movie playback is stopped, visitors can move their arms and body to explore the image. As they move, the visitor's movements are mapped into the image. They see visual feedback of white dots or lines signifying the areas and points selected in the image by their movements. Each selected pixel (the actual image data) is turned into music by mapping the color of the pixel into one of seventeen instrument timbres, and the brightness of the pixel into one of 36 pitches in a Spanish Gypsy scale. The pitches represent 5 octaves on a keyboard, and the Spanish Gypsy scale is a very beautiful and interesting scale. In this way, those who cannot see can experience a highly differentiated and beautiful perceptual signal in the audio domain and thereby perceive the composition and characteristics of the images.

In addition, visitors can hold two vibrating transducers that are stimulated by the music, providing a Left/Right audio signal directly into their hands. These vibrators are called Musica Medica and marketed by Ketano, LLC in New Hampshire. They are successfully being used in some therapeutic contexts and we are exploiting them to expand the perceptual expressions and cognitive opportunities of the exhibit. Thus, for those who cannot hear, they can feel the music the others are hearing, thereby joining in the perceptual experience with their hearing friends. For those who can see, hear, and feel...the combination of the multiple sensory presentations is extremely stimulating and pleasant, as documented by the video clip of a young boy interacting with the exhibit, mentioned above, and input we received during the showing of the exhibit.

Visitors can change cameras at will during either the playback of the movie or when exploring the image. They can reverse direction of the Suns rotation, to support reexamination/review of interesting data, and/or they can select the 'next' image to display. They can also alter how many images are skipped as the movie images are played back. Images are kept in date/time synchronization as much as possible.

The physical structure of the full exhibit consists of an 'H' shaped triangular truss structure (see figure above). It can fit into a 10 by 10 foot exhibit booth space. The structure is flexible in design so that it can be lowered in incremental steps. It will have four legs, connected across each pair of legs and across the middle. A speaker hangs from each leg providing a surround sound experience. The structure will support the data projector and web cam up to a height of 13 feet. At 11 feet high the solar image is about 4 feet wide. Images are displayed at a resolution of 1024 X 1024.

Nearly two million images from the STEREO mission are currently accessible thru the exhibit, all residing on one hard drive. Other imagery data may be loaded into the exhibit, thus extending the application of the exhibit to new space missions as they evolve (IYA2009, Phoenix, IBEX, etc).
Preliminary Impact of the Exhibit
Two students in New Hampshire, who are blind, recently experienced the exhibit and shared with us through emails their thoughts and feelings about it.

The first email is from Keene State College student Andrew Harmon (Figure 3). He spent one and a half hours on the exhibit and then created Braille labels for the exhibit’s image selection and control units.

From: "harmonkj" <harmonkj@metrocast.net>
To: <marty@drsrl.com>
Subject: Thoughts and Thanks
Date: Monday, May 19, 2008 11:17 PM

Dear Mr. Quinn

I thank you for the unique opportunity you provided me this Saturday past. It was extremely educational and very interesting to say the least. The explanation was very thorough, and I was able to pick up the ideas and controls of the process fairly easily. I am honored you allowed me to experience the Sun in all its glory in a brand new way as astronomy has been one of my childhood passions I had to abandon over the years as increasing difficulty took the enjoyment from me. Once I became used to the system, the details of the musical tones were distinct enough that I was able to distinguish the shape of the image, the sunspots as well as the hottest points on the surface of the sun. In the brief time I used the vibrating sensors (about an hour?) I found them useful to hone in on very small spots on the photos. However they were not as useful or detailed as the music when I was exploring the entire landscape. For me there was not as much detail with the vibration as there was with the sound. I may have been concentrating more on the nuances I heard than those I felt. One other thought is that the tracking was a bit tricky. The use of a black glove to allow the sensors to differentiate between my body and the imagery was quite ingenious, as was the solution to stop the synthesizers before proceeding with movement when viewing a frozen image with my body.

I cannot begin to express how excited this experience made me - the possibilities of applications for the equipment you showed me is quite vast, although I will share only a few. The immediate use you mentioned - to adapt any form of visual art (such as the Mona Lisa) to sound and physical sensations would be tremendous. I do not know how many times my mother has attempted with varying success to describe many pictures and imagery of famous places we have visited. I thought the ability to insert some form of audio information, such as a descriptive narrative of a point would be priceless on pictures and on maps. For example, a particular feature, like a crater on the moon or a building in Washington, D.C. or New York City. Other possibilities might lie in the rendering of an image file normally just deleted in alternative texts for the blind, particularly that of books containing sheet music or pictures of paintings. Finally, one possibility is adapting the equipment to assist parents of visually impaired or deaf children by giving them a shared experience in the very same way their children do. There are many other applications, I am sure, since the range is quite literally endless.

You have my sincere thanks for the experience and opportunity of a lifetime. I hope to try this again as your project moves forward. If I can ever be of any assistance please don't hesitate to contact me.

Andrew Harmon

PS ... please do send me any links or literature that I can make available to others attending the Expo. Your work is too exciting and too important. If I can boost attendance at the Planetarium in January, I will. Again, thank you so much.

The second is from Chelsea Duranleau, also a Keene State College student (Figure 4). She spent three hours exploring the exhibit. At the end of the session, I restarted the exhibit to load in a set of art masterpiece images that allowed her to listen to and explore them as music. Hence the reference she makes to art work.

From: "Chelsey Duranleau" <c.duranleau88@hotmail.com>
To: <marty@drsrl.com>
Subject: Rhythmic Sonification Prototype
Date: Thursday, June 05, 2008 4:37 PM

Dear Mr. Quinn,

Thank you very much for showing me the prototype of your rhythmic sonification exhibit. Music is such an integral part of my life and to be able to explore images, scientific data, and art work through it was remarkable. The different instruments and various pitches made it easy to distinguish between different colors and contrasts. I really enjoyed the motion aspect of the exhibit as well. [ED] Thank you so much for showing me your prototype; it was an amazing experience for me, and I know that people, blind and sighted alike, could benefit from such ground-breaking technology.

Sincerely,

Chelsey Duranleau

These encouraging and enthusiastic responses suggest the exhibit may provide a significant learning and investigative platform for those who are blind.