Slideshow
How it Works
The Sensors
Our note-sensing technique is what makes DeepNote unique. Using photodiodes aimed at the 5 notes on the screen, we are able to pick up on the changes in light that are exhibited when a note passes through the sensors. A photodiode works just like a photovoltaic cell (solar panel), by turning absorbed light waves into voltage. Thus, a voltage spike is experienced when light increases. The photodiodes have an 8 nanosecond response time and a small viewing angle, making them ideal for accurate and speedy analysis of notes. Because the voltage generated by this light change is very small (on the scale of a few millivolts), circuitry is used to amplify the signal up to about 3 volts, remove as much noise as possible, and adjust the hysteresis. The result of all of this is a digital square wave where 3 volts indicates a note, and 0 volts indicates no note. The sensors are held in place by a sliding rack that allows them to be adjusted for different size televisions.
The Sensor-Guitar Connection
So now that we have all of these wonderful digital signals, how do we get them to the guitar? The rack and guitar are both equipped with standard 15 pin D-sub ports. A D-sub cable connects the two of them, and can be removed for storage, portability, or whatever else . A D-sub cable is simply a cable with several corresponding pins on each end. A wire runs through the cable for each pin, transmitting digital data in parallel. So, 5 of the pins transmit the colors from the sensors to the guitar (plus one extra wire for good measure), and 2 pins transmit power and ground from our circuitry in the guitar to the sensors (the sensor circuits need power and ground for their amplifiers). On the sensor end, the D-sub connector is wired to the sensors, and on the guitar end, it is wired into our control board.
The Parallax Propeller Chip
The parallax propeller propstick is the brain of DeepNote. It is an 8 core chip that runs at 80MHz. It processes all of the information using the code we wrote in both spin (its interpreted language), and assembly (a low-level CPU language.
The Power Source
DeepNote can be powered by either a 9V battery, or a 5V wall adapter. A DC coaxial jack with a built in switch automatically disconnects battery power and switches to wall power when the plug is inserted. When disconected again, it will switch back to the battery.
The Power Conditioning Circuitry
Using a power regulator and a capacitor, the power circuitry first takes the power provided to it and converts it to 3.3V, the voltage required by the processor. Our guitar has both a 9V battery holster and a connection for a 5V wall adapter. A switch on the front of the guitar turns the processor (and consequently automatic mode) on or off by connecting and disconnecting the conditioning circuitry from ground. A front LED illuminates when power is flowing.
The Delay Switch
Because the distance between the sensors and the bottom of the screen will vary between televisions, our guitar utilizes a hexadecimal switch that allows a selection of the delay time ((0-15)* a multiplier). This switch has 5 connections, one common rail (3.3V), and 4 outputs: 1,2,4,8. By switching on or off those 4 outputs, 16 settings can be achieved. The propeller chip interprets the input and adjusts the delay accordingly.
The Relays and LEDs
Some time spent reverse engineering the x-plorer guitar revealed that buttons, the strum bar, and the whammy can be pressed virtually by bridging the “positive” and “negative” connections associated with the action. In some cases, a negative was shared amongst buttons. This is great because it means that we never have the feed any voltage into the existing guitar circuitry. By using solid-state relays (they work like a switch), we can close the connections when appropriate, thus imitating a button press, strum or whammy. When a button needs to be pressed, the appropriate pin on the processor enters “high mode” and sends a voltage to the relay, telling it to connect the appropriate terminals. Because of this method, the exisiting connections can still be left in tact, allowing the guitar to be played in manual mode. The pin that sends the “high signal” to the relay also sends it in parallel to an LED that illumates when that action is being performed. This allows for debugging, and it makes it look really intense.
The Whammy and StarPower On/Off Switches
In autonmous mode, DeepNote will automatically whammy everything and attempt to activate star power at appropriate times. However, the automatic whammy starts to sound really annoying after a while, and the automatic star power may not be desired if the user wants to decide when to activate it. Thus, two switches were installed that allow these functions to be turned off or on when in automatic mode. They simply connect or disconnect the wire from the relay to the negative terminal of the star power and whammy contacts.
The Software and USB Connection
Naturally, the propeller needs some code to run! The code we wrote does a number of things:
- It analyzes the inputs, performs software filtering, applies the appropriate delay, and outputs the necessary button presses and strums.
- It constantly checks the state of the delay switch pins so the delay distance can be set as per the position of the dial
- It whammys at a constant rate
- It checks the notes it’s cleaned to determine when it should use star power. Then, it uses it when it feels the time is right.
The propeller is programmed through a standard USB to mini-B USB cable. Inside our guitar we have a mini-B to B extension cable plugged into the processor, with the B side mounted in the side of the guitar. This gives us easy access to the programming port when we want to tweak something.
*Learn more about how the DeepNote Bot works in the videos section*
You guys have a serious leg up on your way to becoming Electrical Engineers. I would have loved to get this kind of internship last year before I got into college. Having a reason to code in ASM would have definitely given me more motivation to learn that stuff. Great Job on all of this by the way.
Where in living hell did you get the resources to tackle a project like this?! Did you do this at work or at home?
Dr. Singer was nice enough to provide us with most of what we used (see the team page)
Nice job. Next time though don’t you think you should give it robot fingers so that it can actually touch the keys. Then it could play on anybody’s system. Next time.
Looks like a pretty good way to get the notes detected! Well done! In terms of making the whole system streamlined, is that ever going to be possible? How about the filtering? Did you all come up with those filters by yourselves? Me thinks not…
I noticed that, in the early detection video, the bot’s LEDs would turn off during any held note.
Why was this and how did you correct the problem?
seems simple enough..my recommendation is make suction cups to stick to the tv (keep the sliders). but still, WAY COOL!
war6763: The system could theoretically be a bit more streamlined, but the processing circuity for the photodiodes must remain right next to the diodes, to prevent signal degradation. The filtering program was written by the those on the team page. The amplification circuit was designed with some outside help from an electrical engineer.
TheBlunderbuss: In those videos, the songs were in practice mode, and the notes weren’t actually being played. We were just detecting them. So when the holds weren’t hit, the notes disappeared, and sensor LEDs went off.
Hey awesome project guys
I have a few questions. It seems that you used only one photodiode for each fret, do you place these slightly below the target area (to avoid the rising fire animation)? if you did, then how do you make sure long notes are held? Does the robot hold down the frets until the next note is seen (I’m fairly certain this would work)? Or are the notes released as soon as the robot sees nothing?
How fast are you “moving” your whammy bar? I have a Wii guitar controller and if I have a 50 ms delay between up and down, star power won’t activate by tilting the guitar.