Processing to Unity

The initial idea for this project was to make an interactive educational game with physical balls as the game control to triggers a digital reaction. This idea bridges the gap between real physical and virtual world. The mechanism was for the player to throw the physical balls “into” a virtual scene projected on a wall. The technologies that have been used were Microsoft Kinect for Windows v2 and InFocus X6 projector. The front-end design was created using Adobe Illustrator, in which would be programmed using Processing.

As a digital media designer who does not have a strong programming background, I had to do a number of researchers and experiments. Started from following online tutorial videos, reading and analyzing book and articles, to asking a friend, Nathaniel Mannawi, who has a strong knowledge and experience in Processing, as my tutor and guided me through. The three books (refer to Figure x) by Ben Fry and Casey Reas disclose another level of understanding of Processing from its features, abilities, examples, and they also provide activity workbook for us to have a first-hand experience with the program. Another source I learned from is Daniel Shiffman’s youtube channel, “The Coding Train”. He is an authorized project leader from Processing. Additionally, he also writes on his personal blog regarding the tutorial for Processing, one of them on how to connect Kinect with Processing (Shiffman, n.d.).

After several experiments with Processing, the programme was then applied to be implemented to the Kinect v2. The good thing about Processing is they do provide the library for Kinect v2 thus it was really useful for this exploration, but first, you need to install the Kinect SDK that can be downloaded from the official website (https://www.microsoft.com/en-us/download/details.aspx?id=44561). By benefiting from the depth detection feature in Kinect v2, the ball managed to be detected by Kinect and able to trigger the fireworks effect. The idea behind this was to set specific depth by calibrating the wall area, so whatever object enter the specific depth area will trigger some effects.

Even though Processing is a simplified graphic programming language with the aim to teach non-programmers especially artists and designers (Byrne, 2015), it has limitations in terms of creating complex works. From not having a game engine to restrictions on assets, developing an interactive game would not be suitable with Processing.

The initial idea mentioned in Section 3.2.2 to use physical ball works when programmed using Processing and Kinect, which was helped by Nathan. It triggered Geometric Fireworks (Refer to Section 3.4.1.1.1.2). Later on, the challenge starts to appear along with errors and I could not troubleshoot it, thus wasting a lot of time. Frustrated with Processing, I tried Unity as a replacement and it works. Additionally, I was concerned that the project aesthetic would be affected by asset limitations in Processing, such as 3D assets which would be applied to the project. In which lead me to realize that Processing is better at generating visual art.

Therefore, Unity was then chosen as a better alternative approach as it is a game development software and also prior experience with Unity would definitely benefit the process.