Jaime Cross - Noise Maker At Large

Honours Year Sound Production student at University of Abertay Dundee and general all round sonic nuisance.

I do audio for games, films and animation.

Contact me at jrgcross (at) gmail (dot) com if you're in need of audio for your project.

Honours Update #21 - Reflection

For this last post I plan to evaluate and reflect upon my honours project and the work I have carried out over the past year. I will discuss the positives and negatives of the past year, as well as commenting on the skills and knowledge I’ve gained in the process. I will also detail some of the bigger issues that arose during the project and how they were worked around. While I may be repeating some of the points made previously, I feel it is important to restate them in order to provide a more complete reflection.

As the honours project is essentially an open brief I decided that I would like to look at the technical use of sound in games to some degree. I chose this as I felt that research into emotional feedback and experience relating to sound was something that had been covered in great detail before, while the technical applications of game audio was still an under-researched field. Originally I had wanted to look at how sound physically interacts with an environment and if it could be used as either a navigation aid or as a game mechanic, something akin to the game “Blind Man’s Bluff”. I also wanted to combine this with a visualisation of the sound waves interacting with the virtual environment, but quickly realised that this idea was far too big to try and attempt. Instead I had hoped to work my honours project into another brief I was working on for the Make Something Unreal competition. This would have mean that I could apply my theories directly into a game, and not have to worry about developing the entire game artefact myself, and increase my knowledge of Unreal Development Kit (UDK) in the process. As the team had decided on having as minimal a visual HUD as possible I discussed the possibility of using audio in its place, which eventually lead to my main aim of:

Determining if audio can replace traditional feedback methods such as a visual Heads Up Display (HUD), compass, minimap, objective markers, etc. in a 3D game through the use of environmental and music cues and contextualised beacon sounds, and allow a player to navigate through a 3D world unaided by additional visual information.”

Unfortunately the team didn’t manage to make it to the final of the competition, and for the most part went their separate ways. Two of the coders agree to help me develop something for my honours project, which took off some of the burden of actually producing an artefact, although only one was able to produce anything as the other’s workload became too great. We decided to use Unity3D as the game engine as all of us had experience in using it, and it allows for fairly rapid development.

This was probably the best choice for the project, as the coder and I found UDK fairly cumbersome to use. In particular I found its audio implementation capabilities hampered by a poor interface, which made adding and testing sound sources cumbersome. While Unity3D’s audio engine is slightly weaker by comparison, my familiarity with it and the ability to test and tweak audio parameters quickly made it the better tool to use. This choice did limit developing the game to a single room of the university which featured the Pro version of Unity3D that provides access to that majority of the audio tools that would be needed, such as filters and custom reverb zones. We also tried to implement the audio middleware package FMOD into Unity3D through the Squaretangle plugin, as it is a far more powerful, far more flexible package. It would also have allowed for a more flexible development process, as we could have transferred to the standard Unity3D engine that we both had free access to. Sadly, the lack of documentation slowed down this process, and despite some initial success we failed to get FMOD working properly in Unity3D. Despite this the process was still worthwhile, as it allowed me to delve further into using FMOD and gain a better understanding as to how it is coded into a project. Similarly I developed my knowledge of Unity3D beyond its audio engine, learning how to implement various basic elements and pre-fabs.

The game looked primarily at auditory navigation through a 3D space through the use of a call and response mechanic. This also allowed for some investigation into the semiotics of sound design, how players perceive the meaning of game world objects through audio cues. I also produced an additional experiment to gauge if Head Related Transfer Functions (HRTFs) could provide an audible difference in sound perception.

This was probably the biggest challenge for me, as I had never looked at HRTFs, auditory navigation, semiotics or game design before. Understanding the basic concepts and functionality of HRTFs and semiotics especially took some time, as the latter has centuries’ worth of research and theory behind it. I feel that trying to integrate all of these points into the project was somewhat overambitious, even if they were necessary in proving results.

I was fortunate to find the Slab3D suite which allowed me to test HRTF functionality without having to create my own, and generally experiment with them so that I could tie the theory and practice together. This was also a bit of a turning point for the project, as it had merely assumed that people could perceive 3D sounds properly, and didn’t take their listening habits into consideration. I decided that this was worth investigating to some degree, and while the number of participants could have been higher the results hinted that HRTFs improve sound localisation to some degree.

The game aspect of the project was a mixed bag. It was massively overambitious and I feel I could have improved and refined the designs further if my time management was not so poor, as they made sense on paper but not in game. The cave section in particular was a particular bugbear during development and was the cause of the biggest pipeline issue. The environment mesh of the cave was considered to be one collideable object in Unity3D as opposed to separate walls and floor. Trying to implement different collision sounds for bumping off walls and footsteps was causing the coder some major issues, until we decided that the quickest, albeit somewhat lazy and broken, way to solve the issue was to lay an invisible floor on top of the original to serve as a different collider for the footsteps. While it does work, there are some clipping issues that are caused when the player jumps up platforms. The pathfinding idea in the caves level didn’t work as planned either, and ended up confusing and frustrating participants more than anything else. I decided that, rather than keeping it as a complete negative, it would be worth investigating why this was the case, and as such used it as part of an analysis on semiotics in sound design. The desert and forest levels managed to serve their purpose, although the former may have been to large an area for some.

Reflecting on the project as a whole, I would have liked to have spent more time into actual game design so that the game artefact was slightly more enjoyable as an experience during testing, and allow me to see what gameplay elements could benefit from audio feedback more. I still believe that my ideas are worth investigating further, although perhaps through better or more refined methods. Similarly, if I were to try and carry out the project again I’d ensure that a suitable timeframe was established and ensure that there were enough team members to carry it out.

Honours Update #20 - Game Testing Results

Although a little haphazard, I managed to get some testing done with the game. The questions were:

  • Are you able to locate sounds in the game? (Y/N)
  • On a scale of 1 to 5 how easy is it to locate sounds? (1  very difficult, 5 very easy)
  • Are you able to navigate through a level using the audio cues? (Y/N)
  • On a scale of 1 to 5 how easy is it to complete a level? (1 very difficult, 5 very easy)
  • Are you able to differentiate between each sound cue? (Y/N)
  • Are you able to associate each sound with a certain object in the game? (Y/N)
  • Which level made the best use of the sound cues? (Desert/Cave/Forest)
  • Why?


For the results:

  • 100% of participants said they could navigate through the levels on sound cues
  • 100% of participants were able to associate a sound cue with a specific object
  • 62.5% were able to differentiate between cues
  • 62.5% said the game was of medium difficulty, 37.5% said it was easy

There were some noticeable issues with controls and the camera, so much so that they had to be fine tuned after the first participant to make the test less frustrating. Players had a preference for the Desert level with all participants disliking the Cave level, mainly due to its different sound cues and length. While the sound cue issues were somewhat expected, the camera, controls and Cave level were all design problems that could have been ironed out with some additional gameplay testing. Fortunately they didn’t impede with the main goal of testing audio feedback.

Honours Report #19 - Game Progress

The game artifact has hit a few stumbling blocks along the way, mainly down to it being developed by a single coder and myself. We’ve had to cut some of the proposed features such as the “focus” skill and 4th level due to the lack of time.

Likewise, FMOD implementation didn’t pan out a smoothly as hoped either. Initially we were limited to 5 instances of FMOD triggered events at one time, which was just enough leeway to apply them to the navigational cues (pickups, bats, monsters). Eventually we had the number of instances limited only by the system’s RAM, before Unity decided it didn’t want to co-operate (a running theme, as the number of issues and errors we had with Unity despite changing nothing were numerous). As such, the game had to be built using Unity’s audio engine, and through some confusing implementation issues (such as the spread controls doing different things to pick ups in different levels) we got something that could be used to test for auditory navigation in games, and to a lesser degree the semiotic sound design issues I considered previously.

I’ll detail some of the more pressing issues with development in an evaluation post after the testing has been finished.

Honours Update #18 - 3D Sound Testing Results

I’ve paused the survey on my 3D sound survey after receiving over 20 responses. As part of the questionnaire I also asked about participants’ listening habits, such as how they listen to sound, and the importance of various aspects of audio in games before the actual sound localisation tests. Participants were also asked to use headphones.

As far as results go, they were fairly promising:

  • General importance of in game audio: 55% said very important, 0% found it unimportant (1-5 scale)
  • How they listen: 36% use TV speakers, 27% used a stereo system or headphones
  • Importance of locational cues: 36% very. 0% unimportant (1-5 scale)
  • Importance of audio feedback v. visual feedback: 59% equal, 18% important, 18% slightly less important, 4.5% very important (1-5 scale)

The sweep test also produce some positive results:

  • 100% heard the noise circle around them (1st Sweep)
  • 67% heard the noise move up and down (2nd Sweep)
  • 86% said the sweeps sounded like they were moving in a 3D space

It shows that by using an HRTF people are able of hearing a constant sound moving around them, and the majority can perceive it shifting in height.

The static sources were a mixed bag. Firstly, there were 4 spoilt results due to people not properly understanding the question. This could have been avoided by trying to gather participants in person, but that was one of the risks discussed previously. Generally speaking:

  • HRTF results were more accurate in 80% of cases
  • 81% stated that the HRTF sounds were easier to locate
  • Easiest to locate were Left and Right
  • Most difficult were Above and Below

And here’s the entire series of results in graph form:


So what can I conclude from this data? While not perfect, there is some improvement in sound localisation via HRTF. Additionally, over a quarter of participants stated that they use headphones while playing games, which, if true across the board, seems like a significant enough number for games companies or audio middleware solutions to invest in full HRTF support.

Honours Update #16 - More Context and Testing Prep

I’ve run my 3D sound test idea past my project supervisor and he’s given it the all clear, so I just need to make it now.

In the process I’ve decided to add some history into what HRTFs actually are into my dissertation. It’s essentially a breakdown of how they and sound localisation work. Quite a bit of the background reading has come from a few more papers and 3D Sound For Virtual Reality and Multimedia (Begault, D.),  as well as reading up on Duplex Theory, which describes how humans perceive sound on a horizontal plane due to differences in intensity and time of arrival.

Likewise, I’d like to discuss the history of 3D audio and audio middleware too. Mainly why it hasn’t advanced alongside visuals and where all the tech has disappeared to. I’ll probably cover the Aureal v. Creative law suit and the technology that both companies were developing at the time, and why this seems to be the wall for audio tech. Karen Collin’s Game Sound provides an interesting general history of game audio, but doesn’t focus too much on the later tech side of things.

Read More

Honours Update #15 - 3D Sound Testing

I theorised that my project relied on people being able to localise sound in a 3D space properly, but did actually have any proof that it was doable. So I’ve decided to carry out a little test to see if :

a) People can perceive sound in a 3D space

and

b) If HRTFs have any effect on their ability to do so.

I aim to use SlabScape and Unity3D to carry out the tests, as they will provide the HRTF and non-HRTF cues.

The first part of the test will look to see if participants can hear a sond moving in a 3D environment. Using SlabScape I will create a sound source that will circle the listener at a constant radius to test for lateral localisation (Left-Centre-Right). On a second pass it will begin to move up and down while maintaining the same radius in order to test median localisation. Participants will be asked for a simple yes or no response if they can hear the sounds moving as describes.

The second part will be a comparison between the HRTF and non-HRTF cues. It will involve the participants trying to guess the location of 10 cues coming from 6 static locations and equal distance from the listener: Front, Behind, Left, Right, Above and Below. After answering each set they will be asked which they thought was the easiest to identify. They will not be informed which set is which.

For carrying out the test itself, I’ll either need to book out a timeslot in a room (which would provide some degree of control over the environment) or host it online somehow (for ease and a potentially greater pool of responses). It also means that if the game doesn’t come together as planned I’ll at least have some test results that I can work into my dissertation.

Honours Update #14 - Implementation, Game Progress and Semiotics.

The coder has managed to get FMOD sort of working in the first level of the game. There are a few tweaks to be fixed such as memory leaks and stability, but all signs seem positive. There is a fairly noticeable difference in locating sounds that are implemented using FMOD compared to Unity, so I’m quite glad that it’s getting there.

In general the cave/maze is designed, and the coder managed to generate it Maya using a script which saved on actually trying to model it all. So now it’s a case of populating and testing that, and getting started on the final level. I’m hoping that the bats idea will work alright, as it does seem a little forced.

There are also sound design considerations to look into there as well. I’ve mentioned this briefly before, relating to the “mushroom” sound in the Super Mario games. The book Communications Acoustics (ed. J. Blauert) has a wealth of interesting topics, but one in particular that relates to semiotic sound design (Chapter 8). Semiotics is the study of signs and how they communicate information. People gather this information from signs through semiosis.

For a basic example, colour coding on food labels. Through experience and pre-defined conventions most people associate “red” with danger and “green” with safe, “yellow” being a middle ground. This provides a code which can then be applied to food content labels. A person would associate a “red” fat section with being bad, but a “green” one as being healthy.

This meaning can be obtained through semiosis as something has been communicated (the %Daily Guideline Amount of a certain substance in the food product), by the use of a code (the colours) that has then been decoded and given meaning (due to people’s colour associations).

When playing Mario, people associate the mushroom with a power up due to these pre-established conventions. The sound acts as the code in this case. Breaking that down further:

Power up (thing to be communicated) > audio cue (code) >  player understanding the message (though established patterns and knowledge)

As such, the sound design is important as it is being used to carry the information and meaning of world objects. By establishing one meaning immediately, such as the pick up response from ringing out, will it be possible for players to react to another sound cue that occurs or associate an appropriate meaning towards it, such as the bats and monsters? I’ve tried to keep them sonically different so that there’s no confusion between the three, but actually testing the semantic links will occur later.

Honours Update #13 - Post Crit Week

Crit week didn’t go too well. I didn’t have enough to present as the project is beginning to become a bit too cumbersome.

On the other side, and in typical fashion, I found two useful things that may help. The first is the Squaretangle plugin for Unity: http://www.squaretangle.com/FMODUnity.html

This apparently allows developers to implement the full FMOD system into Unity. This is a bit of a boon as:

a) Unity’s proprietary audio engine isn’t the greatest in terms of features, with a lot of them being locked to Pro which means the development can only really happen in one room of the uni.

b) FMOD implementation means that I can set up all of the sound cues a lot easier and not have to worry about breaking anything in the project. It also means that we’re not reliant on that one room being free all the time.

c) And probably the most important thing, FMOD has some degree of HRTF implementation! From what I’ve seen before using its Audition3D system it tends to be front and back separation with no real height parameters.


Unfortunately it doesn’t seem to come if any real form of documentation beyond some basic examples, but the coder’s having a look though them just now and hopefully we’ll be able to test something out in the actual game.

The second interesting thing I sound was the Slab3D suite of tools: http://slab3d.sonisphere.com/

Of particular interest is SlabScape, as it has some generalised HRTFs built into it, as well as room modelling and a few other things that aren’t all that relevant to the project even if they are interesting. I’m hoping that I can use it for some form of testing, to see if people can locate sounds at different positions or actually hear sounds in a 3D environment at all. It also comes with the API, but I fear that I’d be asking too much of my coder by trying to force that in as well.

Honours Update #12 - Design Document

This is pretty much a copy and paste from the design document I’ve been working on that details the general concept, level and gameplay ideas that I’ve had so far. I’ve taken on the idea of using a narrator to provide some more context to the game based on some of  the points raised by Allan Milne. Obviously this is still a work in progress, so some things may change.

So far we’ve managed to get the first level built with the “ring out” feature implemented. We’re trying to make sure that the camera’s always centered behind the player so that the direction of cues always matches the way the player’s looking. Next up we’re going to start on the cave level, and see if there is some way to implement an audio based jumping puzzle.

Next week is Crit week, so hopefully I’ll have something to show then. But without further ado…

Design Doc

Long description
What is the game?

  • The game is a 3rd person platformer that revolves around the use of sound as a navigation device.

Why create the game?

  • Even at their most interesting, most audio based games are built around a maze mechanic that creates a very limited and defined play experience. This prototype seeks to create a more open game world for the player to navigate, while trying to create an audio only solution to implementing what are considered basic gameplay features (such as jumping puzzles, determining distance). The prototype also looks to investigate players’ dependencies on visual stimuli over audio stimuli, and shall test to see if the reverse can work as well.

Where does the game take place?

  • The game takes place in a minimalist, surreal region. There are 3-4 main “levels”: The inital Desert level; a maze like Cave level; a creature inhabited Forest level; and a surreal, changing landscape. (final two levels may have aspects merged)

What is being controlled?

  • The player controls “The Pale One”, a humanoid character who wakes up in the first area. They can traverse the landscape, and interact with certain objects/creatures with the “Call Out” ability.

What is the aim of the game?

  • The player has to guide the character through the game to find an object/monolith/TBD. The context of the game is provided by a narrator, who implies that they player is controlling a character out of a fairy tale/fable.



Game System
Player Abilities

  • Movement – Walking, running, jumping
  • “Focus” Ability – Filters and attenuates any non-critical audio (such as ambient noise) to give the objective beacons more prominence. The visual elements of the game are removed while in this mode. This ability should allow the player to judge the direction and rough distance of objectives that may not be apparent in the visual feedback.
  • “Ring Out” Ability – Triggers a sonic response from any objectives or creatures within a set radius of the player. Serves as a more localised pathfinding device, and will be the main focus of the second level as the player navigates through a cave maze, as it will trigger waypoints when activated.
  • Head Tilt – Acts in place of camera controls. Instead of rotating a camera to see where an objective lies, the player can instead tilt their head to hear where it lies.

“Camera” Controls

  • As the game is focused on sound over visuals, the traditional, freely moving 3rd person camera has been eschewed for a semi-fixed perspective behind the character. This is to ensure that the sound field in game in always in sync with the visual elements. The player can instead “tilt their head” to some degree in each direction in order to better determine the direction of a sound. Leaning too heavily to the left or right will cause the character to turn in the required direction.

Pick Up Objects

  • “Call Out” ability/Bell – First item found in the first level. Grants the player the “Call Out” Ability.
  • “Keys” – Pick ups that are used to “unlock” the way to the next area. These will react to the “Call Out” ability in addition to having their own sonic signature.

Creatures

  • “Bats” – Act as way points in the “Cave” level. They react to the “Call Out” ability and move in the direction that the player needs to go. The “bats”’ movement loop should finite, moving from its start position, down the tunnel, then back to its original position where it will remain until retriggered. This is to prevent the player from being confused by multiple “bats” moving around the environment.
  • Enemy creatures/TBD – Appear in the “Forest” level. The player needs to avoid these while navigating the landscape. They will be visually blind, and will instead react to the “Call Out” command, turning their attention to the player as they hunt for objectives and waypoints. The player should be able to out run or evade these by leaving the creature’s “territory” or zone of influence. There will be a sound/music cue that will alert the player to the fact that they are being “stalked” and when they have escaped.

World Objects

  • Waypoints – Serve as sonic beacons for the play to use as navigation. These will consist of “landmarks” that provide the player with a reference point on the level and “beacons”, which are used for finer navigation purposes.
  • “Doors” – Used to separate each level. These “unlock” after the player collects the required pick up item(s).

Victory/Failure Conditions

  • The player completes each level by moving though their respective unlocked gates. The game is finished when they reach the monolith at the end of the last level.
  • Failure can only truly occur in the third level, as creature may attack and kill the player. When this happens the player respawns at the level’s entrance, with the narrator providing feedback as to what has occurred.



Narrative Structure

The Narrator

The narrator is used as a crutch to provide context and feedback to the player, as well as moving the game forward. The use of fairy tale/fable plot provides a reason for there being a narrator in the first place, and any hints or feedback will be framed around an appropriate narrative structure. For example: In the first level the player may not be sure of their commands, the narrator can provide information on what to do (use the focus command) and vocalise why certain events occur during that state (the lack of visuals being attributed to the character closing their eyes).

The Character

“The Pale One” is a humanoid character. He/she should look asexual if possible, and the lack of colour/details can be attributed to the lack of visual emphasis in the game.

Level 1 – The Desert

The game starts with the narrator providing some background, with a brief explanation of the story and why the Pale One wakes up in the desert area. The narrator hints about a sound in the distance, and encourages the player to use the “Focus” command to locate the sounds rough position. The player moves towards the sound, and arrive at a raised platform/rock structure. The player jumps onto the structure and gains the “Call Out” ability. The narrator encourages the player to use their new skill, which resonates with a nearby pick up. After collecting this pick up the player hears a rumble/door open in the distance, and moves towards it in order to progress to the next level.

Level 2 – The Caves

The level starts with a brief narration, explaining the presence of “bats” in the level and how they navigate by sound. The player uses their “Call Out” ability, which triggers the “bats” to act as waypoints, indicating the correct path the player should take. The player eventually arrives at a jumping puzzle, where they have to jump up to and across platforms, picking up collectables and gradually climbing towards the exit.

Level 3 – The Forest

The player leaves the cave, and the narrator again sets the scene, hinting at the presence of creatures in the forest. The player must use their previously acquired skills to navigate though the sonically dense forest in order to find the collectables, while avoiding and creatures or escaping from them. When “Focused” the “stalking” audio/music is also filtered, providing an additional potential challenge to the game. The player cannot casually “Call Out” and “Focus”, or  they will be attacked.

Level 3.5/4 – The Shifting Lands

The narrator comments about the final goal that the player has been aiming towards. The level features dynamically altering visual and sonic elements that should make navigating a challenge. After collecting all items the player must then locate the Monolith/Final object.

Game Play

Actions

The player is capable of moving throughout the environment by walking, or running when the appropriate key/button is held.

The player can tilt their character’s head to assist in determining the direction of a sound. The camera should swing slightly in the direction the player chooses when this happens.

There are 3 actions:

Jump – The character jumps. This action is triggered per button press.

Focus – The visual elements of the game fade out, along with any non-critical/environmental audio. This action accents the sounds of any objectives in the area that the player may be too far from to hear or be unable to locate due to the sonic field being cluttered. This mode is toggled on and off.

Call Out – Used to locate objectives or trigger events that are within a certain radius the player. Any objectives/pick up items should emit an “echo” response. The “bats” of the Cave level will also audibly respond to the “call out” and move down their respective tunnel to ensure that the player is heading in the correct direction. The “creatures” of the Forest level will begin to stalk the player if triggered by the “call out” command, and will continue to do so until the player leaves their territory/zone of influence. This action is triggered per button press.