About Project H.In.T

Project Here In Time started as the design project of Akshita Mehta and was continued in her graduation project. I was involved in both the iterations from beginning of prototyping till the end. I contriuted in the project by providing several key insights in areas of interaction design, experimentation, narrative-building, prototyping and testing apart from the development of the project in Unity.

Architecture always means an unchanging deep structure (Brand, 2012). That is, a building is always building and re-building. The central message being that finishing is never finished. A building continues to build itself throughout its life.

So what happens when you stop using the architecture that was built? How do you continue to take them into the future with you? What values determine whether a building or structure should be taken into the future or should it be replaced? These questions became the starting point of this project.

Immersive technology has now achieved sustained momentum and is being applied to many domains creating unforeseen embodied experiences. Can such a technology be applied to heritage that can enable a positive association with our past?

Project Here-In-Time is an exploration of creating such an immersive on-site mixed reality heritage experience that enables the visitor to explore the heritage site in a unique and engaging manner.

Key insights from the previous prototype

Project Here-In-Time - Qutub Minar is the rewamped version of Project Here-In-Time - Adalaj. The core idea is the same - to create an immersive on-site mixed reality experience that enables the visitor to explore the heritage site in a unique and engaging manner.

Project Here-In-Time - Adalaj was a successful first prototype. The gameplay was exciting that created space for a new medium of interaction with the space. Using Vuforia for unity, image targets were collected from site and a narrative was weaved around it. The objective of the project was to navigate through space orienting oneself to the order of the architecture that itself narrates a story of the time gone by.

About Qutub Minar

This 72m ornate tower was built by Qutab-ud-din Aibak, first sultan of the Delhi Sultanate. The tower was built to celebrate Muslim dominance in Delhi after the defeat of Delhi's last Hindu ruler. A site area of 27 temples was demolished to make this complex and only an Iron pillar was left as a remnant of Hindu raj in the region previously. Its construction continued over 3 generations each making additions to the site.

The Qutub Minar complex is a site of multiple narratives spread over several years and even multiple dynasties. While there is visible evidence of some of these narratives, many are presumed or even debated by conflicting schools of thought.

Project Development

A narrative was built arround 'Dr. Q' and 'Time Machine' after assessing various narrative styles and themes.

“Your dear friend Dr. Q, has been mysteriously pulled into the past. You must rescue him! He can communicate with you only through this phone. Follow the instructions he gives you to find him as soon as possible... If he is not found in time, you will lose him forever! “

The narrative was constructed combining several sections which incorporated stories of various past rulers. These rulers will riddle players and upon giving the correct answers the players will be awarded with new 'Hints' which will lead them to the next spot in the heritage campus and closer to finding Dr. Q.

The 3 different steps to complete on-site tasks

1. MOVE

The expanse of the space available for traversing is vast and thus arose the need for some kind of navigation assistance. The narrative has to respond to the technique chosen for this navigation. We developed 2 prototypes to test this:

a) The first was a SLAM (Simultaneous Localization And Mapping) based navigation assist, where a preloaded map on the device continues to show the current location while the target location is already mapped in 2D with the specific locations or routes laid out for the player to follow.

PROS -
Does not rely heavily on the specific GPS coordinates. Once loaded, the navigation is simple and does not require extra computational processing on the device.

CONS -
1] The map takes up heavy real-estate on the precious little screen space available on the device.
2] The tracking is not accurate and reliable. The player is often shown hitting solid walls and triggers jumping movement patterns.
3] The map for Project H-In-T will have to be pre-loaded with the desired locations already drawn out for the player on the static map. This is likely to substantially diminish the experience of the gameplay

b) The other method that can be used to navigate is via GPS based tracking. Once the target location is known to the player, one can follow any suitable route to fall within a specified radius of the target in order to continue playing. This technique requires the use of location tracking abilities of the device.

PROS -
1] The player is free to move within the site as they desire.
2] Taking advantage of the scale and the good connectivity with gps satellites under the open sky, the tracking works with sufficient reliability.
3] Every location the player finds is a reward and is likely to motivate them to explore further. This is likely to substantially diminish the experience of the gameplay

CONS -
Extreme dependency on tracking and computational power of the device.

2. FIND

The location tracking ability only enables the device to know where it is in space. What the user is looking for to trigger the device to have found a specific location, is of equal if not of more importance. The user will have only found the correct spot when the point of interest becomes known to the device. This section explores what these points of interests can be exactly and which aspect of computer vision can be deployed for making them visible to the device.

a) A primary approach can be to search for a specific tangible element on site - a motif, a structural component, a combination of carvings etc. that is a valuable piece of information which the player might not otherwise see. Capturing these exact frames will trigger an action for interaction with the player. This form of interaction was used in the first version of Project H-In-T using Image targets with Vuforia in unity.

PROS -
Can be pin-pointed and will always be specific points in space.

CONS -
1] Given the scale of the space, it can be difficult for the player to search for a very specific element on site.
2] This interaction leaves little scope for what entails after the specific frame has been found. As output will have to be drawn over the currently tracked frame.
3] Most crucially, it will be difficult for the player to position and capture the frame exactly as required without assistance.
4] The narrative of, and, the trackable elements found on, the site do not compliment each other appropriately to enable a meaningful digital experience.

b) Another method that can be applied is location triggered virtual objects. These virtual elements can take 2 forms - Floating / located at a GPS coordinate, like the floating Pokemon in the AR game, Pokemon Go; Or can be attached to a physical element on site. Leveraging the technology of AR core, these virtual elements can be attached to a surface/Image target to become fixed to a physical point in space. The ‘Pokemon Go’ interaction type is already well known and therefore the physically attached virtual objects were tested with prototypes.

PROS -
1] Allows greater flexibility in movement and is not very strenuous search. Thus will require lesser instructions to be designed for directing the player to the point of interest.
2] Availability of AR Core/AR Kit is an advancement from Vuforia which can now detect surfaces that can be used as anchor points. This makes the experience very intuitive and opens a world of possibilities for content representation.
3] This type of search also adds to the excitement of gameplay as every found location will reveal something unknown rather than being an overlay of information.

CONS -
1] Reliability of GPS tracking is not always accurate without additional hardware support.
2] To use ARCore/Kit capabilities, an additional step of finding a surface has to be followed before any virtual element can be augmented.

3. LEARN

The primary objective of Project Here-In-Time is to foster associations with the space they belong to. This means there needs to be something over and above what the user can see themselves already which adds to their presence. While the location tracking and the search technique can be optimzed, the whole experience will only come together when the scene plays.

The factor to keep in mind for designing this scene is to not only provide information, but to provide it in a manner that leads to a revelation!! The player must be awed when the scene finishes and they take back with them something that they had never processed before.

This was a critical learning in the project and went on to influence the script and interactions within the game experience substantially. The feedback of the users also assisted in validating the final use of certain interaction types.

The experience

Having undergone various iterations, the final heritage game experience of Project H-In-T wraps around a simple game mechanic. This choice was based on the fact that there already exists a complex and new medium of interaction with the space for the player to learn on-the-go because of the introduction of AR Core technology.

The learn-ability of the plot and the game objective therefore was kept at minimal cognitive load. This simplicity in turn adds to the experience as the player is now cognitively free to immerse itself into the journey laid forward by the game. It also allows the user to shed-off the need to make navigational choices while on site as the mechanic itself dedicates to this task. The emphasis instead is on critically thinking about the space and its architecture, materiality, technology and history.

To further the concept, the game audio was also selectively chosen to reflect this theme. Credited to Bensound, the track ‘Deep blue’ was manipulated and run on loop as the background score. The repetitive mysterious rhythm creates a sense of inter-dimensional experience while resonating the feeling of being trapped like the state of the game protagonist, Dr. Q. There is a lightness in its overall feel in spite of having a heavy undertone, complementing the game mechanic.

Introducing the game : Story Prelude

Familiarization with UI Design schematic

In the next scene, the basic UI of the game appears and the audio overlay of the character’s voice is used to establish a new addition to the means of communication. This mechanic reinstates the players acceptance as the savior and marks the beginning of the exchange between the game and itself. The use of the icon as a button and the introduction of the progress bar gives the player an understanding of how to navigate through the game.

Part 1 - Find

The game cycle begins immediately after. The first task of the cycle is to decipher the hint given by Dr.Q and identify the target location. A animated vector also hints towards the required action and provides information regarding the players current differential with the target. The progress bar also indicates the find status with a brown dot. Once the player is at location, a pop-up indicates the players successful arrival and the progress bar also reflects the active status in yellow. As the player successfully completes the tasks, the progress updates to green and moves ahead.

Part 2 - Do - Using ARcore

The scene is set in the past and the player accesses that dimension by traversing through the portal. The portal reveals a mystical dimension of the past where it meets relevant characters and solves puzzles in order to get more information about its trapped friend Dr. Q and move forward in the game.

Part 3 - Revelation and learning

Drawn into the world of the past, the player is greeted by a fictional representation of Qutub-ud-dinAibak. At this point, the experience of the game switches and the interaction between the player and the game turns into a trivia. The player now faces a question raised by Aibak that their presence on the site can assist in answering. It allows the player to become self motivated for the discovery of the lesser known fact while also making them feel satisfied and fulfilled on having given the right answer. This methodology not only gives the critical thinking reign in the players hands it also pushes them to think beyond the surface and front of them from an analytical and observational point of view.

It is not a linear mode of transferring information to the player and instead gives room to the player to think for themselves. And when this critical thinking is rewarded, the achievement is emotionally celebrated and gets embedded in the players mind as a memory. In the long run, these visual memories can allow the user to translate and retain their knowledge for much longer and thus create more valuable, collective and shared memory constructs.

Part 4 - Reward

Game Complete

When the player has successfully traversed through the whole site, following the footsteps of Dr. Q, before Dr.Q’s life is over, he can finally meet the protagonist at the end and complete the game. A full figure of Dr. Q is revealed jumping with joy. Leveraging the tech of the game itself, Dr.Q appears to occupy the same physical space as the player and can be interacted with spatially.