HoloLens UX Design Challenge, 2017


NYC Media Lab Summit 2017 Demo, Sep 28th 2017

Featured on NYU Tandon School of Engineering News : Visionary VR/AR Projects From Tandon Community Featured At NYC Media Lab Summit

We demoed our "HoloStudio" at NYC Media Lab Summit 2017 and received a lot of feedback. 


Project Goal

This project goal is aiming to bring an artist gallery or studio work into the holographic world.

During the initial steps for the project, our group members individually went to the Microsoft Store for the 1-hour demo sessions with the HoloLens. From this experience, we learned a lot about the limitation and capabilities of the HoloLens and tried to synthesize these experiences into something that we could use for our project. We also developed these project goals to help guide us:

  1. Create a simple and clear way for a curator to view an artist’s work.

  2. Provide a consistent experience when interacting with holograms.

  3. Design an unobtrusive and intuitive way to navigate the experience.


At the end of the semester, we demoed our project to Nicholas Kamuda, Creative Director for Microsoft HoloLens and Windows Holographic at Microsoft. He also delivered a speech at NYU Tandon's MakerSpace about what work and projects Microsoft HoloLens currently are working on and also his thoughts about mixed reality.

Brainstorming & Concept Map

During Brainstorming, we developed a concept map of the many ways we think users could interact with the holograms and other features. This map is probably going to guide us when we start talking to curators and artists. We also sketched a few of the ideas to help us visualize how this would work. From there, we created an initial Table of Elements to help us determine what we wanted to include in our experience.

Concept Map

Concept Map

Table of elements for the design

Table of elements for the design


As for design techniques, we plan on focusing on World Locked Presence, True Scale, and Persistence, as well as Waypoints and Sense Ratios. We hope that with these techniques in mind, we can combine it with a few usability practices (like having minimal UI elements so the user isn’t too distracted and having audio and visual cues available so the user knows when things are happening in the experience) to fully flesh out this design challenge.

Prototype & Iteration  

For the prototype we made, I think what we did was rally helpful for the project. Before making the first prototype, we still spent much time discussing different approaches for designing the project, even though we already had some ideas last week during the brainstorming.

After that, we came out some simple user flow and based on that user flow we started to make our first prototype. I must say it was a really fun experience prototyping such mixed reality design. Only by using some paper and drawings could clearly show our ideas. I think based on this prototype, we could go talk to more people and get feedbacks for next iteration and further digital prototypes.



After showing our design idea with the first rough paper prototyping video, we gained some good feedbacks. So this week, we tried to think out of box and made some iteration of the design. The idea we came up was having a digital assistant, floating beside the user during the whole mixed reality experience. In order to cut down the user interface elements, we tried to take the advantage of 3D sounds and the voice control. So basically, the digital assistant would guide the users and give instructions at the beginning, then the users were allowed to walk around to see the art pieces by themselves. If they had more questions, they could ask for help from the assistant at anytime.

We made another paper prototyping video of this idea, and also conducted our first user testing this week. The users did give us some feedbacks about the digital assistant, for instance we need to be careful for designing the image of the digital assistant, not to let it distract the users while seeing the art pieces. Over all, the users said they like the idea of having a guiding assistant, and the user flow was quite smooth and easy to understand. This idea made me think of the little alien baby character in the video game in the movie “Her”. I really like how that small guy talked to Theodore. Talking to such a animated avatar was funny but would be really cool to have one. I believe the idea having an A.I. digital assistant was not new, but how to implement this idea into designing a product and let the users enjoy it was another challenge. I think we would conduct more user testing and see if this idea would be our final design. Also, how to develop this concept would be another challenge for us too, and hope we could overcome the technical issues eventually.



After having those insights and the feedbacks from our user testing, which we let four different people try our first paper prototype, we started to work on the iteration for second prototype. The major changes we made was taking off the adjust buttons and putting the little signs instead. Also, we made a new version of our digital assistant “Arty”. We drew several possible solutions for Arty, and in order not to be too distractive to the users, we designed it like a flying little ball with two wings eventually. We haven’t tested to other users with the new Arty yet, will like to do that this week to see how people think about it.

Also, we decided how would the further information be shown. We would bring the users to another virtual room, without other artworks, only with the one they select to see further information and its information would be displayed in that room, and they could go back to the gallery anytime when they finished seeing the whole further information, like the progress of that artwork.

We tried to user test our second prototype, and gathered more feedbacks. To be honest, I found it interesting and difficult to conduct such mixed reality experience user testing with only paper prototype. It would be easy to show people your idea through filming a short prototyping video. People only need to watch what happened in the clip and they could get the idea. However, if we want to let users try out by themselves, many details need to be thought and concerned. We need to tell the testers what exactly they could do and what they couldn’t do, and at the same time, we also need to make sure not giving too much information, because what we’re looking for is the fresh experience and feedbacks from them.

I think we received some good feedbacks from the testing and a lot of them we did not think about before. At  this stage we could not make too many changes, but we still tried to make some iteration based on those feedbacks, for instance, the design of how to show the menu buttons. Also, we got pretty good feedbacks from user testing about the digital assistant idea. People love to interact with Arty and they depend on it a lot. That was an interesting takeaway for me, because sometimes even though I think Arty, the digital assistant idea, was cool and fun, I still have some concerns about how willing and comfortable will people have interacting with Arty when designing it. And this questions then lead me to think more about how important it might be for having a digital assistant in mixed reality world. On mobile phone, I don’t think people rely on digital assistant a lot. A lot of big tech players are pushing this idea, for instance: Siri, Google Assistant, Cortana, and Alexa. People are getting more familar with the digital assistant and even though not using them often, but I think in the future, in virtual reality and mixed reality world, people will get more comfortable with using digital assistants and using them more frequently.



Painting an Artist Information

Painting an Artist Information

Performance and Installation 

Performance and Installation 



We made some render images to visualize how our idea of having a digital assistant Arty in holographic studio would be like, and also how the user could interact with the holographic art works, including sculpture, painting, installation, video, and artist information.

We also made a visual flow instead of traditional user flow design to visualize how the whole experience would be like in our application HoloStudio.


Check our Final documentation for the project: Hellolens Book, Holographic Studio


Group members: Brian Hui, Alexandros Lotsos, Tyler Mechura, Jing Huang, Ankit Ruhela

User Experience Design, Instructor: Dana Karwas