Origin Story

Several weeks ago, we were in the middle of releasing multiple apps like Shengji and Camp. We went to the South Park Commons opening night for the new office and met a new friend, Alex, who we immediately got along with talking about various consumer products. A week later, he asked if we wanted to join him in a hackathon later in the month and we were down.

During this time, we got pretty excited about the release of the Apple Vision Pro. So, we created Stock Tickers+ before even having access to the device 😂. Coincidentally, Alex also had a Vision Pro and we started thinking about what could be possible leveraging the device.

We met up and tried the Apple Vision Pro for the first time. It was incredible, especially the hand and eye-tracking, which was quite novel compared to other AR/VR devices on the market. We were surprised that you could touch your fingers together to produce button clicks, swipe different screens, and press things in midair. The setup for eye and hand tracking was very seamless as well. We also tried the dinosaur app, which seems to be a canonical experience of testing the Vision Pro where you get to see a butterfly within your grasp and dinosaurs fighting with each other. So, we followed our curiosity and started our journey into Apple Vision Pro development from there.

For the hackathon, we had to leverage something related to OpenAI's APIs, so we decided to combine it with the novelty of the Vision Pro. So, we started thinking about what the future would look like. Wouldn't it be cool if we had 3d characters that we can really hang out with, keeping track of everything happening over a timespan of days? In the past, there were things like Tamagotchi, which were fun for folks to grow, care for, and enjoy as they grow from their baby forms to adults. With large language models, we have oracles that respond to you like the old NPC characters from the RPG games of our youth. Now, it seems like those stories are more and more real and a virtual/augmented reality world is almost within reach.

So, we leveraged the device and OpenAI APIs to see if we can make an immersive experience. Our project consisted of a set of scenarios with NPC-like characters like Zoltar (thank you to Jonathan Wegener for introducing this concept to us), legendary investors, and professional debaters. We were aiming for entertaining experiences that let you engage in a natural way producing a semblance of that here:

After producing this in 48 hours, we started ruminating on what was next, since this showed us that there's a lot of potential in spatial computing. We were still excited about the potential of 3D characters that you can interact with.

So, we followed the thread of making a single verticalized experience to see how far that would go. We spent a week planning a narrower set of features and figuring out how to spread our latest iteration. That is now Buddy which is now available on the App Store.

Buddy is the first 3D character companions on the Apple Vision Pro that you can talk to. What we found is that having something real makes it more compelling and desirable for you to talk and respond. It's much more fun than just talking to a text box.

The personification of Buddy really makes it feel like you have a true companion by your side. With animations that match the emotion of a response, we have enhanced the experience of having a companion. You can also tailor Buddy to your specific needs and preferences, ensuring that your friend is the perfect match for you.

Future

We are currently experimenting with iterations of Buddy and want to see how far we can push this concept. There are a lot of potential avenues related to smart pets, more 3d characters, and much more to come.

We believe that the AR/VR space is really early. The Apple Vision Pro has gotten a lot of things right, but it's at an expensive price point and needs a few more iterations to really become mainstream. The latest advances in large language models have helped signficantly and there is a world of possibilities to explore. What an exciting time!

Shoot me a message at [email protected] if you're interested in chatting or if you have any feedback for us.