ChatGPT can have a significant impact on XR development by helping developers to work more efficiently and effectively. ChatGPT can assist XR developers in so many ways, you should go at it! Why?
This article looks at how ChatGPT can make Virtual Reality and Augmented Reality development easier, more efficient, and faster:
Ultimately, I will add my view on whether ChatGPT will change my daily life as an XR freelancer.
ChatGPT is a cutting-edge AI tool created by openAI. It was released in November 2022 and gained 1 million users in just five days. GPT-3 can understand the context of the conversation, which is a significant advancement in chatbot technology. We can take advantage of this ability and “teach” ChatGPT to get our desired results.
ChatGPT can assist XR developers in several ways to produce XR prototypes and applications faster.
ChatGPT potentially enhances the way XR developers search for and access information. While it only partially replaces traditional search tools like Google, Stack Overflow, etc., ChatGPT will become a valuable search tool.
Without further ado, let us take a look at the following examples:
Source: ChatGPT v3, OpenAI
This is already a good starting point. However, whereas ChatGPT is right about OpenCV, we wish it was already possible to access the camera feed to run computer vision algorithms in an XR project that targets the Oculus Quest. ChatGPT’s last suggestions (Leap Motion, OpenHandModel) look weird, especially the latter one XD it started “hallucination.”
So, let us ask a more specific question:
Source: ChatGPT v3, OpenAI
Source: ChatGPT v3, OpenAI
Instead of using OpenAI’s UI for ChatGPT, we can even speed up such research by using a web browser extension and sending such requests while googling. As you may see, we can get code snippets, explanations, and a summary of how to debug and fix issues along the search engine results.
Source: ChatGPT for Google, browser add-on. See the appendix for further links
Source: ChatGPT for Google, browser add-on. See the appendix for further links
ChatGPT can generate scripts for VR environments and interactions, allowing developers to prototype and test ideas quickly.
For the following example, we fed in the below prompt to generate a C# script for a Unity3d and XR Interaction Toolkit project.
Source : ChatGPT, OpenAI
ChatGPT generated the below output:
First, an abstract.
Source : ChatGPT, OpenAI
Followed by the code. In this case, an entire C# class is ready to use in Unity3d. It generated even two lines to unregister Listeners
and a line to do a null
check.
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.Interaction.Toolkit;
public class CreateAndChangeColor : MonoBehaviour
{
public GameObject myCubePrefab;
private GameObject myCube;
public XRController leftController;
public XRController rightController;
public float scale = 0.5f;
public Color startColor = Color.red;
public Color endColor = Color.blue;
private void OnEnable()
{
leftController.onTriggerClicked.AddListener(InstantiateCube);
rightController.onTriggerClicked.AddListener(ChangeColor);
}
private void OnDisable()
{
leftController.onTriggerClicked.RemoveListener(InstantiateCube);
rightController.onTriggerClicked.RemoveListener(ChangeColor);
}
public void InstantiateCube(XRBaseInteractor interactor)
{
myCube = Instantiate(myCubePrefab, leftController.transform.position, Quaternion.identity);
myCube.transform.localScale = new Vector3(scale, scale, scale);
myCube.name = "myCube";
myCube.GetComponent<Renderer>().material.color = startColor;
}
public void ChangeColor(XRBaseInteractor interactor)
{
if(myCube != null)
{
myCube.GetComponent<Renderer>().material.color = endColor;
}
}
}
And finally, it also output a short explanation and “How-To” guide to let us know what should be done in the Unity Editor:
Source: ChatGPT v3
Of course, we should always use our judgment to make sure that the code and the instructions are correct. Thus, take the result as a starting point and maybe a time-saver.
We gave ChatGPT a hint and the following task to correct a few lines of code:
Source: ChatGPT v3
The relevant code snippet modified by ChatGPT looks as follows.
private void OnEnable()
{
leftController.onSelectEnter.AddListener(InstantiateCube);
rightController.onSelectEnter.AddListener(ChangeColor);
}
private void OnDisable()
{
leftController.onSelectEnter.RemoveListener(InstantiateCube);
rightController.onSelectEnter.RemoveListener(ChangeColor);
}
Source: ChatGPT v3
Another use case of text-based scripting with ChatGPT is the generation of Unity Editor scripts. For example, a small tool to count the polygons of a mesh is useful while optimizing an XR app.
Source: ChatGPT, OpenAI
Source: ChatGPT, OpenAI
Neat. However, ChatGPT-3 is known for its ability to generate persuasive yet incorrect answers. Stack Overflow banned users from posting replies or comments produced by ChatGPT. (AI-generated answers temporarily banned on coding Q&A site Stack Overflow )
The moderators of the platform explained:
The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good.
Sample: On-the-fly code generation …
Now, let’s take a look at a more advanced use case, a prototype by Dilmer Valecillos. His prototype demonstrates that we can utilize ChatGPT to generate on the fly code for use in Unity3D.
After setting up a Python environment including ChatGPT Wrapper and dependencies, we create a ChatGPT Web Service (API) in Python. This wrapper will run a headless browser using Flask. ChatGPT Wrapper
To test the communication between ChatGPT and the custom web service, we set up a VS Code REST Client. The service sends prompts to ChatGPT and, in return, receives code from ChatGPT that is generated based on the criteria included in the prompts. All data from and to Unity is sent as JSON data.
The prototype calls the custom service from within Unity to communicate with ChatGPT.
After parsing this JSON response, the generated code is then run in Unity at runtime. For example, we let ChatGPT generate a Monobehavior
script to instantiate a sphere and rotate it over time which is then added to the scene and executed. With Dilmer’s prototype, we even let ChatGPT generate a script to import a car asset randomly picked from Sketchfab, to instantiate and animate it.
The prompts for ChatGPT are created at runtime in a separate component. They specify the task, the name of the generated script (class), method, parameter, and entity names, as well as animation parameters. A set of custom reminders and rules is appended to each prompt to ensure that the generated code is formatted correctly, includes required namespaces, and uses correct types.
The prototype also includes a feature that allows the user to retry the code generation process up to three times in case of errors. This feature is useful for ensuring that the generated code is functional and can help to mitigate errors that may occur during the code generation process.
In conclusion, the prototype demonstrates the potential of using ChatGPT for on-the-fly code generation in Unity3D. It allows for the automation of repetitive tasks and can be used to speed up the game development process. The use of ChatGPT in this way opens up new possibilities for automating and streamlining the process of creating Unity scripts.
Dialogue Creation
To enhance the user experience, ChatGPT can generate dialogue and voice-over scripts for XR characters and language learning.
Incorporating a text-to-speech system, which converts the chatbot's written responses into speech, allows the app user to listen to the chatbot's output text rather than read it. The advantages of using text-to-speech with ChatGPT are clear:
Firstly, it makes the chatbot more accessible to users who prefer auditory information overwritten. This can be particularly helpful for users with visual impairments who may have difficulty reading text on a screen. Additionally, text-to-speech can make the chatbot's responses sound more natural and human-like. The conversion allows for the inclusion of appropriate inflections and emphasis, making the conversation feel more authentic.
We used the Oculus Voice SDK for the following test to convert the user’s speech to text. This text is fed into ChatGPT as a prompt. Once ChatGPT returns its output as JSON data, we parse it, convert it to an audio file and play the latter. Let us watch it in action:
Sample audio files generated on-the-fly in Unity with ChatGPT and Text-to-Speech:
For a more advanced use case, check out the video by Sercan Altundas (Ready Player Me) who creates smart NPCs (non-player characters) using Ready Player Me avatars, Unity Starter Assets, and the OpenAI API
ChatGPT can generate text and visual descriptions for XR environments, objects, and characters, which can be used in mood boards as reference material for artists and designers.
However, creating art prompts can be challenging, but with the help of ChatGPT, it can become much more manageable. ChatGPT is a powerful language model that can generate creative and detailed text descriptions, a crucial aspect of writing art prompts.
One effective strategy when writing art prompts is to divide them into four groupings:
Source: Berenice Terwey
By breaking it down this way, it becomes easier to focus on specific aspects of the prompt and add more detail.
For example, adding more detail would lead to more exciting and creative results when it comes to the description. We can ask ChatGPT to generate more detailed text descriptions to achieve this.
Once we have a detailed text description, we can feed it into another AI that generates images, such as DALL-E (also by Openai), Midjourney, and Stable Diffusion. With this combination, we can generate various visual styles and compositions, making developing new and exciting art prompts easy.
Another advantage of using ChatGPT is that we can continue asking for more information, such as different lighting, illustration styles, or composition ideas. With its capabilities, ChatGPT can help us create art prompts that later result in detailed and visually exciting imagery generated by another AI.
Results of a quick test with ChatGPT and Midjourney
To start, we fed our prompt to Midjourney:
image is a panoramic, spherical, 360° view taken from the top of the tower "Alter Turm", in xxx Germany. It is a sunny day, blue sky view on the streets of xxx. photorealistic style. spherical, 360°
Source: Midjourney
We asked ChatGPT for more detail, it outputs the following, which we then input to Midjourney:
Source: ChatGPT, OpenAI
source: Midjourney
ChatGPT is a powerful tool that can assist in creating art prompts, particularly for XR developers and XR designers. Its ability to understand and interpret human language allows for the generation of detailed art prompts to generate visually interesting imagery through other AI systems.
Quick Intersection: How to write prompts effectively
ChatGPT could assist in testing and debugging VR applications by identifying potential issues. It has promising potential and might set a standard for (low-code) testing solutions to many issues in the future (with emphasis on “future”).
For a simple test, we fed our basic OpenAIClient class into ChatGPT, together with the following prompt:
(NB: This is a very basic class, do not use it in production)
public class OpenAIClient : MonoBehaviour
{
private const string URL = "https://api.openai.com/v1/completions";
[SerializeField]
private string _modelName = "text-davinci-003";
[SerializeField] private InputField _inputPrompt;
[SerializeField] private InputField _inputResults;
[SerializeField] private GameObject _loadingIcon;
[SerializeField] private GC_TextToSpeech_Chatgpt _textToSpeech;private string _apiKey = null;
private bool _isRunning = false;
public void Execute()
{
if (_isRunning)
{
Debug.LogError("already running");
return;
}
_isRunning = true;
_loadingIcon.SetActive(true);
if(!string.IsNullOrEmpty(_inputPrompt.text) && _inputPrompt.text.Length > 10)
MakeRequest();
}
private void Start()
{
LoadAPIKey();
}private void MakeRequest()
{
RequestData requestData = CreateRequestData();
string jsonData = JsonUtility.ToJson(requestData);
byte[] postData = System.Text.Encoding.UTF8.GetBytes(jsonData);UnityWebRequest request = UnityWebRequest.Post(URL, jsonData);
request.uploadHandler = new UploadHandlerRaw(postData);
request.downloadHandler = new DownloadHandlerBuffer();
request.SetRequestHeader("Content-Type", "application/json");
request.SetRequestHeader("Authorization", "Bearer " + _apiKey);
UnityWebRequestAsyncOperation asyncOp = request.SendWebRequest();
asyncOp.completed += (op) =>
{
if (!string.IsNullOrEmpty(request.error))
Debug.LogError(request.error);
else
{
if (!string.IsNullOrEmpty(request.downloadHandler.text))
{
OpenAIAPI responseData = JsonUtility.FromJson<OpenAIAPI> (request.downloadHandler.text);
// clean up, starts sometimes by 2 empty lines
string generatedText =responseData.choices[0].text.TrimStart('\n').TrimStart('\n');
_inputResults.text = generatedText;_textToSpeech.Synthesize(generatedText);
}
}
Reset();
};
}
private void LoadAPIKey()
{
var keypath = Path.Combine(Application.streamingAssetsPath, "apikey.txt");
if (!File.Exists(keypath))
Debug.LogError("Apikey not found at: " + keypath);
_apiKey = File.ReadAllText(keypath).Trim();
Debug.Log("API key loaded, len= " + _apiKey.Length);
}
private RequestData CreateRequestData()
{
return new RequestData()
{
model = _modelName,
prompt = _inputPrompt.text,
temperature = 0.7f,
max_tokens = 256,
top_p = 1,
frequency_penalty = 0,
presence_penalty = 0
};
}
private void Reset()
{
_loadingIcon.SetActive(false);
_isRunning = false;
}
}
For example, we expect ChatGPT to tell us that
UnityWebRequest
instanceStreamingAssets
UnityEngine.JsonUtility
is not the best library to parse JSON dataHere is the output:
UnityWebRequest
which is correct, but it did not recognize that we call SendWebRequest()
on the request
object that returns a UnityWebRequestAsyncOperation
:UnityWebRequestAsyncOperation asyncOp = request.SendWebRequest();
StreamingAssets
folder.request.error
is null or an empty string, when the UnityWebRequestAsyncOperation
has completed. But, we could improve the error handlingUnityWebRequest
is detected by ChatGPT.Though not all of the suggestions by ChatGPT are correct, they may give valuable hints and point out how the code could be improved.
Regarding Unit Testing, we will skip this for now because it does not make much sense to test this on the above OpenAIClient.cs. Unit tests cannot be applied to asynchronous methods. But, I want to point out an article about a Visual Studio Extension that links ChatGPT to this IDE to generate Unit tests.
Writing code documentation is tedious and time-consuming for developers, particularly for large and complex projects. One of the main challenges that product teams face is keeping the documentation up-to-date, consistent, and easy to understand, which requires a lot of time and resources. But what if they could use a tool that could assist in creating and maintaining their documentation? That's where ChatGPT might come in.
One of the ways ChatGPT can assist developers is by providing instant access to relevant information and examples. For example, when a developer team writes documentation for a particular function, they could query ChatGPT for details on how to use that function or method. ChatGPT would provide the appropriate syntax and examples. This saves time and effort, as developers would not have to search through multiple sources or rely on their memory to recall the correct syntax and usage.
ChatGPT could also be helpful by providing suggestions for common pitfalls and mistakes to ensure that the documentation remains comprehensive and easy to understand. For example, ChatGPT could provide suggestions for handling common cases, such as invalid input or missing arguments, if a method takes an argument.
ChatGPT could also be used to verify the accuracy of existing documentation. For example, if developers are unsure about the syntax or usage of a particular function, they could ask ChatGPT for confirmation and clarification.
In complex projects, multiple contributors are working on the creation of documentation. It is challenging to keep a consistent style. ChatGPT could check for spelling and grammar errors and suggest templates and boilerplate text. This helps ensure that the documentation is high quality and easier to understand.
Let's do a small test and let ChatGPT generate documentation of our above OpenAIClient
script.
We fed in the following prompt:
And without further ado, here is the generated output:
This looks promising for more complex projects since the generated descriptions and comments are correct. The output has already formatted text and it provides suggestions for common pitfalls and mistakes.
As a freelance XR developer, I am responsible for implementing XR projects and the ideation, deployment and roll-out, and communication with clients and stakeholders. Just as written in the six steps above, I have utilized ChatGPT as starting point throughout the entire XR development process, from ideation to deployment: I rely on ChatGPT to generate ideas and inspiration for the game and level design, as well as for creating content for team communication with designers and presentations for clients and stakeholders. Additionally, I utilize its capabilities to troubleshoot code and Git issues, generate code snippets, optimize and document code, and even create and modify content within Unity XR projects in real-time. Furthermore, I let ChatGPT generate the necessary documentation and required content for the roll-out of XR applications.
We must remember that ChatGPT v3 is trained on data until 2021. It may need to be fully aware of the more recent updates and advancements in SDKs, frameworks, and APIs. However, we can continuously update its knowledge by providing it with the latest information to teach it accordingly.
Summa summarum, ChatGPT, and generative AI has significantly enhanced and accelerated our entire XR workflow.
As ChatGPT continues to evolve and improve, it will become increasingly capable of generating more human-like responses and understanding the context of conversations. With a more powerful and efficient GPU, the model can process data faster and produce more accurate results. These advancements will benefit XR developers as they continue to create immersive experiences.
In the future, ChatGPT and other generative AI will become essential tools for scaling the creative process and iterating on ideas. While coding will remain essential and generative AI tools will only partially substitute for human input, expertise, and judgment, combining multiple generative AI outputs will replace much of the repetitive work that needs to be considered creative. This includes tasks such as testing, deploying, and writing documentation.
To be a successful XR developer in the future, having a potent combination of deep domain expertise and an action-oriented prototyping mindset will be more than ever crucial. Understanding various generative tools and how to incorporate them into XR applications effectively is essential, from the initial stages of ideation to deployment. This includes having a holistic understanding of the relationships and interfaces between different components and XR-enabling technologies, such as 3D engines, micro-services, serverless backends, visual scripting, digital twins, and AI assistants, and how they can work together seamlessly within the larger XR ecosystem.
As a final tip, I recommend using ChatGPT wherever you can starting from now (it should be easy with all our tips above). You will gain a competitive edge over others and will even be able to offer “generative VR Dev” services. Even though for me, it always sounds crazy that some developers do not want to use generative AI services, many developers out there are not using generative AI to support their development services yet. That’s why the earlier you can get your foot in, the more you can distinguish yourself from others.
Side note: At XR Bootcamp, generative AI is already being integrated into the Foundations & Prototyping Bootcamp curriculum - time to check it out!
Acknowledgment
Thanks to Lucas Martinic, VR Prototyper, Asset Creator & Mentor at XR Bootcamp, for the valuable feedback.
Appendix
Links, Resources