ChatGPT isn’t perfect, but the popular AI chatbot’s access to large language models (LLM) means it can do a lot of things you might not expect, like give all of Tamriel’s NPC inhabitants the ability to hold natural conversations and answer questions about the iconic fantasy world. Uncanny, yes. But it’s a prescient look at how games might one day use AI to reach new heights in immersion.

YouTuber ‘Art from the Machine’ released a video showing off how they modded the much beloved VR version of The Elder Scrolls V: Skyrim.

The mod, which isn’t available yet, ostensibly lets you hold conversations with NPCs via ChatGPT and xVASynth, an AI tool for generating voice acting lines using voices from video games.

Check out the results in the most recent update below:

The latest version of the project introduces Skyrim scripting for the first time, which the developer says allows for lip syncing of voices and NPC awareness of in-game events. While still a little rigid, it feels like a pretty big step towards climbing out of the uncanny valley.

Here’s how ‘Art from the Machine’ describes the project in a recent Reddit post showcasing their work:

A few weeks ago I posted a video demonstrating a Python script I am working on which lets you talk to NPCs in Skyrim via ChatGPT and xVASynth. Since then I have been working to integrate this Python script with Skyrim’s own modding tools and I have reached a few exciting milestones:

NPCs are now aware of their current location and time of day. This opens up lots of possibilities for ChatGPT to react to the game world dynamically instead of waiting to be given context by the player. As an example, I no longer have issues with shopkeepers trying to barter with me in the Bannered Mare after work hours. NPCs are also aware of the items picked up by the player during conversation. This means that if you loot a chest, harvest an animal pelt, or pick a flower, NPCs will be able to comment on these actions.

NPCs are now lip synced with xVASynth. This is obviously much more natural than the floaty proof-of-concept voices I had before. I have also made some quality of life improvements such as getting response times down to ~15 seconds and adding a spell to start conversations.

When everything is in place, it is an incredibly surreal experience to be able to sit down and talk to these characters in VR. Nothing takes me out of the experience more than hearing the same repeated voice lines, and with this no two responses are ever the same. There is still a lot of work to go, but even in its current state I couldn’t go back to playing without this.

SEE ALSO
Rhythm Tooter ‘Trombone Champ’ Coming to Quest in November, Trailer Here

You might notice the actual voice prompting the NPCs is also fairly robotic too, although ‘Art from the Machine’ says they’re using speech-to-text to talk to the ChatGPT 3.5-driven system. The voice heard in the video is generated from xVASynth, and then plugged in during video editing to replace what they call their “radio-unfriendly voice.”

And when can you download and play for yourself? Well, the developer says publishing their project is still a bit of a sticky issue.

“I haven’t really thought about how to publish this, so I think I’ll have to dig into other ChatGPT projects to see how others have tackled the API key issue. I am hoping that it’s possible to alternatively connect to a locally-run LLM model for anyone who isn’t keen on paying the API fees.”

Serving up more natural NPC responses is also an area that needs to be addressed, the developer says.

For now I have it set up so that NPCs say “let me think” to indicate that I have been heard and the response is in the process of being generated, but you’re right this can be expanded to choose from a few different filler lines instead of repeating the same one every time.

And while the video is noticeably sped up after prompts, this mostly comes down to the voice generation software xVASynth, which admittedly slows the response pipeline down since it’s being run locally. ChatGPT itself doesn’t affect performance, the developer says.

SEE ALSO
Nintendo Could Be Making a New Labo VR Add-on for Its Next Switch

This isn’t the first project we’ve seen using chatbots to enrich user interactions. Lee Vermeulen, a long-time VR pioneer and developer behind Modbox, released a video in 2021 showing off one of his first tests using OpenAI GPT 3 and voice acting software Replica. In Vermeulen’s video, he talks about how he set parameters for each NPC, giving them the body of knowledge they should have, all of which guides the sort of responses they’ll give.

Check out Vermeulen’s video below, the very same that inspired ‘Art from the Machine’ to start working on the Skyrim VR mod:

As you’d imagine, this is really only the tip of the iceberg for AI-driven NPC interactions. Being able to naturally talk to NPCs, even if a little stuttery and not exactly at human-level, may be preferable over having to wade through a ton of 2D text menus, or go through slow and ungainly tutorials. It also offers up the chance to bond more with your trusty AI companion, like Skyrim’s Lydia or Fallout 4’s Nick Valentine, who instead of offering up canned dialogue might actually, you know, help you out every once in a while.

And that’s really only the surface level stuff that a mod like ‘Art from the Machine’ might deliver to existing games that aren’t built with AI-driven NPCs. Imagining a game that is actually predicated on your ability to ask the right questions and do your own detective work—well, that’s a role-playing game we’ve never experienced before, either in VR our otherwise.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Scott Hayden
Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 4,000 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • mellott124

    They need to fix the computer sounding voices but otherwise this is fantastic. This will be such a literal game changer. Hopefully we can tell them to stop talking as well. I can imagine meeting an NPC that talks too much and annoys.

    • kool

      It sounds like German people to me!

  • Derek Kent

    Absolutely horrible

    • Ookami

      the concept or the execution?

  • Max-Dmg

    This will be amazing for VR gaming going forward, i hope it develops into a robust feature.

    • philingreat

      I doubt it. It’s way too expensive. Who is going to pay for ChatGPT if it’s part of a game?

      • david vincent

        There are much smaller and cheaper languages models out there and some of them are pretty close to ChatGPT3.5. Check the Llama/Alpaca/Vicuna open-source models.

  • Max-Dmg

    Try and convince a character that they are an NPC and they are in a game and see if they go crazy :P

  • Jonathan Winters III

    This is pure BS, and won’t work as described. How’s the ai going to know all the environmental variables in order to keep up an accurate conversation? Not only that but the voice sounds like poo.

    • scottosaur

      This is a lone amateur dev knocking together a proof of concept. This project will probaby never end up in a polished state.

      But I think it’s pretty compelling as a proof-of-concept. The problems you stated are solvable. The API cost seems like a bigger hurdle, but there’s ways around that, too.

      Honestly, I’m not really all that interested in chatting up Skyrim characters because the game isn’t actually built around that so it’s just “fluff.” But I can imagine a game that uses it much more interestingly, and I can also see generative language being used to avoid the immersion-breaking repetition that games like Skyrim have.

  • Nice experiments… still rough, but nice

  • the vast majority

    After some more refinement this could be incredible.

  • MarcDwonn

    Androids in Skyrim? I find that it breaks lore and immersion. :(

    But of course the future will be interesting if this gets developed to be in a better state.

  • Till Eulenspiegel

    It’s a rough preview of what is to come or not.

    Many developers are fearful of AI, they can be unpredictable – NPCs that have minds of their own can do things that not even the developers anticipated. Things that cannot be controlled will probably never make it to the commercial world.

    Recently Google disclosed something that scared them shitless. One of their AI model had learnt a new language even though it wasn’t trained on it. The devs have no clue how the AI learnt the language. This is one of the few cases recently that made them want to pause AI research until they understand how it works. It’s like Quantum physics – they are creating something that works but they don’t know how or why it works.