[VRchat Review] How do you do facial tracking/face tracking/eye tracking? I can only dress up in Unity, but here’s a simple way I did it with Facial Tracker or QuestPro [as of December 31, 2023].

What is facial tracking and eye tracking anyway?

Full-tracking is an abbreviation for full-body tracking, and its common use is for tracking of the entire body; in VRchat, body tracking is generally referred to as full-tracking. Eye and facial expressions are generally controlled by controllers. Commercial avatars also support full body tracking, but few support eye tracking and facial tracking.

What we mean by “eye tracking and facial tracking support” this time is that we will add “facial tracking” (facial tracking) and “eye tracking” (eye tracking) to avatars, creating avatars that support facial tracking and eye tracking, but

Instead of the traditional full body tracking only
→Full body tracking + eye tracking + facial tracking

We are talking about adding the following to the system The idea is to track facial movements and eye movements as well.

VIVE Facial Tracker Edition (Facial Tracking only)

What is the VIVE Facial Tracker?

I will show you how to deal with facial tracking and face tracking (face tracker), but since those who came before me have already practiced it, including how to deal with and set up face trackers, this article will only explain the sequence of steps I have tried.
A facial tracker is a type of tracking device that recognizes with a camera.

I recently found out that the Facial Tracker is no longer available. What a pity. Dismissed.
We have been using it for a long time, but it looks like we will have to use QuestPro for a while.
Also, I recommend following Tona. He always gives us useful information.

VIVEフェイシャルトラッカーが生産終了。北米VIVE公式サイトからは削除済でECサイトでも軒並み終売。日本でもVIVE公式の在庫を残すのみ。USB接続と設置さえできれば非公式ながらどのHMDでも利用でき、舌の上下左右トラッキングもできる最優秀な顔トラッキングアドオンです。気になる方は確保をお早めに https://t.co/F0W30Ec20w pic.twitter.com/gn6UUrL3x4

— トナ (@tonavrc) December 18, 2023

This site will also feature QuestPro’s Facial Tracking and Eye Tracking at the end of the sale.

What do I need to prepare? (Hardware & Driver Installation)

What do I need to prepare? (Hardware & Driver Installation) As for what to purchase in hardware, VIVE Facial Tracker and in my case, it was Valve Index. Prepare for the fixation. I referred to Nemu’s article.

https://note.com/nemchan_nel/n/nb39ffd9ebef2

VIVE Facial Tracker

If you follow the instructions in this article, you will have no problems around the hardware.

[Hardware] ①Facial tracker, connection attachments to match the HMD

【Hardware】 Simply put, you will need a facial tracker, a doujinshi holder, and a USB converter (USB C to A for Index). USB conversion is not necessary if your HMD is equipped with USBC.However, this article is mainly for Neos, so you need to make it for VRchat.

There are some notes from the officials.

2 disclaimers here:

1. We can only guarantee performance with Pro as that's the exact position we'd originally trained the recognition on.

2. USB A (m) to C (f) is actually against USB-IF specs. These adapters have no spec to conform to, so attempt this at your own risk. https://t.co/NJJpQXT6rv

— Shen Ye (@shen) March 16, 2021

【Drivers] (2) Drivers for which the Facial Tracker works

What do I need to prepare? (Software)

What do I need to prepare? (Software)
Next, though, you will need to set up your avatar and install the software. If you know how to make avatars in Blender, please refer to this video. If you are familiar with Blender and Unity, you can do it by watching the first and second parts of this video.

Avatar: Need to make avatar compatible with PerfectSync.

However, the first part is for people who can handle Blender normally, so I introduced an avatar that supports Perfect Sync.
I bought it from this store. https://endof.booth.pm/items/4276261 
I bought it and installed it after watching the video in the next part. However, it was quite troublesome to make it from scratch, so I installed the following software to handle it.

[Software ed.]VRCFacialOSCAvatarTool、AkaneFacialOSC

VRCFacialOSCAvatarTool
https://booth.pm/ja/items/3718123

AkaneFacialOSC
https://azw.booth.pm/items/3686598

So, we learned how to respond to facial tracking by watching YutanyCH.’s video!

How can I do this in Unity?

Place VRCFacialOSCAvatarTool and Shiro-chan (avatar) in Unity.

QuestPro version (Facial + Eye Tracking version)

How to introduce to VRchat’s famous avatar RearAlice

Not yet implemented & notes for future practice tries. What do I need to get together, and how to introduce it to Avatar Rear Alice, a famous VRchat Avator (QuestPro version)? Everyone’s Rear Alice idol, Rii Nishii, has purchased QuestPro and practiced facial tracking.

I knew that QuestPro could do it, but in the case of Rear Alice, they also sell a tool related to moving faces. That’s great.
I heard that the facial expressions, etc. were introduced with reference to this site. That’s impressive.
It seems that you can easily introduce the avatars that are famous in VRchat by doing the following

An easy way to introduce the RearAlice other

リアアリスの顔が動く神ツールですhttps://t.co/nCAAi629yE #booth_pm

— 西井りい (@nishiirii) November 19, 2023

Rea Alice and Mafuyu’s are available for sale on this site.

https://shopyk.booth.pm/items/5100830

After we have a set of equipment, we will set it up. First of all, here is how to deal with QuestPro -> VRchat.
(4) We will do it while looking at the following settings.
https://qiita.com/Kazu_Sack/items/14460dac1dd149bf1ff5

I installed it myself and it took about 2 hours with VirtualDesktop.
If it doesn’t work, reboot, etc., it works.

Facial tracking support for QuestPro is now complete. Here is the video I tried with the supported avatar.
I tried closing one eye, opening the mouth, and not talking. The only thing left to do now is to adjust the avatar side.

How to install in Avatar (QuestPro Preparation)

Avatars can be created by uploading a pre-purchased rear Alice and then
After uploading the previously purchased Rear Alice, simply follow the steps in the Adjustment Tool for the face area. This also took about an hour.

Since the rear Alice is uploaded on VRchat, please turn on the Facial Tracker and Eye Tracking in the Expression menu.
Turn on the facial tracker and eye tracking.
Then you can.

There are a few different avatars that sell tools for face tracking, like the store below.
There are several places that sell tools for face tracking, such as the following shops. Several avatars that are quite recently famous, such as Maya and Celestia, are sold.
https://fermat.booth.pm/items

This time, we tested QuestPro’s Facial Tracking + Eye Tracker at Rear Alice and found out the following
VIVE facial tracker is sold out & drinks etc. are very hard to get a hold of, so I recommend it because it’s light and very easy to use.

Mafuyu also worked properly.

https://shopyk.booth.pm/items/5151544

Vroid and other users should refer here.

【VRoid編】

What do I need to get together? How to install (QuestPro version)

Here is the video.

How does facial tracking and eye tracking differ between QuestPro and Vive Pro eye?

There are a few things QuestPro cannot do.
Tongue tracking does not work well.

For example, the tongue tracking does not work well. For tracking performance alone, the VIVE Pro eye + facial tracker would be a good choice.

I bought QuestPro and have summarized the process of switching over.

QuestProで、11点無線HMDフルトラ＋Indexコン＋フェイシャル＋アイトラッキングの完全体フルトラになるまでの流れを紹介！【完全体フルトラ（無線フルトラ＋アイトラ＋顔トラ）】【前編】