Intelligent Virtual Agent
- Facial Expression Synthesis
- Hand Gesture Synthesis
- Gaze Estimation
- Multimodal Deep Learning/Multimodal Fusion
- Emotion Computing
- Intention Computing
This project focuses on building the computational foundations to enable human-like virtual characters with the abilities to real-time communicate with human users through natural language, speech signals, gaze behaviors, facial expression and gestures.
While speaking with human users, our virtual characters are able to convey intentions and to express emotions as humans do, but also they are capable of reasoning and perceiving human users' emotions and intentions from their speech signals and behaviors.
This project overlaps the fields of multimodal interaction, computer vision, image processing, natural language processing, speech recognition and synthesis, computer graphics, reinforcement learning, human-computer interaction and social psychology.
An NPC in Justice called Ye Wenzhou
This work focuses on building the computational foundations to analyze and recognize human users' facial expressions but also to predict the virtual characters' expression through facial actions and gaze behaviors.
These computational foundations enable our human-like virtual characters to express and perceive emotions/intentions through facial expression.【Hand Gesture Synthesis】
This work focuses on building the computational foundations to analyze and recognize human users' hand gestures but also to predict the virtual characters' hand gestures.
These computational foundations enable our human-like virtual characters to express and perceive emotions/intentions through hand gestures.【Gaze Estimation】
This work focuses on building the computational foundations to analyze and estimate human users' gaze behaviors but also to predict the virtual characters' gaze behaviors.These computational foundations enable our human-like virtual characters to express and perceive emotions/intentions through gaze behaviors.【Multimodal Deep Learning/Multimodal Fusion】
This work focuses on building the computational foundations to analyze and recognize human users' acoustic, visual and verbal modalities and to learning the temporal contingency between modalities, but also to predict the virtual characters' expression through multimodal verbal and nonverbal behaviors.
These computational foundations enable our human-like virtual characters to express and perceive emotions/intentions through multimodal behaviors.【Emotion Computing】
This work focuses on building the computational foundations to analyze and recognize human users' emotions but also to predict the emotions that the virtual characters have to convey.
These computational foundations enable our human-like virtual characters to express and perceive emotions in face-to-face conversations with human users.Our effort is to study heterogeneous multimodal data including speech signals, natural language, gaze behaviors, facial expressions and hand gestures.【Intention Computing】
This work focuses on building the computational foundations to analyze and recognize human users' intentions but also to predict the emotions that the virtual characters have to convey.
These computational foundations enable our human-like virtual characters to express and perceive intentions in face-to-face conversations with human users.Our effort is to study heterogeneous multimodal data including speech signals, natural language, gaze behaviors, facial expressions and hand gestures.
- Anti-bot Detection Platform
- Match-making System
- Player Action Prediction
- Player Action Reasoning
- Entity Assessment
- Recommendation System
- Identity Verification Service
We analyze players’datasets and extract their features (e.g. digital properties, behavior patterns, social customs, consumption preference) in order to clarify the specific needs of each user group, and distinguish cheaters from normal users. Once the user persona is accomplished, we will be able to provide them personalized service and higher level of safety.
Targeting at the phenomenon that the games bots proliferate in many kinds of games, we build an anti-bot detection platform by combining multi-channel infos, including the account info, the behavior logs and the graph info of players. This platform can not only detect known game bots, but also the muted ones, and is able to evolve during the procedure of detecting and banning.【Match-making System】
The match-making system puts together a certain number of players into teams in a match, after evaluating the performance and ability of the players from various aspects. The goal is to improve the user experience by guaranteeing that players are within the same ability range and form a good synergy inside a team, while ensuring the balance among the teams in one match.【Player Action Prediction】
The player action prediction is to predict the churning, returning, paying actions of players and remind in time the development teams to react properly.【Player Action Reasoning】
The player action reasoning is to analyze the actions of players in a game, such as churning, paying, making friends, etc. We analyze the psychology behind the actions and propose changes accordingly to game designers.【Entity Assessment】
The entity assessment service is to evaluate and profile the abilities of players, the emotions of players, the value of equipments, etc., which provides a reference for the later usages.【Recommendation System】
Players are given different recommendations based on the player demographics, role attributes and equipment values, as well as the game stage of the players. Every player enjoys a unique game experience.【Identity Verification Service】
Tagged pictures with titles are automatically generated to act as CAPTCHA in the game, by leveraging the built-in characteristics of the game pictures. The intelligent CAPTACHAs provide player a better user experience and prevent plugin cracking.
- Transfer Reinforcement Learning
- Distributed Reinforcement Learning
- Hierarchy Reinforcement Learning
- Multi-agent Reinforcement Learning
- Level & Style Controllable AI Bot Generating
- Management Strategy Of Experiment Replay
- Tree Structure Based Reinforcement Learning
- Imitation Learning
- Evolutionary Reinforcement Learning
Have you ever made fun of the stupidity of AI-controlled characters? Hopefully you will receive a deep reinforcement learning AI as your challenging enemy and reliable friend in the near future. Through human-computer and self-combat training, AI may become smarter than you can ever imagine. Deep reinforcement learning AI simulates different difficulty levels and operation strategies perfectly. You can either train it or learn from it.
Arena in Justice
Study the transfer reinforcement learning algorithm to improve the reusability and training efficiency of the model between different game environments.【Distributed Reinforcement Learning】
Study the feasibility and convergence of existing reinforcement learning algorithms on large-scale distributed training platforms.【Hierarchy Reinforcement Learning】
For extremely complex problem, it needs to learn to （automatically） decompose tasks, so that Reinforcement Learning can be solved successfully.【Multi-agent Reinforcement Learning】
In the game scene, there is cooperation and competition between multi-agents, we hope reinforcement learning can train AI with group intelligence.【Level & Style Controllable AI Bot Generating】
According to the needs of actual game AI project, it is possible to automatically generate AI bot with different levels of difficulty and styles.【Management Strategy Of Experiment Replay】
The strategy of experience replay in RL algorithm is improved to enhance the data-efficiency and reduce the forgetting phenomena of the neutral network.【Tree Structure Based Reinforcement Learning】
Balancing the advantages and disadvantages of neural network and tree structure, we hope to develop Reinforcement Learning algorithms that are more efficient, easy to extend, and easy to interpret.【Imitation Learning】
Imitating learning hopes that AI can behave more like a player, and the problems that imitate learning need to solve are:
1、Research on Imitation Learning when the expert trajectories including diverse behaviors.
2、Research on Imitation Learning in the environments with high uncertainty, such as Game problem.
3、Research on the Pervasive Imitation Learning Framework which integrates expert demonstration and environmental rewards.【Evolutionary Reinforcement Learning】
Combining advantages of evolutionary algorithm and reinforcement learning. On the one hand, using evolutionary algorithm to escape from local minima and optimize hyper-parameters simultaneously; on the other hand, adopting reinforcement learning to increase exploit efficiency and resist noise of the environment.【AutoML】
Technologies that automatically training machine learning models. Such as designing neural network architecture, automatic hyper-parameter tuning.
Computer Vision & Graphics
- Face Construction
- Beauty Score
- Gender Classification
- Style Transfer
- Super Resolution
- 3D Reconstruction
- Scenes Generation
- Motion Transfer
- Facial Transfer
We are focusing on avatar animation using facial and skeleton tracking techniques with a simple video camera instead of a professional depth camera (e.g. Kinect sensor). These techniques allow us to capture approximate human facial expressions and poses, reconstruct and display them in a virtual scene. We also assist art designers in constructing game scenes.
Intellect Developing Toolkit
Intellect Developing Toolkit
Calculation of relationship between mesh points with bones based on the relative position and affiliation.【Face Construction】
3D faces generation based on the photos uploaded by players.【Beauty Score】
Evaluate the 3d face created by players based on a trained model.【Gender Classification】
Evaluate gender of the photo uploaded by players.【Style Transfer】
Transfer the game image to any style based on content loss and perceptual loss.【Super Resolution】
Enhance the resolution of images based on Gans.【3D Reconstruction】
3d reconstruction based on images in three views.【Scenes Generation】
Auto generation of scenes based on existing scene data and models.
Fan Creation Platform
Hippo Animator based on Justice
Detect the motion of players based on the visual camera and drive the 3d game character.【Facial Transfer】
Detect the facial of players based on the visual camera and drive the 3d game character.
Natural Language Processing
- Semantic Textual Similarity
- Emotional Conversation Generation
- Personalized Conversation Generation
- Knowledge Aware Conversation Generation
- Context Aware Conversation Generation
- Reading Comprehension
- Spoken Language Understanding
- Dialogue State Tracking
- Aspect-Based Sentiment Analysis
- Word Embedding
- Textual Data Augmentation
We are using perfectly filtered text corpuses to train chatbots with specific temperament labels, by which each and every virtual character can automatically generate dialogues that fit its own personality. Game designers are also able to create a more realistic immersive gaming experience for our players, allow them to control future episode and explore hidden content with their own natural language.
Semantic textual similarity(STS) measures the degree of equivalence in the underlying semantics of paired snippets of text. It lies as an important feature for retrieving the most relevant document in our massive data. Related tasks include paraphrase or duplicate identification, and representation learning of text.【Emotional Conversation Generation】
Perception and expression of human emotions are critical factors to the success of dialogue systems or conversational agents. This problem has seldom been studied in large-scale and industrial settings. We are actively doing research to deal with this problem.【Personalized Conversation Generation】
"Gossiping is fundamental to being human, and this is what separates us from animals." People know each other by chatting. Thus endowing a chatbot system with personality is essential for providing a more immersive conversation experience. There have been a handful of works on this task. We'd also like to make some contributions.【Knowledge Aware Conversation Generation】
An important purpose of conversing with each other is to exchange knowledge. The desire for acquiring knowledge plays an essential role in the history of the Internet. We think providing a more convenient way of accessing extensive knowledge is an important mission of conversational AI systems. However, we believe parsing a user's utterance to SPARQL and then retrieving an answer from a knowledge graph is not all. We want to use knowledge to make the dialogue more informative and appropriate.【Context Aware Conversation Generation】
Context is essential in dialogs. The main drawback of state-of-the-art dialog systems is that they always generate context irrelevant responses. Employment of context in task-oriented dialog systems has been well studied. However, we are more interested with the problems of how to represent context and how to incorporate the context in dialog systems in the open-domain settings.【Reading Comprehension】
Reading Comprehension has been the most actively studied task since the release of SQuAD dataset. Machine Reading Comprehension provides a common framework which facilitates the development of various applications. For example, developing a customer service bot often means preparing a lot of question-answer pairs, a commonsense knowledge question answering bot often incorporates a knowledge base, both of which are expensive. With machine reading comprehension capabilities, people only need to collect unstructured text, which is much easier and cheaper.【Spoken Language Understanding】
Spoken Language Understanding(SLU) is a critical component in dialog systems. Typically, it includes two tasks, speaker intent detection, and semantic slot filling. In different applications, it's occasionally augmented with sentiment analysis, stance analysis to achieve more abundant background information of the user's utterance. We are currently developing a modularized SLU system which incorporates multiple algorithms for different usages.【Dialogue State Tracking】
Dialogue state tacking consists of determining at each turn of a dialog the full representation of what the user wants at that point in the dialog, which contains a goal constraint, a set of requested slots, and the user's dialog act. It allows dialog systems to fulfill tasks like booking live tickets, finding restaurants, etc.. It plays a vital role in applications like virtual assistants.【Aspect-Based Sentiment Analysis】
Aspect-based sentiment analysis is an integral part of public opinion monitoring applications. This technique aims to transform a piece of user review for a specific product into a quintuple consisting of product, aspect, sentiment, opinion holder, and time. We have pushed the state-of-the-art results and actively developing applications to serve multiple products.【Word Embedding】
Word embeddings serve as the basic building block of almost every NLP solution built with a neural network. The performance of word embeddings has a big impact on the downstream applications. For that reason, we pay a lot of attention to developing the best Chinese word embeddings. By incorporating morphological and visual information, we have obtained a series of Chinese word embeddings with state-of-the-art performances over specific downstream applications.【Textual Data Augmentation】
Image data augmentation has been a popular technique in the computer vision community. Except for some basic methods, more advanced techniques like Conditional GANs are also introduced for data augmentation. We hope to introduce data augmentation techniques in NLP tasks. We believe it will not only boost the performances of most NLP tasks but also give us more insights into tasks like text generation, paraphrase identification, semi-supervised learning, etc.
Speech Synthesis & Music Composition
If you get bored listening to the familiar music, let us show you how creative our AI could be. Input enough audio materials you like, and you will be rewarded with a new-created music that only belong to yourself. Speech synthesis, on the other hand, allows NPCs directly talk to players with their own voices instead of verbal communication.【Speech Synthesis】
Speech Synthesis, also known as Text-to-Speech technology, mainly solves the problem of converting text information into audible sound information.
Speech synthesis in game products not only needs to achieve high sound quality that is close to the voice actors, it also needs to be consistent with the character setting of each game character and the emotional development along with the story, so it is a big challenge.【Music Synthesis】
Music Synthesis uses Artificial Intelligence technology to perform soundtrack synthesis in games, to satisfy the players' discerning auditory aesthetics, and give players unique hearing entertainment.【Voice Clone】
Voice Cloning first reproduces the player's timbre with a little speech of the player. Then, while keeping the player's speech content and rhythmic properties (such as modal tones), it exchanges the player's timbre to the target timbre, and make new sound waves.【Voiceprint Recognition】
By modeling the voiceprint of the speaker, it realizes the following two functions: speaker recognition, and speaker confirmation.【Song Recognition】
Quickly pick out matching songs from a massive song library, by recognizing some bars of the song.
FuxiDataSource is the big data fundamental of fuxi lab. It provides large-scale services for data storage and computation based on the long-term game data collected by Netease games. FuxiDataSource integrates with many tools such as Spark, Impala, HBase, ElasticSearch, Kafka and etc. and it offers an integrative service including data transmission, heterogeneous data analysis, data extraction, complex computation and etc. On FuxiDataSource platform, users can customize resilient datasets to acquire dataset service without getting any underlying storage details.
1. Customizable dataset：
Dataset which shields the underlying heterogeneous data storage;
Supporting various structure, e.g. statistical indicators and sequence of behaviours.
2. Multiple computation support:
Real-time streaming computing；
Ad-hoc data analysis.
3. Heterogeneous data source support :
Providing multiple online/offline data processing plans；
Abstracting heterogeneous log and providing consistent data service.
Liberate algorithm researchers , and Automate AI models development until being launched into production
Provided by FuXi Lab, DanLu serves as one of AI cloud platform with complete workflow.It implements GPU cluster management, scheduling, automatic arrangement and recycling based on kubernetes, that facilitates GPU resources utilization.It also offers AI solution management, carries out cloudification for development and training environments by means of docker technology and integrates with GitLab to achieve bidirectional traceability of code commits and model trainings.Furthermore, DanLu delivers one-stop AI serving experience for algorithm researchers to improve work efficiency and operation cost savings.【Product highlights】
1. AI solution management：
Bidirectional traceability on code updates and model trainings；
Online AI services management.
2. Deep learning algorithms and services cloudification：
Images built online, versions management and share；
Development and training environments cloudification；
Support for multiple mainstream deep learning frameworks.
3. GPU cluster management and control：
Cluster management for GPU resources;
Automatic arrangement and recycling of resources;
Real-time monitoring for resources utilization.
NetEase Crowdsourced Data Annotation Platform
NetEase Crowdsourced Data Annotation Platform（https://zb.fuxi.netease.com/about）is developed by Fuxi Lab, which aims to meet the urgent needs of data annotation for AI research in a cost-efficient crowdsourcing mode by making full using of public Internet resource. As a professional AI research department in game field, Fuxi Lab takes the responsibility for promoting the trend of game intelligence, while Crowdsourced Data Annotation Platform provides solid data guarantee for upper-layer applications. The platform will open the functions for data collection and annotation of text, picture, video and speech, which is in favor of the research and technical Implementation in the field of natural language processing, image animation, reinforcement learning and so on. In the future, we will also undertake external requirements to provide customized data services and solutions for third-party customers.【Product highlights】
1. Generalized architecture：
Adopt general data format；
Adopt general component library and automatic interface layout；
Freely customize new patterns.
2. User-friendly design：
Flat vision design；
Both day and night modes to alleviate eyestrain.
Visualized AI FlowChartTool
Visualized Artificial Intelligence Programming Tool is a tool for game designer which innovatively combines traditional method with Reinforcement Learning. It helps to reduce developing difficulty and enhance playing experience.
AI design is an important part of game development. The behavior tree design tool now widely used, is inflexible and hard to get going. Visualized AI Programming Tool shows the AI behavior and game script as flowcharts, which is convenient to read and eidt. Makes it easy for game designer or even player to realize their ideas in game development.Normally, game AI is lack of challenge because it can only increases difficulty through raising numerical value.The machine learning function of the tool support the game to access reinforcement learning algorithm. It brings incredible game experience by making AI behavior more intelligent and diversity.
Visualized Artificial Intelligence Programming Tool, Easy Development , More Fun.
Visualized AI FlowChartTool
1.Graphical Programming Environments.
2.Creating your own neural network is easy.
3.Choosing the algorithm and training directly on the DanLu Platform.
4.You can view and analyze your results at any time.
You can view and analyze your results at any time
Tool and Documentation Downloads: