Virtual Reality and Digital Human Modeling for Ergonomic Assessment in Industrial Product Development: A Patent and Literature Review

Abstract

The late detection of ergonomic component assembly issues during manufacturing processes has an influence on operator well-being and productivity, as well as having a high cost of correction. Although virtual reality may enhance digital human modeling, there is a knowledge gap on the combination of these technologies to assess ergonomics. This study aims to analyze the application of virtual reality and digital human modeling for physical ergonomics assessment during product development in the industry, through a review of patents and the literature. We searched the Derwent Innovation Index, Scopus, and Web of Science databases and found 250 patents and 18 articles. We observed an exponential increase in patents, concentrated among major technological players, and a wide range of technologies being invented. A significant number of studies focuses on the automotive and aviation industries. Despite a relative consensus in the literature on the benefits of integrating virtual reality and digital human modeling to assess physical ergonomics in the early stages of product development, the technologies are seldom combined in the same analysis; moreover, most cases continue to focus on analyzing pre-designed production processes, when resources are completely deployed. These outcomes may provide a reference for practitioners and researchers to develop novel solutions for the early detection of physical ergonomics issues in the industry.

1. Introduction

The product development process is composed of several phases, one of which is defining the manufacturing strategy. The success of this strategy is achieved by starting a project with product and manufacturing process specifications that ensure that it can be produced with minimal impact [1,2].

The late discovery of ergonomic components assembly issues, after the release phase, will have a substantial influence on the cost of the solution in this process. This is because typical production tools are almost finished at this stage of development and modifying them is complex and costly. In contrast, everything in the prior phases of development is generally performed in a virtual model [3].

In terms of ergonomics, finding flaws in the virtual manufacturing model early on decreases the time and cost of finding solutions, while also protecting workers from being exposed to undue risk of harm and improving workplace well-being [1,4,5,6,7].

Therefore, physical ergonomics is an essential principle in the manufacturing process, since it analyzes the connection between physical aspects of humans and workplaces, including human anthropometry, physiology, anatomy, and biomechanics, among other factors [6]. Posture analysis, lifting weights, repetitive movements, and job design are all examples of this domain. As a result, everything in the workplace (tools, devices, items to be handled, etc.) must be designed in such a manner that a worker is able to execute activities with efficient motions, using minimal energy, with low and mitigated risk of injury [8].

Virtual reality (VR) technology can improve the analysis of manufacturing processes. Virtual reality is defined as “a computer-generated digital environment that can be experienced and interacted with as if it were real” [9]. In turn, digital human modeling (DHM) provides a quick, virtual representation of humans, likewise employing a computer to simulate a real-world situation, but without interactivity [10].

Virtual simulation, related to virtual reality and digital human modeling technologies, employs computer modeling and simulation technology, which has been widely utilized for more than half a century in education, health, entertainment, culture, sports, engineering, the military forces, and other fields. However, it has only lately become a valuable tool in the manufacturing industry [3,11].

Despite their various applications, according to the authors’ knowledge, there is no previous work in the literature that reviews scientific and technical knowledge on the combination of virtual reality and digital human modeling to assess physical ergonomics throughout the product design process. Ref. [12] researched the use of immersive virtual reality to enhance industrial workspaces but did not address its integration with digital human modelling. Ref. [13] reviewed the uses and research trends of digital human models in the manufacturing sector but did not focus on ergonomic analyses. As a result, there is still a gap in the knowledge on this subject.

Yet, assessing ergonomic risks during the design phase enables detection of critical issues and the employment of corrective actions early in the process, which is more effective and less costly than later assessment of these risks [2,14,15,16]. Furthermore, virtual reality may improve the realism and effectiveness of virtual manufacturing [13]; supplementing the constraints in the use of digital human modeling technology to assess physical ergonomics, supporting various aspects of the process, such as fidelity issues [17], the requirement for detailed knowledge about the visual behavior of the workers in the task being modelled [18], and the requirement of a high amount of training for engineers [14].

The global virtual reality market is projected to grow from USD 6.30 billion in 2021 to USD 84.09 billion in 2028 [19]. With the emergence of low-cost devices, such as markerless motion capture, an increasing number of organizations are employing them for ergonomics evaluation in the manufacturing sector without fully understanding their overall performance and drawbacks [20]. Therefore, the knowledge on this subject may provide a reference for practitioners and researchers as they continue to develop novel solutions for the early detection of a physical ergonomics problem, improving operator well-being, productivity, and quality, and lowering the cost of corrections.

Thus, this study aims to analyze the application of virtual reality and digital human modeling for physical ergonomics assessment during product development in the industry, through a patents and articles review.

This document is organized as follows: Section 2 describes the materials and methods utilized, Section 3 presents and analyzes the results, and Section 4 provides our conclusions and suggestions for further research.

2. Materials and Methods

This systematic review followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines, which were designed to “help systematic reviewers transparently report why the review was done, what the authors did, and what they found” [21]. In addition, the method described in [22] was used, which encompasses seven steps, as follows: planning, defining the scope, searching the literature, assessing the evidence base, synthetizing, analyzing, and writing.

2.1. Planning

During the planning step, the knowledge bases that were to be explored were defined [22]. The search for document patents was undertaken in the Derwent Innovation Index database, while the search for research articles was conducted in the scientific databases Scopus and Web of Science.

The Derwent Innovation Index database was chosen to conduct the technological prospecting because it contains 39.4 million patent families and 81.1 million patent records, with coverage from 59 worldwide patenting authorities and 2 journal sources. For a search focusing on patents, a database must be assessed using some key criteria, and paid access tools, such as Derwent Innovations, provide some unique resources. As an example, the database of enhanced patent data provides rewritten patent titles, English abstracts of the original patent, a sophisticated classification system, and patent family information with non-convention-equivalent identification [23]. Furthermore, the “Smart Search” tool, which is described further in Section 2.4, leverages artificial intelligence to enhance keyword discovery [24]. As a result, such resources enhance the Derwent Innovation Index database’s ability to retrieve information.

Regarding articles, Scopus and Web of Science were chosen because they are reliable and multi-disciplinary scientific databases of international scope, with comprehensive coverage of citation indexing, allowing the best data from scientific publications.

2.2. Defining the Scope

The defining the scope step results in appropriately formulated research questions [22]. A brainstorming session was held with an interdisciplinary focus group, comprising five experts on product development in the automobile industry and virtual reality. The group selected three pertinent research questions for this systematic review to address, as follows:

• Q1: How are patents that are related to virtual reality and digital human modeling for physical ergonomics assessment during the product design phase in industry characterized?
• Q2: How is current knowledge on virtual reality and digital human modeling for physical ergonomics assessment throughout product development defined, in terms of the application fields, methods, and hardware and software employed?
• Q3: What are the most recent advances, challenges, and opportunities in the combination of virtual reality and digital human modeling in industry for physical ergonomics studies throughout product development phases?

Concerning Q2 and Q3, these characteristics are not abstracted at the same level and are not always mutually exclusive in examined studies; moreover, some research did not describe some of these elements.

2.3. Searching the Literature

The searching the literature step involves exploring the database specified in the planning step using a specific string, based on the questions stated in the defining the scope step [22].

A specialist in the virtual reality-based market of automotive business designed the search strategy. This researcher identified candidate search terms by looking at words in the titles, abstracts, and keywords sections of two known relevant publications. The prospective search phrases were then peer reviewed by five additional members of our research group, with expertise using virtual reality for product design in the automobile sector. The following candidate peer reviewed search terms were applied to the Derwent Innovation Index database: “VIRTUAL REALITY”, “EXTENDED REALITY”, “IMMERS*”, “DIGITAL HUMAN MODELING”, “DHM”, “HUMAN FACTORS”, “ERGO*”, “PRODUCT DEVELOPMENT”, and “PRODUCT DESIGN”.

Then, we entered these search terms in the Derwent Innovation Index database’s “Smart Search” tool to identify relevant keywords. The “Smart Search” analyzed them to form meaningful phrases concerning the technology described in the text and expanded those key terms to include synonyms. Then, “Smart Search” scanned all of Derwent Innovation Index’s patent databases using the extended search keywords and displayed the most relevant patents related to that technology. More information on the technology behind the “Smart Search” resource may be found in [24].

A similar search strategy was used for article retrieval, with minor adjustments to fit the search engine requirements of each database.

The search was carried out in December 2021, and the preliminary screening yielded 497 patent families and 405 articles. A patent family is a set of patents filed with different patenting authorities that refer to the same invention.

2.4. Assessing the Evidence Base

The assessing step employs inclusion and exclusion criteria filters to narrow down the number of documents discovered in the searching the literature step—selecting those that are relevant to the research questions [22]. We reduced the number of documents retrieved during the searching step by combining the following exclusion criteria:

• E1: Documents not written in the English language;
• E2: Published before 2016;
• E3: Publications not related to the industrial areas;
• E4: Patent applications that are no longer active.

We excluded articles published before to 2016, due to the rapid development of virtual reality devices in recent years. According to the literature, “the year 2016 marked a technological breakthrough, with the first public release of technologically mature Virtual Reality headsets” [25]; moreover, 2016 was the year with the greatest rise in virtual reality research globally [26]. Academics and practitioners alike concur that the equipment released in 2016 constituted a “very big breakthrough” for virtual reality applications [27,28].

2.5. Synthetising and Analysing

At this point, the retrieved documents were merged with project-related elements [22]. The documents were submitted to a single screening, in which a reviewer with experience in the automotive industry and virtual reality technology reviewed each document in order to find articles relevant to the research questions defined in the defining the scope step. The documents were chosen based on an examination of their title, abstract, and keywords field, as well as their connection to the project’s purpose—see the following categories:

• Applications in production or manufacturing (rather than, for example, dentistry, medical sectors, or training activities).
• Physical ergonomics studies (rather than other areas of ergonomics, such as emotional or cognitive).
• Interaction with “Product Development” or “Product Design” (some studies addressed maintenance operations, which have a frequency of risk exposure that differs from normal operation).
• Virtual reality—providing users with immersive experiences (since some researchers or database-automated processes associate “mixed reality”, “augmented reality”, or “virtual environment” with immersion characteristics).

Finally, we selected 250 patents grouped by families and 18 scientific articles. Derwent analytical and insights resources were used to analyze the patents. The retrieved documents were uploaded to the Mendeley Reference Manager tool, and Microsoft Excel was used to create spreadsheets and graphics.

The flow from the searching the literature step to the synthetizing step of the systematic review is represented in Figure 1.

3. Results

The research questions Q1, Q2, and Q3 were addressed in order to uncover obstacles, opportunities, and breakthroughs that may be useful in stimulating the use of virtual reality and digital human modeling to undertake ergonomic assessment in industrial product design; the findings are detailed in the sections that follow.

3.1. Patent Mapping

To answer the first research question, patents were analyzed in terms of publishing trends, origin location, top assignees, International Patent Classification (IPC) and technology trends.

• Q1: How are patents that are related to virtual reality and digital human modeling for physical ergonomics assessment during the product design phase in industry characterized?

In terms of patent publishing trends, Figure 2 shows an impressive growth in annual patent publication since 2016, with a significant increase in 2019. The upward trend slowed in 2020, with the probable explanation being the 18-month patent legal secrecy restriction. We omitted 2021 patents from analysis, since they were still being filed at the time of this study.

The fact that there were almost no patent filings in 2016 supports the claim in [26], that the year 2016 marked a technological breakthrough in the domain of virtual reality. Before 2016, commercial virtual reality systems required users to connect a headset, controllers, and sensors to an external high-end computer, which was an expensive, bulky, and inconvenient setup. Thus, the current all-in-one virtual reality systems are a big step forward, occurring only a few years ago [29].

Virtual reality was rated the most often referenced technology in digital human modeling and simulation research and application trends by researchers, developers, and digital human modeling industry stakeholders in 2020 [30]. Furthermore, not only is the global virtual reality market projected to grow from USD 6.30 billion in 2021 to USD 84.09 billion in 2028 [19], but this rise may be much greater, given that the COVID-19 pandemic boosted the use of virtual reality [31]. Therefore, this scenario suggests that the exponential increase in patenting filings, related to virtual reality and digital human modeling, will continue for a few more years.

Figure 3 illustrates patent origin locations. The United States and China have a comparable amount of patent filings, with 44% and 41%, respectively.

Figure 4 illustrates the top 10 patents assignees. The significant concentration of patent filings in a few assignees is intriguing: the top 10 assignees filed 24% of all patents. Microsoft has the most patent filings and is responsible for 20% of the top 10 assignees’ patent filings. Six of the top ten assignees are major players of technology (Microsoft, Apple, Intel, Facebook, IBM, and Bytedance) and two are in industrial automation (Rockwell and Siemens). It is also worth noting that one university (New York University) accounts for 7% of all patents filed in this subject.

Next, we examined at the patent submissions in terms of distribution, based on the International Patent Classification (IPC). The IPC is a method for determining a standard classification for registered patents, allowing the search of and access to the technical information available in documents related to the same topic [32]. This classification facilitates the search for patent filings and allows access to the technical information included within them.

The patents are classified based on their technical content. The International Patent Categorization is a hierarchical patent classification system used in over 100 countries to uniformly classify patent material. It creates a separation into classes and subclasses that are applicable to various technical domains and aids in the standardization of patent classification. Figure 5 illustrates that IPC is composed of letters and numbers and follows a hierarchical pattern [32].

Figure 6 shows the top 10 IPCs of reviewed patent filings. This group’s patents are all classified as Section G (Physics), which contains all physics-related content. As the class level progresses and technological information is described in more depth, the most frequent subclass is G06T (Image Data Processing and Generation, containing the following subgroups: manipulating 3D models or images for computer graphics; 19/20—editing of 3D images, e.g., changing shapes or colors, aligning objects, or positioning parts; 7/00—image analysis), followed by G06F (Electric Digital Data Processing, containing the following subgroups: 3/0176—input arrangements or combined input and output arrangements for interaction between users; 3/0481—computer and interaction techniques, based on graphical user interfaces). Other patents are grouped under G06K (Data Recognition and Presentation), G03H (Holographic Processes or Apparatus), and G02B (Optical Elements, Systems, or Apparatus) subclasses.

In terms of technology trends, Figure 7 illustrates the technologies that are currently being developed and patented. The findings reveal the most current breakthroughs as well as those that are novel and growing. The block chart shows 30 different technology classifications. We determined that the top three technology groups that have led this subject in the previous five years can be classified by inclusion of the following: (1) augmented reality, three-dimensional, processing, rendering, model, and storage medium; (2) computing, transitory, touch, information processing, user, virtual, and management; (3) holographic and hologram. These technologies are present in 72% of the patent records in the set. The number of technologies reflects recent innovations and provides an overview of the “state of the market” and market segmentation. Larger percentages of tech interests indicate saturation, while lesser percentages indicate diversified tech representation.

Since 2020, there has been a substantial increase in the pace of innovation in the sector. The top technology classification (augmented reality, object, three-dimensional, processing, rendering, model, and storage medium) peaked in 2021 and has increased by an average of 11 patent records each year since then. Furthermore, with over 83% of all technologies represented, 2021 had the largest technical diversity. These promising results back up our prior estimates that the exponential growth of virtual reality and digital human modeling research and applications will continue in the coming years.

These data offered an overview of innovations related to virtual reality and digital human modeling for analyzing ergonomics in a broader context. At this point, we conducted a focused search for current inventions that target the integration of virtual reality and digital human modeling technologies in evaluating physical ergonomics in the industry, throughout the product design process.

Table 1. Highlighted patents on integrating virtual reality and digital human modeling.

Ref.	Application Number	Title	Derwent Rewritten Title	Publication Year	Assignee
[33]	US11062520B2	Ergonomic assessment using a wearable device	Portable device of ergonomic scoring system, has processor that provides three-dimensional digital human model located within real environment via display, and moves digital human model relative to virtual surface model	2021	Ford Global Technologies
[34]	CN113331825A	A real-time evaluation method of RULA in virtual reality	Virtual reality-based real-time rapid upper-limb assessment score evaluating method, involves obtaining main sagittal and correction sagittal planes based on joint point	2021	Beijing Institute of Technology

displays two patents that we consider to be noteworthy.

The proposition of US11062520B2 [33] active patent is a portable device with an ergonomic scoring system, used to score ergonomic data in relation to machines, workplaces, and environments. The invention’s novelty is that the portable device has a display that enables users to view a real environment. A computer, connected with this display, includes processors and a memory that stores instructions, executable by the processors. The processor determines a virtual surface model that is associated with a real object in the real environment, using sensor data. The processor provides a three-dimensional digital human model that is located within the real environment via the display, and moves the digital human model relative to the virtual surface model. The processor restricts that the digital human model and the virtual surface model occupy a common 3D space. As a result, the advantage of the invention is that the portable device can display the stresses so that the user can make real-time ergonomic assessment and the ergonomic data can be collected effectively.

The active patent CN113331825A [34] proposes not an apparatus, but a method: a virtual reality-based, real-time Rapid Upper-Limb Assessment (RULA) score evaluation. The RULA is one of the most common methods used to assess the workers’ exposure to ergonomic risk when performing manual upper body tasks, such as hand, neck, and limb twisting [8]. The aim of RULA is to determine the urgency with which a change should be implemented in order to avoid work-associated musculoskeletal disorders. The invention’s novelty is that the proposed method involves collecting an action frame, establishing a space coordinate system, and extracting a joint point coordinate. Then, a limb vector, a main sagittal plane, a correction sagittal plane, and a main projection vector of the limb vector are obtained. Each limb vector is projected into a main sagittal plane and a main limb total score is obtained based on the main limb total score. The invention’s advantage is that the method enables realizing quick, real-time, and accurate evaluation of human–computer efficiency.

3.2. Scientific Mapping

Table 2. List of selected articles.

Ref.	Title	Publication Year
[35]	VR-based Operating Modes and Metaphors for Collaborative Ergonomic Design of Industrial Workstations	2017
[36]	FCA Ergonomics Proactive Approach in Developing New Cars: Virtual Simulations and Physical Validation	2017
[37]	On Immersive Virtual Environments for Assessing Human-driven Assembly of Large Mechanical Parts	2017
[38]	The Benefits of Human-centred Design in Industrial Practices: Re-design of Workstations in Pipe Industry	2017
[39]	Integrating mocap system and immersive reality for efficient human-centred workstation design	2018
[40]	Verification of HTC Vive deployment capabilities for ergonomic evaluations in virtual reality environments	2018
[6]	A comparative study on computer-integrated set-ups to design human-centred manufacturing systems	2019
[1]	An automatic procedure based on virtual ergonomic analysis to promote human-centric manufacturing	2019
[13]	Applications and research trends of digital human models in the manufacturing industry	2019
[8]	Ergonomics simulation in aircraft manufacturing—Methods and potentials	2019
[41]	The effectiveness of virtual environments in developing collaborative strategies between industrial robots and humans	2019
[42]	Application of Virtual Reality to Perform Ergonomic Risk Assessment in Industrialized Construction: Experiment Design	2020
[14]	Natural virtual reality user interface to define assembly sequences for digital human model	2020
[5]	The use and usage of virtual reality technologies in planning and implementing new workstations	2020
[7]	Using Virtual Reality and Smart Textiles to Assess the Design of Workstations	2020
[43]	Virtual Reality (VR): A Review on its Application in Construction Safety	2021
[16]	Virtual reality simulation of human–robot coexistence for an aircraft final assembly line: process evaluation and ergonomics	2021
[30]	Current Trends in Research and Application of Digital Human Modeling	2022

shows the 18 articles selected by the search strategy.

These articles were examined in order to address Q2 and Q3 research questions, which are addressed in the following subsections.

• Q2: How is current knowledge on virtual reality and digital human modeling for physical ergonomics assessment throughout product development defined, in terms of the application fields, methods, and hardware and software employed?

3.2.1. Application Fields

Two of the eighteen articles we reviewed are reviews [13,30]. We excluded them from our analysis since they looked at some of the papers we investigated in this study. Therefore, to avoid counting any research twice, we excluded these two review articles from this quantitative analysis in this section.

Similar to the method in [13], the application field of an article was defined as the industry and/or technological environment in which the application of virtual reality and digital human modeling was carried out. The observed application fields may be grouped into six categories, as follows: (1) automotive industry; (2) aerospace industry; (3) construction industry; (4) industrial plants; (5) energy industry; (6) field unspecified. Figure 8 shows the distribution of the examined studies in each category.

There were 4 articles (25%) which focused on the automotive industry [1,5,7,36]. It was expected that a large number of studies on virtual reality and digital human modeling would be undertaken in the automotive market. These results are consistent with those shown by [13]. The automotive industry’s relevance may be linked to the fact that virtual technologies have long been employed in this sector, being used in a variety of fields and applications, such as production, training, and maintenance, to name a few [44,45,46]. Furthermore, the automotive industry is one of the most mature manufacturing sectors, where cutting-edge technologies are often deployed for the first time [13].

There were 3 studies (19%) investigating the aerospace industry [8,16,37]. The aviation industry is interested in virtual reality and digital human modeling technology for ergonomics assessment for a variety of reasons. Reducing the effect of human errors in production is critical for improving air transportation safety. In general, there is a need to improve aircraft manufacturing performance in order to ensure safe operation at the lowest possible cost [47]. Figure 9 is an example of an aerospace industry study.

There were 2 articles (12%) which focused the construction industry [42,43]. Construction tasks are physically stressful, requiring inadequate and prolonged postures, which can cause pain and injuries. Figure 10 is an example of a construction industry study.

Another 2 studies (12%) were concentrated on industrial plants manufacturing engine [14] and steel pipes [38], and 1 was focused on the energy industry [6].

There were 4 articles (25%) which did not specify the field of application [35,39,40,41]. Ref. [47] asserts that it is common for augmented and virtual reality applications developed by a research team in an academic setting to be examined in their own laboratory using prototypes and items, as shown in Figure 11.

3.2.2. Methods

The articles examined cover a wide range of methods. Some are more frequent and may be found in a variety of applications, while others are specific to the industry sector in which the case study was produced. Some research uses multiple methods, which is justified by the fact that the approaches may serve diverse purposes. Therefore, using more than one of them would constitute more comprehensive research.

Table 3. Methods and measurements used in the studies reviewed.

Ref.	Methods/Measurements
[5,6,30,35,37,38,40,42]	Rapid Upper Limb Assessment (RULA)
[5,37,40]	Rapid Entire Body Assessment (REBA)
[1,30,36]	Ergonomic Assessment Worksheet (EAWS)
[35]	National Institute for Occupational Safety and Health (NIOSH)
[16]	Ovako Working Posture Analysis System (OWAS)
[1]	Methods–Time Measurement (MTM-UAS)
[14,41]	Other (NASA-TLX questionnaire, Kinetic Energy ratio)

groups the methods and metrics used in the research under consideration.

Among the articles reviewed, the RULA method is the most commonly employed, either as the primary method or as a supplementary method. As mentioned previously, RULA is one of the most widely used methods for assessing employees’ exposure to ergonomic risk while performing manual upper body tasks, such as hand, neck, and limb twisting [8].

The second most cited method in the articles reviewed is Rapid Entire Body Assessment (REBA). The method is similar to RULA in principle, but it is utilized for full body posture studies, such as neck, hand, shoulder, leg, and limb twisting. It measures risk based on posture, usage, load weight, length, and frequency of the activity and gives a score to the evaluated task that indicates the level of intervention required to lower the risk of injury. The RULA and REBA methods were primarily intended to serve as a guide to draw attention to activities that offer a high risk of upper-limb disorders; they do not provide quantitative data on injury risk or upper-limb strain [8]. The RULA is more widely employed than the REBA in upper-limb tasks that are more impaired by work-related injuries.

The Ergonomic Assessment Worksheet (EAWS), mentioned in three studies, is a holistic screening method that considers traditional risk factors, such as work strength, frequency, and repetition. It arose from the need to meet the conformities of the various labor legislations of the International Organization for Standardization (ISO) and the European Committee for Standardization (CEN). As a result, it is most commonly used by the European automotive industry and its main characteristic is the comprehensive analysis of all tasks performed in a worker’s daily routine [20]. EAWS includes not only load handling but also posture, action forces, and repetitive activities [8]. EAWS divides the analyses into modules, which makes it possible to separately evaluate various ergonomic conditions, such as work postures and movements with low physical effort, whole-body action forces or hand/finger systems, material handling, and repetitive upper-limb loads, etc. With these assessments, it is feasible to map the many categories of hazards and relate them to the need for corrective and/or preventative action [4]. In reviewed articles, EAWS is used to enable virtual analyses providing a broad evaluation, combining various biomechanical loads that affect workers, and therefore providing a baseline risk assessment [8].

The techniques of the National Institute for Occupational Safety and Health (NIOSH) and Ovako Working Posture Analysis System (OWAS) both received one mention. The American National Institute for Occupational Safety and Health developed the NIOSH method to determine the maximum load to be manually handled and moved in a work activity. The method is designed to evaluate symmetrical and asymmetrical lifting tasks, including lifting with less-than-ideal engagement between the object and the worker’s hands. The tool indicates the estimated weight or load for a certain manual lifting job, under the given postural conditions, that most workers could lift safely over a specified period, and gives a relative estimate of the level of physical stress associated with a manual lifting task or a job involving several lifting tasks [35].

The Ovako Working Posture Analysis System (OWAS) is a method to quickly check the comfort of work postures and determine the urgency of taking corrective measures. The OWAS enables assessment of the relative discomfort of a working posture, based on the positioning of the back, arms, and legs, as well as load requirements, and assigns the assessed posture a score that indicates the urgency of taking corrective action to reduce the potential of the posture to expose workers to injury. The OWAS method ranks postures based on a subjective assessment of discomfort and the effect of each posture on health. The method does not take into account the rate at which different work postures occur, nor the impact of correcting work postures over time [5]. Figure 12 is an example of a study that used the OWAS method.

Among the other methods mentioned, the NASA TLX Questionnaire is noteworthy. It is a method for analyzing total workload, including physical and mental demands, using a weighted average of evaluations in six dimensions, as follows: mental demand, physical demand, time demand, performance, effort, and frustration level. In the reviewed paper, the method was utilized to measure subjective workload and determine an assembly sequence for a digital human model. The job completion time and the number of mistakes made by participants during the task were also measured [14].

One limitation of the methodologies we noticed throughout our search is that the majority of the assessment tools mentioned are not included in the main standards for assessing biomechanical overload. A possible explanation is that the main standards, in general, aim to define a guideline for products, processes, or services, concentrating on what must be done. As an example, International Organization for Standardization (ISO) 11226:2000 specifies recommended limits for static working postures; ISO 11228-1:2003 specifies recommended limits for manual lifting and carrying of loads; ISO 11228-2:2007 provides guidance on the assessment of risk factors considered important in manual pushing and pulling; ISO 11228-3:2007 establishes ergonomic recommendations for repetitive work tasks involving the manual handling of low loads at high frequency [48]. Many of the methodologies mentioned in the articles, such as the OWAS [16] and NIOSH [35], concentrate on how the given methodology must be performed to fulfill the main standards and requirements.

Table A1. Methods, hardware, and software referred to in each article reviewed.

Ref.	Methods, Hardware and Software
[42]	RULA, HTC VIVE, Unreal, Autodesk 3ds Max, Photoshop, SOLIDWORKS
[49]	Powerwall motion capture (single rear-projected wall and two passive stereoscopic projectors and an optical marker-based tracking system), four InfraReds cameras, body markers, IC.IDO
[39]	IMUs motion capture, full body suit, HTC VIVE, Polar V800 BLUETOOTH heart rate monitor, Siemens Jack
[43]	Leap Motion, HTC VIVE, CAVE, SAVES, Multi-user VR, VDC tools, V-REP
[37]	RULA, REBA, Oculus Rift, Kinect, Hand Arm Vibration (HAV), CATIA, Unity3D
[7]	IMUs based motion capture, HTC VIVE, IPS, OpenVR Desktop Portal, Unity 3D
[14]	NASA-TLX questionnaire, HTC VIVE, Leap Motion, Unity 3D, IPS IMMA, SciKit Learn, supervised classification algorithm, Siemens Jack, Dassault HUMAN
[1]	EAWS, MTM-UAS, Motion Capture, Siemens Jack, DELMIA, VBA-coded Excel
[30]	EAWS, RULA, HMD, CAVE, Motion capture, Model-driven, motion planning, key frame, motion synthesis
[35]	RULA, NIOSH, projection system, four-wall immersive room, Digital Mock-Up
[16]	OWAS, HTC VIVE, Noitom’s Perception Neuron Motion Capture
[41]	Kinetic Energy ratio, HTC VIVE, ABB IRB 2600 ROBOT, Kinect, Unity 3D
[6]	RULA, motion capture, 3D printing, 3D active glasses, large screen for rear projection, Barco Galaxy NW-7 projectors, active stereo glasses with active Volfoni Edge RF, Vicon optical tracking system with eight Bonita cameras; a Denon AVR sound system with Dolby surround, DELMIA, CATIA, Haption RTI DELMIA plug-in
[38]	RULA, Vicon Tracker, optic tracking cameras by Vicon motion capture, 3D printing, Delmia, CATIA, Haption RTI Delmia
[5]	RULA, REBA, CAVE, HTC VIVE, IPS IMMA
[36]	EAWS
[40]	RULA, REBA, 2 Kinect, HTC VIVE
[13]	CAVE, head-mounted displays, aixCAVE, Kinect, 3D glasses, motion capture, eMagin Z800 3DVisor, PhaseSpace motion capture, Xsens motion capture, NDI POLARIS optical measurement, CyberGlove, Sensics, Oculus Rift, Vicon, Firefly cameras, Iphone, VITUS XXL laser scanner, HoloLens, Volfoni Edge RF, AVR sound system, 3Ds Max, SmartBody, Unity 3D, Evolver 3D digital avatar, Delmia, Unreal, Adobe Fuse, Mixamo, IPS IMMA VESP, OSG, Jack IPS IMMA, Unity 3D, Middle VR, Vuforia, Jack, VR Juggler, OpenGL, Apple ARKit, UMTRI human shape, CATIA, Blender, RAMSIS

lists the method specified in each of the articles reviewed.

3.2.3. Hardware

Virtual reality is a complex technology that integrates multimedia, sensors, displays, human–machine interactions, ergonomics, simulations, computer graphics, and artificial intelligence technologies to expand human perception; therefore, virtual reality systems often require more than one device to function [13]. Virtual reality hardware can vary greatly depending on the application, ranging from a standard computer to certain display devices, motion capture equipment, and interactive gadgets, such as wearable devices, cameras, and head-mounted displays (HMD), etc.

Table 4. Hardware, equipment, devices, and apparatus used in the studies reviewed.

Ref.	Hardware	Category
[5,7,14,16,39,40,41,42,43]	HTC VIVE	HMD
[13,37]	Oculus Rift
[13]	Microsoft HoloLens
[13]	eMagin Z800 3DVisor
[13]	Sensics
[13,30]	HMD (model unspecified)
[5,13,30,35,43]	CAVE (model unspecified)	CAVE
[13]	CAVE aixCAVE
[13,37,40,41]	Microsoft Kinect	Motion Capture
[6,13,38]	Vicon
[1,6,13]	Motion Capture (model unspecified)
[38]	Cameras	optic tracking
[49]	optical tracking markerbased
[13]	NDI POLARIS
[14,43]	Leap Motion
[49]	Projector passive stereoscopic	Projectors
[6,35]	Projectors (model unspecified)
[6,38]	3D Printer	Other devices
[7,39]	Sensor Inertial Measurement Unit (IMU) (model unspecified)
[6,13]	3D glasses (model unspecified)
[6,13]	Sound System AVR
[49]	body markers, Camera InfraRed (model unspecified), single rear-projected wall
[39]	full body suit, Heart Rate Monitor Polar V800
[37]	Hand Arm Vibration (HAV)
[6]	Large screen for rear projection (model unspecified)
[41]	Robot ABB IRB 2600
[13]	Augmented Reality devices, CyberGlove, Laser scanner VITUS XXL, SmartBody, Smartphone Iphone, Volfoni Edge RF, Camera Firefly

shows the hardware, devices, and apparatus mentioned in the articles reviewed.

We discovered a wide assortment of equipment, both in terms of the overall category and the models used in each category. Some studies did not mention which model was employed. We identified that the most frequently used equipment may be divided into five broad categories, as follows: (1) head-mounted displays, (2) CAVE Automatic Virtual Environment (CAVE), (3) motion capture systems, (4) optic tracking, and (5) projectors. Not only are these categories not mutually exclusive, but the overall structure is often created using a combination of multiple devices.

A virtual reality environment requires the use of both visualization and tracking technology. In the industrial context, the most frequently used virtual reality visualization solutions are projection-based systems and head-mounted displays. Projection virtual reality systems include single or multiple projector-based powerwalls, as well as surrounding, walk-in setups, based on multiple projection screens (e.g., CAVEs). Head-mounted displays are devices attached to the virtual reality user’s head, and typically include one or two displays as the image source, as well as collimating optics between the eyes and the display [5].

Ref. [13] noted that the use of CAVEs and head-mounted displays is rather rare in industry, which they attributed to expensive prices—a significant workload for manufacturing process simulation, since the motions of digital human modeling are driven by actual people—and the time-consuming calibration process.

We found a different scenario, since head-mounted displays were the most commonly utilized devices in the studies we examined. A plausible explanation is that, until recently, the usage of head-mounted displays was limited, owing to their expensive cost and technical limitations. Although the cost of the equipment is still relatively expensive and technical restrictions remain, the price of this equipment has been lowering year after year, technological constraints have improved, and new features have been introduced, leading to a rise in the adoption of head-mounted displays [19].

Before head-mounted displays became more popular, the CAVEs equipment was the most extensively used apparatus. CAVES, on the other hand, are costly, have a poor immersion, and are not particularly portable; therefore, they tend to be supplanted by head-mounted displays. CAVEs were still used in two articles that have been examined; however, one of them was published in 2017, shortly after the significant increase in head-mounted display releases in 2016 [35]. These assumptions are supported by [30], which found that in 2022, 28 percent of presentations used a virtual reality system, whereas just 4 percent used a CAVE setting.

Among the studies that used head-mounted displays, the HTC VIVE was the most frequently mentioned model, significantly ahead of the second most frequently cited model, the Oculus Rift. HTC VIVE is a virtual reality headset comprising a head-mounted display, two wireless handheld controllers, and two lighthouse base stations that generate pulsed infrared lasers. It allows the user to roam about in a 3D environment and interact with it using motion-tracked handheld controllers. The VIVE system contains two 1080 × 1200 resolution panels, one for each eye. Furthermore, the headgear and controllers have 70 infrared sensors, a gyroscope, and an accelerometer. These sensors, together with the two lighthouses, follow the operator’s movements with millimeter accuracy. SteamVR, which runs on Microsoft Windows, is the operating system. A USB connection attaches the VIVE system to the computer [39]. Figure 13 is an example of a study that used HTC VIVE head-mounted displays and Leap Motion optical tracking.

Other head mounted-displays models were mentioned, but they are much less prevalent than the HTC VIVE.

A frequent device combination was head-mounted display with motion capture. This is consistent with the findings in [30], that motion capture is the most common method for manipulating the manikin. Microsoft Kinect and Vicon were the most often utilized motion capture models. Both models were used in a comparable number of studies; however, the Vicon model was used twice by the same researcher in two distinct articles, and a wider range of researchers utilized the Kinect model. Because sensing in single postures while handling the virtual object is currently inadequate, motion capture devices, such as Kinect sensors, are utilized to supplement detection of virtual world players and to reduce occlusion problems [40].

Other hardware with specialized functions was identified. Refs. [7,39] detect body movements via inertial measurement unit (IMU) sensors, which are subsequently converted into equivalent orientation changes in the virtual environment for posture analysis.

A wide range of gadgets, including sound systems, heart rate monitors, haptic devices, body trackers, robots, and 3D printing (as shown in Figure 14), were only mentioned in one study; therefore, these were categorized as “Other”. Table A1 lists the hardware specified in each of the articles reviewed.

3.2.4. Software

Similar to what we observed on hardware devices, we identified a diverse set of editors, programs, engines, and frameworks that were utilized, in terms of both the overall category and the models used in each category. The software employed, like the hardware, is not mutually exclusive; on the contrary, we observed that there is frequently a mix of several complimentary solutions.

Table 5. Software, editors, tools, frameworks, and engines used in the studies reviewed.

Ref.	Software	Category
[7,13,14,37,41]	Unity3D	Game Engine
[13,42]	Unreal
[13,42]	Autodesk 3Ds Max
[6,13,37,38]	CATIA	CAD/CAE and DHM tools
[1,6,13,38]	DELMIA
[5,7,13,14]	Industrial Path Solutions Intelligently Moving Manikin (IPS IMMA)
[1,13,14,39]	Jack Siemens
[6,38]	Haption Real-Time Interaction plug in for DELMIA
[42]	SOLIDWORKS
[14]	Dassault HUMAN
[49]	VR Engineering EIC.IDO	Other
[35]	Digital Mock-up
[7]	VR Steam OpenVR Desktop Portal add on
[42]	Graphics editor Adobe Photoshop
[14]	Machine Learning library SciKit Learn
[13]	Evolver 3D digital avatar, Adobe Fuse, Mixamo, IMMA VESP, OSG, CAVE Unity plugin Middle VR, Augmented Reality related (Vuforia, Apple ARKit), VR Juggler Development framework, OpenGL APPI for rendering vector graphics, UMTRI human shape, Blender, CAD Manikin RAMSIS

shows the software mentioned in the articles reviewed.

Industrial Path Solutions Intelligently Moving Manikin (IPS IMMA) and Jack Siemens, which were utilized by a comparable number of studies, were the most commonly used digital human modeling software.

Digital human modeling software are tools for human representation in a virtual environment that incorporates numerous applications [13]. We observed that some software package is intended for everyday application, whilst others, in addition to having specific uses, give complementary features. DELMIA and CATIA are two such software that simulate and assess various body positions and build virtual manikins that emulate worker motions. The simulations performed using the DELMIA software allow for the creation of a database for fatigue indices, which will accelerate future analyses and facilitate information sharing.

Software is used to assess the ergonomic conditions of the manufacturing process, with a focus on decision assistance for modifying the workplace layout, producing assistive assembly devices, replacing tools, providing specific operator training, and changing routines and assembly processes.

In terms of virtual reality software, the most used is Unity, a platform for game creation that includes a game engine. Ref. [37] selected Unity due to its maturity, its user and developer community, and its zero cost of acquisition.

Instead of Unity, the game engine Unreal was cited in two investigations, for example, [42] justified that Unreal was chosen to develop the virtual reality application because it has a more user-friendly interface than Unity Engine and allows Blueprint Visual Scripting to be used to generate virtual reality interactivity.

Ref. [13] also found that in recent years, game engines such as Unity 3D and Unreal have been applied to virtual manufacturing and digital human modeling. Such game engines can provide helpful application programming interfaces for rendering, interaction, and physics, facilitating the construction of high-quality and high-fidelity virtual environments, as well as a good online documentation [13,37]. Figure 15 is an example of a study’s hardware and software architecture that use the Unity gaming engine.

A wide variety of software, such as plugins, editors, and libraries, among many others, were cited by only one study and have been grouped under the category “Other”.

Table A1 lists the software specified in each of the articles reviewed.

• Q3: What are the most recent advances, challenges, and opportunities in the combination of virtual reality and digital human modeling in industry for physical ergonomics studies throughout product development phases?

The stage of product development at which ergonomic assessments are undertaken is a noteworthy contradiction, which we identified in our study.

On the one hand, as mentioned, there is general consensus in the literature on the benefits of carrying out this assessment early—benefits in terms of time, costs, optimization of the assembly line, human–machine interactions, improvement of the overall performance and well-being of the user, and mitigating hidden dangers, among many others [1,4,6,7,13,31,32,36,38].

Considering that, after the design stage, the product development moves on to manufacturing-related stages, these would be limiting stages for the early identification of the physical ergonomic issues, before the beginning of production. Nonetheless, despite all of these potential gains, we identified just a few cases of virtual reality and digital human modeling being used for physical ergonomics analysis during early product development. We found that the majority of virtual reality and digital human modeling methods of physical ergonomic analysis are oriented to pre-designed production processes with all resources (devices, facilities, equipment, etc.) completely installed.

Another relevant contradiction that we encountered relates to the integration of virtual reality and digital human modeling. In general, the literature examined agrees that digital human modeling enables faster, more accurate, and cost-efficient product development processes [1,4,6,7,13,31,32,36,38], and that it is a useful tool for assessing work assignments and design alternatives without the expenditure of physical mockups and production trials.

However, the literature did point out that digital human modeling has certain limitations when it comes to evaluating physical ergonomics. Constraints include the following: fidelity issues [17]; failure to properly simulate human perceptions and emotions [15]; the need for manual selection of manikin viewpoint, which can be both time consuming and difficult; the digital human modeling user’s comprehensive knowledge of visual behavior of employees in the job being simulated is sometimes lacking [18]; the need for significant engineering training, making its use difficult [14]; an assessment mostly limited to static evaluations of reachability and posture [50].

The sense of immersion and presence, of being part of a virtual setting, are the primary features of virtual reality applications that allow the user to engage with the virtual simulation [11]. Indeed, several of the studies reviewed claimed that the combination of virtual reality with digital human modeling enabled mitigation of some of the digital human modeling constraints. As an example, [51] integrated a computer-based system for simulation, 3D visualization, and the supply of multiple tools to concurrently design the product and the production process, with data from simulations reviewed to assess ergonomic indexes in a preventive and proactive manner. Ref. [17] presented a mixed prototype based on virtual reality and CAD to proactively filter design ideas during the conceptualization phase, before functional prototypes are built, that was able to simulate the change in human performance due to emergency situations. Ref. [5] used a mixed prototype that combined motion capture, digital human modeling, and virtual reality, allowing designers to quickly change the virtual representations of new workstations to test solutions for emerging difficulties, as well as capture, in virtual reality, the motions of prospective workers, while performing work tasks, as an input for digital human modeling to instruct biomechanical human models. The benefits of combining virtual reality and digital human modeling, according to [6], include the following: more realistic replication of workers’ actions into the virtual scene due to motion capture; a higher level of immersion for users; greater simplicity for simulation creation; the lack of direct observation of workers on the shop floor or involvement of production engineers to generate reliable simulations; the ability to conduct interviews with users during task execution.

When virtual reality and digital human modeling are combined, virtual reality provides a cost-effective and safe environment for testing various concepts and hypotheses before they are deployed in the real world. Ref. [41] estimates that, because of virtual reality’s potential to optimize factory layouts before construction, well-designed layouts may save up to 50% of operating costs, and that optimization of manufacturing processes and activities is another area where virtual reality can offer substantial benefits.

However, we experienced a contradiction: on the one hand, the current research indicates the aforementioned benefits of virtual reality enhancing digital human modeling for physical ergonomic evaluations; on the other hand, we discovered that, still, few virtual simulations combine virtual reality and digital human modeling in the same analysis. Since there is no evidence in the articles reviewed that allows us to draw a conclusion on this issue, we cannot be sure of the explanation for the low adoption of the two combined technologies. However, a possible explanation is that there is still a lack of knowledge regarding the benefits of this combination. In general, industrial product development continues to focus on quantifiable features in project stages, such as DHM analysis, while overlooking technologies such as virtual reality, which provides a better knowledge of workers’ or end-users’ viewpoints. In this respect, a lack of understanding of available technologies has a direct influence on their spread, and while researchers perceive a gain, more information is needed to encourage practitioners to employ them.

Another reason is that virtual reality imposes additional constraints that restrict its widespread adoption. In fact, virtual reality technology implementation can pose certain challenges. Ref. [5] found that the heavy emphasis on the sense of touch during assembly processes and the expectation of a physical resistance when interacting with virtual structures can often not be adequately simulated. Ref. [43] addressed the following issues: having only partial control over the user’s approach to moving around in the virtual reality environment; inconsistency between the user’s actual movement and the virtual animation; high setup, implementation, and maintenance costs; a lack of peripheral visual inputs while wearing head-mounted displays; little information on the return on investment for virtual reality usage. Additionally, unexpectedly, rather than improving the sense of immersion, virtual reality can reduce it. Ref. [5] observed that some individuals may feel less present in a virtual environment or may even respond negatively to virtual reality use by having motion sickness-like symptoms, which can impair work performance. As a result, the facial validity of motions tracked in virtual reality may not be as well established as it is in real-world tests on prototypes.

Ref. [30] also encountered this contradiction: virtual reality has been identified as the most important current trend in digital human modeling by researchers, developers, and industry stakeholders, but only around one-third of those they polled in the study stated they use virtual reality to present their simulation results. The author suggests that possible explanations for this low use are the slow integration of virtual reality features into digital human modeling tools, the additional technical expertise required to run such a system, the relative newness of most consumer virtual reality solutions, and the fact that, although relatively few digital human modeling tool users make use of virtual reality technology currently, many more do so than was the case just a few years ago.

The authors highlight an important gap that may exist between researchers’ expectations that virtual reality is a current trend in digital human modeling and actual adoption of the technologies in the field; although academics perceive a clear benefit, further development is required to persuade the digital human modeling community to use the technologies [30].

4. Conclusions

In the last five years, there has been an exponential increase in the number of patents registered by major technological players, such as Microsoft, Apple, Intel, Facebook, IBM, and Bytedance, with a wide range of technologies being invented. This scenario implies the emergence of new devices and equipment concepts. As a result, over the next two or three years, the focus of research will shift toward investigating the potential applications of newly introduced devices. Additionally, the reviewed studies showed that several of the laboratory-tested proposals have the potential to be validated by industry, now in production contexts.

We observed two significant contradictions. Despite a relative consensus in the literature on the many benefits of integrating virtual reality and digital human modeling to assess physical ergonomics in the early stages of product development, currently, virtual reality and digital human modeling are seldom combined in the same ergonomics analysis, and the majority of cases continue to focus on analyzing pre-designed production processes, with all resources (devices, buildings, equipment, etc.) completely deployed.

In our review, we identified potential new research opportunities. We suggest further research on a potential gap between researchers’ expectations, that virtual reality is a current trend in digital human modeling, and the actual adoption of the technologies in the field. We support the assertion in [30], that future research should investigate why virtual reality tools are not extensively utilized, and what is required to make these technologies accessible to digital human modeling tool users. We also suggest that further research should focus on the reason for the lack of studies integrating digital human modeling and virtual reality in the same ergonomic analysis.

Because of the remarkable advancements in technology, a growing number of firms are considering virtual reality and digital human modeling for ergonomics evaluations, without fully understanding their overall benefits and constraints [20].

Our review outcomes, concerning the present patent scenario—the current application fields, methods, and hardware and software used—as well as the recent advances, challenges, and opportunities in the combination of virtual reality and digital human modeling throughout product development, may provide a reference for practitioners and researchers as they continue to develop novel solutions for the early detection of a physical ergonomics issue.