Abstract
This paper implements the Multi-Modal Instruction Agent (here-inafter, MMIA) including a synchronization between audio-gesture modalities, and suggests improved fusion and fission rules depending on SNNR (Signal Plus Noise to Noise Ratio) and fuzzy value for simultaneous multi-modality, based on the embedded KSSL (Korean Standard Sign Language) recognizer using the WPS (Wearable Personal Station) and Voice-XML. Our approach fuses and recognizes the sentence and word-based instruction models that are represented by speech and KSSL, and then translates recognition result that is fissioned according to a weight decision rule into synthetic speech and visual illustration (graphical display by HMD-Head Mounted Display) in real-time. The experimental results, average recognition rates of the MMIA for the prescribed 62 sentential and 152 word instruction models were 94.33% and 96.85% in clean environments, and 92.29% and 92.91% were shown in noisy environments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Gary Perlman., et al.: HCI Bibliography.: Human-Computer Interaction Resources, http://www.hcibib.org/
Kim, J.-H., et al.: Hand Gesture Recognition System using Fuzzy Algorithm and RDBMS for Post PC. In: Wang, L., Jin, Y. (eds.) FSKD 2005. LNCS (LNAI), vol. 3614, pp. 170–175. Springer, Heidelberg (2005)
i.MX21 Processor Data-sheet, http://www.freescale.com/
Kim, S.-G.: Korean Standard Sign Language Tutor, 1st edn. Osung Publishing Company, Seoul (2000)
Kim, J.-H., et al.: An Implementation of KSSL Recognizer for HCI Based on Post Wearable PC and Wireless Networks KES 2006, Lecture Notes in Artificial Intelligence. In: Gabrys, B., Howlett, R.J., Jain, L.C. (eds.) KES 2006. LNCS (LNAI), vol. 4251, pp. 788–797. Springer, Heidelberg (2006)
Chen, C.H.: Fuzzy Logic and Neural Network Handbook. McGraw-Hill, New York (1992)
kandasamy, W.B.V.: Smaranda Fuzzy Algebra. American Research Press, Seattle (2003)
McGlashan, S., et al.: Voice Extensible Markup Language (VoiceXML) Version 2.0. W3C Recommendation (1992), http://www.w3.org
Martin, W.H.: DeciBel – The New Name for the Transmission Unit, Bell System Technical Journal (January 1929)
NIOSH working group.: STRESS... AT WORK NIOSH, Publication No. 99-101,U.S. National Institutes of Occupational Health (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kim, JH., Hong, KS. (2007). SNNR-Based Improved Multi-modal Fusion and Fission Using Fuzzy Value Based on WPS and Web. In: Apolloni, B., Howlett, R.J., Jain, L. (eds) Knowledge-Based Intelligent Information and Engineering Systems. KES 2007. Lecture Notes in Computer Science(), vol 4692. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74819-9_86
Download citation
DOI: https://doi.org/10.1007/978-3-540-74819-9_86
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74817-5
Online ISBN: 978-3-540-74819-9
eBook Packages: Computer ScienceComputer Science (R0)