Release by NVDA Japanese Project (nvdajp), based on NVDA 2012.1 from nvda-project.org.

Release manager: Takuya Nishimoto (nishimotz atmark gmail.com, Twitter @nishimotz)

This release is Japanese localization of the NVDA open-source screen reader for Windows.

Changes from original 2012.1:

• Application icon is changed to the NVDAJP Telescope Goldfish.
• NonConvert key of Japanese keyboard can be used as NVDA modifier key (optional).
• Japanese Text-to-Speech engine JTalk is added (based on Open JTalk, MMDAgent, Mecab project)
• Japanese input method editor (IME) support (for 32bit/64bit environment). Some applicaitions are not fully supported.
  You can disable this feature using “Keyboard settings” – “Support IME (nvdajp)”.
• AppModules for Search Aid and VoicePopper (popular self-voicing applications in Japan) are added.
• Japanese translations added for Input Help Mode: “IME Non convert”, “IME Convert”, “Pause”.
• Character descriptions can be customized by user.
  If characterDescriptions-(locale).dic file is found in userConfig file,
  it supersedes built-in dictionary.
  This feature is requested from the Japanese users, because commercial screen readers use their own character description dictionaries, and the users preferences are so different.
  We are also developing alternative dictionary to replace the current Japanese dictionary.
• Experimental JTalk2 driver: language switching among Japanese and other languages supported by eSpeak.
• Experimental Japanese braille display driver DirectBM (for KGS devices).
  Japanese braille support requires the process of inserting spaces between words.
  Current implementation of nvdajp uses the text analysis module (mecab and naist-jdic) for both JTalk and DirectBM.
  Although we would like to accelerate the development, the development members have very limited number of devices.
• “Braille Viewer” is added in Tools menu. Available for Japanese language.
• Report Time (NVDA+f12) is modified for Japanese environment.
• Help documents are opened by HTML application (HTA) window, not the system’s default web browser.
  This is because Japanese popular voice web browser for screen reader users is difficult to use with NVDA.

What’s New in this version:

• NVDA 2012.1 (rev 5086) is merged.
  http://bzr.nvaccess.org/nvda/releases/2012.1
• Some libraries are updated: liblouis 2.4.1 and espeak 1.46.12
• Some Japanese translation of messages are modified.
• Japanese Input Method Editor (IME) support is improved.
  This work is unfinished. Some limitations and bugs are remained.
• JTalk (Japanese TTS) is updated.
• Dictionary for Japanese text analysis is updated.

We use launchpad.net as bazaar source-code repository.

• sourceforge.jp project site: http://en.sourceforge.jp/projects/nvdajp/
• launchpad.net site: https://code.launchpad.net/nvdajp

Versions of Japanese resources (shown in NVDA log):

• nvdajptext.dll 20111017 mecab:0.98
• nvdajp-jtalk-dic 20120403-053143
• libopenjtalk 20120323-070011

Repositories are as follows:

• lp:~nishimotz/nvdajp/releases_2012.1
• lp:~nishimotz/nvdajp/MiscellaneousDependencies rev 77

I am participating as the project leader of Japanese localization of NVDA since 2011.

Assisitive technlogy is making evolutions, as well as the other technologies.

In Japan, most of the core technologies, such as text input method, speech synthesis, and automatic braille translation, have not been freely available, so the Japanese community is behind the times.
This is the reason I committed to the project so deeply since the earthquake.

nvdajp 2011.3j was released on Dec 23.
Its enhancement includes Japanese TTS, IME support and braille support.
Default speech engine is JTalk, an open-source Japanese TTS, based on Open JTalk.

Functions of reporting Japanese input method has limitations as follows:

(1) IME support does not work properly with Wordpad, Microsoft Word, and Internet Explorer.

(2) With Internet Explorer and some applications, after writing some Japanese characters within a textbox,
pressing Escape key without finishing composition may cause collision of escaping edit mode with canceling Japanese composition.

(3) With 32bit applications, key-echo of Japanese input is delayed, because the report is performed at the key-up events.

IME support function can be disabled using Keyboard setting.

DirectBM is experimental braille driver which supports KGS Japanese braille displays.

Although the activity is separated from the mainstream development now, international cooperation, such as helping development of Chinese text imput support, is very important.

We started discussions with an NPO in Japan to obtain the support. I am also seeking the consultation business related to the software, or a position of full-time researcher where I can keep the engagement of NVDA development. I hope I can inform the good news this year.

Golden Sun Legend Hotel. The room was sound-proofing and confortable.
The hotel manager arranged airport transportations and Halong bay one-day trip (including Kayaking). Currency exchange was also available there. Room PC was Windows 7 Ultimate. WiFi service for guests.

Nha Hang Ngon restaurant, Tran Hung Dao Street. Appeared in the Japanese edition of Vietnam Airlines In-flight magazine.

Green Mango restaurant, 18 Hang Quat street. VISA card available. Spring rolls and eggplant salad.

La Veranda cafe-restaurant. 21A Chu Van An, Q.Ba Dinh, near the Vietnam fine art museum. Free WiFi available.

Le Malraux restaurant, 2 Hang Bong street.

La Place cafe-restaurant, near Joseph’s Cathedral. Upstairs seat.

Green Gecko cafe, Ma May street. Free WiFi available. The service of latte was slow.

Hilton Hanoi Opera. Just visited the lobby lounge. Free WiFi available.

Free WiFi was also available at Tourist Information Center, north of the Hoan Kiem Lake. Cafe available.

This post is excerpt from the draft version of the abstract of doctoral dissertation.
My public hearing for the dissertation will be held this month at Waseda University.
Although the thesis itself is written in Japanese, I am willing to write the related topics here in English.

This paper describes a systematic way of enabling of developers and designers to build information-communication systems successfully with speech technologies, such as speech synthesis and speech recognition. As the results of this work, application systems of speech technologies can be used easily for everyone.

This work also describes four research projects including the development of speech applications and the evaluations of speech interfaces, which are performed based on the proposed methodology.

The first chapter proposes “the principles of interfaces,” consists of (1) basic principles, (2) organization principles, and (3) adoption principles, which are based on the fundamental theories of human-machine interactions.
The most important hypothesis of this work is that these principles are indispensable for accomplishing the projects to investigate on various human interface systems with speech technologies.

The following four chapters show the effectiveness of proposed principles, which comes along with the effectiveness of speech technologies and importance of designs and evaluations of speech interfaces.

One of the application systems proposed here is the S-tgif multimodal drawing system, which uses isolated speech command recognition, mouse and keyboard.
Using speech recognition with the application, average operation time or the number of command inputs can be reduced.

Another proposed system is the AVM asynchronous voice messaging system.
The system displays the voice message as the threaded written words. The user can manipulate voice messages just as if they are text messages. If the participant wants to quote or annotate to a message, the user has only to play the sound and barge into the message while it is playing. Such user interface increases the usefulness of the voice-mail system.

Other two projects are related to the evaluations of speech interface systems.
One of the projects proposed an improvement of the dual-task method to measure the workload of spoken dialog tasks.
In this method, subjects play a game using visual display and keyboard input. We evaluated the effectiveness of our method with a word shadowing task and a spoken dialog application.
Our method can measure relatively small workload differences, such as in word shadowing tasks, which are difficult to measure with the previous works.
Also, our method can identify positions in the dialogs which cause some users significant difficulty.

Another project was related to the ultra fast speech for the computer application of persons with visual disability.
For the evaluation of such speech, consideration of learning effect of the listener is important.
In this research, the learning effects of listening to ultra fast speech with the control of word familiarity were investigated with the considerations of (1) the changes of the familiarity condition during the experiments, and (2) existence or nonexistence of instructions of the familiarity. The experiments to observe the intelligibility and mental workload were performed, using the speech with the speed of approximately 21 mora per second. The results supported the hypothesis that the intelligibility increases and mental workload decreases if the listener is aware of high word familiarity because the access to mental lexicon is promoted.

This thesis also shows the various research projects related to the interface principles, which give the perspectives of human-machine interactions and applications of speech technology.

The most important contribution of this work is discussed in last chapter, which is the separation of general human interface principles and utilization of speech technologies.
The well-organized principles naturally give the guidelines for designing and evaluating interface systems with various applications, modalities and devices. Speech technologies is one of the applications of the principles.

The system is http://rcpt.cc.
Project Information page is available in English.

P.S.
The feed function of my blog system was something wrong.
Now I tested and http://nishimotz.com/feed/rss2/ seems OK.

I enjoyed using Twitter during the conference. Thank you.

Dear new friends,
Usually I tweet in Japanese with @nishimotz account. I prefer using facebook for English conversation. So, please feel free to unfollow me.
Both facebook and @nishimotz account on Twitter can be used for English conversation.
I will tweet in Japanese with @nishimtz account.

My slide and my tweets at ICCHP is as follows: Takuya Nishimoto, Takayuki Watanabe: The Evaluations of Deletion-Based Method and Mixing-Mased Method for Audio CAPTCHAs.

Notice (2010-10-16): Updated slide of this topic (for Interspeech in Sep. 2010) is available.

Twitter Timeline:

tweet time below is JST.

Day 1:

Day 2:

Day 3:

DMCPP in English
DMCPP in Japanese

Please join mailing lists for discussions.

I tried to adopt original pyAA to Python 2.6.x.
At first I obtained the source code (of simpler branch) from CVS repository, then I built it with Visual Studio 2008 and SWIG. I also modified the code so that the Value-property can be accessed correctly under multibyte charactor coding environments.

• pyaa-nishimotz-20100216-python26-lib-site-packages.zip

We are successfully proceeding the work with it at the moment.
At the previous meeting of NVDAjp project, we added some code to NVDA, and verified that the WinEvent of MS-IME 2002 can be captured and the Value property can be accessed using IAccessible interface.

Related pages in Japanese (not yet translated) : pyaa and nvdajp

Notes on Feb 27 : I created a github repository of pyaa.

The assumption is that the human-like spoken dialogs are highly effective. Our proposal is to use the reinforcement-learning for acquiring the strategy how to respond quickly to overlapped utterances, interruptions, or gestures during spoken dialogs between human and machine. Although the research is still in early stage, we hope something like mind-reading will be possible, in other words, the users of spoken dialog systems do not need to say from the beginning to the end.

In another research project with a company during 2007-2008, a prototype of multimodal interface system was developed. In this work, we expected that the visual feedbacks in realtime while the user is speaking may help the user to notice the speaking style or vocabulary is valid or not. Moreover, this exploratory search allows the user to stop speaking even if the speech is not finished, when the user can see the expected results.

Another work related to the effectiveness was conducted by my ex-colleague, Dr. Masahiro Araki.

Input Prediction Method of Speech Front End Processor using Prosodic Information

In general, prosody of speech contains various information. For example, in Japanese, accent information is used for distinguishing homonyms and identifying word boundaries. In this paper, we propose a HMM-based accent type recognition method and, as an application of this method, an input prediction front end processor for dictation. From a few morae inputs, completion candidates that are sorted by input history and by the accent pattern are listed up. We examined two accent usage methods for both registered words and unregistered words, and implemented an input prediction system combining a speech recognizer, a prediction server and an accent usage module.

• Report in Japanese : http://www.gavo.t.u-tokyo.ac.jp/tokutei_pub/houkoku/recog/araki.pdf

• Another publication : Masahiro Araki, Hiroyoshi Ohmiya, and Satoshi Kida, “Input Prediction Method of Speech Front End Processor Using Prosodic Information,” Proceedings of International Conference: Speech Prosody 2004, Nara, Japan, 2004.

This kind of effective voice interfaces had been repeatedly re-invented.
The important point is that the strategy of effective interface highly depends on the situations, the tasks and available modalities.
That’s the reason of my recent interests in the machine learning of such skills.

Our concept of voice interface was inspired by the Hyakunin-isshu game. Hyakunin-isshu (The hundred poems by one hundred poets) is one of the famous game in Japan.

http://www.japanlink.co.jp/ka/play1.htm#Card%20game

Karuta (Card game)

The word karuta is said to have come from the Portuguese carta. Karuta are rectangular, like ordinary playing cards, with pictures or Japanese writing drawn on them. When playing, one player reads out a card for reading (yomi-fuda) and the other players compete to take the picture card (efuda) that matches it; the player who takes the most cards is the winner. As typical Japanese aspects of the game, there are iroha-garuta that contain the Japanese proverbs and poem cards (uta- garuta) on which the poems in 31 syllables known as tanka are written. Nowadays, the game is played principally at New Year.

Hyakunin-isshu (The hundred poems by one hundred poets)

This generally refers to the poetry anthology entitled “Ogura hyakunin-isshu,”compiled by FUJIWARA no Teika(Sadaie). It gathered 100 waka–the classical Japanese poem, specifically in this case Poem in 31 syllables–one each by the most outstanding poets from the Heian Period(794-1185) and the early years of the Kamakura Period(1185-1333). From the time of the Edo Period(1603-1867), this poetry anthology was widely spread and used as the poem cards. The overwhelming majority–43 selections–are love poems, followed by seasonal poems–32 selections. 79 of the poets are male, 21 female, and they express thoughts of love, nature and the seasons with a refinement unique to the Japanese people. It has become well-known as the representative work of classical Japanese literature. It’s one of the essential game of New Year.

In cases of the seven of the hundred cards, the player can decide a single correct card when the game master’s only first syllable is spoken.

This service was built with Ruby and Rails, and hosted by Heroku. Additional “delayed job” option is also used.