A multichannel (multimodal) corpus includes two components. On the one hand, it is a collection of media files that register communication between people. On the other hand, it is an annotation of the communication events stored in the media files. Multichannel corpora differ in a number of parameters, such as the degree of spontaneity of the material (ranging from experimental materials to absolutely free flowing communication), the number of interlocutors (two, three or more) and the nature of the environment (ranging from customized to spontaneous). In accordance with these parameters, our multichannel corpus "Russian Pear Chats and Stories" can be characterized as partly structured input material with established roles for the interlocutors, who, however, can communicate freely; there are four interlocutors; and the communication takes place in a customized environment.
The corpus consists of separate communication episodes (so-called sessions), each involving four participants: a Narrator, a Commentator, a Reteller, and a Listener. The corpus received its name after the so-called “The Pear Film”, a well-known stimulus material used in linguistics which was offered to two of our participants to watch in the beginning of the session. “The Pear Film” was created by a research group under Wallace Chafe at Berkeley in the 1970s (see http://www.linguistics.ucsb.edu/faculty/chafe/pearfilm.htm). The film does not contain any speech acts of the characters, but it depicts a story of the interactions of several characters, among them a gardener who is picking pears, a boy who steals a basket of pears, and others along the way. This film offers a well-structured chain of physical and social events and has long established itself as an excellent way to obtain compact and comparable retellings.
The film is viewed by the participants who later act as a Narrator and a Commentator. The figure shows the general design of the situation in which the communication takes place.
General design of the communication situation
The Narrator tells the Reteller the content of the film. During the interactive stage, which comes next, the Commentator elaborates on the story, and the Reteller asks questions to both interlocutors who have seen the film. After that, the Listener joins in, and the Reteller retells the film to him/her, also as a monologue. In the end, the Listener has to write down yet another, secondary, retelling of the film (see. example, in Russian). The two final stages are important as they encourage all participants to engage in comprehensive and meaningful communication.
The "Russian Pear Chats and Stories" corpus consists of two parts. The first part was collected in the summer of 2015. It contains 24 sessions with a total duration of approximately 9 hours, with an average duration of 23 minutes for each session (ranging from 12 to 38 minutes), and a volume of about 100,000 words; a total of 96 people aged from 18 to 36 years old, 34 of whom were men and 62 of whom were women, took part in the sessions. The second part of the corpus was collected in the summer of 2017. It contains 16 sessions with a total duration of approximately 6 hours, with an average duration of 21 minutes (ranging from 8 to 41 minutes), and a volume of about 60,000 words; 64 people aged from 18 to 36 years old, including 16 men and 48 women, took part in the sessions. Prior to recording, all participants signed informed consent and completed a general and a handedness questionnaire (all forms are in Russian).
For demonstration purposes, the web site offers media files and annotations belonging to three sessions of 2015: #04, #22 and #23. Each session contains 18 media files, among them:
-
4 audio files in the wav format with recording parameters of 48kHz / 16bit:
- from the Narrator's microphone (mono)
- from the Commentator's microphone (mono)
- from the Reteller's microphone (mono)
- with the common microphone installed in the middle of the room (stereo)
-
4 video files in the mjpeg format (avi container):
- an individual video of the Narrator (resolution 1024×736, frame rate 100 fps)
- an individual video of the Commentator (resolution 1024×736, frame rate 100 fps)
- an individual video of the Reteller (resolution 1024×736, frame rate 100 fps)
- a cover shot video (resolution 1024×576; frame rate 50 fps).
-
3 files recorded with the help of eye trackers (avi format, resolution 1024×576, frame rate 25 fps):
- video recorded with the Narrator's eye tracker while watching the stimulus film (suffix -f in the file name)
- video recorded with the Narrator's eye tracker in the course of communication
- video recorded with the Reteller’s eye tracker in the course of communication.
The package also includes lightweight (suffix -lite in the file name) versions of the same files in the avi format (codec H.264), which are convenient to use for orientation purposes.
Each file name includes the number of the session; the code of the participant, i.e. N (Narrator), C (Commentator), R (Reteller), or W (whole, or common, file); as well as its file type: au (audio file), vi (video), or ey (file recorded with the help of an eye tracker).
Along with media files, each session is also provided with annotations which include:
- individual vocal annotations (suffix -voc in the file name), which contains segmental annotations created using the Praat software (in textgrid format), and a verbal-prosodic transcript with indications of the temporal dynamics (suffix -verb, doc format);
- scores vocal transcript which lays out the overall contribution of the three main participants in the session (suffix -voc-scores, xls format);
- referential annotation created using ELAN software (eaf format) which appears as a set of supplementary tiers over the basic vocal markup that contain information on the characteristics of verbal expressions with a specific reference (suffix -ref);
- kinetic annotation created using ELAN (eaf format) that includes annotation of manual gestures (suffix -manu) and cephalic gestures (suffix -ceph); all annotations created in ELAN also contain a supplementary file in the pfsx format;
- oculomotor annotation (suffix -ocul) created in two formats: Excel (xls format) and ELAN (eaf format). Eye tracking data exported to a video scene, and displayed as a video file with a superimposed marker, serve primarily for illustrative purposes, as the data presented therein do not always coincide with the exact timeline that is reflected in the annotation files.
The principles of annotation are described here.