mexca.pipeline ============== .. py:module:: mexca.pipeline .. autoapi-nested-parse:: Build a pipeline to extract emotion expression features from a video file. Classes ------- .. autoapisummary:: mexca.pipeline.Pipeline Module Contents --------------- .. py:class:: Pipeline(face_extractor: Optional[Union[FaceExtractor, FaceExtractorContainer]] = None, speaker_identifier: Optional[Union[SpeakerIdentifier, SpeakerIdentifierContainer]] = None, voice_extractor: Optional[Union[VoiceExtractor, VoiceExtractorContainer]] = None, audio_transcriber: Optional[Union[AudioTranscriber, AudioTranscriberContainer]] = None, sentiment_extractor: Optional[Union[SentimentExtractor, SentimentExtractorContainer]] = None) Build a pipeline to extract emotion expression features from a video file. Takes either component objects or container component objects (or a mix of both) as input. :param face_extractor: Component for detecting and identifying faces as well as extracting facial features. :type face_extractor: FaceExtractor or FaceExtractorContainer, optional, default=None :param speaker_identifier: Component for identifying speech segments and speakers. :type speaker_identifier: SpeakerIdentifier or SpeakerIdentifierContainer, optional, default=None :param voice_extractor: Component for extracting voice features. :type voice_extractor: VoiceExtractor or VoiceExtractorContainer, optional, default=None :param audio_transcriber: Component for transcribing speech segments to text. :type audio_transcriber: AudioTranscriber or AudioTranscriberContainer, optional, default=None :param sentiment_extractor: Component for extracting sentiment from text. :type sentiment_extractor: SentimentExtractor or SentimentExtractorContainer, optional, default=None .. rubric:: Examples Create a pipeline with standard components. >>> from mexca import Pipeline >>> from mexca.audio import SpeakerIdentifier, VoiceExtractor >>> from mexca.text import AudioTranscriber, SentimentExtractor >>> from mexca.video import FaceExtractor >>> num_faces = 2 >>> num_speaker = 2 >>> pipeline = Pipeline( ... face_extractor=FaceExtractor(num_faces=num_faces), ... speaker_identifier=SpeakerIdentifier( ... num_speakers=num_speakers ... ), ... voice_extractor=VoiceExtractor(), ... audio_transcriber=AudioTranscriber(), ... sentiment_extractor=SentimentExtractor() ... ) Create a pipeline with container components. >>> from mexca import Pipeline >>> from mexca.container import AudioTranscriberContainer, FaceExtractorContainer, >>> SentimentExtractorContainer, SpeakerIdentifierContainer, VoiceExtractorContainer >>> num_faces = 2 >>> num_speaker = 2 >>> pipeline = Pipeline( ... face_extractor=FaceExtractorContainer(num_faces=num_faces), ... speaker_identifier=SpeakerIdentifierContainer( ... num_speakers=num_speakers ... ), ... voice_extractor=VoiceExtractorContainer(), ... audio_transcriber=AudioTranscriberContainer(), ... sentiment_extractor=SentimentExtractorContainer() ... ) Create a pipeline with standard *and* container components. >>> from mexca import Pipeline >>> from mexca.audio import SpeakerIdentifier, VoiceExtractor >>> from mexca.container import AudioTranscriberContainer, FaceExtractorContainer, >>> SentimentExtractorContainer >>> num_faces = 2 >>> num_speaker = 2 >>> pipeline = Pipeline( ... face_extractor=FaceExtractorContainer(num_faces=num_faces), ... speaker_identifier=SpeakerIdentifier( # standard ... num_speakers=num_speakers ... ), ... voice_extractor=VoiceExtractor(), # standard ... audio_transcriber=AudioTranscriberContainer(), ... sentiment_extractor=SentimentExtractorContainer() ... ) .. py:method:: apply(filepath: Union[str, collections.abc.Iterable], frame_batch_size: int = 1, skip_frames: int = 1, process_subclip: Tuple[Optional[float]] = (0, None), return_embeddings: bool = False, language: Optional[str] = None, keep_audiofile: bool = False, merge: bool = True, show_progress: bool = True) -> Union[mexca.data.Multimodal, collections.abc.Iterable] Extract emotion expression features from a video file. This is the main function to apply the complete mexca pipeline to a video file. :param filepath: Path to the video file or iterable returning paths to multiple video files. :type filepath: str or collections.abc.Iterable :param frame_batch_size: Size of the batch of video frames that are loaded and processed at the same time. :type frame_batch_size: int, default=1 :param skip_frames: Only process every nth frame, starting at 0. :type skip_frames: int, default=1 :param process_subclip: Process only a part of the video clip. Must be the start and end of the subclip in seconds. `None` indicates the end of the video. :type process_subclip: tuple, default=(0, None) :param return_embeddings: Return embeddings for each detected face. For large input files, this can increase the size of the output substantially as a 512-element vector is stored for each face. Face embeddings are stored in the :class:`video_annotation` attribute of the :class:`Multimodal` object. :type return_embeddings: bool, default=False :param language: The language of the speech that is transcribed. If `None`, the language is detected for each speech segment. :type language: str, optional, default=None :param keep_audiofile: Keeps the audio file after processing. If False, the audio file is only stored temporarily. :type keep_audiofile: bool, default=False :param merge: Whether to merge the output from the different components into a single :class:`polars.LazyFrame`. If `True` (default), the method :func:`merge_features` is called after all components finished processing and a :class:`polars.LazyFrame` is stored at the `features` attribute. If `False`, the method is not called and the `features` attribute is `None`. :type merge: bool, default=True :param show_progress: Enables progress bars and printing info logging messages to the console. The logging is overriden when a custom logger is explicitly created. :type show_progress: bool, default=True :returns: A data class object that contains the extracted merged features in the `features` attribute. See the `Output `_ section for details. If `filepath` is an :class:`collections.abc.Iterable` returns an :class:`collections.abc.Iterable` of :class:`mexca.data.Multimodal` objects. :rtype: Multimodal or collections.abc.Iterable .. seealso:: :obj:`mexca.data.Multimodal` .. rubric:: Examples >>> import polars as pl >>> from mexca.data import Multimodal >>> # Single video file >>> filepath = 'path/to/video' >>> output = pipeline.apply(filepath) >>> assert isinstance(output, Multimodal) True >>> assert isinstance(output.features, pl.LazyFrame) True >>> # List of video files >>> filepaths = ['path/to/video', 'path/to/another/video'] >>> output = pipeline.apply(filepaths) >>> assert isinstance(output, list) True >>> assert [isinstance(r, Multimodal) for r in output] True