# Diarisation

Welcome to our model card for Diarisation. This model card describes our currently deployed diarisation model available via our API.

## **Lelapa-X-Diarisation** 

# **Model Details** 

*Basic information about the model:* Review section 4.1 of the [model cards paper](https://arxiv.org/abs/1810.03993).

| Organization | Lelapa AI |
| :---- | :---- |
| Product | Vulavula |
| Model date | 31 March 2024 |
| Feature | Diarisation |
| Lang | Language Agnostic |
| Domain | N/A |
| Model Name | Lelapa-X-Diarisation |
| Model version | 1.0.0 |
| Model Type | Proprietary Model |

**Information about training algorithms, parameters, fairness constraints or other applied approaches, and features:** Proprietary Model Tested on Audio Data

**License:** Proprietary

**Contact**: info@lelapa.ai

## **Intended use** 

*Use cases envisioned during development:* Review section 4.2 of the [model cards paper](https://arxiv.org/abs/1810.03993).

### **Primary intended uses**

Intended use is governed by the language and domain of the model. The model is intended to be used for audio diarisation. The model is not suitable for settings with multiple speakers (\>4) and thus should be used with caution.


### **Primary intended users**

The Diarisation model can be used for :

* Audio Diarization in the call center domain  
* Improved ASR performance  
* Market Research and Analysis  
* Compliance monitoring for Customer Interactions

### **Out-of-scope use cases** 

The model is  language invariant and cannot  be used in settings different from telephonic or meeting recordings.

## **Factors**

*Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others listed in Section 4.3:* Review section 4.3 of the [model cards paper](https://arxiv.org/abs/1810.03993).

### **Relevant factors** 

Groups:

* Users who recorded utterances used to test the model are diverse across several factors such as age, location, and gender.   
* Performance across groups is underway.

Environmental conditions, Instrumentation and technical attributes:

* Audio utterances are recorded in environments such as rooms, and call centers with a noiseless background. 

## **Metrics** 

*The appropriate metrics to feature in a model card depend on the model being tested. For example, classification systems in which the primary output is a class label differ significantly from systems whose primary output is a score. In all cases, the reported metrics should be determined based on the model’s structure and intended use:* Review section 4.4 of the [model cards paper](https://arxiv.org/abs/1810.03993).

### **Model performance measures** 

The model is evaluated using Diarization Error Rate (DER): the models’ performances are measured by automatic metric. DER It measures the fraction of time that is not attributed correctly to a speaker or to non-speech. DER quantifies the error rate in the diarization output by considering three types of errors:

* Missed Speech: This occurs when the system fails to detect a segment of speech, resulting in a missing speaker label.  
* False Alarm: This happens when the system incorrectly assigns a speaker label to a segment that should not have a speaker label.  
* Speaker Error: This occurs when the system assigns the wrong speaker label to a segment.

The Diarization Error Rate is the sum of these error types, normalized by the total duration of the reference data (ground truth). The smaller the DER, the better the performance.

**DER** \= Missed\_Speech+False Alarm+Speaker Error / Total\_reference\_duration

**Decision thresholds**

No decision thresholds have been specified

### **Evaluation data** 

*All referenced datasets would ideally point to any set of documents that provide visibility into the source and composition of the dataset. Evaluation datasets should include datasets that are publicly available for third-party use. These could be existing datasets or new ones provided alongside the model card analyses to enable further benchmarking.*

Review section 4.5 of the [model cards paper](https://arxiv.org/abs/1810.03993).

### **Datasets** 

* Publicly available datasets in various languages  
* Proprietary call center dataset

### **Motivation**

These datasets have been selected because they are open-source, high-quality, and cover the different languages. These help to challenge the model under different settings and languages and assess whether it can learn voice representations independently from the spoken language.

## **Training data** 

Review section 4.6 of the [model cards paper](https://arxiv.org/abs/1810.03993).

N/A

## **Quantitative analyses** 

*Quantitative analyses should be disaggregated, that is, broken down by the chosen factors. Quantitative analyses should provide the results of evaluating the model according to the chosen metrics, providing confidence interval values when possible.*

Review section 4.7 of the [model cards paper](https://arxiv.org/abs/1810.03993).

### **Unitary results**

| Dataset | AMI | VoxConverse | Proprietary Zulu | AliMeeting |
| ----- | ----- | ----- | ----- | ----- |
| Number of Speakers | \[3, 4\] | \[2, 21\] | 2 | \[3, 10\] |
| Avg Speakers | 3.9375 | 6.47 | 2 | 4.7 |
| Total Duration | 8h 53mn | 43h 15mn | 49h 47mn | 10h 44mn |
| Overlaps Duration | 44mn | 52mn | 2h 51mn | 43mn |
| Number of Overlaps | 2851 | 4416 | 9863 | 4126 |
| Proportion of Overlaps | 8.34% | 2.02 % | 5.73 % | 6.63 % |
| DER | 8.69% | 14.06% | 2.91% | 11.85% |

### **Intersectional result** 

Our experiments show that:

* High DER on other datasets could be due to challenges with speech overlap and the number of speakers.

* Noise reduction impacts audio quality and DER, highlighting a trade-off between noise removal and accuracy.

* The model shows potential for applications with a moderate number of speakers but may struggle with complex scenarios with many speakers.

## 

## **Ethical considerations**

*This section is intended to demonstrate the ethical considerations that went into model development, surfacing ethical challenges and solutions to stakeholders. The ethical analysis does not always lead to precise solutions, but the process of ethical contemplation is worthwhile to inform on responsible practices and next steps in future work:* Review section 4.8 of the [model cards paper](https://arxiv.org/abs/1810.03993).

More details are in the datasheet.

## **Caveats and recommendations** 

*This section should list additional concerns that were not covered in the previous sections.*

Review section 4.9 of the [model cards paper](https://arxiv.org/abs/1810.03993).

This model is limited by its limits in number of speakers between 3-5 max. The results on the benchmark datasets show that the model may not generalize well for datasets with a high amount of speakers.