Transcription - African French
Welcome to our model card for African French Transcription. This model card describes our currently deployed transcription model available via our API.
Approaches to Uncertainty and Variability 4
Model Details
Basic information about the model: Review section 4.1 of the model cards paper.
| Organization | Lelapa AI |
|---|---|
| Product | Vulavula |
| Model date | 31 October 2023 |
| Feature | ASR |
| Lang | French (African) |
| Domain | Government |
| Model Name | Lelapa-X-French |
| Model version | 1.0.0 |
| Model Type | Fine-Tuned Proprietary Model |
Information about training algorithms, parameters, fairness constraints or other applied approaches, and features: Proprietary fine-tuning of a pre-trained model on 15 hours of African French speech data.
License: Proprietary
Contact: info@lelapa.ai
Intended use
Use cases that were envisioned during development: Review section 4.2 of the model cards paper.
Primary intended uses
Intended use is governed by the language and domain of the model. The model is intended to be used transcribe calls conducted in (African) French. The model may not be suitable for the general conversation domain and should be used with extreme caution in high-risk environments.
Primary intended users
Transcription to enable analysis for downstream tasks for African French:
- Enabling search and filter of conversations
- Analysis
Out-of-scope use cases
All domains and languages outside of the government space for African French.
Factors
Factors could include demographic or phenotypic groups, environmental conditions, technical attributes, or others listed in Section 4.3: Review section 4.3 of the model cards paper.
Relevant factors
Groups:
- Cameroon: 2003 in Yaoundé, Cameroon. It has recordings from 84 speakers, 48 male and 36 female.
- CA16: Gabon in June 2016. It has recordings from 125 speakers from Cameroon, Chad, Congo and Gabon.
- Niger: This part was collected from 23 speakers in Niamey, Niger, Oct. 26-30 2015.
Metrics
The appropriate metrics to feature in a model card depend on the model being tested. For example, classification systems in which the primary output is a class label differ significantly from systems whose primary output is a score. In all cases, the reported metrics should be determined based on the model’s structure and intended use: Review section 4.4 of the model cards paper.
Model performance measures
The model is evaluated using WER as well as human evaluation: The models’ performances are measured by both automatic metrics and human evaluation. As an automatic metric, we use the Word Error Rate (WER) which is based on the edit distance also called Levenshtein distance. WER is not a symmetric distance metric, since it measures the number of operations: substitution, deletion, insertion, number of correct words needed to leave a reference sentence A to a predicted sentence B. Read more. As far as human evaluation is concerned, this stage is performed by paid linguists, native speakers of the languages. Evaluation is also done after post-processing techniques are performed on the outputs.
WER: Testing on general African French data
Decision thresholds
No decision thresholds have been specified
Approaches to Uncertainty and Variability
For fairness, robustness, and generalization with respect to languages and datasets, we leveraged standard downsampling and normalization techniques which have proven to be useful.
Evaluation data
All referenced datasets would ideally point to any set of documents that provide visibility into the source and composition of the dataset. Evaluation datasets should include datasets that are publicly available for third-party use. These could be existing datasets or new ones provided alongside the model card analyses to enable further benchmarking.
Review section 4.5 of the model cards paper.
Datasets
- Publicly available datasets in the government domain
Motivation
These datasets have been selected because they are open-source, high-quality, and cover the targeted languages - and utterances are recorded by a variety of speakers living in required regions. These help to capture interesting cultural and linguistic aspects that would be crucial in the development process for better performance.
Preprocessing
Audio files are normalised to a sample rate of 16kHz and transcripts are normalised We also make sure to select actual recordings i.e. recordings that are not just noise or blank.
Training data
Review section 4.6 of the model cards paper.
Quantitative analyses
Quantitative analyses should be disaggregated, that is, broken down by the chosen factors. Quantitative analyses should provide the results of evaluating the model according to the chosen metrics, providing confidence interval values when possible.
Review section 4.7 of the model cards paper.
Unitary results
| WER | |
|---|---|
| WER African Accented French | 5.87% |
Human evaluation
This is a breakdown of the types of errors we are seeing based on a sample of the evaluation dataset.
*Note: some samples suffered from more than 1 type of error
| African_accented_french | # |
|---|---|
| Insertion | 35 |
| Deletion | 10 |
| Substitution | 128 |
Intersectional result
In progress
Ethical considerations
This section is intended to demonstrate the ethical considerations that went into model development, surfacing ethical challenges and solutions to stakeholders. The ethical analysis does not always lead to precise solutions, but the process of ethical contemplation is worthwhile to inform on responsible practices and next steps in future work: Review section 4.8 of the model cards paper.
The model does not contain any personal information..
Caveats and recommendations
This section should list additional concerns that were not covered in the previous sections.
Review section 4.9 of the model cards paper.
Additional caveats are outlined extensively in our Terms and Conditions.