^{NLP - ML - Audio - Speech to text - Transcription}

‍

Background

‍

Customer data is often at the heart of any project start. But whereas textual and image data are typically easier to collect and label, audio data is often more scarce and tricky.

‍

This typically results in a few rounds of “back and forth” at a project start with typical questions such as:

• How much audio data do you need
• How much of that should be labelled
• How should we label the audio data
• Etc.

‍

In this internship, we aim to answer those questions to as specific a degree as possible. For this, you will focus on creating a speech transcription engine that can identify and accurately transcribe your colleagues in various contexts. In this process, you’ll identify key relationships between data quantity, quality, type and model accuracy. You’ll then package all of this into a demo which can take various forms for your fellow agents and the world to use! 

‍

This, dear ML6 Intern agent, is your mission.

‍

Goal

The goals of this internship are as follows:

• Identify the relationship between quantity of data vs accuracy of transcriptions on new data.
• Identify data qualities (if any) which improves transcription accuracy the best.
• Quantify the utility of hotwords in domain adaptation of speech transcription models.
• Identify the differences between the above relationships for English and a lower-resource language (in this case, Nederlands).
• Use all these learnings to create a small demo (web app, smart assistant, chrome extension …)  for ML6 and the world to use! 
• Give a kick-ass presentation to your fellow agents and look cool while doing it!

‍

The adoption of NLP models in real-world use cases has also meant the rise of niche domain-specific language models. Do you need a language model for Polish legal text? Sure thing. Or how about a language model for Swedish medical texts? Look no further. However, in many real-world situations, this begs the question of when we decide to explore using a custom language model or when we decide that an “out-of-the-box” language model will suffice. In some fields like the medical field, this question is trivial as most of the important words (e.g, names of diseases, Latin anatomy language, etc.) are out-of-vocabulary for general-purpose language models but in other domains such as legal texts, technical manuals, etc. it is much less obvious.

‍

The goal of this project is to estimate the expected effect that using a custom language model will have over an existing one based on known heuristics. The current best approach is to compare the overlap of n-grams from your domain-specific texts to those from the texts that the existing solution was trained on and install some (arbitrary) cut-off (i.e, if the n-gram overlap is under 30%, we explore using a custom language model). However, this method is not very quantitative nor very rigorous.

‍

‍

Methodology / Tasks

During this internship, you will:

• Gain experience leveraging state-of-the-art NLP techniques.
• Conduct applied research to solve a concrete real-world problem.
• Let your creative skills loose on a major problem for many companies in a wide range of domains.

‍

Profile / Required skills

• Strong analytical abilities, knowledge of different statistical methods, not scared by mathematics and a familiarity with research studies.
• Strong interest in Computer Vision / NLP / Other subdomain [preferred]
• Familiarity with statistical analysis languages and tools like Python, SQL.
• Excellent verbal and written communication in English.
• You are currently pursuing a degree in computer science or related field.

‍

Internship Duration

The duration of the internship can be flexible and depends on the candidate preference and the project requirements. The typical duration is 6 to 8 weeks. The preferred duration for this specific project is 6 weeks.

‍

Chapters

Our internships and theses are linked to our chapters. A chapter is a cross-squad team of experts in a specific topic to enable knowledge building and sharing across projects. The chapters build knowledge by performing applied research and gathering learnings from projects. This internship falls under the NLP chapter.

‍

Background

Companies have a massive amount of information available within reach, only a few clicks away. The challenge with these large volumes of information is that we, as a human being, are not able to digest it properly. Search Engines like Google are helping to prioritise information based on your query and interests. However, we noticed, with the trend of open data, that not all data is indexed in Google, leaving multiple sources “undiscoverable”. 

‍

One type of information is the local, regional and federal political information. Vast amounts of reports, detailed research documents, … are available for mining but it is difficult to valorize them, since the information is “stuck” in PDF, or, as mentioned, is not indexed by Google. Many political decision documents have become more and more open-source. Albeit on a local municipal level or on a district level, these data and metadata have found their way into Linked Open Data platforms and databases.

‍

In this project, we wish to solve this challenge. We want to enable many companies to reach that information with ease, based on their interests in the political chatter available about their company, line of work or sector.

‍

A traditional approach to this problem is to have literal people spending hours combing through government statements and meeting notes to sift out nuggets of information pertaining to a certain context. A more modern approach however, is to use advanced NLP techniques to do this automatically on a large scale.

Goal

We want to address this gap by creating an end-to-end application that:

• Periodically scrapes relevant websites (e.g. beslissingenvlaamseregering.vlaanderen.be, https://data.dekamer.be/, https://www.vlaamsparlement.be/nl/parlementair-werk/plenaire-vergadering/vergaderingen , …)
• Extract relevant information in a batch-wise or stream-wise fashion
• Expose the functionality through a proper backend
• Develop a basic frontend to interact with the backend in order to get insights

‍

In order to get companies:

• To find the crucial and relevant topics in this vast amount of information, find the needle in the haystack.
• Targeted insights in specifically important events

‍

Your mission, dear ML6 Intern agent, should you choose to accept it, is exactly this!

Functional solution

On a high-level technical perspective, the solution could look as follows:

‍

Of course, things aren’t set in stone, and the finalisation of the functional design, as well as the translation into the technical design, is something that can happen in collaboration with senior engineers at ML6.

Technologies involved

On a machine learning level:

• NLP

• Keyword extraction
• Named entity recognition
• Extractive summarization

• Engineering-wise

• Serverless backend applications
• Microservice architecture
• Apache Beam
• Event-driven task architecture
• Data Warehousing
• Scraping (Scrapy framework)

• General way of working

• Google Cloud Platform
• Trello
• Bitbucket

‍

So if you are a person with a broad set of interests in Machine Learning, Data Engineering and Software Engineering: you are the agent for the job 😎!

‍

Project Description 

In recent times the general trend towards automation has meant that use cases which involve processing large amounts of data are becoming automated. The reasons for this are quite obvious: these are often repetitive time-consuming tasks that are prone to human error and lend themselves to being automated. However, still one of such tasks remains and that is reading. Unfortunately, we can’t automate reading but we can make it faster by highlighting the key information in the text.

‍

The goal of this project is to develop an algorithm that highlights the most important information in a document. However, how we define important information requires some creativity (i.e, it could mean sentences that summarise the text, sentences that are unexpected, etc.). Concretely, we see a major use case for legal texts where we can also exploit the repetitive nature of such documents (e.g, rental contracts are 90% the same because they need a certain legal structure) but we are open to suggestions if there is another field that interests you more where it could also have a big impact.

‍

‍

Methodology / Tasks

During this internship, you will:

• Gain experience leveraging state-of-the-art NLP techniques.
• Conduct applied research to solve a concrete real-world problem.
• Let your creative skills loose on a major problem for many companies in a wide range of domains.

‍

Profile / Required skills

• Strong analytical abilities, knowledge of different statistical methods, not scared by mathematics and a familiarity with research studies.
• Strong interest in Computer Vision / NLP / Other subdomain [preferred]
• Familiarity with statistical analysis languages and tools like Python, SQL.
• Excellent verbal and written communication in English.
• You are currently pursuing a degree in computer science or related field.

‍

Internship Duration

The duration of the internship can be flexible and depends on the candidate preference and the project requirements. The typical duration is 6 to 8 weeks. The preferred duration for this specific project is 6 weeks.

‍

‍

Background

As NLP stands for ‘Natural Language Processing’, it’s not surprising that the field typically dealt with ‘Natural Language’: human-like conversational and written text.

‍

Recently though, the NLP field has been widening it’s field of view to also include processing ‘Programming Language’.

‍

Clear indications of this interesting evolution are:

• The rise of appropriate metrics (eg. CodeGlue)
• The rise of enormous datasets (eg. CodeSearchNet)
• The rise of benchmarks (eg. CodeXGlue)
• The rise of pretrained models (eg. CodeT5, CodeBERT, PLBart, etc.)
• The rise of commercial products (eg. OpenAI Codex / Github CoPilot)

‍

So due time we investigate this emerging field and try to make our mark on it!

‍

One mark we can make on this is to make a commit message generator application. AI is especially useful when used to automate human-like tasks that are not that hard, add value and that humans dislike / skip over. Creating meaningful commit messages definitely falls under this category!

‍

The idea is to train or finetune a seq2seq model (so a model that can convert text to another text) to create a commit message, like illustrated with an example below:

‍

This, dear ML6 Intern agent, is your mission.

‍

Goal

The internships encompasses various steps:

1. Researching the existing field of NLP+Code
2. Identifying relevant architectures to test
3. Identifying or scrape relevant code datasets to use
4. Find a suitable way of representing the input diff data
5. Training a successful seq2seq model
6. Open-sourcing the obtained models (and datasets)
7. Writing a blog post
8. Creating a small cloud-based serverless application to demonstrate the model(s)
9. [extension] creating and open sourcing a vscode plugin that can be used by developers

‍

Project Description 

In recent times anomaly detection has become a major topic within the field of AI. It has particularly gained traction within the domain of computer vision with use cases in defect detection, predictive maintenance, etc. and within the domain of structured data with use cases in fraud detection, spam filtering, etc. However, the development of similar techniques with the domain of NLP remains an understudied subject despite its potential.

‍

The goal of this project is to leverage different NLP techniques to arrive at an algorithm that can accurately highlight anomalous words and/or sentences in a document that you wouldn’t expect to appear in that document. This system would likely exploit the repetitive nature of certain types of documents (e.g, rental contracts are 90% the same because they need a certain legal structure). If successful, such an algorithm could have very impactful use cases in fields such as the legal domain, insurance companies, etc. The concrete approach would be to develop such a system on legal documents but we are open to suggestions if there is another field that interests you more where it could also have a big impact.

‍

‍

Methodology / Tasks

During this internship, you will:

• Gain experience leveraging state-of-the-art NLP techniques.
• Conduct applied research to solve a concrete real-world problem.
• Let your creative skills loose on a major problem for many companies in a wide range of domains.

‍

Profile / Required skills

• Strong analytical abilities, knowledge of different statistical methods, not scared by mathematics and a familiarity with research studies.
• Strong interest in Computer Vision / NLP / Other subdomain [preferred]
• Familiarity with statistical analysis languages and tools like Python, SQL.
• Excellent verbal and written communication in English.
• You are currently pursuing a degree in computer science or related field.

‍

Internship Duration

The duration of the internship can be flexible and depends on the candidate preference and the project requirements. The typical duration is 6 to 8 weeks. The preferred duration for this specific project is 8 weeks.

‍

Project Description

With the increase of globalisation and the current rise in numbers of people moving abroad, learning a new language in an affordable and fun way is becoming more and more important. Parlangi is an e-learning provider which connects native speakers and learners of a specific language. They are invited to use a platform for video calls to talk and improve the language skills of the language learner.

‍

As of now, Parlangi provides different topics of conversations every 10 minutes to engage the speakers. Although this approach is relevant to keep the conversation going, it does have a few potential drawback as it can either be:

‍

1. Intrusive: A disruption of the flow of conversation can occur if both speakers are engaging in the topic of conversation and a topic switch pops up.

2. Infrequent: It might preferable to suggest a topic switch early on if the topic of conversation is not engaging both speakers (dominated by silence) or the topic is engaging only the native speaker (which defeats the purpose of the video call to improve the non-native speakers' speaking skills).

Goal

The goal of this project is to enhance the conversation topic suggestion feature and make it more dynamic. This can help provide a more fun and satisfying experience for the users of the platform. ‍

In order to do so, both the frequency and duration of silence of the conversation as well as the speech frequency of the individual speakers need to be determined. This information can be used to quantify the overall level of engagement of the speakers and suggest conversational topic switch at an appropriate timing.

‍

One method of approaching this problem is by applying speaker diarization techniques on the raw audio recording. Speaker diarization aims at answering the question “who spoke when”. With that, it is feasible to detect both the ‘speech’ moments of the individual speakers as well as the ‘silence’ segments as illustrated below.

This internship isn’t only a great way to leverage your skills working with audio and edge computing but also to do good. Your internship can be rounded up with a blog post where you share your learnings and how you helped Parlangi and its users by improving their experience of learning a new language.

‍

Methodology

You can take a headstart when working on this project, as some work has already been done. There exist many diarization libraries that already implement diarization pipelines to diarize audio recordings. Some initial exploration of those libraries has been done by ML6. However, there is still much work to be done to put this tool in practice. 

• An appropriate diarization framework has to be chosen for this task. Different trade-offs in accuracy/detection speed/resources need to be considered.
• An extension tool has to be implemented to access the audio stream from the open source video platform.
• The results of the diarization algorithm will be used in a control loop algorithm that proposes conversational topics dynamically.

‍

During this internship you will:

• Explore multiple state-of-the-art diarization algorithms.
• Implement an extension tool to access the raw audio data from the video platform. 
• Develop an end-to-end solution for dynamic topic suggestion and integrate it with the Parlangi platform.
• Write a blog post summarising your work..
• Do some good!

‍

Profile / Required skills

• Strong analytical abilities, knowledge of different statistical methods and a familiarity with research studies.
• Working experience in Java development to build a tool that interfaces with the video platformi.
• Strong interest in Speech/Audio processing [preferred].
• Familiarity with  tools like Python.
• Excellent verbal and written communication in English.
• You are currently pursuing a degree in computer science or related field.

‍

Internship Duration

The duration of the internship can be flexible and depends on the candidate preference and the project requirements. The estimated duration for this specific project is 6-8 weeks:

‍

• Week 1: Getting familiar with SoTA diarization algorithms and the open source video platform.
• Week 2-3: Build a tool that interfaces with the video platform to get an audio stream.
• Week 4-5: Integrate the diarization algorithm with the audio stream and build the control flow logic for the dynamic topic suggestions 
• Week 6: Validate the results of the algorithm and write a blogpost 

‍

Chapters

Our internships and theses are linked to our chapters. A chapter is a cross-squad team of experts in a specific topic to enable knowledge building and sharing across projects. The chapters build knowledge by performing applied research and gathering learnings from projects. This internship falls under the Speech/Audio working group which is part of the  Natural Language Processing (NLP) chapter.

‍

Supervisors

Thomas Dehaene: Chapter Lead

Lisa Becker: Machine Learning Engineer and Speech Working Group Lead (daily supervisor)

‍

References

• ML6 diarization algorithm exploration [link]
• Diarization libraries [link]

‍

‍

‍

‍

‍