Skip to main content

Video Captioning using NLP

Natural Language Processing is about taking the languages used by humans in written or spoken form, with all its ambiguity and richness to convey crisp logic, rhetorics, sentiments and bring it into a machine representation.

Lets look at Video Captioning through NLP. Generating captions for image has been done with desirable results but work for captioning videos that follows the actions to form a sort of storyboard is still ongoing.
Motivation for doing this can be for:
  • Prediction of actions. 
  • Security Camera Log.
  • Generating summary of the video.
  • Audio description can help blind people if applied in real time using their viewpoint as input.
  • Allowing for text search in video which takes you to a certain point in the timeline automatically based on query. 
The general followed workflow is like this:
 
The Video is segmented into frames based on Action Localization / Scene Transition / Image Slices. These Frames are processed for noise removal & feature extraction. This is fed into CNN or RNN or both, to generate descriptive captions. The captions are contextually independent and are linked together to form a narrative using NLP techniques such as Coreference resolution, Connective word generation to identify nouns, verbs, etc. across the captions.

This technique is also useful to aligning caption transcript to a video automatically. Using the same procedure and identifying the aligning captions in transcript with captions generated using above techniques.

The above area is highly applicable in the rich social network of Twitter, Facebook, Youtube, etc. that we live in.

References:


Comments

Popular posts from this blog

NLP in Video Games

From the last few decades, NLP (Natural Language Processing) has obtained a high level of success in the field  of Computer Science, Artificial Intelligence and Computational Logistics. NLP can also be used in video games, in fact, it is very interesting to use NLP in video games, as we can see games like Serious Games includes Communication aspects. In video games, the communication includes linguistic information that is passed either through spoken content or written content. Now the question is why and where can we use NLP in video games?  There are some games that are related to pedagogy or teaching (Serious Games). So, NLP can be used in these games to achieve these objectives in the real sense. In other games, one can use the speech control using NLP so that the player can play the game by concentrating only on visuals rather on I/O. These things at last increases the realism of the game. Hence, this is the reason for using NLP in games.  We ...

Discourse Analysis

NLP makes machine to understand human language but we are facing issues like word ambiguity, sarcastic sentiments analysis and many more. One of the issue is to predict correctly relation between words like " Patrick went to the club on last Friday. He met Richard ." Here, ' He' refers to 'Patrick'. This kind of issue makes Discourse analysis one of the important applications of Natural Language Processing. What is Discourse Analysis ? The word discourse in linguistic terms means language in use. Discourse analysis may be defined as the process of performing text or language analysis, which involves text interpretation and knowing the social interactions. Discourse analysis may involve dealing with morphemes, n-grams, tenses, verbal aspects, page layouts, and so on. It is often used to refer to the analysis of conversations or verbal discourse. It is useful for performing tasks, like A naphora Resolution (AR) , Named Entity Recognition (NE...

Cross Language Plagiarism Detection

Plagiarism: the practice of taking someone else work and presenting that as of their own. So its reverse Plagiarism detection is detecting weather a given piece of document is plagiarised or not. Detection of plagiarism was so naive in earlier times ,People used to just compare strings across 2 documents , then as the technology grew people also grew smarter , they started coming with ideas by which they can copy a document while go undetected from detection. In this post i will talk about a particular type of plagiarism detection which is essentially one of hardest in this domain i.e Cross-Language plagiarism detection(CLPD) . People take a piece of document and convert it into some other language and then post it, now our task is to detect weather it is copied from some other language or not ? In this post i will keep my limits to documents only not programming code, though there are many ways as well by which it can be detected that weather a piece of code in converted into some ...