Warsaw University of Technology students developed an algorithm for converting video into comics
A comic strip generated by Comixify from a fragment of the movie "Pulp Fiction". Fig. Comixify. Source: Warsaw University of Technology
The algorithm Comixify for converting video into comics developed at Warsaw University of Technology met with interest around the world. In less than a month, the Comixify website recorded over 180,000 viewers and was used to generate over 22 thousand comics, told PAP Dr. Tomasz Trzciński from Warsaw University of Technology.
Comixify automatically selects the most interesting and important frames from the uploaded video, then arranges them, adjusts to frames and converts them graphically. This is how the comic strips are created.
The authors of the project are students of the Faculty of Electronics and Information Technology of the Warsaw University of Technology: Maciej Pęśko, Adam Svystun and Paweł Andruszkiewicz. Scientific supervisors were the employees of the Department of Computer Graphics of the Institute of Computer Science: Prof. Przemysław Rokita and Dr. Tomasz Trzciński.
Comixify was born from a combination of students` interests in comics and style transfer using machine learning and the department`s previous publications on topics such as predicting the popularity of content on the Internet.
According to a release on the university website, video conversion into comics is divided into two main stages: frame extraction and style transfer. First, representative frames from the video are extracted. To do this, the researchers developed an algorithm based on Reinforcement Learning, the task of which is to intelligently summarize the video.
Thanks to the addition of a module that estimates the quality of the image and its popularity, Comixify - according to its creators - is able to select not only the most representative frames from the film, but also those that have the highest aesthetic and popularity potential.
After extracting the frames, the images are transferred to comic style. To this end, researchers implement a neural style algorithm based on Generative Adversarial Networks (GANs). It is a machine learning technique that uses two different networks: a generator and a discriminator.
During training - the creators explain - the algorithm receives data such as pictures, which the generator uses to create new data instances. The discriminator checks whether the instance comes from a set of training (real) data or from data generated by the generator network (false).
The process continues until the generator creates instances so similar to the training set that the discriminator will not be able to tell the difference, describe the authors of the project.
On Comixify website you can see comics generated from fragments of movies such as "Pulp fiction", "Star Wars: Episode I - The Phantom Menace". The tool developed by the Warsaw University of Technology team can be tested free of charge by uploading own files (up to 50 MB) or using YouTube links.
According to the university`s representatives, the scientific publication describing the algorithm caused a wave of interest around the world. The creators received numerous inquiries from film producers and comic book publishers from Europe and the United States interested in this solution. They are currently looking for financing opportunities for the project and its commercialisation, adds Dr. Tomasz Trzciński.
Researchers from the Warsaw University of Technology intend to improve their project, expand it with new functionalities, including voice recognition and generating speech balloons, that is, adding text to images. (PAP)
szu/ agt/ kap/
tr. RL
Przed dodaniem komentarza prosimy o zapoznanie z Regulaminem forum serwisu Nauka w Polsce.
