Led by the inventor who sold the video launcher to Apple, Panjaya uses deep bite-sized methods to copy video.
There’s a huge opportunity for generative AI in the world of translation, and a startup called Panjaya takes the idea to the next level: an AI-based composition tool for videos that recreates the real voice of someone speaking a new language. , with video and the speaker’s body movements automatically adjusting to naturally match new speech patterns.
After being stealthy for the past three years, the startup unveiled BodyTalk, the first version of its product, and its first external funding of $9.5 million.
Panjaya is the brainchild of Hilik Shani and Ariel Shalom, two deep learning experts who spent most of their professional lives working on deep learning technology for the Israeli government and are now the startup’s general manager and CTO respectively. They hung up their G-man hats in 2021 with the first bite, and 1.5 years ago, they were joined by Guy Piekarz as CEO.
Piekarz is not the founder at Panjaya, but he is a notable name to have: Back in 2013, he sold the startup he created. he did found in Apple. Matcha, as the startup was called, was a morning player, engaged in the discovery and recommendation of broadcast video, and was discovered in the early days of Apple’s TV strategy and broadcasting, when these were more rumors than real products. Matcha was refrigerated and sold for a song: $10 million to $15 million — a pittance considering the significant direction Apple has finally made in broadcast media.
Piekarz stayed with Apple for almost ten years building the Apple TV and then its games stopped. After that, he was introduced to Panjaya through Viola Ventures, one of its backers (others include R-Squared Ventures, JFrog co-founder and CEO Shlomi Ben Haim, Chris Rice, Guy Schory, Ryan Floyd of -Storm Ventures, Ali Behnam of Riviera Partners, and Oded Vardi.
“I had left Apple at the time and was planning to do something completely different,” Piekarz said. “However, seeing the technology demo blew me away, and the rest is history.”
BodyTalk is interesting in how it simultaneously brings several pieces of technology that play on different aspects of synthetic media into a frame.
It starts with voice-based translation that can currently provide translation in 29 languages. The translation is then spoken by a voice imitating a real speaker, which is then set to the original video version where the speaker’s lips and other movements are adjusted to fit the new words and sentences. All of this is automatically created for videos after users upload them to the platform, which also comes with a dashboard that includes additional editing tools. Future plans include an API, and closer to real-time processing. (Currently, BodyTalk is “close to real-time,” taking minutes to process videos, Piekarz said.)
“We use the best type where we need it,” Piekarz said of the company’s use of large-scale modeling languages and other tools. “And we build our AI models where the market doesn’t really have a solution.”
An example of that is the company’s lip sync, which continues. “Our entire lip-syncing engine is born from our AI research team, because we haven’t found anything that comes to that level and quality for so many speakers, angles, and all the business use cases we want to support.”
Its current focus is on B2B only; clients include JFrog and the TED news organization. The company has plans to expand widely into media, especially in areas such as sports, education, marketing, healthcare and medicine.
The resulting translated videos are very unusual, not unlike what you get with deepfakes, although Piekarz wins at the time, which has taken a poor translation over the years that is exactly the opposite of the market the startup is targeting.
“‘Deepfake’ is not something we like,” he said. “We’re looking to avoid that whole term.” Instead, he said, think of Panjaya as part of a “deep real class.”
By targeting only the B2B market, and controlling who has access to its tools, the company is creating “guardrails” around the technology to protect against misuse, he added. He also thinks that in time more tools will be developed, including watermarking, to help detect when videos have been altered to create artificial media, both legal and illegal. “We definitely want to be a part of that and not allow misinformation,” he said.
Not so fine print
There are a number of startups competing with Panjaya in the broader AI-based video translation space, including big names like Vimeo and Eleven Labs, as well as smaller players like Speechify and Synthesis. For all of them, creating ways to improve the way copy works is like swimming against strong waves. That’s because captions have become a very common part of how video is consumed these days.
In TV, it’s due to a number of reasons such as poor speakers, background noise from our busy lives, groaning actors, limited production budgets, and extra sound effects. CBS found in a survey of American TV viewers that more than half of them keep subtitles on “some (21%) or all (34%) of the time.”
But some love captions just because they’re fun to read, and there’s been a whole religion built around that.
In social media and other applications, subtitles are simply baked into the information. TikTok, as one example, began in November 2023 to automatically turn on captions for all videos.
However, there remains a large global market for dubbed content, and although English is often thought of as the language of the Internet, there is evidence from research groups such as CSA that content delivered in native languages gets better engagement, especially in the B2B context. Panjaya’s pitch is that native language content can do even better.
Some of its customers seem to support that view. TED says Talks delivered using the Panjaya tool have seen views increase by 115%, with completion rates doubling for those translated videos.
Source link
/cdn.vox-cdn.com/uploads/chorus_asset/file/25728924/STK133_BLUESKY__B.jpg?w=390&resize=390,220&ssl=1 "Bluesky now has a chat tab in your notification area")