[ad_1]
Final week, MBW printed a broadly shared report about TikTok and its mother or father ByteDance‘s ambitions within the subject of AI-generated music.
Inside that article, we informed you concerning the in depth analysis carried out by ByteDance’s Speech, Audio & Music Intelligence (SAMI) workforce within the subject of AI music.
A kind of analysis tasks describes a generative mannequin referred to as MeLoDy that was skilled, in accordance with ByteDance’s SAMI researchers, on 257,000 hours of music (learn the paper right here) that got here from 6.4 million audio recordsdata.
We additionally informed you about two US patents, secured by ByteDance in current months, that present IP safety within the US for the Chinese language firm’s AI music-related applied sciences.
Now, we’ve uncovered an further batch of AI music-related patent and trademark filings from ByteDance that may elevate various eyebrows within the music business – masking matters together with “knowledge mining” and “music technology”.
Curiously, these US patents and trademark functions had been filed by a mysterious ByteDance affiliate referred to as Lemon Inc.
There’s not numerous details about Lemon Inc. on-line. We do know, nevertheless, that (a) it’s included within the Cayman Islands, and (b) It’s been very busy over the previous couple of years submitting functions to the US Patent and Trademark Workplace. (It even utilized for a guide publishing-related trademark at first of 2023).
Lemon Inc. just isn’t talked about particularly on ByteDance’s company construction diagram (see under, and right here).
Nonetheless, the World Mental Property Group, through its Arbitration reviews, not too long ago described Lemon Inc. as being “affiliated with Bytedance Ltd (“Bytedance”), an Web expertise firm that gives cellular functions (“apps”) equivalent to TikTok”.
Lemon Inc. is to not be confused with one other ByteDance product referred to as Lemon8, the Instagram/Pinterest rival launched within the US final 12 months, which some speculate is being pushed available in the market as a “back-up” within the occasion of a possible TikTok ban.
As reported by Axios final 12 months, Lemon8 was developed by Singapore-based Heliophilia Pte., Ltd., which is registered on the similar deal with as TikTok in Singapore.
Why Lemon Inc. is getting used as a automobile to use for emblems and patents in the US on behalf of ByteDance is a query for an additional day.
For now, feast your eyes on what else ByteDance has been as much as…
Lemon Inc. PAtent 1: ‘Computerized and quick technology of music audio content material for movies’
In what presents the clearest glimpse at what ByteDance is likely to be planning to make use of its AI music-related tech for, its affiliate Lemon Inc owns a patent in the US for a “methodology of routinely and effectively producing music audio content material for movies”.
The patent software was filed in July 2022, granted in September 2023, and is assigned to each Lemon Inc. and ByteDance (see under).
In keeping with the submitting, which you’ll learn for your self right here, this patent covers “methods that allow environment friendly sensible music technology {and professional} music modifying”.
“After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video shall be routinely generated. Such music or audio could also be royalty-free and will be of any size.”
Excerpt from patent
The submitting provides: “After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video shall be routinely generated. Such music or audio could also be royalty-free and will be of any size.”
The submitting explains additional that the tech can extract “data from the video, whereby the extracted data includes data indicating movement pace within the video, data indicating movement saliency within the video, data indicating scene transition within the video, and timing data related to the video”.
It may possibly then generate a “plurality of units of music notes matching the plurality of segments of frames primarily based a minimum of partly on the extracted data utilizing a mannequin, whereby the mannequin is pre-trained and learns to correlate video movement pace with music observe density, correlate video movement saliency with music observe power, correlate video scene transition with music construction, and correlate video timing with music beat”.
Simply to additional spotlight the shut hyperlinks between Lemon Inc, TikTok and ByteDance, right here:
Amongst the named inventors of this tech within the patent submitting are Los Angeles-based researchers named Zhihao Ouyang and Daiyu Zhang, who in accordance with LinkedIn are a Senior Analysis Engineer at TikTok, and ML Analysis Engineer at ByteDance, respectively.
Lemon Inc Patent 2: Computing system and methodology for music technology
Lemon Inc additionally has one other patent pending for a “Computing system and methodology for music technology”.
This patent was filed by Lemon Inc USA in June 2022 and assigned to each Lemon Inc and ByteDance in September 2023 (see under). It was printed in December.
The submitting explains that “Packages have been developed that may generate music primarily based on a lyric inputted by a person” however that the music generated by such applications “typically lacks musical qualities that many individuals recognize, and thus isn’t very song-like”.
It continues: “For instance, auto-generated music from such applications can endure from misalignments in lyrics and melody notes, scattered or disjointed group and music construction, mismatched rhythm tracks, and lack of a catchy repeating melody.
“Because of this, such applications haven’t achieved widespread use. Because of this, a barrier presently exists to fast music growth utilizing such applications.”
The patent software, which you’ll see right here, describes “a music technology system” that options “a processor and a reminiscence operatively coupled to the processor and storing a rhythm template database comprising a plurality of rhythm templates”.
The invention additionally includes “a music technology program saved within the reminiscence and executed by the processor to be configured to obtain a person enter of lyrics, establish a plurality of syllables within the lyrics, decide a syllable sample within the recognized plurality of syllables, match the syllable sample to a specific rhythm template of the plurality of rhythm templates”.
The invention is then proposed to “generate a melody primarily based on the chosen rhythm template, generate a music file encoding the melody and the lyrics, and output the music file encoding the melody and the lyrics”.
The diagram under units out the way it works:
Lemon Inc.’s Emblems
Along with its lively and pending patents within the subject of AI Music, Lemon Inc. has additionally filed for a number of emblems in the US that check with AI, music technology and music tech on the whole.
A kind of emblems listed on the US Patent and Trademark Workplace (USPTO) is for a service referred to as ‘BytePlus‘ which seems to have been formally registered in Might 2023.
TikTok and Lemon Inc mother or father firm ByteDance launched BytePlus in 2021 reportedly as a brand new division to promote its AI expertise. ‘BytePlus’ is listed in ByteDance’s company construction diagram.
The trademark listed on the USPTO describes a “Platform as a Service (PaaS)” and covers what the submitting states are “pc software program platforms for producing and delivering personalised suggestions to prospects within the fields of synthetic intelligence, knowledge mining, knowledge analytics, and machine studying covers a spread of AI-related actions.
The trademark additionally covers “machine-learning primarily based music technology, and machine language translation; software program consultancy providers regarding synthetic intelligence, knowledge mining, knowledge analytics, and machine studying”.
Elsewhere on the USPTO database, a trademark is registered to Lemon Inc. for Mawf, ByteDance’s machine-learning-driven music-making app that analyzes incoming audio indicators after which “re-renders” these indicators utilizing what it says is machine-learned fashions of musical devices.
The Mawf trademark was utilized for in July 2022 and formally registered by the USPTO in September.
The submitting covers “Pc software program, particularly, downloadable software program for creating music movies, recorded music composition software program”.
Lemon Inc. has additionally filed (in Might 2023) two functions to trademark logos for what the filling seems to explain as a music-making app (see under). These logos resemble the logos of ByteDance’s Ripple app.
The appliance described within the pair of filings, which seem like beneath evaluation at present, cowl “Downloadable software program for processing music and audio recordsdata; Downloadable music-composition software program; Downloadable software program for creating, recording, and modifying audio to create music and audio; downloadable software program for creating music and audio; Downloadable cellular software software program enabling customers to share music and audio they created for different customers to view, entry, and remix inside the software; Downloadable software program to be used in music and audio manufacturing“.
Each emblems for various shade variations of the emblem seem to have been filed in April and Might 2023 and have been lively as of January this 12 months.
Lemon Inc. additionally filed a trademark software for ‘Ripple‘ in February 2023 and the mark was solely simply printed on the USPO’s Trademark Official Gazette final month.
Ripple is the AI-powered music-making app launched by ByteDance in June final 12 months that may flip a hummed melody right into a music.
Curiously, Lemon Inc additionally filed a trademark software again in 2021 for SoundOn, which is the DIY music distribution service launched by TikTok in March 2022.
SoundOn lets artists add their music on to TikTok and Resso, and it might additionally distribute artists’ music to platforms like Spotify and Apple Music.
And simply in case you’re nonetheless not satisfied about Lemon Inc’s affiliation with TikTok mother or father ByteDance, Lemon Inc. utilized for a trademark for “ByteDance” in January.
MBW’s reporting on ByteDance’s and TikTok’s AI-related music actions in the US arrives towards a backdrop of two huge ongoing tales involving the corporate and its subsidiary TikTok.
The primary story is a looming TikTok ban in the US, as a result of potential compelled sale of its US operations through a invoice handed by way of the Home of Representatives final week, which is now one massive step nearer to changing into regulation.
The opposite story that we’ve been following intently is TikTok’s falling out with Common Music Group, which refuses to resume its licensing take care of TikTok for various causes, together with the app’s compensation for artists and songwriters, and UMG’s issues about TikTok’s use of AI.
On March 1, Common Music Publishing’s catalog of ~4 million songs turned unlicensed to be used on TikTok, becoming a member of UMG’s portfolio of ~3 million recordings, whose license on TikTok expired (thus far with out renewal) on February 1.
In a assertion issued to UMPG’s songwriters on February 29, the corporate turned a lot of its consideration to the function AI-generated audio is enjoying on TikTok.
UMPG claimed that, thus far, TikTok has not offered Common with any assurances that the platform gained’t prepare its AI fashions on the music firm’s songs.
MBW has been writing for a while now concerning the hypothetical potential for TikTok and different providers to fill their catalogs with music generated by AI, which might, within the course of, dilute the market share of conventional rightsholders.
In February final 12 months, we printed an ‘MBW Reacts’ article asking if TikTok may pull off a “heist” on the music business on this regard, following its aggressive funding in generative AI expertise.
The “heist” we had been referring to: Utilizing licensed music as a cornerstone within the rise of TikTok to nicely over a billion customers globally, earlier than utilizing first-party, AI-created songs to crowd out music owned by conventional music rightsholders on the platform.
We wrote: “With music enjoying such a key function in TikTok’s rise, if main label content material does disappear from the platform – and the hole is someway efficiently crammed by indie and AI-driven creations – TikTok could possibly be mentioned to have pulled off one of many greatest heists in music enterprise historical past. A bait and change for a billion customers.”Music Enterprise Worldwide
[ad_2]
