Some time ago I received the following email about this project.

---
Mr. van Rijn,

I am Darren Briggs, the Chief Technical Officer of Landmark Digital Services, LLC. Landmark Digital Services owns the patents that cover the algorithm used as the basis for your recently posted “Creating Shazam In Java”. While it is not Landmark’s intention to alienate those in the Open Source and Music Information Retrieval community, Landmark must request that you do not ship, deploy or post the code presented in your post. Landmark also requests that in the future you do not ship, deploy or post any portions or versions of this code in its current state or in any modified state.

We hope you understand our position and that we would be legally remiss not to make this request. We appreciate your immediate attention and response.

Best regards,

Darren P. Briggs
Vice President &
Chief Technical Officer
Landmark Digital Services, LLC
---

This scared me a bit, why are they emailing me? I’ve written some code (100% my own) and implemented my own methods for matching music. There are some key differences with the algorithm Shazam uses.

The code isn’t published yet, but I was planning on releasing it under Apache License to the open source community soon.

It was never my intention to release this commercially, I’m just a programmer who likes to work on technical, mathematical algorithms in his spare time. And if enough people ask for the source code, I’d be happy to give it to them. Who would have thought that creating something at home in a weekend could result in a possible patent infringement!?

Just to be sure I asked them confirmation that the email was indeed sent from their company. And second, I’d like to know which patents are in play. Because I just couldn’t think that something this easy (music-fingerprint is a hash, and we do a lookup) can be patented.. Maybe in the States, but in Europe?

I got the following reply from Landmark Digital Services LLC:

---
Mr. van Rijn,

I can confirm that the email you received came from me on behalf of Landmark Digital Services, LLC. If you require a more formal notification, the Landmark legal department can provide you with a legal notification. Please let me know if that would be preferable.

The Landmark patents have been granted around the world, including the US, the EU, individual EU-member countries.

Examples of some of the Landmark patents include:
- System and Methods for Recognizing Sound and Music Signals in High Noise and Distortion
- Robust and Invariant Audio Pattern Matching

We appreciate your compliance with our request.

Best regards,

Darren P. Briggs
Vice President &
Chief Technical Officer
Landmark Digital Services, LLC
---

They are really serious about this. But there is a problem, I still don’t have patent numbers of European patents, so there is no way for me to check the validity of their claims. Once again I asked them for specific patent numbers. And got the following reply:


---
Mr. van Rijn,

The US patent numbers for the two examples I provided you are 6,990,453 and 7,627,477. Note that there are additional issued patents and pending patent applications in the US and Eu that cover these concepts as well.

Best regards,

Darren
P. Briggs
Vice President &
Chief Technical Officer
Landmark Digital Services, LLC
---

Sigh, again two U.S. patent numbers. But well, lets take a step back. Why does Landmark Digital Services think they hold a patent for the concepts used in my code? Even if my code works pretty different from the Shazam code (from which the patents came).

What they describe in the patent is a system which:
1. Make a series of fingerprints of a media file and/or media sample
     (such as audio, but could also be text, video, multimedia, etc)
2. Have a database/hashtable of fingerprints as lookup
3. Compare the set of hashtable hits using their moment in time it happened

This is very vague, basically the only innovative idea is matching the found fingerprints linearly in time. Because the first two steps describe how a hashtable works and creating a hash works. These concepts are not new nor innovative.

But, with a bit of imagination one could (possibly) argue that my code (again, written completely by myself in a weekend with some spare time) does the same thing as the patent describes.

Just to be sure I asked around for advice, including help from the FSF (Free Software Foundation) and the EFF (Electronic Frontier Foundation). They forwarded my questions to Bits of Freedom a Dutch organisation for digital rights.

After a good conversation with Ot van Daalen (from Bits Of Freedom) he suggested I contact Arnoud Engelfriet, a Dutch ICT lawyer and patent attorney with a lot of knowledge about software patents.

In the last couple of days I’ve had quite a few conversations with Arnoud, and he helped me with a lot of my questions.

Here are some of the conclusions:

  • Software companies can make your life very miserable if you don’t comply;
  • Using the Java-Music Match code commercially will likely result in a lawsuit for patent infringement;
  • Releasing the code under an Open Source license on a non-profit website (no ads) is a grey area;
  • Writing/using this code privately can’t be patent infringement in the Netherlands;

And even the Arnoud even mailed me:

---
Als je de software laat staan, loop je de kans dat Landmark Digital Services je een proces aandoet. En zoals gezegd kan dat een fiks bedrag worden.

Translation:
If you leave the software on your website, you run the risk that Landmark Digital Services files a patent infringement lawsuit. And like I told you, this could result in a substantial amount of money.
---

Since I don’t want to end up like Dmitry Sklyarov, with the possibility of a lawsuit, I’m not going to publish the code anymore… Grey area’s with lawsuits roaming around are better to be avoided. Especially if you think about the average cost of a patent lawsuit being 1 to 3 million dollars.

In the latest email I received from Landmark Digital Services they are even asking for more:

---
Mr. Van Rijn,

The two example patent numbers that I sent you are U.S. patents, but each of these patents has also been filed as patent applications in the Netherlands. Also, as I'm sure you are aware, your blogpost may be viewed internationally. As a result, you may contribute to someone infringing our patents in any part of the world.

While we trust your good intentions, yes, we would like you to refrain from releasing the code at all and to remove the blogpost explaining the algorithm.

Thank you for your understanding.

Best regards,

Darren
P. Briggs
Vice President &
Chief Technical Officer
Landmark Digital Services, LLC
---

They are still unable to direct me to the correct Dutch patent numbers. But more shocking, they are now telling me that my blogpost may contribute internationally to patent infringement. But… doesn’t the patent itself describe their algorithm in much more detail? The idea of patents are that the world knows about technology and how it can be used, but they can’t legally commercially exploit it? Now next to asking me not to release the code, they are also asking me to remove the previous blogpost!

This seems like a very unjust threat to me, and for now I’m going to ignore that request. If they decide to file a formal legal complaint I might reconsider taking down the blogpost. The only action I’ll take right now is not releasing the source code.

Other implementations

I’ve also had contact with other people who have implemented this kind of algorithms. Most notible is Dan Ellis. His implementation can be found here: http://labrosa.ee.columbia.edu/~dpwe/resources/matlab/fingerprint/

He hasn’t been contacted (yet?), but he isn’t planning on taking his MatLab implementation down anyway and has agreed for me to place the link here. This raises another interesting question, why are they targetting me, somebody who hasn’t even published the code yet, and not the already published implementation of Dan?!

And if they think its illegal to explain the algorithm, why aren’t they going after this guy? http://laplacian.wordpress.com/2009/01/10/how-shazam-works/

This is where I got the idea to implement the algorithm and it is mentioned in my own first post about the Java Shazam.

Any advice?

So, has anybody else had these kind of experiences? What would you do in this situation?

Maybe I’m going to need it, maybe I’ll just buy a beer, every penny is welcome:

Next:

The patent infrigement story continues here…

ps. I’m sorry John Metcalf: You can stop printing “Free Roy van Rijn” t-shirts…

  • Jorde

    That’s rather strange, considering the EPC (European Patent Convention) excludes Software Patents. (Programs for Computers)

    See article 52: http://www.epo.org/patents/law/legal-texts/html/epc/2000/e/ar52.html

    Apart from that you’re not doing anything commercially and the code is 100% your own. They have absolutely nothing on you.

    Maybe you should change the title of your post, I think that’s probably what got them to target you in the first place. Chances are they didn’t even bother reading your post and just sent that out based on just that.

  • jbs

    No advice, but a couple of observations.

    1. These are, obviously, examples of “process patents” where it is not a device but rather a process or algorithm that is being claimed. Thus, they can patent the underlying concept which could apply to your quite independent later implementation which happens to conform to the process they describe.

    2. Often these patents are less “inventions” than attempts to stake claims to a particular area of technology. They are often done by lawyers rather than engineers, computer scientists, etc. That does not seem to be the case here. From what I can tell, Wang is/was part of the Shazam group and Smith is a Professor of Music at Stanford where Wang got his PhD. So, it looks like they may have legitimate claims to have been involved in the actual invention of the Shazam system/concept.

    3. Landmark Digital Services seems to a small company (they list only three employees on their Web site) located in Nashville that has acquired the Shazam portfolio of patents with the apparent intent of exploiting them for commercial gain. To protect their patents, they would seem to have no choice but to challenge someone who publishes implementation details of concepts included in their patents. Whether or not they have the resources to backup their requests/threats is another matter. As someone has pointed out, patent suits are incredibly expensive, and they may or may not have the resources to sue, but, on the other hand, an individual would have to have deep pockets to defend against such a suit.

    4. As to originality, I agree with your assessment that most of the individual steps in the process you describe and that apparently underlies the Shazam system seem to be based on well-known concepts and algorithms — e.g., hashing. But they have put them together in a novel way that has produced a total system that most people agree is pretty amazing. I think I’m right is saying that the combination of steps would qualify the invention as to originality. As to the vagueness/generality of the patents, they are often written this way to provide maximum coverage while being specific enough to satisfy the examiners.

  • http://dismaldenizen.wordpress.com Aiden

    Unless I’m missing something, it appears as if Shazam and the code in your post are essentially just (different) implementations of acoustic fingerprinting. Such tools have been around for quite a while, and I’m sure that prior art for “Robust and Invariant Audio Pattern Matching” would not be difficult to find (http://en.wikipedia.org/wiki/Acoustic_fingerprint lists a few projects which achieve similar goals). In particular I have been using MusicIP’s Open Fingerprint Architecture in MusicBrainz Picard for a long time.

    I seriously doubt whether Landmark Digital Services could back their claims in court, it seems as if they are just trying to bully you and hope that you fold.

    Don’t give in!

  • Pingback: Creating Shazam in Java - Redcode()

  • http://www.nicholassims.com Nicholas

    1. Make a series of fingerprints of a media file and/or media sample
    (such as audio, but could also be text, video, multimedia, etc)
    2. Have a database/hashtable of fingerprints as lookup
    3. Compare the set of hashtable hits using their moment in time it happened

    Surly the above has already been done by the likes of Nuance (dragon easily speaking) and the like.

  • ftc

    This sort of technology looks very similar to a signal processing technique called an “Energy Detector” with some hashing added on top that is used widely in more than just audio processing. I am not a lawyer by any means but it seems that if you just called it a “Signal Matcher” instead of specifying the purpose of matching audio then you would be in the clear. (Please get legal advice before taking my word for it).

    I wish the best of luck to you and will add Shazam to my ‘List of patent trolls to boycott’ list.

  • Batman

    I would not take their threats too seriously. As you point out yourself, patents protects ideas from being used in commercially competing products after a company has spent tremendous efforts to come up with the idea in the first place. Its not a mechanism designed to hide how the idea is executed. I think their legal threats are empty and manipulative, seeking to eliminate competition which they are in fact NOT shielded against with their limited patent. A fine example of how the patent system is not to be used. I would personally just have ignored them and published the source code just to show them what I think of this kind of abuse.

  • Tom

    Their letter is a bog-standard letter which is just an empty threat. Anti-competition. This most probably was at the whim of Shazam. The lack of details in their letter shouts the obvious emptiness of their threats, and is simply to intimidate you. (Android is an implementation of java… hence their fear!)

    Totally ignore it. The lack of compliance with your requests is another clue to their underhand manipulation. If there really was a problem, you would have been served.

    Even without the references to Shazam, your post outlines a simple music recognition system. Something anyone with basic programming experience could manage.

    Something to line the waste paper bin I think.

  • http://robnonstop.com Robert

    Hi, I find your case very interesting, especially because I want to use similar technology for a very different purpose in an iPhone app.

    I asked the Electronic Frontier Foundation (http://eff.org) and they replied:

    … Thank you for contacting the Electronic Frontier Foundation (EFF). We appreciate you sharing this information with us. We are already aware of this news, but as we rely on our members and supporters to help us stay abreast of developments in our areas of interest, we’re very grateful that you spoke up to make sure.

    As for offering legal assistance, we always consider seriously any requests for legal help, and would do so if this coder came to us.

    It would be great to hear more about this case, maybe you should contact them.

  • Nicolas Franco

    Hi, I really don’t understand all this mess… Based on the well known paper of Avery, and helped by the Dan Ellis code, I also wrote a MatLab/C# algorithm for finding pieces of music on a database. I never publish it because i found it very obvious and easy to do. And now it happens that I would be out of law!!!
    World is becoming crazy or people are bored to work and find new staff so they want to live they whole life from a single idea they got???
    At least in Spain I didn’t found any single patent related with the Shazam algorithm, so I don’t believe they can suit anybody in Spain/Europe.
    Good luck and good job

  • Anticom

    I couldn’t find any patents in Germany either.
    And I’m not sure, wether I’m getting this totally right, but as a short google search revealed, there are plenty of other web services that are in production already and e.g. midomi’s rank at alexa isn’t that bad.
    But since they’re not offering any source code but offering the music search service itself, I’m curious wether that would be patent infringement already.

    Hope that gave you some ideas and that it might have helped. Please don’t give up and keep fighting :)

    Best wishes,
    Timo

  • datenwolf

    I just got here via some ELI5 on Reddit, but just for reference have to get this out of my system:

    If it’s patented, then this thing has been published by Shazam themself. That’s the whole point of a patent, publishing something for the rest of the world to see. In exchange for the publication you receive a monopoly on the thing you patented (for some limited amount of time). Everybody can go to the EPOs or USPOs website, get the patent document from there and reimplement the thing (if you can’t reimplement it, or it describes something trivial, the publication may not hold as a patent).

    The other thing is: Copyright doesn’t work on patents. Copyright works on specific work of literature. For technical designs you have a similar protection, though it’s not a copyright per se (things differ in various legislations, for example in Germany you’ve to register a “Geschmacksmuster” if you want to protect for example some lamp design). But your code is your very own creation. In fact you as it’s author hold its copyright. Shazam can’t claim a single line of code as their copyrighted material. Patented maybe. But not copyrighted.

    Last but not least: In the EU you can file software patents, but they can not be applied here. It’s basically just something you file here, so that you can patent your stuff in the US as well and make some money there.

    So if I’d receive such a stupid letter, my reaction would be, to shove it down the sender’s throat with a explanation how the copyright and patent system works, and that they apparently should go back to law school, because they definitely didn’t pay attention when this came up.

  • Pingback: 用java实现Shazam 译文 - 软件开发联盟()

  • DAWOOD AKRAM

    they tell you they have PENDING patents – they dont matter as they are pending and havent been passed yet – also i think they are trying to bully you,they wont go straight to court. having said that, do you want to take the risk?

  • anony

    sue them back for fraud and harrasment.

  • tko

    They want to privatise just about everything – FUCKING BULLSHIT! This is simple applied math/physics. I am with you 100%

  • _

    Fuck patents.

  • Nick

    Their claims of patent infringement if you ship such code seem legitimate to me, but you must understand that if they can also scare you into taking down your implementation (even if they can’t legally) they will. The only thing you have to be afraid of is if they do take you to court over it, even if you do win it’s still time and money that you had to initially spend. In other words they can make your life difficult.

    If it were me I’d politely tell them I won’t ship it, but I wrote the code and it’s staying up on my blog, although I like money and you can persuade me to take it down for $10,000.

  • gabriel

    Omg. Isn’t a patent a published method of what you are patenting?

    This guy is a retarded. Hopes this serve to him never getting a job anywhere.

    “plz do not publish a thing that may or may not -since I never saw it- because it may be already published in a patent”

    Patent is not trade secret. Fuck, and some company pays that guy!

  • Fadzlan

    Since their patents cover the steps to do the process very closely, what if someone walks throughout the patents and explain line by line what everything in the patent means?

    Now, that would be close to code no? If somebody patent a bike, the patent have to provide how would a bike would be made. If someone explains in a blog post about the method, as long as they do not ship a product infringing the patent, then all is good right?

    Patent is not trade secrets. Its meant to be open to public for them NOT to copy into a different product. Explaining them should not be an infringement. Of course IANAL.

  • http://www.jooq.org Lukas Eder

    Incredible! Shazam a patent troll? But they got sued themselves for infringing very similar patents related to media content discovery:

    http://www.businessinsider.com/shazam-sued-for-patent-infringement-2009-11

    When will patent trolling finally end?

  • floyd raymer

    Here’s a great episode of This American Life on the subject. Infuriating to see how patent trolls work.

  • Aziz

    Does the echoprint open source project infringe on Shazam’s patents as well?

    http://echoprint.me/codegen

  • Felix

    This is just sad sad sad. As a researcher interested in solving problems it shocks me how moneymakers stop us helping each other. Its a shame, really.

  • Stefan

    It’s really sad that the patent system has deteriorated to the point where trivial stuff can be patented. I don’t own a patent, nor have I ever submitted one or even remotely believed that anything that I have ever coded/created could be the topic of a patent. However, if I were to hold myself to the low levels of a typical software patent, then I could literally have submitted thousands of patents.

    The whole patent system is ripe for killing/substantial reform.

  • marco

    It looks like they see you as a potential issue. If they really think so, probably they also think you are talented. did you ever think about the possibility to shut down the issue by proposing them a job opportunity?

    A old motto says that you can make alliances with those you cant beat

  • Robert

    While I do agree on the fact that patents can help when someone blatantly copies a code, piece of artwork or whatever this is ridiculous. You wrote this code yourself. On the topic of patents, they slow down innovation. I think Marco has a point that you are proven to be a threat to them (because of talent naturally), you could bend it into your favor. Crazy world.