Troubling Gluten Testing Data Released by Nima: But Hold the Phone

Troubling Gluten Testing Data Released by Nima: But Hold the Phone

Yesterday the Nima Sensor team released testing data that has caused me a great deal of concern. They are reporting that 24% of packaged foods tested by consumers and uploaded to the Nima app have tested gluten detected. They are also reporting in a new graphic that it is very rare for the Nima to test positive for gluten at levels below 20 parts per million.

BUT this is not the complete story. The Nima is NOT a fully validated ELISA. It is a lateral flow device and there are limitations to these types of testing devices. Of very real concern are false positive results. According to Thomas Grace of Bia Diagnostics, “These methods should never be used for finished product validation in and of themselves, but only in conjunction with a fully certified and validated method such as an ELISA…Something as “innocent” as a pinch of salt or a little vinegar can be all it takes for an LFD to produce a false negative or false positive result.” For more information on LFDs and my interview with Thom and Adrian Rogers from Romer Labs see https://www.glutenfreewatchdog.org/news/lateral-flow-devices-nima-gluten-tox-home-ez-gluten-to-test-food-for-gluten-manufacturer-consumer-use/

Back in January after seeing some early testing data from Nima I emailed the Nima team writing, “If this was my data and I was using a “new” device with a “new” antibody I would wonder why my data was so different from other published and unpublished data.” I suggested to the Nima team that if the percentage of packaged food products labeled gluten-free and testing at/above 20 parts per million of gluten is substantially higher than percentages published in the peer-reviewed scientific literature using a fully validated assay then they should consider the possibility of false positives.

So what is the testing data using the R5 ELISA—an assay scientifically validated by the Prolamin Work Group of Codex via a multi-laboratory ring trial—telling us?

  • A study by the FDA published in the journal Food Chemistry (Food Chemistry 169 (2015) 120–126) tested 275 foods labeled gluten-free. 3 samples (1.1%) of tested foods had gluten levels greater than 20 ppm of gluten.
  • A study by Gluten Free Watchdog published in the journal European Journal of Clinical Nutrition tested 158 products labeled gluten-free. 8 samples (5.1 %) of tested foods had gluten levels greater than or equal to 20 ppm of gluten.
  • According to Bia Diagnostics, LLC they have tested over 14,000 foods labeled gluten-free during the past 2 years. Less than 5% are testing at/above 5 ppm of gluten. Data on the percentage of products testing at/above 20 ppm of gluten is not readily available.
  • Note: A third US study was published in the Journal of Food Protection. While 16 (20.5%) of products labeled gluten-free tested at levels at or above 20 ppm of gluten, the study was flawed. Gluten Free Watchdog communicated at length with the study authors. During the course of our discussions it was determined that the researchers inadvertently used the wrong assay to assess foods for gluten. They meant to use the sandwich R5 ELISA but used the competitive R5 ELISA instead. A competitive ELISA is used to assess foods and ingredients that are fermented or hydrolyzed. The study write-up wrongly states that the sandwich R5 ELISA was used.

Findings simplified…

Nima: 24% testing “gluten detected”

FDA: 1.1% testing at/above 20 ppm

Gluten Free Watchdog: 5.1% testing at/above 20 ppm

Bia Diagnostics: Less than 5% testing at/above 5 ppm

“One of these things is not like the other…”

The bottom line and what I tweeted yesterday to Nima, “If 24% of packaged labeled gluten-free foods are testing “gluten found” & it is “rare” for gluten to be detected below 20 ppm then you should strongly consider false positives. Data to date from peer-reviewed studies published in scientific literature by FDA & Gluten Free Watchdog finding between 1- 5% labeled gluten-free foods testing at/above 20 ppm.”

Gluten Free Watchdog is currently in the process of assessing the Nima with Bia Diagnostics. For information on the strengths and weaknesses of Nima please see the videos and articles posted at https://www.glutenfreewatchdog.org/news/category/nima-sensor/

A couple of additional comments based on comments we’ve received after this post was published…

  1. Some of you have wondered about bias. Might consumers test products that they are concerned about or that caused a reaction. This is possible and would be referred to as sampling bias. At the very least the Nima team should put the results into context just as they would have to in a scientific paper. That said, at Gluten Free Watchdog almost all products tested are subscriber requests. It is likely that our results are potentially biased as well BUT we are still seeing “only” 5% of products testing at/above 20 ppm of gluten.
  2. A false positive result refers to more than just matrix effects (i.e., the composition of the food). The Nima website provides instructions for testing various food products that may or may not be followed by consumers. These directions are often not exact (e.g., pea size amount of food, small amount of food, dilute with water–in other words no measurements are provided). As we reported yesterday for the product that tested low gluten using the Nima but less than 5 ppm using the R5 ELISA: the product contains vinegar and according to the Nima website pure vinegar can not be tested because the acidity of the vinegar damages the test; it is brightly colored and the Nima website recommends diluting brightly colored food with water before testing; and it contains tannins/polyphenols (when testing with the fully validated R5 ELISA the recommendation is to add skim milk powder to the extraction solution to prevent false positives and false negatives).

Share this post

Comments (20)

  • Al Reply

    Yeow! Just the other day, a doctor-type TV show used Nima to test a number of gluten-free menu items from 10 different well-known restaurants. Nima’s results: only one meal reported “No gluten”.

    February 23, 2017 at 1:42 pm
    • Tricia Thompson Reply

      Yup. This is a very big problem and the Nima team owes it to this community to take a step back and figure out what is going on with this device. Just because a device can report a result does not mean the result is accurate.

      February 23, 2017 at 1:45 pm
    • Tricia Thompson Reply

      I also should add that testing data may be different for restaurants as compared to packaged foods. We do not have restaurant testing data using the R5. I am thinking of traveling to the Burlington Vermont area (where Bia Diagnostics is located) and doing some restaurant testing using the Nima and the R5 ELISA.

      February 23, 2017 at 2:13 pm
      • Al Reply

        That’d be great. If you are looking for funding for those tests I’d be willing to contribute.

        February 23, 2017 at 2:45 pm
        • Tricia Thompson Reply

          Hugs to you, Al for always, always being so supportive.

          February 23, 2017 at 3:04 pm
      • Bonnie Reply

        Hi Tricia, if you do testing on restaurant foods, will the process be the same: taking 200g of product, grinding it (to homogenize) and then taking 1g sample to test?

        February 28, 2017 at 7:55 pm
        • Tricia Thompson Reply

          The protocol you mention is for oats because gluten contamination is so spotty and difficult to homogenize. I’m not sure quite how we will test restaurant foods–whether we will mix the whole plate together to homogenize or instead test samples of each individual food on the plate. Maybe we will do both. This is something I will have to give some thought to.

          February 28, 2017 at 8:02 pm
  • Kathleen Warthen Reply

    Two point:
    1) Better false positives than false negatives, right? But more importantly,
    2) is it possible this Nima data is skewed because it comes not from people testing a random sample of GF products, but instead using the test at home on products for which they have a high suspicion of gluten contamination? If I had a Nima I’d probably use it first to test something I think might have made me sick.

    February 23, 2017 at 2:33 pm
  • Erika Reply

    For me, I don’t see false positives as better than false negatives in any way. Already in various GF groups I’m in, folks are buying this device, testing items, and then alerting the group to the fact that the product is now verboten, that the manufacturers have somehow lied, and that this device is the new god of the gf world. Casting a critical eye to this device and the Nima manufacturer’s getting a handle on this quickly would be preferable. I remain skeptical that this will happen, though, and will instead continue to encourage my friends in the gf community to subscribe to GF Watchdog! Thank you, Tricia, for all you do for our community!

    February 23, 2017 at 2:50 pm
    • Tricia Thompson Reply

      Hugs to you, Erika for posting this comment!

      February 23, 2017 at 2:59 pm
    • Al Reply

      I see the same type of activity in the groups that I frequent. Owners of the device won’t listen to reason or current known facts about LFD testing/Nima device (graciously presented by GFW!) Scary.

      February 23, 2017 at 3:48 pm
    • Kathleen Reply

      Your point raises another issue: could Nima face potential legal liability, if false positives are being spread as gospel? It could have a real impact on a food company’s sales if people think there is gluten in a product when there isn’t. I’d be pretty upset if I were a small GF food company being, essentially, libeled.

      February 23, 2017 at 4:21 pm
      • Tricia Thompson Reply

        Jumping in here. You raise an important issue. At Gluten Free Watchdog when we receive test results at/above 20 ppm of gluten there are checks and balances in place to make sure the data is accurate. Additional samples are typically tested, matrix effects are considered, the manufacturer is contacted before results are posted and invited to provide a comment for posting, etc.

        February 23, 2017 at 5:33 pm
  • Sasha Reply

    I was so excited for the Nima, but am now somewhat fearful of what it will do to our quality of life. A friend used it at a restaurant and the Udi’s bun came out as low gluten. They left and went home. Could have been contaminated, or maybe not. And perhaps not enough to cause issues. It troubles me that “low gluten” has a range of 20ppm all the way to 15,000 ppm. Something that is 20 ppm, or even 50, is unlikely to make a celiac sick, whereas 15,000 is. I worry this will lead to us not being able to eat out ever! I communicated this concern to Nima, and was told its up to my discretion and this is just one tool. But if it’s not a precise tool, it could do more harm than good. Not sure what to do in the future. Test and better safe than sorry (but eliminate most restaurants) or not test and risk getting sick.

    February 23, 2017 at 2:55 pm
    • Tricia Thompson Reply

      Also, keep in mind that a low gluten reading may mean a result below 20 ppm (this is per the Nima team). At this point I can only comment on the Nima as it relates to packaged foods as there is no peer-reviewed published data on restaurant foods in the US using a fully validated assay such as the R5. As mentioned I hope to do some of this research this summer.

      February 23, 2017 at 3:04 pm
  • Daniel Reply

    Thank you for this research. How odd it is that you seem to have to do the research to validate Nima rather than the Nima manufacturers themselves. Although I was one of the early consumers who pre-purchased Nima and I was initially very excited, I’m now extremely skeptical. Even more, I’m angry that Nima made it to market without adequate evidence that it is accurate. I’m going to hold off on using it for now. I foresee a class action law suit.

    February 25, 2017 at 3:59 am
  • Charlie Reply

    I have celiac and I am incredibly discerning in what I eat. I do my due diligence and when eating out I use the Nima. When I’ve gotten positives I’ve chosen not to eat that dish. . At best, I’ve saved myself from being glutened. At worst … well I don’t see any downside. I’m grateful to have this tool.

    February 27, 2017 at 1:26 am
  • Lilly Reply

    Thank you so much for this article. I have tried to use the Nima but don’t trust the results at all. Your article confirms all of my doubts. It seems that all of their claims are totally bogus. Great idea for a product, but their technology does not work. I tested Tabouli which came back gluten-free. As did a granola my friend made with wheat flour. I have scoured their website, and hidden on the second page of Testing with Nima it states “No Gluten – No gluten was detected, the sample you tested likely has less than 20ppm of gluten.” Before I received my Nima, they claimed 99% accuracy, but now they don’t state actual accuracy anywhere. “Likely” is not very reassuring. The website also says “Nima will not distinguish whether the gluten level detected is below or above 20ppm.” So what do they distinguish? Also, the box the Nima came in said results in 2 minutes, the inside flyer said 3 minutes, but everything I have tested has taken at least 4 minutes. I am so frustrated with them. I am going to throw away my Nima and sign up for your newsletter. Thanks for being so informative.

    March 6, 2017 at 4:57 pm
    • Tricia Thompson Reply

      Lilly, You seriously tested tabouli (contains bulgur wheat) and granola containing wheat flour and both times the result was a “smiley face?” If possible, can you please retest both products and send me photos of the Nima result. Have you contacted the Nima team about these results?

      March 7, 2017 at 5:05 pm

Leave a Reply

Your email address will not be published. Required fields are marked *