The Data Problem Nobody Warned You About: How Feed Normalisation Failures Are Quietly Destroying Your Diamond Store’s Schema, Rankings, and Agentic Visibility
Author: Erwee Coetzee | Diamond Stack / SEO Gurus Series: Diamond Feed Integration Reading Time: ~24 minutes Published: March 2026
There is a particular kind of technical problem that is almost impossible to diagnose from the outside. It does not generate error messages. It does not produce a visible failure that a developer can point to and say: there, that is broken, fix that. It produces instead a pattern of underperformance that accumulates gradually, that manifests as a persistent gap between expected results and actual results, and that most practitioners — and most store owners — attribute to the wrong causes.
They spend more on content. The gap does not close. They acquire more backlinks. The gap does not close. They upgrade the hosting, optimise the images, improve the page speed. The gap does not close. The gap does not close because the problem is not in any of the places they are looking. The problem is in the data layer that sits beneath all of it — a silent, unglamorous, technically specific failure that is telling search engines and AI systems something different about every diamond on the site than the store owner believes it is telling them.
I am talking about diamond data normalisation. Or more specifically, the failure of it.
I want to begin with a concrete scenario, because the abstract description of this problem does not communicate its practical consequences half as well as a specific example does.
The Feed That Looked Fine
A diamond retailer in Johannesburg contacted me fourteen months after launching their live feed integration. The integration had been built by a competent WooCommerce developer who had done everything the standard integration guide recommended. The Nivoda connection was live. The stones were syncing every four hours. The product pages were populating with diamond data, 360-degree video where available, certificate information, and pricing. The developer had signed off. The store owner had signed off. The integration looked, by every visible measure, complete.
Fourteen months later, the store’s Shopping feed had a 71% product approval rate in Google Merchant Centre. Their diamond product pages generated almost no organic specification query traffic despite indexation. Their AI-mediated search visibility — the query types I described in the previous article on agentic routing — was minimal. Three separate SEO consultants over that fourteen-month period had attributed the underperformance to their domain authority, their content strategy, and their link profile. None of them had looked at the data.
When I audited the feed, here is what I found.
Cut grades were represented in four different formats across the product catalogue. Stones from Nivoda’s main feed used “Excellent.” Stones imported from a secondary RapNet source used “EX.” A batch of manually entered inventory from the store’s own physical stock used “Triple Excellent” for stones with matching cut, polish, and symmetry grades. And a handful of older records, imported from a legacy system during a data migration eighteen months earlier, used a numeric scale — “1” for the top cut grade.
Certificate laboratory names appeared in six variants. “GIA.” “G.I.A.” “Gemological Institute of America.” “GIA Certified.” “GIA – Gemological Institute of America.” And, on a small number of records from one specific data source, “GIA Labs.” All of them representing the same issuing authority. All of them different strings to any system that reads them as data rather than understanding them as human language.
Carat weights were stored at inconsistent precision. Some records showed 1.2. Some showed 1.20. Some showed 1.200. The feed delivered them in different formats depending on the source, and the integration had stored them as delivered without normalisation.
Fluorescence was blank on 34% of stones. Not “None.” Not “No fluorescence.” Blank. The field did not exist in the database record for those stones. When the Product schema was generated for those product pages, the fluorescence property was absent entirely — not because those stones had no fluorescence, but because nobody had handled the case where the feed delivered no fluorescence value.
Origin was absent on 78% of stones. Country of origin, conflict-free certification status, and — for the 23% of the catalogue that was lab-grown — the laboratory-grown declaration required by FTC guidelines, were all either absent or inconsistently present.
Every one of these inconsistencies was invisible to the developer who built the integration. The site looked correct. The data appeared on the product pages. From a human-readable perspective, everything was working. From a structured data perspective, the site was generating schema that was internally inconsistent, factually incomplete, and in some cases actively contradictory — and it had been doing so for fourteen months.
Why Data Inconsistency Damages Rankings in Ways Standard Audits Miss
I want to explain the specific mechanisms by which the failures I have described translate into ranking and visibility underperformance, because understanding the mechanism is the only way to make a convincing case for fixing something that is invisible to the human eye.
The first mechanism is content-schema mismatch. Google’s structured data evaluation process cross-references the values in a page’s JSON-LD schema against the values that appear in the page’s visible content. A diamond product page that shows “1.20ct” in the product title and product description, but whose schema has "weight": 1.2 — without the trailing zero — has a precision mismatch between the content and the schema. The values are mathematically equivalent but they are not the same string. Google’s content-schema reconciliation system flags this as a data quality inconsistency. The page’s product rich result eligibility is reduced.
Multiply this across 40,000 product pages with 20 attributes each and you begin to understand why a store with a fully functioning feed integration and 40,000 indexed product pages can generate minimal Shopping rich result appearances. The individual inconsistencies are small. Their aggregate effect on data quality scoring is not.
The second mechanism is entity vocabulary fragmentation. When Google’s AI systems evaluate a diamond described as “Excellent” cut on one page and “EX” cut on another page of the same site, they are not reading those as synonyms. They are processing them as different attribute values. The semantic distance between “Excellent” and “EX” in the vector space representation of diamond product data is not zero — it is a measurable gap that means an agent routing on “Excellent cut diamonds” does not match “EX” with the same confidence it matches “Excellent.”
For a store whose catalogue contains both “Excellent” and “EX” cut grades for structurally identical stones, this means that a query for “Excellent cut round diamond” will return high-confidence matches from the “Excellent” records and lower-confidence matches from the “EX” records — even though the physical stones are graded identically by GIA. The inconsistency in the data vocabulary is creating a visibility gap between records that should be equally visible.
The third mechanism is the one I described in the previous article but want to reinforce here: absent attributes as active vector distance penalties. A blank fluorescence field does not produce neutral search results for fluorescence-specified queries. It produces actively penalised results, because the AI system’s routing logic treats attribute absence as genuine uncertainty — and uncertainty reduces routing confidence in the same direction as a wrong answer, though less severely.
A store with 34% of its catalogue carrying blank fluorescence fields — as the Johannesburg retailer had — is producing reduced confidence results for every query that specifies fluorescence. Given that “no fluorescence” is one of the most common specification constraints in the research-stage diamond buyer’s query vocabulary, 34% blank coverage is a commercially significant visibility deficit on a very high-intent query category.
The fourth mechanism is the Google Merchant Centre data quality scoring system. Merchant Centre evaluates the quality and completeness of feed data before approving products for Shopping surfaces. A feed with inconsistent attribute vocabulary, missing required fields, and content-schema mismatches generates a low data quality score. Products with data quality issues are either disapproved entirely or given reduced impression share in Shopping placements. The 71% approval rate in the Johannesburg case was a direct consequence of the data normalisation failures — not of the product catalogue, not of the store’s policy compliance, not of any factor the standard troubleshooting guides address.
The wp_postmeta Architecture Problem That Makes This Worse
Before I describe the normalisation solution, I need to address the underlying database architecture problem that makes diamond data normalisation both more important and more technically involved than it would be for a standard e-commerce catalogue.
When WooCommerce stores product attribute data — cut grade, colour grade, clarity grade, fluorescence, and so on — it uses WordPress’s native wp_postmeta table. This table stores data as key-value pairs: one row per attribute per product. A product with 20 attributes generates 20 rows in wp_postmeta.
For a standard e-commerce product catalogue — hundreds or low thousands of products — this architecture performs adequately. The queries that read attribute data for filtering, for schema generation, and for product display are fast enough that the architecture’s limitations do not surface as a performance problem.
For a live diamond feed with 40,000 stones and 20-plus attributes each, the arithmetic becomes catastrophic. 40,000 stones multiplied by 20 attributes equals 800,000 rows in wp_postmeta. When a PHP process needs to read all the attribute data for a specific stone — to generate the Product schema for that stone’s PDP, for example — it is running a lookup against 800,000 rows using a key-value structure that requires a separate database query for each attribute.
The consequence is a Time to First Byte — the time between a browser or crawler requesting a page and the first byte of the response being delivered — that routinely exceeds two to three seconds on diamond product pages under these conditions. In a worst case, during high-traffic periods or when the scheduled feed sync is running simultaneously, TTFB on diamond PDPs can reach four to five seconds.
This matters for normalisation specifically because the Crawlable Shadow — the complete, server-side JSON-LD block that every diamond PDP needs to carry for agentic discoverability and Resilience Floor purposes — requires reading all attribute data at page render time. If that read takes three seconds, the page’s TTFB is three seconds before the HTML has even begun rendering. Googlebot’s crawler, waiting for a response to begin, may abandon the request entirely. The Crawlable Shadow is not built for that page. The stone is not in Google’s structured data index with complete attributes. The agentic routing potential of that page is zero.
The normalisation solution and the database architecture solution are therefore not separate problems. They are the same problem at different layers. Getting the data right and getting the data fast are both prerequisites for the Crawlable Shadow to function — and the Crawlable Shadow is the Resilience Floor that the CLP’s SC formula depends on. All three requirements must be met simultaneously.
The database architecture solution is a custom table — a dedicated MySQL table for diamond inventory data, bypassing wp_postmeta entirely. The table has a dedicated column for each diamond attribute, using appropriate MySQL data types: DECIMAL(5,2) for carat weight, VARCHAR(20) for cut grade, and so on. It is indexed on the attributes most frequently used in filter queries. It is the layer at which normalisation is enforced — every incoming feed record passes through the normalisation mapping before being written to the custom table, so what the table stores is always the canonical, consistent vocabulary regardless of what the feed delivered.
With data in a properly indexed custom table, the database query time for reading a complete diamond record drops from multi-second to under 50 milliseconds on modest hardware. TTFB on diamond PDPs drops to under 200 milliseconds. Googlebot completes every page request. The Crawlable Shadow is built. The Resilience Floor holds.
The Normalisation Mapping: What It Is and How to Build It
The normalisation mapping is a translation layer — a set of rules that converts every known variant of every attribute value into a single canonical form before storage. It is implemented as a function in the feed synchronisation code, applied to every record as it is received from the feed source.
The canonical vocabulary standard I recommend for South African diamond retailers is based on GIA grading terminology, because GIA is the primary certification authority recognised by South African buyers and because GIA’s vocabulary is the most semantically stable and widely understood across both human and machine contexts.
Let me work through the five most consequential normalisation mappings, because understanding these specifically is more useful than a general description of the concept.
Cut grade normalisation.
The canonical scale is the GIA five-grade vocabulary: Excellent, Very Good, Good, Fair, Poor. Every variant from every feed source maps to one of these five values. The specific mappings that appear most frequently in South African diamond retail feeds:
“EX” maps to “Excellent.” “VG” maps to “Very Good.” “GD” maps to “Good.” “Ideal” maps to “Excellent” — this is a marketing term used by some vendors for stones they consider top-quality Excellent cut, but it is not a GIA grade and should not be stored as a canonical attribute value; it can be preserved in a separate marketing notes field if needed. “Triple Excellent” is not a cut grade — it is a shorthand for a stone with Excellent cut, Excellent polish, and Excellent symmetry; normalise the cut field to “Excellent” and ensure the separate polish and symmetry fields are also populated correctly.
Numeric cut grade scales (1, 2, 3 mapped to top, middle, lower grades) should be mapped to the canonical vocabulary based on the scale documentation of the specific feed source that uses them. If the scale documentation is unavailable, treat the records as requiring manual review before publication rather than applying a potentially incorrect mapping.
Certificate laboratory name normalisation.
The canonical forms are the official abbreviations: GIA, IGI, GCAL, AGS, HRD, EGL. Every variant maps to the official abbreviation. “Gemological Institute of America” maps to “GIA.” “G.I.A.” maps to “GIA.” “GIA Certified” maps to “GIA” — “Certified” is a descriptor, not part of the laboratory name. “GIA Labs” maps to “GIA.” “International Gemological Institute” maps to “IGI.” “Gem Certification and Assurance Lab” maps to “GCAL.”
The normalisation function should log every variant it encounters that is not in the mapping table, so that new variants from new feed sources are identified and added to the mapping rather than passing through unnormalised. A variant that is not in the mapping table should be flagged for manual review rather than stored as delivered.
Carat weight precision normalisation.
The canonical format is DECIMAL(5,2) — two decimal places, always. A weight delivered as 1.2 is stored as 1.20. A weight delivered as 1.200 is stored as 1.20. A weight delivered as 1.2034 (some feeds deliver higher precision than certificate-grade accuracy) is rounded to 1.20. The displayed value on the product page should match the schema value exactly — both showing “1.20ct” or both showing “1.20 carats,” never one showing “1.20” and the other showing “1.2.”
Fluorescence normalisation.
This is the most consequential mapping because it addresses the blank-field problem directly. The fluorescence grades used by GIA are: None, Faint, Medium, Strong, Very Strong. Many feeds deliver these correctly for stones where fluorescence has been assessed. The normalisation challenge is the records where fluorescence is absent in the feed data.
The correct handling depends on the reason for absence. For GIA-graded natural diamonds, fluorescence is always assessed — if it is absent from the feed data, it means the feed has not delivered the data, not that the stone has no fluorescence. These records should be flagged for data enrichment — the actual fluorescence grade should be retrieved from the GIA online certificate verification and added to the record. Until enrichment is complete, the record should carry a value of “Unconfirmed” rather than blank, so that the schema reflects genuine uncertainty rather than implied absence.
For stones where the certificate explicitly shows no fluorescence (GIA terminology: “None”), the stored value must be the explicit string “None” — never blank, never null, never zero. “None” and null are different signals to every system that reads them as data.
For laboratory-grown diamonds where fluorescence grading is not included in the certificate, the stored value should be “Not Graded” — explicitly acknowledging that the assessment was not performed, which is different from both “None” (assessed, confirmed absent) and blank (assessment status unknown).
Laboratory-grown declaration normalisation.
Every stone in the catalogue must carry an explicit origin declaration. For natural diamonds: “Natural.” For laboratory-grown diamonds: “Laboratory-Grown” — this is the FTC-compliant term that should appear in both the on-page content and the schema. “Lab-grown,” “lab grown,” “synthetic,” “man-made,” and “cultured” are all variants that should be normalised to “Laboratory-Grown” for schema purposes, while the on-page marketing copy may use the variant preferred for the specific brand context.
The reason this normalisation matters beyond FTC compliance is the same reason all the other normalisations matter: an AI agent routing on a query that specifies “lab grown diamond” will not match with maximum confidence a product whose schema says “lab-grown diamond” (hyphenated) if the query vocabulary and the schema vocabulary do not align. Standardise to the FTC-compliant form and use it consistently everywhere.
Building the Custom Table: The Implementation Sequence
I want to describe the full implementation sequence for the custom database layer and normalisation mapping, because the order of operations matters and doing it in the wrong order creates complications that are difficult to unwind.
Step one: Schema design. Before writing a single line of implementation code, design the custom table with all required columns, correct data types, and appropriate indexes. The table structure is the foundation — changes to it after data has been loaded are costly. The essential columns are: a surrogate primary key, the feed source identifier, the feed’s own stone ID (for deduplication and sync reconciliation), and a dedicated column for every attribute that will be queried, filtered, or included in schema. Attributes used in filter queries must have indexes. Attributes not used in queries but needed for display or schema can be stored as VARCHAR without indexing.
Step two: Normalisation mapping documentation. Before implementing the mapping function, document every known variant for every attribute from every feed source the store uses. This documentation is the reference for the mapping function and should be maintained as a living document — updated every time a new variant is encountered. It is also the audit trail that proves the normalisation is deliberate and consistent rather than ad hoc.
Step three: Feed sync implementation. Build the feed synchronisation process to write to the custom table, applying the normalisation mapping at write time. The sync process should include a validation step that checks each normalised value against the allowed canonical vocabulary before storage, logging any value that does not pass validation for manual review.
Step four: Schema generation. Build the Crawlable Shadow schema generation as a server-side PHP function that reads from the custom table at page render time and outputs the complete JSON-LD block in the initial HTML response. The schema generation function should include null-handling for every optional field — a missing value should produce an explicit “None,” “Not Graded,” or “Unconfirmed” string as appropriate, never an empty schema property.
Step five: Feed decommission. Once the custom table is live and the schema generation is reading from it, the native WooCommerce product attributes (and their wp_postmeta storage) can be reduced to the minimum required for WooCommerce’s commercial functions — price, availability, and the product category hierarchy. The detailed diamond attribute data lives in the custom table, not in wp_postmeta.
Step six: Merchant Centre feed rebuild. The Merchant Centre product feed should be generated from the custom table, not from WooCommerce’s native product attributes. This ensures that the Shopping feed and the on-page schema are drawn from the same normalised source — eliminating the feed-schema mismatch that is one of the most common Ruggedised SEO failure modes I described in the previous article.
The Audit Checklist: Diagnosing Your Current State
If you are running a live diamond feed integration and you have not performed a data normalisation audit, here is the sequence I recommend for establishing your current state before deciding on the remediation scope.
Extract a sample of 100 product records from your current catalogue — a stratified sample that includes records from every feed source you use, records from different time periods, and both feed-integrated and manually entered products if you have both.
For each record, check: cut grade vocabulary (how many distinct values appear for what should be a five-point canonical scale?), certificate laboratory name (how many distinct strings represent the same laboratories?), carat weight precision (is it consistent across all records?), fluorescence field (what percentage are blank or null?), origin declaration (what percentage are blank or null?), and LGD declaration (for any lab-grown stones, is the declaration present and using FTC-compliant terminology?).
The results of this audit will give you a data quality baseline. If the cut grade vocabulary has more than five distinct values, you have a normalisation gap. If more than 5% of fluorescence fields are blank, you have an enrichment requirement. If any LGD stones are missing the laboratory-grown declaration in their schema, you have both an SEO problem and an FTC compliance risk.
For the Johannesburg retailer I described at the start of this article, the audit took four hours. The remediation — building the custom table, implementing the normalisation mapping, migrating the data, rebuilding the feed — took three weeks of development work. Eight months after the remediation was complete, their Merchant Centre product approval rate had moved from 71% to 96%. Their organic specification query traffic had increased by approximately 180% from the pre-remediation baseline. Their Shopping feed impression share had increased substantially. Not because they had added content, acquired backlinks, or changed their pricing. Because the data that had always been there was now telling the same story consistently, completely, and in a vocabulary that every system reading it could understand without ambiguity.
The Compound Effect: Data Quality as a Flywheel
I want to close with the strategic frame that makes data normalisation worth the investment — because without this frame, it is easy to treat it as a one-time technical cleanup rather than the flywheel it actually is.
Every additional data source you add — a new feed supplier, a manually entered product line, a batch import from a trade show — generates new normalisation requirements. Without a systematic normalisation process, every new source adds new vocabulary variants, new blank fields, new inconsistencies. The data quality problem compounds over time as the catalogue grows.
With a systematic normalisation process in place — a documented mapping, a validation step in the sync function, a custom table that enforces canonical vocabulary at the database level — every new data source is normalised on entry. The catalogue grows but the data quality holds. The Merchant Centre approval rate stays high. The Crawlable Shadow remains complete. The agentic routing confidence of every stone in the catalogue is maintained.
This is the flywheel. Not a content flywheel or a link flywheel. A data quality flywheel. A catalogue that is getting larger and staying consistent, rather than getting larger and getting messier.
The stores that build this architecture in year one are the stores whose catalogues in year three are not only larger but categorically more machine-readable than their competitors. Not because they invested more in marketing. Because they invested early in the infrastructure that makes marketing work — the clean, consistent, complete data layer that tells every system that evaluates it exactly what each stone is, in a vocabulary that cannot be misread.
That is the invisible competitive moat that compounds while everything else stays the same. And the foundation of it is a normalisation mapping document, a custom database table, and the discipline to enforce canonical vocabulary at every point where data enters the system.
Erwee Coetzee is the founder of SEO Gurus, a Cape Town-based technical SEO consultancy, and Diamond Stack, a specialist WooCommerce development practice for jewellery e-commerce. He has been active in technical SEO since 2012. Diamond Stack’s live diamond inventory integration service — including custom database architecture, feed normalisation, Crawlable Shadow implementation, and Merchant Centre feed management — is available at diamondstack.co.za. The Coetzee Liquidity Protocol (CLP), which provides the strategic framework underpinning the data architecture described in this article, is published under Creative Commons Attribution 4.0 at seo-gurus.co.za.
