An OCDS data pipeline that turns public procurement releases into a searchable civil-infrastructure market view: sector, stage, location, buyer, cycle time and outcome.
A procurement process can appear in more than one OCDS release as its status changes. Descriptions are written differently, fields can be missing and the eTender portal itself is not a complete census of every procurement process. Counting raw rows as “number of tenders” would therefore be misleading.
I wanted a traceable pipeline that keeps the raw public record, identifies the latest state of each procurement process and classifies civil/road/asphalt opportunities without pretending the classifier is accurate before it has been tested.
I designed the route from official OCDS release to raw snapshot, staging, schema checks, latest-state selection and explainable classification. The remaining step is to test the classifier against a human-labelled production sample before publishing market counts.
Collect official public procurement releases.
Normalise fields and retain the latest state per OCID.
Assign explainable civil/road/asphalt labels and retain the matched terms.
Required before classifier totals are treated as evidence.
With the problem and data purpose clear, the next section shows the result first. The detailed steps below then show exactly how I got there.
The next evidence is a manually labelled production sample with precision, recall, false positives and false negatives. Until then, the pipeline is infrastructure, not a market conclusion.
National Treasury eTender OCDS APIDesigned the OCDS source-to-staging-to-reporting pipeline.
Retained raw release evidence and latest-state logic by OCID.
Built explainable road/asphalt classification rules.
Defined the manual precision/recall validation required before publishing classifier counts.
Public procurement data are available, but useful civil-engineering signals are scattered across releases and free-text descriptions. The decision problem is knowing what work is entering the pipeline, where it is, and how procurement processes move from advertisement to award.
The pipeline keeps the original release first, then extracts reporting fields. That means a classified record can still be traced back to the public procurement payload that produced it.
ocidocidStable procurement identifierbuyer.namebuyer_nameProcuring entitytender.titletender_titleClassification inputtender.statustender_statusProcurement stagetender.tenderPeriod.endDateclosing_dateTimingROW_NUMBER() OVER (PARTITION BY ocid ORDER BY release_date DESC)Keep rn = 1 for the current-state reporting table.
This avoids counting multiple releases of the same procurement process as separate tenders.classification + matched_terms + rule_versionA road/asphalt label is stored with the terms that triggered it.
If a result is wrong during manual review, I can trace which rule caused the false positive or false negative.Precision = TP/(TP+FP) · Recall = TP/(TP+FN)No production percentage is displayed yet.
The numbers will only be calculated after real imported descriptions have been manually labelled.The headline chart gives the decision signal. These additional views show whether the same conclusion survives when the data is sliced another way.
Production market charts are added only after real imported records pass validation.
The technical route from source to reporting table is documented.
I would not publish “X% of tenders are asphalt” as a trusted market statistic yet.
Those metrics decide whether the classifier is fit for the asphalt/tender dashboard.
The Transparency Portal itself states that the published data do not represent every procurement process in South Africa. Some entities provide richer fields than others.
Every interactive chart above already has a CSV button for its exact plotted numbers. These files are the deeper audit trail: source extract, model, SQL/DAX and validation.