Search / Data / WordPressBackend search and data workflow

WordPress-facing search system

SCJPC Search Database System

WordPress-facing search pages for poles and JPA records backed by PostgreSQL, Elasticsearch, RabbitMQ, S3, and Python workflows.

Project summary

Worked on the backend search and migration workflow that kept WordPress as the public layer while heavier data processing lived in Python, PostgreSQL, Elasticsearch, RabbitMQ, and AWS.

2M+ records searchable through WordPress with PostgreSQL, Elasticsearch, RabbitMQ, S3, and Python behind it.

Project summary

Dataset size

2M+ searchable records

Public layer

WordPress search pages

Workflow

Monthly CSV to S3 to queue

Worked on the backend search and migration workflow that kept WordPress as the public layer while heavier data processing lived in Python, PostgreSQL, Elasticsearch, RabbitMQ, and AWS.

Buyer-facing summary

Client problem

The client needed search pages for poles and JPA records, plus a required monthly manual CSV update workflow.

What I delivered

I worked on the Python migration flow, S3 processing, queue handling, PostgreSQL structures, Elasticsearch search, and WordPress-facing search integration.

Business result

The client could keep the public experience in WordPress while heavier data updates and search logic ran through tools built for that workload.

Problem

SCJPC needed search pages for poles and JPA records, but the real challenge was not just the public UI. The data arrived through a client-required monthly CSV process and had to be normalized, compared, migrated, and made searchable.

A normal WordPress database model would have put too much responsibility on the CMS. The better shape was to let WordPress remain the public layer while backend tools handled the data workflow.

The client required manual monthly CSV generation and upload rather than a fully automated upstream data feed.

The search experience had to remain usable through WordPress even though the data shape did not fit normal WordPress querying.

What I built

CSV migration workflow

Built and supported Python-based processing where client-provided CSV files were uploaded to S3, queued, downloaded, compared, and inserted or updated through a controlled migration path.

PostgreSQL and materialized views

Used PostgreSQL as the structured storage layer, with materialized views supporting faster query paths for repeated structured search needs.

Elasticsearch text search

Handled text-heavy search through Elasticsearch so WordPress did not need to fake full search over a specialized, large dataset.

WordPress-facing search pages

Kept the user-facing search experience in WordPress while the backend data and indexing responsibilities lived outside the CMS.

Python migration/import workflowS3-based CSV upload and processing flowRabbitMQ-backed migration handlingPostgreSQL storage and materialized viewsElasticsearch-backed text searchWordPress-facing search pages

Technical decisions

CSV files were uploaded to S3, queued for processing, downloaded, compared against existing records, then inserted or updated through the migration workflow.

PostgreSQL materialized views supported faster structured queries while Elasticsearch handled text-heavy search behavior.

RabbitMQ helped keep longer migration work out of request-time execution.

The most important design choice was separation of responsibilities: WordPress for presentation, PostgreSQL for structured data, Elasticsearch for text search, RabbitMQ for background work, and S3 for upload handoff.

The manual CSV requirement shaped the architecture. Instead of pretending the upstream process was fully automated, the system made the required manual handoff predictable and recoverable.

Outcome

The system made a specialized dataset practical to search from a WordPress-facing interface.

Monthly updates could be handled through a repeatable backend workflow instead of fragile request-time imports.

The project is strong proof of backend architecture, async processing, AWS workflows, WordPress integration, and large-search implementation.

What I would improve

If the upstream data source ever allowed it, I would push toward a more automated feed with stronger validation reports before each migration run.

For a client-required manual workflow, the right goal is not to hide the manual step; it is to make every step after upload consistent and observable.

Tech stack

PythonPostgreSQLElasticsearchRabbitMQAWS S3WordPress

Next step

If you need similar work, let’s talk through the constraints first.

The useful part of a project like this usually starts before code: understanding what the CMS should own, what should live in a backend service, and where integrations or automation can stay maintainable.

Start a conversation

Related work

WordPress Plugin / Media

AudioMazes WordPress Audio Plugin

A custom WordPress audio player plugin with listening history, S3 storage, CDN delivery, and Elementor widget work.