DevOps for Course Catalog Processing on AWS Skip to content
← Back to Blog

DevOps for Course Catalog Processing: Concurrency, Queues, and Operational Reliability

Course catalog PDF processing pipeline with AI extraction and standardized database output

By Michael Walker, CEO of Horus Technology

Horus Technology moved DegreeData's course catalog pipeline to AWS CDK, added SQS-queued admission and Step Functions Distributed Map workflows for batches and chunks, and put concurrency settings under source control. This post covers what changed, the exact July 2025 settings, and why the throughput gain is a forecast rather than a measured result.

Results from DegreeData's benefits-tracking record: time from catalog receipt to availability fell from 3-6 months to same day for standard catalogs (2-3 days for complex formats), cost per catalog fell about 95% (from about $500 to about $31 including QC review), and savings at current volume are about $117,000 a year. The throughput forecast described below is separate and is not presented as a measured result.

What operational problem did DegreeData face?

Catalogs varied in size and format, so a single document could create multiple batches and each batch could create multiple chunks. Increasing parallelism at one level could overload a downstream model API or create more simultaneous Lambda demand than the processing configuration could support.

The DevOps challenge was to make processing demand predictable, failures recoverable, and capacity settings reviewable as the workload evolved.

DegreeData, an education data company, uses course catalogs to provide standardized course and program information to universities and education technology platforms. Its document-processing baseline combined Amazon S3, AWS Lambda, AWS Step Functions, model inference, and database persistence. Large intermediate results also had to fit within workflow payload constraints.

Infrastructure and workflow changes

Horus Technology maintained the processing infrastructure in AWS CDK, with version-controlled definitions for compute, state machines, storage access, and workflow admission. The pipeline used S3 for source documents and intermediate artifacts, Lambda for processing stages, Step Functions for orchestration, and DynamoDB and RDS for processing data and structured outputs.

  • Distributed processing: In July 2024, we introduced separate batch and chunk Distributed Map workflows, with dedicated aggregation and result-joining stages.
  • Queued admission: We added an SQS-backed workflow-start throttle to control the incoming processing demand independently of the work inside each document.
  • Failure handling: We added retry backoff, an error catcher, and a separate text-extraction workflow. We also investigated memory pressure in chunk creation.
  • Capacity controls in code: Batch, chunk, and document concurrency settings could be reviewed and changed through source control rather than left as undocumented console settings.

How did we replace hand-run deploys with CDK?

We rebuilt the API and the processing pipeline as AWS CDK applications, with staging and production as instances of the same stack and every change reviewed through a GitHub pull request. Before the engagement, DegreeData's API was a Go service on AWS Elastic Beanstalk that a previous developer had built, compiled on a workstation and deployed by hand with the Elastic Beanstalk CLI.

Merging to the main branch deployed nothing, a bad deploy was hard to roll back, application secrets sat in a committed configuration file, and no environment could be recreated from source. The rebuild had these parts:

  • One codebase, two environments: Staging and production are instances of the same CDK stack, so a change is exercised in staging before it reaches production.
  • Rollback: AWS CloudFormation rolls back a failed deployment, and a capacity change is reverted by reverting its commit.
  • Secrets and data: Credentials moved to AWS Secrets Manager, and the production database was restored into isolated subnets from a cutover snapshot when the workload moved to us-east-1.
  • Reviewed changes: Every change goes through a GitHub pull request with a build check (a frozen-lockfile install and a Projen drift check) and is merged by Mergify only after the check passes and a reviewer approves. Deployments to staging and then production are run by the delivery engineer with cdk deploy.
  • Automated dependency upgrades: A daily Projen workflow opens dependency-upgrade pull requests that merge only after the build passes. Between November 2024 and October 2025, 33 such upgrades merged, and two high-severity vulnerabilities (CVE-2024-39338 in axios and CVE-2024-4068 in braces) were resolved within 24 hours.

How did we tune concurrency for DegreeData?

In August 2024, we reduced batch concurrency from five to three and chunk concurrency from five to two to protect downstream model capacity, and we also increased retry backoff. In July 2025, we committed a revision that raised both Map concurrency settings to 15 and the document admission threshold from 10 to 75.

The operating configuration changed as we learned where the bottlenecks were. A subsequent investigation recorded chunk-processing invocations capped at two, with approximately 23-25 seconds per invocation and no Lambda throttles in the report's observation window. This identified the configured orchestration cap as a throughput constraint during that window.

The table shows the full July 2025 revision to those capacity controls:

Version-controlled configuration changes in the July 2025 throughput revision
ControlBeforeAfter
Chunk Map concurrency215
Batch Map concurrency315
Document admission threshold1075
SQS batch size15
Artificial delay in chunk Lambda1-30 secondsRemoved

These values are application settings, not AWS service quotas. The changes establish what was implemented. The optimization commit forecast a 7.5x-11x throughput improvement; that forecast is not presented here as a measured production result.

Operational outcomes

The delivered platform had explicit controls for admitting work, distributing batches and chunks, handling failures, and adjusting processing capacity. Its source history records both conservative settings used to protect model capacity and a later revision addressing underutilized processing capacity.

This work established an iterative operating process: investigate observed behavior, identify the constraining layer, change infrastructure and application settings in source control, and reassess downstream demand. A higher Map concurrency setting alone does not establish a proportional improvement in completed documents per hour.

The architectural lesson applied later at VRC

Our subsequent VRC engagement encountered a related orchestration problem. Nested document, patient, and chunk processing multiplied Lambda demand. The January 2026 investigation identified duplicate SQS event-source mappings and demand exceeding a processing function's reserved concurrency.

One VRC remediation explicitly budgeted three document launches, five patient iterations, and three chunk iterations: 45 potential simultaneous calls against reserved concurrency of 50. This made the relationship between queue admission and internal fan-out explicit.

In April 2026, we implemented ECS Fargate workers for VRC that consume SQS messages and run document-processing stages in-process. The worker path removed the per-stage Lambda invocation fan-out and introduced queue-based worker scaling, graceful shutdown, and deployment controls. Inference hosting on ECS was a separate architectural change.

VRC's development validation report records a 1,000-document stress test completed in 556 minutes with zero dead-letter queue messages. Workers scaled from two to ten. GPU capacity was increased manually, and a later GPU autoscaling interruption required manual restoration while workers retried and resumed. These are VRC test results with operational qualifications; DegreeData's documented changes are the queueing, workflow, and capacity improvements described above.

How does Horus help teams run document AI on AWS?

Horus Technology helps teams operate and improve document AI workloads on AWS through infrastructure as code, concurrency and capacity planning, queue-based admission, retry and recovery design, and validation of architectural changes. We evaluate the full processing path, including model APIs, GPU capacity, storage, and databases, when choosing between serverless orchestration and container workers.

For the scope of this work, see our DevOps services for document processing. For architecture and cost review of an existing AWS workload, see our AWS consulting work, and for what running a pipeline like this costs, read how much intelligent document processing costs.

Get started

Tell us what you're trying to solve.

AWS architecture, a GenAI pilot, document processing, a migration off another cloud — tell us where you're stuck and we'll tell you if we can help. We build in your own AWS account, so what's yours stays yours.