How to Split Large CSV Imports into Chunks for AI Content Queues

Splitting a large CSV for an AI content queue means dividing it into bounded units of work — normally by byte size or by row count — and submitting each unit as its own batch or job. Which dimension you choose depends on the limit that binds first: a request-body cap, a row cap, or a per-record budget such as model calls or rate limits.

This guide is written for practitioners who already have a CSV of prompts, product descriptions, leads, or source documents and need to push it through a queue that calls a model. It covers the two documented chunking models, a reproducible splitting procedure, the failure modes that appear most often, and how to verify that every row arrived exactly once. The evidence base here is deliberately narrow: the C3 AI platform documentation on chunk source files and the Pinkfish Batch Service documentation on CSV ingest paths.

Two layers of chunking, and why confusing them causes bugs

Vendors use the word “chunk” for two different things, and the distinction matters when you are designing a queue.

The first layer is file-level chunking for parallel ingestion. In C3 AI, a source is a file or dataset imported into the platform, and chunking breaks each source file into smaller pieces so that each piece can be ingested in parallel rather than waiting for the previous piece to finish. The platform exposes this as configuration: chunkSizeInMb indicates the chunk size in MiB used for chunking content, while minByteSizeToChunk specifies the minimum size in bytes a source file must meet before it becomes eligible for chunking at all. That second parameter is the important one for small files — it exists so that modest files are not fragmented for no benefit.

The second layer is job-level batching. The Pinkfish Batch Service breaks large jobs into manageable pieces processed through queues: a batch is created with a unique batchId containing work items, a workflow processes each item, items are placed in a queue for parallel execution, and completion actions fire when the batch finishes. Batches can be created from a CSV, and the platform documents three ingest paths with different ceilings.

If you split by bytes only to satisfy an upload limit, but your worker bills per model call, you can end up with thousands of tiny chunks and heavy per-chunk overhead. If you split by rows only to keep batches semantically clean, a chunk can still blow past a request-body limit when individual rows are long. Most production pipelines need to respect both.

Start with the documented limits of your ingest path

Before writing any splitting code, find the numbers you are splitting against. For Pinkfish, the documented CSV ingest paths are:

Note what the table implies: hosting the file elsewhere does not remove the size constraint. The csvUrl path is still capped at 10 MB after the platform fetches the file — it changes where the bytes come from, not how many are allowed. The csvKey path is the one designed for genuinely large files, and it is the path most readers of this guide will need.

Step 1: Measure the file before you split it

Get both the byte size and an approximate record count:

wc -c leads.csv
wc -l leads.csv

The byte size tells you which ingest path is even possible. The line count is a warning sign, not a fact: wc -l counts newline characters, not CSV records. A quoted field containing a line break inflates the count, and a file whose final line lacks a trailing newline is undercounted by one. For an accurate record count, parse the file with a real CSV reader rather than counting lines.

Step 2: Decide whether the file needs chunking at all

Chunking is not free. Every additional chunk adds a request, a bookkeeping entry, and a unit that can fail independently. C3 AI’s minByteSizeToChunk parameter exists precisely to encode the rule “do not chunk files that are too small to benefit.” Apply the same discipline manually: if your file is comfortably under the smallest limit in your pipeline — say, a 3 MB CSV against a 10 MB request-body cap — ingest it whole and skip the rest of this procedure.

Step 3: Split by rows and keep the header on every chunk

Row-based splitting is usually the safer default because it cannot cut a record in half. The one thing you must not forget is the header. A chunk without a header is not a valid CSV for most consumers, and a worker that treats the first data row as a header will silently drop it.

import csv

def split_csv(path, rows_per_chunk, out_prefix):
    with open(path, newline="", encoding="utf-8-sig") as src:
        reader = csv.reader(src)
        header = next(reader)
        chunk_index = 1
        out = None
        writer = None
        count = 0
        for row in reader:
            if out is None:
                out = open(
                    f"{out_prefix}_{chunk_index:04d}.csv",
                    "w",
                    newline="",
                    encoding="utf-8",
                )
                writer = csv.writer(out)
                writer.writerow(header)
            writer.writerow(row)
            count += 1
            if count == rows_per_chunk:
                out.close()
                out = None
                chunk_index += 1
                count = 0
        if out is not None:
            out.close()

Reading with encoding="utf-8-sig" strips a byte-order mark if one is present, which prevents a stray character from being glued onto the first header name. Writing with newline="" keeps the CSV module in control of line endings.

The shell tool is faster but has a trap:

split -l 100000 -d --additional-suffix=.csv leads.csv chunk_

split is byte- and line-oriented. It has no concept of a header, so chunks after the first begin with a data row. If you use it, prepend the original header to every generated chunk before enqueueing — and be aware that it will also happily split inside a quoted field that contains a newline.

Step 4: Split by bytes only when payload size is the binding constraint

Byte-based splitting is what chunkSizeInMb describes: choose a target chunk size in MiB and cut the content to match. The danger is that a naive byte cut lands in the middle of a quoted field, producing two invalid records. If your platform performs this chunking internally through configuration, prefer that over hand-rolled byte slicing — the platform can account for record boundaries in a way a byte offset cannot.

If you must do it yourself, the practical compromise is to split by rows first, then measure each resulting chunk and re-split any that exceed the byte cap using a smaller row count. This keeps every chunk CSV-valid while still guaranteeing the size ceiling.

Step 5: Upload and enqueue in the documented order

For the csvKey path, the Pinkfish documentation describes a browser-direct flow. You first request a presigned upload URL, and the request body declares the filename and the exact byte size of the file you are about to upload:

POST /api/queues/{queueId}/items/csv/upload-url
{
  "filename": "leads.csv",
  "sizeBytes": 47185920
}

The platform validates the declared size against the 1 GiB cap and pre-signs an S3 PUT URL bound to that exact Content-Length. The response contains an uploadUrl that you then PUT the file to.

Two ordering rules follow from that design. First, declare the size honestly — the signature is bound to it, so a mismatch breaks the upload rather than being corrected silently. Second, do not modify, re-compress, or re-save the file between requesting the URL and performing the PUT. If your splitting step writes a file and your upload step re-encodes it, the byte count changes and the upload fails.

An illustrative splitting plan (arithmetic, not a tested run)

Suppose a content queue holds 480,000 rows of product copy in a roughly 90 MB CSV. Working through the documented limits: 90 MB exceeds the 10 MB inline request body, and it also exceeds the 10 MB ceiling for csvUrl after fetch, so the csvKey path is the only viable option. At 480,000 rows the file is under the 500,000-row cap, so a single batch would technically be permitted.

One batch, however, means one retry unit. If the workflow fails at row 400,000, the whole batch is the thing you re-run. Splitting into five chunks of 96,000 rows — roughly 18 MB each — keeps every chunk inside both caps while making retries cheap and allowing the queue to process chunks in parallel. These figures are illustrative arithmetic based on the documented caps, not a measured benchmark of any platform.

Failure modes and how to diagnose them

• Header missing on chunks after the first. Symptom: the first data row of each chunk disappears, or the worker rejects the file for an unexpected schema. Fix: re-emit the header in every chunk.
• Duplicate rows after a retry. Symptom: record counts exceed the source. A batch that partially succeeded and was re-run can import the successful portion twice. Fix: give each row or chunk a stable idempotency key, or track chunk-level completion so only failed chunks are resubmitted.
• Request body too large. Symptom: a rejection on the inline csvData path. Fix: reduce rows per chunk, or move to csvUrl or csvKey.
• Row cap exceeded. Symptom: a batch rejected for exceeding 500,000 rows. Fix: split further; the row cap applies per batch, not per file.
• Upload rejected after a presigned URL was issued. Symptom: the PUT fails on a content-length mismatch. Fix: confirm the file was not re-encoded between the size declaration and the upload.
• Corrupted records. Symptom: a row with an odd number of fields. Fix: stop line-based splitting; use a CSV-aware parser that understands quoted newlines.
• Chunks too small. Symptom: throughput drops and per-chunk overhead dominates. Fix: raise the chunk size, or apply a minimum-size threshold in the spirit of minByteSizeToChunk.

Verifying the import

Chunked ingestion fails quietly more often than loudly. Three checks catch most problems. First, sum the record counts of all chunks and compare against an accurate source count taken with a CSV parser — not with wc -l. Second, spot-check the first and last row of each chunk against the source to confirm no boundary rows were lost or duplicated. Third, confirm the queue or batch reached its terminal state and that any completion actions actually fired; a batch that is still listed as in progress after the last chunk uploads usually means one chunk was never enqueued.

Tradeoffs: more chunks is not automatically better

Smaller chunks buy parallelism and finer-grained retries. They also multiply the number of requests, the number of partially completed states you must reconcile, and the amount of bookkeeping you maintain. Larger chunks are simpler to reason about but make every failure more expensive and push you closer to the documented ceilings. The practical target is the largest chunk that stays comfortably under every applicable limit, with a minimum-size floor so that trivially small files are not fragmented at all.

If the queue runs on WordPress

The same principle applies, but the binding constraint is often different. Import plugins and cron-driven jobs on WordPress are typically limited by PHP execution time and memory rather than by a vendor row cap, so a single request that tries to process a 90 MB CSV may time out long before any API limit is reached. Splitting the CSV at the file level and enqueueing each chunk as a separate scheduled job keeps every request short and makes a failed chunk independently retryable. For long-running imports, WP-CLI can execute outside the web request entirely. Check your own host’s limits rather than assuming a particular value — and note that WordPress’s built-in cron is triggered by page visits, which is a poor fit for time-sensitive queue processing.

What to do next

Identify the one limit that binds your pipeline, split by rows with the header preserved on every chunk, apply a minimum-size threshold so small files pass through untouched, and verify the total record count after ingestion rather than trusting the queue’s success message. If you hit a failure mode that is not listed above, the most useful thing you can share with the community is your file size, row count, target platform and its documented limits, and the exact error text — that combination is what makes a chunking problem reproducible for someone else.