Large Datatable Streamer
Processes large datasets in streaming chunks to avoid PAD memory issues with DataTables over ~50K rows. Instead of loading the entire file into memory, reads N rows at a time from Excel, processes each chunk, writes to CSV output, and frees memory before the next chunk.
This script is included with FlowLibs Pro
The full PAD script and its setup notes unlock with a Pro or Team plan. Everything else on this page is free to read.
Provided as-is, without warranty of any kind. Review and test each pattern in a non-production environment before deploying it to live automations. See our Terms.
Problem this solves
PAD slows drastically or crashes with datatables over ~50K rows. Users report memory issues with large Excel files.
Usage Notes
- 1.Memory management is the point. The key action is Variables.ClearDataTable after each chunk write.
- 2.LABEL/GOTO loop pattern is used instead of LOOP WHILE because LOOP WHILE is not in the verified production reference.
- 3.CSV append mode: File.WriteToCSVFile with IfFileExists.Append adds rows without overwriting.
- 4.Excel stays open for the duration of streaming — opened once and read in windows via ReadCells.
- 5.Chunk size tuning: 5,000 rows is conservative. For narrow datasets increase to 10,000-20,000.
- 6.Safe stop (Pattern 6): Insert at the top of the chunk loop for attended runs.