I have a gigantic csv formatted file, 10GB+ uncompressed. There's nothing I can do about it, it's not my file; I'm just trying to import the data in the file. Another organization generates it and I've already tried to get it from them in smaller chunks and lost that battle. Anyways...
To test the viability of native cmdlets, I took a different 3.5GB file (just because it was smaller) and tried to use Import-CSV on it and watched PowerShell's memory exceed 15GB before I cancelled the import (via CTRL-C) and then [gc]::Collect(). So it quickly appeared that if 3.5GB exceeds 15GB in memory, a 10GB file would be monstrous.
So I started using a stream reader to read in the file line by line and acting on that information. What I'd like to know is, is there a native way to convert an array of CSV data to a CSV object (...PSObject that's created from import-csv). I originally thought Convertto-Csv would be the answer it unfortunately does the opposite of what I want: converts an object to raw CSV formatted strings.
I'm currently grabbing about 20K lines, exporting to CSV temp file, and then reimporting, but I wanted to know if there was a way to get around the IOPS hit... a way to convert the data, in memory, to a CSV PSCustom object but using all of the internal PowerShell logic. Any help would be appreciated -- thanks.