Python writing to file slow

Now it seems the next win will be reducing the amount of time spent writing to disk. I'd like to reduce it, if possible.

I can keep a cache of records in memory, but I can't wait until the end and dump it all at once. My first thought would be to keep a cache of records in a list and write them out in batches. Would that be faster? Something like:. Actually, your problem is not that file.

Writing to the disk is slow. There is really nothing you can do about it. It simply takes a very large amount of time to write stuff out to disk. There is almost nothing you can do to speed it up. The ideal is for file.

Batching the writes into groups of did indeed speed up the writes significantly. For this test case the writing rows individually took Batches of took a total of only 0. You can do mmap in python, which might help. Do a test in which you write random lines with the same length, and you will see it's much faster than that.

Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Speed up writing to files Ask Question. Asked 10 years, 11 months ago. Active 7 years, 7 months ago. Viewed 40k times. Improve this question. Python: when writing to a large file, keep the file open or to open it and append to it as needed?

Ask Question. Asked 7 years, 8 months ago. Active 7 years, 8 months ago. Viewed 7k times. I am wondering how to best handle writing to a big file in python. What I am already doing.

At no stage does the whole output need to be in memory. Improve this question. This looks promising Will this allow me to keep the legacy program's input and output in memory without writing to disk? NB I really don't want to change the Fortran program at all. Add a comment. Active Oldest Votes. We can see that using multiple threads indeed speed things up.

Using 1 thread: Depending on your processor s and how many threads you choose to or can spawn, this will definitely speed things up. Reach out on LinkedIn or email. Getting Started We will need xlsx2csv.