I found that I need to scan a 55gb log file nearly 6000 times today. I’ve got a bash script to do the particular job which is relate to files stored in a tape silo, but if I start the process it will probably complete when I retire lol. So I’m thinking of shrinking the log file by only pulling out the information I need and the splitting that into say 500mb files and writing a script in perl to traverse the files smartly. I could also load the small files into memory to speed it up.
I always hear of people having to do this stuff and think “poor bastard”. Now I’m the poor bastard. Lol.