{"by":"JBerlinsky","descendants":68,"id":1603381,"kids":[1603511,1603783,1603459,1603455,1603725,1603615,1603594,1603554,1604402,1603517,1603495,1604384,1603527,1603465,1603581,1603481,1604383,1603531,1603453,1603717,1603544,1603508,1603446,1604178,1603576,1603633,1603447,1603518,1603476,1605146,1604281,1604006,1603448,1603587,1604132,1603485],"score":44,"text":"I've got a ~35GB text file full of data, and I want to parse it so I only have unique results in the end file. In the past, I've had no problem with cat FILE | sort | uniq, but I've never worked with anything of this magnitude. Even running a line count takes an extraordinary amount of time:<p>time cat FILE | wc -l\n2608560847<p>real\t11m18.148s\nuser\t1m35.667s\nsys\t1m33.820s\n[root@server src]#<p>Any suggestions on how I can go about getting unique records from this type of file?","time":1281797492,"title":"Ask HN: Sorting massive text files?","type":"story"}