The Cache: Technology Expert's Forum
 
*
Welcome, Guest. Please login or register. September 18, 2019, 12:36:08 PM

Login with username, password and session length


Pages: [1]
  Print  
Author Topic: Processing large files  (Read 2692 times)
dirk
Global Moderator
Expert
*****
Offline Offline

Posts: 416


View Profile
« on: June 12, 2007, 09:18:57 AM »

Files are normally processed using the for loops below:

Code:
open my $INPUT, '<', $filename
    or croak "Can't open '$filename': $OS_ERROR";
my @lines = <$INPUT>;
close $INPUT;

foreach my $line (@lines) {
    # processing
}

Code:
open my $INPUT, '<', $filename
    or croak "Can't open '$filename': $OS_ERROR";

for my $line (<$INPUT>) {
    # processing
}

close $INPUT;

If the file is a large one, for example with 100 MB or more the script could stop with a memory allocation failure: Out of memory!

The reason is that the array @lines in the first example or the temporary array in the second example will require too much memory.

Therefor a while loop should be used which reads and processes only one line at a time:

Code:
open my $INPUT, '<', $filename
    or croak "Can't open '$filename': $OS_ERROR";

while ( my $line = <$INPUT> ) {
    # processing
}

close $INPUT;

So a file with 2 GB can be processed without any memory problems.


Logged
perkiset
Olde World Hacker
Administrator
Lifer
*****
Offline Offline

Posts: 10096



View Profile
« Reply #1 on: June 12, 2007, 09:59:52 AM »

Such important stuff Dirk - lots of n00bs do not consider streaming data anymore - the PHP function file_get_contents() makes it so easy to grab the whole file into a string variable that people are afraid of the fopen, fread, fseek and fclose style functions - which if you're an oldie (like you'n me  Wink ) you were raised on...

Thanks! Nice one
/p
Logged

It is now believed, that after having lived in one compound with 3 wives and never leaving the house for 5 years, Bin Laden called the U.S. Navy Seals himself.
Pages: [1]
  Print  
 
Jump to:  

Perkiset's Place Home   Best of The Cache   phpMyIDE: MySQL Stored Procedures, Functions & Triggers
Politics @ Perkiset's   Pinkhat's Perspective   
cache
mart
coder
programmers
ajax
php
javascript
Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC
Seo4Smf v0.2 © Webmaster's Talks


Valid XHTML 1.0! Valid CSS!