Opening LARGE text files

[J4P4NM4N]

I made a program that uses <fstream> to open a text file, via

Code:

if (file.is_open()) . . .

It works just fine with a small file of ~3MB, but the file I NEED it to work on is 68MB, and whenever I try to run my program I get this:

Code:

This application has requested the Runtime to terminate in an unusual way.  Please contact the application's support team for more information.

Then Windows pops up a message saying the the application crashed.

FYI: Using GCC on Windows Vista, no IDE. Compiling with g++ -o appName app.cpp.

Also, would this work on some sort of *nix?

[MrD.]

The only thing I can think of is that the file size is causing a buffer overrun, but realistically that size of file should be fine.

Where is it actually crashing in code? I guess you can't really tell since you don't have an IDE.

[uber1337]

I'm not too familiar with C++ but you probably need to read the file one chunk at a time, so you don't load the entire file into the memory(that's what happens with python).
Here is an example:

Code:

// read a file into memory
#include <iostream>
#include <fstream>
using namespace std;

int main () {
  int length;
  char * buffer;

  ifstream is;
  is.open ("test.txt", ios::binary );

  // get length of file:
  is.seekg (0, ios::end);
  length = is.tellg();
  is.seekg (0, ios::beg);

  // allocate memory:
  buffer = new char [length];

  // read data as a block:
  is.read (buffer,length);
  is.close();

  cout.write (buffer,length);

  delete[] buffer;
  return 0;
}

look here:
http://www.cplusplus.com/reference/iostr...ream/read/

[Genesis]

try using different methods instead such as the FILE structure?

[J4P4NM4N]

Ok, I got it.

The issue was the final getline grabbing just a null character at the end of the file, which then got ran through my loops trying to probe it with string.substr() functions and whatnot.

What I still don't get is why this didn't occur on the small file, which had the same exact final lines in the file. Oh well.

I fixed it by probing ahead at each getline and putting the rest of my code in an "if" statement, occurring if it doesn't pass the "Did I only grab a null character?" test.

Thanks for those that posted!

[haphazard]

thats a big text file, why would u need to? haha

[`P R O D I G Y™]

Is there a way to trim them? (Divide files into sections) ?

[Kondry]

There must be away to trim them.

[J4P4NM4N]

Sorry I'm a few months late, but the reason I needed to parse such a huge file was because I was analyzing an exported Event Log from a client's server that was experiencing tons of hacking attempts. I used my program to sort it quickly into which IP addresses were logged, how often they were occurring, which usernames were tried, etc. Looking back on it, yes I could have divided the file up into sets of Event Log entries. It worked, though, so all is well.