Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Opening LARGE text files
#1
I made a program that uses <fstream> to open a text file, via
Code:
if (file.is_open()) . . .

It works just fine with a small file of ~3MB, but the file I NEED it to work on is 68MB, and whenever I try to run my program I get this:

Code:
This application has requested the Runtime to terminate in an unusual way.  Please contact the application's support team for more information.

Then Windows pops up a message saying the the application crashed.

FYI: Using GCC on Windows Vista, no IDE. Compiling with g++ -o appName app.cpp.

Also, would this work on some sort of *nix?
Reply
#2
The only thing I can think of is that the file size is causing a buffer overrun, but realistically that size of file should be fine.

Where is it actually crashing in code? I guess you can't really tell since you don't have an IDE.
[Image: sig.php]
The little boat gently drifted across the pond exactly the way a bowling ball wouldn't.
Reply
#3
I'm not too familiar with C++ but you probably need to read the file one chunk at a time, so you don't load the entire file into the memory(that's what happens with python).
Here is an example:
Code:
// read a file into memory
#include <iostream>
#include <fstream>
using namespace std;

int main () {
  int length;
  char * buffer;

  ifstream is;
  is.open ("test.txt", ios::binary );

  // get length of file:
  is.seekg (0, ios::end);
  length = is.tellg();
  is.seekg (0, ios::beg);

  // allocate memory:
  buffer = new char [length];

  // read data as a block:
  is.read (buffer,length);
  is.close();

  cout.write (buffer,length);

  delete[] buffer;
  return 0;
}
look here:
http://www.cplusplus.com/reference/iostr...ream/read/
[Image: izsyo6.jpg]


Reply
#4
try using different methods instead such as the FILE structure?
Reply
#5
Ok, I got it.

The issue was the final getline grabbing just a null character at the end of the file, which then got ran through my loops trying to probe it with string.substr() functions and whatnot.

What I still don't get is why this didn't occur on the small file, which had the same exact final lines in the file. Oh well.

I fixed it by probing ahead at each getline and putting the rest of my code in an "if" statement, occurring if it doesn't pass the "Did I only grab a null character?" test.

Thanks for those that posted!
Reply
#6
thats a big text file, why would u need to? haha
Reply
#7
Is there a way to trim them? (Divide files into sections) ?
Reply
#8
There must be away to trim them.
Reply
#9
Sorry I'm a few months late, but the reason I needed to parse such a huge file was because I was analyzing an exported Event Log from a client's server that was experiencing tons of hacking attempts. I used my program to sort it quickly into which IP addresses were logged, how often they were occurring, which usernames were tried, etc. Looking back on it, yes I could have divided the file up into sets of Event Log entries. It worked, though, so all is well.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)