While after some research I suspect what I want to do is essentially not possible, I'm looking for some workarounds so any community advice would help.

The project I'm working needs to be able to handle a rather large amount of data, and practically it's a bit much to load into RAM (not impossible, but I'd prefer a more lightweight solution).

I need to be able to save array based information into a hard table file and then use that as the primary information reference. That is, instead of keeping the entire database of information in RAM, I want to be able to selectively load and save the information into a hard file.

But it seems verge is not able to do this without rewriting the file from scratch every single time. So the question is, thusly, does anyone have a way that I can modify only part of a hardfile at one time?

So for example, if my file is:

A B C
D E F
G H I

I want to be able to open the file, overwrite one of the letters in the array, and save it again without having to spool all the data and write a new file.

I could work around this by having the program save hundreds to thousands of individual files, but this isn't a very elegant solution.

Any advice?

Well I'm working on the base core for a Master of Orion style 4X game. In theory I am stretching verge beyond it's intended purpose, but I find the interface very lightweight and easy to manuever through.

I need to generate system data for a potentially large number of stars. This data is brief and uncomplex -- at the moment. But the system files will also contain colony information and this could balloon to pretty large proportions over time. Each star *could* contain up to 204 planet/moon entities, and I can easily see each of these containing hundreds of relevant variables.

If I requisition an integer array of 180x204x250 it consumes 35 megs. I'm just concerned with bloat as other data heavy arrays are added in later on, or if the data needed grows as well. An alternate technique I've used before is using an array of strings and then pulling data tokens from those strings when needed.

I'd prefer to insulate myself against that problem by loading only the colony data being used currently by the system. I'd like to be able to index all the systems into a single table file, but that requires me to load all of the data and rewrite the entire file from scratch each time and I believe a select and replace system would be more efficient because then I can localize all the data into a few files, instead of hundreds. (this makes overwriting save games a much easier task as well).

This doesn't seem to be a terribly efficient approach to the situation. Admittedly, I am learning all this trial by fire, so I might be missing something logic wise. Because I'm using a lot more hard images (not just low resolution tilesets) the images are a bit memory intense too and I want to keep things slim and sexy from the beginning, rather than trying to go back and fix all this at a later date.

Early screenshot looks like this:

http://i141.photobucket.com/albums/r44/Sargana/ss.png

Gradual progress is taking place, but I haven't had much time to work on it lately. Blame Torchlight.