PDA

View Full Version : How about ... replacing /edit/*.txt and /data/*.raw with SQLite ?


PaulBlay
May 29, 2009, 12:43
Serious question: Could, or should, the *_info loading routines and files be replaced with the use of SQLite and a small database?

Rationale: Reading the routines for *_info loading and export/import to *.raw files makes my head hurt. :D

Contraindications: If it is not possible to use SQLite with a major platform supported by Angband then this idea would probably have to be dropped (at least for Vanilla).

Magnate
May 29, 2009, 13:20
Serious question: Could, or should, the *_info loading routines and files be replaced with the use of SQLite and a small database?Undoubtedly it could be done, subject to your caveat about the availability of SQLite for supported platforms.

The question is, how easy will it be for the end user to tailor the game to his/her liking? At the moment the text files provide a (fairly) friendly interface for customising the game. IMO if we move to a database format, it is incumbent upon us to write a user interface to the database.

Otherwise I won't be able to mess with hound rarities!

PaulBlay
May 29, 2009, 13:25
IMO if we move to a database format, it is incumbent upon us to write a user interface to the database.

Otherwise I won't be able to mess with hound rarities!

Producing a utility that a) import/exports text files and b) allows easy editing of the data would seem to be an nice intermediate step between the current system and a possible replacement.

The question is, how easy will it be for the end user to tailor the game to his/her liking?

Theoretically it could be easier than the present system. In that automatic checking of max values, ID numbers, missing/malformed fields could be done. There are, however, a number of "hard coded" numbers in Angband so that needs to be born in mind. Any changes that add / remove records would also break earlier save games (but otherwise could be done with care).

PaulBlay
May 29, 2009, 13:55
In fact I think I'll just go ahead and try it (http://svn.sourceforge.jp/view?view=rev&root=jband&revision=19). The utility should be equally functional for JBand and Angband, and easily altered to other *band variants.

As always, assistance will be welcome. :D

konijn_
May 29, 2009, 15:50
Producing a utility that a) import/exports text files and b) allows easy editing of the data would seem to be an nice intermediate step between the current system and a possible replacement.

Theoretically it could be easier than the present system. In that automatic checking of max values, ID numbers, missing/malformed fields could be done. There are, however, a number of "hard coded" numbers in Angband so that needs to be born in mind. Any changes that add / remove records would also break earlier save games (but otherwise could be done with care).

http://hellband.googlecode.com/svn/trunk/js/hellband.xls

I did away with index numbers, they are evil ;]
And it generates js, not x_info files, but that should be trivial to change.

Doing the exercise of creating this excel sheet has allowed me to find 'bugs' in the data files. Now I know why certain items would never appear ;]
It also was easier to see what pops up where and I adjusted some items more generously and some more stingily. I would advise a similar exercise to any variant maintainer who doesn't know what to do next ;]

T.

pav
May 29, 2009, 15:58
Using SQLite is always a bad idea.

PaulBlay
May 29, 2009, 15:59
Using SQLite is always a bad idea.

So what would you recommend instead?

pav
May 29, 2009, 16:01
I think the current scheme works fine. I would say we can get rid of .raw files and just parse up .txt files on every startup. The main reason for .raw files -- speeding up startup times, is largely obsoleted by today hardware.

PaulBlay
May 29, 2009, 16:53
I think the current scheme works fine.

Not very tinker friendly, but that's not your problem. :D

I would say we can get rid of .raw files and just parse up .txt files on every startup.

I would certainly welcome that.

takkaria
May 29, 2009, 20:36
Serious question: Could, or should, the *_info loading routines and files be replaced with the use of SQLite and a small database?

Rationale: Reading the routines for *_info loading and export/import to *.raw files makes my head hurt. :D

Contraindications: If it is not possible to use SQLite with a major platform supported by Angband then this idea would probably have to be dropped (at least for Vanilla).

I don't think this is particularly useful, for several reasons:
1. text files are easier to edit than databases
2. with a database editor, you have to maintain a database editor too
3. sqlite takes up a lot more space than the current system for very little gain
4. sqlite will be slower than what we have now
5. for all the work it would take, the benefits of sqlite over text files are slim

Nick
May 30, 2009, 00:54
I agree with the naysayers. One of the big contributors to Angband's survival is it's self-containment. The various platform ports need to deal with file-handling and display, and everything else is just C language. And even then we have a raft of implementation issues.

Nice application of the takkaria sig, too :)

saarn
May 30, 2009, 04:57
I'm honestly surprised by the negative reactions. Angband has hundreds of different types of monsters, objects, etc. and a relational db could provide a lot of power for managing them, expanding them, analyzing balance, etc. It would also probably reduce a number of code maintenance concerns by replacing various text parsing code.

The objections I've seen so far are:

1. size: 300K to embed sqlite
2. self-containment: comes as 1 library and 1 header file. s.
3. language: it's ansi c (and public domain too)
4. speed: Seriously? This is Angband, not Crysis. But, if speed is a concern, I would be willing to bet that all of the objects/monster types in Angband could easily fit into memory tables.
5. platforms: seems to be just about everything. It's even the default engine for iphone apps.
6. tinkering: it's freeware, and users can download a client if they want to muck with monsters. There's also tools for importing/exporting to csv if learning SQL is too daunting for end users.

That said-- while I do a lot of work with databases (mysql and postgres), I haven't personally touched sqlite. Does anyone have specific technical reasons sqlite is "always a bad idea?" Is it flakey, does it cause horrors in the build process? Or is this push-back coming from fear of the "technological unknown?"

takkaria
May 30, 2009, 05:02
I'm honestly surprised by the negative reactions. Angband has hundreds of different types of monsters, objects, etc. and a relational db could provide a lot of power for managing them, expanding them, analyzing balance, etc. It would also probably reduce a number of code maintenance concerns by replacing various text parsing code.

The objections I've seen so far are:

1. size: 300K to embed sqlite
2. self-containment: comes as 1 library and 1 header file. s.
3. language: it's ansi c (and public domain too)
4. speed: Seriously? This is Angband, not Crysis. But, if speed is a concern, I would be willing to bet that all of the objects/monster types in Angband could easily fit into memory tables.
5. platforms: seems to be just about everything. It's even the default engine for iphone apps.
6. tinkering: it's freeware, and users can download a client if they want to muck with monsters. There's also tools for importing/exporting to csv if learning SQL is too daunting for end users.

That said-- while I do a lot of work with databases (mysql and postgres), I haven't personally touched sqlite. Does anyone have specific technical reasons sqlite is "always a bad idea?" Is it flakey, does it cause horrors in the build process? Or is this push-back coming from fear of the "technological unknown?"

You missed mine:
1. text files are easier to edit than databases
2. with a database editor, you have to maintain a database editor too

Basically, using SQLite would increase the maintenance burden with not enough benefit to show for it (for me, anyway). Other variant authors may differ, and indeed, do, but I don't think the cost-benefit analysis works out.

Pete Mack
May 30, 2009, 05:39
Do you really want to use SQL as an engine, and what interface would you use? (ODBC?--it's not as readable as the C.)

Or is it that you want to run ad hoc queries? That's a different, and easier, problem--write an editor package that allows you to parse Angband edit files into csv and back, or go directly between the DB and Angband text files.

Atarlost
May 30, 2009, 06:35
You're trying to get us to put a hemi v8 in a model A. It's a waste of effort.

PaulBlay
May 30, 2009, 06:39
You're trying to get us to put a hemi v8 in a model A. It's a waste of effort.

SQLite can be compared to a lot of things, but a hemiv v8 isn't one of them.

Delver
May 31, 2009, 03:00
What about XML? It has at least one C library I know of, and is text based. :)

Actually, I don't think the edit files need to go, at least not yet. If it's not broken, don't fix it, I guess. And giving Angband a full-blown database would probably be overkill.

I remember when Angband had Lua in it (for no well-defined reason, I gathered). I suppose, if Angband kept Lua it could have used Lua files for edit file replacement in addition to scripting, since Lua was originally defined for data definition and is pretty good at that through its table syntax. I'd leave that to Lua-using variants like T.o.M.E. though.

Atarlost
May 31, 2009, 05:53
XML can actually be harder to read because all the tags can and linefeeds can prevent something complex like a spellcasting monster from fitting on a single screen.

RogerN
June 1, 2009, 19:05
I think this is an terrible idea.

1. If your only motivation for switching to a database is to improve the readability of the info loading routines, isn't it better to just rewrite or improve the current loading routines rather than start from scratch with a database?

2. Modifying the *_info text files is quick and easy. If you want to make this even easier, it would be just as simple to write an editor for the *_info files as it would be to write an editor for a database.

3. The format of the *_info files is already very easy (and quick) to parse. Tinkerers can write their own utilities to examine these files and extract information. If you switch to a database, however, then any other utility which wants to retrieve this information will also have to link to a bunch of database stuff.

4. For the most part, Angband is free of dependencies on external libraries. Let's keep it that way.

I do agree that removing the .raw files might be nice. I'd be interested in seeing some benchmarks on slower machines loading .raw vs. .txt, as I suspect the speed difference is not big enough to justify keeping separate .raw files. You only load the game once, and IMO simplicity is better than saving 2 seconds of startup time.

Magnate
June 1, 2009, 19:13
You only load the game once, and IMO simplicity is better than saving 2 seconds of startup time.You load the game every time you die and start a new character - which for some of us is many times in an evening.

Besides, where did people get the impression that keeping the .raw files is all about speed? As I explained earlier in the thread, the .raw files enable the game to be shipped without text files and compiled without (most of) init1.c - it seems to be more about size than speed. I could be wrong - I wasn't around when the design decision was made - but the comments in the code hint that size was the first consideration, not speed. Plus, Angband was developed on university machines where tampering was rife, so it's quite possible that neither size nor speed was the primary reason for the .raw files, but tamper-proofing.

andrewdoull
June 3, 2009, 01:26
Besides, where did people get the impression that keeping the .raw files is all about speed?

The raw files are about speed. I compiled them out the other day to see the savings in size and it was about 40k of code size saved. And this was on Unangband, which has a lot more edit files than Angband.

Andrew

Magnate
June 3, 2009, 07:39
The raw files are about speed. I compiled them out the other day to see the savings in size and it was about 40k of code size saved. And this was on Unangband, which has a lot more edit files than Angband.Right. Now, back in 1991, how significant was 40k? IIRC it was about 1/6 of all available RAM. I reckon that would have mattered quite a lot to the original devs.

I'm *not* saying that speed isn't an issue - it is. But I don't think it was the only reason, or even the primary reason, that the raw files were invented.

david3x3x3
June 8, 2009, 03:31
I do agree that removing the .raw files might be nice. I'd be interested in seeing some benchmarks on slower machines loading .raw vs. .txt, as I suspect the speed difference is not big enough to justify keeping separate .raw files. You only load the game once, and IMO simplicity is better than saving 2 seconds of startup time.

I've been working on getting Angband running on Android. This is a slow machine. It's running as machine code translated to Java bytecode running in a virtual machine on a mobile phone processor. That it runs at all is impressive to me. I estimate that the raw files cut 30 seconds off the startup time on this software/hardware combo. Also, due to how multitasking is handled on the phone, I'd say that I'm more prone to restarting the program than I would be running the program under Windows.