Pyrel dev log

[Derakon]

A few weeks back I described an approach to handling the object model for a roguelike in this thread. I wasn't able to let it rest there, so I started implementing that object model, discovered it worked well, and moved on to adding actual features. So far I'm still at it.

I am legendarily bad at finishing big projects like this, but in the interests of keeping my motivation fresh I'm going to log my progress here. The ultimate goal is to rewrite Angband in Python -- a task several people have attempted in the past without success. I'm hoping that if I'm unable to complete it, I'm at least able to get it to the point that other people can start pitching in to help out. That in mind, eventually I should probably set up a GitHub repo for this project -- only problem is, I'm using Mercurial on my local machine (since I'm far, far more comfortable with it than with Git), and until there's an actual point in having GitHub up, I'd rather not disturb my workflow any more than necessary.

Some high-level stuff: I'm working in Python 2.7, and so far the only external dependency is wxPython, which is doing my windowing, event handling, and display logic. In the interests of making it as simple as possible for new devs to get started, I'm going to try to keep it to just that one external dependency if I can manage it. Standalone applications for OSX and Windows can be made with py2app and py2exe respectively; Linux has apt-get and the like.

I plan to have strong separation between the engine and display layers, so theoretically it should be possible to write e.g. a curses display mode so you can play over SSH or whatever. However, I'll leave that to others to work with. I'm only bothering to implement the "ASCII" display layer because I need it to be able to debug the code.

The way communication between the engine and display layers work is that the display layers provide commands to the engine, the engine updates state, then it hands its state to the display layer to show. Some events will result in prompts to the user; in that case, the engine provides the prompt type, the relevant container of things being prompted for (e.g. items in inventory, monsters in LOS), and a prompt message. The display layer has a free hand in how to display those prompts. For example, the standard Angband UI is keyboard-driven via series of menus, but the display code could decide to show clickable buttons, or even pop up a dialog box instead. All the engine cares about is that a selection is made (or the prompt is cancelled).

[Derakon]

My most recent work has been on loading monster records. I started out with the monster.txt and monster_base.txt files from v4, wrote a simple parser to load those into objects, and then serialized those objects using JSON. Then I removed the Angband files, replaced them with the JSON versions, and wrote a new (much simpler) loader to use them.

There's a number of reasons to do this, but the big two are:
* The old format doesn't cleanly allow for optional values
* The old format is rather inscrutable

There's a third reason which is mostly just a bit of a pleasant bonus: JSON's much easier for programs to load than the Angband file format (you don't need to write your own parser), so writing tools that can make it easier for variant maintainers to add/edit creatures isn't out of the question. It should be fairly easy to integrate into wizard mode, frankly.

I wanted optional values because I want to be able to simplify how display is handled. The idea is that each display mode (e.g. ASCII, small tile, Shockbolt tile, etc.) can attach its own display metadata to each monster record, which the corresponding display module will read out when it needs to draw that monster. That should be more pleasant to work with than having a big mapping of monster index numbers to image files / offsets, which is what I believe is currently being done.

Incidentally, index numbers are completely irrelevant so far.

Anyway, here's an example monster record:

{"index": 24, "name": "Small kobold",
"display": {"ascii": {"color": [255, 255, 0]}},
"templates": ["kobold"],
"hitpoints": 8, "speed": 110,
"visionRange": 20, "alertness": 70, "evasion": 0, "absorption": 0,
"nativeDepth": 1, "rarity": 1, "experienceValue": 5,
"blows": [["HIT", "HURT", "1d5"]],
"flags": ["DROP_60"],
"description": "It is a squat and ugly humanoid figure with a canine face."},

And here's an example monster template:

{"name": "kobold",
"display": {"ascii": {"symbol": "k"}},
"category": "Kobold",
"type": "kobold",
"flags": ["EVIL", "OPEN_DOOR", "BASH_DOOR", "IM_POIS"]},

Basically everything except the symbol and color are ignored, incidentally -- I haven't implemented AI, combat, or even a speed system yet. But speaking of color and symbol, I get arbitrary color selection and UTF-8 support for free.

[grinder]

Originally Posted by Derakon

There's a third reason which is mostly just a bit of a pleasant bonus: JSON's much easier for programs to load than the Angband file format (you don't need to write your own parser), so writing tools that can make it easier for variant maintainers to add/edit creatures isn't out of the question. It should be fairly easy to integrate into wizard mode, frankly.

Consider using YAML instead, as it's a superset of JSON (so no need to worry about it right now) and allows for something valuable for human readers that JSON lacks: inline comments.

[Derakon]

Hm, I'd been using JSON because a) I got it working quickly, and b) it had built-in support in Python via the json module (whereas YAML would require a third-party library). The lack of comments is a problem though. Thanks for pointing it out. We could just have a dummy field in the record that acts as a comment, but that strikes me as rather hackish.

On the flipside, if we assume that most modifications to these records will, in the future, be generated more often by editor programs instead of by hand, it's likely that comments that aren't embedded into the records as data would get blown away regularly anyway (insert comment by hand, someone else makes a change via editor -- boom, comment is gone). In which case, comments-as-data would be preferred, and YAML loses its major convenience.

Incidentally, this morning I switched over object.txt to JSON. Objects and monsters were the roadblocks to implementing a bunch of stuff, so from here I should be clear to start working on inventory, a combat framework, level generation, etc. My plan here is to just sketch out rough details -- for example, for combat I'll be omitting hit chance, critical hits, maybe even multiple blows/round; the main thing is to be able to hit things and have them die. The details can be backfilled later.

EDIT: ha! Writing a conversion from the old object file format to JSON does not equate to being able to create objects from those records! However, that is now working too, at least what I've tested of it -- I can load the records, tell the game to give me a stack of iron spikes, and get a random 6d7 spikes; I can also create daggers, potions, etc. Flavors haven't been implemented yet, and of course none of the items actually do anything, but it's progress.

The codebase is currently 1768 lines long, of which 284 lines are comments and 310 are whitespace. That's a bit light on comments, but not unacceptably so; I've just been being uncharacteristically terse while banging out big structures, for the most part.

[Derakon]

Objects can have object templates now, like creature templates. I spent a bunch of time manually creating a bunch of object templates, only to belatedly realize that Angband already had object templates in object_base.txt. They just aren't called out explicitly like the monster templates are. Oh well; I would have had to make adaptations anyway to handle templated display parameters and a few other things (like crowns always having a base AC of 0).

Example object record:

{"index": 238, "subtype": "Devotion",
"templates": ["amulet"],
"mods": [{"bonus": "1+M3", "flags": ["WIS"]}, {"bonus": "d6", "flags": ["CHR"]}, {"bonus": "1", "flags": ["LIGHT"]}],
"flags": ["SUST_WIS", "SUST_CHR", "", "RES_DARK", "RES_LIGHT", "RES_FIRE", "HOLD_LIFE", "GOOD"],
"allocations": [{"commonness": 10, "minDepth": 70, "maxDepth": 100}]},

Corresponding template entry:

{"type": "amulet", "template_name": "amulet",
"display": {"ascii": {"color": "flavor", "symbol": "\""}},
"weight": 0.3}

The "flavor" color means that the color of the object should derive from its flavor. Once I get around to assigning flavors to objects, that will actually be used. I already have the map of flavor to color ready to be used.

1824 lines, 408 are comments, 208 are whitespace. I'm...having trouble reconciling this count with my earlier count -- how did I manage to turn ~100 lines of whitespace into comments? The command I'm using, for what it's worth, is

find . -name "*py" | grep -v data | xargs cat | grep -P "^\s*#" | grep -cv "#"

The "grep -P" for some reason returns not just lines that have some amount of whitespace followed by a #, but also lines that have no # at all; oh well, I split them out after.

[Derakon]

I'm in the middle of making items equippable. The basic plan here is:

• Each item (or item template) record may include an "equipSlots" field. This is a list of slots the item can be equipped to, like "weapon", "launcher", or "neck". Currently all items in Angband can only be wielded to one type of equip slot, but there are variants that e.g. allow a weapon to be wielded to the shield slot, hence the list.
• Each creature (or creature template) record may include its own "equipSlots" field, which describes the slots the creature has available for wielding items. This is a mapping of slot descriptors (e.g. "on your back", "on your left finger", "providing light") to slot names (e.g. "back", "finger", "light source"). Thus we cover the equipment display as well as non-unique equipment slots (like the two ring fingers). This should make it fairly easy to let variants let the player play as different monster races.
• Whenever a creature equips or unequips an item, we need to recalculate the flags and bonuses they get from gear. We store a cached version of this for ease of lookup; it's this cached version that we use for all other game logic.

I'm also thinking about how to handle the melee/missile stat split. Specifically, there's some weirdness right now with how Rings of Finesse/Prowess apply to missile combat, and you can't currently get off-launcher/ammo slays to apply to missile combat. What I'm thinking is this:

• Some flags and mods apply only to the item that has those flags/mods. For example, EASY_KNOW, IGNORE_ACID. Conceptually we could have items that are more or less resistant to elemental damage too, so flags with associated pvals could go here.
• Some flags apply to the wielder. TELEPATHY, RES_ACID, etc. all go here.
• Some flags apply to one or more equipment slots (or rather the items currently occupying those slots). So you could have a ring that gives you extra finesse but only in melee (a Ring of Fencing?), or a cloak that gives all your other armor slots IGNORE_ACID.

I'm wondering if this is more complexity than gain though. The main benefits I can think of right now are:

* Off-weapon combat bonuses are more obvious about what they affect
* We have a few minor new items that we could create (c.f. anti-acid cloak)

The cost of course is that the complexity of the mods system is increased. Currently mods implicitly affect either the item or the wielder; we can keep that implicit system and avoid having to specify weapon slots for 99% of existing mods. But the code will still need to handle it. Thoughts? Especially if you can think of other interesting things you could do with items that affect other items when equipped.

[Magnate]

Originally Posted by Derakon

Whenever a creature equips or unequips an item, we need to recalculate the flags and bonuses they get from gear. We store a cached version of this for ease of lookup; it's this cached version that we use for all other game logic.

How are you handling player knowledge of player state? Are you keeping two entirely separate caches, one of "real" state and one of "known" state, or are you adding a "known" flag to each element of state, or what? IMO this will be a big determinant of your ID code.

I'm also thinking about how to handle the melee/missile stat split. Specifically, there's some weirdness right now with how Rings of Finesse/Prowess apply to missile combat, and you can't currently get off-launcher/ammo slays to apply to missile combat. What I'm thinking is this:

• Some flags and mods apply only to the item that has those flags/mods. For example, EASY_KNOW, IGNORE_ACID. Conceptually we could have items that are more or less resistant to elemental damage too, so flags with associated pvals could go here.
• Some flags apply to the wielder. TELEPATHY, RES_ACID, etc. all go here.
• Some flags apply to one or more equipment slots (or rather the items currently occupying those slots). So you could have a ring that gives you extra finesse but only in melee (a Ring of Fencing?), or a cloak that gives all your other armor slots IGNORE_ACID.

I'm wondering if this is more complexity than gain though. The main benefits I can think of right now are:

* Off-weapon combat bonuses are more obvious about what they affect
* We have a few minor new items that we could create (c.f. anti-acid cloak)

The cost of course is that the complexity of the mods system is increased. Currently mods implicitly affect either the item or the wielder; we can keep that implicit system and avoid having to specify weapon slots for 99% of existing mods. But the code will still need to handle it. Thoughts? Especially if you can think of other interesting things you could do with items that affect other items when equipped.

I have strong views about this (which you can of course ignore at leisure). I think that added complexity is fine, because it can always always be simplified later. Far better to design in capacity for complexity and ultimately not use it, than try to add it later.

So I'd encourage you to code the mods framework in a way that is completely flexible:
- any mod can affect any entity: @ or any monster(s) or any slot or item, including carried (not worn) items (combine your anti-acid cloak with Sangband's protection blankets), including not-self (dagger that boosts archery but not melee)
- any mod's effect on its affected entities can be on-wield or on-carry (so an artifact rod in the backpack can provide a slowing aura which only affects demons)

Just my 2p. Please keep up the good work. d_m and noz and I were discussing pyrel over a curry the other night and are all very excited about it.

[Derakon]

Originally Posted by Magnate

How are you handling player knowledge of player state? Are you keeping two entirely separate caches, one of "real" state and one of "known" state, or are you adding a "known" flag to each element of state, or what? IMO this will be a big determinant of your ID code.

I'm still working on properly handling equipping items to specific slots -- I haven't had much time to program these last few days and it took a bit longer than anticipated. So the ramifications of equipping items haven't been handled yet. However, here's my plan:

* Every trait of an equippable except for base AC, damage dice, weight, flavor, and of course its item category is considered a "mod" to the item. Some mods are binary (i.e. flags), some are parameterized (i.e. "pvals"). In the item record they're called "flags" and "mods" respectively. Yes, this means that +to-finesse and +to-prowess would be considered item mods.
* Flags are transparently mapped to +1 mods on load. The item has a list of mods that it has -- I'll probably create a new ItemMod class for this.
* Like ItemMods can be merged (e.g. a weapon with multiple Slaying affixes).
* ItemMods can be known or unknown.

Later on I/we can deal with affixes; they'll probably be approached as aggregates of Mods, and if every Mod in the aggregation is learned then the affix is learned too.

I have strong views about this (which you can of course ignore at leisure). I think that added complexity is fine, because it can always always be simplified later. Far better to design in capacity for complexity and ultimately not use it, than try to add it later.

Well, you can go two ways with this. If I tried to make Pyrel be capable of anything then I'd certainly never finish, for example.

So I'd encourage you to code the mods framework in a way that is completely flexible:
- any mod can affect any entity: @ or any monster(s) or any slot or item, including carried (not worn) items (combine your anti-acid cloak with Sangband's protection blankets), including not-self (dagger that boosts archery but not melee)
- any mod's effect on its affected entities can be on-wield or on-carry (so an artifact rod in the backpack can provide a slowing aura which only affects demons)

I have notify-on-pickup and notify-on-wield functions coded in, though of course currently they do nothing (and I'd also need notify-on-remove-from-inventory). I'm busy sketching out the framework, and plan to leave the detail work for later.

I spoke with d_m some about this yesterday evening; making mods targetable does significantly increase complexity even if we disallow second-order effects. I'm not certain I have a good idea for how to implement it cleanly yet; it'll bear some consideration.

Just my 2p. Please keep up the good work. d_m and noz and I were discussing pyrel over a curry the other night and are all very excited about it.

Oh dear, social pressure.

[Magnate]

Originally Posted by Derakon

* Every trait of an equippable except for base AC, damage dice, weight, flavor, and of course its item category is considered a "mod" to the item.

Do we need to distinguish between base AC and +ac? I guess you want the former to be a physical property and the latter magical? That works.

* ItemMods can be known or unknown.

Later on I/we can deal with affixes; they'll probably be approached as aggregates of Mods, and if every Mod in the aggregation is learned then the affix is learned too.

Sure. That's not quite what I was on about though. Consider regen: @ is healing (and getting hungry) faster than usual. There are at least two aspects of knowledge to this: (i) has @ noticed at all and (ii) is @ aware of the source of this change? The latter is the traditional "this property of this item is known", but the former is "this aspect of current state is known", which is different.

Well, you can go two ways with this. If I tried to make Pyrel be capable of anything then I'd certainly never finish, for example.

Hmmm. I see it a bit differently - you don't have to actually code every piece of complexity, merely keep your design able to accommodate it. But I guess we need to look at specifics:

I spoke with d_m some about this yesterday evening; making mods targetable does significantly increase complexity even if we disallow second-order effects. I'm not certain I have a good idea for how to implement it cleanly yet; it'll bear some consideration.

Please forgive me if I've misunderstood, but how does targetable (of entities) increase complexity more than targetable (of slots, which you said you already intended)?

I was thinking some more about this, and thinking that this is closely tied to how you implement effects (which in turn determines spell handling). Because although most item mods affect only attributes of something (usually @'s stats or resists), some of them affect the environment (OF_IMPACT, OF_TELEPATHY etc.). So it's probably worth thinking about effects at least a bit before making irrevocable conclusions about item mods.

Oh dear, social pressure.

But you asked for it :-)

If it helps, I'd happily volunteer to do Pyrel's item gen, if you want to outsource when you're happy with the mod framework. And I'm not sure if he told you this, but d_m wrote his dungeon gen code in python before porting it to V!

[Derakon]

Originally Posted by Magnate

Do we need to distinguish between base AC and +ac? I guess you want the former to be a physical property and the latter magical? That works.

That was my thinking. Base AC is also important for handling acid damage.

Sure. That's not quite what I was on about though. Consider regen: @ is healing (and getting hungry) faster than usual. There are at least two aspects of knowledge to this: (i) has @ noticed at all and (ii) is @ aware of the source of this change? The latter is the traditional "this property of this item is known", but the former is "this aspect of current state is known", which is different.

Don't we assume that if the player is aware of their condition then they also know which piece(s) of gear is causing that condition? Since otherwise @ has do to a bunch of tedious equipment swapping next to salamanders to figure out exactly what's giving him rFire, etc.

Hmmm. I see it a bit differently - you don't have to actually code every piece of complexity, merely keep your design able to accommodate it.

Well, my goal is to implement all of the framework stuff -- the "this is broadly what you are able to do with X". Once that's done I can open up the dev work to more people since the remaining work will mostly be filling in details (e.g. replacing the "generate a static room" code with "generate a random proper dungeon level" code). Item mods are part of the framework, IMO, because getting them wrong makes all aspects of item handling less pleasant. And items are a Big Deal in Angband.

Please forgive me if I've misunderstood, but how does targetable (of entities) increase complexity more than targetable (of slots, which you said you already intended)?

I was waffling on making mods slot-targetable, sorry; I should have made that more clear.

I was thinking some more about this, and thinking that this is closely tied to how you implement effects (which in turn determines spell handling). Because although most item mods affect only attributes of something (usually @'s stats or resists), some of them affect the environment (OF_IMPACT, OF_TELEPATHY etc.). So it's probably worth thinking about effects at least a bit before making irrevocable conclusions about item mods.

I suppose you could consider at some level an item mod to be a permanent passive effect...though I guess you could also have items that cast spells when they are equipped (and then can't be unequipped for awhile), or the like. But I think we can still safely delineate between passive and active mods, and leave the active ones for later.

Originally Posted by magnate2 on IRC

<magnate2> But like you I want to brood on this a little further. I think we need some kind of ontological hierarchy of things in Angband. During my thinking about effects I had got as far as classifying different genui of effects
[04:29] <magnate2> http://trac.rephial.org/wiki/NewEffects
[04:30] <magnate2> So I think I can come up with similar thinking about mods (which you may or may not find helpful :-))

Thanks, I'll check out that page. And if I think of anything about Pyrel that belongs on the wiki instead of in the source documentation, then I'll make a node.

If it helps, I'd happily volunteer to do Pyrel's item gen, if you want to outsource when you're happy with the mod framework. And I'm not sure if he told you this, but d_m wrote his dungeon gen code in python before porting it to V!

As I mentioned earlier, my plan definitely is to get the framework in place so everyone knows how to interact with the different components of the engine, and then set Pyrel loose for others to hack on. To be more specific, here's what I want to get done:

* Item mods, and applying those mods to the player / other items / etc.
* Simple level gen -- just making a small room with some level-appropriate items and monsters
* Simple combat -- letting the player take a whack at monsters, letting monsters take a whack at the player (assuming their incredibly stupid AI lets them try, anyway). This is actually almost done; I just need to affect hitpoints.
* Usable items -- make a potion that can be used to print a message, something like that.
* Savefiles (which will probably be just a serialization of all extant objects, possibly gzipped)
* Character birth and death

That leaves a whole ton of stuff that's really important to gameplay but won't affect the basic engine. Proper AI, v4's combat model, all of the item effects, spells (including monster spells), etc...and honestly the last two items could probably wait until later, if only because we may well be changing the item and monster records and that would mean rewriting the (de)serialization code each time.