Will we get rid of the 32 bit flag variables?

[Bandobras]

Once V starts seeing SVN commits again, I'd be comfortable providing this patch (to see what it does to the source code readability).

Wow, great!

Any moderately-full level's going to have enough nodes (about a thousand should be enough) to take minutes to load if those linked lists are ultimately malloc'd, even with a non-debugging memory manager.

This sounds awful. I have lots of experience of lists, trees, etc. on languages with fast garbage collection (no, I don't mean Java nor Lisp) and I construct and destruct (Edit: not destruct, but abandon, destruction is done by garbage collector) millions of nodes per second, I think. One bit overhead per node, too. (Edit: yeah, I've just tested: my program, a complex compiler, consumes around 1MB of memory per second on my puny 2GHz 32bit processor, which is around a half or a third of a million nodes; but during that second many more nodes are constructed and abandoned during very complex computations.)

And if you're using an alternate method to allocate memory for the linked lists -- any "simplicity gains" are a mirage, even if they're wrapped in an external library.

Once I've worked with C code that had a long cycle of nodes preallocated at initialization and used and released from the cyclic pool. I was very error-prone and looked stupid, but now I begin to understand... You are right that such schemes deduct from the simplicity and efficiency of lists.

** tests then become (bitmap[FLAG/32] & (1<<(FLAG%32))), which a competent optimizing compiler should handle as efficiently as the current code.

You mean this is how it looks like after macro-expansion? But even then it looks better than bitfield variables.

Compile-time detecting when logical AND/OR tests can be merged is a bit trickier; we may have to retain the TR macros to support this transparently.

I'd be surprised if it was a bottleneck, but even then we have low-level access to bitmap[], so we can test whole words of it at will or make specialized macros that depend on location of simlar flag groups inside bitmap[].

[zaimoni]

Originally Posted by Bandobras

This sounds awful.

Agreed. I ran into a much worse version of this while working on a chemical engineering project. (Building a 5,300+ node red-black binary tree was taking hours at 1GHz going through MingW32's standard malloc implementation; ~800 nodes was still taking ~10 minutes.

Unfortunately, the red-black binary tree was backing a std::set which I needed the deduplication of keys effect from. This process was moved from the main flygram suite to a moderately configurable data table distiller.)

The data in these nodes was a pair of long doubles -- trivial construction/destruction. I'm pretty sure that less than one second went into everything other than juggling memory management.

Originally Posted by Bandobras

You mean this is how it looks like after macro-expansion? But even then it looks better than bitfield variables.

Yes.

[zaimoni]

Originally Posted by Bandobras

Wow, great!

Takkaria: yes or no on the proposed patch? (Would need to discuss structure on angband-dev if yes. There are some low-level dependencies that need consideration in the design stage.)

[takkaria]

Originally Posted by zaimoni

Takkaria: yes or no on the proposed patch? (Would need to discuss structure on angband-dev if yes. There are some low-level dependencies that need consideration in the design stage.)

I'm willing to see people's ideas, but I'm not committing yet. If you put together an example of the kind of code it'll result in, then I can give you a much better answer.

[zaimoni]

Originally Posted by Bandobras

You mean this is how it looks like after macro-expansion? But even then it looks better than bitfield variables.

Regrettably, that is not a concrete gain I could rationalize sinking the effort for into Zaiband. There are some low-level dependencies (the A_... stat constants and corresponding pval/sustain flags) that are noticeably harder to hand-verify post-change. [Ideally, these dependencies should be checked in debug-builds explicitly by assert(). In C++, make that static asserts so that it won't even compile if gotten wrong.]

The concrete gains I've seen so far in prototype:
* using C_COPY and C_WIPE in a few places from z-virt.h
* iterating over loops controlled by array size to future-proof load/save code and the flag abstraction code in a few places

[Pete Mack]

various people write:

** tests then become (bitmap[FLAG/32] & (1<<(FLAG%32))), which a competent optimizing compiler should handle as efficiently as the current code.

I don't understand what you want to do with this. It (or something vaguely like it) will work to store flag values, but it wont let you do ordinary bit-vector math.

In particular, what happens if you want to match more than one flag variable at a time? You may want to do things logically like:

flags & (RCONF | RCHAOS) /* Example from NPP */

You can't do this if the high order bits and low order bits are dependent the way they would be in your example.

You just can't OR together array indexes to get valid array indexes. They are ordinal numbers, not orthogonal bits.

[Bandobras]

The original example wasn't mine, but I couldn't ignore that one:

Originally Posted by Pete Mack

  flags & (RCONF | RCHAOS)   /* Example from NPP */

This is one of the many bad things induced by bit arithmetic. Such coding is very error prone, especally when you extend your variant. If the 'flags' bitfields is not the one variable (of a few) that holds the RCONF and RCHAOS bits, you have an error that people may not notice for years. If the RCONF and RCHAOS bits land at opposite sides of a bitfiled boundary you can get another error and you cannot write the code in this way at all, so you get irregular code.

This should be written

object_has_flag(o_ptr, F_CONF) && object_has_flag(o_ptr, F_CHAOS)

and you are safe against all errors I mentioned and then some.

Edit: corrected my code

Edit2: the F_CONF, etc. flags can be the same constant for object flags, monster flags, character flags, etc. because there is plenty of space in a bit vector, so no problem with unused bits. The macros can catch inappropriate flags, though, to point out errors, or they can translate the flags to save space in the bit vector, when really needed.

Edit3: actually, bit vectors are not necessary for the high-level approach I proposed above, bietfields would do, with much messier macros, but even the following mid-level code is quite safe:

check_flag(vecor, F_CONF) && check_flag(vector, F_CHAOS)

[zaimoni]

Originally Posted by Pete Mack

I don't understand what you want to do with this. It (or something vaguely like it) will work to store flag values, but it wont let you do ordinary bit-vector math.

I didn't say this was a good idea, merely that it was possible. The experiment of constructing the patch is worthwhile, even though I cannot do a great job because I'm prejudiced against it.

Originally Posted by Pete Mack

In particular, what happens if you want to match more than one flag variable at a time? You may want to do things logically like:

flags & (RCONF | RCHAOS) /* Example from NPP */

You can't do this if the high order bits and low order bits are dependent the way they would be in your example.

In C, you're hung with a high-level interface. Thus, use macro-wrappers so you can do this.

If I was making the final decision for V (I'm not, I merely am willing to provide patching as a code experiment), based on what I'm seeing in prototype I'd stop with:
* sanity-checking the flags for low-level dependencies with assert() [V probably needs this regardless]
* make a policy decision on whether converting to bitvector arrays is worth future-proofing, and C_COPY/C_WIPE applicability.
* making a policy decision on whether the trivial syntax change for newbie bit test/set/update is worth supporting two macro sets. If it is, make a policy decision on whether to use enum or #define for the flag index version. C dictates going from flag index to bitmap. See angband-dev for what the #define changes will look like for object flags in the enum case.
* Design requirement: leave in the bitvector math capability on array elements, as it's moronic to ditch that.

In C++ with Boost libraries, "just" use a template function with two parameters and a conditional type definition to detect whether or not the two flags are in the same bucket. You lose all C++ compilers too archaic for Substitution Failure Is Not An Error (SFINAE). [While not relevant to this construction, a number of Boost libraries do imply not using MSVC++'s ISO-breaking Safe STL.]

[zaimoni]

Originally Posted by Bandobras

This is one of the many bad things induced by bit arithmetic. Such coding is very error prone, especally when you extend your variant. If the 'flags' bitfields is not the one variable (of a few) that holds the RCONF and RCHAOS bits, you have an error that people may not notice for years.

This is what automatic unit tests are for -- in any language, not just C. C just makes it easier by standardizing the NDEBUG flag for omitting them in release builds.

Originally Posted by Bandobras

If the RCONF and RCHAOS bits land at opposite sides of a bitfiled boundary you can get another error and you cannot write the code in this way at all

Wrong. Pete's example works perfectly fine as long as the two flags really are in the same bitfield bucket. (EDIT: Assumed "another error" was accurate. As a bitfield boundary is well-defined only within a bitvector, this is the same error as your first example.)

Originally Posted by Bandobras

The macros can catch inappropriate flags, though, to point out errors, or they can translate the flags to save space in the bit vector, when really needed.

Almost wrong. You need Boost Preprocessor to do this cleanly at all (the *one* Boost library that can live in C), and the standard release version wipes out at 256 (hard-limiting you to a 256-bit bitvector).