Multiplayer Variants Network Manifesto

[Flambard]

First off, apologies to forum moderators, as this thread would mainly concern multiplayer variants devs. I chose this forum because it's a neutral ground, hopefully it will serve us well. Thanks.

TL;DR
=====

We can't unify server codebases, and probably can't unify client codebases
(yet), BUT WHAT WE CAN DO IS: unify the network protocol itself.

Longer version
==============

I think we didn't have a proper talk in 10+ years, so I want to catchup
with you guys. I've been reading PWMA and TomeNET patch notes, and I've
browsing your code, so I *think* I kinda understand what's going on in
your projects.

So, while PW was busy refactoring the code and brining everything up-to-date
with Angband 4+ (still think that's insane amount of work, *_*), and TomeNET
continued to introduce astonishing MMO features to the game, MAngband devs
have been busy... doing what exactly? If you compare 2 last major versions,
there have been no significant game-play changes. Same monsters, same spells,
same mechanics... So what was going on?

We were working on a good common network codebase, that MAngband variants
could use and prosper from. The end-goal is to make the protocol so fluid,
that we remove client incompatibility once and for all.

Has this work been done? No, we still have miles to go. But an important,
first, step has been made: we have removed about 70% of protocol rigidness.

An example is in order. Let's say you're developing a PWMAngband variant,
and decide you want some kind of talk-to-NPC feature. Then, you realize,
that a new network packet is needed. Well, easily enough, you add a PKT_
define, write some Send_dialog_option/Receive_dialog_option functions,
and, *boom*, you're done. Time to realase a new version.

Also, *wham*, you've broken compatibility with your own game, and your
parent variant. With no way of going back. And thus you have another
TomeNET

(And yes, I've noticed the 9000 'if server_version()' checks in tomenet,
so I know there *is* some compatibility within single game, and,
we've recently started doing the same thing in mang (see 'if client_version'),
and it's horrible!)

Now, this situation is completely insane! I don't know of ONE network
protocol, that doesn't support some kind of extension mechanism. You don't
need a different web-browser to connect to different sites (ahem) or
a new IRC client for each IRC server (although their "protocol" is
semi-random with crazy lack of standards). I know we all have good reasons
to make *this* or *that* change to the protocol, and those reasons can
come up unexpectedly, but we SHOULD NOT and CAN NOT shift this burden
to our PLAYERS.

(In fact, I believe we shall drop everything and work on this FIRST,
and only then move on with whatever plans for our games we have in mind.)

So, in my mind, this works like this: every client (mangclient,
tomenet.client, pwmangband-client) can connect to every server
(mangband, tomenet.server, pwmangband). We still develop our clients
separately, and they each have unique features (maybe to be brushed off
over time, cause it surely can be a single program), so players are
free to choose their favourite. And it's OK, if some feature would
only work with a correct combo (i.e. tomenet.client<->tomenet.server to see
*weather*, other clients simply lack this cool feature).

So players would be advised to used specialized clients. But they definitely
should be able to JOIN any server, run around a bit, and decide if this
game is to their liking. (And I'd prefer a unified metaserver too,
but let's cross this bridge when we reach it)

Again, I don't mind separate metas, and variants with hardcoded server
addresses, I'm just saying we're doing a great disservice to our players.
I mean, are we competing? We're basicaly a fork of the same game with 90%
(nowdays, less so) shared code.

Also, a similar thing happened in Angband history, all the different
ports (DOS, OS/2, Win, UNIX variants) were separate forks, to be then
merged/unified under common protocol (Z-Term). Now, admittably Z-Term
is whack, and our goal is somewhat harder, but it's definitely doable.

I mean, every other network protocol managed to do it. The dream goal
is to publish a "final" version of the standard, so that new games,
in new languages, can continue to be developed in this ecosystem. That's
it.

So, in a nutshell, the proposal is:
- generalize our primitives, and use that, instead of being overly-
specialized.
- add "infinite" extension mechanism so we can add any new network
packets, and clients could support them at will.

What has been done in MAng:

- PKT_STRUCT_INFO packet (and schema) for tranferring structs such as player
class. I've seen PWMA uses it (THANK YOU!!!), but the structs themselves
are incompatible (mainly due to V), so we'll have to re-think those.
Ideas welcome!
- PKT_STREAM packet to stream cave data (line/char) in a unified way.
Clients can subscribe/unsubscribe to streams, negotiate their dimensions,
pick RLE methods, etc.
- PKT_INDICATOR packet which replaces all the PKT_FOOD, PKT_FEAR, etc,
in a generalized way.
- PKT_COMMAND packet which replaces all the PKT_QUAFF, PKT_EAT, etc,
in a generalized way.

So this covers like 70% of our basic needs, but there's still lots
of hardcoded stuff, like shopping, etc.

On the part of the extension mechanism, there's been NO CODE written
yet, but I think it should be something simple, like:

- At connect time, server sends a number of

PKT_EXTENSION "weather" "Some info"

Clients have a map for those, so they known precisely what they support.

So they send

PKT_EXTENSION "weather" "Yes-please-turn-it-on"

From then on, Server starts sending PKT_WEATHER packets. Client asked
for those itself, so it should be able to parse them properly and
not choke.

In addition, let's say client wants to SEND some PKT_WEATHER packets
(ill example). Again, since this extension has been announced as supported,
it can safely send it it.

At this step, server authors can differentiate between variants
and versions (say, PKT_INVEN is different in MAng150 and MAng153,
this mechanism can safely express this). So I would probably name
the new inven packet something like "mang153-inven". Or maybe,
I'm doing some tomenet hacking, so I'd simple add a "tomenet-weather"
extension.

HOWEVER, at some point in the future, we can consolidate those into
common groups, and then (if we want to), move it from extensions
territory to PROTOCOL PROPER (and yes, we'll still have to break
protocol from time to time, but we should *always* stride to
make it more generic, while doing it, not more specific).

(I will experiment with this approach, comments welcome).

What else I have in mind:

- We're not developing an ARCADE server-compatible protocol. We are
developing a network protocol to play roguelike games of the Moria
family. That means range of 'q' to quaff commands, Which Item (a-e)
inventory management, inscriptions, macros, etc. If you think about
it from this perspective, it should be extremely doable.

- PWMAngband, and TomeNET devs: *PLEASE* drop your old netcode
and adopt the new one from mang. It will a) make your games
legal from FOSS/LICENSE stand point, b) clean up 12 layers
of UDP-to-TCP mess, c) remove silly bugs like NET_VERIFY FAILED
d) add some useful patterns so the logic is not all over the place.

I don't ask to adopt our network protocol (although, I would really
appreciate it, like I said, from handshake to actual gameplay,
it has been designed with extensibility in mind), I'm just talking
about the base netcode and sockbuf i/o.

(Also, if you want to re-license it, or something, just ask, it's
all doable).

- From my end, I'm *really interested in collaboration*, don't want
to impose any design decisions on you guys (some of mang150 ideas were
good, some weren't), and am willing to add PKT_ definitions to mang,
which it doesn't need at all.

- For example, I'm going to work on the accounts packets (designed
after PWMA or TomeNET, or maybe can design something better?) and add
those to MAngband, although we are not planning on adding "Accounts"
as a feature *ever*.

- If we go through with this, at some point, we all will have to
release a client/server pair that *completely* breaks protocol
compatibility within our own games. Yup, every single player
in each community will have to upgrade. BUT, from then on,
it's paradise

OK, I think this was long enough, if you are not convinced by now..
Please contact me (and each other), let's do it!!!

Contact me right here, on Discord https://discordapp.com/invite/U5B65Rh, or name a place and I shall be summoned.

[tangar]

*sign*

It's time to unity our strength, to overcome our differences and to give a new breath to the genre which we all love so much!