I tried to clarify my thoughts a bit and put them at http://plaza.v-wave.com/kestrel/myidea.html.
Still needs some work but might make a little more sense.
Looking forward to your comments ... :)
First, let me state for the record that I'm no C/C++ expert. Second, let me state for the record that I am a pessimist. Third, let me state for the record that I've seen QuakeC and written a mod in it. From what I can tell of the source you jotted down on that page, it looks like a near-direct translation of QuakeC code. Horrid. Fourth, let me stop enumerating things for the record.
I did in fact read your page entirely and thought "Ain't gonna happen, SteQve" most of the way through the whole thing.
I think if you stare at the wall, let your eyes glaze over for a bit, and think about what is realistically attainable, you'll come to the conclusion that grafting code is grafting code. If you do it line by line and create a new single thing from 2 sources, you at least were forced to realize everything going on in both sources. If you kinda sorta graft certain blocks here and there you probably end up with code that might as well be renamed TimeBomb.
I didn't follow some of the point you were making about function pointers (I imagine if you were sitting in front of me saying the same thing the web page says, it would make sense to me). A lot of what you wrote down contained too many ideas per paragraph to digest well (for me). That may well be the result of wanting to jot notes down and run like mad hoping someone would take the lead and work on it.
I can't think of any project I've ever seen do what you want to do. I mean...break it down. Quake II mods are full-blown programs in disguise. The fact that Quake II mods are part of a game we play for fun does not in anyway diminish the possible complexity. I think the notion of source reuse at this level is more enticing because mod authors don't want to spend Lots of Real Time on Quake II mods -- "Hey, this mod was supposed to be a fun little thing that served a specific purpose. This is getting hard. Where is all my free time going? This should be easier because this is just for a game I play. Dammit!"
A library of code that mod authors use? Quite possibly.
A new cool way to weave random things together into a cohesive running program? I just don't see it. It failed pretty solidly when they tried to do it with QuakeC (I forget the name of the project). Did I mention I'm a pessimist?
Having said all of that, I hope I'm wrong and it all irons out.
Hey Steve, I appeciate your candor in your post (http://www1.shore.net/~steqve/qm/standards.html) about Mod Merging (or whatever you want to call it) and you obviously have a good sense of homour (humor, sorry); not because of your technical proposals, but simply because of the comments in the article.
I would have thought the better solution to the merging problem would be addressed to some extent by an object oriented approach, using virtual C++ functions; i.e.:
class MyMod1 : public Quake2Mod
{
public:
virtual void T_Damgage(a, b, c);
.
.
.
};
class MyMod2 : public MyMod2
{
public:
virtual void T_Damage(a, b, c);
.
.
.
};
and in the implementation of MyMod2::T_Damage you would call the base class
before doing any mod work:
void MyMod2::T_Damage(a, b, c)
{
MyMod1::T_Damage(a,b,c);
< mod code here >
}
This would probably work for some mods, but you may have to resort to
re-implementing the base class function if it does something the
derived-class mod function doesn't want doing. Anyway, at least C++ gives
you a structure in which to implement these functions but there are still
the following problems:
1) The C source would need to be reimplemented/designed in C++ (a losing battle given the bugfixes that are/will be coming out).
2) You need to change the API slightly (haven't looked at it yet) so that it calls through the virtual function (in our example this is T_Damage, so Quake2Mod->T_Damage(a,b,c) would need to be called). This would require additional abitration code at the API level of the mod.
3) Most mod authors won't release their source code, due to the many lame mod coders out there who are more than willing to plagerize it. This is a fundemental problem -- how many mods for Quake include source ? (This is not a retorical question; I don't know the answer, but would imagine there are few).
An alternative solution to all this is to use do the mod merge manually using the traditional UNIX tools of "diff" and "patch" (patch is Larry Wall's legiondary program that can read context diff's and apply the changes). This would at least help the mod merger get most of the code into the combined source tree, but there is still potentially serious coding that needs doing. At least with this method the mod author is not required to "buy into" third-party schemes, which, I think, there would be a problem selling to most.
All the best mate, Andy.
I read you're article on Mod standardization... this is a pretty standard problem with modular systems, and one that has really good solutions and really bad solutions.
DLL's are bad in terms of portability, but great for adding on modules at runtime. I've done some work with Apache (Web server) under NT and it's a hell of alot better than expecting my clients to recompile the web server on their machine (no matter how much fun they think it is).
And as far as other languages go, it would be a BAD idea to use Java or Lisp, for precisely the reasons you cited. (Although Java isn't really an "interpreted" language). Bottom line is, Carmack and crew have obviously spent alot of effort tuning the hell out of Q2, and throwing anything in the loop that's not lean/fast C/C++ code will slow down the whole parade.
Your basic solution is a good one and there are techniques (like continuation-passing) that will allow you to do the equivalent of sub-procedural blending between mods, without having to carve up event functionality into little pieces. One can establish a policy framework where modules can state what kind of combination they expect to be a part of, and the standard event processing code can then pick the appropriate handler(s). In some cases, a mod might get left out in the cold if it's antisocial enough to expect no-blending, but not high enough on the priority list to get chosen.
The coding issues aren't all that hard, but I agree with your bottom line, specifically in terms of the policy of choosing handlers, it's not an easy problem to solve. But it could be fun.
-david
let me add some more comments. i am unsure as to how quake ii loads the dll's but if it does it as other c programs do then if you have two dll's that contain the same function name then only one is going to get called. more then likely it will be the last one loaded. in a worse case the program one not know which to call and then would crash. i guess i will have to write a small app to test the therory out, but i do agree that mixing mods will be an amost insurmountable problem.
Well, I for one read your article on mod standardization and I agree with it for the most part, although I don't think it would be as difficult as you seem to. What scares me is the prospect of having to merge the changes *back* into newer versions of gamex86.dll released by id.
The idea you had of having a list of function pointers is perfectly valid and quite easy. The call semantics are a bit different than your code:
while (method) {
damage = (*method)(attacker, inflictor, victim) * damage;
etc()
}
Personally, I would probably go the lazy route and keep an array of function
pointers. This would have a maximum number of mods loaded at one time, but
who's going to load 20 mods anyway? It's easier to maintain an array than a
linked list, and all of the function pointer arrays could be stuck in a big
"mods" array, which makes everything more concise.
We'd also need an ability for a mod to return "this is my value", "I'm not returning any value", and "this is my value and I don't want you to call any other mods". That kind of thing.
It's an interesting problem, anyway. Perhaps I'll tackle it. I don't feel that the Java and OOP idea that someone else came up with is feasible, since it's just too ambitious for no real gain. Object-Oriented programming is very useful for larger systems, but at its core it's little more than a very good encapsulation system. Just like C modules and naming conventions.
I'm babbling a bit, but the reason I actually wrote you was to say I really like the idea of cutting the granularity of functions down. That'll make it much easier to avoid conflicts.
I have been reading all of the various rants on mod standardization. The ability to standardize the interfaces in and interaction between any number of mod DLLs is a very challenging issue, but one of great importance. Just imagine the quantity and quality of mods available if they could be broken up into individual plugins that each do a certain job and interact in a standard way. People could mix and match mods to create a very unique game experience, both offline and online.
A lot of people comment on how it hasn't been done before, and that we would have no idea where to start, that it wouldn't work, and that it would be then given up on. A lot of people have commented on how complex it would be to create such a system, and it is, but it has been done.
We must look to the industry for our inspiration (and to save us headaches while trying to design this system) We can find three levels of plugins in use in the industry.
1. Mutually exclusive plugins - Quake, Quake2, Photoshop, Sound Forge - many plugins can be installed, but only one can be running at any one time 2. Simultaneous programmatically configured plugins - Microsoft ActiveMovie is a good example of this - any number of plugins supporting the standard interface can be installed on the system, but only an applications developer can snap them all together to define the inputs, the processing chain in the middle, and the output 3. Simultaneous user-configured plugins - 3D Studio Max - this is the ultimate in plugin architecture and multithreading
We might as well ignore the first two levels and concentrate on Level 3. I think that a lot of people are thinking that we have to create a Level, where the plugins themselves magically know how to work with every other plugin in the world. That's where the impossible part comes in. However, Level 3 is very possible and is already finding great success in 3D Studio Max.
In case you don't know about Max, it is a 3D modeling and animation system that was developed from the ground up for Windows NT starting as soon as NT 3.1 came out. The software was released when NT3.51 was still current, and the shell preview was available for it, though they didn't recommend you be running it. The software not only supports, but is entirely based on it's plugin architecture. It's really quite amazing.
The plugin architecture defines a standard set of object types - primitives (cube, cylinder, teapot, etc.), object modifiers (bend, twist, squash, etc.), world space modifiers (wind, gravity, etc.), helpers, animation systems, atmospheric effects, renderers, etc. Each of those object types exposes a standard interface which the plugin developer must support. The system design defines how each object type interacts with each other object type, and each plugin is a black box into which data is entered and data returns. Each plugin can expose properties via integration with the Max user interface, and the system allows these properties to be modified over time (to create animation) and to be serialized to disk for storage and retrieval.
The definition of these object types (classes) and how each interacts (the interface) is where the true complexity lies. If that task is done correctly, creating the plugins is quite simple.
The catch is this - we also must design the mechanism by which the user connects these plugins together. Unlike Max, where the user is always interactively working with these objects, in the Quake world we want the user to be able to use some simple setup software or script to select, configure, and connect these various plugins. This is where we will have to be the most creative, as even the mighty Max SDK doesn't really cover this issue (except perhaps where it discusses serialization)
Get your hands on any and all documentation you can find on the 3D Studio Max Plugin SDK. It doesn't really matter whether it's R1 or R2 information, as the architecture is still the same (R2 is mostly R1 with tons of plugins bundled with it :-) ). Get your hands on Max itself and see how these things can transparently interact, think of how a Quake2 user could configure and connect their plugins (or how someone could script it for others to use). I'm sure lots of people you know have a copy of Max that you can try out. I might be the only one with the dongle, but that's another issue. Check http://www.ktx.com for any information they might have. Download their SDK documentation and have a read. E-mail Steve or myself (jeremyg@sniper.net) or the mailing list once it's up and running to further discuss this important issue. I sure hope Carmack looks at this kind of system for Trinity, so even if we're not totally successful developing such a system for Q2, perhaps we can convince him to incorporate it into id's future products. Either way, it will be a fun and challenging effort that I know I'm looking forward to.
Jeremy Gray jeremyg@sniper.net Sniper Network Gaming Group http://www.sniper.net You bring the PC, we'll supply the competition
if quake2 was totally rewritten as an OO prog, it would be more feasible. With c++ polymorphism, virtual functions etc.. it would be more possible. but, anyway....baby steps right... Mod suggestion Weapon DLL. the code for a new weapon is in a single external DLL, the mod loads this DLL. The major change to the base code would be to reorganize the item list to allow new weapons. could be done with a dynamtic allocation scheme (even under pure ANSI C). if you could do that (dynamically add and subtract from the item list) then it would be simple to dynamically add weapons.
Yes, I read your Quake Mod article and yes I thought it was very interesting. I just wouldn't think it would be that difficult to combine multiple mods together, but who knew.
After reading your article, I got to thinking (which is dangerous). Since it appears you are of the C/C++ programming type, I thought I would bounce this off of you. Are you familiar with the COM/ActiveX architecture that Microsoft is currently pushing as "the future of programming"? I know, you think Microsoft and immediately all these bad notions of cross-platform incompatibility, bloated DLLs, etc., come to mind, but I think this is different. COM (the Component Object Model) is a binary cross-platform standard that can be thought of as a method of dynamically linking and invoking objects. The best part is, COM is language independent (though it lends itself nicely to OO languages like C++ and Java). I'll try to keep this as simple as possible, since it's quite a lot to swallow.
What COM allows you to do is group sets of functions into interfaces. So, you might have a Damage interface, which contains functions for calculating and inflicting damage, another might be a Weapons interface, which handles all the aspects of drawing, firing, etc. of a weapon. This in and of itself is no big deal. However, the advantage to this is that one mod doesn't need to know anything about the other COM objects on the system or what interfaces they contain. Mod X can actually ask Mod Y whether it supports any Damage interfaces, and if so, it can handle the multiple mod situation appropriately. If no other objects are running, or there are no other interfaces for which it is concerned, the mod operates as it normally would. This might be a lot easier than trying to walk a (potentially long) linked list of pointers to functions and hope to call those functions correctly.
So a mod would be a server for these COM objects (by this I mean that the mod would be responsible for instantiating the object and initializing it appropriately). There would be some kind of central "mod controller" that could arbitrate any conflicts, perhaps by letting one object be responsible for the final damage calculation, etc. There might be a user interface which allows the user to configure how these objects should interact, right down to enabling or disabling specific functionality in a specific mod.
I'm not sure how much of this is just techno-babble and how much actually makes sense. I, unfortunately, have just started learning about programming my own mods, so I'm not yet familiar with all its intricacies. It just seems to me that the COM/ActiveX (which are pretty much the same, incidentally) architecture could go a long way in resolving this problem.
Just a thought, I certainly don't expect there to be any action on this issue (by anyone) in the near future.
If you haven't already, take a look at http://www.qhunter.com/Q2MMDLL.html -- it's a bit stale, but then I just moved and I'm still unpacking. Your recommended approach and mine are, basically, the same. I hadn't got down to the nitty-gritty, but, it seemed that it was going to be necessary to partition the Q2 API even further as you suggest.
Anyway, take a look at my stuff and tell me what you think.
Peace.
-- sky G.
I enjoyed reading your article. Here's my two cent's worth:
(Disclaimer: I do almost no programming, but I manage real programmers, so my input may be worth less than two cents.)
1. Unless id itself rewrites Quake II to support multiple mods, you will probably never get a majority consensus. The best you can hope for is that the authors whose mods you like agree to a standard.
2. As a first step, how about a standard that only allows mods to declare what code they change. If two standard-following mods don't declare the same sets of code, they should both work at the same time.
3. As a second step, have a standard that allows mods to declare what code that they MUST change, and what code that they'd LIKE to change. If damage mods is a small part of your CTF mod, your mod could agree to let the weapons mod grab that code. If two or more mods can't agree, they'd notify the user what code was in conflict.
Mod standards. Yeah, well, it's a toughie. But I may have coincidentally come up with a simple, elegantish solution in my efforts to create a modular set of mods (no pun) that I can switch in and out from the console. When I want to modify (for example) PutClientInServer, I add one line only to it, either at the beginning (prefix) or at the end (postfix):
if (JOJO_PutClientInServer(ent)) return;
and extern out JOJO_PutClientInServer(edict_t *) to a header file of its own. That way, it can override PutClientInServer completely (if it's prefix) by returning 1, or just adding to it by returning 0 or being postfix. If my man Hook wants to add his own stuff, he writes a function HOOK_PutClientInServer(edict_t *) and stick that in before mine, etc etc. Is this painless? Well, since nobody listens to me, all I can guarantee is that someone else wanting to add one of my patches to theirs only has to write two lines of code (there's a #include too :). And really, that's all we can ask of each other. You're right in that programmers are never going to stick to standards (just look at what Cash et al. have done to the C syntax with the Q2 API for crying out loud! Not that I dislike it, mind :) but we can at least try to make it easy for each other, even if we expect no real return investment. I want my code to be as widely used as possible; I don't really care what the context is. Thus, standards aside, I want it to be as easy as possible for others to use and this seems the best way to me.
Write me back and tell me what you think; I think I feel an article of my own coming on...
Date: Sun, 11 Jan 1998 11:00:05 +1100
Perhaps we are aiming too high with mod standardisation. Maybe for most purposes all we need is a standard plugin interface for things such as, weapons, monsters and bots. This would be much easier to implement and for most novice users that want to 'plug' in a bot or weapon this would be a dream. The more large mods would of course not be possible, but that might not matter.
Karl