Hi! If you're speaking about UT2004DeathMatch class, than whenever anything goes wrong, it will tear down the whole system (server, observers, bots) + throw an exception at you.

Thus ... smooth execution == no exception + correct result

... anything goes wrong == exception is thrown

Best,
Jimmy

Good to know. I'll probably try it out once I figure out the best way to send custom controllers to the jar for the bot. In the meantime, I have a few questions about how I'm currently implementing things. I'm seeing the following error:

FatalErrorEvent[
    Component:  ComponentBus[UT^2-9@192.168.1.104/61391a07-7839-8649-e458-00ff16f3a913, running=true, queue length=0]
    Message:    Exception happened during the event propagation.
    Cause:      class java.lang.NullPointerException: null (at cz.cuni.amis.pogamut.base3d.worldview.impl.VisionWorldView.objectUpdatedEvent(VisionWorldView.java:145))
    Stacktrace:
        cz.cuni.amis.pogamut.base.component.bus.event.impl.FatalErrorPropagatingEvent.(FatalErrorPropagatingEvent.java:11)
        cz.cuni.amis.pogamut.base.component.bus.ComponentBus.processQueue(ComponentBus.java:688)
        cz.cuni.amis.pogamut.base.component.bus.ComponentBus.event(ComponentBus.java:628)
        cz.cuni.amis.pogamut.base.component.bus.ComponentBus.eventTransactional(ComponentBus.java:654)
        cz.cuni.amis.pogamut.base.component.bus.event.ComponentBusEvents.eventTransactional(ComponentBusEvents.java:86)
        cz.cuni.amis.pogamut.base.component.bus.event.ComponentBusEvents.stoppingTransactional(ComponentBusEvents.java:232)
        cz.cuni.amis.pogamut.base.agent.impl.AbstractAgent.stop(AbstractAgent.java:583)
        pogamut.killers.BotKiller.call(BotKiller.java:38)
        pogamut.killers.BotKiller.call(BotKiller.java:15)
        java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        java.util.concurrent.FutureTask.run(FutureTask.java:138)
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        java.lang.Thread.run(Thread.java:619)
    Caused by: class java.lang.NullPointerException: null (at cz.cuni.amis.pogamut.base3d.worldview.impl.VisionWorldView.objectUpdatedEvent(VisionWorldView.java:145))
        cz.cuni.amis.pogamut.base3d.worldview.impl.VisionWorldView.objectUpdatedEvent(VisionWorldView.java:145)
        cz.cuni.amis.pogamut.base.communication.worldview.impl.EventDrivenWorldView.innerNotify(EventDrivenWorldView.java:124)
        cz.cuni.amis.pogamut.base.communication.worldview.impl.EventDrivenWorldView.notifyImmediately(EventDrivenWorldView.java:238)
        cz.cuni.amis.pogamut.base3d.worldview.impl.BatchAwareWorldView.notifyImmediately(BatchAwareWorldView.java:96)
        cz.cuni.amis.pogamut.ut2004.agent.module.sensomotoric.Raycasting.cleanUp(Raycasting.java:67)
        cz.cuni.amis.pogamut.base.agent.module.AgentModule.stop(AgentModule.java:183)
        cz.cuni.amis.pogamut.base.agent.module.AgentModule$1.stop(AgentModule.java:146)
        cz.cuni.amis.pogamut.base.component.controller.ComponentController.stop(ComponentController.java:1369)
        cz.cuni.amis.pogamut.base.component.controller.ComponentController.stoppingChangedByStoppingEvent(ComponentController.java:1147)
        cz.cuni.amis.pogamut.base.component.controller.ComponentController.stoppingEvent(ComponentController.java:892)
        cz.cuni.amis.pogamut.base.component.controller.ComponentController.access$700(ComponentController.java:122)
        cz.cuni.amis.pogamut.base.component.controller.ComponentController$8.notify(ComponentController.java:220)
        cz.cuni.amis.pogamut.base.component.controller.ComponentController$8.notify(ComponentController.java:216)
        cz.cuni.amis.pogamut.base.component.bus.ComponentBus.notifyListenersC(ComponentBus.java:444)
        cz.cuni.amis.pogamut.base.component.bus.ComponentBus.innerRaiseEvent(ComponentBus.java:549)
        cz.cuni.amis.pogamut.base.component.bus.ComponentBus.processQueue(ComponentBus.java:682)
        cz.cuni.amis.pogamut.base.component.bus.ComponentBus.event(ComponentBus.java:628)
        cz.cuni.amis.pogamut.base.component.bus.ComponentBus.eventTransactional(ComponentBus.java:654)
        cz.cuni.amis.pogamut.base.component.bus.event.ComponentBusEvents.eventTransactional(ComponentBusEvents.java:86)
        cz.cuni.amis.pogamut.base.component.bus.event.ComponentBusEvents.stoppingTransactional(ComponentBusEvents.java:232)
        cz.cuni.amis.pogamut.base.agent.impl.AbstractAgent.stop(AbstractAgent.java:583)
        pogamut.killers.BotKiller.call(BotKiller.java:38)
        pogamut.killers.BotKiller.call(BotKiller.java:15)
        java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        java.util.concurrent.FutureTask.run(FutureTask.java:138)
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        java.lang.Thread.run(Thread.java:619)
]

BotKiller is a class I made that runs in a separate thread. Here is its call method:

public Boolean call() throws Exception {
        System.out.println(b.getName() + ": Killing " + b);
        try {
            b.stop();
        } catch (ComponentCantStopException nostop) {
            System.out.println("Can't stop, then kill: " + b.getName());
            try {
                b.kill();
            } catch (Exception e) {
                System.out.println("Exception within the kill for " + b.getName());
            }
        } catch (NullPointerException npe) {
            System.out.println("NullPointerException on bot kill: " + npe);
        } catch (Exception e) {
            System.out.println("Exception on bot kill: " + e);
        }
        System.out.println(b.getName() + ": Done killing " + b);
        return true;
    }

I want to catch and suppress these errors. What exactly are FatalErrors in Pogamut, and how can I suppress them like Exceptions? Since I only call this when killing a server, I don't care if errors happen on shutdown. Also, I find it strange that even when I get a FatalError with a NullPointerException, the code in my catch (NullPointerException npe) block is never executed (I never see the result of the println).

Caused by: class java.lang.NullPointerException: null (at cz.cuni.amis.pogamut.base3d.worldview.impl.VisionWorldView.objectUpdatedEvent(VisionWorldView.java:145))
        cz.cuni.amis.pogamut.base3d.worldview.impl.VisionWorldView.objectUpdatedEvent(VisionWorldView.java:145)
        cz.cuni.amis.pogamut.base.communication.worldview.impl.EventDrivenWorldView.innerNotify(EventDrivenWorldView.java:124)
        cz.cuni.amis.pogamut.base.communication.worldview.impl.EventDrivenWorldView.notifyImmediately(EventDrivenWorldView.java:238)
        cz.cuni.amis.pogamut.base3d.worldview.impl.BatchAwareWorldView.notifyImmediately(BatchAwareWorldView.java:96)
        cz.cuni.amis.pogamut.ut2004.agent.module.sensomotoric.Raycasting.cleanUp(Raycasting.java:67)

This is most likely to be the culprit. Raycasting module and cleanUp() ... we're doing something experimental there ;-) ... I'll check it out.

Unfortunately, not this weekend as I'm going on a short vacation.

Best,
Jimmy

Regarding FatalErrors.

This is something you CAN'T suppress... it is core mechanism that is notifying the whole agent system (i.e., different threads) that some exception has been raised somewhere == meaning that some part of the agent has screwed up == you can't say what will happen next == the only thing you can do is to kill the whole system.

This is not anything you should be worrying about (I think)... just use "KILL()" if you can't "STOP" it as you're doing and you should be fine. If not, please describe the scenario in greater detail, because this feature is so so omni-present (any part of the code may throw an exception) that it is hard to guess what went wrong...

Best, Jimmy

Regarding the FatalErrors:

I'm wondering what the scope of "the whole agent system" is. Like I said, I think that my code is working fine despite seeing these errors,
but with a multithreaded system it is hard to know. I'm just concerned about the communication going on between Threads. If one bot
experiences a FatalError, then what Threads does it tell about that fact? Will the FatalError interfere with other bots (which themselves consist
of many Threads)? Will they interfere with other servers (even more Threads)?

If bots are aware of which Threads are associated with them, then I guess the FatalErrors have this information as well, and can avoid messing
with anything else. I guess I'm in non-standard territory since I'm running so many Threads/servers/bots in a single execution.

Anyway, enjoy your vacation!

Oh I'm silly I should have described it earlier.

Every agent works for itself so FatalError will bring down only single agent. But I think UT2004BotRunner observes bot for you and tear others down as well if something bad happens.
UT2004DeathMatch definitely does that for your bots in order to ensure the correctness of your app.

Jimmy right from the bus

Ok, some more questions related to server/bot termination. I've discovered that my code definitely has a memory leak,
and maybe also a Thread leak, with the help of the NetBeans profiler. I verified a few things:

If I run my bot by itself against an opponent on an externally launched server, there is no memory/thread leak.

If I run my evolution code, but make it launch only one server with an unlimited time limit (so basically, one server
gets launched with some bots), there is no memory/thread leak.

If I evolve using just one server at a time (no multithreading, but still using the same code that can launch multiple server threads),
with a population size of 1 for several generations, the code leaks both memory and threads.

I'm not sure of the cause, but I have some theories and a little data that I'm hoping will help.
I'm wondering if whenever I stop/kill a server/bot, the resulting FatalErrorEvent is not closing
all the running threads, and therefore not freeing the memory used by objects in those threads.
How do I completely wipe out all objects and threads created by a server/bot? Simply using stop/kill
doesn't seem to work. I've noticed in some of the Pogamut code interesting use of various sync tools,
such as listeners for state changes, countdown latches, and things like that, but before I start
trying to copy this code, I was wondering if anyone could help explain what is necessary and how/why
it works.

Also, although the profiler lets me look at the allocated objects, it crashes whenever I try to use the
"Record stack trace for allocations" option, so I'm not able to find the exact sources of the classes
that are causing the problem. However, from looking at the number, age, and surviving generations
of the different classes, I can say that the following seem to be causing problems:

HashMaps, ArrayLists, Locations, Strings, char arrays and Object arrays. The char arrays are inside the
Strings, and the Object Arrays are inside the ArrayLists. Strings are probably the keys for a lot of the HashMaps,
and Locations are probably what's being looked up. From all of this, I have the feeling that navigation and
path planning information is not being cleared out when the bots die, but maybe I'm jumping to conclusions.
Without the ability to follow the stack traces, I can't know for certain.

On the thread side, RMI connections, sockets, and TCP connections seem to be responsible. These threads
tend to stay alive long after they should be closed.

I can provide more specific profiler data if needed. I would really like to figure this out.

> Ok, some more questions related to server/bot termination. I've discovered that my code definitely has a memory leak,
> and maybe also a Thread leak, with the help of the NetBeans profiler. I verified a few things:

> If I run my bot by itself against an opponent on an externally launched server, there is no memory/thread leak.

Good.

> If I run my evolution code, but make it launch only one server with an unlimited time limit (so basically, one server
> gets launched with some bots), there is no memory/thread leak.

This means using UCCWrapper?

> If I evolve using just one server at a time (no multithreading, but still using the same code that can launch multiple server threads),
> with a population size of 1 for several generations, the code leaks both memory and threads.

You mean code as a whole or yours? Or this could not have been verified?

> I'm not sure of the cause, but I have some theories and a little data that I'm hoping will help.
> I'm wondering if whenever I stop/kill a server/bot, the resulting FatalErrorEvent is not closing
> all the running threads, and therefore not freeing the memory used by objects in those threads.
> How do I completely wipe out all objects and threads created by a server/bot? Simply using stop/kill
> doesn't seem to work. I've noticed in some of the Pogamut code interesting use of various sync tools,
> such as listeners for state changes, countdown latches, and things like that, but before I start
> trying to copy this code, I was wondering if anyone could help explain what is necessary and how/why
> it works.

No, FatalErrorEvent is propagated to every single thread agent thread. I.e., whenever FatalErrorEvent happens
it will gracefully tear down ONE agent, if this is failing -> there is a bug.

Such event might be then sensed by other parts of Pogamut such as
UT2004BotRunner that might then kill all other agents.

The key point here is to discover which threads were not killed. This could be found out from the debugger itself
(no need to use profiler) either in Eclipse or NetBeans... whenever such FatalErrorEvent occurs and your JVM won't terminate
(meaning there are still some threads running), simply pause JVM and examine how many threads it has and what their names
are. You will notice that all threads (that Pogamut Library is spawning) are nicely named, so it will be easy to distinguish between them.

> Also, although the profiler lets me look at the allocated objects, it crashes whenever I try to use the
> "Record stack trace for allocations" option, so I'm not able to find the exact sources of the classes
> that are causing the problem. However, from looking at the number, age, and surviving generations
> of the different classes, I can say that the following seem to be causing problems:

Probably Guice and byte-code weaving are to blame... but I'm not sure.

> HashMaps, ArrayLists, Locations, Strings, char arrays and Object arrays. The char arrays are inside the
> Strings, and the Object Arrays are inside the ArrayLists. Strings are probably the keys for a lot of the HashMaps,
> and Locations are probably what's being looked up. From all of this, I have the feeling that navigation and
> path planning information is not being cleared out when the bots die, but maybe I'm jumping to conclusions.
> Without the ability to follow the stack traces, I can't know for certain.

No HashMap is using String as a key in Pogamut, this is wrong - we always use Token which is "named" String
that has much faster equals() method. When you're using profiler, there is a problem that all objects in the core
consist of many lists / maps / Strings, thus it seems that these objects leak the most, but they are roots of the
mountain of objects that leaked just because some "high-level object" were not gc()ed. Thus you should look for objects
name "XYZModule" or "XYZAgent" or "XYZBot" which will be probably at the end of the list of objects (when sorted
according to number of instances).

> On the thread side, RMI connections, sockets, and TCP connections seem to be responsible. These threads
> tend to stay alive long after they should be closed.

There is JMX problem - to be able to publish agent's JMX interface one need to have demon threads running RMI registry.
If those registry are not shut down properly (via PogamutPlatform.getPlatform().close();) it will run forever.

> I can provide more specific profiler data if needed. I would really like to figure this out.

Try JProfiler for your code, it is paid, but you can obtain 14 days trial license that is very powerful, much more useful
than NetBeans profiler.

=======

The thing we probably need the most right now is the list of threads that were not stopped after some error.

Cheers!
Jimmy

>> If I run my evolution code, but make it launch only one server with an unlimited time limit (so basically, one server
>> gets launched with some bots), there is no memory/thread leak.
>
>This means using UCCWrapper?

Yes. This scenario does not involve any bots or servers shutting down, which
I think is the source of the problem. I should do a test where I use one server,
but I load and kill several bots ...

>> If I evolve using just one server at a time (no multithreading, but still using the same code that can launch multiple server threads),
>> with a population size of 1 for several generations, the code leaks both memory and threads.
>
>You mean code as a whole or yours? Or this could not have been verified?

Can't be verified, but I think the answer might be a little of both. See answers below:

>The key point here is to discover which threads were not killed. This could be found out from the debugger itself
>(no need to use profiler) either in Eclipse or NetBeans... whenever such FatalErrorEvent occurs and your JVM won't terminate
>(meaning there are still some threads running), simply pause JVM and examine how many threads it has and what their names
>are. You will notice that all threads (that Pogamut Library is spawning) are nicely named, so it will be easy to distinguish between them.

I actually did use the profiler, but to look at which Threads were running the whole time. In particular,
with each new server, several threads are created, but some of these are still around when the server and
its bots are killed. Some of these were mine, and I've dealt with them. The threads on your end that are not
going away are threads that start with:

RMI TCP Connection
JMX server connection timeout
ClientNotifForwarder

The fact that these don't go away could be a consequence either of how I'm starting or stopping the servers.

>> HashMaps, ArrayLists, Locations, Strings, char arrays and Object arrays. The char arrays are inside the
>> Strings, and the Object Arrays are inside the ArrayLists. Strings are probably the keys for a lot of the HashMaps,
>> and Locations are probably what's being looked up. From all of this, I have the feeling that navigation and
>> path planning information is not being cleared out when the bots die, but maybe I'm jumping to conclusions.
>> Without the ability to follow the stack traces, I can't know for certain.
>
>No HashMap is using String as a key in Pogamut, this is wrong - we always use Token which is "named" String
>that has much faster equals() method.

Actually, Tokens seem to be leaking too, as are UnrealIds and something called WeakReference. I know that
UnrealId's contain Strings, and I'm guessing Tokens do too, so that explains why Strings are leaking.

>When you're using profiler, there is a problem that all objects in the core
>consist of many lists / maps / Strings, thus it seems that these objects leak the most, but they are roots of the
>mountain of objects that leaked just because some "high-level object" were not gc()ed. Thus you should look for objects
>name "XYZModule" or "XYZAgent" or "XYZBot" which will be probably at the end of the list of objects (when sorted
>according to number of instances).

The most long lived class with "Module" at the end is AgentModule$1, which corresponds to the ComponentControlHelper
within AgentModule. PlayerMaps within Players also looks like it might be staying around too long.

>> On the thread side, RMI connections, sockets, and TCP connections seem to be responsible. These threads
>> tend to stay alive long after they should be closed.
>
>There is JMX problem - to be able to publish agent's JMX interface one need to have demon threads running RMI registry.
>If those registry are not shut down properly (via PogamutPlatform.getPlatform().close();) it will run forever.

As I mentioned above, there are some RMI/JMX threads that are created when a new server is created, and stay around
even after the server is killed. I think these are in addition to the demon threads, which I see at the top of the
profiler timeline: RMI TCP Accept, RMI Scheduler, RMI Reaper.

Are you saying that I should run PogamutPlatform.getPlatform().close() when my code terminates after the final generation,
or should I be running it after the initial settings are loaded and the first server is launched? Should I run it every
time I kill a server?

>> I can provide more specific profiler data if needed. I would really like to figure this out.
>
>Try JProfiler for your code, it is paid, but you can obtain 14 days trial license that is very powerful, much more useful
>than NetBeans profiler.

I will do that.

However, the main reason I'm doing all of this is for the Humanlike Bots/BorPtize competitions, which use a slightly
modified version of Gamebots. Unfortunately, the mod being used for the upcoming competition in June doesn't seem
to work with Pogamut 3.2: There are message parsing errors and, most alarmingly, my bot can't choose to use the
Link Gun/Judging gun. I put the following code in the logic() method:

weaponry.changeWeapon(ItemType.LINK_GUN);
body.getShooting().shoot();

But when I went to the bot, it was firing the Assault Rifle. But all players START with the Link Gun in Botprize, since it
is the judging gun. My 3.1 bot works though, so I'm going to have to go back to that for the competition. Still, I would like
to figure this problem out, but since I'm using 3.1 for a while, I may not be as focused on it.

> I actually did use the profiler, but to look at which Threads were running the whole time. In particular,
> with each new server, several threads are created, but some of these are still around when the server and
> its bots are killed. Some of these were mine, and I've dealt with them. The threads on your end that are not
> going away are threads that start with:
>
> RMI TCP Connection
> JMX server connection timeout
> ClientNotifForwarder

If those are all threads that are hanging in the air, than JMX is to blame... nothing else at all. I'm not expert on JMX implementation
and it certainly worries me a lot that it is so hard to use properly, sadly. Well are you using following code in your Main class?

public static void main(String[] args) {
  // NO CODE HERE
  try{
      ...
      waitTillAllJobsAreDone();
  } finally {
    PogamutPlatform.getPlatform().close()
  }
  // NO CODE HERE -> JVM IS GOING TO BE SHUTDOWN AFTER THIS POINT
}

If you do not close PogamutPlatform, RMI registry won't never ever shutdown.

> Actually, Tokens seem to be leaking too, as are UnrealIds and something called WeakReference. I know that
> UnrealId's contain Strings, and I'm guessing Tokens do too, so that explains why Strings are leaking.

WeakReferences should be gc()ed finally.

Regarding Tokens and UnrealIds, yes you're right, they are leaking - there is Singleton (manager) for them that
is keeping them all, not letting even one of them go...

I could provide you with "clear()" method. But that could have disastrous effects when used in the middle of
Pogamut agent's executions as Maps will stop working correctly resulting in strange errors, crashes, misbehaviour, etc.

>However, the main reason I'm doing all of this is for the Humanlike Bots/BorPtize competitions, which use a slightly
> modified version of Gamebots.

We're sorry but BotPrize is fixing GameBots implementation for long time and we need GB to be flexible as many errors
can be solved by just adjusting them or adding some more attribs to their messages, etc. Thus current version of Pogamut
is not suitable for BotPrize competition. I think that you will have to use older versions of Pogamut. Does version 3.1 work?

======

Bit different side of view, what is the most problematic point about "server termination"? Is it the case that you just can't
make JVM to shutdown? I mean, our usual scenario for large simulations / evolutions / tournaments goes like this:

You create a batch file that will be sequentially executing JVMs and each JVM will

1) instantiate its own UCCWrapper
2) perform one task
3) terminate

Batch file will always wait till previous JVM exits ... thus you won't suffer from leaks / GB2004 bugs related to bots staying in the game
after UT2004Bot is killed, etc.

Best,
Jimmy

Concerning PogamutPlatform.getPlatform().close(), I wasn't using that before, but it looks like this just makes sure
everything is closed at program termination. I'm more concerned about threads leaking during execution and
both slowing down and stopping execution.

Concerning the clear() method for Tokens and UnrealIds: I would like to have a clear() method. I would only
use it between generations of evolution, when no servers are running, so it shouldn't cause any problems.
However, since I'm working in Pogamut 3.1 for the Humanlike bots competition, I should probably just implement
this myself. I have my own checkout of the 3.1 code that I've already modified in small ways. Could you just give
me a general idea where I should put this method and how to implement it? I imagine it would just reset a
bunch of member variables.

I understand that it's not your job to make things work with BotPrize etc. Gamebots is complicated enough as it is.

Your recommended method for running multiple batches sounds like it might save me some headaches,
but with the competition coming up so soon, I'm not sure I want to try writing a bunch of new code right now.
My current code almost works like I want. I will definitely try it in the future though.

Thanks again for all of the help!

By the way, I'm not 100% sure, but I think I have fixed the RMI thread leak problem by
modifying a few files. I put the following code at the end of the reset() method in
AbstractUT2004Server. In order to do this I had to make controller of the connection
have public instead of private access.

if(connection != null){
                    this.connection.controller.manualStop("MANUAL STOP: Server reset()");
                }

I'm not sure whether this has any unwanted side-effects, but it seems to be working ok
for me.

Hmm, if this prevents Pogamut from leaking threads ... very suspicious... but it may explain why Bots sometimes do not get disconnected from the server.

I think that the code you've added won't do any harm since it is in reset() method. But I do not see why this is related to RMI. I'll try to remember it and put it
on my "Pogamut TODO list".

Thanks for reporting!

Cheers,
Jimmy