PreviousNext…

The wiki 16 Jan 2005

As many of you know, the wiki has died. I suspect massive memory leaks courtesy of LS2J. Of course, it’s pretty tricky to troubleshoot issues like this on a server which isn’t yours! Occasionally a page will load, but usually the rendering fails, and all the user is left with is the subject of the page in question.

I make use of a Web Query Open agent to create a new WikiPage object in a custom Lotusscript library. This class does all the heavy lifting, including referencing some ORO regexp code in Julian’s excellent script library (see his LS2J examples database). This ran happily for three days before dying yesterday afternoon.

My suspicions of a leak are bolstered by the fact that I get this error message when I try to trigger a test agent (this time pure Java) on the server:

HTTP Web Server: Lotus Notes Exception - JVM: Attempt to retrieve Java agent attachments failed.

I understand that this error can mean that your JVM’s heap size is trashed, which would make sense. The agent logs that Domino Developer Network make available to its users aren’t showing anything untoward, and debug statements in the code are showing no issues… so there we are. Flummoxing. I’ve logged a support call, so we shall see.

In the meantime, my apologies to uses of the Lotusphere wiki. Hopefully normal service will resume soon. You can edit your content as much as you like. You just can’t see it (!)

Filed under

Comments

  1. Only piece of advice I can offer is to ALWAYS recycle notesDocument objects in Java code - the latent garbage collector never seems to get rid of them. But then again, everyone knows about this, so I'd be surprised if this was the case.

    Perhaps it crashed when I tried to extend the "Party" page with my list ? :-)

    Bummer.

    ---* BillWild Bill#
  2. Since it's not your server you may not be able to do this, but you could try referencing the ORO JAR file from the hard drive instead of as an "attachment" to the script library. Either put it in the jvm\lib\ext directory, or put it anywhere and add it to the JAVAUSERCLASSES in the Notes.ini file. Then remove it from the script library so it's being read from the hard drive.

    Just something you could try, if you're allowed. Maybe the problem is just referencing the JAR file as an included/attached file in the script library. I know that sort of thing is supposed to work, but maybe it's buggy on your version of Domino.

    - Julian
    Julian Robichaux#
  3. Dan at DDN kindly rebooted the server, and the problem is resolved, although of course this may well be temporary given my LS2J coding efforts: I must have mullered the JVM.

    I ensure that there’s code to release object references and the like, but the help is rather scant on exactly what needs to be done in this regard with LS2J (as Bill points out, Domino Java needs heaps of recycling (pun intended, sorry), but what about Lotusscript wrappers?)

    I may have to check out m’learned colleague’s custom class for LS2J shenanigans. You can see it here:

    http://www.swapcode.com/domdoc/page/lscode012.htmlBen Poole#
  4. Hi,

    I've heard the same LS2J issues from Manfred Dillman in development effort of his rss reader.:
    [quote]
    Version 1.2 | 2004-07-05

    * Completely replaced the Java code with LotusScript (Win API calls). There were memory leaks on the agent manager when running the agent on a Domino Server which reappeared with Domino version 6.5.2.
    [/quote]
    (from: http://www.madicon.de/content/view/39/52/)

    If I remember right, he even talked with IBM support in Germany and got no solution. But not sure.
    I am highly interested in this issue.
    Until now I suspect that the memory leak is buried deep down in closedSource LS2J code, which I don't find funny.

    Sun (and other Java stakeholders) should start to think about charging Lotus for image loss of Java. In a german notes forum a lot of people seemed to believe that "java has lots of memory leak problems". I had tried my best to convince them that it was Lotus with LS2J.

    Axel
    Axel Janssen#
  5. Ben. Thank you for the link.
    Haven't seen it. This looks like a good workaround. Creates extra burden on the developer, but might make the whole thing work more than 3 days.
    But why the hell this is not documented on Lotus/IBM site?!?
    Similar gotchas with Workplace if the thing will ever arouse more interest as the 0 interest it has now and they'll get killed on http://www.javablogs.com

    AxelAxel Janssen#
  6. Interesting, thanks Axel. I must admit, third party talk of leaks and so on always had me shying away from LS2J, which is why I’ve only started to use it now.

    To be honest, it’s trivial to re-code the agent to work as a pure Java process, but I like how it looks at the moment, so I shall wait and see. I could really do with some good profiling tools on a server of my own at the moment! :-)

    Definitely agree about documentation though: Lotus need to be a lot clearer in this regard, as they do with what counts as “restricted” and “unrestricted” in terms of Java agent security: a very early incarnation of OpenWiki used the gnu regexp classes, but I had to pull all of that and use a Lotusscript parser because the Java code only ran in unrestricted mode (that in itself took a lot of testing to prove, as the Domino box doesn’t give many clues here).Ben Poole#
  7. I've revisited LS2J a few times - everytime I have found too many problems with memory leaks etc - even when you are careful to recycle everything - so my conclusion is always the same - all java or all lotusscript.

    It still annoys me you have to watch every single domino object you create and recycle it - the viewentry one is the one I always see in "the field" which causes memory issues - having to recycle the document part even if you have never referenced it - where would most people pick up on that?Steve Castledine#
  8. LOL! I spent HOURS troubleshooting that exact error this weekend thinking it was my app had suddenly died after being stable for a year. All the time it was the JVM was hozed. At least it was for a good cause. I was getting paranoid my template was corrupted or something… even asked Jake to send me the jar file again. heheh…. glad I can laugh at it. Good luck sorting it, mates!

    JerryJerry Carter#
  9. The recycle() is not that big issue. Ted Neward writes in his very good "Effective Enterprise Java" that the Java Garbage Collector is good at freeing memory of Java Object, but not good at freeing memory of some C/C++-stuff, wrapped by a Java Object. You have to explicitedly free memory with:
    - SWT (uses native widgets): dispose()
    - JDBC: close() does that, but its the same if you don't use pure Java JDBC-Driver (type 1, 2 and 3 me thinks).
    - Presumably JNI
    The problem ist that a lot of people don't know so it should have been better documented. Axel Janssen#
  10. Sorry, I don’t follow what you’re saying here. The Domino Java objects are wrappered C code, so they do need explicit recycling, as we’ve discussed. This isn’t brilliantly documented in Domino Designer Help, but you can find this stuff out and address it. It has to be done.

    When it comes to LS2J however, the developer is in trouble: the leaks happen in a place that we can’t touch. Whilst my wiki code now frees up all relevant objects (JavaObject, JavaSession, and so on), it still leaks.

    So, LS2J. Good idea, iffy implementation: often the story of Notes’ life!Ben Poole#
  11. I have had the same problem with 2 java agents. More work-a-rounds are tried without any luck. Among those I have tried are:
    - recycling
    - setting parameters in java.policy under jvm\lib\security
    - setting parameters in java.security under jvm\lib\security
    - placing jar files under Domino folder jvm\lib\ext instead of attaching them to the java agent
    - downgrading the versions of the jar files used in agent (commons-net and axis)
    - restarting server to clean up any 'dirt' in JVM

    My last go was to start the agent via LEI. Goal ! The agent started and ran carelessly. Checking the ACL revealed that the server was not in the ACL. (LEI is setup up on another server)

    My advice is check the ACL, server must be stated. For agents running as a web service check the 'Run on behalf of' plus the 'Set run time security level' properties on agent, too. Insert the server name in 'Run on behalf of' and set 'Security level' to 3. I my case the headage disappeared.
    Kim Petersen#

Comments on this post are now closed.

About

I’m a developer / general IT wrangler, specialising in web apps, the mobile web, enterprise Java and the odd Domino system.

Best described as a simpleton, but kindly. Read more…