You've edited your vmparams/.bat file, right?
Specifically, you need to add the following somewhere in your arguments (not at the end of the line):
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=51324 -Dorg.codehaus.janino.source_debugging.enable=true
The first enables debugging and tells Java what port your debugger should connect on. If you want the game to pause loading until the debugger is connected change suspend=n to suspend=y. That will let you ensure the debugger attachment is working.
The second part of the above line enables debugging for loose scripts, not just jarred code (that part might not be necessary, but I included it just in case debugging isn't enabled in the embedded Janino compiler...).
Once you have that added, you just add your breakpoints in Netbeans (click the line number, a pink shape should appear), compile your code (if it's a jar), launch Starsector, then go back in Netbeans and go to Debug/Attach Debugger. You'd need the following settings for the line I posted above:
Once the debugger is attached and the game hits your breakpoint, the game suspends and you can carefully step through your code in the IDE line-by-line and watch the variables change as it runs. There are special controls for this found under Debug (everything in the section that starts with Step Over). Step Into will enter a method and step through its code, Step Over just executes the current line (without stepping into a method if it's called in that line), Step Out runs the entire current method, Continue runs until the next breakpoint is hit.
If there's anything I missed, let me know. There's also a decent
Netbeans guide if you're interested.