Fractal Softworks Forum

Please login or register.

Login with username, password and session length

Author Topic: Upgrading JRE, Obfuscator  (Read 2211 times)

boom-mug

  • Ensign
  • *
  • Posts: 6
    • View Profile
Upgrading JRE, Obfuscator
« on: May 17, 2020, 06:48:45 AM »

Would it be possible to use an obfuscator that is JDK9+ compatible (11 is LTS).  I was just testing some of the later openjdk builds and ran into this :

Error: LinkageError occurred while loading main class com.fs.starfarer.StarfarerLauncher
        java.lang.ClassFormatError: Illegal method name "null.new" in class com/fs/starfarer/StarfarerLauncher


I'm not sure if it's alot of trouble or if you have alot of downstream processes connected to it etc...
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 24128
    • View Profile
Re: Upgrading JRE, Obfuscator
« Reply #1 on: May 17, 2020, 08:33:56 AM »

Hi - ah, sorry, unfortunately that's not a good option. Or, rather - it'd involve messing with the build process and testing against JREs that the game doesn't technically support, in any case, so it's not something I could easily do.

(I also suspect this isn't an issue with JDK9 specifically but rather with the OpenJDK implementation of it, but I'm not 100% sure.)
Logged

TJJ

  • Admiral
  • *****
  • Posts: 1905
    • View Profile
Re: Upgrading JRE, Obfuscator
« Reply #2 on: May 18, 2020, 05:32:20 PM »

Yeah, it's a fix for a bug that exists in Oracle VMs.

The JVM spec has never allowed periods in method names (4.2.2. Unqualified Names), but it was a rule that wasn't explicitly enforced by Oracle's VM.
It looks like the OpenJDK is more rigorous in implementing the JVM specification.

IMO such an obfuscation technique isn't a good idea in the 1st place, as it does nothing to hinder deobfuscation yet creates compatibility risks due to the bytecode now being out-of-spec. (as the OP has discovered).
Logged

boom-mug

  • Ensign
  • *
  • Posts: 6
    • View Profile
Re: Upgrading JRE, Obfuscator
« Reply #3 on: May 19, 2020, 02:09:19 PM »

Hi - ah, sorry, unfortunately that's not a good option. Or, rather - it'd involve messing with the build process and testing against JREs that the game doesn't technically support, in any case, so it's not something I could easily do.

(I also suspect this isn't an issue with JDK9 specifically but rather with the OpenJDK implementation of it, but I'm not 100% sure.)

Thanks, it's no problem.

I saw the . and was like wuuuuuh????
Logged

necauqua

  • Ensign
  • *
  • Posts: 4
    • View Profile
    • necauqua.dev
Re: Upgrading JRE, Obfuscator
« Reply #4 on: July 04, 2022, 05:45:34 PM »

Hello!
Sorry for necroposting, but I wrote a script which fixes the field and method names in obfuscated JAR files by replacing dots with underscores.
After running it on *_obf.jar files, Starsector now works completely fine on latest openjdk8 :)

You'll have to run it with Python tho
(Also please consult a developer you know before running random Python scripts from the internet!)

class_format_fixer.py
#!/usr/bin/env python3

# Usage ./class_format_fixer.py <list of jar files>
#
# A (relatively) simple Python script that parses class files in given JARs
# and creates new JARs with dots in field and method names replaced with
# underscores.
# It's needed because some old JVMs (?) allowed dots there despite the JVM spec
# forbidding them - newer (?, maybe Oracle vs OpenJDK thing) JVMs refuse to
# load such classes, which this script does fix.
#
# Main reason of creation is to run a game called Starsector on NixOS :)
#
# Author: Anton Bulakh <[email protected]>

import sys
import struct
import shutil
from zipfile import ZipFile
from io import BytesIO

CONST_SIZES = [0, 0, 0, 4, 4, 8, 8, 2, 2, 4, 4, 4, 4, 0, 0, 3, 2, 4, 4, 2, 2]

UTF_8 = 1
LONG = 5
DOUBLE = 6
NAME_AND_TYPE = 12


def read(format, stream):
    data = struct.unpack_from(format, stream.getbuffer(), stream.tell())
    stream.seek(struct.calcsize(format), 1)
    return data[0] if len(data) == 1 else data


def fix_jar(filename):
    temp = filename + '.temp'
    with ZipFile(temp, 'w') as output, \
         ZipFile(filename) as zip:
        for info in zip.infolist():
            data, fixed = zip.read(info), None
            if info.filename.endswith('.class'):
                fixed = fix_class(data)
            output.writestr(info, fixed or data)
            if fixed:
                print(f'changed {info.filename}')

    shutil.move(temp, filename)


def fix_class(bytecode):
    stream = BytesIO(bytecode)

    magic = read('>Ixxxx', stream)
    if magic != 0xCAFEBABE:
        raise RuntimeError('Bad magic number')

    constant_pool = read_constants(stream)
    stream.seek(2 * read('>xxxxxxH', stream), 1)

    constants_to_fix = []

    # fields and methods
    for _ in range(2):
        for _ in range(read('>H', stream)):
            constants_to_fix.append(read('>xxHxx', stream))
            for _ in range(read('>H', stream)):
                stream.seek(read('>xxI', stream), 1)

    for type, constant in constant_pool:
        if type == 2:
            constants_to_fix.append(constant)

    offsets_to_fix = []

    for constant in constants_to_fix:
        type, constant = constant_pool[constant]
        if type != 1:
            raise RuntimeError(f'Constant #{constant} is not a Utf_info')
        pos, data = constant
        local_offset = data.find(b'.')
        if local_offset != -1:
            offsets_to_fix.append((data, pos + local_offset))

    if len(offsets_to_fix) != 0:
        bytecode = bytearray(bytecode)

        for string, offset in offsets_to_fix:
            if bytecode[offset] == 46:
                bytecode[offset] = 95
                print(f'fixed "{str(string)[2:-1]}"')

        return bytecode


def read_constants(stream):
    constant_pool_count = read('>H', stream)

    constant_pool = [(0, None)]

    while len(constant_pool) < constant_pool_count:
        tag = read('>B', stream)
        if tag > len(CONST_SIZES) or tag == 0 or tag == 2:
            raise RuntimeError(f'Unknown constant tag: {tag}')
        if tag == UTF_8:
            length = read('>H', stream)
            constant_pool.append((1, (stream.tell(), stream.read(length))))
        elif tag == NAME_AND_TYPE:
            constant_pool.append((2, read('>Hxx', stream)))
        else:
            stream.seek(CONST_SIZES[tag], 1)
            if tag in (LONG, DOUBLE):
                constant_pool.append((0, None))
            constant_pool.append((0, None))
    return constant_pool


if __name__ == '__main__':
    for f in sys.argv[1:]:
        fix_jar(f)

[close]

But when running it on jdk >8 we hit methods that were removed from DirectBuffer which isn't fixable without actually reverse engineering stuff and replacing those method calls with something different :/
I mean at the script level I could try just removing them but that'll add a ginant memory leak to the game I think :)

edit: a quickfix for the script, it was missing some dots but it worked on jdk8 - found out about it when trying stuff with jdk11
« Last Edit: July 05, 2022, 06:54:14 PM by necauqua »
Logged

necauqua

  • Ensign
  • *
  • Posts: 4
    • View Profile
    • necauqua.dev
Re: Upgrading JRE, Obfuscator
« Reply #5 on: July 09, 2022, 08:42:06 AM »

Hello!
Sorry for pinging this thread again, but I found out that this should be a non-issue if you update your yGuard version to 2.7.2 where they address this exact issue, or even 2.8 which is under MIT now

I care about this because I'm playing the game on NixOS and there we replace the bundled JDK with the one from nix packages, which is 8 since java7 is eol and not present in any official channels
Had to write this script and have my own package overlay which runs it to fix the jars.
Also, I've seen around the forum that when people upgrade to Java 8 for various reasons they all use the specific version of Java which didn't yet check for dots and say that newer versions are 'unsupported', 'unstable' and so on, while the issue was the obfuscator :)

And then I was just having fun kind of, trying to bring the game to newer Java

My findings on how to play the game on Java versions so far:

Java 8:
Just fix the obfuscation thing, by either upgrading the obfuscator or running my script from above after the fact (obviously not something the official developer would do, but worked for me yay).
Optionally add `-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSConcurrentMTEnabled -XX:+DisableExplicitGC` arguments I've seen on the forums by someone claiming it makes GC much better if you're on Java 8.

Java 11:
1. Remove/replace a call to DirectBuffer.cleaner (in 1 place) - there I believe that the buffer can be just GCd, I replaced that method with a no-op and played the game for an hour and it seemed to work and not OOM (ask me for the bytecode modification Python script I guess if you're not the developer and you need this, but read the third point)
2. Add JAXB api and core libraries in place of the ones removed from Java, change import of IndentingXMLStreamWriter from internal to one present in those libs (in 1 place) - didn't even have to edit bytecode here, just wrote a shim and added it to classpath, heh
3. Well, -Djava.util.Arrays.useLegacyMergeSort=true is not a thing anymore and I got this crash about inconsistent Comparator implementation - didn't bother to fix it myself but this will have to be done.
apparently it's still a thing up to java 17, how did I get that crash once then?. Well if it wasn't a thing I know an easy fix - just use your own mergesort instead of Collections.sort which does timsort that has this issue with inconsistent comparators.

Java 17:
Most things were done on Java 11, now we just add a few --add-opens to the command line (or jar manifests) and remove one VM option that was removed (-XX:+CompilerThreadHintNoPreempt)
--add-opens
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.text=ALL-UNNAMED
--add-opens=java.desktop/java.awt.font=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
[close]

Now there everything seemed to work fine - loading, saving and playing the game (not modded heh) for a while with an exception of those sorting crashes :)

P.S. Found a forum bug: if you insert a shruggie (it has a japanese symbol in it, utf8 issue.?) you can't post bc you get a database error
« Last Edit: July 13, 2022, 08:26:44 PM by necauqua »
Logged

Alex

  • Administrator
  • Admiral
  • *****
  • Posts: 24128
    • View Profile
Re: Upgrading JRE, Obfuscator
« Reply #6 on: July 09, 2022, 09:08:14 AM »

Hi - thank you for all the info! This is good stuff.

(Yeah, the forum has a hard time with non-ascii, unfortunately. I need to look into it at some point!)
Logged

necauqua

  • Ensign
  • *
  • Posts: 4
    • View Profile
    • necauqua.dev
Re: Upgrading JRE, Obfuscator
« Reply #7 on: July 13, 2022, 05:41:05 PM »

Okay, omg, hear read this. So I was writing a javaagent which will fix all the issues, but it somehow didn't help with the incorrect names - I figured that the JVM verifier checks the real bytecode and not one transformed by the agent.

I went into the openjdk sources to look if that's true and look what I've found there - searching for copper found gold: with those options `-XX:+UnlockDiagnosticVMOptions -XX:-BytecodeVerificationRemote` you can disable bytecode verification and the game happily runs on jdk17 (with my other couple of fixes from above), or just with those flags on the latest official java 8 (well, openjdk8 in my case)


But you still will have to update the obfuscator at some point in the future heh

ps: Also the `-XX:+CompilerThreadHintNoPreempt` is a deprecated (had to remove it for java17 above) Solaris-only(!) argument, do you specifically support Solaris? :)
« Last Edit: July 13, 2022, 08:40:47 PM by necauqua »
Logged

necauqua

  • Ensign
  • *
  • Posts: 4
    • View Profile
    • necauqua.dev
Re: Upgrading JRE, Obfuscator
« Reply #8 on: November 14, 2022, 11:22:36 AM »

Another little necropost just so that it does not get lost in the Discord logs - for anyone who's interested I made the javaagent to fix sun.nio.ch.DirectBuffer.cleaner() issue and add the necessary libs for jdk11, as well as instructions on how I got the game to run up to the jdk17, here - https://github.com/necauqua/starsector-fixes

This forum thread is pretty googleable :shrug:
Logged

IGdood

  • Commander
  • ***
  • Posts: 197
    • View Profile
Re: Upgrading JRE, Obfuscator
« Reply #9 on: November 26, 2022, 04:28:19 PM »

I take it this is not as easy as simply renaming a jre folder and replacing a few files like the jre7 to jre8 upgrade?

Logged