Skip to content

Sidra's integration runtimes using OpenJDK

This page intends to clarify some concepts about integration runtimes and give a quick guide about which cost-free runtime environment can be used and how.

Java Runtime is needed

The target physical format for the data ingested with Sidra is Parquet files, being optimal for Databricks. The IR agent does not have the native ability to (de)serialize to Parquet; for these, Java libraries are needed. And the IR agent installation kit does not include a Java Runtime. Hence, a Java Runtime must be manually installed on the node, after installing the IR agent.

This can be easily done using the Java installer, which sets everything necessary.

Mind the licensing

Recently, Oracle changed the licensing terms for the Java Runtime Environment (JRE): for production environment, fees may apply. The usage of the JRE may not be free-of-charge.

Fortunately, Oracle is contributing to an open-source version - OpenJDK - that can be used. See the install instructions.

Microsoft, with its commitment to protect its customers from licensing claims, is also publishing their own build of the open-source OpenJDK, with an .MSI installer too: The Microsoft Build of OpenJDK.

PATH and Registry keys

Unfortunately, the OpenJDK versions above, including the Microsoft's OpenJDK installer -commitment to respecting licenses and disimbursment- don't set the Registry keys that the Integration Runtime agent needs for calling the JRE when processing for Parquet files.

According to the troubleshooting article here, the IR agent:

  1. Needs to be same bitness (64-bit) as the JRE, hence, both will be installed under C:\Program Files\ (not C:\Program Files (x86)\).
  2. Checks for the installed version of JRE under HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment, value CurrentVersion.
  3. Retrieves location of the JRE from HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment\<versionNumber>, value JavaHome.
  4. Locates the bin\server folder in the path retrieved.
  5. Loads the Java Virtual Machine library, jvm.dll; if it is not present, the bin\client is probed for the same DLL.

In addition, for it, is recommendable to set the environment variables for the JRE too:

  • The variable JAVA_HOME should point to the installation folder; for example:
    C:\Program Files\Microsoft\jdk-17.0.3.7-hotspot\
  • The variable PATH should includes the path to the binaries of JRE; for example:
    C:\Program Files\Microsoft\jdk-17.0.3.7-hotspot\bin

Example

On installing the Oracle's Java 8, the installer is adding Registry keys under:
HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft

Exporting that key, we end up with a text file bearing the .REG extension, looking simmilar to the below. Then, Oracle's Java 8 may be uninstalled.

The default Java location when we install using the Microsoft's build of OpenJDK, on Windows, is:
C:\Program Files\Microsoft\jdk-17.0.3.7-hotspot\

After adapting the Registry file - the corresponding keys - updating the version and location of the JRE and disregarding the browser plugin entries:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft]

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Plug-in]

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Plug-in\11.333.2]
"JavaHome"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot"
"UseJava2IExplorer"=dword:00000001
"UseNewJavaPlugin"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment]
"CurrentVersion"="1.17"
"BrowserJavaVersion"="11.333.2"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment\1.17]
"RuntimeLib"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin\\server\\jvm.dll"
"JavaHome"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot"
"MicroVersion"="0"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment\1.17.0.3.7-hotspot]
"JavaHome"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot"
"MicroVersion"="0"
"RuntimeLib"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin\\server\\jvm.dll"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment\1.17.0.3.7-hotspot\MSI]
"INSTALLDIR"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\"
"JU"=""
"OEMUPDATE"=""
"FROMVERSION"="NA"
"FROMVERSIONFULL"=""
"PRODUCTVERSION"="17.0.3.7-hotspot"
"EULA"=""
"JAVAUPDATE"="1"
"AUTOUPDATECHECK"="1"
"AUTOUPDATEDELAY"=""
"FullVersion"="17.0.3.7-hotspot"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Update]

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Update\Policy]
"Country"="ES"
"PostStatusUrl"=https://sjremetrics.java.com/b/ss//6

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start]
"CurrentVersion"="11.333.2"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start\1.0.1]
"Home"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start\1.0.1_02]
"Home"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start\1.0.1_03]
"Home"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start\1.0.1_04]
"Home"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start\1.2]
"Home"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start\1.2.0_01]
"Home"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start\11.333.2]
"Home"="C:\\Program Files\\Microsoft\\jdk-17.0.3.7-hotspot\\bin"

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start Caps]

[HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Web Start Caps\11.333.2]
"JNLPProtocol"=dword:00000001
"JNLPAssociation2"=dword:00000001

Visual C++ runtime

The Java Virtual Machine of the JRE - <jre-path>\bin\server\jvm.dll - takes a dependency on Visual C++ runtime libraries, as illustrated below:

# Using Visual Studio Build Tools, location:
# C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64\
.\dumpbin.exe /DEPENDENTS "C:\Program Files\Microsoft\jdk-17.0.3.7-hotspot\bin\server\jvm.dll"
...
 Image has the following dependencies:
    ...
    VCRUNTIME140.dll
    VCRUNTIME140_1.dll
    ...

Process Monitor traces show that C:\Windows\System32\vcruntime140_1.dll is being loaded (among others); if Visual C++ runtime libs are missing, then jvm.dll of the JRE can't be loaded either.

For the above specific version of Microsoft's build of OpenJDK, installing the version 17 from the Microsoft Visual C++ Redistributable downloads page, then applying the above Registry keys, resulted in a working Data Factory Integration Runtime node, one able to process Parquet files.


Sidra Ideas Portal


Last update: 2022-07-14
Back to top