Extracting metadata from Apache Hive for migration

Apache Hive

Perform the steps in the Apache Hive section Extract metadata and query logs from your data warehouse to extract your Apache Hive metadata. You can then upload the metadata to your Cloud Storage bucket containing your migration files.

HDFS

Run the following command to extract extract metadata from HDFS using the dwh-migration-dumper tool.

dwh-migration-dumper \
  --connector hdfs \
  --host HDFS-HOST \
  --port HDFS-PORT \
  --output gs://MIGRATION-BUCKET/hdfs-dumper-output.zip \
  --assessment \

Replace the following:

• HDFS-HOST: the HDFS NameNode hostname
• HDFS-PORT: the HDFS NameNode port number. You can skip this argument if you are using the default 8020 port.
• MIGRATION-BUCKET: the Cloud Storage bucket that you are using to store the migration files.

This command extracts metadata from HDFS to a file named hdfs-dumper-output.zip in the MIGRATION-BUCKET directory.

There are several known limitations when extracting metadata from HDFS:

• Some tasks in this connector are optional and can fail, logging a full stack trade in the output. As long as the required tasks have succeeded and the hdfs-dumper-output.zip is generated, then you can proceed with the HDFS migration.
• The extraction process might fail or run slower than expected if the configured thread pool size is too large. If you are encountering these issues, we recommend decreasing the thread pool size using the command line argument --thread-pool-size.

Apache Ranger

Run the following command to extract extract metadata from Apache Ranger using the dwh-migration-dumper tool.

dwh-migration-dumper \
  --connector ranger \
  --host RANGER-HOST \
  --port 6080 \
  --user RANGER-USER \
  --password RANGER-PASSWORD \
  --ranger-scheme RANGER-SCHEME \
  --output gs://MIGRATION-BUCKET/ranger-dumper-output.zip \
  --assessment \

Replace the following:

• RANGER-HOST: the hostname of the Apache Ranger instance
• RANGER-USER: the username of the Apache Ranger user
• RANGER-PASSWORD: the password of the Apache Ranger user
• RANGER-SCHEME: specify if Apache Ranger is using http or https. Default value is http.
• MIGRATION-BUCKET: the Cloud Storage bucket that you are using to store the migration files.

You can also include the following optional flags:

• --kerberos-auth-for-hadoop: replaces --user and --password, if Apache Ranger is protected by kerberos instead of basic authentication. You must run the kinit command before the dwh-migration-dumper tool tool to use this flag.
• --ranger-disable-tls-validation: include this flag if the https certificate used by the API is self signed. For example, when using Cloudera.

This command extracts metadata from Apache Ranger to a file named ranger-dumper-output.zip in the MIGRATION-BUCKET directory.

Cloudera

Run the following command to extract metadata from Cloudera using the dwh-migration-dumper tool.

dwh-migration-dumper \
  --connector cloudera-manager \
  --url CLOUDERA-URL \
  --user CLOUDERA-USER \
  --password CLOUDERA-PASSWORD \
  --output gs://MIGRATION-BUCKET/cloudera-dumper-output.zip \
  --yarn-application-types APPLICATION-TYPES \
  --pagination-page-size PAGE-SIZE \
  --assessment \

Replace the following:

• CLOUDERA-URL: the URL for Cloudera Manager
• CLOUDERA-USER: the username of the Cloudera user
• CLOUDERA-PASSWORD: the password of the Cloudera user
• MIGRATION-BUCKET: the Cloud Storage bucket that you are using to store the migration files.
• APPLICATION-TYPES: (Optional) list of all existing application types from Hadoop YARN. For example, SPARK, MAPREDUCE.
• PAGE-SIZE: (Optional) specify how much data is fetched from 3rd party services, like the Hadoop YARN API. The default value is 1000, which represents 1000 entities per request.

This command extracts metadata from Cloudera to a file named dwh-migration-cloudera.zip in the MIGRATION-BUCKET directory.

Apache Hive query logs

Perform the steps in the Apache Hive section Extract query logs with the hadoop-migration-assessment logging hook to extract your Apache Hive query logs. You can then upload the logs to your Cloud Storage bucket containing your migration files.