sqoop从mysql导数据到hdfs,出现java.lang.ClassNotFoundException: Class QueryResult not found

运行sqoop从postgresql/mysql导入数据到hdfs, 结果出现如下错误:

2025-07-18 16:59:13,624 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/datasophon/hadoop-3.3.3
Note: /opt/sqoop/bin/QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2025-07-18 16:59:14,807 INFO orm.CompilationManager: Writing jar file: /opt/sqoop/bin/QueryResult.jar
2025-07-18 16:59:15,442 INFO tool.ImportTool: Destination directory /dmp/ods/riskctr/ri_base_auth_user deleted.
2025-07-18 16:59:15,444 INFO mapreduce.ImportJobBase: Beginning query import.
2025-07-18 16:59:15,445 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2025-07-18 16:59:15,450 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2025-07-18 16:59:15,462 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2025-07-18 16:59:15,528 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2025-07-18 16:59:15,668 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2025-07-18 16:59:15,668 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2025-07-18 16:59:15,881 INFO db.DBInputFormat: Using read commited transaction isolation
2025-07-18 16:59:15,908 INFO mapreduce.JobSubmitter: number of splits:1
2025-07-18 16:59:16,010 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local384333610_0001
2025-07-18 16:59:16,011 INFO mapreduce.JobSubmitter: Executing with tokens: []
2025-07-18 16:59:16,133 INFO mapred.LocalDistributedCacheManager: Creating symlink: /data/mapred/local/job_local384333610_0001_9a218038-4c6b-4055-bc1b-06386f28f0cb/libjars <- /home/hive/libjars/*
2025-07-18 16:59:16,135 WARN fs.FileUtil: Command 'ln -s /data/mapred/local/job_local384333610_0001_9a218038-4c6b-4055-bc1b-06386f28f0cb/libjars /home/hive/libjars/*' failed 1 with: ln: failed to create symbolic link ‘/home/hive/libjars/*’: No such file or directory

2025-07-18 16:59:16,136 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /data/mapred/local/job_local384333610_0001_9a218038-4c6b-4055-bc1b-06386f28f0cb/libjars <- /home/hive/libjars/*
2025-07-18 16:59:16,136 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/hive384333610/.staging/job_local384333610_0001/libjars as file:/data/mapred/local/job_local384333610_0001_9a218038-4c6b-4055-bc1b-06386f28f0cb/libjars
2025-07-18 16:59:16,197 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2025-07-18 16:59:16,197 INFO mapreduce.Job: Running job: job_local384333610_0001
2025-07-18 16:59:16,198 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2025-07-18 16:59:16,203 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2025-07-18 16:59:16,203 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2025-07-18 16:59:16,203 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2025-07-18 16:59:16,237 INFO mapred.LocalJobRunner: Waiting for map tasks
2025-07-18 16:59:16,237 INFO mapred.LocalJobRunner: Starting task: attempt_local384333610_0001_m_000000_0
2025-07-18 16:59:16,253 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2025-07-18 16:59:16,253 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2025-07-18 16:59:16,265 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2025-07-18 16:59:16,423 INFO db.DBInputFormat: Using read commited transaction isolation
2025-07-18 16:59:16,441 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
2025-07-18 16:59:16,447 INFO mapred.LocalJobRunner: map task executor complete.
2025-07-18 16:59:16,453 WARN mapred.LocalJobRunner: job_local384333610_0001
java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class QueryResult not found
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class QueryResult not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
        at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:270)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:266)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:528)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: Class QueryResult not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
        ... 12 more
2025-07-18 16:59:17,201 INFO mapreduce.Job: Job job_local384333610_0001 running in uber mode : false
2025-07-18 16:59:17,202 INFO mapreduce.Job:  map 0% reduce 0%
2025-07-18 16:59:17,204 INFO mapreduce.Job: Job job_local384333610_0001 failed with state FAILED due to: NA
2025-07-18 16:59:17,208 INFO mapreduce.Job: Counters: 0
2025-07-18 16:59:17,211 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2025-07-18 16:59:17,212 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 1.742 seconds (0 bytes/sec)
2025-07-18 16:59:17,213 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2025-07-18 16:59:17,213 INFO mapreduce.ImportJobBase: Retrieved 0 records.
2025-07-18 16:59:17,213 ERROR tool.ImportTool: Import failed: Import job failed!
[hive@dmp-rdb-svr1 ~]$ sqoop import --connect jdbc:postgresql://10.188.6.241:5432/postgres --username root --password XcMg*Crm^240606 --query " SELECT username, display_name, user_sn, native_name, email, password as passwd, valid_from, valid_to, status, mobile, user_type, sso_uid, gender, birth_date, id_number, id_type, src_sys,  created_by,  updated_by FROM ri_base.ri_base_auth_user where \$CONDITIONS " --target-dir /dmp/ods/riskctr/ri_base_auth_user --delete-target-dir --hive-import --hive-database testdb --hive-table ri_base_auth_user --null-string '\\N' --null-non-string '\\N' --fields-terminated-by "," -m 1 --bindir /opt/sqoop
Warning: /opt/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
2025-07-18 16:59:27,845 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2025-07-18 16:59:27,872 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2025-07-18 16:59:27,959 INFO manager.SqlManager: Using default fetchSize of 1000
2025-07-18 16:59:27,963 INFO tool.CodeGenTool: Beginning code generation
2025-07-18 16:59:28,579 INFO manager.SqlManager: Executing SQL statement:  SELECT username, display_name, user_sn, native_name, email, password as passwd, valid_from, valid_to, status, mobile, user_type, sso_uid, gender, birth_date, id_number, id_type, src_sys,  created_by,  updated_by FROM ri_base.ri_base_auth_user where  (1 = 0)
2025-07-18 16:59:28,622 INFO manager.SqlManager: Executing SQL statement:  SELECT username, display_name, user_sn, native_name, email, password as passwd, valid_from, valid_to, status, mobile, user_type, sso_uid, gender, birth_date, id_number, id_type, src_sys,  created_by,  updated_by FROM ri_base.ri_base_auth_user where  (1 = 0)
2025-07-18 16:59:28,672 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/datasophon/hadoop-3.3.3
Note: /opt/sqoop/QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2025-07-18 16:59:29,703 INFO orm.CompilationManager: Writing jar file: /opt/sqoop/QueryResult.jar
2025-07-18 16:59:30,383 INFO tool.ImportTool: Destination directory /dmp/ods/riskctr/ri_base_auth_user deleted.
2025-07-18 16:59:30,385 INFO mapreduce.ImportJobBase: Beginning query import.
2025-07-18 16:59:30,386 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2025-07-18 16:59:30,392 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2025-07-18 16:59:30,404 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2025-07-18 16:59:30,483 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2025-07-18 16:59:30,595 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2025-07-18 16:59:30,595 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2025-07-18 16:59:30,818 INFO db.DBInputFormat: Using read commited transaction isolation
2025-07-18 16:59:30,844 INFO mapreduce.JobSubmitter: number of splits:1
2025-07-18 16:59:30,942 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1949585164_0001
2025-07-18 16:59:30,942 INFO mapreduce.JobSubmitter: Executing with tokens: []
2025-07-18 16:59:31,068 INFO mapred.LocalDistributedCacheManager: Creating symlink: /data/mapred/local/job_local1949585164_0001_605ef3df-a215-49ff-9c31-163424ab22ef/libjars <- /home/hive/libjars/*
2025-07-18 16:59:31,071 WARN fs.FileUtil: Command 'ln -s /data/mapred/local/job_local1949585164_0001_605ef3df-a215-49ff-9c31-163424ab22ef/libjars /home/hive/libjars/*' failed 1 with: ln: failed to create symbolic link ‘/home/hive/libjars/*’: No such file or directory

2025-07-18 16:59:31,071 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /data/mapred/local/job_local1949585164_0001_605ef3df-a215-49ff-9c31-163424ab22ef/libjars <- /home/hive/libjars/*
2025-07-18 16:59:31,071 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/hive1949585164/.staging/job_local1949585164_0001/libjars as file:/data/mapred/local/job_local1949585164_0001_605ef3df-a215-49ff-9c31-163424ab22ef/libjars
2025-07-18 16:59:31,117 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2025-07-18 16:59:31,118 INFO mapreduce.Job: Running job: job_local1949585164_0001
2025-07-18 16:59:31,119 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2025-07-18 16:59:31,124 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2025-07-18 16:59:31,124 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2025-07-18 16:59:31,125 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2025-07-18 16:59:31,158 INFO mapred.LocalJobRunner: Waiting for map tasks
2025-07-18 16:59:31,159 INFO mapred.LocalJobRunner: Starting task: attempt_local1949585164_0001_m_000000_0
2025-07-18 16:59:31,175 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2025-07-18 16:59:31,175 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2025-07-18 16:59:31,186 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2025-07-18 16:59:31,334 INFO db.DBInputFormat: Using read commited transaction isolation
2025-07-18 16:59:31,351 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
2025-07-18 16:59:31,356 INFO mapred.LocalJobRunner: map task executor complete.
2025-07-18 16:59:31,364 WARN mapred.LocalJobRunner: job_local1949585164_0001
java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class QueryResult not found
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class QueryResult not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
        at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:270)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:266)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.java:528)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: Class QueryResult not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
        ... 12 more

网上各种解决方案,有人说要把QueryResult复制到$SQOOP_HOME/lib,这显然治标不治本,QueryResult应该是根据查询语句的字段动态产生的,这样肯定每个任务都得手动复制一下才能跑。

有人说加一个参数 --bindir=$SQOOP_HOME/lib,这方法可行,但会残留文件在lib目录。

经试验,我发现只要正确设置了SQOOP_HOME变量,就不会报错了。如

 export SQOOP_HOME=/opt/sqoop

后来发现应该是权限问题:

chmod 777 -R  /data/mapred/local /tmp/sqoop-hive

你可能感兴趣的:(大数据,sqoop,mysql,hdfs)