Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hdfs_to_hdfs 加入参数--preserveMeta报错 #102

Open
zhenghang2 opened this issue Jun 2, 2023 · 1 comment
Open

hdfs_to_hdfs 加入参数--preserveMeta报错 #102

zhenghang2 opened this issue Jun 2, 2023 · 1 comment

Comments

@zhenghang2
Copy link

Error: java.io.IOException: Exception raised while copying meta org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to com.aliyun.jindodata.common.JindoHadoopSystem at com.aliyun.jindodata.distcp.task.CpTask.handleSetMetaFailures(CpTask.java:292) at com.aliyun.jindodata.distcp.task.CpTask.syncDirMetaStatus(CpTask.java:335) at com.aliyun.jindodata.distcp.task.CpTask.loop(CpTask.java:124) at com.aliyun.jindodata.distcp.task.CpTask.loop(CpTask.java:41) at com.aliyun.jindodata.distjob.task.DistTask.run(DistTask.java:62) at com.aliyun.jindodata.distcp.DistCpMapper.map(DistCpMapper.java:41) at com.aliyun.jindodata.distcp.DistCpMapper.map(DistCpMapper.java:17) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) Caused by: java.lang.ClassCastException: org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to com.aliyun.jindodata.common.JindoHadoopSystem at com.aliyun.jindodata.distcp.task.CpTask.syncDirMetaStatus(CpTask.java:330) ... 13 more
这个是什么原因?大佬们解答一下?

@yanbinyang
Copy link
Collaborator

Error: java.io.IOException: Exception raised while copying meta org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to com.aliyun.jindodata.common.JindoHadoopSystem at com.aliyun.jindodata.distcp.task.CpTask.handleSetMetaFailures(CpTask.java:292) at com.aliyun.jindodata.distcp.task.CpTask.syncDirMetaStatus(CpTask.java:335) at com.aliyun.jindodata.distcp.task.CpTask.loop(CpTask.java:124) at com.aliyun.jindodata.distcp.task.CpTask.loop(CpTask.java:41) at com.aliyun.jindodata.distjob.task.DistTask.run(DistTask.java:62) at com.aliyun.jindodata.distcp.DistCpMapper.map(DistCpMapper.java:41) at com.aliyun.jindodata.distcp.DistCpMapper.map(DistCpMapper.java:17) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172) Caused by: java.lang.ClassCastException: org.apache.hadoop.hdfs.DistributedFileSystem cannot be cast to com.aliyun.jindodata.common.JindoHadoopSystem at com.aliyun.jindodata.distcp.task.CpTask.syncDirMetaStatus(CpTask.java:330) ... 13 more 这个是什么原因?大佬们解答一下?

麻烦提供一下运行的环境,如HDFS版本,及使用的jindo distcp版本。并且描述一下需求,如果只是单纯的HDFS to HDFS场景,其实可以考虑社区版本的hadoop distcp。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants