Skip to content

[CARBONDATA-4304] Initialize CarbonEnv will try to create folder on local filesystem if storelocation is set without scheme#4234

Open
jack86596 wants to merge 1 commit intoapache:masterfrom
jack86596:storepath
Open

[CARBONDATA-4304] Initialize CarbonEnv will try to create folder on local filesystem if storelocation is set without scheme#4234
jack86596 wants to merge 1 commit intoapache:masterfrom
jack86596:storepath

Conversation

@jack86596
Copy link
Contributor

Why is this PR needed?

If storelocation doesn't have scheme, file system is treated as local by default, not the file system defined by fs.defaultFS.

What changes were proposed in this PR?

  1. call fs.makeQualified and get the final storepath with correct scheme.
  2. use storepath with scheme to create the folder, keep storepath without scheme in CarbonProperties, because storepath in CarbonProperties will be used to compare with spark.sql.warehouse.dir which doesn't have scheme also.

Does this PR introduce any user interface change?

  • No

Is any new testcase added?

  • Yes

…ocal filesystem if storepath is set without scheme
@CarbonDataQA2
Copy link

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6049/

@CarbonDataQA2
Copy link

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4305/

@CarbonDataQA2
Copy link

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/439/

@jack86596
Copy link
Contributor Author

retest this please

@CarbonDataQA2
Copy link

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6051/

@CarbonDataQA2
Copy link

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4307/

@CarbonDataQA2
Copy link

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/441/

@kunal642
Copy link
Contributor

@jack86596 does this problem exists only with "carbon.storelocation" property?

@jack86596
Copy link
Contributor Author

@jack86596 does this problem exists only with "carbon.storelocation" property?

yes

@kunal642
Copy link
Contributor

As carbon.storelocation is deprecated and would be removed soon, I don't think we should use it or fix issues related to it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants