[CARBONDATA-4346] Remove list files while query and invalid cache by ShreelekhyaG · Pull Request #4287 · apache/carbondata

ShreelekhyaG · 2022-07-04T14:01:55Z

Why is this PR needed?

Performance degradation for Incremental updates is observed in the partition table.

During the update, in the prune step we are listing files from segment path to get the carbondata files and create fileNameToMetaInfoMapping map. On incremental update for partition table, the number of invalid files keep on increasing each time which is causing the degradation in createCarbonDataFileBlockMetaInfoMapping method.
Example: Assume a single partition with 1000 carbondata files.
Perform 1st update: adds 900 new carbondata files.
Perform 2nd update (same update query): adds another 900 carbondata files. Now the files added by 1st update are invalid.
Perform query: It does list files. Here, considers invalid files also and adds to fileNameToMetaInfoMapping map.
The number of invalid files keeps on increasing with each update which is causing the degradation in creating fileNameToMetaInfoMapping map.

Invalid segments cache is not removed after delete/update.

What changes were proposed in this PR?

Instead of listing files, made a change to get the carbon file from the file name and create BlockMetaInfo directly in createBlockMetaInfo.
Impact when tested on a single partition with 100 segments:
- There is significant improvement observed in the Incremental update operation.
- 95% improvement seen in 1st time select count(*) operation. Because in select count(*) flow it was listing files for each segment and the map was not reused.
Impact when tested on a non-partition table with 100 segments:
- Almost the same or no improvement for the non-partition table
Clearing invalid/deleted segments from cache after delete and update.

Does this PR introduce any user interface change?

No

Is any new testcase added?

Yes

CarbonDataQA2 · 2022-07-04T14:16:10Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/772/

CarbonDataQA2 · 2022-07-04T14:17:10Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6386/

CarbonDataQA2 · 2022-07-04T14:17:42Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4641/

CarbonDataQA2 · 2022-07-04T17:22:22Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6387/

CarbonDataQA2 · 2022-07-04T17:29:03Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4642/

CarbonDataQA2 · 2022-07-04T18:04:22Z

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/773/

...in/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala

core/src/main/java/org/apache/carbondata/core/util/BlockletIndexUtil.java

CarbonDataQA2 · 2022-07-08T12:46:40Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6390/

CarbonDataQA2 · 2022-07-08T13:19:29Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4645/

CarbonDataQA2 · 2022-07-08T13:29:00Z

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/776/

Indhumathi27 · 2022-07-11T11:41:34Z

LGTM

core/src/main/java/org/apache/carbondata/core/util/BlockletIndexUtil.java

CarbonDataQA2 · 2022-07-12T15:59:20Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6393/

CarbonDataQA2 · 2022-07-12T16:00:53Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4650/

CarbonDataQA2 · 2022-07-12T16:35:34Z

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/779/

ShreelekhyaG · 2022-07-13T04:58:11Z

retest this please

CarbonDataQA2 · 2022-07-13T07:46:49Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6395/

CarbonDataQA2 · 2022-07-13T07:52:38Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4652/

CarbonDataQA2 · 2022-07-13T08:38:14Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/781/

ShreelekhyaG · 2022-07-13T09:09:25Z

retest this please

CarbonDataQA2 · 2022-07-13T09:49:26Z

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4653/

CarbonDataQA2 · 2022-07-13T09:50:26Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/782/

CarbonDataQA2 · 2022-07-13T12:40:00Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6396/

akashrn5 · 2022-07-14T03:53:51Z

LGTM

ShreelekhyaG · 2022-07-14T05:03:16Z

retest this please

CarbonDataQA2 · 2022-07-14T07:42:04Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6397/

CarbonDataQA2 · 2022-07-14T07:47:37Z

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4654/

CarbonDataQA2 · 2022-07-14T08:08:10Z

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/783/

shenjiayu17 · 2022-10-09T15:16:42Z

Build Failed with Spark 2.4.5, Please check CI http://159.138.8.58:12602/job/ApacheCarbon_PR_Builder_2.4.5/4659/

ShreelekhyaG force-pushed the updatePerfAndCache branch from 0415996 to c63d26c Compare July 4, 2022 14:54

ShreelekhyaG changed the title ~~[WIP] Remove list files while query and invalid cache~~ [CARBONDATA-4346] Remove list files while query and invalid cache Jul 6, 2022

Indhumathi27 reviewed Jul 7, 2022

View reviewed changes

...in/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForUpdateCommand.scala Outdated Show resolved Hide resolved

Indhumathi27 reviewed Jul 7, 2022

View reviewed changes

core/src/main/java/org/apache/carbondata/core/util/BlockletIndexUtil.java Outdated Show resolved Hide resolved

Indhumathi27 reviewed Jul 7, 2022

View reviewed changes

core/src/main/java/org/apache/carbondata/core/util/BlockletIndexUtil.java Show resolved Hide resolved

ShreelekhyaG force-pushed the updatePerfAndCache branch from c63d26c to 94c40e9 Compare July 8, 2022 09:59

akashrn5 reviewed Jul 12, 2022

View reviewed changes

core/src/main/java/org/apache/carbondata/core/util/BlockletIndexUtil.java Show resolved Hide resolved

ShreelekhyaG force-pushed the updatePerfAndCache branch from 94c40e9 to 6e26954 Compare July 12, 2022 13:07

remove list files while query and invalid cache

b6175a7

ShreelekhyaG force-pushed the updatePerfAndCache branch from 6e26954 to b6175a7 Compare July 13, 2022 05:50

Conversation

ShreelekhyaG commented Jul 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why is this PR needed?

What changes were proposed in this PR?

Does this PR introduce any user interface change?

Is any new testcase added?

Uh oh!

CarbonDataQA2 commented Jul 4, 2022

Uh oh!

CarbonDataQA2 commented Jul 4, 2022

Uh oh!

CarbonDataQA2 commented Jul 4, 2022

Uh oh!

CarbonDataQA2 commented Jul 4, 2022

Uh oh!

CarbonDataQA2 commented Jul 4, 2022

Uh oh!

CarbonDataQA2 commented Jul 4, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CarbonDataQA2 commented Jul 8, 2022

Uh oh!

CarbonDataQA2 commented Jul 8, 2022

Uh oh!

CarbonDataQA2 commented Jul 8, 2022

Uh oh!

Indhumathi27 commented Jul 11, 2022

Uh oh!

Uh oh!

CarbonDataQA2 commented Jul 12, 2022

Uh oh!

CarbonDataQA2 commented Jul 12, 2022

Uh oh!

CarbonDataQA2 commented Jul 12, 2022

Uh oh!

ShreelekhyaG commented Jul 13, 2022

Uh oh!

CarbonDataQA2 commented Jul 13, 2022

Uh oh!

CarbonDataQA2 commented Jul 13, 2022

Uh oh!

CarbonDataQA2 commented Jul 13, 2022

Uh oh!

ShreelekhyaG commented Jul 13, 2022

Uh oh!

CarbonDataQA2 commented Jul 13, 2022

Uh oh!

CarbonDataQA2 commented Jul 13, 2022

Uh oh!

CarbonDataQA2 commented Jul 13, 2022

Uh oh!

akashrn5 commented Jul 14, 2022

Uh oh!

ShreelekhyaG commented Jul 14, 2022

Uh oh!

CarbonDataQA2 commented Jul 14, 2022

Uh oh!

CarbonDataQA2 commented Jul 14, 2022

Uh oh!

CarbonDataQA2 commented Jul 14, 2022

Uh oh!

shenjiayu17 commented Oct 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ShreelekhyaG commented Jul 4, 2022 •

edited

Loading