Introduce XPU scope profiler extending existing XPU profiler plugin

[moksiuc]

Summary:

As XPU became a PyTorch built-in device, the profiler support is indispensable part of functionality completeness. In this PR, the XPU scope profiler is introduced by extending existing XPU profiler plugin. The XPU scope profiler is built on the foundation of intel PTI toolkit (https://github.com/intel/pti-gpu), and underlying SYCL runtime. It allows to gather XPU hardware metrics. The LIBKINETO_NOXPUPTI option is used to enable or disable the whole XPU profiler plugin during kineto build stage.

Changes:

• Added new ActivityType : XPU_SCOPE_PROFILER, enabling the new scope profiler
• Outputs XPU hardware metrics from the new scope profiler in Perfetto counters display mode ("C")
• Added gtest

[moksiuc]

@EikanWang, @gujinghui

[gujinghui]

@moksiuc It's great that we are going to update our PTI integration code, and introduce new profiler path.
Could you help address below questions?

1. The alternative of ScopeProfiler is the RangeProfiler for CUDA? Looks like the RangeProfler is not enabled in PyTorch by default so far. Do you know why?
2. This PR is too huge to review. Can we split it to several PRs? For example, one PR for code refactor or cleanup per kineto or PTI changes, one or two PRs for ScopeProfiler, one PR for ChromeTraceLogger enhancement, and add test cases for each PRs.
3. BTW, CUDA provides CUDA_DRIVER activity to trace the driver actions. We should provide L0 actions as the counterpart, right? I remember, PTI should be able to do that. Do we have plan to cover it?
   

   kineto/libkineto/src/ActivityType.cpp

   Line 29 in 1e30d37

   {"cuda_driver", ActivityType::CUDA_DRIVER},

[moksiuc]

1. The alternative of ScopeProfiler is the RangeProfiler for CUDA? Looks like the RangeProfler is not enabled in PyTorch by default so far. Do you know why?
   It is enabled by providing experimental_config=_ExperimentalConfig(...). I don't know why it is this way but we are enabling our profiler the same way. One of the reasons may be that Range/Scope profiler requires parameters like HW metrics names that are passed through _ExperimentalConfig.

[moksiuc]

3. BTW, CUDA provides CUDA_DRIVER activity to trace the driver actions. We should provide L0 actions as the counterpart, right? I remember, PTI should be able to do that. Do we have plan to cover it?
   

   kineto/libkineto/src/ActivityType.cpp

   Line 29 in 1e30d37

   {"cuda_driver", ActivityType::CUDA_DRIVER},

For sure not in this PR. I'll add this to our list of tasks.

[moksiuc]

2. This PR is too huge to review. Can we split it to several PRs? For example, one PR for code refactor or cleanup per kineto or PTI changes, one or two PRs for ScopeProfiler, one PR for ChromeTraceLogger enhancement, and add test cases for each PRs.

Extracted clean up and adding config for scope profiler to separate PR's.
This one should be much smaller afterwards.
Currently I don't see further areas of extracting separate PR's as what would remain is full scope profiler implementation with tests and we'd like not to introduce half of the implementation that is not working functionally.

[gujinghui]

This PR is split to #1177, #1180, and more.

[gujinghui]

@moksiuc Let's close this PR.

[moksiuc]

@gujinghui this is the core of the scope profiler. When 2 smaller parts are merged this one would have only core profiler left.

[gujinghui]

@divyanshk This PR is ready now. Could you help review? Thanks.

[divyanshk]

Left some comments. Important one is in ActivityType.h

I need to look again at output_json.cpp.

[gujinghui]

@divyanshk @scotts all comments have been addressed. Could you help review again? Thanks.