OpenPAI Backlog

[scarlett2018]

This issue is a long term backlog for planning and discussion. Please feel free to add "the top in your mind" to this issue directly.

• 2021 Feb Release Plan 2021 Feb Release Plan #5253
• 2021 March Draft Plan 2021 March Release Plan #5346

Last Updated on 02/01/2021

Top Focus Scenario

• Support HiveD Management experience
• AI Optimized Workload E2E support
  • use case study for internal & external typical workloads

---

Multi-Cloud/Multi-Cluster
- Hybrid installation
- multi/cross cluster management
- multi-cluster job scheduling

Job management

• Favorite Jobs

AutoScaler

• new added node's provisioning

Engineering Excellent

• CI/CD
• Build process

More examples for the current examples

Add user team support

• Jupyter Notebook support

• FC detect the preemption, gc, eviction, etc. Yuqi Wang (@yqwang-ms)

• FC rescale Framework Yuqi Wang (@yqwang-ms)

• Update the official docker image (comparing with deepo) Zhiyuan He (@hzy46)

Installation experience

• Topology planning
• Installation pre-check (especially for installation that start from non-clean env)
• Azure installation practices
• SKU support for complex job types
• Network plugin deployment
• migrate python2 to python3 Upgrade Python 2 to Python 3 #5042
• A kube runtime plugin to export parameters to env A kube runtime plugin to export parameters to env #4636

[fanyangCS]

#3872

[scarlett2018]

History Backlog Info Backup, only for reference.

Brainstorming on 2020/09/10

[Planned in #4898 ] Cell as sku in hived scheduler.

• Refactor duplicate deployment code.
• Refactor job submission form.

[Planned in #4898] Support dynamic sku types in different vc.

• Remaining works in New RestServer Architecture: RestServer -> DB -> ApiServer New RestServer Architecture: RestServer -> DB -> ApiServer #4651

Zhiyuan He (@hzy46) :
[Planned in #4898] pod/event watcher to support #4649
[Planned in #4898 ] multi cluster management detailed design & review

• merge previous doc change Fix document using DRI tickets #4828 Add distributed training examples of PyTorch #4821
• db gcer
• archive job (tag) Support delete/archive jobs #4453
• dbcontroller optimization items Database poller will be dead locked when 30000+ concurrent RPC calls happen #4849 Memory issues for framework watcher and poller in DBController #4850 Database controller performance optimization candidate items #4794
• make DBController tolerant to wrong framework request make DBController tolerant to wrong framework request #4889
• ui to let admin add storage how to realize this sentence: "please use RESTful API to assign storage to the groups of the user"?  #4806

Guoxin (@suiguoxin): #4789
[Planned in #4898] Alert-manager: Kill low-gpu-utilization jobs, tag abnormal jobs, Cordon node with k8s API when GPU GCC Error
[Planned in #4898] Job tags : DB table, rest-server API, web-portal refactor

Yi Yi (@yiyione):
[Planned in #4898] Group management page in webportal
[Planned in #4898] VC request management for user and admin

• VSCode client bug fix

Debuggy (@debuggy):

• marketplace related items 2020 Sept Release Plan openpaimarketplace#60
  [Planned in 2020 Sept ~ Oct release plan #4898]Grammar check
  • NNI job integration -- Feature Engineering
  • add info for items in submit job page
  • add "Upload" feature for dataset, set upload standard and workflow
• job submit refactor
  • new submit workflow design
  • team storage
  • sku
• job detail work part 2
  • job retry more detail
  • show events in job detail page

Per Task Retry History

• FC generates Task history snapshots (mocked Task CRD object with UID) Yuqi Wang (@yqwang-ms)
• Fluentd + DB to collect above snapshots Zhiyuan He (@hzy46)
• WebPortal expose the Per Task Retry History (and failure counts grouped by failure type within one job?) Debuggy (@debuggy)

Brainstorming on 5/14 - last updated at 5/28

• Candidate ideas about PAI experience improvements Candidate ideas about PAI experience improvements #3338

Multi-Cloud/Multi-Cluster
Zhiyuan He (@hzy46)
- Hybrid installation
- multi/cross cluster management
- multi-cluster job scheduling

AutoScaler YundongYe (@ydye)

• new added node's provisioning

Support for large scale cluster

• (June) set up latency benchmark report for 100 virtual nodes --> 1500 virtual nodes * 8 job YundongYe (@ydye)
• based on the stress and performance test result, analyze and plan for improvements

GPU Utilization

• (June) improve training cost ROI for EDU scenario Zhiyuan He (@hzy46)

• improve current internal training workload utilization Yi Yi (@yiyione)

• (June) Performance improvement for DB (Use Database as History Server Use Database as History Server #4610) Zhiyuan He (@hzy46)

More examples for the current examples

Add user team support

Surfacing more backend error to users in Job Details Page #4649
qfyin (@qfyin) ,Yuqi Wang (@yqwang-ms)
potential failure case: storage mount failed

• Surfacing more backend error to users (GCC error, out of storage, etc.)

• K8S Pod keep waiting in creating container, such as ErrImagePull K8S Pod keep waiting in creating container, such as ErrImagePull #3572

• GPU Fairness Usage Binyang Li (@Binyang2014)

• Usability improvement based on user study and feedback Yangou-ms (@yangou1988)

• Service plug-in (user customized) Zhiyuan He (@hzy46)

• Jupyter Notebook support Zhiyuan He (@hzy46)

• Runtime interface Binyang Li (@Binyang2014)

• HiveD Improvements (i.e. aware per GPU bad, etc.) Yuqi Wang (@yqwang-ms) Hanyu Zhao (@zhypku)

• FC detect the preemption, gc, eviction, etc. Yuqi Wang (@yqwang-ms)

• FC rescale Framework Yuqi Wang (@yqwang-ms)

• Update the official docker image (comparing with deepo) Zhiyuan He (@hzy46)

Installation experience

• Topology planning

• Installation pre-check (especially for installation that start from non-clean env)

• Azure installation practices

• SKU support for complex job types

• Network plugin deployment

• Support for new hardware

• migrate python2 to python3 Upgrade Python 2 to Python 3 #5042

• A kube runtime plugin to export parameters to env A kube runtime plugin to export parameters to env #4636

These are incomplete items from v0.17

• Grafana metrics link does not have time interval

• Skip intra communication for elastic job (Binyang P1 - Skip intra communication for elastic tasks #3708)

• Bring back debug mode (Binyang Reserve container after job failed. #3643, Better Job Debugging Experience for End Users #2210)

• replace disk cleaner with k8s pod eviction policy ([pure-k8s] Disk usage monitor module #3340, [pure k8s] K8s's pod eviction may not trigger when job container uses too much disk space #3765)

• When job is waiting, show the detailed waiting reason somewhere in the job page. (Show queueing duration and running duration in job detail #3847)

• Archive logs to system provided storage

• Exec to job containers on webportal

• persist service log (persistent log for PAI services #2129)

• differentiate opportunistic job and normal jobs in VC usage page.

• improve user group adding to VC experience

• improve HiveD configuration experience

• Show job priority in webportal

• introduce a new job state "initialization" (A job (or a pod of a job) may get stuck in a state neither running nor waiting #4141)

• open ssh connection in browser

• ssh key management and simplifying user behavior of ssh plugin

• open ssh connection in browser

• favorite jobs My favorite jobs and Jobs shared to me #2749

• actual resource usage of a job V1.0.0/1 allocated job resource usage API #4695

• Favorite Jobs

---

Engineering Excellent

• CI/CD
• Build process

---

Low Priority Postponed items

• (Postponed from Aug 2020 release) Elastic DL and Job level scale up and scale down - Yuqi Wang (@yqwang-ms) check with Ming/NNI and other integration scenarios - TODO: summarize the to-the-customer level scenarios supports