[LIVY-527] added autocompletion api and implementation for ipython by wangqiaoshi · Pull Request #119 · apache/livy

wangqiaoshi · 2018-10-08T07:53:12Z

What changes were proposed in this pull request?

add complete function in PySparkJobProcessor interface, so ReplDriver can get completion message from python

How was this patch tested?

post /sessions/{sessionId}/completion
set kind=pyspark

wangqiaoshi · 2018-10-08T07:56:03Z

should set
spark.pyspark.python ${ipython-path}

codecov-io · 2018-10-08T08:52:24Z

Codecov Report

Merging #119 into master will decrease coverage by 2.44%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master     #119      +/-   ##
============================================
- Coverage     70.94%   68.49%   -2.45%     
+ Complexity      922      893      -29     
============================================
  Files           100      100              
  Lines          5500     5606     +106     
  Branches        828      840      +12     
============================================
- Hits           3902     3840      -62     
- Misses         1062     1214     +152     
- Partials        536      552      +16

Impacted Files	Coverage Δ	Complexity Δ
...scala/org/apache/livy/repl/PythonInterpreter.scala	`47.53% <0%> (-0.6%)`	`9 <0> (ø)`
.../main/java/org/apache/livy/rsc/ReplJobResults.java	`0% <0%> (-100%)`	`0% <0%> (-2%)`
...ain/java/org/apache/livy/rsc/driver/Statement.java	`0% <0%> (-100%)`	`0% <0%> (-6%)`
...in/java/org/apache/livy/rsc/driver/AddFileJob.java	`0% <0%> (-100%)`	`0% <0%> (-3%)`
...ava/org/apache/livy/rsc/driver/StatementState.java	`0% <0%> (-93.34%)`	`0% <0%> (-9%)`
...rc/main/java/org/apache/livy/rsc/BaseProtocol.java	`52.12% <0%> (-22.35%)`	`1% <0%> (ø)`
.../java/org/apache/livy/rsc/driver/SparkEntries.java	`46.87% <0%> (-17.19%)`	`7% <0%> (-3%)`
...main/java/org/apache/livy/rsc/ContextLauncher.java	`66.01% <0%> (-15.62%)`	`13% <0%> (-5%)`
...c/src/main/java/org/apache/livy/rsc/RSCClient.java	`75.15% <0%> (-8.7%)`	`20% <0%> (-6%)`
...ain/scala/org/apache/livy/utils/SparkYarnApp.scala	`73.23% <0%> (-6.34%)`	`33% <0%> (ø)`
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d87a348...1cd30e1. Read the comment docs.

vanzin · 2018-10-08T18:50:25Z

Please follow the guidelines in the contribution guide (linked in the PR template). Mainly:

file a bug with a description of the problem / feature
provide a proper description of your PR (which will be the commit message when merged).

wangqiaoshi · 2018-10-09T02:49:59Z

Fix the bug : The completion function doesn't work in pyspark kind.

description:
when pyspark run in ipython environment , kind pyspark completion work.
this pr add complete function in PySparkJobProcessor interface, so ReplDriver can get completion message from python

How was this patch tested?

post /sessions/{sessionId}/completion
set kind=pyspark

vanzin · 2018-10-09T15:50:03Z

Please follow the contribution guide. It's linked in the PR template. In case you missed it:
https://livy.incubator.apache.org/community/

You're supposed to replace the PR template with your description of the changes. And file a bug.

vanzin

Some style issues, otherwise looks ok.

Can a test be written for this (e.g. in InteractiveIT)? Not sure how easy it is to hit the completion API from tests.

vanzin · 2018-12-19T22:38:28Z

    def addFile(self, uri_path):
        job_context.sc.addFile(uri_path)

+


That's a lot of blank lines. Surprising the style checker didn't complain. Should be one line between methods.

vanzin · 2018-12-19T22:38:37Z

+
+    def complete(self,code,cursor_pos=None):
+        results = []
+        ip_completer =global_dict.get('ip_completer')


space after =

vanzin · 2018-12-19T22:41:31Z

+        from IPython.core.completer import IPCompleter
+        ip = get_ipython()
+        ip_completer = IPCompleter(ip,global_namespace=global_dict)
+        global_dict['ip_completer'] = ip_completer


Does the completer need to be visible to user statements? Otherwise it seems you can just use a regular global variable (like job_context or local_tmp_dir_path in this file).

Completer will be different when python interpreter is different. The number of completer global variable will be two,I don't know if that's appropriate?

I see what you mean (ip_completer vs. p_completer).

I think it would be better to keep a single variable for the completer, and a boolean that says whether to use the IPython API.

But my main question was: do they need to be stored in global_dict or can they just be a global variable in this module instead?

ok,I see what you mean.

vanzin · 2018-12-19T22:41:50Z

+
+
+
+    def complete(self,code,cursor_pos=None):


space after ,

vanzin · 2018-12-19T22:42:03Z

+        ip_completer =global_dict.get('ip_completer')
+        p_completer = global_dict.get("p_completer")
+        if ip_completer is not None:
+            results = ip_completer.complete(code,cursor_pos=cursor_pos)[1]


space after ,

vanzin · 2018-12-19T22:42:52Z

    sys.stderr = UnicodeDecodingStringIO()

    spark_major_version = os.getenv("LIVY_SPARK_MAJOR_VERSION")
+    get_completer()


get assumes something is being returned. initialize_completer() is probably a better name.

vanzin · 2018-12-19T22:43:17Z

+        __IPYTHON__
+        from IPython.core.completer import IPCompleter
+        ip = get_ipython()
+        ip_completer = IPCompleter(ip,global_namespace=global_dict)


space after ,

Tagar · 2019-01-15T23:51:37Z

ping @wangqiaoshi - that's a great improvement!
do you mind fixing those minor issues after the last review
thank you!

wangqiaoshi · 2019-01-17T14:46:34Z

ping @wangqiaoshi - that's a great improvement!
do you mind fixing those minor issues after the last review
thank you!

ok

wangqiaoshi · 2019-02-02T04:19:11Z

Some style issues, otherwise looks ok.

Can a test be written for this (e.g. in InteractiveIT)? Not sure how easy it is to hit the completion API from tests.

I have a doubt, InteractiveIT isn't find Kind . Because module livy - integration - the test not introduced livy - core - the parent, but Kind in module livy - core - the parent. I can't run InteractiveIT

vanzin · 2019-03-14T22:33:53Z

Sorry, totally missed your question. I'm also not sure I understand the question.

It would be really good to have some test for this. It doesn't need to be an integration tests, but it needs to be tested somehow. Otherwise the changes look good.

balamuruganvg · 2019-05-03T09:46:55Z

What's the plan to merge this?

jepma · 2019-05-20T13:15:23Z

Yes, it would be nice to implement this feature 👍

Tagar · 2019-05-20T21:46:21Z

ping @wangqiaoshi

cyrilledelabre · 2019-07-25T08:15:35Z

Hi @wangqiaoshi , any plan to merge this ?
is still something missing ?
thanks!

tsurushun · 2019-09-05T04:13:24Z

Hi @wangqiaoshi, you would be doing a great favor for a lot of people if you could merge this.
Thanks!

mnoji · 2019-09-05T04:20:42Z

Hello @wangqiaoshi ,
Please merge your great job!!

Jassy1994 · 2019-09-05T09:53:10Z

Hello @wangqiaoshi
I use jupyter pyspark kernel in livy yarn-cluster mode, and merged this patch in my livy。but jupyter notebook still cannot auto complete. I find jupyter auto completion seems not use livy autocompletion api，May you have some advices about this situation?
Thanks

rragundez · 2019-11-02T08:28:09Z

Any updates on merging this feature @wangqiaoshi ?

namanmishra91 · 2020-02-24T10:06:06Z

+def initialize_completer():
+    global ip_completer,p_completer
+    try:
+        __IPYTHON__


@wangqiaoshi Since fake_shell runs within a standard python process and not within an ipython shell, how will IPCompleter be initialized?

agsachin · 2022-04-21T04:59:21Z

Any updates on merging this feature @wangqiaoshi @vanzin ?

aabdullah-getguru · 2023-01-29T23:10:26Z

Anyone?

github-actions · 2026-06-29T01:09:38Z

This pull request has been automatically marked as stale because it has had no activity for at least 3 months. If you are still working on this change or plan to move it forward, please leave a comment or push a new commit so we know to keep it open. Otherwise, this PR will be closed automatically in about one month. Thank you for your contribution to Apache Livy!

add pyspark Completion function

da2c07a

mkdify scala style

039a63a

wangqiaoshi changed the title ~~added autocompletion api and implementation for ipython~~ [LIVY-527] added autocompletion api and implementation for ipython Oct 12, 2018

增加python版本的代码提示

e4ced91

wangqiaoshi mentioned this pull request Nov 3, 2018

Enable Jupyter tab completion for Spark and PySpark kernel jupyter-incubator/sparkmagic#274

Open

vanzin reviewed Dec 19, 2018

View reviewed changes

qiaoshi.wang added 2 commits January 20, 2019 22:46

修复样式的问题

44570f5

添加ip_completer, p_completer变量

1cd30e1

namanmishra91 reviewed Feb 24, 2020

View reviewed changes

github-actions Bot added the stale label Jun 29, 2026

		def addFile(self, uri_path):
		job_context.sc.addFile(uri_path)

Uh oh!

Conversation

wangqiaoshi commented Oct 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

wangqiaoshi commented Oct 8, 2018

Uh oh!

codecov-io commented Oct 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vanzin commented Oct 8, 2018

Uh oh!

wangqiaoshi commented Oct 9, 2018

Fix the bug : The completion function doesn't work in pyspark kind.

How was this patch tested?

Uh oh!

vanzin commented Oct 9, 2018

Uh oh!

vanzin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tagar commented Jan 15, 2019

Uh oh!

wangqiaoshi commented Jan 17, 2019

Uh oh!

wangqiaoshi commented Feb 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanzin commented Mar 14, 2019

Uh oh!

balamuruganvg commented May 3, 2019

Uh oh!

jepma commented May 20, 2019

Uh oh!

Tagar commented May 20, 2019

Uh oh!

cyrilledelabre commented Jul 25, 2019

Uh oh!

tsurushun commented Sep 5, 2019

Uh oh!

mnoji commented Sep 5, 2019

Uh oh!

Jassy1994 commented Sep 5, 2019

Uh oh!

rragundez commented Nov 2, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agsachin commented Apr 21, 2022

Uh oh!

aabdullah-getguru commented Jan 29, 2023

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

wangqiaoshi commented Oct 8, 2018 •

edited

Loading

codecov-io commented Oct 8, 2018 •

edited

Loading

wangqiaoshi commented Feb 2, 2019 •

edited

Loading