Android: Position in Collection not supported by Flatlist, SectionList, VirtualizedList, or ScrollView

[fabOnReact]

Issue react/react-native#30977
Pull Request react/react-native#33180
Related react/react-native#31666

[fabOnReact]

• port flatlist example
• test
• list of scenarios to tests Android: Position in Collection not supported by Flatlist, SectionList, VirtualizedList, or ScrollView #6 (comment)
• read sourcecode
• evaluate if the default to columnCount of 1 is correct 5d3e3f8
• save test cases
• post test cases in pull request
• discuss alternative implementations Android: Position in Collection not supported by Flatlist, SectionList, VirtualizedList, or ScrollView #6 (comment)

[fabOnReact]

<FlatList numColumns={undefined} /> should not trigger Runtime Error NoSuchKey exception columnCount when enabling TalkBack.

<FlatList numColumns={undefined} />

Expected Result:
No Runtime Error

Actual Result:
Enabling TalkBack triggers a Runtime Error NoSuchKeyException columnCount

CLICK TO OPEN RUNTIME ERROR

CLICK TO OPEN DETAILS

https://github.com/fabriziobertoglio1987/react-native/blob/e36f46a003a0c279d6d7e84bdfefc0b1a61d5cb7/Libraries/Lists/FlatList.js#L671-L675

https://github.com/fabriziobertoglio1987/react-native/blob/e36f46a003a0c279d6d7e84bdfefc0b1a61d5cb7/ReactAndroid/src/main/java/com/facebook/react/views/scroll/ReactHorizontalScrollView.java#L201

numColums default was removed with commit 7d5895d

CLICK TO OPEN TESTS RESULTS - WITHOUT RUNTIME

numColumnsNoRuntime.mp4

CLICK TO OPEN TESTS RESULTS - WITH RUNTIME

numColumnsRuntime.mp4

Solved with fabOnReact/react-native@062cdcd

[fabOnReact]

Evaluate Alternative Implementations

• calculating number of childs on the native side.

The ReactScrollView has a child node ReactViewGroup. It is not possible to retrieve the number of child from ReactViewGroup, because getChildCount was over-ridden and returns null.
https://github.com/fabriziobertoglio1987/react-native-notes/blob/5d3e3f8abd1cb9ece663db88137aded23e4c50d8/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactShadowNodeImpl.java#L256-L258

[fabOnReact]

Scenarios Previously Tested in react/react-native#31666 (comment) (VIDEO1 and VIDEO2)

• Horizontal FlatList
• Vertical FlatList
• Horizontal FlatList
• Horizontal FlatList when fixed layout
• FlatList when numColumns > 1
• FlatList when numColumns > 1 and fixed layout
• Simple 1 column FlatList
• Simple 1 column FlatList - Fixed layout
• Nested FlatList

New Scenarios

• read diff Android: Position in Collection not supported by Flatlist, SectionList, VirtualizedList, or ScrollView #6 (comment)
  * understand the use of CollectionInfo properties by changing the values of rowSpan, columnSpan and columnIndex
• change values of different props used in the branch (for ex. numColumns to undefined)
• find potential issues in diff
• do random tests
• learn more about TalkBack with FlatList.
  - Special TalkBack features with Android ScrollView
  - TalkBack Shortcuts
  - Change TalkBack settings and test them with FlatList

[fabOnReact]

CollectionItemInfo

Doubts and Tougths

These values are hardcoded.

1. What do they represent in the UI/functionality?
2. Which functionality we are not supporting?
3. When developers try to use unsupported functionality, do they receive a log message or alert that they are not supported?
4. Do we get a runtime or crash in the application because we did not handle/test these scenarios?

CollectionItemInfo in ReactNative

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/Libraries/Lists/FlatList.js#L619
https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/Libraries/Lists/FlatList.js#L621
https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/Libraries/Lists/FlatList.js#L651-L653
https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/Libraries/Lists/FlatList.js#L622

The information are read by TalkBack from AccessibilityNodeInfo:

• rowSpan, columnSpan will always be 1
• heading is always false

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L230-L238

CollectionItemInfo in AndroidX

https://github.com/androidx/androidx/blob/150112ca4e1e670fb4eb6756993f3a25df0a5da9/core/core/src/main/java/androidx/core/view/accessibility/AccessibilityNodeInfoCompat.java#L881

    /**
     * Class with information if a node is a collection item.
     * <p>
     * A collection item is contained in a collection, it starts at
     * a given row and column in the collection, and spans one or
     * more rows and columns. For example, a header of two related
     * table columns starts at the first row and the first column,
     * spans one row and two columns.
     * </p>
     */
    public static class CollectionItemInfoCompat {

        final Object mInfo;

        /**
         * Returns a cached instance if such is available otherwise a new one.
         *
         * @param rowIndex The row index at which the item is located.
         * @param rowSpan The number of rows the item spans.
         * @param columnIndex The column index at which the item is located.
         * @param columnSpan The number of columns the item spans.
         * @param heading Whether the item is a heading. This should be set to false and the newer
         *                {@link AccessibilityNodeInfoCompat#setHeading(boolean)} used to identify
         *                headings.
         * @param selected Whether the item is selected.
         * @return An instance.
         */

Video Tests

Header

2022-02-10.14-58-38.mp4

Behaviour without changing the values

2022-02-11.08-12-03.mp4

[fabOnReact]

Columns in FlatList with multiple columns are not announced

• A blind user uses TalkBack to navigate items in the FlatList.
• The FlatList has 2 columns and 6 rows (grid layout)
• TalkBack announces each of the items and their column and row number

Expected Result:
Talkback announces:

1. Marketplace row 1 column 1
2. FriendsList row 1 column 2
3. Groups row 2 column 1
4. Pages row 2 column 2
5. Fifth Item row 3 column 1
6. Sixth Item row 3 column 2

Actual Result:
Talkback announces:

1. Marketplace row 1
2. FriendsList
3. Groups row 2
4. Pages
5. Fifth Item row 3
6. Sixth Item

The TalkBack announcements are correct when using 1 column.

CLICK TO OPEN EXAMPLE

data

const DATA = [
  {
    id: 'bd7acbea-c1b1-46c2-aed5-3ad53abb28ba',
    title: 'Marketplace',
  },
  {
    id: '3ac68afc-c605-48d3-a4f8-fbd91aa97f63',
    title: 'FriendsList',
  },
  {
    id: '58694a0f-3da1-471f-bd96-145571e29d72',
    title: 'Groups',
  },
  {
    id: 'bd7acbea-c1b1-46c2-aed5-3ad53abb8bbb',
    title: 'Pages',
  },
  {
    id: '3ac68afc-c605-48d3-a4f8-fbd91aa97676',
    title: 'Fifth Item',
  },
  {
    id: '58694a0f-3da1-471f-bd96-145571e27234',
    title: 'Sixth Item',
  },
];
// and more items

JSX

<>
      <FlatList
        keyExtractor={item => item.id}
        numColumns={2}
        pagingEnabled={true}
        data={DATA}
        renderItem={renderItem}
        keyExtractor={item => item.toString()}
      />
</>

CLICK TO OPEN TESTS RESULTS - EXPECTED BEHAVIOUR

gridViewExpectedBehaviour.mp4

CLICK TO OPEN TESTS RESULTS - ACTUAL BEHAVIOUR

columnsInSpanNotAnnounced.mp4

[fabOnReact]

SOLUTION FOR ISSUE #6 (comment)

The issue is caused by this 2 lines. Commenting them and replacing them with a random integer number fixes the issue.

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/ReactAndroid/src/main/java/com/facebook/react/views/scroll/ReactScrollView.java#L210-L211

• The correct column numbers are announced when moving between different columns.
• The correct row numbers are announced when moving between different rows.

CLICK TO OPEN VIDEO - NOT ANNOUNCING COLUMN

notAnnouncingColumn.mp4

CLICK TO OPEN VIDEO - ANNOUNCING COLUMN

announcingColumn.mp4

CLICK TO OPEN VIDEO - ANNOUNCING COLUMN AND ROW

announcingColumnAndRow.mp4

CLICK TO OPEN VIDEO - SCROLL PAGE BEFORE

With 3 columns - Before the Change

verifyBehaviourWithoutChange.mp4

CLICK TO OPEN VIDEO - SCROLL PAGE AFTER

With 3 columns - After the Change

notAnnouncingCorrectListCount.mp4

CLICK TO OPEN MORE INFO

Removing the lines below (or replacing grid with list) would remove the row/column announcement.

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L409

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/Libraries/Lists/FlatList.js#L698

more info at #6 (comment)

The index is set correctly for columns 1 and 2, but TalkBack does not announce them.

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/Libraries/Lists/FlatList.js#L620

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L237

The reason is that our Java Implementation of FlatList numColums does not use GridView, but simply uses ScrollView formatting data as a 2 columns table.

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/Libraries/Lists/FlatList.js#L492-L506

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/ReactAndroid/src/main/java/com/facebook/react/views/scroll/ReactScrollView.java#L67

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/widget/ScrollView.java#L56-L80

https://github.com/aosp-mirror/platform_frameworks_base/blob/1ac46f932ef88a8f96d652580d8105e361ffc842/core/java/android/widget/GridView.java#L54-L69

https://github.com/fabriziobertoglio1987/react-native-notes/blob/875076952dc8658d2cd61026411dbc24e05e90a3/ReactAndroid/src/main/java/com/facebook/react/uimanager/ReactAccessibilityDelegate.java#L146

Removing the above line:

• TalkBack announces the item when scrolling pages
• Rows are not announced anymore

Adding the accessibilityRole grid adds the announcement for the rows.

CLICK TO OPEN TESTS RESULTS

removingRole.mp4

[fabOnReact]

Test Cases

2. ScreenReader announcing the wrong page ranges when scrolling up

The user scrolls up from the last page to the previous page.
The page includes items from 10 to 21.

CLICK TO OPEN PAGES PREVIEW

LAST PAGE	PREVIOUS PAGE

Expected Result:
Talkback announces the page Thirteen Item row 5 column 1 then says showing item 10 to 21 of 29

Actual Result:
Scrolling Down: Same as Expected Result
Scrolling Up: Talkback announces the page Thirteen Item row 5 column 1 then says showing item 10 to 19 of 29

CLICK TO OPEN SOURCECODE

<FlatList
  keyExtractor={item => item.id}
  numColumns={3}
  pagingEnabled={true}
  data={DATA}
  renderItem={renderItem}
  keyExtractor={item => item.toString()}
/>

missingLastTwoCells.mp4

[fabOnReact]

INVESTIGATING ISSUE #6 (comment)

Differences when scrolling with pagingEnabled without TalkBack and with TalkBack

Without Talkback (left):
The screen displays items 10 to 19 of 29

With Talkback (right):
The screen displays items 13 to 21 of 29

The ScrollView functionality with paging enabled is different between TalkBack and No-TalkBack.
The method getScrollDelta (inspired by pr react/react-native#25105) was designed not for TalkBack, but for a FlatList with pagingEnabled={true}.

https://github.com/fabriziobertoglio1987/react-native-notes/blob/79777766ae8eec8594c1f0947f28a8965ef26653/ReactAndroid/src/main/java/com/facebook/react/views/scroll/ReactScrollView.java#L401-L405

WITHOUT TALKBACK	WITH TALKBACK

VIDEO OF THE ISSUE

pagingEnableItemsOnScreen.mp4

CLICK TO OPEN OTHER NOTES

The issue is caused by isPartiallyScrolledInView not computing correctly visible childs of ScrollView.

https://github.com/fabriziobertoglio1987/react-native-notes/blob/79777766ae8eec8594c1f0947f28a8965ef26653/ReactAndroid/src/main/java/com/facebook/react/views/scroll/ReactScrollView.java#L158

https://github.com/fabriziobertoglio1987/react-native-notes/blob/79777766ae8eec8594c1f0947f28a8965ef26653/ReactAndroid/src/main/java/com/facebook/react/views/scroll/ReactScrollView.java#L407-L412

https://stackoverflow.com/a/4629167/7295772

The issue seems caused by isVisible

isVisible.mp4

Debugging isPartiallyScrolledInView

loggingIsVisible.mp4

[fabOnReact]

SOLUTION FOR ISSUE #6 (comment)

• Loop through each column
• Using height instead of width
• fix issues with isPartiallyScrolledInView and getScrollDelta
• find a callback to run event.setFromIndex(0) when ScrollView reaches top of the list
• Executing the logic for a specific event TYPE_WINDOW_CONTENT_CHANGED

Video of Debugging Values return in isMostlyScrolledInView for Item 10 and 20 while scrolling down

2022-02-15.13-44-27.mp4

Video of Debugging Values return in isMostlyScrolledInView for Item 10 and 20 while scrolling up

2022-02-15.14-00-22.mp4

Testing and troubleshooting the implementation

fabOnReact/react-native@048c5b9 and fabOnReact/react-native@602e81a

2022-02-15.16-53-32.mp4

[fabOnReact]

Test Cases

3. Not announcing correctly the first page items range

TalkBack announces the first page

Expected Result:
Showing items 1 to 19 of 29

Actual Result:
Showing items 10 of 29

CLICK TO OPEN VIDEO - NOT ANNOUNCING FIRST ITEM FIRST PAGE

notAnnouncingItem0.mp4

CLICK TO OPEN VIDEO - ANNOUNCING FIRST ITEM FIRST PAGE

announcingFirstItemFirstPage.mp4

Solved with cbe286d