Added accessibility APIs for introspecting interactive windows.

1. The old introspection model was allowing querying only the active window
   which is the one the user is touching or the focused one if no window is
   touched. This was limiting as auto completion drop downs were not inspectable,
   there was not way to know when the IME toggles, non-focusable windows were
   not inspectable if the user taps them as until a screen-reader starts
   introspecting the users finger is up, accessibility focus was limited to
   only one window and the user couldn't use gestures to visit the whole UI,
   and other things I can't remember right now.

   The new APIs allow getting all interactive windows, i.e. ones that a
   sighted user can interact with. This prevents an accessibility service
   from interacting with content a sighter user cannot. The list of windows
   can be obtained from an accessibility service or the host window from an
   accessibility node info. Introspecting windows obey the same rules for
   introspecting node, i.e. the service has to declare this capability
   in its manifest.

   When some windows change accessibility services receive a new type
   of event. Initially the types of windows is very limited. We provide
   the bounds in screen, layer, and some other properties which are
   enough for a client to determined the spacial and hierarchical
   relationship of the windows.

2. Update the documentation in AccessibilityService for newer event types.

3. LongArray was not removing elements properly.

4. Composite accessibility node ids were not properly constructed as they
   are composed of two ints, each taking 32 bits. However, the values for
   undefined were -1 so composing a 64 long from -1, -1 prevents from getting
   back these values when unpacking.

5. Some apps were generating inconsistent AccessibilityNodeInfo tree. Added
   a check that enforces such trees to be well formed on dev builds.

6. Removed an necessary code for piping the touch exploration state to
   the policy as it should just use the AccessibilityManager from context.

7. When view's visibility changed it was not firing an event to notify
   clients it disappeared/appeared. Also ViewGroup was sending accessibility
   events for changes if the view is included for accessibility but this is
   wrong as there may be a service that want all nodes, hence events from them.
   The accessibility manager service takes care of delivering events from
   not important for accessibility nodes only to services that want such.

8. Several places were asking for prefetching of sibling but not predecessor
   nodes which resulted in prefetching of unconnected subtrees.

9. The local AccessibilityManager implementation was relying on the backing
   service being ready when it is created but it can be fetched from a context
   before that. If that happens the local manager was in a broken state forever.
   Now it is more robust and starts working properly once the backing service
   is up. Several places were lacking locking.

bug:13331285

Change-Id: Ie51166d4875d5f3def8d29d77973da4b9251f5c8

4 files changed, 171 insertions, 34 deletions

diff --git a/core/java/android/accessibilityservice/AccessibilityService.java b/core/java/android/accessibilityservice/AccessibilityService.java
index 1e3d5be..b01d92c 100644
--- a/core/java/android/accessibilityservice/AccessibilityService.java
+++ b/core/java/android/accessibilityservice/AccessibilityService.java
@@ -23,14 +23,18 @@ import android.os.IBinder;
 import android.os.Looper;
 import android.os.Message;
 import android.os.RemoteException;
+import android.os.SystemClock;
 import android.util.Log;
 import android.view.KeyEvent;
 import android.view.accessibility.AccessibilityEvent;
 import android.view.accessibility.AccessibilityInteractionClient;
 import android.view.accessibility.AccessibilityNodeInfo;
+import android.view.accessibility.AccessibilityWindowInfo;
 
 import com.android.internal.os.HandlerCaller;
 
+import java.util.List;
+
 /**
  * An accessibility service runs in the background and receives callbacks by the system
  * when {@link AccessibilityEvent}s are fired. Such events denote some state transition
@@ -180,28 +184,37 @@ import com.android.internal.os.HandlerCaller;
  * event generation has settled down.</p>
  * <h3>Event types</h3>
  * <ul>
- * <li>{@link AccessibilityEvent#TYPE_VIEW_CLICKED}
- * <li>{@link AccessibilityEvent#TYPE_VIEW_LONG_CLICKED}
- * <li>{@link AccessibilityEvent#TYPE_VIEW_FOCUSED}
- * <li>{@link AccessibilityEvent#TYPE_VIEW_SELECTED}
- * <li>{@link AccessibilityEvent#TYPE_VIEW_TEXT_CHANGED}
- * <li>{@link AccessibilityEvent#TYPE_WINDOW_STATE_CHANGED}
- * <li>{@link AccessibilityEvent#TYPE_NOTIFICATION_STATE_CHANGED}
- * <li>{@link AccessibilityEvent#TYPE_TOUCH_EXPLORATION_GESTURE_START}
- * <li>{@link AccessibilityEvent#TYPE_TOUCH_EXPLORATION_GESTURE_END}
- * <li>{@link AccessibilityEvent#TYPE_VIEW_HOVER_ENTER}
- * <li>{@link AccessibilityEvent#TYPE_VIEW_HOVER_EXIT}
- * <li>{@link AccessibilityEvent#TYPE_VIEW_SCROLLED}
- * <li>{@link AccessibilityEvent#TYPE_VIEW_TEXT_SELECTION_CHANGED}
- * <li>{@link AccessibilityEvent#TYPE_WINDOW_CONTENT_CHANGED}
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_CLICKED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_LONG_CLICKED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_FOCUSED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_SELECTED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_TEXT_CHANGED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_WINDOW_STATE_CHANGED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_NOTIFICATION_STATE_CHANGED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_TOUCH_EXPLORATION_GESTURE_START}</li>
+ * <li>{@link AccessibilityEvent#TYPE_TOUCH_EXPLORATION_GESTURE_END}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_HOVER_ENTER}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_HOVER_EXIT}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_SCROLLED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_TEXT_SELECTION_CHANGED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_WINDOW_CONTENT_CHANGED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_ANNOUNCEMENT}</li>
+ * <li>{@link AccessibilityEvent#TYPE_GESTURE_DETECTION_START}</li>
+ * <li>{@link AccessibilityEvent#TYPE_GESTURE_DETECTION_END}</li>
+ * <li>{@link AccessibilityEvent#TYPE_TOUCH_INTERACTION_START}</li>
+ * <li>{@link AccessibilityEvent#TYPE_TOUCH_INTERACTION_END}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_ACCESSIBILITY_FOCUSED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_WINDOWS_CHANGED}</li>
+ * <li>{@link AccessibilityEvent#TYPE_VIEW_ACCESSIBILITY_FOCUS_CLEARED}</li>
  * </ul>
  * <h3>Feedback types</h3>
  * <ul>
- * <li>{@link AccessibilityServiceInfo#FEEDBACK_AUDIBLE}
- * <li>{@link AccessibilityServiceInfo#FEEDBACK_HAPTIC}
- * <li>{@link AccessibilityServiceInfo#FEEDBACK_AUDIBLE}
- * <li>{@link AccessibilityServiceInfo#FEEDBACK_VISUAL}
- * <li>{@link AccessibilityServiceInfo#FEEDBACK_GENERIC}
+ * <li>{@link AccessibilityServiceInfo#FEEDBACK_AUDIBLE}</li>
+ * <li>{@link AccessibilityServiceInfo#FEEDBACK_HAPTIC}</li>
+ * <li>{@link AccessibilityServiceInfo#FEEDBACK_AUDIBLE}</li>
+ * <li>{@link AccessibilityServiceInfo#FEEDBACK_VISUAL}</li>
+ * <li>{@link AccessibilityServiceInfo#FEEDBACK_GENERIC}</li>
+ * <li>{@link AccessibilityServiceInfo#FEEDBACK_BRAILLE}</li>
  * </ul>
  * @see AccessibilityEvent
  * @see AccessibilityServiceInfo
@@ -443,8 +456,41 @@ public abstract class AccessibilityService extends Service {
     }
 
     /**
+     * Gets the windows on the screen. This method returns only the windows
+     * that a sighted user can interact with, as opposed to all windows.
+     * For example, if there is a modal dialog shown and the user cannot touch
+     * anything behind it, then only the modal window will be reported
+     * (assuming it is the top one). For convenience the returned windows
+     * are ordered in a descending layer order, which is the windows that
+     * are higher in the Z-order are reported first.
+     * <p>
+     * <strong>Note:</strong> In order to access the windows your service has
+     * to declare the capability to retrieve window content by setting the
+     * {@link android.R.styleable#AccessibilityService_canRetrieveWindowContent}
+     * property in its meta-data. For details refer to {@link #SERVICE_META_DATA}.
+     * Also the service has to opt-in to retrieve the interactive windows by
+     * setting the {@link AccessibilityServiceInfo#FLAG_RETRIEVE_INTERACTIVE_WINDOWS}
+     * flag.
+     * </p>
+     *
+     * @return The windows if there are windows and the service is can retrieve
+     *         them, otherwise an empty list.
+     */
+    public List<AccessibilityWindowInfo> getWindows() {
+        return AccessibilityInteractionClient.getInstance().getWindows(mConnectionId);
+    }
+
+    /**
      * Gets the root node in the currently active window if this service
-     * can retrieve window content.
+     * can retrieve window content. The active window is the one that the user
+     * is currently touching or the window with input focus, if the user is not
+     * touching any window.
+     * <p>
+     * <strong>Note:</strong> In order to access the root node your service has
+     * to declare the capability to retrieve window content by setting the
+     * {@link android.R.styleable#AccessibilityService_canRetrieveWindowContent}
+     * property in its meta-data. For details refer to {@link #SERVICE_META_DATA}.
+     * </p>
      *
      * @return The root node if this service can retrieve window content.
      */
@@ -584,12 +630,13 @@ public abstract class AccessibilityService extends Service {
 
         static final int NO_ID = -1;
 
-        private static final int DO_SET_SET_CONNECTION = 10;
-        private static final int DO_ON_INTERRUPT = 20;
-        private static final int DO_ON_ACCESSIBILITY_EVENT = 30;
-        private static final int DO_ON_GESTURE = 40;
-        private static final int DO_CLEAR_ACCESSIBILITY_NODE_INFO_CACHE = 50;
-        private static final int DO_ON_KEY_EVENT = 60;
+        private static final int DO_SET_SET_CONNECTION = 1;
+        private static final int DO_ON_INTERRUPT = 2;
+        private static final int DO_ON_ACCESSIBILITY_EVENT = 3;
+        private static final int DO_ON_GESTURE = 4;
+        private static final int DO_CLEAR_ACCESSIBILITY_CACHE = 5;
+        private static final int DO_ON_KEY_EVENT = 6;
+        private static final int DO_ON_WINDOWS_CHANGED = 7;
 
         private final HandlerCaller mCaller;
 
@@ -624,8 +671,8 @@ public abstract class AccessibilityService extends Service {
             mCaller.sendMessage(message);
         }
 
-        public void clearAccessibilityNodeInfoCache() {
-            Message message = mCaller.obtainMessage(DO_CLEAR_ACCESSIBILITY_NODE_INFO_CACHE);
+        public void clearAccessibilityCache() {
+            Message message = mCaller.obtainMessage(DO_CLEAR_ACCESSIBILITY_CACHE);
             mCaller.sendMessage(message);
         }
 
@@ -635,6 +682,13 @@ public abstract class AccessibilityService extends Service {
             mCaller.sendMessage(message);
         }
 
+        @Override
+        public void onWindowsChanged(int[] windowIds) {
+            Message message = mCaller.obtainMessageO(DO_ON_WINDOWS_CHANGED, windowIds);
+            mCaller.sendMessage(message);
+        }
+
+        @Override
         public void executeMessage(Message message) {
             switch (message.what) {
                 case DO_ON_ACCESSIBILITY_EVENT: {
@@ -642,12 +696,19 @@ public abstract class AccessibilityService extends Service {
                     if (event != null) {
                         AccessibilityInteractionClient.getInstance().onAccessibilityEvent(event);
                         mCallback.onAccessibilityEvent(event);
-                        event.recycle();
+                        // Make sure the event is recycled.
+                        try {
+                            event.recycle();
+                        } catch (IllegalStateException ise) {
+                            /* ignore - best effort */
+                        }
                     }
                 } return;
+
                 case DO_ON_INTERRUPT: {
                     mCallback.onInterrupt();
                 } return;
+
                 case DO_SET_SET_CONNECTION: {
                     mConnectionId = message.arg1;
                     IAccessibilityServiceConnection connection =
@@ -658,18 +719,22 @@ public abstract class AccessibilityService extends Service {
                         mCallback.onSetConnectionId(mConnectionId);
                         mCallback.onServiceConnected();
                     } else {
-                        AccessibilityInteractionClient.getInstance().removeConnection(mConnectionId);
+                        AccessibilityInteractionClient.getInstance().removeConnection(
+                                mConnectionId);
                         AccessibilityInteractionClient.getInstance().clearCache();
                         mCallback.onSetConnectionId(AccessibilityInteractionClient.NO_ID);
                     }
                 } return;
+
                 case DO_ON_GESTURE: {
                     final int gestureId = message.arg1;
                     mCallback.onGesture(gestureId);
                 } return;
-                case DO_CLEAR_ACCESSIBILITY_NODE_INFO_CACHE: {
+
+                case DO_CLEAR_ACCESSIBILITY_CACHE: {
                     AccessibilityInteractionClient.getInstance().clearCache();
                 } return;
+
                 case DO_ON_KEY_EVENT: {
                     KeyEvent event = (KeyEvent) message.obj;
                     try {
@@ -685,9 +750,40 @@ public abstract class AccessibilityService extends Service {
                             }
                         }
                     } finally {
-                        event.recycle();
+                        // Make sure the event is recycled.
+                        try {
+                            event.recycle();
+                        } catch (IllegalStateException ise) {
+                            /* ignore - best effort */
+                        }
                     }
                 } return;
+
+                case DO_ON_WINDOWS_CHANGED: {
+                    final int[] windowIds = (int[]) message.obj;
+
+                    // Update the cached windows first.
+                    // NOTE: The cache will hold on to the windows so do not recycle.
+                    if (windowIds != null) {
+                        AccessibilityInteractionClient.getInstance().removeWindows(windowIds);
+                    }
+
+                    // Let the client know the windows changed.
+                    AccessibilityEvent event = AccessibilityEvent.obtain(
+                            AccessibilityEvent.TYPE_WINDOWS_CHANGED);
+                    event.setEventTime(SystemClock.uptimeMillis());
+                    event.setSealed(true);
+
+                    mCallback.onAccessibilityEvent(event);
+
+                    // Make sure the event is recycled.
+                    try {
+                        event.recycle();
+                    } catch (IllegalStateException ise) {
+                        /* ignore - best effort */
+                    }
+                } break;
+
                 default :
                     Log.w(LOG_TAG, "Unknown message type " + message.what);
             }
diff --git a/core/java/android/accessibilityservice/AccessibilityServiceInfo.java b/core/java/android/accessibilityservice/AccessibilityServiceInfo.java
index bdc4fdd..4f9ba59 100644
--- a/core/java/android/accessibilityservice/AccessibilityServiceInfo.java
+++ b/core/java/android/accessibilityservice/AccessibilityServiceInfo.java
@@ -284,6 +284,27 @@ public class AccessibilityServiceInfo implements Parcelable {
     public static final int FLAG_REQUEST_FILTER_KEY_EVENTS = 0x00000020;
 
     /**
+     * This flag indicates to the system that the accessibility service wants
+     * to access content of all interactive windows. An interactive window is a
+     * window that can be touched by a sighted user when explore by touch is not
+     * enabled. If this flag is not set your service will not receive
+     * {@link android.view.accessibility.AccessibilityEvent#TYPE_WINDOWS_CHANGED}
+     * events, calling AccessibilityService{@link AccessibilityService#getWindows()
+     * AccessibilityService.getWindows()} will return an empty list, and {@link
+     * AccessibilityNodeInfo#getWindow() AccessibilityNodeInfo.getWindow()} will
+     * return null.
+     * <p>
+     * Services that want to set this flag have to declare the capability
+     * to retrieve window content in their meta-data by setting the attribute
+     * {@link android.R.attr#canRetrieveWindowContent canRetrieveWindowContent} to
+     * true, otherwise this flag will be ignored. For how to declare the meta-data
+     * of a service refer to {@value AccessibilityService#SERVICE_META_DATA}.
+     * </p>
+     * @see android.R.styleable#AccessibilityService_canRetrieveWindowContent
+     */
+    public static final int FLAG_RETRIEVE_INTERACTIVE_WINDOWS = 0x00000040;
+
+    /**
      * The event types an {@link AccessibilityService} is interested in.
      * <p>
      *   <strong>Can be dynamically set at runtime.</strong>
@@ -302,6 +323,15 @@ public class AccessibilityServiceInfo implements Parcelable {
      * @see android.view.accessibility.AccessibilityEvent#TYPE_VIEW_SCROLLED
      * @see android.view.accessibility.AccessibilityEvent#TYPE_VIEW_TEXT_SELECTION_CHANGED
      * @see android.view.accessibility.AccessibilityEvent#TYPE_WINDOW_CONTENT_CHANGED
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_TOUCH_INTERACTION_START
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_TOUCH_INTERACTION_END
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_ANNOUNCEMENT
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_GESTURE_DETECTION_START
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_GESTURE_DETECTION_END
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_VIEW_ACCESSIBILITY_FOCUSED
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_VIEW_ACCESSIBILITY_FOCUS_CLEARED
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_VIEW_TEXT_TRAVERSED_AT_MOVEMENT_GRANULARITY
+     * @see android.view.accessibility.AccessibilityEvent#TYPE_WINDOWS_CHANGED
      */
     public int eventTypes;
 
@@ -354,6 +384,7 @@ public class AccessibilityServiceInfo implements Parcelable {
      * @see #FLAG_REQUEST_ENHANCED_WEB_ACCESSIBILITY
      * @see #FLAG_REQUEST_FILTER_KEY_EVENTS
      * @see #FLAG_REPORT_VIEW_IDS
+     * @see #FLAG_RETRIEVE_INTERACTIVE_WINDOWS
      */
     public int flags;
 
@@ -449,7 +480,7 @@ public class AccessibilityServiceInfo implements Parcelable {
                     com.android.internal.R.styleable.AccessibilityService_accessibilityFeedbackType,
                     0);
             notificationTimeout = asAttributes.getInt(
-                    com.android.internal.R.styleable.AccessibilityService_notificationTimeout, 
+                    com.android.internal.R.styleable.AccessibilityService_notificationTimeout,
                     0);
             flags = asAttributes.getInt(
                     com.android.internal.R.styleable.AccessibilityService_accessibilityFlags, 0);
@@ -861,6 +892,8 @@ public class AccessibilityServiceInfo implements Parcelable {
                 return "FLAG_REPORT_VIEW_IDS";
             case FLAG_REQUEST_FILTER_KEY_EVENTS:
                 return "FLAG_REQUEST_FILTER_KEY_EVENTS";
+            case FLAG_RETRIEVE_INTERACTIVE_WINDOWS:
+                return "FLAG_RETRIEVE_INTERACTIVE_WINDOWS";
             default:
                 return null;
         }
diff --git a/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl b/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
index c5e3d43a..edd8727 100644
--- a/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
+++ b/core/java/android/accessibilityservice/IAccessibilityServiceClient.aidl
@@ -18,6 +18,7 @@ package android.accessibilityservice;
 
 import android.accessibilityservice.IAccessibilityServiceConnection;
 import android.view.accessibility.AccessibilityEvent;
+import android.view.accessibility.AccessibilityWindowInfo;
 import android.view.KeyEvent;
 
 /**
@@ -35,7 +36,9 @@ import android.view.KeyEvent;
 
     void onGesture(int gesture);
 
-    void clearAccessibilityNodeInfoCache();
+    void clearAccessibilityCache();
 
     void onKeyEvent(in KeyEvent event, int sequence);
+
+    void onWindowsChanged(in int[] changedWindowIds);
 }
diff --git a/core/java/android/accessibilityservice/IAccessibilityServiceConnection.aidl b/core/java/android/accessibilityservice/IAccessibilityServiceConnection.aidl
index 3df06b5..5f7a17d 100644
--- a/core/java/android/accessibilityservice/IAccessibilityServiceConnection.aidl
+++ b/core/java/android/accessibilityservice/IAccessibilityServiceConnection.aidl
@@ -21,6 +21,7 @@ import android.accessibilityservice.AccessibilityServiceInfo;
 import android.view.MagnificationSpec;
 import android.view.accessibility.AccessibilityNodeInfo;
 import android.view.accessibility.IAccessibilityInteractionConnectionCallback;
+import android.view.accessibility.AccessibilityWindowInfo;
 
 /**
  * Interface given to an AccessibilitySerivce to talk to the AccessibilityManagerService.
@@ -53,6 +54,10 @@ interface IAccessibilityServiceConnection {
         int action, in Bundle arguments, int interactionId,
         IAccessibilityInteractionConnectionCallback callback, long threadId);
 
+    AccessibilityWindowInfo getWindow(int windowId);
+
+    List<AccessibilityWindowInfo> getWindows();
+
     AccessibilityServiceInfo getServiceInfo();
 
     boolean performGlobalAction(int action);