actiontype = SCREEN_INTERACTION

performAction on accessibility node. Let them tap on a view in an app and Key Mapper will detect this. throw action error if isn't in the correct app.

• Have textinputlayouts with EditTexts and Switches inside to pick..

These are all compulsory...

• action (click, long click, scroll forward, scroll backward, focus, select, dismiss)
• package name
• UI element text
  optional...
• UI element id (viewIdResourceName). Not all views have this and only show this option on API 18+.
• UI element hinttext
• UI element tooltipText

Thank you for looking into this feature request, as well as the tap coordinates action (it's very cool). How feasible would an action that clicks a user provided/cropped .png file of an UI element be? E.g. the user gives Key Mapper a cropped screenshot of a button to tap, Key Mapper searches for it on the screen (assuming it has the accessibility service), and if found Key Mapper taps the center of it?

It is technically possible (not sure on the latest Android versions) but there is no elegant way to do it. Lets say the screen is just a picture, finding another picture of random resolution and accuracy in another picture isn't trivial. What is the use case for providing a cropped .png of a UI element? If the UI element is a standard UI element that subclasses View.java then the accessibility can find it with the accessibility nodes API thing.

Sorry I was unclear in my last reply, I thought "UI element" was a generic term rather than specific to Android. Some usecases I can think of that don't involve Android UI elements are tapping a button on a webpage (e.g. GH notifications) or a VNC client connected to a remote host. A .png search feature could be used as a fallback when a normal Android UI isn't being used

Yeah, UI element is a generic term. I'm not sure which terminology to use so I'll just use "UI element". The user will have to be a bit familiar with how accessibility service nodes work to understand how this feature works. It is up to the app developers to make their custom Views/UI elements behave nicely with accessibility services, which is why UI elements in some apps can't be interacted with by an accessibility service. The standard views already behave nicely like Buttons and RecyclerViews. Good point, there are useful use-cases for this but I am very unlikely to pursue it since it isn't trivial and there isn't much demand for it.

I thought I'd throw the idea out there. Thank you for the work you put into this great app.

Yup, thanks for the ideas