M5Core2 library fork that supports multi-touch and provides Arduino-style virtual buttons [update: and gestures and events] [update2: MERGED !]

felmue

Hello @Rop

Thank you for the update. Unfortunately no luck for me. When I run your code with the latest M5Core2 from your visual branch I get a crash as soon as the code tries to instantiate the button in screen1() function.

When I declare the buttons globally the code works, but as you mentioned they don't go away.

I wonder if the button object is too large to be declared in a function?

Thanks
Felix

felmue

Hi @Rop

Update2: I've created a pull-request with below fixes for your convenience.

https://github.com/ropg/M5Core2/pull/2

I have found the reason for the crash: drawFn is not NULL, yet not set correctly which causes the crash when calling drawFn(this, bc) on line 212.

void Button::draw(ButtonColors bc) {
  // use locally set draw function if aplicable, global one otherwise
  if (drawFn) {
    drawFn(this, bc);

I've fixed it by modifying two lines (137, 157) in M5Button.h and initialising drawFn with nullptr.

void (*drawFn)(Button* b, ButtonColors bc) = nullptr;

Update: Just found another crash when using new and delete. Fixed by setting _freeFont to nullptr (lines 108 and 125).

const GFXfont* _freeFont = nullptr;

Thanks
Felix

Rop

@felmue Thanks! I fixed it but in the cpp and not in the .h. I prefer to not have too much in the header files.

The version that's online now (I merged 'visual' into master on my repo) has fixes. Also there's now a copy of the original M5Stack library on my Github as well, it has the identical M5Button library file with Buttons and Events. Allows you to compile

#include <M5Stack.h>

void setup() {
  M5.begin();
  M5.BtnA.setLabel("Test");
  M5.BtnB.setLabel("Wow !");
  M5.BtnC.setLabel("Amazing !");
  M5.BtnA.off = M5.BtnB.off = M5.BtnC.off = {BLUE, WHITE, NODRAW};
  M5.BtnA.on = M5.BtnB.on = M5.BtnC.on = {RED, WHITE, NODRAW};
  M5.Events.addHandler(eventDisplay);
  M5.Buttons.draw();
}

void loop() {
  M5.update();
}

void eventDisplay(Event& e) {
  Serial.printf("\n%-12s %-18s", e.typeName(), e.objName());
  if (e.type == E_RELEASE || e.type == E_PRESSED) Serial.printf("%5d ms", e.duration);
}

(I'll post this wider after a bit more testing and documenting.)

felmue

Hi @Rop

playing with your latest code. Cool stuff.

You probably might want to remove or comment the Serial.println() in below function.

bool Button::pressedFor(uint32_t ms) {
	Serial.println(_time);
	return (_state && _time - _lastChange >= ms);
}

Cheers
Felix

Rop

@felmue Removed that line.

I just updated my M5Stack classic version of the library. Buttons have a few new features, such as myButton.cancel() which stops any further high-level event processing for that button, only E_RELEASE will be sent after that. It's used internally for gestures, but you can also use it for something like this:

#include <M5Stack.h>

// Shortcuts: the & means it's a reference to the same object
Button& A = M5.BtnA;
Button& B = M5.BtnB;
Button& C = M5.BtnC;

// New buttons, not tied to hardware pins, so off until set with setState()
Button AB = Button(55, 193, 102, 21, true, "BtnAB");
Button BC = Button(161, 193, 102, 21, true, "BtnBC");

void setup() {
  M5.begin();
  A.off = B.off = C.off = AB.off = BC.off = {BLUE, WHITE, NODRAW};
  A.on  = B.on  = C.on  = AB.on  = BC.on  = {RED,  WHITE, NODRAW};
  C.tapTime = 0;
  B.longPressTime = 700;
  M5.Events.addHandler(eventDisplay, E_ALL - E_TOUCH - E_RELEASE);
  M5.Events.addHandler(buttonDown, E_TOUCH);
  M5.Events.addHandler(buttonUp, E_RELEASE);
  M5.Buttons.draw();
}

void loop() {
  M5.update();
}

void buttonDown(Event& e) {
    if (A && B) {
        A.cancel();
        B.cancel();
        AB.setState(true);
    }
    if (B && C) {
        B.cancel();
        C.cancel();
        BC.setState(true);
    }
}

void buttonUp(Event& e) {
    if (!(A && B)) AB.setState(false);
    if (!(B && C)) BC.setState(false);
}


void eventDisplay(Event& e) {
  Serial.printf("\n%-14s %-18s", e.typeName(), e.objName());
  if (e.type == E_RELEASE || e.type == E_PRESSED) Serial.printf("%5d ms", e.duration);
}

These two M5ez-like combi-buttons now behave like real buttons, with doubleclick and all. The event viewer in this example does not present the - noisy - E_TOUCH and E_RELEASE events but all the others. As you can see one button has no taps, another fires a longpress event after 700 ms. Also if (myButton.isPressed() ... can now be shortened to if (myButton) ...

Things have become cleaner and simpler on the inside, and there is now only one state-machine for touch and regular buttons. Will update the Core2 version later or tomorrow.

Rop

New version is up.

Changes:

Introduced M5.background, the button object that is underneath all others and gets the events if nobody else does. So all events now have a button attached. E_BTNONLY pseudo-event removed.
Various state-machine tweaks.
Added keywords.txt for Arduino syntax highlighting.
Internally cleaner, some functionality moved from M5Touch to M5Buttons.
Added M5.Touch.dump() to see FT6336 registers.
M5.begin() now shows touch: FT6336 ready (fw id 0x10 rel 1, lib 0x300E) as one of the lines.
M5.Touch.stale is the location of the last past touch that is kept in memory of the FT6336. Can be used in light sleeping: only wake the ESP32 up two times a second and still see if someone touched.

Rop

This post is deleted!

Rop

I just pushed a new version that handles two fingers on same button (background is a button) properly.

With that version, try this (it will be an example later) to nicely visualise what the sensor sees.

 #include <M5Core2.h>

const int r = 50;

void setup() {
  M5.begin();
  M5.Events.addHandler(moveEvent, E_MOVE);
  M5.Events.addHandler(touchEvent, E_TOUCH);
  M5.Events.addHandler(releaseEvent, E_RELEASE);
}

void loop() {
  M5.update();
}

void moveEvent(Event &e) {
  M5.Lcd.drawCircle(e.from.x, e.from.y, r, BLACK);
  M5.Lcd.drawCircle(e.to.x, e.to.y, r, e.finger ? BLUE : RED);
}

void touchEvent(Event &e) {
  M5.Lcd.drawCircle(e.to.x, e.to.y, r, e.finger ? BLUE : RED);
}

void releaseEvent(Event &e) {
  M5.Lcd.drawCircle(e.to.x, e.to.y, r, BLACK);
}

Rop

Or, even shorter (and with slightly thicker and better visible circles):

#include <M5Core2.h>

const int r = 50;

void setup() {
  M5.begin();
  M5.Lcd.fillScreen(WHITE);
}

void loop() {
  M5.update();
  Event& e = M5.background.event;
  if (e & (E_MOVE | E_RELEASE)) circle(e & E_MOVE ? e.from : e.to, WHITE);
  if (e & (E_TOUCH | E_MOVE)) circle(e.to, e.finger ? BLUE : RED);
}

void circle(Point p, uint16_t c) {
  M5.Lcd.drawCircle(p.x, p.y, r, c);
  M5.Lcd.drawCircle(p.x, p.y, r + 2, c);
}

Rop

New version is up. You can now make gestures that have only directions. So that whole bit in the first part of the example is now:

// Defines gestures
Gesture swipeRight("swipe right", 160, RIGHT, 30, true);
Gesture swipeDown("swipe down", 120, DOWN, 30, true);
Gesture swipeLeft("swipe left", 160, LEFT, 30, true);
Gesture swipeUp("swipe up", 120, UP, 30, true);

The first number is the minimum distance, the RIGHT, LEFT, etc are just #defines for compass directions, the 30 thereafter is 'plus or minus 30 degrees'. The Gesture definitions have the same rot1 flag (the true at the end above) to denote that the direction is relative to the screen in rotation 1. You can still specify two zones for start and finish before the name of the gesture. If you want to specify just one, say ANYWHERE for the other. This direction stuff works because support for it is now in the underlying Point class. You can also say pointA.directionTo(PointB) or if (PointA.isDirectionTo(PointB, 0, 15)) ... to see if it's due north plus or minus 15 deg. Event has the same functions without the To because it has a start and end point in there already. So if you can want to see if a key was dragged upwards you just put something like that in the E_DRAGGED handler.
Key repeat with myButton.repeatDelay and repeatInterval, gives repeated E_PRESSING events.
The Events object is gone and its (few) functions are now functions of Buttons. Last time I move stuff around like that, I swear.
All my code is now formatted (with clang-format) for easier reading and reworked to be maximally understandable. Will work in some more documentation on how I did things, esp. rotation and other tricky bits. (The .clang-format config file is in the root, see the comment in there for usage. I chose Google's style guide with a few minor tweaks, because that was closest to what the rest of the library was doing already anyway.)
I expect to be submitting a Pull Request sometime in the next few days after I finish with the documentation.
I'd be very happy with any test reports.

vkichline

Rop, M5Button.cpp, line 176 is:

    Serial.println(_time);

...making it hard to catch any of my own debug output.
It would be much easier to open issues if you enabled Issues on https://github.com/ropg/M5Core2/tree/visual.

Rop

@vkichline Ooops, further up I had said:

The version that's online now (I merged 'visual' into master on my repo) has fixes

but I did not delete the 'visual' branch. My bad. Use the master branch. Much nicer. Some changes, you'll see in the example. (gestures now have directions as well as start and end zones, all optional.)

Will enable issues also...

vkichline

This won't compile for me, it conflicts with another library I need and admire: ezTime.
I get this over and over again:

C:/users/van/.platformio/lib/M5Core2/src/utility/M5Button.h:379:14: error: expected identifier before numeric constant
 #define NONE 0x0120  // Special color value: transparent
              ^
C:/users/van/.platformio/lib/ezTime/src/ezTime.h:69:2: note: in expansion of macro 'NONE'
  NONE, 
  ^

If this specific case is fixed, it's likely to cause conflicts elsewhere. Maybe name spacing could be used to make a macro as general as NONE more compatible.

vkichline

Rop, I've opened issues on your fork, and will continue to do so as I work through the library.
I wrote a goal-based UI evaluation tool for Touch and posted it here: https://github.com/vkichline/TouchGoal . Please give it a try!
It asks you to accomplish 16 Touch tasks in random order and keeps score, telling you what you missed at the end. You have up to 8 seconds for each task, but they end as soon as you successfully complete them, so it's pretty fast.
I had written some tools to spew event names, but didn't get a feel for how Touch worked in real life; this test tool provides that feedback.

I expected to have issues with gestures based on my serial spewing tests, but they seem quite solid. However, I usually fail to get all the double taps. Does double-tapping work better for you?
Age and fast-twitch motor neurons have a lot of sway over double-click success, it may be a bit too hot for me. Can the delay be adjusted?
If you can breeze through this test w/o failures, that would tell you the UI is doing what you expect it to do.

Rop

@vkichline Thanks!! Shouldn't have used NONE, neither here nor in ezTime. Its now NODRAW (again) in this library. i like TouchGoal. Fixed all the issues, I think. The documentation has been completely overhauled.

Rop

@vkichline In fact, shall we include TouchGoal as an example with the M5Core2 library?

vkichline

You're welcome to include it! Some additional tests for what shouldn't happen and a few comments about the best way to detect things would make it more valuable.
It's always nice to have a comprehensive suite to walk through after making 'improvements'. I could see TouchGoal being useful for testing subclasses of buttons, zones, etc.
When you think about it being used for testing subclasses, are there any additions you can think of? It's mostly positive testing now, few negative checks. (For example, you should not get this event before that event in this case...)

vkichline

Tried your changes and I can successfully double-tap every time now. Excellent!
I will add long presses to TouchGoal as soon as I figure them out. But today's my anniversary, so until later!

Rop

I think it's a wrap...

That is: on something this size there will always be more work to do and corners that could be even cleaner. But i think it works, does the job, is well-documented and at the very least doesn't break anything else.

So please give it a good test and tell me what y'all think, and then I'll make the PRs for M5Core2 (with M5Touch) and M5Stack (just m5Button).

@vkichline By all means wrap up TouchGoal so it can be included.

Everyone: please give the documentation a read and maybe write a tiny thing yourself to see if it works like you would expect... I'd love some more really basic simple examples if you can find the time...

vkichline

I've updated TouchGoal with long touches and better syntax. It's ready to go. BTW, I consistently get 100% scores now, so Touch is doing exactly what I want it to do.

I've added another testing program you may be interested in as well: TouchView.
It displays events to the screen as well as to the serial port. I like to test in an easy chair.
The A button turns E_MOVE an and off, B enables/disable long touches, and C toggles key repeat.
A good way to look for unexpected events. Looks clean to me.

I did report one bug I'd call a show stopper; #22. (I added a lot of issues, so it could get lost in the soup):

Run events_buttons_gestures_rotation
Double-tap top-left button. Turns blue. Good.
Swipe up. Display inverts. Good.
Double-tap button named "top-left" (now at bottom-right.) Button named "bottom-left" (in top-right position) becomes blue. Error.

Also, the file M5Touch.h didn't get updated; it has at least one example program (the first example in the top level file) which does not compile. I opened detailed issues at https://github.com/ropg/M5Core2/issues. It looks like a lot but most are typos.

I know I'm a noisy gong, always with the nits and quibbles, but on the whole this is excellent! I'm surprised at how reliably the gestures work; I took a stab at that myself and it was far from trivial. The two-finger support and rotation really go above and beyond as well! A huge leap forward for the Core2 platform.