US 20100211920A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2010/0211920 A1 Westerman et al. (43) Pub. Date: Aug. 19, 2010 (54) DETECTING AND INTERPRETING REAL-WORLD AND SECURITY GESTURES ON TOUCH AND HOVER SENSITIVE DEVICES (76) Inventors: Wayne Carl Westerman, San Francisco, CA (US); Myra Mary Haggerty, San Mateo, CA (U S) Correspondence Address: APPLE C/O MORRISON AND FOERSTER,LLP LOS ANGELES 555 WEST FIFTH STREET SUITE 3500 LOS ANGELES, CA 90013-1024 (US) (21) Appl. No.: 12/753,209 (22) Filed: Apr. 2, 2010 (62) Related US. Application Data Division of application No. 11/818,500,?led on Jun. 13, 2007. (60) Provisional application No. 60/879,191,?led on Jan. 6, 2007. Publication Classi?cation (51) Int. Cl. G06F 3/033 (2006.01) (52) US. Cl...... 715/863 (57) ABSTRACT Real-World gestures such as hand or?nger movements/ orientations that are generally recognized to mean certain things (e.g., an OK hand signal generally indicates an a?ir mative response) can be interpreted by a touch or hover sen sitive device to more e?iciently and accurately effect intended operations. These gestures can include, but are not limited to, OK gestures, grasp everything gestures, stamp of approval gestures, circle select gestures, X to delete gestures, knock to inquire gestures, hitchhiker directional gestures, and shape gestures. In addition, ges tures can be used to provide identi?cation and allow or deny access to applications,?les, and the like. FILE WINDOW FILE 1 FILE 2 FILE 3,F'LEV// FILE 5 FILE 6 506 501
Patent Application Publication Aug. 19, 2010 Sheet 1 0f 44 US 2010/0211920 A1 mm. mm.emewwo zéwomm,. ow \ momwwoomm 35% o: A. Sago. mm) [Y 56 [mm mm wmdhwwo All Om
Patent Application Publication Aug. 19, 2010 Sheet 2 0f 44 US 2010/0211920 A1.. I \ Pe?phgals Display Device,13. 'Z A ' _ MT Panel Processor Host 12 i > Processor,- 14 Mulli-touch subsystem 277 lgrogram tora e 159 RAM 16 MT High-Voltage Driver v _ Subsystem Decoder Level analog channel Driver Outputs Shiiler/ channels scan Logic 20 ' Driver > > logic > n 21 22 17 1s 19 '. i _ i MT Panel 23/ Row inputs control Signals Mulli-touch panel 24 XCsig i 26 Fig. 2
Patent Application Publication Aug. 19, 2010 Sheet 3 0f 44 US 2010/0211920 A1 100 x READ'IMAGE FROM TOUCHSCREEN CONVERT IMAGE TO COLLECTION OR LIST OF FEATURES ' K. o A Y \A O C) CLASSIFY AND GROUP FEATURES CALCULATE KEY PARAMETERS OF FEATURE GROUPS K; ASSOCIATE GROUP(S) TO U.l. ELEMENT(S) 112 PERFORM ACTION PROVIDE USER FEEDBACK FIG. 3
Patent Application Publication Aug. 19, 2010 Sheet 4 0f 44 US 2010/0211920 A1 TOUCH 1, 120\ I (ID1) ' TOUCH2 (ID2) A [Z1 1212A 1228 FIG. 4B
Patent Application Publication Aug. 19, 2010 Sheet 5 0f 44 US 2010/0211920 A1 130 130 FIG. 5
Patent Application Publication Aug. 19, 2010 Sheet 6 0f 44 US 2010/0211920 A1 152 < -- \\ RECEIVE GROUP OF FEATURES '. IS THERE A CHANGE IN # OF FEATURES? N0 YES 158 156 " \ CALCULATE CURRENT \ CALCULATE INITIAL PARAMETER VALUES PARAMETER VALUES REPORT INITIAL AND 160 > CURRENT / PARAMETER VALUES v - FIG. 6 END
Patent Application Publication Aug. 19, 2010 Sheet 7 0f 44 US 2010/0211920 A1 FILE 1 FILE 2 FILE 3 J00 V////\// FILE 5 FILE 6 FIG. 7A 501 FILE WINDOW 1 DESKTOP i FILE 1 502 FILE 2 H F LE3 503 F _ \ v ) \F\ LE WINDOWZ / FILE 4/ V500
Patent Application Publication Aug. 19, 2010 Sheet 8 0f 44 US 2010/0211920 A1 FILE WINDOW 501 FILE WINDOW FILE I OPEN 504 FILE 2 PRINT / FILE 3 F LE4 50o CUT COPY/f <// J, 505 FILE 5 CREATE H TCUT /506 FILE 6 DELETE
Patent Application Publication Aug. 19, 2010 Sheet 9 0f 44 US 2010/0211920 A1 FIG.7E FILE 1 FILE 2 FILE 3 ILE/5 FI/LE6 /F LE/ F LE8 F LE9 FILEIO FILE WINDOW fsos 501 502 J/_ I 720 TOUCH ASSOCIATED W/ HYPERLINK? OPFEN HYPERLINK DESTINATION FIG. 76
Patent Application Publication Aug. 19, 2010 Sheet 10 0f 44 US 2010/0211920 A1 DETECT PRESENCE OF FINGERS ON TOUCH SCREEN 710 712 ONE FINGER OR TWO FINGERS? ONE FINGER ASSOCIATED WITH A SELECTABLE OBJECT? TWO FINGERS TOUCH ASSOCIATED WITH A SELECTABLE OBJECT? YES SELECT OBJECT i 714 YES SELECT OBJECT DETECT A DOUBLE TAP? DETECT A TAP BY ONE FINGER? INVOKE DOUBLE CLICK ACTION INVOKE RIGHT CLICK ACTION 719 FIG. 7F 718 )
Patent Application Publication Aug. 19, 2010 Sheet 11 0f 44 US 2010/0211920 A1 SCROLLABLE ENVIRONMENT? 724 INVOKE SCROLLING ACTION SCROLL AT 1X SPEED IF ONE FINGER TOUCH DETECTED SCROLL AT 2X SPEED IF TWO FINGER TOUCH IS DETECTED /\ 725 FIG. 7H
Patent Application Publication Aug. 19, 2010 Sheet 12 0f 44 US 2010/0211920 A1 1x SCROLL f 727 0)) << / 726 727 mf \Jl <<o>> FIG.7J L728 I 501
Patent Application Publication Aug. 19, 2010 Sheet 13 0f 44 US 2010/0211920 A1 000 / 729 UUUUU UUUUU 730 731 731 m W :1 I: I: I: l Ll L_ l _] FIG. 7K 501
Patent Application Publication Aug. 19, 2010 Sheet 14 0f 44 US 2010/0211920 A1 /178 170 \\/ 176 172
. Patent Application Publication Aug. 19, 2010 Sheet 15 0f 44 US 2010/0211920 A1 59, <2: 22K NwFK. mi: $2. $.91. (#2
Patent Application Publication Aug. 19, 2010 Sheet 16 0f 44 US 2010/0211920 A1 /1s2 184B 184A 184A 184 FIG. 80 180 K 184A l 184A 1848 184A
Patent Application Publication Aug. 19, 2010 Sheet 17 0f 44 US 2010/0211920 A1 5&7.. N2
Patent Application Publication Aug. 19, 2010 Sheet 18 0f 44 US 2010/0211920 A1 182 / v 188/ 190 w. I ELEMENT FIG. 8E 180 K,...''C J?", k 188
Patent Application Publication Aug. 19, 2010 Sheet 19 0f 44 US 2010/0211920 A1 ///@ /// I FIG. 86 \_/ V CLICK V. FIG. 8F