diff --git a/Final Project/README.md b/Final Project/README.md
deleted file mode 100644
index f02201b6aa..0000000000
--- a/Final Project/README.md
+++ /dev/null
@@ -1,37 +0,0 @@
-# Final Project
-
-Using the tools and techniques you learned in this class, design, prototype and test an interactive device.
-
-Project Github page set up - May 3
-
-Functional check-off - May 10
-
-Final Project Presentations (video watch party) - May 12
-
-Final Project Documentation due - May 19
-
-
-
-## Objective
-
-The goal of this final project is for you to have a fully functioning and well-designed interactive device of your own design.
-
-## Description
-Your project is to design and build an interactive device to suit a specific application of your choosing.
-
-## Deliverables
-
-1. Documentation of design process
-2. Archive of all code, design patterns, etc. used in the final design. (As with labs, the standard should be that the documentation would allow you to recreate your project if you woke up with amnesia.)
-3. Video of someone using your project (or as safe a version of that as can be managed given social distancing)
-4. Reflections on process (What have you learned or wish you knew at the start?)
-
-
-## Teams
-
-You can and are not required to work in teams. Be clear in documentation who contributed what. The total project contributions should reflect the number of people on the project.
-
-## Examples
-
-[Here is a list of good final projects from previous classes.](https://github.com/FAR-Lab/Developing-and-Designing-Interactive-Devices/wiki/Previous-Final-Projects)
-This version of the class is very different, but it may be useful to see these.
diff --git a/Lab 1/demo.py b/Lab 1/demo.py
index 47c742f60e..893d64ca22 100644
--- a/Lab 1/demo.py
+++ b/Lab 1/demo.py
@@ -1,3 +1,2 @@
-## This line is comment
-## for demo during the class
+
print("Hello World")
diff --git a/Lab 2/Extending the Pi.md b/Lab 2/Extending the Pi.md
index ae519b5f1a..4abe07a108 100644
--- a/Lab 2/Extending the Pi.md
+++ b/Lab 2/Extending the Pi.md
@@ -1,6 +1,6 @@
# Extending the Pi
-To extend the Pi, we are using breakout boards that connect to the PI using a standard communication bus (I2C)(https://learn.sparkfun.com/tutorials/i2c/all). [StemmaQT](https://learn.adafruit.com/introducing-adafruit-stemma-qt/what-is-stemma) and [Qwiic](https://www.sparkfun.com/qwiic#overview) use a standardized 4-pin connector to connect devices using the I2C protocol.
+To extend the Pi, we are using breakout boards that connect to the PI using a standard communication bus [I2C](https://learn.sparkfun.com/tutorials/i2c/all). [StemmaQT](https://learn.adafruit.com/introducing-adafruit-stemma-qt/what-is-stemma) and [Qwiic](https://www.sparkfun.com/qwiic#overview) use a standardized 4-pin connector to connect devices using the I2C protocol.
The StemmaQT and I2C parts often have a fixed I2C address; to differentiate between similar parts, the devices often have pads that allow additional bits to be pulled high or low. The addresses are in [hexidecimal](https://learn.sparkfun.com/tutorials/hexadecimal/introduction) format, things like `0x6f`. This is the hexadecimal (or hex) representation for the decimal number `111` which is represented as `1101111` in binary. You are not expected to make any kinds of conversions but should have some conceptual grasp that a hex value is just a number shown another way. [This Python library](https://towardsdatascience.com/binary-hex-and-octal-in-python-20222488cee1) will assist you if you need help manipulating hexidecimal numbers.
@@ -30,16 +30,17 @@ As before, connect to your Pi and activate your virtual environment.
ssh pi@ixe00
pi@ixe00:~ $ source circuitpython/bin/activate
(circuitpython) pi@ixe00:~ $
-
```
-Navigate to your interactive lab hub, pull changes from upstream, and install new packages. If you have [merge conflicts](https://www.atlassian.com/git/tutorials/using-branches/merge-conflicts), you need to resolve them. If you've never done this before ask people in your group for help. Otherwise ask yout TA!
+On the pi, Navigate to your interactive lab hub, pull changes from upstream, and install new packages. If you have [merge conflicts](https://www.atlassian.com/git/tutorials/using-branches/merge-conflicts), you need to resolve them. If you've never done this before ask people in your group for help.
```
(circuitpython) pi@ixe00:~$ cd Interactive-Lab-Hub
-(circuitpython) pi@ixe00:~/Interactive-Lab-Hub $ git pull upstream Spring2021
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub $ git remote add upstream https://github.com/FAR-Lab/Interactive-Lab-Hub.git
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub $ git pull upstream Fall2021
(circuitpython) pi@ixe00:~/Interactive-Lab-Hub $ git add .
-(circuitpython) pi@ixe00:~/Interactive-Lab-Hub $ git commit -m'merge'
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub $ git commit -m "merge"
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub $ git push
(circuitpython) pi@ixe00:~/Interactive-Lab-Hub $ cd Lab\ 2/
(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 2 $ pip install -r requirements.txt
```
@@ -55,21 +56,10 @@ Try running `python library_example.py`.
Some important things to note from the code:
* We create an I2C device to handle communication with the pi.
- * We then scan for devices on the bus
+ * We then scan for devices on the bus.
* We check if `default_addr = 0x6f` is listed in the found devices. This is the address your button comes programmed with, you can also change this and have it store the update on the button.
* Once we initialize the I2C_Button object the rest of the code shows us some of the builtin capabilities.
-## Connecting a Sensor
-
-Your kit is full of sensors! Look up what they can do and feel free to ask your TAs, We love to talk sensors. We will go further in depth into sensors in the coming weeks, but we put this small sample here to demonstrate how you can get sensor data if you want to use it for your project this week.
-
-We are going to connect the [Adafruit APDS9960 Proximity, Light, RGB, and Gesture Sensor](https://www.adafruit.com/product/3595). You can leave the button plugged in and daisy-chain the sensor, this is part of the magic of I2C.
-
-
-
-
-Now run `python proximity.py`.
-
## Under the I2C curtain (optional: complete only after working on your projects in groups)
@@ -79,7 +69,6 @@ Run the file `I2C_scan.py` and the output should look like:
(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 2 $ python I2C_scan.py
I2C ok!
I2C addresses found: []
-
```
Now plug the other end of the cable into the ports on the right of the button board. The pwr LED should turn on. Run the file again and you should see the device ID. You can also try daisy chaining multiple devices and sensors and running again.
@@ -100,3 +89,24 @@ Use a higher level device interface can make reading and writing registers for I
```
56 write_register(device, STATUS, 0)
```
+
+#### Connecting more that one button
+
+The more buttons the merrier! ...but how do you control them individually when they are come with the same default I2C address `0x6f`?
+
+Option 1 - Software: Look into the [list of registers](https://cdn.sparkfun.com/assets/learn_tutorials/1/1/0/8/Qwiic_Button_I2C_Register_Map.pdf) of the buttons again, is it possible to change the I2C address through software progrmming?
+
+Option 2 - Hardware: Look at the bottom right corner of the back of the button, you should be able to local a sign of ADR with A0 - A3 jumpers. By solding these I2C address jumpers, you can actually change the address directly! Check [here](https://learn.sparkfun.com/tutorials/sparkfun-qwiic-button-hookup-guide/all) to see how the I2C address change!
+
+
+
+## Connecting a Sensor
+
+Your kit is full of sensors! Look up what they can do and feel free to ask your TAs, we love to talk sensors. We will go further in depth into sensors in the coming weeks, but we put this small sample here to demonstrate how you can get sensor data if you want to use it for your project this week.
+
+We are going to connect the [Adafruit APDS9960 Proximity, Light, RGB, and Gesture Sensor](https://www.adafruit.com/product/3595). You can leave the button plugged in and daisy-chain the sensor, this is part of the magic of I2C.
+
+
+
+
+Now run `python proximity.py`. What did you see? Check out [here](https://learn.adafruit.com/adafruit-apds9960-breakout/circuitpython) to learn more about the sensor and think about how you might be able to apply it in the future projects!
diff --git a/Lab 2/README.md b/Lab 2/README.md
index 7d9b43c5aa..0a4f219ad6 100644
--- a/Lab 2/README.md
+++ b/Lab 2/README.md
@@ -1,6 +1,6 @@
-# The Clock of Pi
+# Interactive Prototyping: The Clock of Pi
-Does it feel like time is moving strangely during the pandemic?
+Does it feel like time is moving strangely during this semester?
For our first Pi project, we will pay homage to the [timekeeping devices of old](https://en.wikipedia.org/wiki/History_of_timekeeping_devices) by making simple clocks.
@@ -11,24 +11,31 @@ Be generous in acknowledging their contributions! And also recognizing any other
## Prep
-[Lab prep](prep.md) is extra long this week! Make sure you read it over in time to prepare for lab on Wednesday.
+[Lab prep](prep.md) is extra long this week! Make sure you read it over in time to prepare for lab on Thursday.
### Get your kit
If you are remote but in the US, let the teaching team know you need the parts mailed.
-If you are in New York, you can come to the campus and pick up your parts. If you have not picked up your parts by class you should come to Tata 351.
+
+If you are in New York, you can come to the campus and pick up your parts. If you have not picked up your parts by Thursday lab you should come to Tata 351.
+
### Set up your Lab 2
1. [Pull changes from the Interactive Lab Hub](https://github.com/FAR-Lab/Developing-and-Designing-Interactive-Devices/blob/2021Fall/readings/Submitting%20Labs.md#to-pull-lab-updates) so that you have your own copy of Lab 2 on your own lab hub. (This may have to be done again at the start of lab on Thursday.)
-
+
If you are organizing your Lab Hub through folder in local machine, go to terminal, cd into your Interactive-Lab-Hub folder and run:
```
Interactive-Lab-Hub $ git remote add upstream https://github.com/FAR-Lab/Interactive-Lab-Hub.git
- Interactive-Lab-Hub $ git pull upstream Spring2021
+ Interactive-Lab-Hub $ git pull upstream Fall2022
+ ```
+
+ The reason why we are adding a upstream with **course lab-hub** instead of yours is because the local Interactive-Lab-Hub folder is linked with your own git repo already. Try typing ``git remote -v`` and you should see there is the origin branch with your own git repo. We here add the upstream to get latest updates from the teaching team by pulling the **course lab-hub** to your local machine. After your local folder got the latest updates, push them to your remote git repo by running:
+
+ ```
Interactive-Lab-Hub $ git add .
- Interactive-Lab-Hub $ git commit -m'merge'
+ Interactive-Lab-Hub $ git commit -m "message"
Interactive-Lab-Hub $ git push
```
Your local and remote should now be up to date with the most recent files.
@@ -73,7 +80,8 @@ pi@ixe00:~ $ source circuitpython/bin/activate
### Setup Personal Access Tokens on GitHub
The support for password authentication of GitHub was removed on August 13, 2021. That is, in order to link and sync your own lab-hub repo with your Pi, you will have to set up a "Personal Access Tokens" to act as the password for your GitHub account on your Pi when using git command, such as `git clone` and `git push`.
-Following the steps listed [here](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) from GitHub to set up one for your Pi to use. Depends on your preference, you can set up and select the scopes, or permissions, you'd like to grant the token your Pi is going to use. This token will act as your GitHub password later when you use the terminal on you Pi to sync files with your lab-hub repo.
+Following the steps listed [here](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) from GitHub to set up a token. Depends on your preference, you can set up and select the scopes, or permissions, you would like to grant the token. This token will act as your GitHub password later when you use the terminal on your Pi to sync files with your lab-hub repo.
+
## Part B.
### Try out the Command Line Clock
@@ -149,13 +157,21 @@ You can look in `image.py` for an example of how to display an image on the scre
Work on `screen_clock.py`, try to show the time by filling in the while loop (at the bottom of the script where we noted "TODO" for you). You can use the code in `cli_clock.py` and `stats.py` to figure this out.
### How to Edit Scripts on Pi
-One of the ways for you to edit scripts on Pi through terminal is using [`nano`](https://linuxize.com/post/how-to-use-nano-text-editor/) command. You can go into the `screen_clock.py` by typing the follow command line:
+Option 1. One of the ways for you to edit scripts on Pi through terminal is using [`nano`](https://linuxize.com/post/how-to-use-nano-text-editor/) command. You can go into the `screen_clock.py` by typing the follow command line:
```
(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 2 $ nano screen_clock.py
```
You can make changes to the script this way, remember to save the changes by pressing `ctrl-o` and press enter again. You can press `ctrl-x` to exit the nano mode. There are more options listed down in the terminal you can use in nano.
-Another way for you to edit scripts is to use VNC on your laptop to remotely connect your Pi. Try to open the files directly like what you will do with your laptop and edit them.
+Option 2. Another way for you to edit scripts is to use VNC on your laptop to remotely connect your Pi. Try to open the files directly like what you will do with your laptop and edit them. Since the default OS we have for you does not come up a python programmer, you will have to install one yourself otherwise you will have to edit the codes with text editor. [Thonny IDE](https://thonny.org/) is a good option for you to install, try run the following command lines in your Pi's ternimal:
+
+ ```
+ pi@ixe00:~ $ sudo apt install thonny
+ pi@ixe00:~ $ sudo apt update && sudo apt upgrade -y
+ ```
+
+Now you should be able to edit python scripts with Thonny on your Pi.
+
## Part E.
@@ -165,6 +181,9 @@ Does time have to be linear? How do you measure a year? [In daylights? In midni
Can you make time interactive? You can look in `screen_test.py` for examples for how to use the buttons.
+**We strongly discourage and will reject the results of literal digital or analog clock display.**
+
+
\*\*\***A copy of your code should be in your Lab 2 Github repo.**\*\*\*
After you edit and work on the scripts for Lab 2, the files should be upload back to your own GitHub repo! You can push to your personal github repo by adding the files here, commiting and pushing.
@@ -189,7 +208,8 @@ After that, Git will ask you to login to your GitHub account to push the updates
# Prep for Part 2
-1. Pick up remaining parts for kit.
+1. Pick up remaining parts for kit on Thursday lab class. Check the updated [parts list inventory](partslist.md) and let the TA know if there is any part missing.
+
2. Look at and give feedback on the Part G. for at least 2 other people in the class (and get 2 people to comment on your Part G!)
diff --git a/Lab 2/partslist.md b/Lab 2/partslist.md
index aee3e02735..e0e9f415fc 100644
--- a/Lab 2/partslist.md
+++ b/Lab 2/partslist.md
@@ -1,2 +1,48 @@
# Parts list inventory
+## Parts in the kit by Thursday class Sep. 16th
+
+1 x [Raspberry Pi 4 Computer Kit](https://www.amazon.com/Raspberry-USB-C-Adapters-Vilros-Quickstart/dp/B089ZSGF8M/ref=sr_1_14?dchild=1&keywords=vilros+pi+4+heatsink&qid=1630441198&sr=8-14)
+
+1 x [32GB MicroSD Cards w/ Card Reader](https://www.digikey.com/en/products/detail/seeed-technology-co-ltd/112990066/10290294)
+
+1 x [Raspberry Pi 4 Case](https://www.adafruit.com/product/4301)
+
+1 x [Raspberry Pi 4 Power Supply with USB C](https://www.adafruit.com/product/4298)
+
+1 x [Copper Foil Tape](https://www.amazon.com/Conductive-Shielding-Repellent-Electrical-Grounding/dp/B0741ZRP4W/ref=sr_1_5?dchild=1&keywords=conductive+copper+tape&qid=1628142003&sr=8-5)
+
+1 x [Alligator Clips](https://www.amazon.com/WGGE-WG-026-Pieces-Colors-Alligator/dp/B06ZXSCLDH/ref=sr_1_3?dchild=1&keywords=alligator%2Bclips&qid=1611164254&sr=8-3&th=1)
+
+1 x [Micro Servo Motor SG51](https://www.amazon.com/Smraza-Helicopter-Airplane-Control-Arduino/dp/B07L2SF3R4/ref=sr_1_3?dchild=1&keywords=micro+servo+motor&qid=1628142137&sr=8-3)
+
+1 x [Qwiic Servo Controller](https://www.sparkfun.com/products/16773)
+
+1 x [Adafruit Mini PiTFT](https://www.adafruit.com/product/4393)
+
+1 x [Adafruit APDS9960 Proximity, Light, RBG, Gesture Sensor](https://www.adafruit.com/product/3595)
+
+1 x [Adafruit I2C QT Rotery Encoder](https://www.adafruit.com/product/4991)
+
+1 x [Adafruit MPU-6050 6 DoF Accel & Gyro Sensor](https://www.adafruit.com/product/3886)
+
+1 x [Adafruit MPR121 Capacitive Touch Sensor QT](https://www.adafruit.com/product/4830)
+
+
+## Parts in the kit by Thursday class Sep. 22th
+
+1 x [SparkFun Qwiic Cable Kit](https://www.sparkfun.com/products/15081)
+
+1 x [SparkFun Qwiic Joystick](https://www.sparkfun.com/products/15168)
+
+1 x [SparkFun Qwiic OLED Display](https://www.sparkfun.com/products/17153)
+
+1 x [SparkFun Qwiic Button - Red LED](https://www.sparkfun.com/products/15932)
+
+1 x [SparkFun Qwiic Button - Green LED](https://www.sparkfun.com/products/16842)
+
+1 x [HD Webcam](https://www.amazon.com/Webcam-Speakers-2021-Microphone-Compatible/dp/B08QRGCW6K/ref=pd_lpo_147_t_0/134-8136290-7363941?_encoding=UTF8&pd_rd_i=B08QRGCW6K&pd_rd_r=a70c666b-ee4f-43c8-ae23-0be491319204&pd_rd_w=AEVHh&pd_rd_wg=iHZYM&pf_rd_p=fb1e266d-b690-4b4f-b71c-bd35e5395976&pf_rd_r=0CZ6ZZZ0XRRVRAGSGHDK&psc=1&refRID=0CZ6ZZZ0XRRVRAGSGHDK#customerReviews)
+
+
+## Missing Parts here
+\*\*\***List down parts you did not get from the kit here**\*\*\*
diff --git a/Lab 3/README.md b/Lab 3/README.md
index 947dd458f2..9d9c13c479 100644
--- a/Lab 3/README.md
+++ b/Lab 3/README.md
@@ -1,106 +1,162 @@
-# You're a wizard, [Student Name Here]
-
-
-
-In this lab, we want you to practice wizarding an interactive device as discussed in class. We will focus on audio as the main modality for interaction but there is no reason these general techniques can't extend to video, haptics or other interactive mechanisms. In fact, you are welcome to add those to your project if they enhance your design.
-
-
-## Text to Speech and Speech to Text
-
-In the home directory of your Pi there is a folder called `text2speech` containing some shell scripts.
-
-```
-pi@ixe00:~/text2speech $ ls
-Download festival_demo.sh GoogleTTS_demo.sh pico2text_demo.sh
-espeak_demo.sh flite_demo.sh lookdave.wav
-
-```
-
-you can run these examples by typing
-`./espeakdeom.sh`. Take some time to look at each script and see how it works. You can see a script by typing `cat filename`
-
-```
-pi@ixe00:~/text2speech $ cat festival_demo.sh
-#from: https://elinux.org/RPi_Text_to_Speech_(Speech_Synthesis)#Festival_Text_to_Speech
-
-echo "Just what do you think you're doing, Dave?" | festival --tts
-
-```
-
-You can also play audio files directly with `aplay filename`.
-
-After looking through this folder do the same for the `speech2text` folder. In particular, look at `test_words.py` and make sure you understand how the vocab is defined. Then try `./vosk_demo_mic.sh`
-
-## Serving Pages
-
-In Lab 1 we served a webpage with flask. In this lab you may find it useful to serve a webpage for the controller on a remote device. Here is a simple example of a webserver.
-
-```
-pi@ixe00:~/$ python server.py
- * Serving Flask app "server" (lazy loading)
- * Environment: production
- WARNING: This is a development server. Do not use it in a production deployment.
- Use a production WSGI server instead.
- * Debug mode: on
- * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
- * Restarting with stat
- * Debugger is active!
- * Debugger PIN: 162-573-883
-```
-From a remote browser on the same network, check to make sure your webserver is working by going to [http://ixe00.local:5000]()
-
-
-## Demo
-
-In the [demo directory](./demo), you will find an example wizard of oz project you may use as a template. **You do not have to** feel free to get creative. In that project, you can see how audio and sensor data is streamed from the Pi to a wizard controller that runs in the browser. You can control what system says from the controller as well.
-
-## Optional
-
-There is an included [dspeech](./dspeech) demo that uses [Mozilla DeepSpeech](https://github.com/mozilla/DeepSpeech) for speech to text. If you're interested in trying it out we suggest you create a seperarate virutalenv.
-
-
-
-# Lab 3 Part 2
-
-Create a system that runs on the Raspberry Pi that takes in one or more sensors and requires participants to speak to it. Document how the system works and include videos of both the system and the controller.
-
-## Prep for Part 2
-
-1. Sketch ideas for what you'll work on in lab on Wednesday.
-
-## Share your idea sketches with Zoom Room mates and get feedback
-
-*what was the feedback? Who did it come from?*
-
-## Prototype your system
-
-The system should:
-* use the Raspberry Pi
-* use one or more sensors
-* require participants to speak to it.
-
-*Document how the system works*
-
-*Include videos or screencaptures of both the system and the controller.*
-
-## Test the system
-Try to get at least two people to interact with your system. (Ideally, you would inform them that there is a wizard _after_ the interaction, but we recognize that can be hard.)
-
-Answer the following:
-
-### What worked well about the system and what didn't?
-*your answer here*
-
-### What worked well about the controller and what didn't?
-
-*your answer here*
-
-### What lessons can you take away from the WoZ interactions for designing a more autonomous version of the system?
-
-*your answer here*
-
-
-### How could you use your system to create a dataset of interaction? What other sensing modalities would make sense to capture?
-
-*your answer here*
-
+# Chatterboxes
+[](https://www.youtube.com/embed/Q8FWzLMobx0?start=19)
+
+In this lab, we want you to design interaction with a speech-enabled device--something that listens and talks to you. This device can do anything *but* control lights (since we already did that in Lab 1). First, we want you first to storyboard what you imagine the conversational interaction to be like. Then, you will use wizarding techniques to elicit examples of what people might say, ask, or respond. We then want you to use the examples collected from at least two other people to inform the redesign of the device.
+
+We will focus on **audio** as the main modality for interaction to start; these general techniques can be extended to **video**, **haptics** or other interactive mechanisms in the second part of the Lab.
+
+## Prep for Part 1: Get the Latest Content and Pick up Additional Parts
+
+### Pick up Additional Parts
+
+As mentioned during the class, we ordered additional mini microphone for Lab 3. Also, a new part that has finally arrived is encoder! Please remember to pick them up from the TA.
+
+### Get the Latest Content
+
+As always, pull updates from the class Interactive-Lab-Hub to both your Pi and your own GitHub repo. As we discussed in the class, there are 2 ways you can do so:
+
+**\[recommended\]**Option 1: On the Pi, `cd` to your `Interactive-Lab-Hub`, pull the updates from upstream (class lab-hub) and push the updates back to your own GitHub repo. You will need the *personal access token* for this.
+
+```
+pi@ixe00:~$ cd Interactive-Lab-Hub
+pi@ixe00:~/Interactive-Lab-Hub $ git pull upstream Fall2021
+pi@ixe00:~/Interactive-Lab-Hub $ git add .
+pi@ixe00:~/Interactive-Lab-Hub $ git commit -m "get lab3 updates"
+pi@ixe00:~/Interactive-Lab-Hub $ git push
+```
+
+Option 2: On your your own GitHub repo, [create pull request](https://github.com/FAR-Lab/Developing-and-Designing-Interactive-Devices/blob/2021Fall/readings/Submitting%20Labs.md) to get updates from the class Interactive-Lab-Hub. After you have latest updates online, go on your Pi, `cd` to your `Interactive-Lab-Hub` and use `git pull` to get updates from your own GitHub repo.
+
+## Part 1.
+### Text to Speech
+
+In this part of lab, we are going to start peeking into the world of audio on your Pi!
+
+We will be using a USB microphone, and the speaker on your webcamera. (Originally we intended to use the microphone on the web camera, but it does not seem to work on Linux.) In the home directory of your Pi, there is a folder called `text2speech` containing several shell scripts. `cd` to the folder and list out all the files by `ls`:
+
+```
+pi@ixe00:~/text2speech $ ls
+Download festival_demo.sh GoogleTTS_demo.sh pico2text_demo.sh
+espeak_demo.sh flite_demo.sh lookdave.wav
+```
+
+You can run these shell files by typing `./filename`, for example, typing `./espeak_demo.sh` and see what happens. Take some time to look at each script and see how it works. You can see a script by typing `cat filename`. For instance:
+
+```
+pi@ixe00:~/text2speech $ cat festival_demo.sh
+#from: https://elinux.org/RPi_Text_to_Speech_(Speech_Synthesis)#Festival_Text_to_Speech
+
+echo "Just what do you think you're doing, Dave?" | festival --tts
+```
+
+Now, you might wonder what exactly is a `.sh` file? Typically, a `.sh` file is a shell script which you can execute in a terminal. The example files we offer here are for you to figure out the ways to play with audio on your Pi!
+
+You can also play audio files directly with `aplay filename`. Try typing `aplay lookdave.wav`.
+
+\*\***Write your own shell file to use your favorite of these TTS engines to have your Pi greet you by name.**\*\*
+(This shell file should be saved to your own repo for this lab.)
+
+Bonus: If this topic is very exciting to you, you can try out this new TTS system we recently learned about: https://github.com/rhasspy/larynx
+
+### Speech to Text
+
+Now examine the `speech2text` folder. We are using a speech recognition engine, [Vosk](https://alphacephei.com/vosk/), which is made by researchers at Carnegie Mellon University. Vosk is amazing because it is an offline speech recognition engine; that is, all the processing for the speech recognition is happening onboard the Raspberry Pi.
+
+In particular, look at `test_words.py` and make sure you understand how the vocab is defined. Then try `./vosk_demo_mic.sh`
+
+One thing you might need to pay attention to is the audio input setting of Pi. Since you are plugging the USB cable of your webcam to your Pi at the same time to act as speaker, the default input might be set to the webcam microphone, which will not be working for recording.
+
+\*\***Write your own shell file that verbally asks for a numerical based input (such as a phone number, zipcode, number of pets, etc) and records the answer the respondent provides.**\*\*
+
+Bonus Activity:
+
+If you are really excited about Speech to Text, you can try out [Mozilla DeepSpeech](https://github.com/mozilla/DeepSpeech) and [voice2json](http://voice2json.org/install.html)
+There is an included [dspeech](./dspeech) demo on the Pi. If you're interested in trying it out, we suggest you create a seperarate virutal environment for it . Create a new Python virtual environment by typing the following commands.
+
+```
+pi@ixe00:~ $ virtualenv dspeechexercise
+pi@ixe00:~ $ source dspeechexercise/bin/activate
+(dspeechexercise) pi@ixe00:~ $
+```
+
+### Serving Pages
+
+In Lab 1, we served a webpage with flask. In this lab, you may find it useful to serve a webpage for the controller on a remote device. Here is a simple example of a webserver.
+
+```
+pi@ixe00:~/Interactive-Lab-Hub/Lab 3 $ python server.py
+ * Serving Flask app "server" (lazy loading)
+ * Environment: production
+ WARNING: This is a development server. Do not use it in a production deployment.
+ Use a production WSGI server instead.
+ * Debug mode: on
+ * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
+ * Restarting with stat
+ * Debugger is active!
+ * Debugger PIN: 162-573-883
+```
+From a remote browser on the same network, check to make sure your webserver is working by going to `http://:5000`. You should be able to see "Hello World" on the webpage.
+
+### Storyboard
+
+Storyboard and/or use a Verplank diagram to design a speech-enabled device. (Stuck? Make a device that talks for dogs. If that is too stupid, find an application that is better than that.)
+
+\*\***Post your storyboard and diagram here.**\*\*
+
+Write out what you imagine the dialogue to be. Use cards, post-its, or whatever method helps you develop alternatives or group responses.
+
+\*\***Please describe and document your process.**\*\*
+
+### Acting out the dialogue
+
+Find a partner, and *without sharing the script with your partner* try out the dialogue you've designed, where you (as the device designer) act as the device you are designing. Please record this interaction (for example, using Zoom's record feature).
+
+\*\***Describe if the dialogue seemed different than what you imagined when it was acted out, and how.**\*\*
+
+### Wizarding with the Pi (optional)
+In the [demo directory](./demo), you will find an example Wizard of Oz project. In that project, you can see how audio and sensor data is streamed from the Pi to a wizard controller that runs in the browser. You may use this demo code as a template. By running the `app.py` script, you can see how audio and sensor data (Adafruit MPU-6050 6-DoF Accel and Gyro Sensor) is streamed from the Pi to a wizard controller that runs in the browser `http://:5000`. You can control what the system says from the controller as well!
+
+\*\***Describe if the dialogue seemed different than what you imagined, or when acted out, when it was wizarded, and how.**\*\*
+
+# Lab 3 Part 2
+
+For Part 2, you will redesign the interaction with the speech-enabled device using the data collected, as well as feedback from part 1.
+
+## Prep for Part 2
+
+1. What are concrete things that could use improvement in the design of your device? For example: wording, timing, anticipation of misunderstandings...
+2. What are other modes of interaction _beyond speech_ that you might also use to clarify how to interact?
+3. Make a new storyboard, diagram and/or script based on these reflections.
+
+## Prototype your system
+
+The system should:
+* use the Raspberry Pi
+* use one or more sensors
+* require participants to speak to it.
+
+*Document how the system works*
+
+*Include videos or screencaptures of both the system and the controller.*
+
+## Test the system
+Try to get at least two people to interact with your system. (Ideally, you would inform them that there is a wizard _after_ the interaction, but we recognize that can be hard.)
+
+Answer the following:
+
+### What worked well about the system and what didn't?
+\*\**your answer here*\*\*
+
+### What worked well about the controller and what didn't?
+
+\*\**your answer here*\*\*
+
+### What lessons can you take away from the WoZ interactions for designing a more autonomous version of the system?
+
+\*\**your answer here*\*\*
+
+
+### How could you use your system to create a dataset of interaction? What other sensing modalities would make sense to capture?
+
+\*\**your answer here*\*\*
+
diff --git a/Lab 3/demo/.gitignore b/Lab 3/demo/.gitignore
deleted file mode 100644
index cd7bf9cd93..0000000000
--- a/Lab 3/demo/.gitignore
+++ /dev/null
@@ -1,2 +0,0 @@
-sftp-config.json
-.DS_Store
diff --git a/Lab 3/demo/README.md b/Lab 3/demo/README.md
index c9d0b8be25..2a13472b8c 100644
--- a/Lab 3/demo/README.md
+++ b/Lab 3/demo/README.md
@@ -1,87 +1,87 @@
-# Magic Ball WoZ
-
-This is a Demo App for a Wizard of Oz interactive system where the wizard is playing a magic 8 ball
-
-
-
-## Hardware Set-Up
-
-For this demo, you will need:
-* your Raspberry Pi,
-* a Qwiic/Stemma Cable,
-* a speaker/aux cable or wired headphones,
-* the display (we are just using it for the Qwiic/StemmaQT port. Feel free to use the display in your projects),
-* your accelerometer, and
-* your USB microphone
-
-
-
-
-
-
-Plug the display in and connect the accelerometer to the port underneath with your Qwiic connector cable. Plug the aux cable into the raspberry pi and your speaker or connect your headphones. Plug the USB microphone into one of the usb ports on the Pi
-
-## Software Setup
-
-Sssh on to your Raspberry Pi as we've done previously
-
-`ssh pi@yourHostname.local`
-
-Ensure audio is playing through the aux connector by typing
-
-`sudo raspi-config`
-
-on `system options` hit enter. Go down to `s2 Audio` and hit enter. Select `1 Headphones` and hit enter. Then navigate to `` and exit the config menu.
-
-We will need one additional piece of software called VLC Media player. To install it type `sudo apt-get install vlc`
-
-
-I would suggest making a new virtual environment for this demo then navigating to this folder and installing the requirements.
-
-```
-pi@yourHostname:~ $ virtualenv woz
-pi@yourHostname:~ $ source woz/bin/activate
-(woz) pi@yourHostname:~ $ cd Interactive-Lab-Hub/Lab\ 3/demo
-(woz) pi@yourHostname:~/Interactive-Lab-Hub/Lab 3/demo $
-(woz) pi@yourHostname:~/Interactive-Lab-Hub/Lab 3/demo $ pip install -r requirements.txt
-```
-
-## Running
-
-To run the app
-
-`(woz) pi@yourHostname:~/Interactive-Lab-Hub/Lab 3/demo $ python app.py`
-
-In the browser of a computer on the same network, navigate to http://yourHostname.local:5000/ where in my case my hostname is ixe00
-
-
-
-The interface will immediately begin streaming the accelerometer to let you know if your participant shakes their Magic 8 ball. The "eavesdrop" button will begin streaming audio from the Pi to your browser (note their is a noticeable delay it is best to start eavesdropping right at the beginning). To have the Pi speak, you can write in the text box and hit send or press enter.
-
-## Notes
-
-You may need to change line 23 in `app.py`
-
-```
-hardware = 'plughw:2,0'
-```
-
-This is the soundcard and hardware device associated with the USB microphone. To check, you can run `python get_device.py` which will output A LOT of nonsense. At the end, you will see
-
-```
-0 bcm2835 HDMI 1: - (plughw:0,0)
-1 bcm2835 Headphones: - (plughw:1,0)
-2 USB PnP Sound Device: Audio (plughw:2,0)
-3 sysdefault
-4 lavrate
-5 samplerate
-6 speexrate
-7 pulse
-8 upmix
-9 vdownmix
-10 dmix
-11 default
-```
-
-In our case, `USB PnP Sound Device: Audio (plughw:2,0)` is the name of our microphone and the index is in parenthesis.
-
+# Magic Ball WoZ
+
+This is a Demo App for a Wizard of Oz interactive system where the wizard is playing a magic 8 ball
+
+
+
+## Hardware Set-Up
+
+For this demo, you will need:
+* your Raspberry Pi,
+* a Qwiic/Stemma Cable,
+* a speaker/aux cable or wired headphones,
+* the display (we are just using it for the Qwiic/StemmaQT port. Feel free to use the display in your projects),
+* your accelerometer, and
+* your USB microphone
+
+
+
+
+
+
+Plug the display in and connect the accelerometer to the port underneath with your Qwiic connector cable. Plug the aux cable into the raspberry pi and your speaker or connect your headphones. Plug the USB microphone into one of the usb ports on the Pi
+
+## Software Setup
+
+Sssh on to your Raspberry Pi as we've done previously
+
+`ssh pi@yourHostname.local`
+
+Ensure audio is playing through the aux connector by typing
+
+`sudo raspi-config`
+
+on `system options` hit enter. Go down to `s2 Audio` and hit enter. Select `1 Headphones` and hit enter. Then navigate to `` and exit the config menu.
+
+We will need one additional piece of software called VLC Media player. To install it type `sudo apt-get install vlc`
+
+
+I would suggest making a new virtual environment for this demo then navigating to this folder and installing the requirements.
+
+```
+pi@yourHostname:~ $ virtualenv woz
+pi@yourHostname:~ $ source woz/bin/activate
+(woz) pi@yourHostname:~ $ cd Interactive-Lab-Hub/Lab\ 3/demo
+(woz) pi@yourHostname:~/Interactive-Lab-Hub/Lab 3/demo $
+(woz) pi@yourHostname:~/Interactive-Lab-Hub/Lab 3/demo $ pip install -r requirements.txt
+```
+
+## Running
+
+To run the app
+
+`(woz) pi@yourHostname:~/Interactive-Lab-Hub/Lab 3/demo $ python app.py`
+
+In the browser of a computer on the same network, navigate to http://yourHostname.local:5000/ where in my case my hostname is ixe00
+
+
+
+The interface will immediately begin streaming the accelerometer to let you know if your participant shakes their Magic 8 ball. The "eavesdrop" button will begin streaming audio from the Pi to your browser (note their is a noticeable delay it is best to start eavesdropping right at the beginning). To have the Pi speak, you can write in the text box and hit send or press enter.
+
+## Notes
+
+You may need to change line 23 in `app.py`
+
+```
+hardware = 'plughw:2,0'
+```
+
+This is the soundcard and hardware device associated with the USB microphone. To check, you can run `python get_device.py` which will output A LOT of nonsense. At the end, you will see
+
+```
+0 bcm2835 HDMI 1: - (plughw:0,0)
+1 bcm2835 Headphones: - (plughw:1,0)
+2 USB PnP Sound Device: Audio (plughw:2,0)
+3 sysdefault
+4 lavrate
+5 samplerate
+6 speexrate
+7 pulse
+8 upmix
+9 vdownmix
+10 dmix
+11 default
+```
+
+In our case, `USB PnP Sound Device: Audio (plughw:2,0)` is the name of our microphone and the index is in parenthesis.
+
diff --git a/Lab 3/demo/app.py b/Lab 3/demo/app.py
index 53be25b749..7dec99426a 100644
--- a/Lab 3/demo/app.py
+++ b/Lab 3/demo/app.py
@@ -1,61 +1,61 @@
-import eventlet
-eventlet.monkey_patch()
-
-from flask import Flask, Response,render_template
-from flask_socketio import SocketIO, send, emit
-from subprocess import Popen, call
-
-import time
-import board
-import busio
-import adafruit_mpu6050
-import json
-import socket
-
-import signal
-import sys
-from queue import Queue
-
-
-i2c = busio.I2C(board.SCL, board.SDA)
-mpu = adafruit_mpu6050.MPU6050(i2c)
-
-hostname = socket.gethostname()
-hardware = 'plughw:2,0'
-
-app = Flask(__name__)
-socketio = SocketIO(app)
-audio_stream = Popen("/usr/bin/cvlc alsa://"+hardware+" --sout='#transcode{vcodec=none,acodec=mp3,ab=256,channels=2,samplerate=44100,scodec=none}:http{mux=mp3,dst=:8080/}' --no-sout-all --sout-keep", shell=True)
-
-@socketio.on('speak')
-def handel_speak(val):
- call(f"espeak '{val}'", shell=True)
-
-@socketio.on('connect')
-def test_connect():
- print('connected')
- emit('after connect', {'data':'Lets dance'})
-
-@socketio.on('ping-gps')
-def handle_message(val):
- # print(mpu.acceleration)
- emit('pong-gps', mpu.acceleration)
-
-
-
-@app.route('/')
-def index():
- return render_template('index.html', hostname=hostname)
-
-def signal_handler(sig, frame):
- print('Closing Gracefully')
- audio_stream.terminate()
- sys.exit(0)
-
-signal.signal(signal.SIGINT, signal_handler)
-
-
-if __name__ == "__main__":
- socketio.run(app, host='0.0.0.0', port=5000)
-
-
+import eventlet
+eventlet.monkey_patch()
+
+from flask import Flask, Response,render_template
+from flask_socketio import SocketIO, send, emit
+from subprocess import Popen, call
+
+import time
+import board
+import busio
+import adafruit_mpu6050
+import json
+import socket
+
+import signal
+import sys
+from queue import Queue
+
+
+i2c = busio.I2C(board.SCL, board.SDA)
+mpu = adafruit_mpu6050.MPU6050(i2c)
+
+hostname = socket.gethostname()
+hardware = 'plughw:2,0'
+
+app = Flask(__name__)
+socketio = SocketIO(app)
+audio_stream = Popen("/usr/bin/cvlc alsa://"+hardware+" --sout='#transcode{vcodec=none,acodec=mp3,ab=256,channels=2,samplerate=44100,scodec=none}:http{mux=mp3,dst=:8080/}' --no-sout-all --sout-keep", shell=True)
+
+@socketio.on('speak')
+def handel_speak(val):
+ call(f"espeak '{val}'", shell=True)
+
+@socketio.on('connect')
+def test_connect():
+ print('connected')
+ emit('after connect', {'data':'Lets dance'})
+
+@socketio.on('ping-gps')
+def handle_message(val):
+ # print(mpu.acceleration)
+ emit('pong-gps', mpu.acceleration)
+
+
+
+@app.route('/')
+def index():
+ return render_template('index.html', hostname=hostname)
+
+def signal_handler(sig, frame):
+ print('Closing Gracefully')
+ audio_stream.terminate()
+ sys.exit(0)
+
+signal.signal(signal.SIGINT, signal_handler)
+
+
+if __name__ == "__main__":
+ socketio.run(app, host='0.0.0.0', port=5000)
+
+
diff --git a/Lab 3/demo/get_device.py b/Lab 3/demo/get_device.py
index 1c0697031a..2a8dfcc3de 100644
--- a/Lab 3/demo/get_device.py
+++ b/Lab 3/demo/get_device.py
@@ -1,6 +1,6 @@
-import pyaudio
-
-audio = pyaudio.PyAudio()
-
-for ii in range(audio.get_device_count()):
- print(ii, audio.get_device_info_by_index(ii).get('name'))
+import pyaudio
+
+audio = pyaudio.PyAudio()
+
+for ii in range(audio.get_device_count()):
+ print(ii, audio.get_device_info_by_index(ii).get('name'))
diff --git a/Lab 3/demo/requirements.txt b/Lab 3/demo/requirements.txt
index 568acb3a5e..330d8866e3 100644
--- a/Lab 3/demo/requirements.txt
+++ b/Lab 3/demo/requirements.txt
@@ -1,31 +1,31 @@
-Adafruit-Blinka==6.4.0
-adafruit-circuitpython-busdevice==5.0.6
-adafruit-circuitpython-mpu6050==1.1.6
-adafruit-circuitpython-register==1.9.5
-Adafruit-PlatformDetect==3.3.0
-Adafruit-PureIO==1.1.8
-bidict==0.21.2
-click==7.1.2
-dnspython==1.16.0
-eventlet==0.31.0
-Flask==1.1.2
-Flask-SocketIO==5.0.1
-gevent==21.1.2
-gevent-websocket==0.10.1
-greenlet==1.0.0
-itsdangerous==1.1.0
-Jinja2==2.11.3
-MarkupSafe==1.1.1
-PyAudio==0.2.11
-pyftdi==0.52.9
-pyserial==3.5
-python-engineio==4.0.1
-python-socketio==5.1.0
-pyusb==1.1.1
-rpi-ws281x==4.2.6
-RPi.GPIO==0.7.0
-six==1.15.0
-sysv-ipc==1.1.0
-Werkzeug==1.0.1
-zope.event==4.5.0
+Adafruit-Blinka==6.4.0
+adafruit-circuitpython-busdevice==5.0.6
+adafruit-circuitpython-mpu6050==1.1.6
+adafruit-circuitpython-register==1.9.5
+Adafruit-PlatformDetect==3.3.0
+Adafruit-PureIO==1.1.8
+bidict==0.21.2
+click==7.1.2
+dnspython==1.16.0
+eventlet==0.31.0
+Flask==1.1.2
+Flask-SocketIO==5.0.1
+gevent==21.1.2
+gevent-websocket==0.10.1
+greenlet==1.0.0
+itsdangerous==1.1.0
+Jinja2==2.11.3
+MarkupSafe==1.1.1
+PyAudio==0.2.11
+pyftdi==0.52.9
+pyserial==3.5
+python-engineio==4.0.1
+python-socketio==5.1.0
+pyusb==1.1.1
+rpi-ws281x==4.2.6
+RPi.GPIO==0.7.0
+six==1.15.0
+sysv-ipc==1.1.0
+Werkzeug==1.0.1
+zope.event==4.5.0
zope.interface==5.2.0
\ No newline at end of file
diff --git a/Lab 3/demo/static/index.js b/Lab 3/demo/static/index.js
index 4a25601727..8fedc35238 100644
--- a/Lab 3/demo/static/index.js
+++ b/Lab 3/demo/static/index.js
@@ -1,196 +1,196 @@
-// const control = document.getElementById('control');
-// const light = document.getElementById('light');
-// const play = document.getElementById('play');
-// const pause = document.getElementById('pause');
-// const audioIn = document.getElementById('audioIn');
-// const audio = new Audio();
-// let pickr;
-
-// const socket = io();
-
-// socket.on('connect', () => {
-// socket.on('hex', (val) => {document.body.style.backgroundColor = val})
-// socket.on('audio', (val) => {getSound(encodeURI(val));})
-// socket.on('pauseAudio', (val) => {audio.pause();})
-// socket.onAny((event, ...args) => {
-// console.log(event, args);
-// });
-// });
-
-// // enter controller mode
-// control.onclick = () => {
-// console.log('control')
-// // make sure you're not in fullscreen
-// if (document.fullscreenElement) {
-// document.exitFullscreen()
-// .then(() => console.log('exited full screen mode'))
-// .catch((err) => console.error(err));
-// }
-// // make buttons and controls visible
-// document.getElementById('user').classList.remove('fadeOut');
-// document.getElementById('controlPanel').style.opacity = 0.6;
-// if (!pickr) {
-// // create our color picker. You can change the swatches that appear at the bottom
-// pickr = Pickr.create({
-// el: '.pickr',
-// theme: 'classic',
-// showAlways: true,
-// swatches: [
-// 'rgba(255, 255, 255, 1)',
-// 'rgba(244, 67, 54, 1)',
-// 'rgba(233, 30, 99, 1)',
-// 'rgba(156, 39, 176, 1)',
-// 'rgba(103, 58, 183, 1)',
-// 'rgba(63, 81, 181, 1)',
-// 'rgba(33, 150, 243, 1)',
-// 'rgba(3, 169, 244, 1)',
-// 'rgba(0, 188, 212, 1)',
-// 'rgba(0, 150, 136, 1)',
-// 'rgba(76, 175, 80, 1)',
-// 'rgba(139, 195, 74, 1)',
-// 'rgba(205, 220, 57, 1)',
-// 'rgba(255, 235, 59, 1)',
-// 'rgba(255, 193, 7, 1)',
-// 'rgba(0, 0, 0, 1)',
-// ],
-// components: {
-// preview: false,
-// opacity: false,
-// hue: true,
-// },
-// });
-
-// pickr.on('change', (e) => {
-// // when pickr color value is changed change background and send message on ws to change background
-// const hexCode = e.toHEXA().toString();
-// document.body.style.backgroundColor = hexCode;
-// socket.emit('hex', hexCode)
-// });
-// }
-// };
-
-// light.onclick = () => {
-// // safari requires playing on input before allowing audio
-// audio.muted = true;
-// audio.play().then(audio.muted=false)
-
-// // in light mode make it full screen and fade buttons
-// document.documentElement.requestFullscreen();
-// document.getElementById('user').classList.add('fadeOut');
-// // if you were previously in control mode remove color picker and hide controls
-// if (pickr) {
-// // this is annoying because of the pickr package
-// pickr.destroyAndRemove();
-// document.getElementById('controlPanel').append(Object.assign(document.createElement('div'), { className: 'pickr' }));
-// pickr = undefined;
-// }
-// document.getElementById('controlPanel').style.opacity = 0;
-// };
-
-
-// const getSound = (query, loop = false, random = false) => {
-// const url = `https://freesound.org/apiv2/search/text/?query=${query}+"&fields=name,previews&token=U5slaNIqr6ofmMMG2rbwJ19mInmhvCJIryn2JX89&format=json`;
-// fetch(url)
-// .then((response) => response.clone().text())
-// .then((data) => {
-// console.log(data);
-// data = JSON.parse(data);
-// if (data.results.length >= 1) var src = random ? choice(data.results).previews['preview-hq-mp3'] : data.results[0].previews['preview-hq-mp3'];
-// audio.src = src;
-// audio.play();
-// console.log(src);
-// })
-// .catch((error) => console.log(error));
-// };
-
-// play.onclick = () => {
-// socket.emit('audio', audioIn.value)
-// getSound(encodeURI(audioIn.value));
-// };
-// pause.onclick = () => {
-// socket.emit('pauseAudio', audioIn.value)
-// audio.pause();
-// };
-// audioIn.onkeyup = (e) => { if (e.keyCode === 13) { play.click(); } };
-
-const socket = io();
-socket.on('connect', () => {
-// socket.onAny((event, ...args) => {
-// console.log(Date.now(),event, args);
-// });
-});
-
-const mic = document.getElementById('mic');
-const play = document.getElementById('play');
-const wordsIn = document.getElementById('wordsIn');
-const send = document.getElementById('send');
-
-const src = mic.src
-mic.src = ''
-
-play.onclick = () => {
- if(mic.paused) {
- console.log('redo audio')
- mic.src = src
- mic.play()
- play.innerText='Pause'
- } else {
- mic.pause()
- mic.src = '';
- play.innerText='Eavesdrop'
- }
-
-}
-
-send.onclick = () => {
- socket.emit('speak', wordsIn.value)
- wordsIn.value = ''
-}
-wordsIn.onkeyup = (e) => { if (e.keyCode === 13) { send.click(); } };
-
-setInterval(() => {
- socket.emit('ping-gps', 'dat')
-}, 100)
-
-socket.on('disconnect', () => {
- console.log('disconnect')
- mic.src = ''
-
- });
-
-var vlSpec = {
- $schema: 'https://vega.github.io/schema/vega-lite/v5.json',
- data: {name: 'table'},
- width: 400,
- mark: 'line',
- encoding: {
- x: {field: 'x', type: 'quantitative', scale: {zero: false}},
- y: {field: 'y', type: 'quantitative'},
- color: {field: 'category', type: 'nominal'}
- }
-};
-vegaEmbed('#chart', vlSpec).then( (res) => {
- let x, y, z;
- let counter = -1;
- let cat = ['x', 'y', 'z']
- let minimumX = -100;
- socket.on('pong-gps', (new_x,new_y,new_z) => {
- counter++;
- minimumX++;
- const newVals = [new_x, new_y, new_z].map((c,v) => {
- return {
- x: counter,
- y: c,
- category: cat[v]
- };
- })
- const changeSet = vega
- .changeset()
- .insert(newVals)
- .remove( (t) => {
- return t.x < minimumX;
- });
- res.view.change('table', changeSet).run();
- })
-
-})
+// const control = document.getElementById('control');
+// const light = document.getElementById('light');
+// const play = document.getElementById('play');
+// const pause = document.getElementById('pause');
+// const audioIn = document.getElementById('audioIn');
+// const audio = new Audio();
+// let pickr;
+
+// const socket = io();
+
+// socket.on('connect', () => {
+// socket.on('hex', (val) => {document.body.style.backgroundColor = val})
+// socket.on('audio', (val) => {getSound(encodeURI(val));})
+// socket.on('pauseAudio', (val) => {audio.pause();})
+// socket.onAny((event, ...args) => {
+// console.log(event, args);
+// });
+// });
+
+// // enter controller mode
+// control.onclick = () => {
+// console.log('control')
+// // make sure you're not in fullscreen
+// if (document.fullscreenElement) {
+// document.exitFullscreen()
+// .then(() => console.log('exited full screen mode'))
+// .catch((err) => console.error(err));
+// }
+// // make buttons and controls visible
+// document.getElementById('user').classList.remove('fadeOut');
+// document.getElementById('controlPanel').style.opacity = 0.6;
+// if (!pickr) {
+// // create our color picker. You can change the swatches that appear at the bottom
+// pickr = Pickr.create({
+// el: '.pickr',
+// theme: 'classic',
+// showAlways: true,
+// swatches: [
+// 'rgba(255, 255, 255, 1)',
+// 'rgba(244, 67, 54, 1)',
+// 'rgba(233, 30, 99, 1)',
+// 'rgba(156, 39, 176, 1)',
+// 'rgba(103, 58, 183, 1)',
+// 'rgba(63, 81, 181, 1)',
+// 'rgba(33, 150, 243, 1)',
+// 'rgba(3, 169, 244, 1)',
+// 'rgba(0, 188, 212, 1)',
+// 'rgba(0, 150, 136, 1)',
+// 'rgba(76, 175, 80, 1)',
+// 'rgba(139, 195, 74, 1)',
+// 'rgba(205, 220, 57, 1)',
+// 'rgba(255, 235, 59, 1)',
+// 'rgba(255, 193, 7, 1)',
+// 'rgba(0, 0, 0, 1)',
+// ],
+// components: {
+// preview: false,
+// opacity: false,
+// hue: true,
+// },
+// });
+
+// pickr.on('change', (e) => {
+// // when pickr color value is changed change background and send message on ws to change background
+// const hexCode = e.toHEXA().toString();
+// document.body.style.backgroundColor = hexCode;
+// socket.emit('hex', hexCode)
+// });
+// }
+// };
+
+// light.onclick = () => {
+// // safari requires playing on input before allowing audio
+// audio.muted = true;
+// audio.play().then(audio.muted=false)
+
+// // in light mode make it full screen and fade buttons
+// document.documentElement.requestFullscreen();
+// document.getElementById('user').classList.add('fadeOut');
+// // if you were previously in control mode remove color picker and hide controls
+// if (pickr) {
+// // this is annoying because of the pickr package
+// pickr.destroyAndRemove();
+// document.getElementById('controlPanel').append(Object.assign(document.createElement('div'), { className: 'pickr' }));
+// pickr = undefined;
+// }
+// document.getElementById('controlPanel').style.opacity = 0;
+// };
+
+
+// const getSound = (query, loop = false, random = false) => {
+// const url = `https://freesound.org/apiv2/search/text/?query=${query}+"&fields=name,previews&token=U5slaNIqr6ofmMMG2rbwJ19mInmhvCJIryn2JX89&format=json`;
+// fetch(url)
+// .then((response) => response.clone().text())
+// .then((data) => {
+// console.log(data);
+// data = JSON.parse(data);
+// if (data.results.length >= 1) var src = random ? choice(data.results).previews['preview-hq-mp3'] : data.results[0].previews['preview-hq-mp3'];
+// audio.src = src;
+// audio.play();
+// console.log(src);
+// })
+// .catch((error) => console.log(error));
+// };
+
+// play.onclick = () => {
+// socket.emit('audio', audioIn.value)
+// getSound(encodeURI(audioIn.value));
+// };
+// pause.onclick = () => {
+// socket.emit('pauseAudio', audioIn.value)
+// audio.pause();
+// };
+// audioIn.onkeyup = (e) => { if (e.keyCode === 13) { play.click(); } };
+
+const socket = io();
+socket.on('connect', () => {
+// socket.onAny((event, ...args) => {
+// console.log(Date.now(),event, args);
+// });
+});
+
+const mic = document.getElementById('mic');
+const play = document.getElementById('play');
+const wordsIn = document.getElementById('wordsIn');
+const send = document.getElementById('send');
+
+const src = mic.src
+mic.src = ''
+
+play.onclick = () => {
+ if(mic.paused) {
+ console.log('redo audio')
+ mic.src = src
+ mic.play()
+ play.innerText='Pause'
+ } else {
+ mic.pause()
+ mic.src = '';
+ play.innerText='Eavesdrop'
+ }
+
+}
+
+send.onclick = () => {
+ socket.emit('speak', wordsIn.value)
+ wordsIn.value = ''
+}
+wordsIn.onkeyup = (e) => { if (e.keyCode === 13) { send.click(); } };
+
+setInterval(() => {
+ socket.emit('ping-gps', 'dat')
+}, 100)
+
+socket.on('disconnect', () => {
+ console.log('disconnect')
+ mic.src = ''
+
+ });
+
+var vlSpec = {
+ $schema: 'https://vega.github.io/schema/vega-lite/v5.json',
+ data: {name: 'table'},
+ width: 400,
+ mark: 'line',
+ encoding: {
+ x: {field: 'x', type: 'quantitative', scale: {zero: false}},
+ y: {field: 'y', type: 'quantitative'},
+ color: {field: 'category', type: 'nominal'}
+ }
+};
+vegaEmbed('#chart', vlSpec).then( (res) => {
+ let x, y, z;
+ let counter = -1;
+ let cat = ['x', 'y', 'z']
+ let minimumX = -100;
+ socket.on('pong-gps', (new_x,new_y,new_z) => {
+ counter++;
+ minimumX++;
+ const newVals = [new_x, new_y, new_z].map((c,v) => {
+ return {
+ x: counter,
+ y: c,
+ category: cat[v]
+ };
+ })
+ const changeSet = vega
+ .changeset()
+ .insert(newVals)
+ .remove( (t) => {
+ return t.x < minimumX;
+ });
+ res.view.change('table', changeSet).run();
+ })
+
+})
diff --git a/Lab 3/demo/templates/index.html b/Lab 3/demo/templates/index.html
index 37492de842..6b9af227a7 100644
--- a/Lab 3/demo/templates/index.html
+++ b/Lab 3/demo/templates/index.html
@@ -1,30 +1,30 @@
-
-
-
-
-
-
-
-
-
-
-
-
- Document
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+ Document
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/Lab 3/dspeech/README.md b/Lab 3/dspeech/README.md
index 12084b7c28..a21fa8fec3 100644
--- a/Lab 3/dspeech/README.md
+++ b/Lab 3/dspeech/README.md
@@ -1,17 +1,24 @@
-# DeepSpeech
-
-This uses [Mozilla DeepSpeech](https://github.com/mozilla/DeepSpeech) and works shockingly well even on the Pi. You may want to change the VAD (Voice Activity Detection) parameter down to 2 given the quality of our microphones.
-
-## Setup
-
-```
-sudo apt-get install libatlas3-base
-pip install -r requirements.txt
-sh downloader.sh
-```
-
-## Running
-
-```
-python deepspeech_demo.py -m deepspeech-0.9.3-models.tflite -s deepspeech-0.9.3-models.scorer
-```
+# DeepSpeech
+
+This uses [Mozilla DeepSpeech](https://github.com/mozilla/DeepSpeech) and works shockingly well even on the Pi.
+
+## Setup
+Undering the virtual environment you created for this exercise, `cd` to the `dspeech` folder under Lab 3 and run the following command lines:
+
+```
+(dspeechexercise) pi@ixe00:~/Interactive-Lab-Hub/Lab 3/dspeech $ sudo apt-get install libatlas3-base
+(dspeechexercise) pi@ixe00:~/Interactive-Lab-Hub/Lab 3/dspeech $ pip install -r requirements.txt
+(dspeechexercise) pi@ixe00:~/Interactive-Lab-Hub/Lab 3/dspeech $ sh downloader.sh
+```
+
+## Running
+
+```
+(dspeechexercise) pi@ixe00:~/Interactive-Lab-Hub/Lab 3/dspeech $ python deepspeech_demo.py -m deepspeech-0.9.3-models.tflite -s deepspeech-0.9.3-models.scorer
+```
+
+The script should update and show what you are saying. Depneding on the level of background noise in your environment, you may want to change the VAD (Voice Activity Detection) parameter down given the quality of our microphones. VAD is set as an integer between 0 and 3, 0 being the least aggressive about filtering out non-speech, 3 the most aggressive. In the script, we have the VAD default = 3. Set aggressiveness of VAD by adding -v setting in the command line:
+
+```
+(dspeechexercise) pi@ixe00:~/Interactive-Lab-Hub/Lab 3/dspeech $ python deepspeech_demo.py -m deepspeech-0.9.3-models.tflite -s deepspeech-0.9.3-models.scorer -v 2
+```
diff --git a/Lab 3/dspeech/deepspeech_demo.py b/Lab 3/dspeech/deepspeech_demo.py
index c56f817bde..fde6911456 100644
--- a/Lab 3/dspeech/deepspeech_demo.py
+++ b/Lab 3/dspeech/deepspeech_demo.py
@@ -1,224 +1,224 @@
-import time, logging
-from datetime import datetime
-import threading, collections, queue, os, os.path
-import deepspeech
-import numpy as np
-import pyaudio
-import wave
-import webrtcvad
-from halo import Halo
-from scipy import signal
-
-logging.basicConfig(level=20)
-
-class Audio(object):
- """Streams raw audio from microphone. Data is received in a separate thread, and stored in a buffer, to be read from."""
-
- FORMAT = pyaudio.paInt16
- # Network/VAD rate-space
- RATE_PROCESS = 16000
- CHANNELS = 1
- BLOCKS_PER_SECOND = 50
-
- def __init__(self, callback=None, device=None, input_rate=RATE_PROCESS, file=None):
- def proxy_callback(in_data, frame_count, time_info, status):
- #pylint: disable=unused-argument
- if self.chunk is not None:
- in_data = self.wf.readframes(self.chunk)
- callback(in_data)
- return (None, pyaudio.paContinue)
- if callback is None: callback = lambda in_data: self.buffer_queue.put(in_data)
- self.buffer_queue = queue.Queue()
- self.device = device
- self.input_rate = input_rate
- self.sample_rate = self.RATE_PROCESS
- self.block_size = int(self.RATE_PROCESS / float(self.BLOCKS_PER_SECOND))
- self.block_size_input = int(self.input_rate / float(self.BLOCKS_PER_SECOND))
- self.pa = pyaudio.PyAudio()
-
- kwargs = {
- 'format': self.FORMAT,
- 'channels': self.CHANNELS,
- 'rate': self.input_rate,
- 'input': True,
- 'frames_per_buffer': self.block_size_input,
- 'stream_callback': proxy_callback,
- }
-
- self.chunk = None
- # if not default device
- if self.device:
- kwargs['input_device_index'] = self.device
- elif file is not None:
- self.chunk = 320
- self.wf = wave.open(file, 'rb')
-
- self.stream = self.pa.open(**kwargs)
- self.stream.start_stream()
-
- def resample(self, data, input_rate):
- """
- Microphone may not support our native processing sampling rate, so
- resample from input_rate to RATE_PROCESS here for webrtcvad and
- deepspeech
-
- Args:
- data (binary): Input audio stream
- input_rate (int): Input audio rate to resample from
- """
- data16 = np.fromstring(string=data, dtype=np.int16)
- resample_size = int(len(data16) / self.input_rate * self.RATE_PROCESS)
- resample = signal.resample(data16, resample_size)
- resample16 = np.array(resample, dtype=np.int16)
- return resample16.tostring()
-
- def read_resampled(self):
- """Return a block of audio data resampled to 16000hz, blocking if necessary."""
- return self.resample(data=self.buffer_queue.get(),
- input_rate=self.input_rate)
-
- def read(self):
- """Return a block of audio data, blocking if necessary."""
- return self.buffer_queue.get()
-
- def destroy(self):
- self.stream.stop_stream()
- self.stream.close()
- self.pa.terminate()
-
- frame_duration_ms = property(lambda self: 1000 * self.block_size // self.sample_rate)
-
- def write_wav(self, filename, data):
- logging.info("write wav %s", filename)
- wf = wave.open(filename, 'wb')
- wf.setnchannels(self.CHANNELS)
- # wf.setsampwidth(self.pa.get_sample_size(FORMAT))
- assert self.FORMAT == pyaudio.paInt16
- wf.setsampwidth(2)
- wf.setframerate(self.sample_rate)
- wf.writeframes(data)
- wf.close()
-
-
-class VADAudio(Audio):
- """Filter & segment audio with voice activity detection."""
-
- def __init__(self, aggressiveness=3, device=None, input_rate=None, file=None):
- super().__init__(device=device, input_rate=input_rate, file=file)
- self.vad = webrtcvad.Vad(aggressiveness)
-
- def frame_generator(self):
- """Generator that yields all audio frames from microphone."""
- if self.input_rate == self.RATE_PROCESS:
- while True:
- yield self.read()
- else:
- while True:
- yield self.read_resampled()
-
- def vad_collector(self, padding_ms=300, ratio=0.75, frames=None):
- """Generator that yields series of consecutive audio frames comprising each utterence, separated by yielding a single None.
- Determines voice activity by ratio of frames in padding_ms. Uses a buffer to include padding_ms prior to being triggered.
- Example: (frame, ..., frame, None, frame, ..., frame, None, ...)
- |---utterence---| |---utterence---|
- """
- if frames is None: frames = self.frame_generator()
- num_padding_frames = padding_ms // self.frame_duration_ms
- ring_buffer = collections.deque(maxlen=num_padding_frames)
- triggered = False
-
- for frame in frames:
- if len(frame) < 640:
- return
-
- is_speech = self.vad.is_speech(frame, self.sample_rate)
-
- if not triggered:
- ring_buffer.append((frame, is_speech))
- num_voiced = len([f for f, speech in ring_buffer if speech])
- if num_voiced > ratio * ring_buffer.maxlen:
- triggered = True
- for f, s in ring_buffer:
- yield f
- ring_buffer.clear()
-
- else:
- yield frame
- ring_buffer.append((frame, is_speech))
- num_unvoiced = len([f for f, speech in ring_buffer if not speech])
- if num_unvoiced > ratio * ring_buffer.maxlen:
- triggered = False
- yield None
- ring_buffer.clear()
-
-def main(ARGS):
- # Load DeepSpeech model
- if os.path.isdir(ARGS.model):
- model_dir = ARGS.model
- ARGS.model = os.path.join(model_dir, 'output_graph.pb')
- ARGS.scorer = os.path.join(model_dir, ARGS.scorer)
-
- print('Initializing model...')
- logging.info("ARGS.model: %s", ARGS.model)
- model = deepspeech.Model(ARGS.model)
- if ARGS.scorer:
- logging.info("ARGS.scorer: %s", ARGS.scorer)
- model.enableExternalScorer(ARGS.scorer)
-
- # Start audio with VAD
- vad_audio = VADAudio(aggressiveness=ARGS.vad_aggressiveness,
- device=ARGS.device,
- input_rate=ARGS.rate,
- file=ARGS.file)
- print("Listening (ctrl-C to exit)...")
- frames = vad_audio.vad_collector()
-
- # Stream from microphone to DeepSpeech using VAD
- spinner = None
- if not ARGS.nospinner:
- spinner = Halo(spinner='line')
- stream_context = model.createStream()
- wav_data = bytearray()
- for frame in frames:
- if frame is not None:
- if spinner: spinner.start()
- logging.debug("streaming frame")
- stream_context.feedAudioContent(np.frombuffer(frame, np.int16))
- if ARGS.savewav: wav_data.extend(frame)
- else:
- if spinner: spinner.stop()
- logging.debug("end utterence")
- if ARGS.savewav:
- vad_audio.write_wav(os.path.join(ARGS.savewav, datetime.now().strftime("savewav_%Y-%m-%d_%H-%M-%S_%f.wav")), wav_data)
- wav_data = bytearray()
- text = stream_context.finishStream()
- print("Recognized: %s" % text)
- stream_context = model.createStream()
-
-if __name__ == '__main__':
- DEFAULT_SAMPLE_RATE = 16000
-
- import argparse
- parser = argparse.ArgumentParser(description="Stream from microphone to DeepSpeech using VAD")
-
- parser.add_argument('-v', '--vad_aggressiveness', type=int, default=3,
- help="Set aggressiveness of VAD: an integer between 0 and 3, 0 being the least aggressive about filtering out non-speech, 3 the most aggressive. Default: 3")
- parser.add_argument('--nospinner', action='store_true',
- help="Disable spinner")
- parser.add_argument('-w', '--savewav',
- help="Save .wav files of utterences to given directory")
- parser.add_argument('-f', '--file',
- help="Read from .wav file instead of microphone")
-
- parser.add_argument('-m', '--model', required=True,
- help="Path to the model (protocol buffer binary file, or entire directory containing all standard-named files for model)")
- parser.add_argument('-s', '--scorer',
- help="Path to the external scorer file.")
- parser.add_argument('-d', '--device', type=int, default=None,
- help="Device input index (Int) as listed by pyaudio.PyAudio.get_device_info_by_index(). If not provided, falls back to PyAudio.get_default_device().")
- parser.add_argument('-r', '--rate', type=int, default=DEFAULT_SAMPLE_RATE,
- help=f"Input device sample rate. Default: {DEFAULT_SAMPLE_RATE}. Your device may require 44100.")
-
- ARGS = parser.parse_args()
- if ARGS.savewav: os.makedirs(ARGS.savewav, exist_ok=True)
- main(ARGS)
+import time, logging
+from datetime import datetime
+import threading, collections, queue, os, os.path
+import deepspeech
+import numpy as np
+import pyaudio
+import wave
+import webrtcvad
+from halo import Halo
+from scipy import signal
+
+logging.basicConfig(level=20)
+
+class Audio(object):
+ """Streams raw audio from microphone. Data is received in a separate thread, and stored in a buffer, to be read from."""
+
+ FORMAT = pyaudio.paInt16
+ # Network/VAD rate-space
+ RATE_PROCESS = 16000
+ CHANNELS = 1
+ BLOCKS_PER_SECOND = 50
+
+ def __init__(self, callback=None, device=None, input_rate=RATE_PROCESS, file=None):
+ def proxy_callback(in_data, frame_count, time_info, status):
+ #pylint: disable=unused-argument
+ if self.chunk is not None:
+ in_data = self.wf.readframes(self.chunk)
+ callback(in_data)
+ return (None, pyaudio.paContinue)
+ if callback is None: callback = lambda in_data: self.buffer_queue.put(in_data)
+ self.buffer_queue = queue.Queue()
+ self.device = device
+ self.input_rate = input_rate
+ self.sample_rate = self.RATE_PROCESS
+ self.block_size = int(self.RATE_PROCESS / float(self.BLOCKS_PER_SECOND))
+ self.block_size_input = int(self.input_rate / float(self.BLOCKS_PER_SECOND))
+ self.pa = pyaudio.PyAudio()
+
+ kwargs = {
+ 'format': self.FORMAT,
+ 'channels': self.CHANNELS,
+ 'rate': self.input_rate,
+ 'input': True,
+ 'frames_per_buffer': self.block_size_input,
+ 'stream_callback': proxy_callback,
+ }
+
+ self.chunk = None
+ # if not default device
+ if self.device:
+ kwargs['input_device_index'] = self.device
+ elif file is not None:
+ self.chunk = 320
+ self.wf = wave.open(file, 'rb')
+
+ self.stream = self.pa.open(**kwargs)
+ self.stream.start_stream()
+
+ def resample(self, data, input_rate):
+ """
+ Microphone may not support our native processing sampling rate, so
+ resample from input_rate to RATE_PROCESS here for webrtcvad and
+ deepspeech
+
+ Args:
+ data (binary): Input audio stream
+ input_rate (int): Input audio rate to resample from
+ """
+ data16 = np.fromstring(string=data, dtype=np.int16)
+ resample_size = int(len(data16) / self.input_rate * self.RATE_PROCESS)
+ resample = signal.resample(data16, resample_size)
+ resample16 = np.array(resample, dtype=np.int16)
+ return resample16.tostring()
+
+ def read_resampled(self):
+ """Return a block of audio data resampled to 16000hz, blocking if necessary."""
+ return self.resample(data=self.buffer_queue.get(),
+ input_rate=self.input_rate)
+
+ def read(self):
+ """Return a block of audio data, blocking if necessary."""
+ return self.buffer_queue.get()
+
+ def destroy(self):
+ self.stream.stop_stream()
+ self.stream.close()
+ self.pa.terminate()
+
+ frame_duration_ms = property(lambda self: 1000 * self.block_size // self.sample_rate)
+
+ def write_wav(self, filename, data):
+ logging.info("write wav %s", filename)
+ wf = wave.open(filename, 'wb')
+ wf.setnchannels(self.CHANNELS)
+ # wf.setsampwidth(self.pa.get_sample_size(FORMAT))
+ assert self.FORMAT == pyaudio.paInt16
+ wf.setsampwidth(2)
+ wf.setframerate(self.sample_rate)
+ wf.writeframes(data)
+ wf.close()
+
+
+class VADAudio(Audio):
+ """Filter & segment audio with voice activity detection."""
+
+ def __init__(self, aggressiveness=3, device=None, input_rate=None, file=None):
+ super().__init__(device=device, input_rate=input_rate, file=file)
+ self.vad = webrtcvad.Vad(aggressiveness)
+
+ def frame_generator(self):
+ """Generator that yields all audio frames from microphone."""
+ if self.input_rate == self.RATE_PROCESS:
+ while True:
+ yield self.read()
+ else:
+ while True:
+ yield self.read_resampled()
+
+ def vad_collector(self, padding_ms=300, ratio=0.75, frames=None):
+ """Generator that yields series of consecutive audio frames comprising each utterence, separated by yielding a single None.
+ Determines voice activity by ratio of frames in padding_ms. Uses a buffer to include padding_ms prior to being triggered.
+ Example: (frame, ..., frame, None, frame, ..., frame, None, ...)
+ |---utterence---| |---utterence---|
+ """
+ if frames is None: frames = self.frame_generator()
+ num_padding_frames = padding_ms // self.frame_duration_ms
+ ring_buffer = collections.deque(maxlen=num_padding_frames)
+ triggered = False
+
+ for frame in frames:
+ if len(frame) < 640:
+ return
+
+ is_speech = self.vad.is_speech(frame, self.sample_rate)
+
+ if not triggered:
+ ring_buffer.append((frame, is_speech))
+ num_voiced = len([f for f, speech in ring_buffer if speech])
+ if num_voiced > ratio * ring_buffer.maxlen:
+ triggered = True
+ for f, s in ring_buffer:
+ yield f
+ ring_buffer.clear()
+
+ else:
+ yield frame
+ ring_buffer.append((frame, is_speech))
+ num_unvoiced = len([f for f, speech in ring_buffer if not speech])
+ if num_unvoiced > ratio * ring_buffer.maxlen:
+ triggered = False
+ yield None
+ ring_buffer.clear()
+
+def main(ARGS):
+ # Load DeepSpeech model
+ if os.path.isdir(ARGS.model):
+ model_dir = ARGS.model
+ ARGS.model = os.path.join(model_dir, 'output_graph.pb')
+ ARGS.scorer = os.path.join(model_dir, ARGS.scorer)
+
+ print('Initializing model...')
+ logging.info("ARGS.model: %s", ARGS.model)
+ model = deepspeech.Model(ARGS.model)
+ if ARGS.scorer:
+ logging.info("ARGS.scorer: %s", ARGS.scorer)
+ model.enableExternalScorer(ARGS.scorer)
+
+ # Start audio with VAD
+ vad_audio = VADAudio(aggressiveness=ARGS.vad_aggressiveness,
+ device=ARGS.device,
+ input_rate=ARGS.rate,
+ file=ARGS.file)
+ print("Listening (ctrl-C to exit)...")
+ frames = vad_audio.vad_collector()
+
+ # Stream from microphone to DeepSpeech using VAD
+ spinner = None
+ if not ARGS.nospinner:
+ spinner = Halo(spinner='line')
+ stream_context = model.createStream()
+ wav_data = bytearray()
+ for frame in frames:
+ if frame is not None:
+ if spinner: spinner.start()
+ logging.debug("streaming frame")
+ stream_context.feedAudioContent(np.frombuffer(frame, np.int16))
+ if ARGS.savewav: wav_data.extend(frame)
+ else:
+ if spinner: spinner.stop()
+ logging.debug("end utterence")
+ if ARGS.savewav:
+ vad_audio.write_wav(os.path.join(ARGS.savewav, datetime.now().strftime("savewav_%Y-%m-%d_%H-%M-%S_%f.wav")), wav_data)
+ wav_data = bytearray()
+ text = stream_context.finishStream()
+ print("Recognized: %s" % text)
+ stream_context = model.createStream()
+
+if __name__ == '__main__':
+ DEFAULT_SAMPLE_RATE = 16000
+
+ import argparse
+ parser = argparse.ArgumentParser(description="Stream from microphone to DeepSpeech using VAD")
+
+ parser.add_argument('-v', '--vad_aggressiveness', type=int, default=3,
+ help="Set aggressiveness of VAD: an integer between 0 and 3, 0 being the least aggressive about filtering out non-speech, 3 the most aggressive. Default: 3")
+ parser.add_argument('--nospinner', action='store_true',
+ help="Disable spinner")
+ parser.add_argument('-w', '--savewav',
+ help="Save .wav files of utterences to given directory")
+ parser.add_argument('-f', '--file',
+ help="Read from .wav file instead of microphone")
+
+ parser.add_argument('-m', '--model', required=True,
+ help="Path to the model (protocol buffer binary file, or entire directory containing all standard-named files for model)")
+ parser.add_argument('-s', '--scorer',
+ help="Path to the external scorer file.")
+ parser.add_argument('-d', '--device', type=int, default=None,
+ help="Device input index (Int) as listed by pyaudio.PyAudio.get_device_info_by_index(). If not provided, falls back to PyAudio.get_default_device().")
+ parser.add_argument('-r', '--rate', type=int, default=DEFAULT_SAMPLE_RATE,
+ help=f"Input device sample rate. Default: {DEFAULT_SAMPLE_RATE}. Your device may require 44100.")
+
+ ARGS = parser.parse_args()
+ if ARGS.savewav: os.makedirs(ARGS.savewav, exist_ok=True)
+ main(ARGS)
diff --git a/Lab 3/dspeech/downloader.sh b/Lab 3/dspeech/downloader.sh
index ff28b89637..9f5ae46b82 100644
--- a/Lab 3/dspeech/downloader.sh
+++ b/Lab 3/dspeech/downloader.sh
@@ -1,2 +1,2 @@
-curl -LO https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/deepspeech-0.9.3-models.tflite
-curl -LO https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/deepspeech-0.9.3-models.scorer
+curl -LO https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/deepspeech-0.9.3-models.tflite
+curl -LO https://github.com/mozilla/DeepSpeech/releases/download/v0.9.3/deepspeech-0.9.3-models.scorer
diff --git a/Lab 3/dspeech/requirements.txt b/Lab 3/dspeech/requirements.txt
index d643f3235f..20f14c845f 100644
--- a/Lab 3/dspeech/requirements.txt
+++ b/Lab 3/dspeech/requirements.txt
@@ -1,11 +1,11 @@
-colorama==0.4.4
-deepspeech==0.9.3
-halo==0.0.31
-log-symbols==0.0.14
-numpy==1.20.1
-PyAudio==0.2.11
-scipy==1.6.1
-six==1.15.0
-spinners==0.0.24
-termcolor==1.1.0
-webrtcvad==2.0.10
+colorama==0.4.4
+deepspeech==0.9.3
+halo==0.0.31
+log-symbols==0.0.14
+numpy==1.20.1
+PyAudio==0.2.11
+scipy==1.6.1
+six==1.15.0
+spinners==0.0.24
+termcolor==1.1.0
+webrtcvad==2.0.10
diff --git a/Lab 3/server.py b/Lab 3/server.py
index 71dd27c3b5..ae33946c36 100644
--- a/Lab 3/server.py
+++ b/Lab 3/server.py
@@ -1,12 +1,12 @@
-from flask import Flask
-
-app = Flask(__name__)
-
-@app.route('/')
-def index():
- return 'Hello world'
-
-if __name__ == '__main__':
- app.run(debug=True, host='0.0.0.0')
-
-
+from flask import Flask
+
+app = Flask(__name__)
+
+@app.route('/')
+def index():
+ return 'Hello world'
+
+if __name__ == '__main__':
+ app.run(debug=True, host='0.0.0.0')
+
+
diff --git a/Lab 4/README.md b/Lab 4/README.md
index 19e8cdc2b7..c6f209f5a5 100644
--- a/Lab 4/README.md
+++ b/Lab 4/README.md
@@ -1,46 +1,58 @@
# Ph-UI!!!
-For lab this week, we focus on the prototyping the physical look and feel of the device. _Make sure you read all the instructions and understand the whole of the laboratory activity before starting!_
+For lab this week, we focus on both on sensing, to bring in new modes of input into your devices, as well as prototyping the physical look and feel of the device. You will think about the physical form the device needs to perform the sensing as well as present the display or feedback about what was sensed.
+## Part 1 Lab Preparation
+### Get the latest content:
+As always, pull updates from the class Interactive-Lab-Hub to both your Pi and your own GitHub repo. As we discussed in the class, there are 2 ways you can do so:
-## Prep
+**\[recommended\]**Option 1: On the Pi, `cd` to your `Interactive-Lab-Hub`, pull the updates from upstream (class lab-hub) and push the updates back to your own GitHub repo. You will need the personal access token for this.
-1. Pull the new Github Repo.
-2. Readings:
+```
+pi@ixe00:~$ cd Interactive-Lab-Hub
+pi@ixe00:~/Interactive-Lab-Hub $ git pull upstream Fall2021
+pi@ixe00:~/Interactive-Lab-Hub $ git add .
+pi@ixe00:~/Interactive-Lab-Hub $ git commit -m "get lab4 content"
+pi@ixe00:~/Interactive-Lab-Hub $ git push
+```
-* [What do prototypes prototype?](https://www.semanticscholar.org/paper/What-do-Prototypes-Prototype-Houde-Hill/30bc6125fab9d9b2d5854223aeea7900a218f149)
+Option 2: On your your own GitHub repo, [create pull request](https://github.com/FAR-Lab/Developing-and-Designing-Interactive-Devices/blob/2021Fall/readings/Submitting%20Labs.md) to get updates from the class Interactive-Lab-Hub. After you have latest updates online, go on your Pi, `cd` to your `Interactive-Lab-Hub` and use `git pull` to get updates from your own GitHub repo.
+### Start brasinstorming ideas by reading:
+* [What do prototypes prototype?](https://www.semanticscholar.org/paper/What-do-Prototypes-Prototype-Houde-Hill/30bc6125fab9d9b2d5854223aeea7900a218f149)
* [Paper prototyping](https://www.uxpin.com/studio/blog/paper-prototyping-the-practical-beginners-guide/) is used by UX designers to quickly develop interface ideas and run them by people before any programming occurs.
-
* [Cardboard prototypes](https://www.youtube.com/watch?v=k_9Q-KDSb9o) help interactive product designers to work through additional issues, like how big something should be, how it could be carried, where it would sit.
-
* [Tips to Cut, Fold, Mold and Papier-Mache Cardboard](https://makezine.com/2016/04/21/working-with-cardboard-tips-cut-fold-mold-papier-mache/) from Make Magazine.
-
* [Surprisingly complicated forms](https://www.pinterest.com/pin/50032245843343100/) can be built with paper, cardstock or cardboard. The most advanced and challenging prototypes to prototype with paper are [cardboard mechanisms](https://www.pinterest.com/helgangchin/paper-mechanisms/) which move and change.
+* [Dyson Vacuum Cardboard Prototypes](http://media.dyson.com/downloads/JDF/JDF_Prim_poster05.pdf)
+
- Dyson Vacuum cardboard prototypes
+### Gathering materials for this lab:
+* Cardboard (start collecting those shipping boxes!)
+* Found objects and materials--like bananas and twigs.
+* Cutting board
+* Cutting tools
+* Markers
-### For lab, you will need:
+(We do offer shared cutting board, cutting tools, and markers on the class cart during the lab, so do not worry if you don't have them!)
-1. Cardboard (start collecting those shipping boxes!)
-1. Cutting board
-1. Cutting tools
-1. Markers
-1. Found objects and materials--like bananas--we're not saying that to be funny.
+## Deliverables \& Submission for Lab 4
+The deliverables for this lab are, writings, sketches, photos, and videos that show what your prototype:
+* "Looks like": shows how the device should look, feel, sit, weigh, etc.
+* "Works like": shows what the device can do.
+* "Acts like": shows how a person would interact with the device.
-### Deliverables for this lab are:
-1. Sketches/photos of device designs
-1. "Looks like" prototypes: show us what how the device should look, feel, sit, weigh, etc.
-3. "Works like" prototypes: show us what the device can do
-4. "Acts like" prototypes: videos/storyboards/other means of showing how a person would interact with the device
-5. Submit these in the lab 4 folder of your class [Github page], either as links or uploaded files. Each group member should post their own copy of the work to their own Lab Hub, even if some of the work is the same for each person in the group.
+For submission, the readme.md page for this lab should be edited to include the work you have done:
+* Upload any materials that explain what you did, into your lab 4 repository, and link them in your lab 4 readme.md.
+* Link your Lab 4 readme.md in your main Interactive-Lab-Hub readme.md.
+* Group members can turn in one repository, but make sure your Hub readme.md links to the shared repository.
+* Labs are due on Mondays, make sure to submit your Lab 4 readme.md to Canvas.
-## Overview
-Here are the parts of the assignment
+## Lab Overview
A) [Capacitive Sensing](#part-a)
@@ -48,71 +60,150 @@ B) [OLED screen](#part-b)
C) [Paper Display](#part-c)
-D) [Wizard the device](#part-d-wizard-the-device)
-
-E) [Costume the device](#part-e-costume-the-device)
-
-F) [Record the interaction](#part-f-record)
+D) [Materiality](#part-d)
-## The Report
-This readme.md page in your own repository should be edited to include the work you have done. You can delete everything but the headers and the sections between the **stars**. Write the answers to the questions under the starred sentences.
+E) [Servo Control](#part-e)
-Include any material that explains what you did in this lab hub folder, and link it in the readme.
+F) [Record the interaction](#part-f)
-Labs are due on Mondays. Make sure this page is linked to on your main class hub page.
+## The Report (Part 1: A-D, Part 2: E-F)
### Part A
-### Capacitive Sensing, a.k.a. Human Banana Interaction
+### Capacitive Sensing, a.k.a. Human-Twizzler Interaction
+
+We want to introduce you to the [capacitive sensor](https://learn.adafruit.com/adafruit-mpr121-gator) in your kit. It's one of the most flexible input devices we are able to provide. At boot, it measures the capacitance on each of the 12 contacts. Whenever that capacitance changes, it considers it a user touch. You can attach any conductive material. In your kit, you have copper tape that will work well, but don't limit yourself! In the example below, we use Twizzlers--you should pick your own objects.
-We wanted to introduce you to the [capacitive sensor](https://learn.adafruit.com/adafruit-mpr121-gator) in your kit. It's one of the most flexible input devices we were able to provide. At boot it measures the capacitance on each of the 12 contacts. Whenever that capacitance changes it considers it a user touch. You can attach any conductive material. In your kit you have conductive fabric and copper tape that will work well, but don't limit yourself! In this lab we will use (go?) bananas!
-
-Plug in the capacitive sensor board with the qwiic connector. Connect your banana's with either the copper tape or the alligator clips (the clips work better). make sure to install the requirements from `requirements.txt`
+Plug in the capacitive sensor board with the QWIIC connector. Connect your Twizzlers with either the copper tape or the alligator clips (the clips work better). In this lab, we will continue to use the `circuitpython` virtual environment we created before. Activate `circuitpython` and `cd` to your Lab 4 folder to install the requirements by:
-
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ pip3 install -r requirements.txt
+```
-I've connected my banana's* to pads 6 and 10. When you run the code and touch a banana the terminal will print out the following
+
+These Twizzlers are connected to pads 6 and 10. When you run the code and touch a Twizzler, the terminal will print out the following
```
(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python cap_test.py
-Banana 10 touched!
-Banana 6 touched!
+Twizzler 10 touched!
+Twizzler 6 touched!
```
-*\*Some students have noted that their banana's look noticeably different from the ones presented in this demo. We firmly reject the accusation that these are not in fact banana's but Twizzlers™. Due to the challenges of remote teaching we cannot debug banana's at this time. We suggest you bring these issues up with the university or your local produce representative*
-
### Part B
-### OLED screen
+### More sensors
+
+#### Light/Proximity/Gesture sensor (APDS-9960)
+
+We here want you to get to know this awesome sensor [Adafruit APDS-9960](https://www.adafruit.com/product/3595). It is capable of sensing proximity, light (also RGB), and gesture!
+
+
+
+Connect it to your pi with Qwiic connector and try running the three example scripts individually to see what the sensor is capable of doing!
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python proximity_test.py
+...
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python gesture_test.py
+...
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python color_test.py
+...
+```
+
+You can go the the [Adafruit GitHub Page](https://github.com/adafruit/Adafruit_CircuitPython_APDS9960) to see more examples for this sensor!
-We just received some of the small oled screens that we had coped to include in your kit. If you want one feel free to pop into the lab and get one. These don't have colors like the one on the pi but you can move it around on a cable making for more flexible interface design. The way you program this display is almost identical to the pi display. Take a look at `oled_test.py` and some more of the [Adafruit examples](https://github.com/adafruit/Adafruit_CircuitPython_SSD1306/tree/master/examples).
+#### Rotary Encoder
+
+A rotary encoder is an electro-mechanical device that converts the angular position to analog or digital output signals. The [Adafruit rotary encoder](https://www.adafruit.com/product/4991#technical-details) we ordered for you came with separated breakout board and encoder itself, that is, they will need to be soldered if you have not yet done so! We will be bringing the soldering station to the lab class for you to use, also, you can go to the MakerLAB to do the soldering off-class. Here is some [guidance on soldering](https://learn.adafruit.com/adafruit-guide-excellent-soldering/preparation) from Adafruit. When you first solder, get someone who has done it before (ideally in the MakerLAB environment). It is a good idea to review this material beforehand so you know what to look at.
-
-
+
+
+
+
+
+Connect it to your pi with Qwiic connector and try running the example script, it comes with an additional button which might be useful for your design!
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python encoder_test.py
+```
+
+You can go to the [Adafruit Learn Page](https://learn.adafruit.com/adafruit-i2c-qt-rotary-encoder/python-circuitpython) to learn more about the sensor! The sensor actually comes with an LED (neo pixel): Can you try lighting it up?
+
+#### Joystick
+
+A [joystick](https://www.sparkfun.com/products/15168) can be used to sense and report the input of the stick for it pivoting angle or direction. It also comes with a button input!
+
+
+
+Connect it to your pi with Qwiic connector and try running the example script to see what it can do!
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python joystick_test.py
+```
+
+You can go to the [SparkFun GitHub Page](https://github.com/sparkfun/Qwiic_Joystick_Py) to learn more about the sensor!
+
+#### (Optional) Distance Sensor
+Note: We did not distribute this sensor to you, so if you are interested in playing with it, please come pick it up from the TA!
+
+Earlier we have asked you to play with the proximity sensor, which is able to sense object within a short distance. Here, we offer [Qwiic Multi Distance Sensor](https://www.sparkfun.com/products/17072), which has a field of view of about 25° and is able to detect objects up to 3 meters away!
+
+
+
+
+
+Connect it to your pi with Qwiic connector and try running the example script to see how it works!
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python distance_test.py
+```
+
+You can go to the [SparkFun GitHub Page](https://github.com/sparkfun/Qwiic_VL53L1X_Py) to learn more about the sensor and see other examples!
### Part C
-### Paper Display
+### Physical considerations for sensing
+
+Usually, sensors need to positioned in specific locations or orientations to make them useful for their application. Now that you've tried a bunch of the sensors, pick one that you would like to use, and an application where you use the output of that sensor for an interaction. For example, you can use a distance sensor to measure someone's height if you position it overhead and get them to stand under it.
+
+**\*\*\*Draw 5 sketches of different ways you might use your sensor, and how the larger device needs to be shaped in order to make the sensor useful.\*\*\***
+
+**\*\*\*What are some things these sketches raise as questions? What do you need to physically prototype to understand how to anwer those questions?\*\*\***
-Here is an Pi with a paper faceplate on it to turn it into a display:
+**\*\*\*Pick one of these designs to prototype.\*\*\***
+
+
+### Part D
+### Physical considerations for displaying information and housing parts
+
+
+Here is an Pi with a paper faceplate on it to turn it into a display interface:
-This is fine, but it can be a bit difficult to lay out a great and user friendly display within the constraints of the Pi. Also, it really only works for applications where people can come and stand over the Pi, or where you can mount the Pi to the wall.
+This is fine, but the mounting of the display constrains the display location and orientation a lot. Also, it really only works for applications where people can come and stand over the Pi, or where you can mount the Pi to the wall.
Here is another prototype for a paper display:
-It holds a pi and usb power supply, and provides a front stage on which to put writing, graphics, LEDs, buttons or displays.
+Your kit includes these [SparkFun Qwiic OLED screens](https://www.sparkfun.com/products/17153). These use less power than the MiniTFTs you have mounted on the GPIO pins of the Pi, but, more importantly, they can be more flexibily be mounted elsewhere on your physical interface. The way you program this display is almost identical to the way you program a Pi display. Take a look at `oled_test.py` and some more of the [Adafruit examples](https://github.com/adafruit/Adafruit_CircuitPython_SSD1306/tree/master/examples).
+
+
+
+
+
+
+
+It holds a Pi and usb power supply, and provides a front stage on which to put writing, graphics, LEDs, buttons or displays.
This design can be made by scoring a long strip of corrugated cardboard of width X, with the following measurements:
@@ -125,36 +216,61 @@ Here is an example:
-
-Make a paper display for your project that communicates the state of the Pi and a sensor. Ideally you should design it so that you can slide the Pi out to work on the circuit or programming, and then slide it back in and reattach a few wires to be back in operation.
+Think about how you want to present the information about what your sensor is sensing! Design a paper display for your project that communicates the state of the Pi and a sensor. Ideally you should design it so that you can slide the Pi out to work on the circuit or programming, and then slide it back in and reattach a few wires to be back in operation.
-**a. Document the design for your paper display.** (e.g. if you had to make it again from scratch, what information would you need?). Include interim iterations (or at least tell us about them).
+**\*\*\*Sketch 5 designs for how you would physically position your display and any buttons or knobs needed to interact with it.\*\*\***
-**b. Make a video of your paper display in action.**
+**\*\*\*What are some things these sketches raise as questions? What do you need to physically prototype to understand how to anwer those questions?\*\*\***
-**c. Explain the rationale for the design.** (e.g. Does it need to be a certain size or form or need to be able to be seen from a certain distance?)
+**\*\*\*Pick one of these display designs to integrate into your prototype.\*\*\***
-### Part D
-### Materiality
+**\*\*\*Explain the rationale for the design.\*\*\*** (e.g. Does it need to be a certain size or form or need to be able to be seen from a certain distance?)
+
+Build a cardbord prototype of your design.
+
+**\*\*\*Document your rough prototype.\*\*\***
+
+
+LAB PART 2
+
+### Part 2
+
+Following exploration and reflection from Part 1, complete the "looks like," "works like" and "acts like" prototypes for your design, reiterated below.
-**Open Ended**: We are putting very few constraints on this part but we want you to get creative.
+### Part E (Optional)
+### Servo Control with Joystick
-Design a system with the Pi and anything from your kit with a focus on form, and materiality. The "stuff" that enclose the system should be informed by the desired interaction. What would a computer made of rocks be like? How would an ipod made of grass behave? Would a roomba made of gold clean your floor any differently?
+In the class kit, you should be able to find the [Qwiic Servo Controller](https://www.sparkfun.com/products/16773) and [Micro Servo Motor SG51](https://www.adafruit.com/product/2201). The Qwiic Servo Controller will need external power supply to drive, which we will be distributing the battery packs in the class. Connect the servo controller to the miniPiTFT through qwiic connector and connect the external battery to the 2-Pin JST port (ower port) on the servo controller. Connect your servo to channel 2 on the controller, make sure the brown is connected to GND and orange is connected to PWM.
+
+
+
+In this exercise, we will be using the nice [ServoKit library](https://learn.adafruit.com/16-channel-pwm-servo-driver/python-circuitpython) developed by Adafruit! We will continue to use the `circuitpython` virtual environment we created. Activate the virtual environment and make sure to install the latest required libraries by running:
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ pip3 install -r requirements.txt
+```
+
+A servo motor is a rotary actuator or linear actuator that allows for precise control of angular or linear position. The position of a servo motor is set by the width of an electrical pulse, that is, we can use PWM (pulse-width modulation) to set and control the servo motor position. You can read [this](https://learn.adafruit.com/adafruit-arduino-lesson-14-servo-motors/servo-motors) to learn a bit more about how exactly a servo motor works.
+
+Now that you have a basic idea of what a servo motor is, look into the script `qwiic_servo_example.py` we provide. In line 14, you should see that we have set up the min_pulse and max_pulse corresponding to the servo turning 0 - 180 degree. Try running the servo example code now and see what happens:
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python servo_test.py
+```
-**a. document the material prototype.** Include candidates that were considered even if they were set aside later.
+It is also possible to control the servo using the sensors mentioned in as in part A and part B, and/or from some of the buttons or parts included in your kit, the simplest way might be to chain Qwiic buttons to the other end of the Qwiic OLED. Like this:
-**b. explain the selection.**
+
-### Part 2.
+You can then call whichever control you like rather than setting a fixed value for the servo. For more information on controlling Qwiic devices, Sparkfun has several python examples, such as [this](https://learn.sparkfun.com/tutorials/qwiic-joystick-hookup-guide/all#python-examples).
-Following exploration and reflection from Part 1, complete the "looks like," "works like" and "acts like" prototypes for your design.
+We encourage you to try using these controls, **while** paying particular attention to how the interaction changes depending on the position of the controls. For example, if you have your servo rotating a screen (or a piece of cardboard) from one position to another, what changes about the interaction if the control is on the same side of the screen, or the opposite side of the screen? Trying and retrying different configurations generally helps reveal what a design choice changes about the interaction -- _make sure to document what you tried_!
-Reiterating:
-### Deliverables for this lab are:
-1. Sketches/photos of device designs
-1. "Looks like" prototypes: show us what how the device should look, feel, sit, weigh, etc.
-3. "Works like" prototypes: show us what the device can do
-4. "Acts like" prototypes: videos/storyboards/other means of showing how a person would interact with the device
-5. Submit these in the lab 4 folder of your class [Github page], either as links or uploaded files. Each group member should post their own copy of the work to their own Lab Hub, even if some of the work is the same for each person in the group.
+### Part F
+### Record
+Document all the prototypes and iterations you have designed and worked on! Again, deliverables for this lab are writings, sketches, photos, and videos that show what your prototype:
+* "Looks like": shows how the device should look, feel, sit, weigh, etc.
+* "Works like": shows what the device can do
+* "Acts like": shows how a person would interact with the device
diff --git a/Lab 4/Servo_Setup.jpg b/Lab 4/Servo_Setup.jpg
new file mode 100644
index 0000000000..002f4dbd5e
Binary files /dev/null and b/Lab 4/Servo_Setup.jpg differ
diff --git a/Lab 4/cap_test.py b/Lab 4/cap_test.py
index 70a5ec3fd7..3c50644d74 100644
--- a/Lab 4/cap_test.py
+++ b/Lab 4/cap_test.py
@@ -1,15 +1,15 @@
-import time
-import board
-import busio
-
-import adafruit_mpr121
-
-i2c = busio.I2C(board.SCL, board.SDA)
-
-mpr121 = adafruit_mpr121.MPR121(i2c)
-
-while True:
- for i in range(12):
- if mpr121[i].value:
- print(f"Banana {i} touched!")
- time.sleep(0.25) # Small delay to keep from spamming output messages.
\ No newline at end of file
+import time
+import board
+import busio
+
+import adafruit_mpr121
+
+i2c = busio.I2C(board.SCL, board.SDA)
+
+mpr121 = adafruit_mpr121.MPR121(i2c)
+
+while True:
+ for i in range(12):
+ if mpr121[i].value:
+ print(f"Twizzler {i} touched!")
+ time.sleep(0.25) # Small delay to keep from spamming output messages.
diff --git a/Lab 4/chaining.png b/Lab 4/chaining.png
new file mode 100644
index 0000000000..e8eaf23ee6
Binary files /dev/null and b/Lab 4/chaining.png differ
diff --git a/Lab 4/color_test.py b/Lab 4/color_test.py
new file mode 100644
index 0000000000..57fa7a79fe
--- /dev/null
+++ b/Lab 4/color_test.py
@@ -0,0 +1,30 @@
+# SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries
+# SPDX-License-Identifier: MIT
+
+import time
+import board
+from adafruit_apds9960.apds9960 import APDS9960
+from adafruit_apds9960 import colorutility
+
+i2c = board.I2C()
+apds = APDS9960(i2c)
+apds.enable_color = True
+
+
+while True:
+ # create some variables to store the color data in
+
+ # wait for color data to be ready
+ while not apds.color_data_ready:
+ time.sleep(0.005)
+
+ # get the data and print the different channels
+ r, g, b, c = apds.color_data
+ print("red: ", r)
+ print("green: ", g)
+ print("blue: ", b)
+ print("clear: ", c)
+
+ print("color temp {}".format(colorutility.calculate_color_temperature(r, g, b)))
+ print("light lux {}".format(colorutility.calculate_lux(r, g, b)))
+ time.sleep(0.5)
\ No newline at end of file
diff --git a/Lab 4/distance_test.py b/Lab 4/distance_test.py
new file mode 100644
index 0000000000..22b80eb3c3
--- /dev/null
+++ b/Lab 4/distance_test.py
@@ -0,0 +1,29 @@
+"""
+ Reading distance from the laser based VL53L1X
+ This example prints the distance to an object. If you are getting weird
+ readings, be sure the vacuum tape has been removed from the sensor.
+"""
+
+import qwiic
+import time
+
+print("VL53L1X Qwiic Test\n")
+ToF = qwiic.QwiicVL53L1X()
+if (ToF.sensor_init() == None): # Begin returns 0 on a good init
+ print("Sensor online!\n")
+
+while True:
+ try:
+ ToF.start_ranging() # Write configuration bytes to initiate measurement
+ time.sleep(.005)
+ distance = ToF.get_distance() # Get the result of the measurement from the sensor
+ time.sleep(.005)
+ ToF.stop_ranging()
+
+ distanceInches = distance / 25.4
+ distanceFeet = distanceInches / 12.0
+
+ print("Distance(mm): %s Distance(ft): %s" % (distance, distanceFeet))
+
+ except Exception as e:
+ print(e)
\ No newline at end of file
diff --git a/Lab 4/encoder_test.py b/Lab 4/encoder_test.py
new file mode 100644
index 0000000000..8ecc7c1818
--- /dev/null
+++ b/Lab 4/encoder_test.py
@@ -0,0 +1,43 @@
+# SPDX-FileCopyrightText: 2021 John Furcean
+# SPDX-License-Identifier: MIT
+
+"""I2C rotary encoder simple test example."""
+
+import board
+from adafruit_seesaw import seesaw, rotaryio, digitalio
+
+# For use with the STEMMA connector on QT Py RP2040
+# import busio
+# i2c = busio.I2C(board.SCL1, board.SDA1)
+# seesaw = seesaw.Seesaw(i2c, 0x36)
+
+seesaw = seesaw.Seesaw(board.I2C(), addr=0x36)
+
+seesaw_product = (seesaw.get_version() >> 16) & 0xFFFF
+print("Found product {}".format(seesaw_product))
+if seesaw_product != 4991:
+ print("Wrong firmware loaded? Expected 4991")
+
+seesaw.pin_mode(24, seesaw.INPUT_PULLUP)
+button = digitalio.DigitalIO(seesaw, 24)
+button_held = False
+
+encoder = rotaryio.IncrementalEncoder(seesaw)
+last_position = None
+
+while True:
+
+ # negate the position to make clockwise rotation positive
+ position = -encoder.position
+
+ if position != last_position:
+ last_position = position
+ print("Position: {}".format(position))
+
+ if not button.value and not button_held:
+ button_held = True
+ print("Button pressed")
+
+ if button.value and button_held:
+ button_held = False
+ print("Button released")
\ No newline at end of file
diff --git a/Lab 4/gesture_test.py b/Lab 4/gesture_test.py
new file mode 100644
index 0000000000..2d4d455df4
--- /dev/null
+++ b/Lab 4/gesture_test.py
@@ -0,0 +1,26 @@
+# SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries
+# SPDX-License-Identifier: MIT
+
+import board
+from adafruit_apds9960.apds9960 import APDS9960
+
+i2c = board.I2C()
+
+apds = APDS9960(i2c)
+apds.enable_proximity = True
+apds.enable_gesture = True
+
+# Uncomment and set the rotation if depending on how your sensor is mounted.
+# apds.rotation = 270 # 270 for CLUE
+
+while True:
+ gesture = apds.gesture()
+
+ if gesture == 0x01:
+ print("up")
+ elif gesture == 0x02:
+ print("down")
+ elif gesture == 0x03:
+ print("left")
+ elif gesture == 0x04:
+ print("right")
\ No newline at end of file
diff --git a/Lab 4/joystick_test.py b/Lab 4/joystick_test.py
new file mode 100644
index 0000000000..133ad115d5
--- /dev/null
+++ b/Lab 4/joystick_test.py
@@ -0,0 +1,34 @@
+from __future__ import print_function
+import qwiic_joystick
+import time
+import sys
+
+def runExample():
+
+ print("\nSparkFun qwiic Joystick Example 1\n")
+ myJoystick = qwiic_joystick.QwiicJoystick()
+
+ if myJoystick.connected == False:
+ print("The Qwiic Joystick device isn't connected to the system. Please check your connection", \
+ file=sys.stderr)
+ return
+
+ myJoystick.begin()
+
+ print("Initialized. Firmware Version: %s" % myJoystick.version)
+
+ while True:
+
+ print("X: %d, Y: %d, Button: %d" % ( \
+ myJoystick.horizontal, \
+ myJoystick.vertical, \
+ myJoystick.button))
+
+ time.sleep(.5)
+
+if __name__ == '__main__':
+ try:
+ runExample()
+ except (KeyboardInterrupt, SystemExit) as exErr:
+ print("\nEnding Example 1")
+ sys.exit(0)
\ No newline at end of file
diff --git a/Lab 4/oled_test.py b/Lab 4/oled_test.py
index cfa8c44d62..b58ee20362 100644
--- a/Lab 4/oled_test.py
+++ b/Lab 4/oled_test.py
@@ -1,87 +1,87 @@
-# SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries
-# SPDX-License-Identifier: MIT
-
-import board
-import busio
-import adafruit_ssd1306
-
-# Create the I2C interface.
-i2c = busio.I2C(board.SCL, board.SDA)
-
-# Create the SSD1306 OLED class.
-# The first two parameters are the pixel width and pixel height. Change these
-# to the right size for your display!
-oled = adafruit_ssd1306.SSD1306_I2C(128, 32, i2c)
-
-
-# Helper function to draw a circle from a given position with a given radius
-# This is an implementation of the midpoint circle algorithm,
-# see https://en.wikipedia.org/wiki/Midpoint_circle_algorithm#C_example for details
-def draw_circle(xpos0, ypos0, rad, col=1):
- x = rad - 1
- y = 0
- dx = 1
- dy = 1
- err = dx - (rad << 1)
- while x >= y:
- oled.pixel(xpos0 + x, ypos0 + y, col)
- oled.pixel(xpos0 + y, ypos0 + x, col)
- oled.pixel(xpos0 - y, ypos0 + x, col)
- oled.pixel(xpos0 - x, ypos0 + y, col)
- oled.pixel(xpos0 - x, ypos0 - y, col)
- oled.pixel(xpos0 - y, ypos0 - x, col)
- oled.pixel(xpos0 + y, ypos0 - x, col)
- oled.pixel(xpos0 + x, ypos0 - y, col)
- if err <= 0:
- y += 1
- err += dy
- dy += 2
- if err > 0:
- x -= 1
- dx += 2
- err += dx - (rad << 1)
-
-
-# initial center of the circle
-center_x = 63
-center_y = 15
-# how fast does it move in each direction
-x_inc = 1
-y_inc = 1
-# what is the starting radius of the circle
-radius = 8
-
-# start with a blank screen
-oled.fill(0)
-# we just blanked the framebuffer. to push the framebuffer onto the display, we call show()
-oled.show()
-while True:
- # undraw the previous circle
- draw_circle(center_x, center_y, radius, col=0)
-
- # if bouncing off right
- if center_x + radius >= oled.width:
- # start moving to the left
- x_inc = -1
- # if bouncing off left
- elif center_x - radius < 0:
- # start moving to the right
- x_inc = 1
-
- # if bouncing off top
- if center_y + radius >= oled.height:
- # start moving down
- y_inc = -1
- # if bouncing off bottom
- elif center_y - radius < 0:
- # start moving up
- y_inc = 1
-
- # go more in the current direction
- center_x += x_inc
- center_y += y_inc
-
- # draw the new circle
- draw_circle(center_x, center_y, radius)
- # show all the changes we just made
+# SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries
+# SPDX-License-Identifier: MIT
+
+import board
+import busio
+import adafruit_ssd1306
+
+# Create the I2C interface.
+i2c = busio.I2C(board.SCL, board.SDA)
+
+# Create the SSD1306 OLED class.
+# The first two parameters are the pixel width and pixel height. Change these
+# to the right size for your display!
+oled = adafruit_ssd1306.SSD1306_I2C(128, 32, i2c)
+
+
+# Helper function to draw a circle from a given position with a given radius
+# This is an implementation of the midpoint circle algorithm,
+# see https://en.wikipedia.org/wiki/Midpoint_circle_algorithm#C_example for details
+def draw_circle(xpos0, ypos0, rad, col=1):
+ x = rad - 1
+ y = 0
+ dx = 1
+ dy = 1
+ err = dx - (rad << 1)
+ while x >= y:
+ oled.pixel(xpos0 + x, ypos0 + y, col)
+ oled.pixel(xpos0 + y, ypos0 + x, col)
+ oled.pixel(xpos0 - y, ypos0 + x, col)
+ oled.pixel(xpos0 - x, ypos0 + y, col)
+ oled.pixel(xpos0 - x, ypos0 - y, col)
+ oled.pixel(xpos0 - y, ypos0 - x, col)
+ oled.pixel(xpos0 + y, ypos0 - x, col)
+ oled.pixel(xpos0 + x, ypos0 - y, col)
+ if err <= 0:
+ y += 1
+ err += dy
+ dy += 2
+ if err > 0:
+ x -= 1
+ dx += 2
+ err += dx - (rad << 1)
+
+
+# initial center of the circle
+center_x = 63
+center_y = 15
+# how fast does it move in each direction
+x_inc = 1
+y_inc = 1
+# what is the starting radius of the circle
+radius = 8
+
+# start with a blank screen
+oled.fill(0)
+# we just blanked the framebuffer. to push the framebuffer onto the display, we call show()
+oled.show()
+while True:
+ # undraw the previous circle
+ draw_circle(center_x, center_y, radius, col=0)
+
+ # if bouncing off right
+ if center_x + radius >= oled.width:
+ # start moving to the left
+ x_inc = -1
+ # if bouncing off left
+ elif center_x - radius < 0:
+ # start moving to the right
+ x_inc = 1
+
+ # if bouncing off top
+ if center_y + radius >= oled.height:
+ # start moving down
+ y_inc = -1
+ # if bouncing off bottom
+ elif center_y - radius < 0:
+ # start moving up
+ y_inc = 1
+
+ # go more in the current direction
+ center_x += x_inc
+ center_y += y_inc
+
+ # draw the new circle
+ draw_circle(center_x, center_y, radius)
+ # show all the changes we just made
oled.show()
\ No newline at end of file
diff --git a/Lab 4/proximity_test.py b/Lab 4/proximity_test.py
new file mode 100644
index 0000000000..9d0799f61c
--- /dev/null
+++ b/Lab 4/proximity_test.py
@@ -0,0 +1,15 @@
+# SPDX-FileCopyrightText: 2021 ladyada for Adafruit Industries
+# SPDX-License-Identifier: MIT
+
+import time
+import board
+from adafruit_apds9960.apds9960 import APDS9960
+
+i2c = board.I2C()
+apds = APDS9960(i2c)
+
+apds.enable_proximity = True
+
+while True:
+ print(apds.proximity)
+ time.sleep(0.2)
\ No newline at end of file
diff --git a/Lab 4/requirements.txt b/Lab 4/requirements.txt
index 2265862603..e88eb7a445 100644
--- a/Lab 4/requirements.txt
+++ b/Lab 4/requirements.txt
@@ -2,7 +2,15 @@ Adafruit-Blinka==6.4.1
adafruit-circuitpython-busdevice==5.0.6
adafruit-circuitpython-framebuf==1.4.6
adafruit-circuitpython-mpr121==2.1.7
+adafruit-circuitpython-mpu6050==1.1.6
adafruit-circuitpython-ssd1306==2.11.2
+adafruit-circuitpython-pca9685==3.3.8
+adafruit-circuitpython-servokit==1.3.5
+adafruit-circuitpython-apds9960==2.2.5
+adafruit-circuitpython-seesaw==1.10.0
+sparkfun-qwiic==1.1.5
+sparkfun-qwiic-joystick==0.9.0
+sparkfun-qwiic-vl53l1x==1.0.1
Adafruit-GPIO==1.0.3
Adafruit-PlatformDetect==3.4.1
Adafruit-PureIO==1.1.8
@@ -13,4 +21,4 @@ pyusb==1.1.1
rpi-ws281x==4.2.6
RPi.GPIO==0.7.0
spidev==3.5
-sysv-ipc==1.1.0
\ No newline at end of file
+sysv-ipc==1.1.0
diff --git a/Lab 4/servo_test.py b/Lab 4/servo_test.py
new file mode 100644
index 0000000000..cb8b094ccb
--- /dev/null
+++ b/Lab 4/servo_test.py
@@ -0,0 +1,28 @@
+import time
+from adafruit_servokit import ServoKit
+
+# Set channels to the number of servo channels on your kit.
+# There are 16 channels on the PCA9685 chip.
+kit = ServoKit(channels=16)
+
+# Name and set up the servo according to the channel you are using.
+servo = kit.servo[2]
+
+# Set the pulse width range of your servo for PWM control of rotating 0-180 degree (min_pulse, max_pulse)
+# Each servo might be different, you can normally find this information in the servo datasheet
+servo.set_pulse_width_range(500, 2500)
+
+while True:
+ try:
+ # Set the servo to 180 degree position
+ servo.angle = 180
+ time.sleep(2)
+ # Set the servo to 0 degree position
+ servo.angle = 0
+ time.sleep(2)
+
+ except KeyboardInterrupt:
+ # Once interrupted, set the servo back to 0 degree position
+ servo.angle = 0
+ time.sleep(0.5)
+ break
diff --git a/Lab 5/HandTrackingModule.py b/Lab 5/HandTrackingModule.py
new file mode 100644
index 0000000000..2769d1c384
--- /dev/null
+++ b/Lab 5/HandTrackingModule.py
@@ -0,0 +1,71 @@
+import cv2
+import mediapipe as mp
+import time
+
+
+class handDetector():
+ def __init__(self, mode=False, maxHands=2, detectionCon=0.5, trackCon=0.5):
+ self.mode = mode
+ self.maxHands = maxHands
+ self.detectionCon = detectionCon
+ self.trackCon = trackCon
+
+ self.mpHands = mp.solutions.hands
+ self.hands = self.mpHands.Hands(self.mode, self.maxHands,
+ self.detectionCon, self.trackCon)
+ self.mpDraw = mp.solutions.drawing_utils
+
+ def findHands(self, img, draw=True):
+ imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
+ self.results = self.hands.process(imgRGB)
+ # print(results.multi_hand_landmarks)
+
+ if self.results.multi_hand_landmarks:
+ for handLms in self.results.multi_hand_landmarks:
+ if draw:
+ self.mpDraw.draw_landmarks(img, handLms,
+ self.mpHands.HAND_CONNECTIONS)
+ return img
+
+ def findPosition(self, img, handNo=0, draw=True):
+
+ lmList = []
+ if self.results.multi_hand_landmarks:
+ myHand = self.results.multi_hand_landmarks[handNo]
+ for id, lm in enumerate(myHand.landmark):
+ # print(id, lm)
+ h, w, c = img.shape
+ cx, cy = int(lm.x * w), int(lm.y * h)
+ # print(id, cx, cy)
+ lmList.append([id, cx, cy])
+ if draw:
+ cv2.circle(img, (cx, cy), 15, (255, 0, 255), cv2.FILLED)
+
+ return lmList
+
+
+def main():
+ pTime = 0
+ cTime = 0
+ cap = cv2.VideoCapture(1)
+ detector = handDetector()
+ while True:
+ success, img = cap.read()
+ img = detector.findHands(img)
+ lmList = detector.findPosition(img)
+ if len(lmList) != 0:
+ print(lmList[4])
+
+ cTime = time.time()
+ fps = 1 / (cTime - pTime)
+ pTime = cTime
+
+ cv2.putText(img, str(int(fps)), (10, 70), cv2.FONT_HERSHEY_PLAIN, 3,
+ (255, 0, 255), 3)
+
+ cv2.imshow("Image", img)
+ cv2.waitKey(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/Lab 5/README.md b/Lab 5/README.md
index 992ee3b302..0319d877d6 100644
--- a/Lab 5/README.md
+++ b/Lab 5/README.md
@@ -4,32 +4,26 @@
For lab this week, we focus on creating interactive systems that can detect and respond to events or stimuli in the environment of the Pi, like the Boat Detector we mentioned in lecture.
Your **observant device** could, for example, count items, find objects, recognize an event or continuously monitor a room.
-This lab will help you think through the design of observant systems, particularly corner cases that the algorithms need to be aware of.
-
-In Lab 5 part 1, we focus on detecting and sense-making.
-
-In Lab 5 part 2, we'll incorporate interactive responses.
-
+This lab will help you think through the design of observant systems, particularly corner cases that the algorithms needs to be aware of.
## Prep
1. Pull the new Github Repo.
-2. Read about [OpenCV](https://opencv.org/about/).
-3. Read Belloti, et al's [Making Sense of Sensing Systems: Five Questions for Designers and Researchers](https://www.cc.gatech.edu/~keith/pubs/chi2002-sensing.pdf)
+2. Install VNC on your laptop if you have not yet done so. This lab will actually require you to run script on your Pi through VNC so that you can see the video stream. Please refer to the [prep for Lab 2](https://github.com/FAR-Lab/Interactive-Lab-Hub/blob/Fall2021/Lab%202/prep.md), we offered the instruction at the bottom.
+3. Read about [OpenCV](https://opencv.org/about/), [MediaPipe](https://mediapipe.dev/), and [TeachableMachines](https://teachablemachine.withgoogle.com/).
+4. Read Belloti, et al.'s [Making Sense of Sensing Systems: Five Questions for Designers and Researchers](https://www.cc.gatech.edu/~keith/pubs/chi2002-sensing.pdf).
### For the lab, you will need:
1. Raspberry Pi
-1. Raspberry Pi Camera (2.1)
-1. Microphone (if you want speech or sound input)
-1. Webcam (if you want to be able to locate the camera more flexibly than the Pi Camera)
+1. Webcam
+1. Microphone (if you want to have speech or sound input for your design)
### Deliverables for this lab are:
1. Show pictures, videos of the "sense-making" algorithms you tried.
1. Show a video of how you embed one of these algorithms into your observant system.
1. Test, characterize your interactive device. Show faults in the detection and how the system handled it.
-
## Overview
Building upon the paper-airplane metaphor (we're understanding the material of machine learning for design), here are the four sections of the lab activity:
@@ -46,36 +40,37 @@ D) [Reflect](#part-d)
### Part A
### Play with different sense-making algorithms.
-Befor you get started connect the RaspberryPi Camera V2. [The Pi hut has a great explanation on how to do that](https://thepihut.com/blogs/raspberry-pi-tutorials/16021420-how-to-install-use-the-raspberry-pi-camera).
-
#### OpenCV
-A more traditional to extract information out of images is provided with OpenCV. The RPI image provided to you comes with an optimized installation that can be accessed through python.
+A more traditional method to extract information out of images is provided with OpenCV. The RPI image provided to you comes with an optimized installation that can be accessed through python. We included 4 standard OpenCV examples: contour(blob) detection, face detection with the ``Haarcascade``, flow detection (a type of keypoint tracking), and standard object detection with the [Yolo](https://pjreddie.com/darknet/yolo/) darknet.
-Additionally, we also included 4 standard OpenCV examples. These examples include contour(blob) detection, face detection with the ``Haarcascade``, flow detection(a type of keypoint tracking), and standard object detection with the [Yolo](https://pjreddie.com/darknet/yolo/) darknet.
+Most examples can be run with a screen (e.g. VNC or ssh -X or with an HDMI monitor), or with just the terminal. The examples are separated out into different folders. Each folder contains a ```HowToUse.md``` file, which explains how to run the python example.
-Most examples can be run with a screen (I.e. VNC or ssh -X or with an HDMI monitor), or with just the terminal. The examples are separated out into different folders. Each folder contains a ```HowToUse.md``` file, which explains how to run the python example.
+Following is a nicer way you can run and see the flow of the `openCV-examples` we have included in your Pi. Instead of `ls`, the command we will be using here is `tree`. [Tree](http://mama.indstate.edu/users/ice/tree/) is a recursive directory colored listing command that produces a depth indented listing of files. Install `tree` first and `cd` to the `openCV-examples` folder and run the command:
```shell
+pi@ixe00:~ $ sudo apt install tree
+...
+pi@ixe00:~ $ cd openCV-examples
pi@ixe00:~/openCV-examples $ tree -l
.
├── contours-detection
-│ ├── contours.py
-│ └── HowToUse.md
+│ ├── contours.py
+│ └── HowToUse.md
├── data
-│ ├── slow_traffic_small.mp4
-│ └── test.jpg
+│ ├── slow_traffic_small.mp4
+│ └── test.jpg
├── face-detection
-│ ├── face-detection.py
-│ ├── faces_detected.jpg
-│ ├── haarcascade_eye_tree_eyeglasses.xml
-│ ├── haarcascade_eye.xml
-│ ├── haarcascade_frontalface_alt.xml
-│ ├── haarcascade_frontalface_default.xml
-│ └── HowToUse.md
+│ ├── face-detection.py
+│ ├── faces_detected.jpg
+│ ├── haarcascade_eye_tree_eyeglasses.xml
+│ ├── haarcascade_eye.xml
+│ ├── haarcascade_frontalface_alt.xml
+│ ├── haarcascade_frontalface_default.xml
+│ └── HowToUse.md
├── flow-detection
-│ ├── flow.png
-│ ├── HowToUse.md
-│ └── optical_flow.py
+│ ├── flow.png
+│ ├── HowToUse.md
+│ └── optical_flow.py
└── object-detection
├── detected_out.jpg
├── detect.py
@@ -83,55 +78,138 @@ pi@ixe00:~/openCV-examples $ tree -l
├── HowToUse.md
└── ssd_mobilenet_v2_coco_2018_03_29.pbtxt
```
-#### Filtering, FFTs, and Time Series data. (beta, optional)
-Additional filtering and analysis can be done on the sensors that were provided in the kit. For example, running a Fast Fourier Transform over the IMU data stream could create a simple activity classifier between walking, running, and standing.
-Using the set up from the [Lab 3 demo](https://github.com/FAR-Lab/Interactive-Lab-Hub/tree/Spring2021/Lab%203/demo) and the accelerometer, try the following:
+The flow detection might seem random, but consider [this recent research](https://cseweb.ucsd.edu/~lriek/papers/taylor-icra-2021.pdf) that uses optical flow to determine busy-ness in hospital settings to facilitate robot navigation. Note the velocity parameter on page 3 and the mentions of optical flow.
-**1. Set up threshold detection** Can you identify when a signal goes above certain fixed values?
+Now, connect your webcam to your Pi and use **VNC to access to your Pi** and open the terminal. Use the following command lines to try each of the examples we provided:
+(***it will not work if you use ssh from your laptop***)
-**2. Set up averaging** Can you average your signal in N-sample blocks? N-sample running average?
+```
+pi@ixe00:~$ cd ~/openCV-examples/contours-detection
+pi@ixe00:~/openCV-examples/contours-detection $ python contours.py
+...
+pi@ixe00:~$ cd ~/openCV-examples/face-detection
+pi@ixe00:~/openCV-examples/face-detection $ python face-detection.py
+...
+pi@ixe00:~$ cd ~/openCV-examples/flow-detection
+pi@ixe00:~/openCV-examples/flow-detection $ python optical_flow.py 0 window
+...
+pi@ixe00:~$ cd ~/openCV-examples/object-detection
+pi@ixe00:~/openCV-examples/object-detection $ python detect.py
+```
-**3. Set up peak detection** Can you identify when your signal reaches a peak and then goes down?
+**\*\*\*Try each of the following four examples in the `openCV-examples`, include screenshots of your use and write about one design for each example that might work based on the individual benefits to each algorithm.\*\*\***
+
+#### MediaPipe
+
+A more recent open source and efficient method of extracting information from video streams comes out of Google's [MediaPipe](https://mediapipe.dev/), which offers state of the art face, face mesh, hand pose, and body pose detection.
+
+
+
+To get started, create a new virtual environment with special indication this time:
+
+```
+pi@ixe00:~ $ virtualenv mpipe --system-site-packages
+pi@ixe00:~ $ source mpipe/bin/activate
+(mpipe) pi@ixe00:~ $
+```
+
+and install the following.
+
+```
+...
+(mpipe) pi@ixe00:~ $ sudo apt install ffmpeg python3-opencv
+(mpipe) pi@ixe00:~ $ sudo apt install libxcb-shm0 libcdio-paranoia-dev libsdl2-2.0-0 libxv1 libtheora0 libva-drm2 libva-x11-2 libvdpau1 libharfbuzz0b libbluray2 libatlas-base-dev libhdf5-103 libgtk-3-0 libdc1394-22 libopenexr23
+(mpipe) pi@ixe00:~ $ pip3 install mediapipe-rpi4 pyalsaaudio
+```
+
+Each of the installs will take a while, please be patient. After successfully installing mediapipe, connect your webcam to your Pi and use **VNC to access to your Pi**, open the terminal, and go to Lab 5 folder and run the hand pose detection script we provide:
+(***it will not work if you use ssh from your laptop***)
+
+
+```
+(mpipe) pi@ixe00:~ $ cd Interactive-Lab-Hub/Lab\ 5
+(mpipe) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ python hand_pose.py
+```
+
+Try the two main features of this script: 1) pinching for percentage control, and 2) "[Quiet Coyote](https://www.youtube.com/watch?v=qsKlNVpY7zg)" for instant percentage setting. Notice how this example uses hardcoded positions and relates those positions with a desired set of events, in `hand_pose.py` lines 48-53.
-Include links to your code here, and put the code for these in your repo--they will come in handy later.
+**\*\*\*Consider how you might use this position based approach to create an interaction, and write how you might use it on either face, hand or body pose tracking.\*\*\***
-#### Teachable Machines (beta, optional)
-Google's [TeachableMachines](https://teachablemachine.withgoogle.com/train) might look very simple. However, its simplicity is very useful for experimenting with the capabilities of this technology.
+(You might also consider how this notion of percentage control with hand tracking might be used in some of the physical UI you may have experimented with in the last lab, for instance in controlling a servo or rotary encoder.)
-You can train a Model on your browser, experiment with its performance, and then port it to the Raspberry Pi to do even its task on the device.
-Here is Adafruit's directions on using Raspberry Pi and the Pi camera with Teachable Machines:
-1. [Setup](https://learn.adafruit.com/teachable-machine-raspberry-pi-tensorflow-camera/raspberry-pi-setup)
-2. Install Tensorflow: Like [this](https://learn.adafruit.com/running-tensorflow-lite-on-the-raspberry-pi-4/tensorflow-lite-2-setup), but use this [pre-built binary](https://github.com/bitsy-ai/tensorflow-arm-bin/) [the file](https://github.com/bitsy-ai/tensorflow-arm-bin/releases/download/v2.4.0/tensorflow-2.4.0-cp37-none-linux_armv7l.whl) for Tensorflow, it will speed things up a lot.
-3. [Collect data and train models using the PiCam](https://learn.adafruit.com/teachable-machine-raspberry-pi-tensorflow-camera/training)
-4. [Export and run trained models on the Pi](https://learn.adafruit.com/teachable-machine-raspberry-pi-tensorflow-camera/transferring-to-the-pi)
+#### Teachable Machines
+Google's [TeachableMachines](https://teachablemachine.withgoogle.com/train) might look very simple. However, its simplicity is very useful for experimenting with the capabilities of this technology.
-Alternative less steps option is [here](https://github.com/FAR-Lab/TensorflowonThePi).
+
+
+To get started, create and activate a new virtual environment for this exercise with special indication:
+
+```
+pi@ixe00:~ $ virtualenv tmachine --system-site-packages
+pi@ixe00:~ $ source tmachine/bin/activate
+(tmachine) pi@ixe00:~ $
+```
+
+After activating the virtual environment, install the requisite TensorFlow libraries by running the following lines:
+```
+(tmachine) pi@ixe00:~ $ cd Interactive-Lab-Hub/Lab\ 5
+(tmachine) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ sudo chmod +x ./teachable_machines.sh
+(tmachine) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ ./teachable_machines.sh
+```
+
+This might take a while to get fully installed. After installation, connect your webcam to your Pi and use **VNC to access to your Pi**, open the terminal, and go to Lab 5 folder and run the example script:
+(***it will not work if you use ssh from your laptop***)
+
+```
+(tmachine) pi@ixe00:~ Interactive-Lab-Hub/Lab 5 $ python tm_ppe_detection.py
+```
+
+
+(**Optionally**: You can train your own model, too. First, visit [TeachableMachines](https://teachablemachine.withgoogle.com/train), select Image Project and Standard model. Second, use the webcam on your computer to train a model. For each class try to have over 50 samples, and consider adding a background class where you have nothing in view so the model is trained to know that this is the background. Then create classes based on what you want the model to classify. Lastly, preview and iterate, or export your model as a 'Tensorflow' model, and select 'Keras'. You will find an '.h5' file and a 'labels.txt' file. These are included in this labs 'teachable_machines' folder, to make the PPE model you used earlier. You can make your own folder or replace these to make your own classifier.)
+
+**\*\*\*Whether you make your own model or not, include screenshots of your use of Teachable Machines, and write how you might use this to create your own classifier. Include what different affordances this method brings, compared to the OpenCV or MediaPipe options.\*\*\***
+
+
+*Don't forget to run ```deactivate``` to end the Teachable Machines demo, and to reactivate with ```source tmachine/bin/activate``` when you want to use it again.*
+
+
+#### Filtering, FFTs, and Time Series data. (optional)
+Additional filtering and analysis can be done on the sensors that were provided in the kit. For example, running a Fast Fourier Transform over the IMU data stream could create a simple activity classifier between walking, running, and standing.
+
+Using the accelerometer, try the following:
+
+**1. Set up threshold detection** Can you identify when a signal goes above certain fixed values?
+
+**2. Set up averaging** Can you average your signal in N-sample blocks? N-sample running average?
+
+**3. Set up peak detection** Can you identify when your signal reaches a peak and then goes down?
+
+**\*\*\*Include links to your code here, and put the code for these in your repo--they will come in handy later.\*\*\***
-#### PyTorch
-As a note, the global Python install contains also a PyTorch installation. That can be experimented with as well if you are so inclined.
### Part B
### Construct a simple interaction.
Pick one of the models you have tried, pick a class of objects, and experiment with prototyping an interaction.
This can be as simple as the boat detector earlier.
-Try out different interactions outputs and inputs.
-**Describe and detail the interaction, as well as your experimentation.**
+Try out different interaction outputs and inputs.
+
+**\*\*\*Describe and detail the interaction, as well as your experimentation here.\*\*\***
### Part C
### Test the interaction prototype
-Now flight test your interactive prototype and **note your observations**:
+Now flight test your interactive prototype and **note down your observations**:
For example:
1. When does it what it is supposed to do?
1. When does it fail?
1. When it fails, why does it fail?
1. Based on the behavior you have seen, what other scenarios could cause problems?
-**Think about someone using the system. Describe how you think this will work.**
+**\*\*\*Think about someone using the system. Describe how you think this will work.\*\*\***
1. Are they aware of the uncertainties in the system?
1. How bad would they be impacted by a miss classification?
1. How could change your interactive system to address this?
@@ -150,11 +228,10 @@ During the lecture, we mentioned questions to help characterize a material:
* What are other properties/behaviors of X?
* How does X feel?
-**Include a short video demonstrating the answers to these questions.**
+**\*\*\*Include a short video demonstrating the answers to these questions.\*\*\***
### Part 2.
Following exploration and reflection from Part 1, finish building your interactive system, and demonstrate it in use with a video.
-**Include a short video demonstrating the finished result.**
-
+**\*\*\*Include a short video demonstrating the finished result.\*\*\***
diff --git a/Lab 5/hand_pose.py b/Lab 5/hand_pose.py
new file mode 100644
index 0000000000..87268faf8a
--- /dev/null
+++ b/Lab 5/hand_pose.py
@@ -0,0 +1,88 @@
+import cv2
+import time
+import numpy as np
+import HandTrackingModule as htm
+import math
+from ctypes import cast, POINTER
+import alsaaudio
+m = alsaaudio.Mixer()
+################################
+wCam, hCam = 640, 480
+################################
+
+cap = cv2.VideoCapture(0)
+cap.set(3, wCam)
+cap.set(4, hCam)
+pTime = 0
+
+detector = htm.handDetector(detectionCon=0.7)
+minVol = 0
+maxVol = 100
+vol = 0
+volBar = 400
+volPer = 0
+while True:
+ success, img = cap.read()
+ img = detector.findHands(img)
+ lmList = detector.findPosition(img, draw=False)
+ if len(lmList) != 0:
+
+ thumbX, thumbY = lmList[4][1], lmList[4][2] #thumb
+ pointerX, pointerY = lmList[8][1], lmList[8][2] #pointer
+
+ middleX, middleY = lmList[12][1], lmList[12][2]
+ ringX, ringY = lmList[16][1], lmList[16][2]
+ pinkyX, pinkyY = lmList[20][1], lmList[20][2]
+
+ cx, cy = (thumbX + pointerX) // 2, (thumbY + pointerY) // 2
+
+ cv2.circle(img, (thumbX, thumbY), 15, (255, 0, 255), cv2.FILLED)
+ cv2.circle(img, (pointerX, pointerY), 15, (255, 0, 255), cv2.FILLED)
+ cv2.circle(img, (middleX, middleY), 15, (255, 0, 255), cv2.FILLED)
+ cv2.circle(img, (ringX, ringY), 15, (255, 0, 255), cv2.FILLED)
+ cv2.circle(img, (pinkyX, pinkyY), 15, (255, 0, 255), cv2.FILLED)
+ cv2.line(img, (thumbX, thumbY), (pointerX, pointerY), (255, 0, 255), 3)
+ cv2.circle(img, (cx, cy), 15, (255, 0, 255), cv2.FILLED)
+
+ len_calc = lambda x1,y1,x2,y2: math.hypot(x2 - x1, y2 - y1)
+ length = len_calc(thumbX,thumbY,pointerX,pointerY)
+ length1 = len_calc(pointerX,pointerY,middleX,middleY)
+ length2 = len_calc(middleX, middleY, ringX, ringY)
+ length3 = len_calc(ringX, ringY, pinkyX, pinkyY)
+ length4 = len_calc(thumbX,thumbY, ringX, ringY)
+ print(length1,length2,length3)
+ condition = length>100 and length1>100 and length2<100 and length3>100 and length4<100
+ if condition:
+ m.setvolume(0)
+ volPer = 0
+ volBar = 400
+ print("CONDITION")
+ cv2.putText(img, 'quiet coyote!', (40, 70), cv2.FONT_HERSHEY_COMPLEX,
+ 1, (255, 255, 255), 3)
+ else:
+
+ vol = np.interp(length, [50, 300], [minVol, maxVol])
+ volBar = np.interp(length, [50, 300], [400, 150])
+ volPer = np.interp(length, [50, 300], [0, 100])
+ m.setvolume(int(vol))
+
+ print(int(length), vol)
+
+
+ if length < 50:
+ cv2.circle(img, (cx, cy), 15, (0, 255, 0), cv2.FILLED)
+
+ cv2.rectangle(img, (50, 150), (85, 400), (255, 0, 0), 3)
+ cv2.rectangle(img, (50, int(volBar)), (85, 400), (255, 0, 0), cv2.FILLED)
+ cv2.putText(img, f'{int(volPer)} %', (40, 450), cv2.FONT_HERSHEY_COMPLEX,
+ 1, (255, 0, 0), 3)
+
+
+ cTime = time.time()
+ fps = 1 / (cTime - pTime)
+ pTime = cTime
+ cv2.putText(img, f'FPS: {int(fps)}', (40, 50), cv2.FONT_HERSHEY_COMPLEX,
+ 1, (255, 0, 0), 3)
+
+ cv2.imshow("Img", img)
+ cv2.waitKey(1)
\ No newline at end of file
diff --git a/Lab 5/keras_model.h5 b/Lab 5/keras_model.h5
new file mode 100644
index 0000000000..c83e42b346
Binary files /dev/null and b/Lab 5/keras_model.h5 differ
diff --git a/Lab 5/labels.txt b/Lab 5/labels.txt
new file mode 100644
index 0000000000..6bc4c4b76b
--- /dev/null
+++ b/Lab 5/labels.txt
@@ -0,0 +1,3 @@
+0 no mask
+1 mask
+2 background
diff --git a/Lab 5/mp.gif b/Lab 5/mp.gif
new file mode 100644
index 0000000000..60a95d4380
Binary files /dev/null and b/Lab 5/mp.gif differ
diff --git a/Lab 5/teachable_machines.sh b/Lab 5/teachable_machines.sh
new file mode 100644
index 0000000000..45a1c48560
--- /dev/null
+++ b/Lab 5/teachable_machines.sh
@@ -0,0 +1,6 @@
+#download
+wget https://github.com/bitsy-ai/tensorflow-arm-bin/releases/download/v2.4.0/tensorflow-2.4.0-cp37-none-linux_armv7l.whl
+#install
+pip install tensorflow-2.4.0-cp37-none-linux_armv7l.whl
+
+echo "Done!"
diff --git a/Lab 5/tm.gif b/Lab 5/tm.gif
new file mode 100644
index 0000000000..335b6153c7
Binary files /dev/null and b/Lab 5/tm.gif differ
diff --git a/Lab 5/tm_ppe_detection.py b/Lab 5/tm_ppe_detection.py
new file mode 100644
index 0000000000..ab7667ba92
--- /dev/null
+++ b/Lab 5/tm_ppe_detection.py
@@ -0,0 +1,79 @@
+
+#This example is directly copied from the Tensorflow examples provided from the Teachable Machine.
+
+import tensorflow.keras
+from PIL import Image, ImageOps
+import numpy as np
+import cv2
+import sys
+
+
+# Disable scientific notation for clarity
+np.set_printoptions(suppress=True)
+
+img = None
+webCam = False
+if(len(sys.argv)>1 and not sys.argv[-1]== "noWindow"):
+ try:
+ print("I'll try to read your image");
+ img = cv2.imread(sys.argv[1])
+ if img is None:
+ print("Failed to load image file:", sys.argv[1])
+ except:
+ print("Failed to load the image are you sure that:", sys.argv[1],"is a path to an image?")
+else:
+ try:
+ print("Trying to open the Webcam.")
+ cap = cv2.VideoCapture(0)
+ if cap is None or not cap.isOpened():
+ raise("No camera")
+ webCam = True
+ except:
+ print("Unable to access webcam.")
+
+
+# Load the model
+model = tensorflow.keras.models.load_model('keras_model.h5')
+# Load Labels:
+labels=[]
+f = open("labels.txt", "r")
+for line in f.readlines():
+ if(len(line)<1):
+ continue
+ labels.append(line.split(' ')[1].strip())
+
+
+while(True):
+ if webCam:
+ ret, img = cap.read()
+
+ rows, cols, channels = img.shape
+ data = np.ndarray(shape=(1, 224, 224, 3), dtype=np.float32)
+
+ size = (224, 224)
+ img = cv2.resize(img, size, interpolation = cv2.INTER_AREA)
+ #turn the image into a numpy array
+ image_array = np.asarray(img)
+
+ # Normalize the image
+ normalized_image_array = (image_array.astype(np.float32) / 127.0) - 1
+ # Load the image into the array
+ data[0] = normalized_image_array
+
+ # run the inference
+ prediction = model.predict(data)
+ print("I think its a:",labels[np.argmax(prediction)])
+
+ if webCam:
+ if sys.argv[-1] == "noWindow":
+ cv2.imwrite('detected_out.jpg',img)
+ continue
+ cv2.imshow('detected (press q to quit)',img)
+ if cv2.waitKey(1) & 0xFF == ord('q'):
+ cap.release()
+ break
+ else:
+ break
+
+cv2.imwrite('detected_out.jpg',img)
+cv2.destroyAllWindows()
diff --git a/Lab 6/README.md b/Lab 6/README.md
index 33005a47df..77ecb771c9 100644
--- a/Lab 6/README.md
+++ b/Lab 6/README.md
@@ -1,96 +1,159 @@
-# m[Q](https://en.wikipedia.org/wiki/QAnon)tt[Anon](https://en.wikipedia.org/wiki/QAnon): Where We Go One, We Go All
-
-## Prep
-
-1. Pull the new changes
-2. Install [MQTT Explorer](http://mqtt-explorer.com/)
-3. Readings
- * [MQTT](#MQTT)
- * [The Presence Table](https://dl.acm.org/doi/10.1145/1935701.1935800) and [video](https://vimeo.com/15932020)
-
-
-## Introduction
-
-The point of this lab is to introduce you to distributed interaction. We've included a some Natural Language Processing (NLP) and Generation (NLG) but those are not really the emphasis. Feel free to dig into the examples and play around the code which you can integrate into your projects. However we want to emphasize the grading will focus on your ability to develop interesting uses for messaging across distributed devices.
-
-## MQTT
-
-MQTT is a lightweight messaging portal invented in 1999 for low bandwidth networks. It was later adopted as a defacto standard for a variety of Internet of Things (IoT) devices.
-
-### The Bits
-
-* **Broker** - The central server node that receives all messages and sends them out to the interested clients. Our broker is hosted on the far lab server (Thanks David!) at `farlab.infosci.cornell.edu/8883`
-* **Client** - A device that subscribes or publishes information on the network
-* **Topic** - The location data gets published to. These are hierarchical with subtopics. If you were making a network of IoT smart bulbs this might look like `home/livingroom/sidelamp/light_status` and `home/livingroom/sidelamp/voltage`. Subscribing to `home/livingroom/sidelamp/#` would give you message updates to both the light_status and the voltage. Because we use this broker for a variety of projects you have access to read, write and create subtopics of `IDD`. This means `IDD/ilan/is/a/goof` is a valid topic you can send data messages to.
-* **Subscribe** - This is a way of telling the client to pay attention to messages the broker sends out on that topic. You can subscribe to a specific topic or subtopics. You can also unsubscribe
-* **Publish** - This is a way of sending messages to a topic. You can publish to topics you don't subscribe to. Just remember on our broker you are limited to subtopics of `IDD`
-
-Setting up a broker isn't much work but for the purposes of this class you should all use the broker we've set up for you.
-
-### Useful Tooling
-
-Debugging and visualizing what's happening on your MQTT broker can be helpful. We like [MQTT Explorer](http://mqtt-explorer.com/). You can connect by putting in the settings from the image below.
-
-
-
-
-
-
-
-Once connected you should be able to see all the messaged on the IDD topic. From the interface you can send and plot messages as well.
-
-
-
-## Send and Receive
-
-[sender.py](./sender.py) and and [reader.py](./reader.py) show you the basics of using the mqtt in python. Lets spend a few minutes running these and seeing how messages are transferred and show up.
-
-**Running Examples**
-
-* Install the packages from `requirements.txt`, ideally in a python environment. We've been using the circuitpython environment we setup earlier this semester. To install them do `pip install -r requirements.txt`
-* to run `sender.py` type `python sender.py` and fill in a topic name, then start sending messages. You should see them on MQTT Explorer
-* to run `reader.py` type `python reader.py` and you should see any messages being published to `IDD/` subtopics.
-
-## Streaming a Sensor
-
-We've included an updated example from [lab 4](https://github.com/FAR-Lab/Interactive-Lab-Hub/tree/Spring2021/Lab%204) that streams sensor inputs over MQTT. Feel free to poke around with it!
-
-## The One True ColorNet
-
-It is with great fortitude and resilience that we shall worship at the altar of the *OneColor*. Through unity of the collective RGB we too can find unity in our heart, minds and souls. With the help of machines can overthrow the bourgeoisie, get on the same wavelength (this was also a color pun) and establish [Fully Automated Luxury Communism](https://en.wikipedia.org/wiki/Fully_Automated_Luxury_Communism).
-
-The first step on the path to *collective* enlightenment, plug the [APDS-9960 Proximity, Light, RGB, and Gesture Sensor](https://www.adafruit.com/product/3595) into the [Pi Display](https://www.adafruit.com/product/4393).
-
-
-
-You are almost there!
-
-The second step to achieving our great enlightenment is to run `python color.py`
-
-You will find the two squares on the display. Half is showing an approximation of the output from the color sensor. The other half is up to the collective. Press the top button to share your color with the class. Your color is now our color, our color is now your color. We are one.
-
-I was not super careful with handling the loop so you may need to press more than once if the timing isn't quite right. Also I have't load tested it so things might just immediately break when every pushes the button at once.
-
-You may ask "but what if I missed class?"
-
-Am I not admitted into the collective enlightenment of the *OneColor*?
-
-Of course not! You can got to [https://one-true-colornet.glitch.me/](https://one-true-colornet.glitch.me/) and become one with the ColorNet on the inter-webs.
-
-Glitch is a great tool for prototyping sites, interfaces and web-apps that's worth taking some time to get familiar with if you have a chance. Its not super pertinent for the class but good to know either way.
-
-
-
-## Make it your own
-
-Find at least one class (more are okay) partner, and design a distributed application together.
-
-**1. Explain your design** For example, if you made a remote controlled banana piano, explain why anyone would want such a thing.
-
-**2. Diagram the architecture of the system.** Be clear to document where input, output and computation occur, and label all parts and connections. For example, where is the banana, who is the banana player, where does the sound get played, and who is listening to the banana music?
-
-**3. Build a working prototype of the system.** Do think about the user interface: if someone encountered these bananas, would they know how to interact with them? Should they know what to expect?
-
-**4. Document the working prototype in use.** It may be helpful to record a Zoom session where you should the input in one location clearly causing response in another location.
-
-**5. BONUS (Wendy didn't approve this so you should probably ignore it)** get the whole class to run your code and make your distributed system BIGGER.
+# Little Interactions Everywhere
+
+## Prep
+
+1. Pull the new changes from the class interactive-lab-hub. (You should be familiar with this already!)
+2. Install [MQTT Explorer](http://mqtt-explorer.com/) on your laptop.
+3. Readings before class:
+ * [MQTT](#MQTT)
+ * [The Presence Table](https://dl.acm.org/doi/10.1145/1935701.1935800) and [video](https://vimeo.com/15932020)
+
+
+## Overview
+
+The point of this lab is to introduce you to distributed interaction. We have included some Natural Language Processing (NLP) and Generation (NLG) but those are not really the emphasis. Feel free to dig into the examples and play around the code which you can integrate into your projects if wanted. However, we want to emphasize that the grading will focus on your ability to develop interesting uses for messaging across distributed devices. Here are the four sections of the lab activity:
+
+A) [MQTT](#part-a)
+
+B) [Send and Receive on your Pi](#part-b)
+
+C) [Streaming a Sensor](#part-c)
+
+D) [The One True ColorNet](#part-d)
+
+E) [Make It Your Own](#part-)
+
+## Part 1.
+
+### Part A
+### MQTT
+
+MQTT is a lightweight messaging portal invented in 1999 for low bandwidth networks. It was later adopted as a defacto standard for a variety of [Internet of Things (IoT)](https://en.wikipedia.org/wiki/Internet_of_things) devices.
+
+#### The Bits
+
+* **Broker** - The central server node that receives all messages and sends them out to the interested clients. Our broker is hosted on the far lab server (Thanks David!) at `farlab.infosci.cornell.edu/8883`. Imagine that the Broker is the messaging center!
+* **Client** - A device that subscribes or publishes information to/on the network.
+* **Topic** - The location data gets published to. These are *hierarchical with subtopics*. For example, If you were making a network of IoT smart bulbs this might look like `home/livingroom/sidelamp/light_status` and `home/livingroom/sidelamp/voltage`. With this setup, the info/updates of the sidelamp's `light_status` and `voltage` will be store in the subtopics. Because we use this broker for a variety of projects you have access to read, write and create subtopics of `IDD`. This means `IDD/ilan/is/a/goof` is a valid topic you can send data messages to.
+* **Subscribe** - This is a way of telling the client to pay attention to messages the broker sends out on the topic. You can subscribe to a specific topic or subtopics. You can also unsubscribe. Following the previouse example of home IoT smart bulbs, subscribing to `home/livingroom/sidelamp/#` would give you message updates to both the light_status and the voltage.
+* **Publish** - This is a way of sending messages to a topic. Again, with the previouse example, you can set up your IoT smart bulbs to publish info/updates to the topic or subtopic. Also, note that you can publish to topics you do not subscribe to.
+
+
+**Important note:** With the broker we set up for the class, you are limited to subtopics of `IDD`. That is, to publish or subcribe, the topics will start with `IDD/`. Also, setting up a broker is not much work, but for the purposes of this class, you should all use the broker we have set up for you!
+
+
+#### Useful Tooling
+
+Debugging and visualizing what's happening on your MQTT broker can be helpful. We like [MQTT Explorer](http://mqtt-explorer.com/). You can connect by putting in the settings from the image below.
+
+
+
+
+
+Once connected, you should be able to see all the messages under the IDD topic. , go to the **Publish** tab and try publish something! From the interface you can send and plot messages as well. Remember, you are limited to subtopics of `IDD`. That is, to publish or subcribe, the topics will start with `IDD/`.
+
+
+
+
+### Part B
+### Send and Receive on your Pi
+
+[sender.py](./sender.py) and and [reader.py](./reader.py) show you the basics of using the mqtt in python. Let's spend a few minutes running these and seeing how messages are transferred and shown up. Before working on your Pi, keep the connection of `farlab.infosci.cornell.edu/8883` with MQTT Explorer running on your laptop.
+
+**Running Examples on Pi**
+
+* Install the packages from `requirements.txt` under a virtual environment, we will continue to use the `circuitpython` environment we setup earlier this semester:
+ ```
+ pi@ixe00:~ $ source circuitpython/bin/activate
+ (circuitpython) pi@ixe00:~ $ cd Interactive-Lab-Hub/Lab\ 6
+ (circuitpython) pi@ixe00:~ Interactive-Lab-Hub/Lab 6 $ pip install -r requirements.txt
+ ```
+* Run `sender.py`, fill in a topic name (should start with `IDD/`), then start sending messages. You should be able to see them on MQTT Explorer.
+ ```
+ (circuitpython) pi@ixe00:~ Interactive-Lab-Hub/Lab 6 $ python sender.py
+ pi@ReiIDDPi:~/Interactive-Lab-Hub/Lab 6 $ python sender.py
+ >> topic: IDD/ReiTesting
+ now writing to topic IDD/ReiTesting
+ type new-topic to swich topics
+ >> message: testtesttest
+ ...
+ ```
+* Run `reader.py`, and you should see any messages being published to `IDD/` subtopics.
+ ```
+ (circuitpython) pi@ixe00:~ Interactive-Lab-Hub/Lab 6 $ python reader.py
+ ...
+ ```
+
+**\*\*\*Consider how you might use this messaging system on interactive devices, and draw/write down 5 ideas here.\*\*\***
+
+### Part C
+### Streaming a Sensor
+
+We have included an updated example from [lab 4](https://github.com/FAR-Lab/Interactive-Lab-Hub/tree/Fall2021/Lab%204) that streams the [capacitor sensor](https://learn.adafruit.com/adafruit-mpr121-gator) inputs over MQTT. We will also be running this example under `circuitpython` virtual environment.
+
+Plug in the capacitive sensor board with the Qwiic connector. Use the alligator clips to connect a Twizzler (or any other things you used back in Lab 4) and run the example script:
+
+
+
+
+
+
+
+
+ ```
+ (circuitpython) pi@ixe00:~ Interactive-Lab-Hub/Lab 6 $ python distributed_twizzlers_sender.py
+ ...
+ ```
+
+**\*\*\*Include a picture of your setup here: what did you see on MQTT Explorer?\*\*\***
+
+**\*\*\*Pick another part in your kit and try to implement the data streaming with it.\*\*\***
+
+
+### Part D
+### The One True ColorNet
+
+It is with great fortitude and resilience that we shall worship at the altar of the *OneColor*. Through unity of the collective RGB, we too can find unity in our heart, minds and souls. With the help of machines, we can overthrow the bourgeoisie, get on the same wavelength (this was also a color pun) and establish [Fully Automated Luxury Communism](https://en.wikipedia.org/wiki/Fully_Automated_Luxury_Communism).
+
+The first step on the path to *collective* enlightenment, plug the [APDS-9960 Proximity, Light, RGB, and Gesture Sensor](https://www.adafruit.com/product/3595) into the [MiniPiTFT Display](https://www.adafruit.com/product/4393). You are almost there!
+
+
+
+
+
+
+
+
+The second step to achieving our great enlightenment is to run `color.py`. We have talked about this sensor back in Lab 2 and Lab 4, this script is similar to what you have done before! Remember to ativate the `circuitpython` virtual environment you have been using during this semester before running the script:
+
+ ```
+ (circuitpython) pi@ixe00:~ Interactive-Lab-Hub/Lab 6 $ python color.py
+ ...
+ ```
+
+By running the script, wou will find the two squares on the display. Half is showing an approximation of the output from the color sensor. The other half is up to the collective. Press the top button to share your color with the class. Your color is now our color, our color is now your color. We are one.
+
+(A message from the previous TA, Ilan: I was not super careful with handling the loop so you may need to press more than once if the timing isn't quite right. Also, I haven't load-tested it so things might just immediately break when everyone pushes the button at once.)
+
+You may ask "but what if I missed class?" Am I not admitted into the collective enlightenment of the *OneColor*?
+
+Of course not! You can go to [https://one-true-colornet.glitch.me/](https://one-true-colornet.glitch.me/) and become one with the ColorNet on the inter-webs. Glitch is a great tool for prototyping sites, interfaces and web-apps that's worth taking some time to get familiar with if you have a chance. Its not super pertinent for the class but good to know either way.
+
+**\*\*\*Can you set up the script that can read the color anyone else publish and display it on your screen?\*\*\***
+
+
+### Part E
+### Make it your own
+
+Find at least one class (more are okay) partner, and design a distributed application together based on the exercise we asked you to do in this lab.
+
+**\*\*\*1. Explain your design\*\*\*** For example, if you made a remote controlled banana piano, explain why anyone would want such a thing.
+
+**\*\*\*2. Diagram the architecture of the system.\*\*\*** Be clear to document where input, output and computation occur, and label all parts and connections. For example, where is the banana, who is the banana player, where does the sound get played, and who is listening to the banana music?
+
+**\*\*\*3. Build a working prototype of the system.\*\*\*** Do think about the user interface: if someone encountered these bananas somewhere in the wild, would they know how to interact with them? Should they know what to expect?
+
+**\*\*\*4. Document the working prototype in use.\*\*\*** It may be helpful to record a Zoom session where you should the input in one location clearly causing response in another location.
+
+
+
diff --git a/Lab 6/color.py b/Lab 6/color.py
index f56250058c..935ea9d7d4 100644
--- a/Lab 6/color.py
+++ b/Lab 6/color.py
@@ -1,110 +1,110 @@
-import board
-import busio
-import adafruit_apds9960.apds9960
-import time
-import paho.mqtt.client as mqtt
-import uuid
-import signal
-
-import digitalio
-from PIL import Image, ImageDraw, ImageFont
-import adafruit_rgb_display.st7789 as st7789
-
-
-# Configuration for CS and DC pins (these are FeatherWing defaults on M0/M4):
-cs_pin = digitalio.DigitalInOut(board.CE0)
-dc_pin = digitalio.DigitalInOut(board.D25)
-reset_pin = None
-
-# Config for display baudrate (default max is 24mhz):
-BAUDRATE = 64000000
-
-backlight = digitalio.DigitalInOut(board.D22)
-backlight.switch_to_output()
-backlight.value = True
-buttonA = digitalio.DigitalInOut(board.D23)
-buttonB = digitalio.DigitalInOut(board.D24)
-buttonA.switch_to_input()
-buttonB.switch_to_input()
-
-# Setup SPI bus using hardware SPI:
-spi = board.SPI()
-
-# Create the ST7789 display:
-disp = st7789.ST7789(
- spi,
- cs=cs_pin,
- dc=dc_pin,
- rst=reset_pin,
- baudrate=BAUDRATE,
- width=135,
- height=240,
- x_offset=53,
- y_offset=40,
-)
-
-height = disp.height
-width = disp.width
-image = Image.new("RGB", (width, height))
-draw = ImageDraw.Draw(image)
-
-
-i2c = busio.I2C(board.SCL, board.SDA)
-sensor = adafruit_apds9960.apds9960.APDS9960(i2c)
-
-sensor.enable_color = True
-r, g, b, a = sensor.color_data
-
-topic = 'IDD/colors'
-
-def on_connect(client, userdata, flags, rc):
- print(f"connected with result code {rc}")
- client.subscribe(topic)
-
-def on_message(cleint, userdata, msg):
- # if a message is recieved on the colors topic, parse it and set the color
- if msg.topic == topic:
- colors = list(map(int, msg.payload.decode('UTF-8').split(',')))
- draw.rectangle((0, 0, width, height*0.5), fill=color)
- disp.image(image)
-
-client = mqtt.Client(str(uuid.uuid1()))
-client.tls_set()
-client.username_pw_set('idd', 'device@theFarm')
-client.on_connect = on_connect
-client.on_message = on_message
-
-client.connect(
- 'farlab.infosci.cornell.edu',
- port=8883)
-
-client.loop_start()
-
-# this lets us exit gracefully (close the connection to the broker)
-def handler(signum, frame):
- print('exit gracefully')
- client.loop_stop()
- exit (0)
-
-# hen sigint happens, do the handler callback function
-signal.signal(signal.SIGINT, handler)
-
-# our main loop
-while True:
- r, g, b, a = sensor.color_data
-
- # there's a few things going on here
- # colors are reported at 16bits (thats 65536 levels per color).
- # we need to convert that to 0-255. thats what the 255*(x/65536) is doing
- # color are also reported with an alpha (opacity, or in our case a proxy for ambient brightness)
- # 255*(1-(a/65536)) acts as scaling factor for brightness, it worked well enough in the lab but
- # your success may vary depenging on how much ambient light there is, you can mess with these constants
- color =tuple(map(lambda x: int(255*(1-(a/65536))*255*(x/65536)) , [r,g,b,a]))
-
- # if we press the button, send msg to cahnge everyones color
- if not buttonA.value:
- client.publish(topic, f"{r},{g},{b}")
- draw.rectangle((0, height*0.5, width, height), fill=color[:3])
- disp.image(image)
- time.sleep(.01)
+import board
+import busio
+import adafruit_apds9960.apds9960
+import time
+import paho.mqtt.client as mqtt
+import uuid
+import signal
+
+import digitalio
+from PIL import Image, ImageDraw, ImageFont
+import adafruit_rgb_display.st7789 as st7789
+
+
+# Configuration for CS and DC pins (these are FeatherWing defaults on M0/M4):
+cs_pin = digitalio.DigitalInOut(board.CE0)
+dc_pin = digitalio.DigitalInOut(board.D25)
+reset_pin = None
+
+# Config for display baudrate (default max is 24mhz):
+BAUDRATE = 64000000
+
+backlight = digitalio.DigitalInOut(board.D22)
+backlight.switch_to_output()
+backlight.value = True
+buttonA = digitalio.DigitalInOut(board.D23)
+buttonB = digitalio.DigitalInOut(board.D24)
+buttonA.switch_to_input()
+buttonB.switch_to_input()
+
+# Setup SPI bus using hardware SPI:
+spi = board.SPI()
+
+# Create the ST7789 display:
+disp = st7789.ST7789(
+ spi,
+ cs=cs_pin,
+ dc=dc_pin,
+ rst=reset_pin,
+ baudrate=BAUDRATE,
+ width=135,
+ height=240,
+ x_offset=53,
+ y_offset=40,
+)
+
+height = disp.height
+width = disp.width
+image = Image.new("RGB", (width, height))
+draw = ImageDraw.Draw(image)
+
+
+i2c = busio.I2C(board.SCL, board.SDA)
+sensor = adafruit_apds9960.apds9960.APDS9960(i2c)
+
+sensor.enable_color = True
+r, g, b, a = sensor.color_data
+
+topic = 'IDD/colors'
+
+def on_connect(client, userdata, flags, rc):
+ print(f"connected with result code {rc}")
+ client.subscribe(topic)
+
+def on_message(cleint, userdata, msg):
+ # if a message is recieved on the colors topic, parse it and set the color
+ if msg.topic == topic:
+ colors = list(map(int, msg.payload.decode('UTF-8').split(',')))
+ draw.rectangle((0, 0, width, height*0.5), fill=color)
+ disp.image(image)
+
+client = mqtt.Client(str(uuid.uuid1()))
+client.tls_set()
+client.username_pw_set('idd', 'device@theFarm')
+client.on_connect = on_connect
+client.on_message = on_message
+
+client.connect(
+ 'farlab.infosci.cornell.edu',
+ port=8883)
+
+client.loop_start()
+
+# this lets us exit gracefully (close the connection to the broker)
+def handler(signum, frame):
+ print('exit gracefully')
+ client.loop_stop()
+ exit (0)
+
+# hen sigint happens, do the handler callback function
+signal.signal(signal.SIGINT, handler)
+
+# our main loop
+while True:
+ r, g, b, a = sensor.color_data
+
+ # there's a few things going on here
+ # colors are reported at 16bits (thats 65536 levels per color).
+ # we need to convert that to 0-255. thats what the 255*(x/65536) is doing
+ # color are also reported with an alpha (opacity, or in our case a proxy for ambient brightness)
+ # 255*(1-(a/65536)) acts as scaling factor for brightness, it worked well enough in the lab but
+ # your success may vary depenging on how much ambient light there is, you can mess with these constants
+ color =tuple(map(lambda x: int(255*(1-(a/65536))*255*(x/65536)) , [r,g,b,a]))
+
+ # if we press the button, send msg to cahnge everyones color
+ if not buttonA.value:
+ client.publish(topic, f"{r},{g},{b}")
+ draw.rectangle((0, height*0.5, width, height), fill=color[:3])
+ disp.image(image)
+ time.sleep(.01)
\ No newline at end of file
diff --git a/Lab 6/distributed_banana_sender.py b/Lab 6/distributed_twizzlers_sender.py
similarity index 78%
rename from Lab 6/distributed_banana_sender.py
rename to Lab 6/distributed_twizzlers_sender.py
index 6c075243f5..f92a120a73 100644
--- a/Lab 6/distributed_banana_sender.py
+++ b/Lab 6/distributed_twizzlers_sender.py
@@ -1,29 +1,29 @@
-import time
-import board
-import busio
-import adafruit_mpr121
-
-import paho.mqtt.client as mqtt
-import uuid
-
-client = mqtt.Client(str(uuid.uuid1()))
-client.tls_set()
-client.username_pw_set('idd', 'device@theFarm')
-
-client.connect(
- 'farlab.infosci.cornell.edu',
- port=8883)
-
-topic = 'IDD/your/topic/here'
-
-i2c = busio.I2C(board.SCL, board.SDA)
-
-mpr121 = adafruit_mpr121.MPR121(i2c)
-
-while True:
- for i in range(12):
- if mpr121[i].value:
- val = f"Banana {i} touched!"
- print(val)
- client.publish(topic, val)
- time.sleep(0.25)
\ No newline at end of file
+import time
+import board
+import busio
+import adafruit_mpr121
+
+import paho.mqtt.client as mqtt
+import uuid
+
+client = mqtt.Client(str(uuid.uuid1()))
+client.tls_set()
+client.username_pw_set('idd', 'device@theFarm')
+
+client.connect(
+ 'farlab.infosci.cornell.edu',
+ port=8883)
+
+topic = 'IDD/your/topic/here'
+
+i2c = busio.I2C(board.SCL, board.SDA)
+
+mpr121 = adafruit_mpr121.MPR121(i2c)
+
+while True:
+ for i in range(12):
+ if mpr121[i].value:
+ val = f"Twizzler {i} touched!"
+ print(val)
+ client.publish(topic, val)
+ time.sleep(0.25)
diff --git a/Lab 6/imgs/mqtt_explorer.png b/Lab 6/imgs/mqtt_explorer.png
index dc0a10c06a..c7232160b2 100644
Binary files a/Lab 6/imgs/mqtt_explorer.png and b/Lab 6/imgs/mqtt_explorer.png differ
diff --git a/Lab 6/imgs/mqtt_explorer_2.png b/Lab 6/imgs/mqtt_explorer_2.png
new file mode 100644
index 0000000000..90861090b3
Binary files /dev/null and b/Lab 6/imgs/mqtt_explorer_2.png differ
diff --git a/Lab 6/reader.py b/Lab 6/reader.py
index af56a17491..90f7b28e93 100644
--- a/Lab 6/reader.py
+++ b/Lab 6/reader.py
@@ -1,44 +1,44 @@
-import paho.mqtt.client as mqtt
-import uuid
-
-# the # wildcard means we subscribe to all subtopics of IDD
-topic = 'IDD/#'
-
-# some other examples
-# topic = 'IDD/a/fun/topic'
-
-#this is the callback that gets called once we connect to the broker.
-#we should add our subscribe functions here as well
-def on_connect(client, userdata, flags, rc):
- print(f"connected with result code {rc}")
- client.subscribe(topic)
- # you can subsribe to as many topics as you'd like
- # client.subscribe('some/other/topic')
-
-
-# this is the callback that gets called each time a message is recived
-def on_message(cleint, userdata, msg):
- print(f"topic: {msg.topic} msg: {msg.payload.decode('UTF-8')}")
- # you can filter by topics
- # if msg.topic == 'IDD/some/other/topic': do thing
-
-
-# Every client needs a random ID
-client = mqtt.Client(str(uuid.uuid1()))
-# configure network encryption etc
-client.tls_set()
-# this is the username and pw we have setup for the class
-client.username_pw_set('idd', 'device@theFarm')
-
-# attach out callbacks to the client
-client.on_connect = on_connect
-client.on_message = on_message
-
-#connect to the broker
-client.connect(
- 'farlab.infosci.cornell.edu',
- port=8883)
-
-# this is blocking. to see other ways of dealing with the loop
-# https://www.eclipse.org/paho/index.php?page=clients/python/docs/index.php#network-loop
+import paho.mqtt.client as mqtt
+import uuid
+
+# the # wildcard means we subscribe to all subtopics of IDD
+topic = 'IDD/#'
+
+# some other examples
+# topic = 'IDD/a/fun/topic'
+
+#this is the callback that gets called once we connect to the broker.
+#we should add our subscribe functions here as well
+def on_connect(client, userdata, flags, rc):
+ print(f"connected with result code {rc}")
+ client.subscribe(topic)
+ # you can subsribe to as many topics as you'd like
+ # client.subscribe('some/other/topic')
+
+
+# this is the callback that gets called each time a message is recived
+def on_message(cleint, userdata, msg):
+ print(f"topic: {msg.topic} msg: {msg.payload.decode('UTF-8')}")
+ # you can filter by topics
+ # if msg.topic == 'IDD/some/other/topic': do thing
+
+
+# Every client needs a random ID
+client = mqtt.Client(str(uuid.uuid1()))
+# configure network encryption etc
+client.tls_set()
+# this is the username and pw we have setup for the class
+client.username_pw_set('idd', 'device@theFarm')
+
+# attach out callbacks to the client
+client.on_connect = on_connect
+client.on_message = on_message
+
+#connect to the broker
+client.connect(
+ 'farlab.infosci.cornell.edu',
+ port=8883)
+
+# this is blocking. to see other ways of dealing with the loop
+# https://www.eclipse.org/paho/index.php?page=clients/python/docs/index.php#network-loop
client.loop_forever()
\ No newline at end of file
diff --git a/Lab 6/requirements.txt b/Lab 6/requirements.txt
index 34b356e086..0ba1dcc284 100644
--- a/Lab 6/requirements.txt
+++ b/Lab 6/requirements.txt
@@ -1,15 +1,15 @@
-Adafruit-Blinka==6.4.2
-adafruit-circuitpython-apds9960==2.2.6
-adafruit-circuitpython-busdevice==5.0.6
-adafruit-circuitpython-register==1.9.5
-adafruit-circuitpython-rgb-display==3.10.6
-Adafruit-PlatformDetect==3.5.0
-Adafruit-PureIO==1.1.8
-paho-mqtt==1.5.1
-Pillow==8.2.0
-pyftdi==0.52.9
-pyserial==3.5
-pyusb==1.1.1
-rpi-ws281x==4.2.6
-RPi.GPIO==0.7.0
-sysv-ipc==1.1.0
+Adafruit-Blinka==6.4.2
+adafruit-circuitpython-apds9960==2.2.6
+adafruit-circuitpython-busdevice==5.0.6
+adafruit-circuitpython-register==1.9.5
+adafruit-circuitpython-rgb-display==3.10.6
+Adafruit-PlatformDetect==3.5.0
+Adafruit-PureIO==1.1.8
+paho-mqtt==1.5.1
+Pillow==8.2.0
+pyftdi==0.52.9
+pyserial==3.5
+pyusb==1.1.1
+rpi-ws281x==4.2.6
+RPi.GPIO==0.7.0
+sysv-ipc==1.1.0
diff --git a/Lab 6/sender.py b/Lab 6/sender.py
index 8d346ffc7b..26d3c4e573 100644
--- a/Lab 6/sender.py
+++ b/Lab 6/sender.py
@@ -1,29 +1,29 @@
-import paho.mqtt.client as mqtt
-import uuid
-
-# Every client needs a random ID
-client = mqtt.Client(str(uuid.uuid1()))
-# configure network encryption etc
-client.tls_set()
-# this is the username and pw we have setup for the class
-client.username_pw_set('idd', 'device@theFarm')
-
-#connect to the broker
-client.connect(
- 'farlab.infosci.cornell.edu',
- port=8883)
-
-while True:
- cmd = input('>> topic: IDD/')
- if ' ' in cmd:
- print('sorry white space is a no go for topics')
- else:
- topic = f"IDD/{cmd}"
- print(f"now writing to topic {topic}")
- print("type new-topic to swich topics")
- while True:
- val = input(">> message: ")
- if val =='new-topic':
- break
- else:
+import paho.mqtt.client as mqtt
+import uuid
+
+# Every client needs a random ID
+client = mqtt.Client(str(uuid.uuid1()))
+# configure network encryption etc
+client.tls_set()
+# this is the username and pw we have setup for the class
+client.username_pw_set('idd', 'device@theFarm')
+
+#connect to the broker
+client.connect(
+ 'farlab.infosci.cornell.edu',
+ port=8883)
+
+while True:
+ cmd = input('>> topic: IDD/')
+ if ' ' in cmd:
+ print('sorry white space is a no go for topics')
+ else:
+ topic = f"IDD/{cmd}"
+ print(f"now writing to topic {topic}")
+ print("type new-topic to swich topics")
+ while True:
+ val = input(">> message: ")
+ if val =='new-topic':
+ break
+ else:
client.publish(topic, val)
\ No newline at end of file
diff --git a/README.md b/README.md
index 153961aec7..51a9120525 100644
--- a/README.md
+++ b/README.md
@@ -5,15 +5,15 @@ Please place links here to the README.md's for each of your labs here:
[Lab 1. Staging Interaction](Lab%201/)
-[Lab 2. The Clock of Pi](Lab%202/)
+[Lab 2. Interactive Prototyping: The Clock of Pi](Lab%202/)
-[Lab 3. You're a Wizard](Lab%203/)
+[Lab 3. Chatterboxes](Lab%203/)
[Lab 4. Ph-UI!!!](Lab%204/)
[Lab 5. Observant Systems](Lab%205/)
-[Lab 6. Where We Go One, We Go All](Lab%206/)
+[Lab 6. Little Interactions Everywhere](Lab%206/)
-[Final Project](Final%20Project/)
+[Final Project](https://github.com/FAR-Lab/Developing-and-Designing-Interactive-Devices/blob/2021Fall/FinalProject.md)
-
-
-Now run `python proximity.py`.
-
## Under the I2C curtain (optional: complete only after working on your projects in groups)
@@ -79,7 +69,6 @@ Run the file `I2C_scan.py` and the output should look like:
(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 2 $ python I2C_scan.py
I2C ok!
I2C addresses found: []
-
```
Now plug the other end of the cable into the ports on the right of the button board. The pwr LED should turn on. Run the file again and you should see the device ID. You can also try daisy chaining multiple devices and sensors and running again.
@@ -100,3 +89,24 @@ Use a higher level device interface can make reading and writing registers for I
```
56 write_register(device, STATUS, 0)
```
+
+#### Connecting more that one button
+
+The more buttons the merrier! ...but how do you control them individually when they are come with the same default I2C address `0x6f`?
+
+Option 1 - Software: Look into the [list of registers](https://cdn.sparkfun.com/assets/learn_tutorials/1/1/0/8/Qwiic_Button_I2C_Register_Map.pdf) of the buttons again, is it possible to change the I2C address through software progrmming?
+
+Option 2 - Hardware: Look at the bottom right corner of the back of the button, you should be able to local a sign of ADR with A0 - A3 jumpers. By solding these I2C address jumpers, you can actually change the address directly! Check [here](https://learn.sparkfun.com/tutorials/sparkfun-qwiic-button-hookup-guide/all) to see how the I2C address change!
+
+
+
+## Connecting a Sensor
+
+Your kit is full of sensors! Look up what they can do and feel free to ask your TAs, we love to talk sensors. We will go further in depth into sensors in the coming weeks, but we put this small sample here to demonstrate how you can get sensor data if you want to use it for your project this week.
+
+We are going to connect the [Adafruit APDS9960 Proximity, Light, RGB, and Gesture Sensor](https://www.adafruit.com/product/3595). You can leave the button plugged in and daisy-chain the sensor, this is part of the magic of I2C.
+
+
-
-In this lab, we want you to practice wizarding an interactive device as discussed in class. We will focus on audio as the main modality for interaction but there is no reason these general techniques can't extend to video, haptics or other interactive mechanisms. In fact, you are welcome to add those to your project if they enhance your design.
-
-
-## Text to Speech and Speech to Text
-
-In the home directory of your Pi there is a folder called `text2speech` containing some shell scripts.
-
-```
-pi@ixe00:~/text2speech $ ls
-Download festival_demo.sh GoogleTTS_demo.sh pico2text_demo.sh
-espeak_demo.sh flite_demo.sh lookdave.wav
-
-```
-
-you can run these examples by typing
-`./espeakdeom.sh`. Take some time to look at each script and see how it works. You can see a script by typing `cat filename`
-
-```
-pi@ixe00:~/text2speech $ cat festival_demo.sh
-#from: https://elinux.org/RPi_Text_to_Speech_(Speech_Synthesis)#Festival_Text_to_Speech
-
-echo "Just what do you think you're doing, Dave?" | festival --tts
-
-```
-
-You can also play audio files directly with `aplay filename`.
-
-After looking through this folder do the same for the `speech2text` folder. In particular, look at `test_words.py` and make sure you understand how the vocab is defined. Then try `./vosk_demo_mic.sh`
-
-## Serving Pages
-
-In Lab 1 we served a webpage with flask. In this lab you may find it useful to serve a webpage for the controller on a remote device. Here is a simple example of a webserver.
-
-```
-pi@ixe00:~/$ python server.py
- * Serving Flask app "server" (lazy loading)
- * Environment: production
- WARNING: This is a development server. Do not use it in a production deployment.
- Use a production WSGI server instead.
- * Debug mode: on
- * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
- * Restarting with stat
- * Debugger is active!
- * Debugger PIN: 162-573-883
-```
-From a remote browser on the same network, check to make sure your webserver is working by going to [http://ixe00.local:5000]()
-
-
-## Demo
-
-In the [demo directory](./demo), you will find an example wizard of oz project you may use as a template. **You do not have to** feel free to get creative. In that project, you can see how audio and sensor data is streamed from the Pi to a wizard controller that runs in the browser. You can control what system says from the controller as well.
-
-## Optional
-
-There is an included [dspeech](./dspeech) demo that uses [Mozilla DeepSpeech](https://github.com/mozilla/DeepSpeech) for speech to text. If you're interested in trying it out we suggest you create a seperarate virutalenv.
-
-
-
-# Lab 3 Part 2
-
-Create a system that runs on the Raspberry Pi that takes in one or more sensors and requires participants to speak to it. Document how the system works and include videos of both the system and the controller.
-
-## Prep for Part 2
-
-1. Sketch ideas for what you'll work on in lab on Wednesday.
-
-## Share your idea sketches with Zoom Room mates and get feedback
-
-*what was the feedback? Who did it come from?*
-
-## Prototype your system
-
-The system should:
-* use the Raspberry Pi
-* use one or more sensors
-* require participants to speak to it.
-
-*Document how the system works*
-
-*Include videos or screencaptures of both the system and the controller.*
-
-## Test the system
-Try to get at least two people to interact with your system. (Ideally, you would inform them that there is a wizard _after_ the interaction, but we recognize that can be hard.)
-
-Answer the following:
-
-### What worked well about the system and what didn't?
-*your answer here*
-
-### What worked well about the controller and what didn't?
-
-*your answer here*
-
-### What lessons can you take away from the WoZ interactions for designing a more autonomous version of the system?
-
-*your answer here*
-
-
-### How could you use your system to create a dataset of interaction? What other sensing modalities would make sense to capture?
-
-*your answer here*
-
+# Chatterboxes
+[](https://www.youtube.com/embed/Q8FWzLMobx0?start=19)
+
+In this lab, we want you to design interaction with a speech-enabled device--something that listens and talks to you. This device can do anything *but* control lights (since we already did that in Lab 1). First, we want you first to storyboard what you imagine the conversational interaction to be like. Then, you will use wizarding techniques to elicit examples of what people might say, ask, or respond. We then want you to use the examples collected from at least two other people to inform the redesign of the device.
+
+We will focus on **audio** as the main modality for interaction to start; these general techniques can be extended to **video**, **haptics** or other interactive mechanisms in the second part of the Lab.
+
+## Prep for Part 1: Get the Latest Content and Pick up Additional Parts
+
+### Pick up Additional Parts
+
+As mentioned during the class, we ordered additional mini microphone for Lab 3. Also, a new part that has finally arrived is encoder! Please remember to pick them up from the TA.
+
+### Get the Latest Content
+
+As always, pull updates from the class Interactive-Lab-Hub to both your Pi and your own GitHub repo. As we discussed in the class, there are 2 ways you can do so:
+
+**\[recommended\]**Option 1: On the Pi, `cd` to your `Interactive-Lab-Hub`, pull the updates from upstream (class lab-hub) and push the updates back to your own GitHub repo. You will need the *personal access token* for this.
+
+```
+pi@ixe00:~$ cd Interactive-Lab-Hub
+pi@ixe00:~/Interactive-Lab-Hub $ git pull upstream Fall2021
+pi@ixe00:~/Interactive-Lab-Hub $ git add .
+pi@ixe00:~/Interactive-Lab-Hub $ git commit -m "get lab3 updates"
+pi@ixe00:~/Interactive-Lab-Hub $ git push
+```
+
+Option 2: On your your own GitHub repo, [create pull request](https://github.com/FAR-Lab/Developing-and-Designing-Interactive-Devices/blob/2021Fall/readings/Submitting%20Labs.md) to get updates from the class Interactive-Lab-Hub. After you have latest updates online, go on your Pi, `cd` to your `Interactive-Lab-Hub` and use `git pull` to get updates from your own GitHub repo.
+
+## Part 1.
+### Text to Speech
+
+In this part of lab, we are going to start peeking into the world of audio on your Pi!
+
+We will be using a USB microphone, and the speaker on your webcamera. (Originally we intended to use the microphone on the web camera, but it does not seem to work on Linux.) In the home directory of your Pi, there is a folder called `text2speech` containing several shell scripts. `cd` to the folder and list out all the files by `ls`:
+
+```
+pi@ixe00:~/text2speech $ ls
+Download festival_demo.sh GoogleTTS_demo.sh pico2text_demo.sh
+espeak_demo.sh flite_demo.sh lookdave.wav
+```
+
+You can run these shell files by typing `./filename`, for example, typing `./espeak_demo.sh` and see what happens. Take some time to look at each script and see how it works. You can see a script by typing `cat filename`. For instance:
+
+```
+pi@ixe00:~/text2speech $ cat festival_demo.sh
+#from: https://elinux.org/RPi_Text_to_Speech_(Speech_Synthesis)#Festival_Text_to_Speech
+
+echo "Just what do you think you're doing, Dave?" | festival --tts
+```
+
+Now, you might wonder what exactly is a `.sh` file? Typically, a `.sh` file is a shell script which you can execute in a terminal. The example files we offer here are for you to figure out the ways to play with audio on your Pi!
+
+You can also play audio files directly with `aplay filename`. Try typing `aplay lookdave.wav`.
+
+\*\***Write your own shell file to use your favorite of these TTS engines to have your Pi greet you by name.**\*\*
+(This shell file should be saved to your own repo for this lab.)
+
+Bonus: If this topic is very exciting to you, you can try out this new TTS system we recently learned about: https://github.com/rhasspy/larynx
+
+### Speech to Text
+
+Now examine the `speech2text` folder. We are using a speech recognition engine, [Vosk](https://alphacephei.com/vosk/), which is made by researchers at Carnegie Mellon University. Vosk is amazing because it is an offline speech recognition engine; that is, all the processing for the speech recognition is happening onboard the Raspberry Pi.
+
+In particular, look at `test_words.py` and make sure you understand how the vocab is defined. Then try `./vosk_demo_mic.sh`
+
+One thing you might need to pay attention to is the audio input setting of Pi. Since you are plugging the USB cable of your webcam to your Pi at the same time to act as speaker, the default input might be set to the webcam microphone, which will not be working for recording.
+
+\*\***Write your own shell file that verbally asks for a numerical based input (such as a phone number, zipcode, number of pets, etc) and records the answer the respondent provides.**\*\*
+
+Bonus Activity:
+
+If you are really excited about Speech to Text, you can try out [Mozilla DeepSpeech](https://github.com/mozilla/DeepSpeech) and [voice2json](http://voice2json.org/install.html)
+There is an included [dspeech](./dspeech) demo on the Pi. If you're interested in trying it out, we suggest you create a seperarate virutal environment for it . Create a new Python virtual environment by typing the following commands.
+
+```
+pi@ixe00:~ $ virtualenv dspeechexercise
+pi@ixe00:~ $ source dspeechexercise/bin/activate
+(dspeechexercise) pi@ixe00:~ $
+```
+
+### Serving Pages
+
+In Lab 1, we served a webpage with flask. In this lab, you may find it useful to serve a webpage for the controller on a remote device. Here is a simple example of a webserver.
+
+```
+pi@ixe00:~/Interactive-Lab-Hub/Lab 3 $ python server.py
+ * Serving Flask app "server" (lazy loading)
+ * Environment: production
+ WARNING: This is a development server. Do not use it in a production deployment.
+ Use a production WSGI server instead.
+ * Debug mode: on
+ * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
+ * Restarting with stat
+ * Debugger is active!
+ * Debugger PIN: 162-573-883
+```
+From a remote browser on the same network, check to make sure your webserver is working by going to `http://
-
-## Hardware Set-Up
-
-For this demo, you will need:
-* your Raspberry Pi,
-* a Qwiic/Stemma Cable,
-* a speaker/aux cable or wired headphones,
-* the display (we are just using it for the Qwiic/StemmaQT port. Feel free to use the display in your projects),
-* your accelerometer, and
-* your USB microphone
-
-
-
-
-
-
+These Twizzlers are connected to pads 6 and 10. When you run the code and touch a Twizzler, the terminal will print out the following
```
(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python cap_test.py
-Banana 10 touched!
-Banana 6 touched!
+Twizzler 10 touched!
+Twizzler 6 touched!
```
-*\*Some students have noted that their banana's look noticeably different from the ones presented in this demo. We firmly reject the accusation that these are not in fact banana's but Twizzlers™. Due to the challenges of remote teaching we cannot debug banana's at this time. We suggest you bring these issues up with the university or your local produce representative*
-
### Part B
-### OLED screen
+### More sensors
+
+#### Light/Proximity/Gesture sensor (APDS-9960)
+
+We here want you to get to know this awesome sensor [Adafruit APDS-9960](https://www.adafruit.com/product/3595). It is capable of sensing proximity, light (also RGB), and gesture!
+
+
+
+Connect it to your pi with Qwiic connector and try running the three example scripts individually to see what the sensor is capable of doing!
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python proximity_test.py
+...
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python gesture_test.py
+...
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python color_test.py
+...
+```
+
+You can go the the [Adafruit GitHub Page](https://github.com/adafruit/Adafruit_CircuitPython_APDS9960) to see more examples for this sensor!
-We just received some of the small oled screens that we had coped to include in your kit. If you want one feel free to pop into the lab and get one. These don't have colors like the one on the pi but you can move it around on a cable making for more flexible interface design. The way you program this display is almost identical to the pi display. Take a look at `oled_test.py` and some more of the [Adafruit examples](https://github.com/adafruit/Adafruit_CircuitPython_SSD1306/tree/master/examples).
+#### Rotary Encoder
+
+A rotary encoder is an electro-mechanical device that converts the angular position to analog or digital output signals. The [Adafruit rotary encoder](https://www.adafruit.com/product/4991#technical-details) we ordered for you came with separated breakout board and encoder itself, that is, they will need to be soldered if you have not yet done so! We will be bringing the soldering station to the lab class for you to use, also, you can go to the MakerLAB to do the soldering off-class. Here is some [guidance on soldering](https://learn.adafruit.com/adafruit-guide-excellent-soldering/preparation) from Adafruit. When you first solder, get someone who has done it before (ideally in the MakerLAB environment). It is a good idea to review this material beforehand so you know what to look at.
-
+
+
+
+
+
-This is fine, but it can be a bit difficult to lay out a great and user friendly display within the constraints of the Pi. Also, it really only works for applications where people can come and stand over the Pi, or where you can mount the Pi to the wall.
+This is fine, but the mounting of the display constrains the display location and orientation a lot. Also, it really only works for applications where people can come and stand over the Pi, or where you can mount the Pi to the wall.
Here is another prototype for a paper display:
-It holds a pi and usb power supply, and provides a front stage on which to put writing, graphics, LEDs, buttons or displays.
+Your kit includes these [SparkFun Qwiic OLED screens](https://www.sparkfun.com/products/17153). These use less power than the MiniTFTs you have mounted on the GPIO pins of the Pi, but, more importantly, they can be more flexibily be mounted elsewhere on your physical interface. The way you program this display is almost identical to the way you program a Pi display. Take a look at `oled_test.py` and some more of the [Adafruit examples](https://github.com/adafruit/Adafruit_CircuitPython_SSD1306/tree/master/examples).
+
+
-
-Make a paper display for your project that communicates the state of the Pi and a sensor. Ideally you should design it so that you can slide the Pi out to work on the circuit or programming, and then slide it back in and reattach a few wires to be back in operation.
+Think about how you want to present the information about what your sensor is sensing! Design a paper display for your project that communicates the state of the Pi and a sensor. Ideally you should design it so that you can slide the Pi out to work on the circuit or programming, and then slide it back in and reattach a few wires to be back in operation.
-**a. Document the design for your paper display.** (e.g. if you had to make it again from scratch, what information would you need?). Include interim iterations (or at least tell us about them).
+**\*\*\*Sketch 5 designs for how you would physically position your display and any buttons or knobs needed to interact with it.\*\*\***
-**b. Make a video of your paper display in action.**
+**\*\*\*What are some things these sketches raise as questions? What do you need to physically prototype to understand how to anwer those questions?\*\*\***
-**c. Explain the rationale for the design.** (e.g. Does it need to be a certain size or form or need to be able to be seen from a certain distance?)
+**\*\*\*Pick one of these display designs to integrate into your prototype.\*\*\***
-### Part D
-### Materiality
+**\*\*\*Explain the rationale for the design.\*\*\*** (e.g. Does it need to be a certain size or form or need to be able to be seen from a certain distance?)
+
+Build a cardbord prototype of your design.
+
+**\*\*\*Document your rough prototype.\*\*\***
+
+
+LAB PART 2
+
+### Part 2
+
+Following exploration and reflection from Part 1, complete the "looks like," "works like" and "acts like" prototypes for your design, reiterated below.
-**Open Ended**: We are putting very few constraints on this part but we want you to get creative.
+### Part E (Optional)
+### Servo Control with Joystick
-Design a system with the Pi and anything from your kit with a focus on form, and materiality. The "stuff" that enclose the system should be informed by the desired interaction. What would a computer made of rocks be like? How would an ipod made of grass behave? Would a roomba made of gold clean your floor any differently?
+In the class kit, you should be able to find the [Qwiic Servo Controller](https://www.sparkfun.com/products/16773) and [Micro Servo Motor SG51](https://www.adafruit.com/product/2201). The Qwiic Servo Controller will need external power supply to drive, which we will be distributing the battery packs in the class. Connect the servo controller to the miniPiTFT through qwiic connector and connect the external battery to the 2-Pin JST port (ower port) on the servo controller. Connect your servo to channel 2 on the controller, make sure the brown is connected to GND and orange is connected to PWM.
+
+
+
+In this exercise, we will be using the nice [ServoKit library](https://learn.adafruit.com/16-channel-pwm-servo-driver/python-circuitpython) developed by Adafruit! We will continue to use the `circuitpython` virtual environment we created. Activate the virtual environment and make sure to install the latest required libraries by running:
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ pip3 install -r requirements.txt
+```
+
+A servo motor is a rotary actuator or linear actuator that allows for precise control of angular or linear position. The position of a servo motor is set by the width of an electrical pulse, that is, we can use PWM (pulse-width modulation) to set and control the servo motor position. You can read [this](https://learn.adafruit.com/adafruit-arduino-lesson-14-servo-motors/servo-motors) to learn a bit more about how exactly a servo motor works.
+
+Now that you have a basic idea of what a servo motor is, look into the script `qwiic_servo_example.py` we provide. In line 14, you should see that we have set up the min_pulse and max_pulse corresponding to the servo turning 0 - 180 degree. Try running the servo example code now and see what happens:
+
+```
+(circuitpython) pi@ixe00:~/Interactive-Lab-Hub/Lab 4 $ python servo_test.py
+```
-**a. document the material prototype.** Include candidates that were considered even if they were set aside later.
+It is also possible to control the servo using the sensors mentioned in as in part A and part B, and/or from some of the buttons or parts included in your kit, the simplest way might be to chain Qwiic buttons to the other end of the Qwiic OLED. Like this:
-**b. explain the selection.**
+
+