Resolves#1584
We've been using Docker+QEMU to emulate 32-bit ARM on AMD64, but this PR
switches to ARM-native CircleCI instances, which skips the emulation.
This achieves a speedup on building Debian packages from about 4m down
to about 70s.
Because Debian package building is fast again, we're changing our CI
workflow to build a Debian package on every commit rather than just on
`master` or just when we manually trigger it.
<a data-ca-tag
href="https://codeapprove.com/pr/tiny-pilot/tinypilot/1588"><img
src="https://codeapprove.com/external/github-tag-allbg.png" alt="Review
on CodeApprove" /></a>
---------
Co-authored-by: Michael Lynch <git@mtlynch.io>
Resolves https://github.com/tiny-pilot/tinypilot/issues/1501Resolves#1267
This change gets rid of the `ustreamer_capture_device` Ansible var
(TC358743 vs. HDMI to USB) and the config file where we stored the
result. Instead, we'll track capture device based on whether
`dtoverlay=tc358743` appears in `/boot/config.txt`.
In theory, relying on `/boot/config.txt` will cause incorrect behavior
if a user enables the TC358743 overlay even though they're using HDMI to
USB. That's unlikely to happen in practice, so I think it's worth the
tradeoff, as this assumption eliminates a **lot** of complexity.
As a side-effect of eliminating `ustreamer_capture_device`, we can
remove a lot of cruft from the bundle install file.
### Background
We originally added the "uStreamer settings file" because we needed a
way to keep track of whether the device was using a HDMI to USB dongle
(like a DIY TinyPilot) or a TC358743-based HDMI to CSI chip (like a
Voyager).
At the time of the original implementation, the uStreamer Ansible role
was independent of TinyPilot, so it couldn't write to TinyPilot
settings. So, we created a "uStreamer settings file" at
`/home/ustreamer.config.yml` that would essentially persist the setting
for whether the system was TC358743-based so that the initiator of the
Ansible role wouldn't have to re-specify it every time.
### Implementation notes
* I'm broadening the scope a little bit by tackling #1267 at the same
time, but I feel like the major cost of both those issues is the testing
to make sure everything is correct, so I might as well fix and test both
at the same time.
* With this change, the Ansible role no longer adds the
`dtoverlay=tc358743` line to `/boot/config.txt` because it's now the
user's responsibility to do that in advance if they've got a DIY
TinyPilot with an HDMI to CSI chip. For Pro, Packer will take
responsibility for adding that line.
* The Ansible role still adds the `dtoverlay=tc358743-audio` line.
* `bundler/bundle/install` used to be responsible for creating
`/home/tinypilot/settings.yml` if it didn't already exist, but this
change moves that responsibility to `tinypilot.postinst`.
* I'm deliberately leaving a `debug` task in the Ansible role because I
think it's helpful to have the output in logs.
* This change drops the `ustreamer_persistent` var and instead always
launches ustreamer with the `--persistent` flag. The setting used to be
unofficially configurable via settings, but we're dropping that
configurability (see
https://github.com/tiny-pilot/tinypilot-pro/issues/972).
### Documentation impact
After we merge this, we need to update our
[wiki](https://github.com/tiny-pilot/tinypilot/wiki/Installation-Options#example-tc358743-hdmi-to-csi-capture-chip)
with the new TC358743 install method, which should simplify down to:
```bash
if ! grep --silent '^dtoverlay=tc358743$' /boot/config.txt; then
echo 'dtoverlay=tc358743' | sudo tee --append /boot/config.txt
fi
```
Corresponding changes in Pro:
https://github.com/tiny-pilot/tinypilot-pro/compare/no-capture-pro?expand=1
(difference is just Packer config)
### Manual tests
I performed the following manual tests to verify we're not breaking
functionality.
- [x] Verify TinyPilot Community install with HDMI to USB dongle
- [x] Verify TinyPilot Community install with HDMI to CSI bridge
- [x] Verify TinyPilot Pro Hobbyist image
- [x] Verify TinyPilot Pro Voyager image
- [x] Verify installing TinyPilot Pro bundle on top of TinyPilot Pro
2.6.0 Voyager image
- [x] Verify installing TinyPilot Pro bundle on top of TinyPilot Pro
2.6.0 Hobbyist image
<a data-ca-tag
href="https://codeapprove.com/pr/tiny-pilot/tinypilot/1554"><img
src="https://codeapprove.com/external/github-tag-allbg.png" alt="Review
on CodeApprove" /></a>
Resolves https://github.com/tiny-pilot/tinypilot/issues/1548
This PR moves the comments that explains how to trigger a code-based
bundle build from `.circleci/config.yml` to
`.circleci/continue_config.yml` and updates the contributing docs to
reference the correct CircleCI config file.
So the correct ways to [manually trigger a bundle
build](1ea663b0ee/CONTRIBUTING.md (installing-a-nightly-bundle))
is either the:
1. Code-based way
> 1. In [the CircleCI configuration](/.circleci/continue_config.yml),
change the `bundle_build_branch` parameter’s `default` value from
`master` to `<< pipeline.git.branch >>`.
> 1. Push your changes as branch to GitHub. CircleCI will now
automatically build bundles for all subsequent commits of that branch.
> 1. Before eventually merging your feature branch, remember to revert
the `bundle_build_branch` parameter to the original value (i.e.,
`master`)
3. CircleCI web UI way
> 1. On CircleCI’s overview page for the respective branch, click the
“Trigger Pipeline” button.
> 1. In the dialog, enter `bundle_build_branch` as parameter name, and
the (exact) name of the branch as parameter value.
> 1. Click the “Trigger Pipeline” button in the dialog. CircleCI will
now build a bundle for the latest commit of that branch.
<a data-ca-tag
href="https://codeapprove.com/pr/tiny-pilot/tinypilot/1561"><img
src="https://codeapprove.com/external/github-tag-allbg.png" alt="Review
on CodeApprove" /></a>
Currently, when we want to build a bundle off a PR, we need to
temporarily push a [code change in the Circle
configuration](b20762c924/.circleci/continue_config.yml (L10-L13)),
similar to
dde6828649.
Once we are done with testing, we need to revert that Circle config
change, before merging the PR eventually.
I just discovered that there is a way to trigger the complete build
pipeline on the fly, **without** having to push any code changes:
1. On [CircleCI’s overview page for the respective
branch](https://app.circleci.com/pipelines/github/tiny-pilot/tinypilot?branch=circle-manual-trigger),
click the “Trigger Pipeline“ button, which opens a dialog
2. As parameter, enter `bundle_build_branch` as parameter name, and the
current branch name as value, in this case `circle-manual-trigger`
3. Click the “Trigger Pipeline” button
<img width="1234" alt="Screenshot 2023-07-20 at 19 30 57"
src="https://github.com/tiny-pilot/tinypilot/assets/83721279/7b108baa-7442-430a-97ec-b4b1f6afac75">
The prerequisite for this to work is that we additionally declare the
`bundle_build_branch` parameter in the top-level `config.yml` file, and
not just in the `continue_config.yml` file. Otherwise, CircleCI wouldn’t
recognize the manual trigger parameter, and refuse to run the pipeline.
I think this would be a useful option for us, especially in order to run
one-off builds on the fly. The “old” method would continue to work, so
it’s only an additional method.
<a data-ca-tag
href="https://codeapprove.com/pr/tiny-pilot/tinypilot/1518"><img
src="https://codeapprove.com/external/github-tag-allbg.png" alt="Review
on CodeApprove" /></a>
Bundles are no longer built automatically off feature branches, so [we
have to manually trigger this via a one-line change in the CircleCI
config](https://github.com/tiny-pilot/tinypilot/pull/1442). This PR
updates the documentation and describes the procedure.
Along the way, I also changed the list numbering style from `1. 2. 3.`
to `1. 1. 1.`. I think that’s how we generally seem to be doing it in
Markdown files, and it spares us from having to update all consecutive
numbers when prepending a new bullet.
<a data-ca-tag
href="https://codeapprove.com/pr/tiny-pilot/tinypilot/1472"><img
src="https://codeapprove.com/external/github-tag-allbg.png" alt="Review
on CodeApprove" /></a>
Resolves https://github.com/tiny-pilot/tinypilot/issues/1073.
In our discussion in the [previous draft
PR](https://github.com/tiny-pilot/tinypilot/pull/1407) around dev
scripts and workflow documentation for remote testing, we realized that
it might be too tricky for us to provide easy-to-use dev scripts that
work in all environments and setups out of the box.
We could still add such dev scripts at a later point. For now, this PR
updates and extends our documentation. I think that’s also a good way to
share tips and workflows (although it’s prose), and to align ones own
workflows with best practices that others are using. I tried to cover
all topics that I could think of, and that appeared useful or important
to me. The structure is more or less loose, but I thought it makes sense
to group things into sections.
I also removed the section on SSH agent forwarding. This should be
obsolete since we abandoned the git-based update workflow.
If there is anything else that we can add, please let me know.
<a data-ca-tag
href="https://codeapprove.com/pr/tiny-pilot/tinypilot/1414"><img
src="https://codeapprove.com/external/github-tag-allbg.png" alt="Review
on CodeApprove" /></a>
This change integrates the
[dh-virtualenv](https://github.com/spotify/dh-virtualenv) extension to
debhelper to install TinyPilot's Python virtual environment as part of
the Debian package install.
This seemed like it would be easy, but it turned out to be fairly
complicated. Putting the virtualenv into the Debian package makes the
Debian package architecture-specific, so we have to spin up QEMU and
manage architecture-specific bits throughout the build process.
## What this PR does
### Reduces disk writes on customer devices
The advantage of this change is that on customer devices, this reduces
disk writes. Instead of installing/compiling Python packages on the
device, most of that work is done ahead of time, so we're putting less
wear on the customer's microSD.
### Speeds up installs/updates
Similarly to the reduction in disk writes, it should speed up installs.
We don't have to go through the install process for the virtualenv on
the end-user's device because the packages are pre-provisioned in the
Debian package.
### Reduces Ansible code
This change continues our "war on Ansible," reimplementing Ansible logic
as Debian package logic.
### Makes TinyPilot's Debian package architecture-dependent
Previously, the TinyPilot Debian package could declare an architecture
of `all` because it didn't contain any native binaries. Using
`dh-virtualenv` means that TinyPilot's third-party libraries are now
packaged inside the Debian package, and some of the dependencies are
architecture-dependent, which means the package as a whole is
architecture dependent.
### Only builds AMD64 in PRs
Because we have to emulate ARMv7 from CI, this change slows down Debian
package building significantly. It's about a 20x slowdown, increasing
build time for the Debian package from ~30s to ~6m.
I considered building in ARM on CircleCI, but we need 32-bit ARM, and
CircleCI only supports 64-bit.
>We currently do not provide support for 32-bit Arm architectures. Only
64-bit arm64 architectures are supported at this time.
https://circleci.com/docs/using-arm/#limitations
With that in mind, we now only build AMD64 in PRs and ARMv7+AMD64 in
master/releases.
## Notes
### Lintian Override Syntax
I find the documentation for [Lintian
overrides](https://lintian.debian.org/manual/index.html#overrides)
really confusing. I got the overrides by copying errors from Lintian and
pasting them into the overrides. What I put in doesn't seem to match
Lintian's documented syntax, but when I try to adjust the overrides to
match my understanding of the documentation, the overrides fail.
### Debian package hardening
In an effort to resolve a [`hardening-no-relro` lintian
issue](https://app.circleci.com/pipelines/github/tiny-pilot/tinypilot/3386/workflows/aaa8762f-0791-4805-b2aa-14663ece04dd/jobs/18387?invite=true#step-106-19),
I enabled [Debian package
"hardening"](https://wiki.debian.org/Hardening#Using_Hardening_Options):
79fe3e6a37/debian-pkg/debian/rules (L3-L6)
But to be honest, I don't know what it really does for our package. It
didn't resolve the lintian issue, but it seems to be something to do
with package security, so I kept it in.
### `install-from-source` script
To get the `install-from-source` script working again, I mostly just
needed to [clean out previous ansible
roles](4ebe94a86b/bundler/create-bundle (L75-L77))
before creating the bundle.
<a data-ca-tag
href="https://codeapprove.com/pr/tiny-pilot/tinypilot/1352"><img
src="https://codeapprove.com/external/github-tag-allbg.png" alt="Review
on CodeApprove" /></a>
---------
Co-authored-by: Jan Heuermann <github-jotaen4tinypilot@reg.jotaen.net>
Co-authored-by: Jason Wallace <jdeanwallace@gmail.com>
I've found during development, it's sometimes convenient to clone the
repo and make changes directly on the device and then install those
changes. This script automates the process of installing TinyPilot from
the local repo.
This also makes it a little easier for newcomers to the project to make
changes and then test the changes on a real device. Before the update
overhaul, `quick-install` let users do that, but then the update
overhaul brought Gatekeeper into the install path, which prevents users
from installing their own code. This allows users to easily install
their own code again.
Related: #1246
<a data-ca-tag
href="https://codeapprove.com/pr/tiny-pilot/tinypilot/1242"><img
src="https://codeapprove.com/external/github-tag-allbg.png" alt="Review
on CodeApprove" /></a>
* Document TinyPilot's conventions for web components
* Revise text
* Fix quotation marks
* Clarify wording
* Add section on parameterizing style rules
* Fix code snippet termination
* Add a link for shadow DOM and clarify wording
* Keep track of FE dependencies in `package.json` file
* Setup unit tests with Mocha
* Fix Circle config
* Pin dependencies to the exact version
* Upgrade NodeJS to the latest available LTS
* Add FE test script to `build` script
* Replace dummy test with real one
* Setup remote script for pulling and running a specific revision
* Execute git operations with root user (to make use of ssh agent forwarding)
* Make sure we don’t overwrite non-commited changes
* Describe basic SSH setup for testing on real device
* Correct paht
* Improve instructions
* Always fetch from all remotes
* Don’t rely on the root user, but support all users with passwordless sudo permissions
* Update URLs for new TinyPilot org
I've moved tinypilot from mtlynch/tinypilot to tiny-pilot/tinypilot, so this makes all the URLs match the new location.
* Fix spelling of tiny
They aren't really something an end-user should change. They are more useful to developers working on the code.
For end-users, the preferred way to change options is through the settings.yml file and advanced installation (linked to in the README).
The PR template is meant for new contributors, but it pops up for every PR, even from regular contributors. It's creating too much of a cost for regular developers and buying too little with newcomers, so I'm getting rid of the PR template.
* Add mock scripts for development
This enables TinyPilot developers to mock out two privileged functions during development:
* Change hostname
* Collect debug logs
Once the developer enables mock scripts on their system, they can use these two features and have it work (sort of) like it works on a real device.
This still leaves a few functions unmocked:
* Shutdown/restart (because we call those by absolute path to system scripts, we can't mock them out)
* Update: The update logic is pretty complex and involves launching a systemd process, so it's non-trivial to mock this out.
Related: #501
* Handle case where /opt/tinypilot-privileged is a real directory
