Wednesday, December 23, 2020

How to install Parallel Desktop and Windows 10 on Mac M1

Mac M1 can install Parallel Desktop Preview and run x64 Windows app

1) Install PD for M1 Mac from https://www.parallels.com/blogs/parallels-desktop-apple-silicon-mac/ and click Try Technical Preview. Need to login and register.

Direct download link
MD5 checksum 5fc71f89fcc67ed13bac1995604a962f
Activation key You have to register and login to activate the product

2) Install Windows 10 Client ARM64 Insider Preview download from https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewARM64. Need to login and register as Insider Preview Program.
Direct download link>
>
3) Update Win10 ARM to version 21277 which can install and run x64 Windows programs.
Have to login Windows Insider Program and set to "Dev Channel" to recieve preview updates.


Sunday, December 20, 2020

Buid docker image for M1 Mac with development tools and vnc

(1) With the release of docker preview for Mac M1, it is now possible to build some arm64 images.
Docker M1 preview
https://desktop.docker.com/mac/m1preview/Docker-AppleSilicon-Preview7.dmg

(2) Install the above preview and then use terminal the tools. The build script is based on Android SDK docker image for x64 and modify it for arm64. Moreover, swift 5.3.1, clang 10.0.0, kotlin 1.4.21, gradle 6.7.1, gcc 9.3.0, java 11, firefox, visual studio code and codeblocks are added for development purpose.

(3) Shell script run in Mac docker info # check if docker is installed properly in Mac M1
mkdir -p ${HOME}/android_sdk_vnc
cd ${HOME}/android_sdk_vnc
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/sshd-banner
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/supervisord.conf
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/vnc/supervisord_vncserver.conf
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/vnc/vncpass.sh
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/vnc/watchdog.sh

(4) Create the following Dockfile
Dockerfile    Select all
cat >Dockerfile <<'HEREEOF' # ====================================================================== # # Android SDK Docker Image # ====================================================================== # # Base image # ---------------------------------------------------------------------- # FROM arm64v8/ubuntu:20.04 # install essential tools RUN apt-get update && \ apt-get dist-upgrade -y && \ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends git wget wget2 zip unzip && \ apt-get install -y build-essential vim # install Android SDK RUN DEBIAN_FRONTEND=noninteractive apt-get install -y android-sdk && \ rm -f /usr/lib/android-sdk/build-tools/27.0.1 # install Android SDK cmdline tools ARG ANDROID_SDK_VERSION=6858069 ENV ANDROID_SDK_ROOT /usr/lib/android-sdk RUN wget2 -q https://dl.google.com/android/repository/commandlinetools-linux-${ANDROID_SDK_VERSION}_latest.zip && \ unzip commandlinetools*linux*${ANDROID_SDK_VERSION}*.zip && \ mv cmdline-tools tools && \ mkdir -p ${ANDROID_SDK_ROOT}/cmdline-tools && \ mv tools ${ANDROID_SDK_ROOT}/cmdline-tools/ && \ (cd /usr/lib/android-sdk/cmdline-tools; ln -s tools latest) && \ rm -f commandlinetools*linux*.zip # download and install Gradle ARG GRADLE_VERSION=6.7.1 ARG GRADLE_DIST=bin RUN wget2 -q https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-${GRADLE_DIST}.zip && \ rm -fr /usr/share/gradle && \ unzip gradle*.zip -d /usr/share && \ rm gradle*.zip && \ mv /usr/share/gradle-${GRADLE_VERSION} /usr/share/gradle # download and install Kotlin compiler ARG KOTLIN_VERSION=1.4.10 RUN wget2 -q https://github.com/JetBrains/kotlin/releases/download/v${KOTLIN_VERSION}/kotlin-compiler-${KOTLIN_VERSION}.zip && \ unzip *kotlin*.zip && \ mv kotlinc /usr/share && \ rm *kotlin*.zip # install swift5 compiler RUN wget -qO- https://packagecloud.io/install/repositories/swift-arm/release/script.deb.sh | bash RUN apt install -y swiftlang # setup adb server EXPOSE 5037 # set the environment variables ARG JDK_VERSION=1.11.0 ENV JAVA_HOME /usr/lib/jvm/java-${JDK_VERSION}-openjdk-arm64 ENV GRADLE_HOME /usr/share/gradle ENV KOTLIN_HOME /usr/share/kotlinc ENV ANDROID_SDK_ROOT /usr/lib/android-sdk ENV ANDROID_HOME /usr/lib/android-sdk ENV ANDROID_TOOLS /usr/lib/android-sdk/tools ENV ANDROID_PLATFORMS /usr/lib/android-sdk/platforms ENV ANDROID_PLATFORM_TOOLS /usr/lib/android-sdk/platform-tools ENV ANDROID_BUILD_TOOLS /usr/lib/android-sdk/build-tools ENV ANDROID_CMDLINE_TOOLS /usr/lib/android-sdk/cmdline-tools/tools ENV PATH ${PATH}:${GRADLE_HOME}/bin:${KOTLIN_HOME}/bin:${ANDROID_BUILD_TOOLS}/debian:${ANDROID_TOOLS}/bin:${ANDROID_PLATFORM_TOOLS}:${ANDROID_CMDLINE_TOOLS}/bin:${ANDROID_SDK_ROOT}/emulator # install system image and accept licence, installation of emulator will fail for arm64 system RUN yes Y | ${ANDROID_CMDLINE_TOOLS}/bin/sdkmanager --install "platform-tools" "system-images;android-24;google_apis;arm64-v8a" "platforms;android-24" "build-tools;30.0.3" && \ wget https://github.com/qhuyduong/arm_adb/releases/download/v1.0.39-aarch64/adb && \ chmod +x adb && mv adb /usr/lib/android-sdk/platform-tools/ # install and configure SSH server EXPOSE 22 ADD sshd-banner /etc/ssh/ # ADD authorized_keys /tmp/ RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends openssh-server supervisor locales && \ mkdir -p /var/run/sshd /var/log/supervisord && \ locale-gen en en_US en_US.UTF-8 && \ apt-get remove -y locales && apt-get autoremove -y && \ FILE_SSHD_CONFIG="/etc/ssh/sshd_config" && \ echo "\nBanner /etc/ssh/sshd-banner" >> $FILE_SSHD_CONFIG && \ echo "\nPermitUserEnvironment=yes" >> $FILE_SSHD_CONFIG && \ ssh-keygen -q -N "" -f /root/.ssh/id_rsa && \ FILE_SSH_ENV="/root/.ssh/environment" && \ touch $FILE_SSH_ENV && chmod 600 $FILE_SSH_ENV && \ printenv | grep "JAVA_HOME\|GRADLE_HOME\|KOTLIN_HOME\|ANDROID_SDK_ROOT\|LD_LIBRARY_PATH\|PATH" >> $FILE_SSH_ENV && \ echo "\nauth required pam_env.so envfile=$FILE_SSH_ENV" >> /etc/pam.d/sshd && \ FILE_AUTH_KEYS="/root/.ssh/authorized_keys" && \ touch $FILE_AUTH_KEYS && chmod 600 $FILE_AUTH_KEYS && \ for file in /tmp/*.pub; \ do if [ -f "$file" ]; then echo "\n" >> $FILE_AUTH_KEYS && cat $file >> $FILE_AUTH_KEYS && echo "\n" >> $FILE_AUTH_KEYS; fi; \ done && \ (rm /tmp/*.pub 2> /dev/null || true) # install and configure VNC server ENV USER root ENV DISPLAY :1 EXPOSE 5901 ADD vncpass.sh /tmp/ ADD watchdog.sh /usr/local/bin/ ADD supervisord_vncserver.conf /etc/supervisor/conf.d/ RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends xfce4 xfce4-goodies xfonts-base dbus-x11 tightvncserver expect && \ apt-get install -y --no-install-recommends firefox codeblocks && \ chmod +x /tmp/vncpass.sh; sync && \ /tmp/vncpass.sh && \ rm /tmp/vncpass.sh && \ apt-get remove -y expect && apt-get autoremove -y && \ FILE_SSH_ENV="/root/.ssh/environment" && \ echo "DISPLAY=:1" >> $FILE_SSH_ENV # install and patch Visual Studio Code RUN apt update && apt install -y libsecret-1-0 libxss1 libgbm1 && apt-get clean RUN wget -O code_arm64.deb https://aka.ms/linux-arm64-deb && dpkg -i code_arm64.deb && rm code_arm64.deb RUN sed 's/BIG-REQUESTS/_IG-REQUESTS/' /usr/lib/aarch64-linux-gnu/libxcb.so.1.1.0 | tee /usr/share/code/libxcb.so.1 > /dev/null ADD supervisord.conf /etc/supervisor/conf.d/ CMD ["/usr/bin/supervisord"] HEREEOF


(5) build the Dockfile
docker build -f Dockerfile -t arm64v8/android-sdk-vnc .


(6) Copy Mac host id_rsa.pub to authorized_keys and start the container
cp ~/.ssh/authorized_keys .
cat ~/.ssh/id_rsa.pub >> authorized_keys


# To start the container as daemon
docker run -d --name android-sdk-vnc -p 5901:5901 -p 2222:22 -p 5037:5037 \
-v $(pwd)/authorized_keys:/root/.ssh/authorized_keys arm64v8/android-sdk-vnc

# To connect the container via ssh
ssh root@<IP address of your Mac M1> -p 2222

# To stop the container
docker container stop android-sdk-vnc


(7) To connect to the vnc of the docker image after starting the container
Use the Mac Finder "Go" -> "Connect To Server" and enter "vnc://0.0.0.0:5901"
For other machines in the network enter "vnc://<IP address of your Mac M1>:5901"

The passwords are in the vncpass.sh when building which are
password="android"
password_view_only="docker"

(8) There is a common problem if you are connecting remotely using Mac Terminal, so the solution is to not forward your locale. Edit /etc/ssh/ssh_config and comment out SendEnv LANG LC_* line.

(9) Testing swift
# attach to the running container
docker exec -it android-sdk-vnc bash
# test
mkdir -p ${HOME}/Projects
cd ${HOME}/Projects
git clone https://github.com/apple/example-package-dealer.git
cd example-package-dealer
swift run Dealer




If you need port forwarding on macOS, see this article https://www.papercut.com/kb/Main/MacPortForwarding

Saturday, December 19, 2020

How to setup debian vm with VNC on New Mac Mini M1

(1) Download the ubuntu kernel and images
cd ${HOME}
mkdir debianvm
curl -o - https://cloud-images.ubuntu.com/releases/focal/release/unpacked/ubuntu-20.04-server-cloudimg-arm64-vmlinuz-generic | gunzip > debianvm/vmlinuz
curl -o debianvm/initrd https://cloud-images.ubuntu.com/releases/focal/release/unpacked/ubuntu-20.04-server-cloudimg-arm64-initrd-generic
curl -o - https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-arm64.tar.gz | tar zxvC debianvm

(2) download the raspbian arm64 image
curl -OL https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2020-08-24/2020-08-20-raspios-buster-arm64.zip
unzip 2020-08-20-raspios-buster-arm64.zip -d debianvm

(3) Copy and expand the disk to say 38G (Don't use this method to expand disk on external HD)
cp debianvm/focal-server-cloudimg-arm64.img debianvm/raspberry-arm64.img
dd if=/dev/zero of=debianvm/raspberry-arm64.img seek=40000000 obs=1024 count=0

(4) Compile and use the latest vftool which is enhanced to mount multiple disks.
cd ${HOME}
git clone https://github.com/evansm7/vftool
cd vftool && xcodebuild

(5) start the vm say
screen -S vm
cd ${HOME}/debianvm
${HOME}/vftool/build/Release/vftool -k vmlinuz -i initrd -d focal-server-cloudimg-arm64.img -d raspberry-arm64.img -d 2020-08-20-raspios-buster-arm64.img -m 2048 -a "console=hvc0"
Detach the screen using Ctrl-A D

(6) start the tty say
screen -S tty /dev/ttys001

(7) prepare the ubuntu cloud image
mkdir /mnt
mount /dev/vda /mnt
chroot /mnt

touch /etc/cloud/cloud-init.disabled

echo 'root:root' | chpasswd

ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519

# add fixed IP
cat <<EOF > /etc/netplan/01-dhcp.yaml 
network:
    renderer: networkd
    ethernets:
        enp0s1:
            dhcp4: true
            addresses: [192.168.64.33/24]
    version: 2
EOF

# add IP when login prompt
cat <<EOF >> /etc/issue
enp0s1: \4

EOF

exit
umount /dev/vda



(8) copy the rootfs to raspberry-arm64.img
#Detach the screen using Ctrl-A D
screen -r vm # resume vm screen # stop the vm by ctrl-C and restart the ubuntu vm
cd ${HOME}/debianvm
${HOME}/vftool/build/Release/vftool -k vmlinuz -i initrd -d focal-server-cloudimg-arm64.img -d raspberry-arm64.img -d 2020-08-20-raspios-buster-arm64.img -m 2048 -a "console=hvc0 root=/dev/vda"

#Detach the screen using Ctrl-A D
#Notice the tty number when starting the vm and start the tty say
screen -S tty /dev/ttys001

# login with id root and password root
# and copy the rootfs to the expanded disk raspberry-arm64.img
dd if=/dev/vdc2 of=/dev/vdb bs=64K conv=noerror,sync
exit

(9) prepare the raspberry-arm64 image
# stop the vm and restart the vm
# Control-A D to detach the tty screen # resume vm screen screen -r vm # Control-C to stop the vm cd ${HOME}/debianvm
${HOME}/vftool/build/Release/vftool -k vmlinuz -i initrd -d raspberry-arm64.img -m 2048 -a "console=hvc0"
mkdir /mnt
mount /dev/vda /mnt
chroot /mnt

echo 'root:root' | chpasswd

#!!! delete or comment out the lines in /etc/fstab, or there will be "A start job is running on /dev/disk/.." blocking!!
cat <<EOF > /etc/fstab
proc            /proc           proc    defaults          0       0
#PARTUUID=ad09722e-01  /boot           vfat    defaults          0       2
#PARTUUID=ad09722e-02  /               ext4    defaults,noatime  0       1
EOF

ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519

mkdir -p /etc/network/interfaces.d/
cat <<EOF > /etc/network/interfaces.d/enp0s1 
allow-hotplug enp0s1
iface enp0s1 inet dhcp
auto lo
iface lo inet loopback
EOF

cat <<EOF >> /etc/dhcpcd.conf
# add fixed IP
interface enp0s1
static ip_address=192.168.64.10/24
EOF

# put mac host ${HOME}/.ssh/id.rsa.pub key here
mkdir -p /root/.ssh
cat <<EOF > /root/.ssh/authorized_keys
ssh-rsa <mac id.rsa.pub key here>
EOF

# add IP when login prompt
cat <<EOF >> /etc/issue
enp0s1: \4

EOF

exit
umount /dev/vda

(10) login raspberry OS arm64 with id root and password root
# stop the vm and restart the vm with
cd ${HOME}/debianvm
${HOME}/vftool/build/Release/vftool -k vmlinuz -i initrd -d raspberry-arm64.img -m 2048 -a "console=hvc0 root=/dev/vda rw"
# set date say
date -s "29 DEC 2020 14:54:00"
resize2fs /dev/vda
dpkg-reconfigure tzdata # set timezone
apt update
systemctl disable rpi-eeprom-update.service
systemctl disable rng-tools.service

(11) install utilities in raspberry os that can handle qcow2 disk image conversion and resize
apt install qemu-utils libguestfs-tools
hostname -I # get ip address

(13) Install Realvncserver and connect from Mac
passwd pi # set user pi password
# login as user pi
login pi
# Use raspi-config to enable SSH and VNC
sudo apt install realvnc-vnc-server
sudo raspi-config
# Choose 5 Interfacing Options -> P2 SSH
# Choose 5 Interfacing Options -> P3 VNC

# enter vnc key, can put this in /etc/rc.local without sudo
sudo vnclicense -add VKUPN-MTHHC-UDHGS-UWD76-6N36A

# setup vncpasswd for user mode, only user mode can run browser
vncpasswd -user

# edit config file to have these two lines for vnc password auth
echo -e "Authentication=VncAuth" | tee -a ~/.vnc/config.d/vncserver-x11
echo -e "Encryption=PreferOff" | tee -a ~/.vnc/config.d/vncserver-x11
cp ~/.vnc/config.d/vncserver-x11 ~/.vnc/config.d/Xvnc
# start service
vncserver

#put these in rc.local to automate after reboot, Raspberry Pi hardware has bundled vnclicense and no need to add it
/usr/bin/vnclicense -add VKUPN-MTHHC-UDHGS-UWD76-6N36A
sudo -H -u pi /usr/bin/vncserver

#check which port is listening
netstat -tlpn | grep -i listen

This would configure VNC in Virtual Mode
# Screen Sharing Connect from Mac
# In Finder, Go -> Connect to Server, enter IP address and port number of the vm e.g.
# Enter the vncpassword to connect, and connection needs port number
vnc://192.168.64.10:5901

For configuring VNC in Service Mode, please refer to https://www.raspberrypi.org/documentation/remote-access/vnc/
For differences in these modes, please refer to https://help.realvnc.com/hc/en-us/articles/360002253238-Understanding-VNC-Server-Modes
#Start or stop the service with:
# systemctl (start|stop) vncserver-x11-serviced.service
#Mark or unmark the service to be started at boot time with:
# systemctl (enable|disable) vncserver-x11-serviced.service
#sudo systemctl start vncserver-x11-serviced.service
For Service Mode the connection from Mac is (no port number), vnc://192.168.64.10

If you need port forwarding on macOS, so that you can vnc connect from other machines on the LAN to this vm, see this article https://www.papercut.com/kb/Main/MacPortForwarding That is run
echo "
rdr pass on lo0 inet proto tcp from any to self port 5910 -> 192.168.64.10 port 5901
rdr pass on en0 inet proto tcp from any to any port 5910 -> 192.168.64.10 port 5901
rdr pass on wlan0 inet proto tcp from any to any port 5910 -> 192.168.64.10 port 5901
" | sudo pfctl -ef -
Then, test connect from other machines to :5901 on the LAN.

# To install Visual Studio Code
wget -O code_arm64.deb https://aka.ms/linux-arm64-deb && sudo dpkg -i code_arm64.deb && rm code_arm64.deb
sed 's/BIG-REQUESTS/_IG-REQUESTS/' /usr/lib/aarch64-linux-gnu/libxcb.so.1.1.0 | sudo tee /usr/share/code/libxcb.so.1 > /dev/null

Demo - VNC to rapberrypi desktop vm screen capture

P.S. Only vm can run realvncserver, docker environment can only run lightweight vncserver e.g. tightvncserver

(14) For qcow2 image, the conversion to raw image is
wget https://cloud.debian.org/images/cloud/buster/20201214-484/debian-10-generic-arm64-20201214-484.qcow2
qemu-img dd -f qcow2 -O raw bs=64k if=debian-10-generic-arm64-20201214-484.qcow2 of=debian-10.img
# scp debian-10.img to the Mac Mini host and prepare a new expanded debian-arm64.img similar to the raspberry-arm64.img as above
# After restarting vm and mounting images, the dd command to extract rootfs is
# Assume debian-arm64.img is vdb and debian-10.img is vdc
dd if=/dev/vdc1 of=/dev/vdb bs=64K conv=noerror,sync
# For debian image, add this to /etc/hosts 127.0.1.1 debian

Thursday, December 3, 2020

How to run docker on the new Mac Mini M1

(1) Reference to this article https://finestructure.co/blog/2020/11/27/running-docker-on-apple-silicon-m1

(2) The problem of this method of running Ubuntu iso live image is not persistent.

(3) So here is the improved method to add persistent storage to this vm.

(4) # First create an empty image data say 50G in Mac Mini
dd if=/dev/zero of=data.img bs=1m count=51200

(5) # Change the command to invoke the vm to include persistent and increase cpu to 2
./vftool -k vmlinuz -i initrd -c focal-desktop-arm64.iso -d data.img -p 2 -m 4096 -a "console=hvc0 persistent rw"
./vftool -k vmlinuz -i initrd -d data.img -c focal-desktop-arm64.iso -p 2 -m 4096 -a "console=hvc0 persistent rw"

P.S.
If you compile the latest vftool, which has added the ability to add more image files and the order of the mounted device will follow the parameters you passed in. The article was written using the first version of the vftool and will mount the data.img as the /dev/vda and the cdrom image as /dev/vdb. So if you use the latest version vftool compiled, you have to check the device name of your mounted data.img.

(6) # Attach the vm and create 3 partitions for the created image
sudo fdisk /dev/vda
# Press (n)(p)(1) to create new partition 1
# Press (enter) to accept starting sector and set size to say +25G
# Press (n)(p)(2) to create new partition 2
# Press (enter) to accept starting sector and set size to say +15G
# Press (n)(p)(3) to create new partition 3
# Accept the default options and starting block and ending block
# Press (w) to write to partition

(6) # Format partition as ext4 with labels casper-rw and home-rw in vm, this create the persistent medium for the rw partition.
sudo mkfs.ext4 /dev/vda1 -L docker-data
sudo mkfs.ext4 /dev/vda2 -L casper-rw
sudo mkfs.ext4 /dev/vda3 -L home-rw

(7) # exit shell and exit vm by ctrl-c
exit

(8) # start vm again and use a new tab using screen command to attach to the vm e.g.
screen -S docker /dev/ttys003

(9) # Create folder to mount /dev/vda1
sudo mkdir -p /mnt/docker-data
# Add this mount commands in /etc/rc.local in order to persist as editing /etc/fstab is ignored in this iso image
cat << EOF | sudo tee -a /etc/rc.local 
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

mount /dev/vda1 -t ext4 /mnt/docker-data
exit 0
EOF

cat << EOF | sudo tee -a /etc/systemd/system/rc-local.service
[Unit]
 Description=/etc/rc.local Compatibility
 ConditionPathExists=/etc/rc.local

[Service]
 Type=forking
 ExecStart=/etc/rc.local start
 TimeoutSec=0
 StandardOutput=tty
 RemainAfterExit=yes
 SysVStartPriority=99

[Install]
 WantedBy=multi-user.target
EOF

# enable rc.local see this.
sudo chmod +x /etc/rc.local
sudo systemctl start rc-local.service
sudo systemctl status rc-local.service

(10) Follow the reference article to install docker related packages.
If you have problem when installing docker. Try this = previous verion
apt install docker-ce=5:19.03.14~3-0~ubuntu-focal

(11) # Move docker data-root to the newly created partition
sudo service docker stop
cat << EOF | sudo tee -a /etc/docker/daemon.json 
{
"storage-driver": "overlay2",
"data-root": "/mnt/docker-data"
}
EOF
sudo rsync -aP /var/lib/docker/ /mnt/docker-data
sudo mv /var/lib/docker /var/lib/docker.old
sudo service docker start

(12) Additionally, install docker-compose
sudo add-apt-repository universe
sudo apt install -y python3-pip
sudo pip3 -v install docker-compose

(13) Test docker
docker run hello-world

docker run -e MYSQL_ROOT_PASSWORD=rootpassword -e MYSQL_USER=wpuser -e MYSQL_PASSWORD=wpuserpassword -e MYSQL_DATABASE=wordpressdb --name wordpressdb -d arm64v8/mariadb
docker run -e WORDPRESS_DB_USER=wpuser -e WORDPRESS_DB_PASSWORD=wpuserpassword -e WORDPRESS_DB_NAME=wordpressdb -p 8080:80 --link wordpressdb:mysql --name wordpress -d arm64v8/wordpress
# show IP address
ip addr | grep inet
# test using browser e.g. http://192.168.64.19:8080/

(14) Test docker-compose 1 (wordpress and mysql)
# create docker-compose.yaml
cd $HOME
mkdir my-wordpress
cd my-wordpress
cat > docker-compose.yaml <<EOF
version: '3'
services:
  db:
    image: arm64v8/mariadb
    volumes:
      - "./.data/db:/var/lib/mysql"
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: rootpassword
      MYSQL_DATABASE: wordpressdb
      MYSQL_USER: wpuser
      MYSQL_PASSWORD: wpuserpassword

  wordpress:
    depends_on:
      - db
    image: arm64v8/wordpress
    links:
      - db:mysql
    ports:
      - "8000:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_NAME: wordpressdb
      WORDPRESS_DB_USER: wpuser
      WORDPRESS_DB_PASSWORD: wpuserpassword
EOF
# Start docker-compose
docker-compose up -d
# test using browser e.g. http://192.168.64.19:8000/
# after testing bring it down
docker-compose down -v

(15) Test docker-compose 2 (mongo-express)
# create docker-compose.yaml
cd $HOME
mkdir my-mongoexpress
cd my-mongoexpress
cat > docker-compose.yaml <<EOF
version: '3.1'
services:
  mongo:
    image: arm64v8/mongo
    volumes:
      - ./data:/data/db
    ports:
      - '27018:27017'
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: 'root'
      MONGO_INITDB_ROOT_PASSWORD: 'example'
      MONGO_INITDB_DATABASE: 'db'

  mongo-express:
    image: arm64v8/mongo-express
    links:
      - mongo
    ports:
      - "8081:8081"
    restart: always
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: 'root'
      ME_CONFIG_MONGODB_ADMINPASSWORD: 'example'
      ME_CONFIG_MONGODB_ENABLE_ADMIN: 'true'
      ME_CONFIG_MONGODB_AUTH_DATABASE: 'db'
      ME_CONFIG_OPTIONS_EDITORTHEME: 'ambiance'
      ME_CONFIG_BASICAUTH_USERNAME: 'user'
      ME_CONFIG_BASICAUTH_PASSWORD: 'password'
EOF
# Start docker-compose
docker-compose up -d --remove-orphans
# test using browser e.g. http://192.168.64.19:8081/ with user password
# after testing bring it down
docker-compose down -v

(16) Change password and install openssh-server
sudo apt install -y openssh-server
# change /etc/ssh/sshd_config PermitEmptyPasswords to yes
PermitEmptyPasswords yes

(17) Use another terminal session to connect e.g.
ssh-copy-id ubuntu@192.168.64.19
ssh ubuntu@192.168.64.19

(18) This iso image has desktop Applications, so it is possible to install vnc server and connect to it via Remote Desktop or vnc://localhost:590x. To install vnc server
sudo apt install -y tightvncserver
vncserver
# restart vncserver in vm
vncserver -localhost
# create ssh tunnel in Mac, where x is the desktop number when starting vncserver
ssh ubuntu@192.168.64.19 -L 590x:localhost:590x # Reference : https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-20-04
https://cat.pdx.edu/platforms/mac/remote-access/vnc-to-linux/

P.S. follow up post for the docker M1 preview and building of docker image with vnc server.
https://iphonesdkdev.blogspot.com/2020/12/20201205.html



If you need port forwarding on macOS to connect to this vm service (e.g. ssh or vnc), see this article https://www.papercut.com/kb/Main/MacPortForwarding

How to run on Headless Mac Mini M1 with nohup and screen

(1) Need a HDMI dummy simulator attached to Mac Mini for full speed operation. Be sure to have HDMI 2.0 and 4K support and some dummy simulators have Pass-Through EDID Emulator (also with 4K support) so that you can attach 4K monitor or via screen sharing in HD when working on it with attached keyboard and mouse.

(2) Disable sleep mode and set never sleep for HD and monitor on System Preferences of Mac Mini.

(3) Enable Sharing and SSH login on Mac Mini.

(4) generate ssh key-pair on mac mini and your terminal or your phone. If using Android phone, recommend Termux app for remote shell access.
e.g. ssh-keygen -t rsa # generate key pair on Mac Mini and remote client

(5) copy the ssh key-pair to mac mini if using another terminal from mac.
e.g. ssh-copy-id user@x.x.x.x

(6) Login in Mac Mini via ssh remotely
e.g. ssh user@x.x.x.x

(7) # Apple Compressor command line batch submission example (no need for nohup)
/Applications/Compressor.app/Contents/MacOS/Compressor -batchname "My First Batch" \
-jobpath ~/Movies/MySource_x265.mp4 \
-settingpath /Applications/Compressor.app/Contents/Resources/Settings/ProRes/proRes422Name.compressorsetting \
-locationpath ~/Movies/MyOutput_ProRes422.mov

(8) # install youtube-dl, prerequisite: Command Line Tools for Xcode 12.2 for Python 3.8 to be installed
sudo mkdir -p /usr/local/bin
sudo curl -L https://github.com/l1ving/youtube-dl/releases/latest/download/youtube-dl -o /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl
youtube-dl --version
sudo youtube-dl -U # update youtube-dl
youtube-dl --version # check version after update
cd $HOME/Downloads
mkdir -p youtube
cd youtube
youtube-dl -f mp4 <youtube url>

(9) Either use nohup or screen command for persistent jobs.

# nohup example
nohup bash -c 'youtube-dl -f 401 https://youtu.be/8rPB4A3zDnQ' &> nohup.youtubedl.4k_2.out &

(10)screen example (suitable for command utility with tty input
# create a screen and run python script
screen -S tensorflow
conda activate python38
python cnn.py
# leave (detach) the screen by Control-A d

# create second screen and run handbrake conversion
screen -S handbrake
conda deactivate
$HOME/Downloads/HandBrakeCLI -i $HOME/Downloads/8KVIDEO120FPS-8rPB4A3zDnQ_4.1G.mp4 -o $HOME/Downloads/8KVIDEO_x265.mp4 -e x265 -q24
# leave (detach) the screen by Control-A d

# list all the screen ids
screen -ls

# Reattach to one of the screen e.g. 6473.tensorflow
screen -r 6473

# Exit Screen if the process terimated
exit

Saturday, November 21, 2020

How to install tensorflow for the new Mac M1 hardware

Prerequisite: Xcode 12.2 and Command Line Tools for Xcode 12.2

(1) # Download the archive for this repo from https://github.com/apple/tensorflow_macos/releases
cd $HOME/Downloads/
curl -fsSLO https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha0/tensorflow_macos-0.1alpha0.tar.gz
tar xzvf tensorflow_macos-0.1alpha0.tar.gz
/bin/bash ./tensorflow_macos/install_venv.sh --help

(2) # Download Miniconda from https://conda-forge.org/blog/posts/2020-10-29-macos-arm64/

(3) # Install Miniconda and after installtion, exit shell and login again
/bin/bash -c "$(curl -fsSL https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh)"

(4) # Once it's installed, create a Python 3.8 env by running
conda create --name python38 python=3.8

(5) # Put a path to where the arm64 libraries are. For example...
libs="$HOME/Downloads/tensorflow_macos/arm64/"

(6) # Replace this with the path of your Conda environment
env="$HOME/miniforge3/envs/python38"

(7) # upgrade
conda upgrade -c conda-forge pip setuptools cached-property six

(8) # activate env
conda activate python38
# conda deactivate

(9) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/grpcio-1.33.2-cp38-cp38-macosx_11_0_arm64.whl"

(10) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/h5py-2.10.0-cp38-cp38-macosx_11_0_arm64.whl"

(11) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/numpy-1.18.5-cp38-cp38-macosx_11_0_arm64.whl"

(12) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/tensorflow_addons-0.11.2+mlcompute-cp38-cp38-macosx_11_0_arm64.whl"

(13) # install these
conda install -c conda-forge -y absl-py
conda install -c conda-forge -y astunparse
conda install -c conda-forge -y gast
conda install -c conda-forge -y opt_einsum
conda install -c conda-forge -y termcolor
conda install -c conda-forge -y typing_extensions
conda install -c conda-forge -y wheel
conda install -c conda-forge -y typeguard

pip install tensorboard

pip install wrapt flatbuffers tensorflow_estimator google_pasta keras_preprocessing protobuf



(14) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/tensorflow_macos-0.1a0-cp38-cp38-macosx_11_0_arm64.whl"

(15) # Run this to test
time python tftest.py


tftest.py    Select all
from datetime import datetime import numpy as np import tensorflow as tf from tensorflow.python.compiler.mlcompute import mlcompute mlcompute.set_mlc_device(device_name="cpu") # tensorflow:Eager mode on GPU is extremely slow. So use CPU instead print("Hello, Tensorflow! ", end='') print(tf.__version__) print("start" , datetime.now()) X_raw = np.array([2013, 2014, 2015, 2016, 2017, 2018], dtype=np.float32) y_raw = np.array([12000, 14000, 15000, 16500, 17500, 19000], dtype=np.float32) X = (X_raw - X_raw.min()) / (X_raw.max() - X_raw.min()) y = (y_raw - y_raw.min()) / (y_raw.max() - y_raw.min()) X = tf.constant(X) y = tf.constant(y) a = tf.Variable(initial_value=0.) b = tf.Variable(initial_value=0.) variables = [a, b] num_epoch = 10000 optimizer = tf.keras.optimizers.SGD(learning_rate=1e-3) for e in range(num_epoch): with tf.GradientTape() as tape: y_pred = a * X + b loss = 0.5 * tf.reduce_sum(tf.square(y_pred - y)) grads = tape.gradient(loss, variables) optimizer.apply_gradients(grads_and_vars=zip(grads, variables)) print(a, b) print("end" , datetime.now())




(16) Test run this cnn.py https://github.com/apple/tensorflow_macos/issues/25

Saturday, October 24, 2020

How to use SpriteKit in SwiftUI and preview SKScene in Xcode 12

This demo the code using Xcode 12.0.1 under macOSX 10.15.7 The iOS14 has a new SpriteView for the SKScene, this demo a player movement with on-screen joystick control. This file does not need the creation of sks file and using code to generate the SKNodes required. The graphics assets and touches codes are borrowed from this Elton Game - Introduction to Sprite Kit. https://designcode.io/spritekit-intro

Create this swift file for iOS14 and you can preview it in SwiftUI under Xcode 12


SKContentView.swift.sh    Select all
// // SKContentView.swift // import SwiftUI import SpriteKit class GameScene: SKScene { // Nodes var player : SKNode? var joystick : SKNode? var joystickKnob : SKNode? // Boolean var joystickAction = false // Measure var knobRadius : CGFloat = 50.0 // Sprite Engine added in Lesson 2 var previousTimeInterval : TimeInterval = 0 let playerSpeed = 4.0 override func didMove(to view: SKView) { physicsBody = SKPhysicsBody(edgeLoopFrom: frame) let player = SKSpriteNode(imageNamed: "player/1") self.player = player let joystick = SKSpriteNode(imageNamed: "arrow") self.joystick = joystick let joystickKnob = SKSpriteNode(imageNamed: "knob") self.joystickKnob = joystickKnob joystick.addChild(joystickKnob) player.position = CGPoint(x: frame.midX, y: frame.size.height / 5 * 3) joystick.position = CGPoint(x: 80, y: 80) addChild(player) addChild(joystick) view.showsFPS = true view.showsNodeCount = true } } extension GameScene { // Touch Began override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) { for touch in touches { if let joystickKnob = joystickKnob { let location = touch.location(in: joystick!) joystickAction = joystickKnob.frame.contains(location) } } } // Touch Moved override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) { guard let joystick = joystick else { return } guard let joystickKnob = joystickKnob else { return } if !joystickAction { return } // Distance for touch in touches { let position = touch.location(in: joystick) let length = sqrt(pow(position.y, 2) + pow(position.x, 2)) let angle = atan2(position.y, position.x) if knobRadius > length { joystickKnob.position = position } else { joystickKnob.position = CGPoint(x: cos(angle) * knobRadius, y: sin(angle) * knobRadius) } } } // Touch End added in Lesson 2 override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) { for touch in touches { let xJoystickCoordinate = touch.location(in: joystick!).x let xLimit: CGFloat = 200.0 if xJoystickCoordinate > -xLimit && xJoystickCoordinate > xLimit { resetKnobPosition() } } } } // MARKS: Action added in Lesson 2 extension GameScene { func resetKnobPosition() { let initialPoint = CGPoint(x:0, y:0) let moveBack = SKAction.move(to: initialPoint, duration:0.1) moveBack.timingMode = .linear joystickKnob?.run(moveBack) joystickAction = false } } // MARKS: Game Loop added in Lesson 2 extension GameScene { override func update( _ currentTime: TimeInterval) { let deltaTime = currentTime - previousTimeInterval previousTimeInterval = currentTime // player movement guard let joystickKnob = joystickKnob else { return } let xPosition = Double(joystickKnob.position.x) let yPosition = Double(joystickKnob.position.y) let displacement = CGVector(dx:deltaTime * xPosition * playerSpeed, dy:deltaTime * yPosition * playerSpeed) let move = SKAction.move(by: displacement, duration:0) player?.run(move) } } struct SKContentView: View { var scene: SKScene { let scene = GameScene() scene.size = CGSize(width: 300, height: 400) scene.scaleMode = .fill return scene } var body: some View { SpriteView(scene: scene) .frame(width: 300, height: 400) .edgesIgnoringSafeArea(.all) } } struct SKContentView_Previews: PreviewProvider { static var previews: some View { SKContentView() } }






player/1
arrow
knob

Saturday, October 17, 2020

How to download WWDC2020 videos and subtitles

The scripts for previous WWDC are here https://iphonesdkdev.blogspot.com/2017/07/how-to-fetch-wwdc-2017-video-subtitle.html

Create and Run this script wwdc2020_fetch_srt.sh to fetch WWDC2020 subtitle


wwdc2020_fetch_en.srt.sh    Select all
#!/bin/bash # @Last Modified by: javacom # @Last Modified time: 2020-10-17 WWDC_YEAR=2020; # WWDC_SESSION_PREFIX=https://developer.apple.com/videos/play/wwdc$WWDC_YEAR; WWDC_LOCAL_DIR=$(basename $WWDC_SESSION_PREFIX); detect_video_m3u8 () { local session_url=$WWDC_SESSION_PREFIX/$SESSION_ID/; local session_html=$(curl -s $session_url); local video_url=$(echo "$session_html" | grep .m3u8 | grep $SESSION_ID | head -n1 | sed "s#.*\"\(https://.*m3u8\)\".*#\1#"); echo "$session_html" | grep .mp4 | grep $SESSION_ID | sed "s#.*\"\(https://.*mp4\).*\".*#\1#" | while read mp4_url; do local mp4_filename=$(basename $mp4_url | cut -d. -f1); local srt_filename=$mp4_filename.en.srt; # local srt_filename=$mp4_filename.zh.srt; echo "> Subtitle local: $WWDC_LOCAL_DIR/$srt_filename" >&2; > $WWDC_LOCAL_DIR/$srt_filename; done echo "$video_url"; echo "> Video: $video_url" >&2; } detect_subtitle_m3u8 () { local video_url=$1; # en subtitle local subtitle_uri=$(curl -s $video_url | grep "LANGUAGE=\"en\",URI" | sed "s#.*URI=\"\(.*\)\"#\1#"); # zh subtitle #local subtitle_uri=$(curl -s $video_url | grep "LANGUAGE=\"zh\"" | sed "s#.*URI=\"\(.*\)\"#\1#"); local subtitle_url=$subtitle_uri; [[ "$subtitle_uri" != http* ]] && { subtitle_url=$(dirname $video_url)/$subtitle_uri; } echo "$subtitle_url"; echo "> Subtitle: $subtitle_url" >&2; } download_subtitle_contents () { local subtitle_url=$1; echo "> Downloading... " local subtitle_base_url=$(dirname $subtitle_url); curl -s $subtitle_url | grep "webvtt" | while read webvtt; do local subtitle_webvtt=$subtitle_base_url/$webvtt; #echo "- get $subtitle_webvtt"; local subtitle_content=$(curl -s $subtitle_webvtt); # en subtitle ls $WWDC_LOCAL_DIR/"wwdc$WWDC_YEAR"_"$SESSION_ID"*.en.srt | while read srt_file; do # zh subtitle # ls $WWDC_LOCAL_DIR/"wwdc$WWDC_YEAR"_"$SESSION_ID"*.zh.srt | while read srt_file; do echo "$subtitle_content" >> $srt_file; done done } main () { [ ! -d $WWDC_LOCAL_DIR ] && { mkdir $WWDC_LOCAL_DIR; } #Year 2020 change {3\} to {3,5\} curl -s $WWDC_SESSION_PREFIX | grep /videos/play/wwdc$WWDC_YEAR | sed "s#.*/videos/play/wwdc$WWDC_YEAR/\([0-9]\{3,5\}\).*#\1#" | sort | uniq | while read SESSION_ID; do #echo "SESSION_ID is" $SESSION_ID local video_url=$(detect_video_m3u8 $SESSION_ID); local subtitle_url=$(detect_subtitle_m3u8 $video_url); download_subtitle_contents $subtitle_url; done } main;




Run this shell script to format as SRT subtitle

shellscript.sh    Select all
WWDC_YEAR=2020; # cd wwdc$WWDC_YEAR mkdir -p sd mkdir -p hd for i in *_sd.??.srt; do sed -e '/WEBVTT/d;/X-TIMESTAMP/d;s/align.middle line.*$//;' $i | awk '/^[0-9]{2}:[0-9]{2}:/ {seen[$0]++; skipduplicated=0} {if (seen[$0]>1) skipduplicated=1; if (!skipduplicated) print $0}' | awk -v RS="" '{gsub("\n", "-Z"); print}' | awk '$0 !~/^WEB/ {print $0}' | uniq | awk '{printf "\n%s-Z%s", NR,$0 }' | awk -v ORS="\n\n" '{gsub("-Z", "\n"); print}' | sed -e 's/.A:middle$//g;s/&gt;/>/g;s/&lt;/</g;1,2d;' > sd/$i; done for i in *_hd.??.srt; do sed -e '/WEBVTT/d;/X-TIMESTAMP/d;s/align.middle line.*$//;' $i | awk '/^[0-9]{2}:[0-9]{2}:/ {seen[$0]++; skipduplicated=0} {if (seen[$0]>1) skipduplicated=1; if (!skipduplicated) print $0}' | awk -v RS="" '{gsub("\n", "-Z"); print}' | awk '$0 !~/^WEB/ {print $0}' | uniq | awk '{printf "\n%s-Z%s", NR,$0 }' | awk -v ORS="\n\n" '{gsub("-Z", "\n"); print}' | sed -e 's/.A:middle$//g;s/&gt;/>/g;s/&lt;/</g;1,2d;' > hd/$i; done




Run this script wwdc2020_fetch_mp4.sh to download all mp4 (HD and SD) videos (also works for 2021)

wwdc2020_fetch_mp4.sh    Select all
#!/bin/bash # @Last Modified by: javacom # @Last Modified time: 2020-10-17 WWDC_YEAR=2020; # change to 2021 also works for WWDC2021 # WWDC_SESSION_PREFIX=https://developer.apple.com/videos/play/wwdc$WWDC_YEAR; WWDC_LOCAL_DIR=$(basename $WWDC_SESSION_PREFIX); download_mp4_video () { local session_url=$WWDC_SESSION_PREFIX/$SESSION_ID/; local session_html=$(curl -s $session_url); local video_url=$(echo "$session_html" | grep .m3u8 | grep $SESSION_ID | head -n1 | sed "s#.*\"\(https://.*m3u8\)\".*#\1#"); echo "$session_html" | grep .mp4 | grep $SESSION_ID | sed "s#.*\"\(https://.*mp4\).*\".*#\1#" | while read mp4_url; do local mp4_filename=$(basename $mp4_url); if [ -e $WWDC_LOCAL_DIR/$mp4_filename ] then echo "> MP4 already existed : $WWDC_LOCAL_DIR/$mp4_filename" >&2; echo "> To resume broken download use curl -C - --connect-timeout 1200 -o $WWDC_LOCAL_DIR/$mp4_filename $mp4_url" >&2; echo " " >&2; else echo "> MP4 Downloading... : $mp4_url" >&2; curl --connect-timeout 120 -o $WWDC_LOCAL_DIR/$mp4_filename $mp4_url fi done } main () { [ ! -d $WWDC_LOCAL_DIR ] && { mkdir $WWDC_LOCAL_DIR; } #Year 2020 change {3\} to {3,5\} curl -s $WWDC_SESSION_PREFIX | grep /videos/play/wwdc$WWDC_YEAR | sed "s#.*/videos/play/wwdc$WWDC_YEAR/\([0-9]\{3,5\}\).*#\1#" | sort | uniq | while read SESSION_ID; do download_mp4_video $SESSION_ID; done } main;






Run this script to rename the video or subtitles to proper title (HD & SD) videos

wwdc2020_rename_title.sh    Select all
#!/bin/sh # @Last Modified by: javacom # @Last Modified time: 2020-10-17 #sd video curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_sd.mp4" \] \&\& mv "wwdc2020_\1_sd.mp4" "wwdc2020_\1_sd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.mp4"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash #hd video curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_hd.mp4" \] \&\& mv "wwdc2020_\1_hd.mp4" "wwdc2020_\1_hd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.mp4"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash #sd video en.srt curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_sd.en.srt" \] \&\& mv "wwdc2020_\1_sd.en.srt" "wwdc2020_\1_sd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.en.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash #hd video en.srt curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_hd.en.srt" \] \&\& mv "wwdc2020_\1_hd.en.srt" "wwdc2020_\1_hd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.en.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash #sd video zh.srt curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_sd.zh.srt" \] \&\& mv "wwdc2020_\1_sd.zh.srt" "wwdc2020_\1_sd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.zh.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash #hd video zh.srt curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_hd.zh.srt" \] \&\& mv "wwdc2020_\1_hd.zh.srt" "wwdc2020_\1_hd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.zh.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash # For WWDC2021 WWDC_YEAR=2021; VIDEO=sd; curl -s https://developer.apple.com/videos/wwdc${WWDC_YEAR}/ | grep -B1 "video-title" | sed -e "s#[[:space:]]\{10,\}<a href=./videos/play/wwdc202./\([0-9]\{3,5\}\).*#\[ -f \"wwdc${WWDC_YEAR}-\1_${VIDEO}.mp4\" \] \&\& mv \"wwdc${WWDC_YEAR}-\1_${VIDEO}.mp4\" \"wwdc${WWDC_YEAR}-\1_sd_#" -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.mp4"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | sed "s#[\/@:’\'\,]#_#g" | /bin/bash






# Example nohup commnad for Linux Download

nohup bash -c 'cd $HOME/Downloads/WWDC/; ./wwdc2020_fetch_mp4.sh' &> nohup.wwdc2020.mp4.out &

nohup bash -c 'cd $HOME/Downloads/WWDC/; ./wwdc2020_fetch_en.srt.sh' &> nohup.wwdc2020.en.srt.out &

# or use screen utility to download
screen -S wwdcdownloadmp4
bash wwdc2020_fetch_mp4.sh



If want to remove special characters in filename "brew install rename" and use this command to rename
rename "s/[\@:’\'\,]//g" *.mp4 *.srt