tag:blogger.com,1999:blog-62459630271529069872024-03-14T04:56:48.907+08:00iPhone Software DevelopmentUnknownnoreply@blogger.comBlogger125125tag:blogger.com,1999:blog-6245963027152906987.post-75515743932172714432022-12-09T14:59:00.005+08:002022-12-09T15:04:55.948+08:00How to install games on Asahi Linux<a href="https://www.facebook.com/groups/758982148551933" target="_blank">https://www.facebook.com/groups/758982148551933</a>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrnBt5L_pvmKIXwv0IWzEN9ybNEln4CE8B6_7LjiV1q-aWzKH6IVuOa_W_hRDAOGszmnK0KH1B0FC-RxEuzmUmSp_FH2_yzV7ldPqXKVlopvdlirj8pFMfdMAVq_p_QGBI3LRJYJmd3plMeWkj3Dx0QAojjZfGL6U1ULj90ZfW17kdMe-hmw7xdEV_/s1283/neverball.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="600" data-original-height="837" data-original-width="1283" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrnBt5L_pvmKIXwv0IWzEN9ybNEln4CE8B6_7LjiV1q-aWzKH6IVuOa_W_hRDAOGszmnK0KH1B0FC-RxEuzmUmSp_FH2_yzV7ldPqXKVlopvdlirj8pFMfdMAVq_p_QGBI3LRJYJmd3plMeWkj3Dx0QAojjZfGL6U1ULj90ZfW17kdMe-hmw7xdEV_/s600/neverball.png"/></a></div>javacomhttp://www.blogger.com/profile/10745838995139328314noreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-65833797284752465712022-08-23T05:27:00.002+08:002022-08-23T05:28:05.824+08:00My Asahi Linux Desktop<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgk8G1ZSk-4B37W4xwc86aHw7wvn-NbksadK3waTawQ6JCOLvmsZbTdvKd8eqYk9YrY29CfQLc0toHDUogDgZ4cXz9JhCI4DF33iCQHHFpEeO0xHk3PendEXHM2lxwwRppqnma0Nq3qQ-nx3RCSwFKOSsTO6P23tCvbjTAoC-FfZprJ4gICqGFNnm0WcQ/s2560/AsahiLinuxDesktop.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1600" data-original-width="2560" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgk8G1ZSk-4B37W4xwc86aHw7wvn-NbksadK3waTawQ6JCOLvmsZbTdvKd8eqYk9YrY29CfQLc0toHDUogDgZ4cXz9JhCI4DF33iCQHHFpEeO0xHk3PendEXHM2lxwwRppqnma0Nq3qQ-nx3RCSwFKOSsTO6P23tCvbjTAoC-FfZprJ4gICqGFNnm0WcQ/s400/AsahiLinuxDesktop.png"/></a></div>
See my facebook <a href="https://www.facebook.com/groups/758982148551933/" target="_blank">https://www.facebook.com/groups/758982148551933/</a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-24139642685579952652022-04-19T15:33:00.016+08:002022-04-19T17:23:29.145+08:00How to use M1 Max / M1 Ultra machine to crack password using brute forceM1 Max and M1 Ultra has lots of GPU power for cracking password. For brute force attach we can use hashcat and John the Ripper.<br/>
<br/>
(1) Installation<br/>
<br/>
(1.1) Install XCode and Command Line Utilities from Apple<br/>
(1.2) Install <a href="https://stackoverflow.com/questions/66666134/how-to-install-homebrew-on-m1-mac" target="_blank">Homebrew</a> <br/>
(1.3) Install John the Ripper and add path by using Terminal command<br/>
<pre>brew install john-jumbo
export PATH=/opt/homebrew/Cellar/john-jumbo/1.9.0/share/john/:$PATH</pre><br/>
(1.4) Install rar and. unrar for testing (as the HomeBrew version might not working.
<pre>cd $(HOME)/Downloads
wget https://www.rarlab.com/rar/rarmacos-arm-611.tar.gz
sudo cp rar/rar /usr/local/bin
sudo cp rar/unrar /usr/local/bin
</pre><br/>
(1.5)Install hashcat from git<br/>
<pre>cd $(HOME)/Downloads
git clone https://github.com/hashcat/hashcat.git
cd hashcat
make
</pre><br/>
(2) Testing <br/>
<pre># create a test.rar file with password of 1234 for testing
cd $(HOME)/Downloads
mkdir -p testhashcat
cd testhascat
echo 'I am testing' > test.txt
rar a -P<span style="color:blue">1234</span> test.rar test.txt
</pre>
Use John the Ripper to generate hash<br/>
% rar2john test.rar<br/>
test.rar:<span style="color:blue">$rar5$16$852481e911dc38c66cec2fbe8e9a825b$15$e1c319e802eec30efab80c6bd7470468$8$f9c3bfde638768a5</span><br/>
<br/>
And copy the hash value in <span style="color:blue">blue color</span> and it will be used again in hashcat <br/>
<br/>
Use hashcat to brute force attack<br/>
<pre>
../hashcat/hashcat -m 13000 -w 3 -a 3 \
'<span style="color:blue">$rar5$16$852481e911dc38c66cec2fbe8e9a825b$15$e1c319e802eec30efab80c6bd7470468$8$f9c3bfde638768a5</span>' \
--increment --increment-min 2 --increment-max 8 '?d?d?d?d?d?d?d?d' --potfile-path=rarfile.pot -o testrar.out.txt
</pre>
<pre>
Explanation of the parameters used in hashcat
-m 13000 means --hash-type RAR5 see here https://hashcat.net/wiki/doku.php?id=example_hashes
-a 3 means attack-mode Brute-force
-w 3 Workload Profiles High
--increment --increment-min 2 --increment-max 8
means increment the guess password with minimum of 2 and maximum of 8
?d means digit as below
Built-in charsets are
?l = abcdefghijklmnopqrstuvwxyz
?u = ABCDEFGHIJKLMNOPQRSTUVWXYZ
?d = 0123456789
?h = 0123456789abcdef
?H = 0123456789ABCDEF
?s = «space»!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
?a = ?l?u?d?s
</pre><br/>
<br/>
<br/>
If finished, the password will be in testrar.out.txt
$rar5$16$852481e911dc38c66cec2fbe8e9a825b$15$e1c319e802eec30efab80c6bd7470468$8$f9c3bfde638768a5:<span style="color:blue">1234</span>
<br/><br/><br/>
Testing of zip file with password<br/>
<pre>
zip -e test.zip test.txt
zip2john test.zip
</pre><br/>
<pre>
% zip2john test.zip
ver 1.0 efh 5455 efh 7875 test.zip/test.txt PKZIP Encr: 2b chk, TS_chk, cmplen=25, decmplen=13, crc=A491CD37
test.zip/test.txt:<span style="color:blue">$pkzip2$1*2*2*0*19*d*a491cd37*0*42*0*19*a491*796a*55a56112ff0d2913127e2502764b2f8044e7975a5a23014084*$/pkzip2$</span>:test.txt:test.zip::test.zip
</pre><br/>
<pre>
../hashcat/hashcat -m 17210 -w 3 -a 3 \
'<span style="color:blue">$pkzip2$1*2*2*0*19*d*a491cd37*0*42*0*19*a491*796a*55a56112ff0d2913127e2502764b2f8044e7975a5a23014084*$/pkzip2$</span>' \
--increment --increment-min 2 --increment-max 8 '?d?d?d?d?d?d?d?d' --potfile-path=zipfile.pot -o testzip.out.txt
% cat testzip.out.txt
$pkzip2$1*2*2*0*19*d*a491cd37*0*42*0*19*a491*796a*55a56112ff0d2913127e2502764b2f8044e7975a5a23014084*$/pkzip2$:<span style="color:blue">1256</span>
</pre>
<br/><br/><br/>
Testing of pdf file with password<br/>
<pre>
% perl pdf2john.pl testpdf.pdf
testpdf.pdf:<span style="color:blue">$pdf$4*4*128*-4*1*16*9d60ea4e1b8444818557a392476b3ab3*32*13f7b5e82e85872782a2795121fc850d00000000000000000000000000000000*32*7f8b9892322afb0dccd7a4259da14c65e9d5009abb26bae6d1708fed9dc60edc</span>
../hashcat/hashcat -m 10500 -w 3 -a 3 \
'<span style="color:blue">$pdf$4*4*128*-4*1*16*9d60ea4e1b8444818557a392476b3ab3*32*13f7b5e82e85872782a2795121fc850d00000000000000000000000000000000*32*7f8b9892322afb0dccd7a4259da14c65e9d5009abb26bae6d1708fed9dc60edc</span>' \
--increment --increment-min 2 --increment-max 8 '?d?d?d?d?d?d?d?d' --potfile-path=pdffile.pot -o testpdf.out.txt
</pre>
<br/>
<pre>
note: hash-type for pdf
10400 PDF 1.1 - 1.3 (Acrobat 2 - 4)
10410 PDF 1.1 - 1.3 (Acrobat 2 - 4), collider #1
10420 PDF 1.1 - 1.3 (Acrobat 2 - 4), collider #2
10500 PDF 1.4 - 1.6 (Acrobat 5 - 8)
10600 PDF 1.7 Level 3 (Acrobat 9)
10700 PDF 1.7 Level 8 (Acrobat 10 - 11)
</pre>
<br/>
Testing of office file with password<br/>
<pre>
% python office2john.py test.docx
test.docx:<span style="color:blue">$office$*2013*100000*256*16*561f4dcaaac333e7c06d150f9ea5aea2*ef4e7b026217124561ecb865b324eac4*e9ef4a859f2c81581db0e27d9ce48e6451b82cd1641941e8adc10dc5600969cb</span>
../hashcat/hashcat -m 9600 -w 3 -a 3 \
'<span style="color:blue">$office$*2013*100000*256*16*561f4dcaaac333e7c06d150f9ea5aea2*ef4e7b026217124561ecb865b324eac4*e9ef4a859f2c81581db0e27d9ce48e6451b82cd1641941e8adc10dc5600969cb</span>' \
--increment --increment-min 2 --increment-max 8 '?h?h?h?h?h?h?h?h' --potfile-path=officefile.pot -o testdocx.out.txt
</pre>
<br/>
<pre>
note: hash-type for office doc
9400 MS Office 2007
9500 MS Office 2010
9600 MS Office 2013
</pre>
<br/><br/><br/><br/>
If you don't have the machine for cracking purpose, you can do this <a href="https://www.youtube.com/watch?v=s4jhe4XsoZI&t=1305s" target="_blank">using cloud GPU</a> for a fee.
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-19120626881742961942022-04-17T15:47:00.029+08:002022-04-18T08:48:07.934+08:00How to install Ubuntu on Intel Macbook Air/Pro with T2 Security ChipCurrently you cannot easily install Linux onto an Apple computer which uses the T2 security chip because the Linux Kernel with the T2 support is not included in any of the currently released distributions as a default kernel. And cannot use the built-in keyboard, trackpad, bluetooth and neither WiFi nor ethernet.<br/>
<br/>
(1) There is a patched Ubuntu ISO for <a href="https://support.apple.com/HT208862" target="_blank">Macbook Air/Pro (2018-2020)</a> with T2 Security Chip hardware here. <a href="https://github.com/marcosfad/mbp-ubuntu/releases" target="_blank">https://github.com/marcosfad/mbp-ubuntu/releases</a><br/>
<br/>
(2) This ISO image requires you to install on the internal SSD and if you are on the lastest macOS Monterey, the internal SSD would be encrypted. It is not recommended to dual booting Macbook with both macOS and Linux unless you have good backups and are prepared to perform a clean install or restore from backup since you will at some point most likely do something which will prevent either OS from booting and possibly risk losing access to all the data in one or both operating systems.<br/>
<br/>
(3) So it is probably better to use USB to boot up for testing out Linux on Macbook first.<br/> You can download this <a href="https://mega.nz/file/zk5CELaR#hyOSXjHLlkxD9eK0ywIRSanIcwEBdqNxPkIJikFEABQ" target="_blank">installed ISO images here (2.64GB)</a> and burn it to an USB stick with 128GB using <a href="https://www.balena.io/etcher/" target="_blank">balenaEtcher for Mac</a>. You don't need to unzip the downloaded zip file as balenaEtcher can read zip file and write to the USB stick directly. The image has an extra 10GB exFAT partition, so that data can be exchanged offline with PC or Mac.<br/>
<br/>
(4) The Macbook should be Disable Secure Boot and Set allowed boot media based on the instructions here <a href="https://support.apple.com/HT208330" target="_blank">https://support.apple.com/HT208330</a><br/>
<br/>
(5) Then plug in the USB and restart the Macbook and press <span style="color:blue">option</span> key while bootup, then select EFI boot to startup Ubuntu. The installed version of this Ubuntu image, username and password are all <span style="color:blue">ubuntu</span><br/>
<br/>
(6) This patched Ubuntu ISO should worked for the butterfly keyboard, function keys, touch pad, bluetooth, display and audio. But the microphone might not be working. If the wifi is not working initially, the fix is here using the Terminal (control-option-T) commands<br/>
<br/><br/>
<span style="font-size: large;"><a href="https://github.com/marcosfad/mbp-ubuntu/issues/67" target="_blank">Fix wifi issue of ubuntu-mbp image</a></span><br />
<pre>sudo dpkg -i /usr/src/iso-firmware.deb
#edit /etc/NetworkManager/NetworkManager.conf as below
sudo editor /etc/NetworkManager/NetworkManager.conf
</pre>
<br /><dl class="codebox">
<dt>NetworkManager.conf <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>[main]
plugins=ifupdown,keyfile
[ifupdown]
managed=false
[device]
wifi.scan-rand-mac-address=no
</code></dl>
<pre>
#edit /etc/NetworkManager/conf.d/wifi_backend.conf and add # sign to stop iwd as below
sudo editor /etc/NetworkManager/conf.d/wifi_backend.conf
</pre>
<br /><dl class="codebox">
<dt>wifi_backend.conf <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#[device]
#wifi.backend=iwd
</code></dl>
<br /><br />
Then
<pre>
# restart NetworkManager
sudo systemctl restart NetworkManager
sudo modprobe brcmfmac
</pre>
Then reboot Ubuntu...<br /><br />
<br /><br />
<span style="font-size: large;">Ubuntu Short Cut Key on Mac</span><br />
<u>Move window to left or right</u><br />
command + arrow (e.g. left or right)<br />
<br />
<u>Switch desktop</u><br />
control + option + arrow (e.g. up or down)<br />
<br />
<u>Move current window to desktop command + shift + page up / page down</u><br />
command + shift + fn + arrow (e.g. up or down)<br />
<br />
<u>Show all desktops</u><br />
command<br />
<br />
<u>Change window</u><br />
command + tab<br />
<br />
<br /><br />
<span style="font-size: large;">Further Installation of Waydroid and sideload of Android apk on Ubuntu</span><br />
<br /><dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>sudo apt update
sudo apt install python3-pip lxc curl wget -y
sudo pip install pyclip
export DISTRO="focal" && sudo curl https://repo.waydro.id/waydroid.gpg --output /usr/share/keyrings/waydroid.gpg && echo "deb [signed-by=/usr/share/keyrings/waydroid.gpg] https://repo.waydro.id/ $DISTRO main" > ~/waydroid.list && sudo mv ~/waydroid.list /etc/apt/sources.list.d/waydroid.list
sudo apt update
sudo apt install dkms -y
# if old Linux Kernel get this https://github.com/anbox/anbox-modules
# With Linux Kernel 5.17 (that is >=5.7), download and build Linux modules from here https://github.com/choff/anbox-modules
wget https://github.com/choff/anbox-modules/archive/refs/heads/master.zip
unzip master.zip
cd anbox-modules-master/
sudo ./INSTALL.sh
# check Linux kernel modules installed anbox-ashmem & anbox-binder
dkms status
# install waydroid
sudo apt install waydroid -y
</code></dl>
<pre>
#sudo editor /etc/resolv.conf as
nameserver 8.8.8.8
nameserver 8.8.4.4
</pre>
<pre>
#sudo editor /etc/gbinder.d/anbox.conf as
[Protocol]
/dev/binder = aidl2
/dev/vndbinder = aidl2
/dev/hwbinder = hidl
[ServiceManager]
/dev/binder = aidl2
/dev/vndbinder = aidl2
/dev/hwbinder = hidl
</pre>
<br /><br />
<pre>
Logout Ubuntu (PC keyboard: control-alt-delete / Mac keyboard: control-option-fn-delete)
And login again with "Ubuntu on Wayland" (setting at the bottom right hand corner)
</pre>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhZYiB-cuht9jni7uLXkwgsfTQrRwfRNY8nud3hdagU5V_grxN8EGgi71dXz2KTHutxFiBkLT7CC6T4rK_57sFkrbpXDhXVW347uQlXxHiz0dj0K7jCjM1J_YW4fiY_UctCEiWg_5lSK-7OyCPIdrXxM65t-6MMAJ0zIHwlbbj1fYSO4YxMcLn5pXcAg/s339/waylands.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="320" data-original-height="339" data-original-width="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhZYiB-cuht9jni7uLXkwgsfTQrRwfRNY8nud3hdagU5V_grxN8EGgi71dXz2KTHutxFiBkLT7CC6T4rK_57sFkrbpXDhXVW347uQlXxHiz0dj0K7jCjM1J_YW4fiY_UctCEiWg_5lSK-7OyCPIdrXxM65t-6MMAJ0zIHwlbbj1fYSO4YxMcLn5pXcAg/s320/waylands.png"/></a></div>
<br/>
<br/>
<pre>
# Download waydroid image (752.66 MB) then init
sudo waydroid init
sudo systemctl start waydroid-container
waydroid session start
# sideload apk (x86_64 apk only)
waydroid app install myapplication.apk
</pre>
<br/>
<b><u>Color correction of WayDroid</u></b><br/>
Go to Settings App -> Accessibility -> Color inversion (on)<br/>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5HWtyVK2rzQF6Lvb3ZwkKe5Vy0rY05t4GOtCM13Q_3TGPN5o__KPdu5GDG6IceCekus32LznZaQFInSYjfGSktroxs72w2i4JG0wcy1vKmZmOrFstE18Lp2WXwHkgO66ddfjVkPVolwjqHonst2hJah_a5k-XgPN2WhiNj2Bws-1bOZbQA7Da62Eerw/s1090/colorinversion.jpeg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="320" data-original-height="449" data-original-width="1090" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5HWtyVK2rzQF6Lvb3ZwkKe5Vy0rY05t4GOtCM13Q_3TGPN5o__KPdu5GDG6IceCekus32LznZaQFInSYjfGSktroxs72w2i4JG0wcy1vKmZmOrFstE18Lp2WXwHkgO66ddfjVkPVolwjqHonst2hJah_a5k-XgPN2WhiNj2Bws-1bOZbQA7Da62Eerw/s320/colorinversion.jpeg"/></a></div>
<br/>
<span style="font-size: large;">If you want to install Ubuntu Mate Desktop, the terminal command is</span><br/>
<pre>
sudo apt-get update
sudo apt-get install ubuntu-mate-desktop
</pre>
<!--
bash -c "DISPLAY=:0 mate-screenshot -a"
-->
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-42428336863870114052022-04-09T09:27:00.022+08:002022-04-13T07:20:03.806+08:00How to erase Mac SSD for downgrade macOS to older versionIt used to be using created bootable USB stick in order to downgrade macOS to older version, <a href="https://support.apple.com/HT201372" target="_blank">as in here</a>.<br/>
<br/>
But with the new macOS Monterey and with the T2 Security Chip of modem Mac, your Macintosh SSD will be encrypted and for Intel Mac, the bootcamp partition will also encrypted.<br/>
<br/>
As a result, you cannot install macOS from USB stick to the encrptyed SSD, even if you have enabled boot from media in the Startup Security Utility.<br/>
<br/>
The downgrade path will now require you to <a href="https://support.apple.com/HT201065" target="_blank">erase all content and settings</a> first.<br/>If you haven't do so, you have to do Erase Mac as below before installing any macOS version.
<br/>
In addition, if you want to downgrade macOS, you have to do<br/>
<br/>
Step (1) Backup all your important data and be sure you have a good Internet connection for the remaining steps<br/>
<br/>
Step (2) <span style="color:Red">Erase Mac</span>, <a href="https://support.apple.com/guide/mac-help/mh27903/mac" target="_blank">reboot Mac and select Erase Mac from the menu</a>. This will wipe and remove all the whole Macintosh SSD, including the encrypted Macintosh partition and bootcamp partition if any, you might need to authenticate by entering the admin password of your existing MacOS to do this. You can use Internet Recovery to the original macOS version (Option-Shift-Command-R), or else it will default to the latest macOS Monterey (Option-Command-R).<br/>
<br/>
Step (3) <span style="color:Red">Activate Mac</span>, reboot Mac and enter Recovery by pressing Command-R and activate the Mac.<br/>
<br/>
Step (4) <span style="color:Red">Internet Recovery</span>, reboot Mac and enter Recovery by pressing Command-R and use <a href="https://support.apple.com/guide/mac-help/mchlp1599/12.0/mac/12.0" target="_blank">Internet Recovery to choose install original version of macOS</a>.<br/>
<br/>
Step (5) <span style="color:Red">Enable boot from media</span>, After successful installation of the old macOS version, <a href="https://support.apple.com/HT201372" target="_blank">create bootable USB stick of your required macOS version based on this guide from Apple</a>. Reboot Mac and <a href="https://support.apple.com/HT208198" target="_blank">enable boot from media from in the Startup Security Utility</a>.<br/>
<br/>
Step (5) <span style="color:Red">Install macOS from USB stick</span>, reboot again and press Option to choose the USB stick to install from macOS.<br/><br/>
<br/><br/><br/>
<!--
https://igotoffer.com/apple/category/macbook
https://github.com/marcosfad/mbp-ubuntu/releases/tag/v20.04-5.17.1-1
https://askubuntu.com/questions/55868/installing-broadcom-wireless-drivers
https://gist.github.com/gbrow004/096f845c8fe8d03ef9009fbb87b781a4
https://www.xmodulo.com/onscreen-virtual-keyboard-linux.html
https://wiki.debian.org/MacBook/Wireless
-->
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-61330705006077796992021-06-10T11:42:00.006+08:002021-06-10T17:55:41.434+08:00How to download WWDC2021 videos and subtitlesThe scripts for previous WWDC are here
https://iphonesdkdev.blogspot.com/2020/10/how-to-download-wwdc2020-videos-and.html<br /><br />
https://iphonesdkdev.blogspot.com/2017/07/how-to-fetch-wwdc-2017-video-subtitle.html<br /><br />
<span style="font-size: large;">Create and Run this script wwdc2021_fetch_srt.sh to fetch WWDC2021 subtitle</span><br />
<br />
<br />
<dl class="codebox">
<dt>wwdc2021_fetch_en.srt.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/bin/bash
# @Last Modified by: javacom
# @Last Modified time: 2021-06-10
WWDC_YEAR=2021;
#
WWDC_SESSION_PREFIX=https://developer.apple.com/videos/play/wwdc$WWDC_YEAR;
WWDC_LOCAL_DIR=$(basename $WWDC_SESSION_PREFIX);
detect_video_m3u8 () {
local session_url=$WWDC_SESSION_PREFIX/$SESSION_ID/;
local session_html=$(curl -s $session_url);
local video_url=$(echo "$session_html" | grep .m3u8 | grep $SESSION_ID | head -n1 | sed "s#.*\"\(https://.*m3u8\)\".*#\1#");
echo "$session_html" | grep .mp4 | grep $SESSION_ID | sed "s#.*\"\(https://.*mp4\).*\".*#\1#" | while read mp4_url; do
local mp4_filename=$(basename $mp4_url | cut -d. -f1);
local srt_filename=$mp4_filename.en.srt;
# local srt_filename=$mp4_filename.zh.srt;
echo "> Subtitle local: $WWDC_LOCAL_DIR/$srt_filename" >&2;
> $WWDC_LOCAL_DIR/$srt_filename;
done
echo "$video_url";
echo "> Video: $video_url" >&2;
}
detect_subtitle_m3u8 () {
local video_url=$1;
# en subtitle
local subtitle_uri=$(curl -s $video_url | grep "LANGUAGE=\"eng\"" | grep "TYPE=SUBTITLES" | sed "s#.*URI=\"\(.*\)\"#\1#" | sed "s#.FORCED=NO##");
# zh subtitle
#local subtitle_uri=$(curl -s $video_url | grep "LANGUAGE=\"zh\"" | sed "s#.*URI=\"\(.*\)\"#\1#");
local subtitle_url=$subtitle_uri;
[[ "$subtitle_uri" != http* ]] && {
subtitle_url=$(dirname $video_url)/$subtitle_uri;
}
echo "$subtitle_url";
echo "> Subtitle: $subtitle_url" >&2;
}
download_subtitle_contents () {
local subtitle_url=$1;
echo "> Downloading... "
local subtitle_base_url=$(dirname $subtitle_url);
curl -s $subtitle_url | grep "webvtt" | while read webvtt; do
local subtitle_webvtt=$subtitle_base_url/$webvtt;
#echo "- get $subtitle_webvtt";
local subtitle_content=$(curl -s $subtitle_webvtt);
# en subtitle
ls $WWDC_LOCAL_DIR/"wwdc$WWDC_YEAR"-"$SESSION_ID"*.en.srt | while read srt_file; do
# zh subtitle
# ls $WWDC_LOCAL_DIR/"wwdc$WWDC_YEAR"-"$SESSION_ID"*.zh.srt | while read srt_file; do
echo "$subtitle_content" >> $srt_file;
done
done
}
main () {
[ ! -d $WWDC_LOCAL_DIR ] && {
mkdir $WWDC_LOCAL_DIR;
}
#Year 2020/2021 change {3\} to {3,5\}
curl -s $WWDC_SESSION_PREFIX | grep /videos/play/wwdc$WWDC_YEAR | sed "s#.*/videos/play/wwdc$WWDC_YEAR/\([0-9]\{3,5\}\).*#\1#" | sort | uniq | while read SESSION_ID; do
#echo "SESSION_ID is" $SESSION_ID
local video_url=$(detect_video_m3u8 $SESSION_ID);
local subtitle_url=$(detect_subtitle_m3u8 $video_url);
download_subtitle_contents $subtitle_url;
done
}
main;
</code></dl>
<br /><br /><br /><br />
<span style="font-size: large;">Run this shell script to format as SRT subtitle</span><br />
<br />
<dl class="codebox">
<dt>shellscript.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>WWDC_YEAR=2021;
#
cd wwdc$WWDC_YEAR
mkdir -p sd
mkdir -p hd
for i in *_sd.??.srt; do sed -e '/WEBVTT/d;/X-TIMESTAMP/d;s/align.middle line.*$//;' $i | awk '/^[0-9]{2}:[0-9]{2}:/ {seen[$0]++; skipduplicated=0} {if (seen[$0]>1) skipduplicated=1; if (!skipduplicated) print $0}' | awk -v RS="" '{gsub("\n", "-Z"); print}' | awk '$0 !~/^WEB/ {print $0}' | uniq | awk '{printf "\n%s-Z%s", NR,$0 }' | awk -v ORS="\n\n" '{gsub("-Z", "\n"); print}' | sed -e 's/.A:middle$//g;s/&gt;/>/g;s/&lt;/</g;1,2d;' > sd/$i; done
for i in *_hd.??.srt; do sed -e '/WEBVTT/d;/X-TIMESTAMP/d;s/align.middle line.*$//;' $i | awk '/^[0-9]{2}:[0-9]{2}:/ {seen[$0]++; skipduplicated=0} {if (seen[$0]>1) skipduplicated=1; if (!skipduplicated) print $0}' | awk -v RS="" '{gsub("\n", "-Z"); print}' | awk '$0 !~/^WEB/ {print $0}' | uniq | awk '{printf "\n%s-Z%s", NR,$0 }' | awk -v ORS="\n\n" '{gsub("-Z", "\n"); print}' | sed -e 's/.A:middle$//g;s/&gt;/>/g;s/&lt;/</g;1,2d;' > hd/$i; done
</code></dl>
<br /><br /><br /><br />
<span style="font-size: large;">Run this script wwdc2021_fetch_mp4.sh to download all mp4 (HD and SD) videos (also works for 2021)</span><br />
<br />
<dl class="codebox">
<dt>wwdc2021_fetch_mp4.sh  <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/bin/bash
# @Last Modified by: javacom
# @Last Modified time: 2021-06-10
WWDC_YEAR=2021;
#
WWDC_SESSION_PREFIX=https://developer.apple.com/videos/play/wwdc$WWDC_YEAR;
WWDC_LOCAL_DIR=$(basename $WWDC_SESSION_PREFIX);
download_mp4_video () {
local session_url=$WWDC_SESSION_PREFIX/$SESSION_ID/;
local session_html=$(curl -s $session_url);
local video_url=$(echo "$session_html" | grep .m3u8 | grep $SESSION_ID | head -n1 | sed "s#.*\"\(https://.*m3u8\)\".*#\1#");
echo "$session_html" | grep .mp4 | grep $SESSION_ID | sed "s#.*\"\(https://.*mp4\).*\".*#\1#" | while read mp4_url; do
local mp4_filename=$(basename $mp4_url);
if [ -e $WWDC_LOCAL_DIR/$mp4_filename ]
then
echo "> MP4 already existed : $WWDC_LOCAL_DIR/$mp4_filename" >&2;
echo "> To resume broken download use curl -C - --connect-timeout 1200 -o $WWDC_LOCAL_DIR/$mp4_filename $mp4_url" >&2;
echo " " >&2;
else
echo "> MP4 Downloading... : $mp4_url" >&2;
curl --connect-timeout 120 -o $WWDC_LOCAL_DIR/$mp4_filename $mp4_url
fi
done
}
main () {
[ ! -d $WWDC_LOCAL_DIR ] && {
mkdir $WWDC_LOCAL_DIR;
}
#Year 2020/2021 change {3\} to {3,5\}
curl -s $WWDC_SESSION_PREFIX | grep /videos/play/wwdc$WWDC_YEAR | sed "s#.*/videos/play/wwdc$WWDC_YEAR/\([0-9]\{3,5\}\).*#\1#" | sort | uniq | while read SESSION_ID; do
download_mp4_video $SESSION_ID;
done
}
main;
</code></dl><br /><br />
<br /><br /><br /><br />
<span style="font-size: large;">Run this script to rename the video or subtitles to proper title (HD & SD) videos</span><br />
<br />
<dl class="codebox">
<dt>wwdc2021_rename_title.sh  <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/bin/sh
# @Last Modified by: javacom
# @Last Modified time: 2021-6-10
# For WWDC2021 video files, VIDEO=sd or VIDEO=hd
WWDC_YEAR=2021; VIDEO=sd; curl -s https://developer.apple.com/videos/wwdc${WWDC_YEAR}/ | grep -B1 "video-title" | sed -e "s#[[:space:]]\{10,\}<a href=./videos/play/wwdc202./\([0-9]\{3,5\}\).*#\[ -f \"wwdc${WWDC_YEAR}-\1_${VIDEO}.mp4\" \] \&\& mv \"wwdc${WWDC_YEAR}-\1_${VIDEO}.mp4\" \"wwdc${WWDC_YEAR}-\1_${VIDEO}_#" -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.mp4"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | sed "s#[\/@:’\'\,]#_#g" | /bin/bash
# For WWDC2021 English subtitle files, VIDEO=sd or VIDEO=hd
WWDC_YEAR=2021; VIDEO=sd; curl -s https://developer.apple.com/videos/wwdc${WWDC_YEAR}/ | grep -B1 "video-title" | sed -e "s#[[:space:]]\{10,\}<a href=./videos/play/wwdc202./\([0-9]\{3,5\}\).*#\[ -f \"wwdc${WWDC_YEAR}-\1_${VIDEO}.en.srt\" \] \&\& mv \"wwdc${WWDC_YEAR}-\1_${VIDEO}.en.srt\" \"wwdc${WWDC_YEAR}-\1_${VIDEO}_#" -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.en.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | sed "s#[\/@:’\'\,]#_#g" | /bin/bash
</code></dl><br /><br />
<br /><br /><br /><br />
# Example nohup commnad for Linux Download<br/><br/>
nohup bash -c 'cd $HOME/Downloads/WWDC/; ./wwdc2021_fetch_mp4.sh' &> nohup.wwdc2021.mp4.out &<br/><br/>
nohup bash -c 'cd $HOME/Downloads/WWDC/; ./wwdc2021_fetch_en.srt.sh' &> nohup.wwdc2021.en.srt.out &<br/><br/>
# or use screen utility to download<br/>
screen -S wwdcdownloadmp4<br/>
bash wwdc2021_fetch_mp4.sh<br/><br/>
<br/><br/>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-29394558882277241482021-04-08T21:54:00.013+08:002021-04-10T16:07:47.705+08:00How to write subrountine call in macOS assembly code for x86_64 and arm64(1) Demo the subrountine call in macOS assembly code for x86_64 and arm64. For M1 Mac with Rosetta 2 installed, it can compile and run x86_64 and arm64 binary after installation of Xcode<br/>
<dl class="codebox">
<dt>callfactorial.c <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>/*
* An application that illustrates calling the factorial function defined elsewhere.
*/
#include <stdio.h>
#include <inttypes.h>
uint64_t factorial(unsigned n);
int main() {
for (unsigned i = 0; i < 20; i++) {
printf("factorial(%2u) = %llu\n", i, factorial(i));
}
}
</code></dl>
<br /><br />
<dl class="codebox">
<dt>factorial.s <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># ----------------------------------------------------------------------
# A 64-bit recursive implementation of the function
#
# uint64_t factorial(unsigned n)
#
# implemented recursively with x86_64 and arm64 assembly code
# -----------------------------------------------------------------------
.globl _factorial
.text
#ifdef __arm64__
.align 4
#endif
_factorial:
#ifdef __x86_64__
cmp $1, %rdi # n <= 1?
jnbe L1 # if not, go do a recursive call
mov $1, %rax # otherwise return 1
ret
#endif
#ifdef __arm64__
cmp x8, #1 //# n > 1?
b.gt L1 //# if yes, go do a recursive call
mov x0, #1 //# otherwise return 1
ret
#endif
L1:
#ifdef __x86_64__
push %rdi # save n on stack (also aligns %rsp!)
dec %rdi # n-1
call _factorial # factorial(n-1), result goes in %rax
pop %rdi # restore n
imul %rdi, %rax # n * factorial(n-1), stored in %rax
ret
#endif
#ifdef __arm64__
STP X8, LR, [SP, #-16]! //# push x8 and LR(x30)
// LR is to return from subroutine
subs x8, x8, #1 //# n-1
bl _factorial //# factorial(n-1), result goes in x0
LDP X8, LR, [SP], #16 //# pop x8 and LR(x30)
mul x0, x0, x8 //# n * factorial(n-1), stored in x0
ret
#endif
</code></dl>
<br /><br />
(2) To compile with -g and codesign the program so as to debug in lldb under macOS.<br/>
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># To compile and codesign x86_64 version
clang factorial.s callfactorial.c -g -o callfactorial_arm64 -arch x86_64 && codesign --entitlement entitlements --force -s - callfactorial_x86_64
# To compile and codesign arm64 version
clang factorial.s callfactorial.c -g -o callfactorial_arm64 -arch arm64 && codesign --entitlement entitlements --force -s - callfactorial_arm64
</code></dl>
<br /><br />
(3) To debug using lldb<br/>
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>lldb callfactorial_x86_64
# or
lldb callfactorial_arm64
# lldb debug session for arm64 - useful commands
(lldb) breakpoint set --name main --name factorial
(lldb) breakpoint list
(lldb) run
(lldb) step
(lldb) po i
(lldb) reg read x0 x8 lr pc
(lldb) reg read -f t cpsr
# lldb debug session for x86_64 - useful commands
(lldb) reg read -f d rax rdi rflags
(lldb) reg read -f t rflags
# print the address value in the stackpointer for x86_64
(lldb) p *(int **)$sp
# hint: to search lldb command history use ctrl-r
</code></dl>
<br /><br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-42063013305650418612021-04-03T18:19:00.014+08:002021-04-09T14:39:33.315+08:00How to use inline assembly language for M1 Mac for x86_64 and arm64(1) Demo the inline assembly code for x86_64 and arm64 <br/>
<dl class="codebox">
<dt>add-inline.c <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#include <stdio.h>
// compile with command line
// clang add-inline.c -o add-inline_x86_64 -arch x86_64
// clang add-inline.c -o add-inline_arm64 -arch arm64
// disassemble with
// otool -otV add-inline_x86_64
// otool -otV add-inline_arm64
int main(int argc, const char * argv[]) {
int a = 10;
int b = 25;
int ans = 0;
#ifdef __x86_64__
__asm__( "add %2,%1;\n" // %1 += %2 add source to destination
"mov %1,%0;\n" // move data from %1 to %0
: "=r"(ans)
: "r"(a), "r"(b)
: );
#endif
#ifdef __arm64__
__asm__( "add %w0,%w1,%w2;\n" // load %w0 = %w1 + %w2
: "=r"(ans)
: "r"(a), "r"(b)
: );
#endif
printf("The answer is %d\n",ans);
return 0;
}
</code></dl>
<br /><br />
(2) Demo the assembly source code as a separate function for x86_64 and arm64 <br/>
<dl class="codebox">
<dt>add-main.c <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#include <stdio.h>
// compile with command line
// clang add-main.c add.s -o add-main_x86_64 -arch x86_64
// clang add-main.c add.s -o add-main_arm64 -arch arm64
#include <stdio.h>
int add(int x, int y);
int main(int argc, const char * argv[]) {
int ans = add(15,40);
printf("The answer is %d\n",ans);
return 0;
}
</code></dl><br/><br/>
<dl class="codebox">
<dt>add.s <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>.text
.globl _add
.align 2
_add:
#ifdef __x86_64__
add %esi,%edi // %edi += %esi, source is the first
mov %edi,%eax // move data from %edi to %eax
// x86_64 calling convention
// rdi, rsi, rdx, rcx, r8, r9
// The 32-bit general purpose registers are edi, esi, edx, ecx, r8d, r9d instead.
// The 16-bit general purpose registers are di, si, dx, cx, r8w, r9w instead.
// The syscall number is placed in rax
// see https://sigsegv.pl/osx-bsd-syscalls/
// Return value is in rax
#endif
#ifdef __arm64__
add w0,w0,w1 // load w0 with w0+w1, destination is the first
// calling convention for arm64
// x0,x1,x2,x3,x4,x5,x6,x7 or r0 to r7
// The 32-bit general purpose registers are w0-w7 instead.
// The syscall number is placed in r8
// Return value is in x0
// see https://wiki.cdot.senecacollege.ca/wiki/Syscalls
#endif
ret
</code></dl><br/><br/>
<br /><br />
<dl class="codebox">
<dt>entitlements <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code><?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.get-task-allow</key>
<true/>
</dict>
</plist>
</code></dl><br/><br/>
(3) Compile with<br />
clang add-inline.c -o add-inline_x86_64 -arch x86_64<br />
clang add-inline.c -o add-inline_arm64 -arch arm64<br />
clang add-main.c add.s -o add-main_x86_64 -arch x86_64<br />
clang add-main.c add.s -o add-main_arm64 -arch arm64<br />
<br /><br />
(4) Disassemble with (e.g.)<br />
otool -otV add-inline_arm64
<br /><br />
(5) Install Rosetta 2 on M1 to run the x86_64 version (e.g.)<br />
./add-main_x86_64
<br /><br />
(6) codesign and debug with lldb (e.g.)<br />
clang -g -o add-inline_x86_64 add-inline.c -arch x86_64<br />
codesign --entitlement entitlements --force -s - add-inline_x86_64<br />
lldb add-inline_x86_64<br />
(lldb) breakpoint set --file add-inline.c --line 7<br />
<br /><br />
(7) floating point example for x86_64 and arm64<br />
compile with <br />
clang sum.s callsum.c -o callsum_x86_64 -arch x86_64<br />
clang sum.s callsum.c -o callsum_arm64 -arch arm64<br />
Debug and codesign similar to above example<br />
<dl class="codebox">
<dt>callsum.c <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>/*
* callsum.c
*
* Illustrates how to call the sum.s function wrote in assembly language.
*/
// clang sum.s callsum.c -o callsum_x86_64 -arch x86_64
// clang sum.s callsum.c -o callsum_arm64 -arch arm64
#include <stdio.h>
double sum(double[], unsigned);
int main() {
double test[] = {
40.5, 26.7, 21.9, 1.5, -40.5, -23.4
};
printf("%20.7f\n", sum(test, 6));
printf("%20.7f\n", sum(test, 2));
printf("%20.7f\n", sum(test, 0));
printf("%20.7f\n", sum(test, 3));
return 0;
}
</code></dl><br/><br/>
<dl class="codebox">
<dt>sum.s <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># -----------------------------------------------------------------------
# A 64-bit function that returns the sum of the elements in a
# floating-point array for x86_64 and arm64. The function has prototype:
#
# double sum(double[] array, unsigned length)
# -----------------------------------------------------------------------
.global _sum
.text
.align 2
_sum:
#ifdef __x86_64__
xorpd %xmm0, %xmm0 // initialize the sum to 0
// floats are passed in xmm0
cmp $0, %rsi // special case for length = 0
je done
#endif
#ifdef __arm64__
movi d0, #0 // initialize the sum to 0
// floats are passed in s0-7 and doubles in the d0-7 registers.
cmp x1, #0 // special case for length = 0
b.eq done
#endif
next:
#ifdef __x86_64__
addsd (%rdi), %xmm0 // add in the current array element, return floating point value in xmm0
add $8, %rdi // move to next array element
dec %rsi // count down
jnz next // if not done counting, continue
#endif
#ifdef __arm64__
ldr d16, [x0] // load the float into d16
// floats in s0-7 and doubles in the d0-7 registers.
fadd d0, d0, d16 // add in the current array element, return floating point value in d0
add x0, x0, #8 // move to next array element
subs x1, x1, #1 // count down
cbnz w1, next // if not done counting, continue
#endif
done:
ret
</code></dl><br/><br/>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-8693885104178947292020-12-23T11:03:00.014+08:002020-12-23T15:20:38.449+08:00How to install Parallel Desktop and Windows 10 on Mac M1Mac M1 can install Parallel Desktop Preview and run x64 Windows app <br /><br />
1) Install PD for M1 Mac from <a href="https://www.parallels.com/blogs/parallels-desktop-apple-silicon-mac/" target="_blank">https://www.parallels.com/blogs/parallels-desktop-apple-silicon-mac/</a> and click Try Technical Preview. Need to login and register. <br/> <br />
<a href="https://mega.nz/file/25pUHBgK#Sp6ZvLkFSzjDYKUUTIFIV0VyS5Bp5tLQNiCPljBGvR8" target="_blank">Direct download link</a><br />
MD5 checksum 5fc71f89fcc67ed13bac1995604a962f<br />
Activation key You have to register and login to activate the product<br /><br />
2) Install Windows 10 Client ARM64 Insider Preview download from <a href="https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewARM64" target="_blank">https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewARM64</a>. Need to login and register as Insider Preview Program.<br />
<a href="https://mega.nz/file/i5wTnKpL#gYKS4OTwngr4pxK8diyVBwxq9dIKvI_OS9lltuS-T5A" target="_blank">Direct download link</a>><br />><br />
3) Update Win10 ARM to version 21277 which can install and run x64 Windows programs.<br />
Have to login Windows Insider Program and set to "Dev Channel" to recieve preview updates.<br />
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-tyoLecDhWFI/X-LtJYWNxWI/AAAAAAAA3H0/RBvLdyv5uDEHGA-3px0Tt3wyoGgNIuZtgCLcBGAsYHQ/s749/windowsinsider.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="659" data-original-width="749" src="https://1.bp.blogspot.com/-tyoLecDhWFI/X-LtJYWNxWI/AAAAAAAA3H0/RBvLdyv5uDEHGA-3px0Tt3wyoGgNIuZtgCLcBGAsYHQ/s400/windowsinsider.png"/></a></div>
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-OLtGuUymUZE/X-LvvTSZlFI/AAAAAAAA3IM/4lgnZNzca7AnXiNvGWFZaoV3soIzG4sjgCLcBGAsYHQ/s672/21227.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="207" data-original-width="672" src="https://1.bp.blogspot.com/-OLtGuUymUZE/X-LvvTSZlFI/AAAAAAAA3IM/4lgnZNzca7AnXiNvGWFZaoV3soIzG4sjgCLcBGAsYHQ/s400/21227.png"/></a></div>
<br /><br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-2668972846416013352020-12-20T18:46:00.035+08:002021-05-02T21:29:24.561+08:00Buid docker image for M1 Mac with development tools and vnc(1) With the release of docker preview for Mac M1, it is now possible to build some arm64 images.<br/>
Docker M1 preview<br/>
<a href="https://desktop.docker.com/mac/m1preview/Docker-AppleSilicon-Preview7.dmg" target="_blank">https://desktop.docker.com/mac/m1preview/Docker-AppleSilicon-Preview7.dmg</a><br/><br/>
(2) Install the above preview and then use terminal the tools. The build script is based on Android SDK docker image for x64 and modify it for arm64. Moreover, swift 5.3.1, clang 10.0.0, kotlin 1.4.21, gradle 6.7.1, gcc 9.3.0, java 11, firefox, visual studio code and codeblocks are added for development purpose.<br/><br/>
(3) Shell script run in Mac
docker info # check if docker is installed properly in Mac M1<br/>
mkdir -p ${HOME}/android_sdk_vnc<br/>
cd ${HOME}/android_sdk_vnc<br/>
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/sshd-banner<br/>
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/supervisord.conf<br/>
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/vnc/supervisord_vncserver.conf<br/>
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/vnc/vncpass.sh<br/>
curl -O https://raw.githubusercontent.com/thyrlian/AndroidSDK/master/android-sdk/vnc/watchdog.sh<br/><br/>
(4) Create the following Dockfile
<dl class="codebox">
<dt>Dockerfile <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>cat >Dockerfile <<'HEREEOF'
# ====================================================================== #
# Android SDK Docker Image
# ====================================================================== #
# Base image
# ---------------------------------------------------------------------- #
FROM arm64v8/ubuntu:20.04
# install essential tools
RUN apt-get update && \
apt-get dist-upgrade -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends git wget wget2 zip unzip && \
apt-get install -y build-essential vim
# install Android SDK
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y android-sdk && \
rm -f /usr/lib/android-sdk/build-tools/27.0.1
# install Android SDK cmdline tools
ARG ANDROID_SDK_VERSION=6858069
ENV ANDROID_SDK_ROOT /usr/lib/android-sdk
RUN wget2 -q https://dl.google.com/android/repository/commandlinetools-linux-${ANDROID_SDK_VERSION}_latest.zip && \
unzip commandlinetools*linux*${ANDROID_SDK_VERSION}*.zip && \
mv cmdline-tools tools && \
mkdir -p ${ANDROID_SDK_ROOT}/cmdline-tools && \
mv tools ${ANDROID_SDK_ROOT}/cmdline-tools/ && \
(cd /usr/lib/android-sdk/cmdline-tools; ln -s tools latest) && \
rm -f commandlinetools*linux*.zip
# download and install Gradle
ARG GRADLE_VERSION=6.7.1
ARG GRADLE_DIST=bin
RUN wget2 -q https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-${GRADLE_DIST}.zip && \
rm -fr /usr/share/gradle && \
unzip gradle*.zip -d /usr/share && \
rm gradle*.zip && \
mv /usr/share/gradle-${GRADLE_VERSION} /usr/share/gradle
# download and install Kotlin compiler
ARG KOTLIN_VERSION=1.4.10
RUN wget2 -q https://github.com/JetBrains/kotlin/releases/download/v${KOTLIN_VERSION}/kotlin-compiler-${KOTLIN_VERSION}.zip && \
unzip *kotlin*.zip && \
mv kotlinc /usr/share && \
rm *kotlin*.zip
# install swift5 compiler
RUN wget -qO- https://packagecloud.io/install/repositories/swift-arm/release/script.deb.sh | bash
RUN apt install -y swiftlang
# setup adb server
EXPOSE 5037
# set the environment variables
ARG JDK_VERSION=1.11.0
ENV JAVA_HOME /usr/lib/jvm/java-${JDK_VERSION}-openjdk-arm64
ENV GRADLE_HOME /usr/share/gradle
ENV KOTLIN_HOME /usr/share/kotlinc
ENV ANDROID_SDK_ROOT /usr/lib/android-sdk
ENV ANDROID_HOME /usr/lib/android-sdk
ENV ANDROID_TOOLS /usr/lib/android-sdk/tools
ENV ANDROID_PLATFORMS /usr/lib/android-sdk/platforms
ENV ANDROID_PLATFORM_TOOLS /usr/lib/android-sdk/platform-tools
ENV ANDROID_BUILD_TOOLS /usr/lib/android-sdk/build-tools
ENV ANDROID_CMDLINE_TOOLS /usr/lib/android-sdk/cmdline-tools/tools
ENV PATH ${PATH}:${GRADLE_HOME}/bin:${KOTLIN_HOME}/bin:${ANDROID_BUILD_TOOLS}/debian:${ANDROID_TOOLS}/bin:${ANDROID_PLATFORM_TOOLS}:${ANDROID_CMDLINE_TOOLS}/bin:${ANDROID_SDK_ROOT}/emulator
# install system image and accept licence, installation of emulator will fail for arm64 system
RUN yes Y | ${ANDROID_CMDLINE_TOOLS}/bin/sdkmanager --install "platform-tools" "system-images;android-24;google_apis;arm64-v8a" "platforms;android-24" "build-tools;30.0.3" && \
wget https://github.com/qhuyduong/arm_adb/releases/download/v1.0.39-aarch64/adb && \
chmod +x adb && mv adb /usr/lib/android-sdk/platform-tools/
# install and configure SSH server
EXPOSE 22
ADD sshd-banner /etc/ssh/
# ADD authorized_keys /tmp/
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends openssh-server supervisor locales && \
mkdir -p /var/run/sshd /var/log/supervisord && \
locale-gen en en_US en_US.UTF-8 && \
apt-get remove -y locales && apt-get autoremove -y && \
FILE_SSHD_CONFIG="/etc/ssh/sshd_config" && \
echo "\nBanner /etc/ssh/sshd-banner" >> $FILE_SSHD_CONFIG && \
echo "\nPermitUserEnvironment=yes" >> $FILE_SSHD_CONFIG && \
ssh-keygen -q -N "" -f /root/.ssh/id_rsa && \
FILE_SSH_ENV="/root/.ssh/environment" && \
touch $FILE_SSH_ENV && chmod 600 $FILE_SSH_ENV && \
printenv | grep "JAVA_HOME\|GRADLE_HOME\|KOTLIN_HOME\|ANDROID_SDK_ROOT\|LD_LIBRARY_PATH\|PATH" >> $FILE_SSH_ENV && \
echo "\nauth required pam_env.so envfile=$FILE_SSH_ENV" >> /etc/pam.d/sshd && \
FILE_AUTH_KEYS="/root/.ssh/authorized_keys" && \
touch $FILE_AUTH_KEYS && chmod 600 $FILE_AUTH_KEYS && \
for file in /tmp/*.pub; \
do if [ -f "$file" ]; then echo "\n" >> $FILE_AUTH_KEYS && cat $file >> $FILE_AUTH_KEYS && echo "\n" >> $FILE_AUTH_KEYS; fi; \
done && \
(rm /tmp/*.pub 2> /dev/null || true)
# install and configure VNC server
ENV USER root
ENV DISPLAY :1
EXPOSE 5901
ADD vncpass.sh /tmp/
ADD watchdog.sh /usr/local/bin/
ADD supervisord_vncserver.conf /etc/supervisor/conf.d/
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends xfce4 xfce4-goodies xfonts-base dbus-x11 tightvncserver expect && \
apt-get install -y --no-install-recommends firefox codeblocks && \
chmod +x /tmp/vncpass.sh; sync && \
/tmp/vncpass.sh && \
rm /tmp/vncpass.sh && \
apt-get remove -y expect && apt-get autoremove -y && \
FILE_SSH_ENV="/root/.ssh/environment" && \
echo "DISPLAY=:1" >> $FILE_SSH_ENV
# install and patch Visual Studio Code
RUN apt update && apt install -y libsecret-1-0 libxss1 libgbm1 && apt-get clean
RUN wget -O code_arm64.deb https://aka.ms/linux-arm64-deb && dpkg -i code_arm64.deb && rm code_arm64.deb
RUN sed 's/BIG-REQUESTS/_IG-REQUESTS/' /usr/lib/aarch64-linux-gnu/libxcb.so.1.1.0 | tee /usr/share/code/libxcb.so.1 > /dev/null
ADD supervisord.conf /etc/supervisor/conf.d/
CMD ["/usr/bin/supervisord"]
HEREEOF
</code></dl>
<br /><br />
(5) build the Dockfile<br />
<pre>docker build -f Dockerfile -t arm64v8/android-sdk-vnc .</pre><br /><br />
(6) Copy Mac host id_rsa.pub to authorized_keys and start the container<br />
<pre>cp ~/.ssh/authorized_keys .
cat ~/.ssh/id_rsa.pub >> authorized_keys</pre><br /><br />
# To start the container as daemon<br />
<pre>docker run -d --name android-sdk-vnc -p 5901:5901 -p 2222:22 -p 5037:5037 \
-v $(pwd)/authorized_keys:/root/.ssh/authorized_keys arm64v8/android-sdk-vnc</pre><br />
# To connect the container via ssh<br />
<pre>ssh root@<IP address of your Mac M1> -p 2222</pre><br />
# To stop the container<br />
<pre>docker container stop android-sdk-vnc</pre><br /><br />
(7) To connect to the vnc of the docker image after starting the container<br />
Use the Mac Finder "Go" -> "Connect To Server" and enter "vnc://0.0.0.0:5901"<br />
For other machines in the network enter "vnc://<IP address of your Mac M1>:5901"<br /><br />
The passwords are in the vncpass.sh when building which are<br />
password="android"<br />
password_view_only="docker"<br /><br />
(8) There is a common problem if you are connecting remotely using Mac Terminal, so the solution is to not forward your locale. Edit /etc/ssh/ssh_config and comment out SendEnv LANG LC_* line.<br /><br />
(9) Testing swift<br />
# attach to the running container<br />
<pre>docker exec -it android-sdk-vnc bash</pre>
<pre>
# test
mkdir -p ${HOME}/Projects
cd ${HOME}/Projects
git clone https://github.com/apple/example-package-dealer.git
cd example-package-dealer
swift run Dealer</pre><br /><br />
<br /><br />
If you need port forwarding on macOS, see this article
https://www.papercut.com/kb/Main/MacPortForwarding
<br /><br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-75004134013967237912020-12-19T14:47:00.072+08:002021-05-02T23:19:45.485+08:00How to setup debian vm with VNC on New Mac Mini M1(1) Download the ubuntu kernel and images<br/>
cd ${HOME}<br/>
mkdir debianvm<br/>
curl -o - https://cloud-images.ubuntu.com/releases/focal/release/unpacked/ubuntu-20.04-server-cloudimg-arm64-vmlinuz-generic | gunzip > debianvm/vmlinuz<br/>
curl -o debianvm/initrd https://cloud-images.ubuntu.com/releases/focal/release/unpacked/ubuntu-20.04-server-cloudimg-arm64-initrd-generic<br/>
curl -o - https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-arm64.tar.gz | tar zxvC debianvm<br/><br/>
(2) download the raspbian arm64 image<br/>
curl -OL https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2020-08-24/2020-08-20-raspios-buster-arm64.zip<br/>
unzip 2020-08-20-raspios-buster-arm64.zip -d debianvm<br/><br/>
(3) Copy and expand the disk to say 38G (<span style="color:red">Don't use this method to expand disk on external HD)</span><br/>
cp debianvm/focal-server-cloudimg-arm64.img debianvm/raspberry-arm64.img<br/>
dd if=/dev/zero of=debianvm/raspberry-arm64.img seek=40000000 obs=1024 count=0<br/><br/>
(4) Compile and use the latest vftool which is enhanced to mount multiple disks.<br/>
cd ${HOME}<br/>
git clone https://github.com/evansm7/vftool<br/>
cd vftool && xcodebuild<br/><br/>
(5) start the vm say<br/>
screen -S vm<br/>
cd ${HOME}/debianvm<br/>
${HOME}/vftool/build/Release/vftool -k vmlinuz -i initrd -d focal-server-cloudimg-arm64.img -d raspberry-arm64.img -d 2020-08-20-raspios-buster-arm64.img -m 2048 -a "console=hvc0"<br/>
Detach the screen using Ctrl-A D<br/><br/>
(6) start the tty say <br/>
screen -S tty /dev/ttys001<br/><br/>
(7) prepare the ubuntu cloud image<br/>
<pre>
mkdir /mnt
mount /dev/vda /mnt
chroot /mnt
touch /etc/cloud/cloud-init.disabled
echo 'root:root' | chpasswd
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519
# add fixed IP
cat <<EOF > /etc/netplan/01-dhcp.yaml
network:
renderer: networkd
ethernets:
enp0s1:
dhcp4: true
addresses: [192.168.64.33/24]
version: 2
EOF
# add IP when login prompt
cat <<EOF >> /etc/issue
enp0s1: \4
EOF
exit
umount /dev/vda
</pre><br/><br/>
(8) copy the rootfs to raspberry-arm64.img<br/>
#Detach the screen using Ctrl-A D<br/>
screen -r vm # resume vm screen
# stop the vm by ctrl-C and restart the ubuntu vm<br/>
cd ${HOME}/debianvm<br/>
${HOME}/vftool/build/Release/vftool -k vmlinuz -i initrd -d focal-server-cloudimg-arm64.img -d raspberry-arm64.img -d 2020-08-20-raspios-buster-arm64.img -m 2048 -a "console=hvc0 root=/dev/vda"<br/><br/>
#Detach the screen using Ctrl-A D<br/>
#Notice the tty number when starting the vm and start the tty say <br/>
screen -S tty /dev/ttys001<br/><br/>
# login with id root and password root<br/>
# and copy the rootfs to the expanded disk raspberry-arm64.img<br/>
dd if=/dev/vdc2 of=/dev/vdb bs=64K conv=noerror,sync<br/>
exit<br/><br/>
(9) prepare the raspberry-arm64 image<br/>
# stop the vm and restart the vm<br/>
# Control-A D to detach the tty screen
# resume vm screen
screen -r vm
# Control-C to stop the vm
cd ${HOME}/debianvm<br/>
${HOME}/vftool/build/Release/vftool -k vmlinuz -i initrd -d raspberry-arm64.img -m 2048 -a "console=hvc0"<br/>
<pre>
mkdir /mnt
mount /dev/vda /mnt
chroot /mnt
echo 'root:root' | chpasswd
#!!! delete or comment out the lines in /etc/fstab, or there will be "A start job is running on /dev/disk/.." blocking!!
cat <<EOF > /etc/fstab
proc /proc proc defaults 0 0
#PARTUUID=ad09722e-01 /boot vfat defaults 0 2
#PARTUUID=ad09722e-02 / ext4 defaults,noatime 0 1
EOF
ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa
ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa
ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519
mkdir -p /etc/network/interfaces.d/
cat <<EOF > /etc/network/interfaces.d/enp0s1
allow-hotplug enp0s1
iface enp0s1 inet dhcp
auto lo
iface lo inet loopback
EOF
cat <<EOF >> /etc/dhcpcd.conf
# add fixed IP
interface enp0s1
static ip_address=192.168.64.10/24
EOF
# put mac host ${HOME}/.ssh/id.rsa.pub key here
mkdir -p /root/.ssh
cat <<EOF > /root/.ssh/authorized_keys
ssh-rsa <mac id.rsa.pub key here>
EOF
# add IP when login prompt
cat <<EOF >> /etc/issue
enp0s1: \4
EOF
exit
umount /dev/vda
</pre><br/>
(10) login raspberry OS arm64 with id root and password root<br/>
# stop the vm and restart the vm with<br/>
cd ${HOME}/debianvm<br/>
${HOME}/vftool/build/Release/vftool -k vmlinuz -i initrd -d raspberry-arm64.img -m 2048 -a "console=hvc0 root=/dev/vda rw"<br/>
# set date say<br/>
date -s "29 DEC 2020 14:54:00"<br/>
resize2fs /dev/vda<br/>
dpkg-reconfigure tzdata # set timezone<br/>
apt update<br/>
systemctl disable rpi-eeprom-update.service<br/>
systemctl disable rng-tools.service<br/><br/>
(11) install utilities in raspberry os that can handle qcow2 disk image conversion and resize<br/>
apt install qemu-utils libguestfs-tools<br/>
hostname -I # get ip address<br/><br/>
(13) Install Realvncserver and connect from Mac<br/>
passwd pi # set user pi password<br/>
# login as user pi<br/>
login pi<br/>
# Use raspi-config to enable SSH and VNC<br/>
sudo apt install realvnc-vnc-server<br/>
sudo raspi-config<br/>
# Choose 5 Interfacing Options -> P2 SSH<br/>
# Choose 5 Interfacing Options -> P3 VNC<br/><br/>
# enter vnc key, can put this in /etc/rc.local without sudo <br/>
sudo vnclicense -add VKUPN-MTHHC-UDHGS-UWD76-6N36A<br/><br/>
# setup vncpasswd for user mode, only user mode can run browser<br/>
vncpasswd -user<br/><br/>
# edit config file to have these two lines for vnc password auth<br/>
echo -e "Authentication=VncAuth" | tee -a ~/.vnc/config.d/vncserver-x11<br/>
echo -e "Encryption=PreferOff" | tee -a ~/.vnc/config.d/vncserver-x11<br/>
cp ~/.vnc/config.d/vncserver-x11 ~/.vnc/config.d/Xvnc<br/>
# start service<br/>
vncserver<br/><br/>
#put these in rc.local to automate after reboot, Raspberry Pi hardware has bundled vnclicense and no need to add it<br/>
/usr/bin/vnclicense -add VKUPN-MTHHC-UDHGS-UWD76-6N36A<br/>
sudo -H -u pi /usr/bin/vncserver<br/><br/>
#check which port is listening<br/>
netstat -tlpn | grep -i listen<br/><br/>
This would configure VNC in <span style="background-color:gray; color:#add8e6">Virtual Mode</span><br/>
# Screen Sharing Connect from Mac<br/>
# In Finder, Go -> Connect to Server, enter IP address and port number of the vm e.g.<br/>
# Enter the vncpassword to connect, and connection needs port number<br />
vnc://192.168.64.10:5901<br/><br/>
For configuring VNC in <span style="background-color:black; color:#add8e6">Service Mode</span>, please refer to https://www.raspberrypi.org/documentation/remote-access/vnc/<br/>
For differences in these modes, please refer to https://help.realvnc.com/hc/en-us/articles/360002253238-Understanding-VNC-Server-Modes<br/>
#Start or stop the service with:<br/>
# systemctl (start|stop) vncserver-x11-serviced.service<br/>
#Mark or unmark the service to be started at boot time with:<br/>
# systemctl (enable|disable) vncserver-x11-serviced.service<br/>
#sudo systemctl start vncserver-x11-serviced.service<br/>
For Service Mode the connection from Mac is (no port number), vnc://192.168.64.10<br/><br/>
If you need port forwarding on macOS, so that you can vnc connect from other machines on the LAN to this vm, see this article
https://www.papercut.com/kb/Main/MacPortForwarding
That is run
<pre>
echo "
rdr pass on lo0 inet proto tcp from any to self port 5910 -> 192.168.64.10 port 5901
rdr pass on en0 inet proto tcp from any to any port 5910 -> 192.168.64.10 port 5901
rdr pass on wlan0 inet proto tcp from any to any port 5910 -> 192.168.64.10 port 5901
" | sudo pfctl -ef -
</pre>
Then, test connect from other machines to <IPaddress of macOS host>:5901 on the LAN.
<br/><br/>
# To install Visual Studio Code<br/>
wget -O code_arm64.deb https://aka.ms/linux-arm64-deb && sudo dpkg -i code_arm64.deb && rm code_arm64.deb<br/>
sed 's/BIG-REQUESTS/_IG-REQUESTS/' /usr/lib/aarch64-linux-gnu/libxcb.so.1.1.0 | sudo tee /usr/share/code/libxcb.so.1 > /dev/null<br/><br/>
<b>Demo - VNC to rapberrypi desktop vm screen capture</b>
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-k_eRDhsFVAw/X_Uj2riCX2I/AAAAAAAA3Ko/RWMIY9rhPfI4GM461sygca8GsOsVuL92wCLcBGAsYHQ/s2048/Screenshot_20210106_103755_com.realvnc.viewer.android.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="1280" data-original-width="2048" src="https://1.bp.blogspot.com/-k_eRDhsFVAw/X_Uj2riCX2I/AAAAAAAA3Ko/RWMIY9rhPfI4GM461sygca8GsOsVuL92wCLcBGAsYHQ/s400/Screenshot_20210106_103755_com.realvnc.viewer.android.jpg"/></a></div><br />
<!--
<div class="separator" style="clear: both;"><ahref="https://1.bp.blogspot.com/-7Iye4B02-FI/X-tA-47rTDI/AAAAAAAA3Io/M_3BcDKnQ-c-mAOUSfD7wY1vBaprmlSrQCLcBGAsYHQ/s1224/vmvncout.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="400" data-original-height="621" data-original-width="1224" src="https://1.bp.blogspot.com/-7Iye4B02-FI/X-tA-47rTDI/AAAAAAAA3Io/M_3BcDKnQ-c-mAOUSfD7wY1vBaprmlSrQCLcBGAsYHQ/s400/vmvncout.png"/></a></div><br/>
-->
P.S. Only vm can run realvncserver, docker environment can only run lightweight vncserver e.g. tightvncserver<br/><br/>
(14) For qcow2 image, the conversion to raw image is<br/>
wget https://cloud.debian.org/images/cloud/buster/20201214-484/debian-10-generic-arm64-20201214-484.qcow2<br/>
qemu-img dd -f qcow2 -O raw bs=64k if=debian-10-generic-arm64-20201214-484.qcow2 of=debian-10.img<br/>
# scp debian-10.img to the Mac Mini host and prepare a new expanded debian-arm64.img similar to the raspberry-arm64.img as above<br/>
# After restarting vm and mounting images, the dd command to extract rootfs is<br/>
# Assume debian-arm64.img is vdb and debian-10.img is vdc<br/>
dd if=/dev/vdc1 of=/dev/vdb bs=64K conv=noerror,sync<br/>
# For debian image, add this to /etc/hosts
127.0.1.1 debian
<br/><br/>
<!--
# deb [arch=arm64] https://download.docker.com/linux/debian buster stable
# apt install docker-ce=5:19.03.14~3-0~debian-buster containerd.io=1.4.3-1
-->
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-36005017462502611512020-12-03T22:55:00.055+08:002021-05-02T23:22:01.031+08:00How to run docker on the new Mac Mini M1(1) Reference to this article <a href="https://finestructure.co/blog/2020/11/27/running-docker-on-apple-silicon-m1" target="_blank">https://finestructure.co/blog/2020/11/27/running-docker-on-apple-silicon-m1</a><br/><br/>
(2) The problem of this method of running Ubuntu iso live image is not persistent.<br/><br/>
(3) So here is the improved method to add persistent storage to this vm.<br/><br/>
(4) # First create an empty image data say 50G in Mac Mini<br/>
dd if=/dev/zero of=data.img bs=1m count=51200<br/><br/>
(5) # Change the command to invoke the vm to include persistent and increase cpu to 2<br/>
<s>./vftool -k vmlinuz -i initrd -c focal-desktop-arm64.iso -d data.img -p 2 -m 4096 -a "console=hvc0 persistent rw"</s> <br/> ./vftool -k vmlinuz -i initrd -d data.img -c focal-desktop-arm64.iso -p 2 -m 4096 -a "console=hvc0 persistent rw"<br/><br/>
P.S.<br/>
If you compile the latest vftool, which has added the ability to add more image files and the order of the mounted device will follow the parameters you passed in. The article was written using the first version of the vftool and will mount the data.img as the /dev/vda and the cdrom image as /dev/vdb. So if you use the latest version vftool compiled, you have to check the device name of your mounted data.img.<br/><br/>
(6) # Attach the vm and create 3 partitions for the created image<br/>
sudo fdisk /dev/vda<br/>
# Press (n)(p)(1) to create new partition 1<br/>
# Press (enter) to accept starting sector and set size to say +25G<br/>
# Press (n)(p)(2) to create new partition 2<br/>
# Press (enter) to accept starting sector and set size to say +15G<br/>
# Press (n)(p)(3) to create new partition 3<br/>
# Accept the default options and starting block and ending block<br/>
# Press (w) to write to partition<br/><br/>
(6) # Format partition as ext4 with labels casper-rw and home-rw in vm, this create the persistent medium for the rw partition.<br/>
sudo mkfs.ext4 /dev/vda1 -L docker-data<br/>
sudo mkfs.ext4 /dev/vda2 -L casper-rw<br/>
sudo mkfs.ext4 /dev/vda3 -L home-rw<br/><br/>
(7) # exit shell and exit vm by ctrl-c<br/>
exit<br/><br/>
(8) # start vm again and use a new tab using screen command to attach to the vm e.g.<br/>
screen -S docker /dev/ttys003 <br/><br/>
(9) # Create folder to mount /dev/vda1<br/>
sudo mkdir -p /mnt/docker-data<br/>
# Add this mount commands in /etc/rc.local in order to persist as editing /etc/fstab is ignored in this iso image<br/>
<pre>
cat << EOF | sudo tee -a /etc/rc.local
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
mount /dev/vda1 -t ext4 /mnt/docker-data
exit 0
EOF</pre><br/>
<pre>
cat << EOF | sudo tee -a /etc/systemd/system/rc-local.service
[Unit]
Description=/etc/rc.local Compatibility
ConditionPathExists=/etc/rc.local
[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99
[Install]
WantedBy=multi-user.target
EOF</pre><br/>
# enable rc.local see <a href="https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd" target="_blank">this</a>.<br/>
sudo chmod +x /etc/rc.local<br/>
sudo systemctl start rc-local.service<br/>
sudo systemctl status rc-local.service<br/><br/>
(10) Follow the <a href="https://finestructure.co/blog/2020/11/27/running-docker-on-apple-silicon-m1" target="_blank">reference article</a> to install docker related packages.<br/>
If you have problem when installing docker. Try this = previous verion<br/>
apt install docker-ce=5:19.03.14~3-0~ubuntu-focal<br/><br/>
(11) # Move docker data-root to the newly created partition<br/>
sudo service docker stop<br/>
<pre>
cat << EOF | sudo tee -a /etc/docker/daemon.json
{
"storage-driver": "overlay2",
"data-root": "/mnt/docker-data"
}
EOF
</pre>
sudo rsync -aP /var/lib/docker/ /mnt/docker-data<br/>
sudo mv /var/lib/docker /var/lib/docker.old<br/>
sudo service docker start<br/><br/>
(12) Additionally, install docker-compose<br/>
sudo add-apt-repository universe<br/>
sudo apt install -y python3-pip<br/>
sudo pip3 -v install docker-compose<br/><br/>
(13) Test docker<br/>
docker run hello-world<br/><br/>
docker run -e MYSQL_ROOT_PASSWORD=rootpassword -e MYSQL_USER=wpuser -e MYSQL_PASSWORD=wpuserpassword -e MYSQL_DATABASE=wordpressdb --name wordpressdb -d arm64v8/mariadb<br/>
docker run -e WORDPRESS_DB_USER=wpuser -e WORDPRESS_DB_PASSWORD=wpuserpassword -e WORDPRESS_DB_NAME=wordpressdb -p 8080:80 --link wordpressdb:mysql --name wordpress -d arm64v8/wordpress<br/>
# show IP address<br/>
ip addr | grep inet<br/>
# test using browser e.g. http://192.168.64.19:8080/<br/><br/>
(14) Test docker-compose 1 (wordpress and mysql)<br/>
# create docker-compose.yaml <br/>
cd $HOME<br/>
mkdir my-wordpress<br/>
cd my-wordpress<br/>
<pre>
cat > docker-compose.yaml <<EOF
version: '3'
services:
db:
image: arm64v8/mariadb
volumes:
- "./.data/db:/var/lib/mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: rootpassword
MYSQL_DATABASE: wordpressdb
MYSQL_USER: wpuser
MYSQL_PASSWORD: wpuserpassword
wordpress:
depends_on:
- db
image: arm64v8/wordpress
links:
- db:mysql
ports:
- "8000:80"
restart: always
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_NAME: wordpressdb
WORDPRESS_DB_USER: wpuser
WORDPRESS_DB_PASSWORD: wpuserpassword
EOF
</pre>
# Start docker-compose<br/>
docker-compose up -d<br/>
# test using browser e.g. http://192.168.64.19:8000/<br/>
# after testing bring it down<br/>
docker-compose down -v<br/><br/>
(15) Test docker-compose 2 (mongo-express)<br/>
# create docker-compose.yaml <br/>
cd $HOME<br/>
mkdir my-mongoexpress<br/>
cd my-mongoexpress<br/>
<pre>
cat > docker-compose.yaml <<EOF
version: '3.1'
services:
mongo:
image: arm64v8/mongo
volumes:
- ./data:/data/db
ports:
- '27018:27017'
restart: always
environment:
MONGO_INITDB_ROOT_USERNAME: 'root'
MONGO_INITDB_ROOT_PASSWORD: 'example'
MONGO_INITDB_DATABASE: 'db'
mongo-express:
image: arm64v8/mongo-express
links:
- mongo
ports:
- "8081:8081"
restart: always
environment:
ME_CONFIG_MONGODB_ADMINUSERNAME: 'root'
ME_CONFIG_MONGODB_ADMINPASSWORD: 'example'
ME_CONFIG_MONGODB_ENABLE_ADMIN: 'true'
ME_CONFIG_MONGODB_AUTH_DATABASE: 'db'
ME_CONFIG_OPTIONS_EDITORTHEME: 'ambiance'
ME_CONFIG_BASICAUTH_USERNAME: 'user'
ME_CONFIG_BASICAUTH_PASSWORD: 'password'
EOF
</pre>
# Start docker-compose<br/>
docker-compose up -d --remove-orphans<br/>
# test using browser e.g. http://192.168.64.19:8081/ with user password<br/>
# after testing bring it down<br/>
docker-compose down -v<br/><br/>
(16) Change password and install openssh-server<br/>
sudo apt install -y openssh-server<br/>
# change /etc/ssh/sshd_config PermitEmptyPasswords to yes<br/>
PermitEmptyPasswords yes<br/><br/>
(17) Use another terminal session to connect e.g. <br/>
ssh-copy-id ubuntu@192.168.64.19<br/>
ssh ubuntu@192.168.64.19<br/><br/>
(18) This iso image has desktop Applications, so it is possible to install vnc server and connect to it via Remote Desktop or vnc://localhost:590x. To install vnc server <br/>
sudo apt install -y tightvncserver<br/>
vncserver<br/>
# restart vncserver in vm<br/>
vncserver -localhost<br/>
# create ssh tunnel in Mac, where x is the desktop number when starting vncserver <br/>
ssh ubuntu@192.168.64.19 -L 590x:localhost:590x
# Reference : <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-20-04" target="_blank">https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-20-04
</a><br/><a href="https://cat.pdx.edu/platforms/mac/remote-access/vnc-to-linux/" target="_blank">https://cat.pdx.edu/platforms/mac/remote-access/vnc-to-linux/</a>
<br/><br/>
P.S. follow up post for the docker M1 preview and building of docker image with vnc server.<br/>
<a href="https://iphonesdkdev.blogspot.com/2020/12/20201205.html" target="_blank">https://iphonesdkdev.blogspot.com/2020/12/20201205.html</a><br/><br/><br/><br/>
If you need port forwarding on macOS to connect to this vm service (e.g. ssh or vnc), see this article
<a href="https://www.papercut.com/kb/Main/MacPortForwarding" target="_blank">https://www.papercut.com/kb/Main/MacPortForwarding</a>
<br/><br/>
<!--
sudo dd if=/dev/vdb of=/dev/vda bs=64K conv=noerror,sync
sudo dd if=/dev/vdb2 of=/dev/vda1 conv=noerror,sync
sudo dd if=/dev/vdb1 of=/dev/vda2 conv=noerror,sync
sudo dd if=/cdrom/casper/filesystem.squashfs of=/dev/vda
-->
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-83774323726211959942020-12-03T00:52:00.012+08:002020-12-03T15:37:22.745+08:00How to run on Headless Mac Mini M1 with nohup and screen<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-Q3MmAuMGOx8/X8fJP_MWPZI/AAAAAAAA3Dw/AFfKBSywDUoWrrqIgpUMlF-5MXxPsXk0wCLcBGAsYHQ/s1158/Screenshot%2B2020-12-03%2Bat%2B1.03.27%2BAM.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" height="160" data-original-height="1158" data-original-width="902" src="https://1.bp.blogspot.com/-Q3MmAuMGOx8/X8fJP_MWPZI/AAAAAAAA3Dw/AFfKBSywDUoWrrqIgpUMlF-5MXxPsXk0wCLcBGAsYHQ/s320/Screenshot%2B2020-12-03%2Bat%2B1.03.27%2BAM.png"/></a></div>
(1) Need a HDMI dummy simulator attached to Mac Mini for full speed operation. Be sure to have HDMI 2.0 and 4K support and some dummy simulators have Pass-Through EDID Emulator (also with 4K support) so that you can attach 4K monitor or via screen sharing in HD when working on it with attached keyboard and mouse.<br/><br/>
(2) Disable sleep mode and set never sleep for HD and monitor on System Preferences of Mac Mini.<br/><br/>
(3) Enable Sharing and SSH login on Mac Mini.<br/><br/>
(4) generate ssh key-pair on mac mini and your terminal or your phone. If using Android phone, recommend Termux app for remote shell access.<br/>
e.g. ssh-keygen -t rsa # generate key pair on Mac Mini and remote client<br/><br/>
(5) copy the ssh key-pair to mac mini if using another terminal from mac.<br/>
e.g. ssh-copy-id user@x.x.x.x<br/><br/>
(6) Login in Mac Mini via ssh remotely<br/>
e.g. ssh user@x.x.x.x<br/><br/>
(7) # Apple Compressor command line batch submission example (no need for nohup)<br/>
/Applications/Compressor.app/Contents/MacOS/Compressor -batchname "My First Batch" \<br/>
-jobpath ~/Movies/MySource_x265.mp4 \<br/>
-settingpath /Applications/Compressor.app/Contents/Resources/Settings/ProRes/proRes422Name.compressorsetting \ <br/>
-locationpath ~/Movies/MyOutput_ProRes422.mov<br/><br/>
(8) # install youtube-dl, prerequisite: Command Line Tools for Xcode 12.2 for Python 3.8 to be installed<br/>
sudo mkdir -p /usr/local/bin<br/>
sudo curl -L https://github.com/l1ving/youtube-dl/releases/latest/download/youtube-dl -o /usr/local/bin/youtube-dl<br/>
sudo chmod a+rx /usr/local/bin/youtube-dl<br/>
youtube-dl --version<br/>
sudo youtube-dl -U # update youtube-dl<br/>
youtube-dl --version # check version after update<br/>
cd $HOME/Downloads<br/>
mkdir -p youtube<br/>
cd youtube<br/>
youtube-dl -f mp4 <youtube url><br/><br/>
(9) Either use nohup or screen command for persistent jobs.<br/><br/>
# nohup example <br/>
nohup bash -c 'youtube-dl -f 401 https://youtu.be/8rPB4A3zDnQ' &> nohup.youtubedl.4k_2.out &<br/><br/>
(10)screen example (suitable for command utility with tty input <br/>
# create a screen and run python script<br/>
screen -S tensorflow<br/>
conda activate python38<br/>
python cnn.py<br/>
# leave (detach) the screen by Control-A d<br/><br/>
# create second screen and run handbrake conversion<br/>
screen -S handbrake<br/>
conda deactivate<br/>
$HOME/Downloads/HandBrakeCLI -i $HOME/Downloads/8KVIDEO120FPS-8rPB4A3zDnQ_4.1G.mp4 -o $HOME/Downloads/8KVIDEO_x265.mp4 -e x265 -q24<br/>
# leave (detach) the screen by Control-A d<br/><br/>
# list all the screen ids<br/>
screen -ls<br/><br/>
# Reattach to one of the screen e.g. 6473.tensorflow<br/>
screen -r 6473<br/><br/>
# Exit Screen if the process terimated<br/>
exit<br/><br/>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-71924501374506703402020-11-21T13:24:00.021+08:002020-12-03T01:23:25.271+08:00How to install tensorflow for the new Mac M1 hardwarePrerequisite: Xcode 12.2 and Command Line Tools for Xcode 12.2<br/><br/>
(1) # Download the archive for this repo from https://github.com/apple/tensorflow_macos/releases<br/>
cd $HOME/Downloads/<br/>
curl -fsSLO https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha0/tensorflow_macos-0.1alpha0.tar.gz<br/>
tar xzvf tensorflow_macos-0.1alpha0.tar.gz<br/>
/bin/bash ./tensorflow_macos/install_venv.sh --help <br/><br/>
(2) # Download Miniconda from https://conda-forge.org/blog/posts/2020-10-29-macos-arm64/<br/><br/>
(3) # Install Miniconda and after installtion, exit shell and login again<br/>
/bin/bash -c "$(curl -fsSL https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh)"<br/><br/>
(4) # Once it's installed, create a Python 3.8 env by running <br/>
conda create --name python38 python=3.8<br/><br/>
(5) # Put a path to where the arm64 libraries are. For example...<br/>
libs="$HOME/Downloads/tensorflow_macos/arm64/"<br/><br/>
(6) # Replace this with the path of your Conda environment<br/>
env="$HOME/miniforge3/envs/python38"<br/><br/>
(7) # upgrade <br/>
conda upgrade -c conda-forge pip setuptools cached-property six<br/><br/>
(8) # activate env <br/>
conda activate python38<br/>
# conda deactivate<br/><br/>
(9) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/grpcio-1.33.2-cp38-cp38-macosx_11_0_arm64.whl"<br/><br/>
(10) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/h5py-2.10.0-cp38-cp38-macosx_11_0_arm64.whl"<br/><br/>
(11) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/numpy-1.18.5-cp38-cp38-macosx_11_0_arm64.whl"<br/><br/>
(12) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/tensorflow_addons-0.11.2+mlcompute-cp38-cp38-macosx_11_0_arm64.whl"<br/><br/>
(13) # install these <br/>
<pre>
conda install -c conda-forge -y absl-py
conda install -c conda-forge -y astunparse
conda install -c conda-forge -y gast
conda install -c conda-forge -y opt_einsum
conda install -c conda-forge -y termcolor
conda install -c conda-forge -y typing_extensions
conda install -c conda-forge -y wheel
conda install -c conda-forge -y typeguard
pip install tensorboard
pip install wrapt flatbuffers tensorflow_estimator google_pasta keras_preprocessing protobuf
</pre><br/><br/>
(14) pip install --upgrade -t "$env/lib/python3.8/site-packages/" --no-dependencies --force "$libs/tensorflow_macos-0.1a0-cp38-cp38-macosx_11_0_arm64.whl"<br/><br/>
(15) # Run this to test<br/>
<pre>time python tftest.py</pre><br/><br/>
<dl class="codebox">
<dt>tftest.py <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>from datetime import datetime
import numpy as np
import tensorflow as tf
from tensorflow.python.compiler.mlcompute import mlcompute
mlcompute.set_mlc_device(device_name="cpu")
# tensorflow:Eager mode on GPU is extremely slow. So use CPU instead
print("Hello, Tensorflow! ", end='')
print(tf.__version__)
print("start" , datetime.now())
X_raw = np.array([2013, 2014, 2015, 2016, 2017, 2018], dtype=np.float32)
y_raw = np.array([12000, 14000, 15000, 16500, 17500, 19000], dtype=np.float32)
X = (X_raw - X_raw.min()) / (X_raw.max() - X_raw.min())
y = (y_raw - y_raw.min()) / (y_raw.max() - y_raw.min())
X = tf.constant(X)
y = tf.constant(y)
a = tf.Variable(initial_value=0.)
b = tf.Variable(initial_value=0.)
variables = [a, b]
num_epoch = 10000
optimizer = tf.keras.optimizers.SGD(learning_rate=1e-3)
for e in range(num_epoch):
with tf.GradientTape() as tape:
y_pred = a * X + b
loss = 0.5 * tf.reduce_sum(tf.square(y_pred - y))
grads = tape.gradient(loss, variables)
optimizer.apply_gradients(grads_and_vars=zip(grads, variables))
print(a, b)
print("end" , datetime.now())
</code></dl>
<br/><br/><br/><br/>
(16) Test run this cnn.py https://github.com/apple/tensorflow_macos/issues/25<br/><br/>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-29754759772185755602020-10-24T16:06:00.011+08:002020-10-24T17:55:13.188+08:00How to use SpriteKit in SwiftUI and preview SKScene in Xcode 12This demo the code using Xcode 12.0.1 under macOSX 10.15.7
The iOS14 has a new SpriteView for the SKScene, this demo a player movement with on-screen joystick control. This file does not need the creation of sks file and using code to generate the SKNodes required. The graphics assets and touches codes are borrowed from this Elton Game - Introduction to Sprite Kit. https://designcode.io/spritekit-intro
<br /><br />
<span style="font-size: large;">Create this swift file for iOS14 and you can preview it in SwiftUI under Xcode 12 </span><br />
<br />
<br />
<dl class="codebox">
<dt>SKContentView.swift.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>//
// SKContentView.swift
//
import SwiftUI
import SpriteKit
class GameScene: SKScene {
// Nodes
var player : SKNode?
var joystick : SKNode?
var joystickKnob : SKNode?
// Boolean
var joystickAction = false
// Measure
var knobRadius : CGFloat = 50.0
// Sprite Engine added in Lesson 2
var previousTimeInterval : TimeInterval = 0
let playerSpeed = 4.0
override func didMove(to view: SKView) {
physicsBody = SKPhysicsBody(edgeLoopFrom: frame)
let player = SKSpriteNode(imageNamed: "player/1")
self.player = player
let joystick = SKSpriteNode(imageNamed: "arrow")
self.joystick = joystick
let joystickKnob = SKSpriteNode(imageNamed: "knob")
self.joystickKnob = joystickKnob
joystick.addChild(joystickKnob)
player.position = CGPoint(x: frame.midX, y: frame.size.height / 5 * 3)
joystick.position = CGPoint(x: 80, y: 80)
addChild(player)
addChild(joystick)
view.showsFPS = true
view.showsNodeCount = true
}
}
extension GameScene {
// Touch Began
override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
for touch in touches {
if let joystickKnob = joystickKnob {
let location = touch.location(in: joystick!)
joystickAction = joystickKnob.frame.contains(location)
}
}
}
// Touch Moved
override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
guard let joystick = joystick else { return }
guard let joystickKnob = joystickKnob else { return }
if !joystickAction { return }
// Distance
for touch in touches {
let position = touch.location(in: joystick)
let length = sqrt(pow(position.y, 2) + pow(position.x, 2))
let angle = atan2(position.y, position.x)
if knobRadius > length {
joystickKnob.position = position
} else {
joystickKnob.position = CGPoint(x: cos(angle) * knobRadius, y: sin(angle) * knobRadius)
}
}
}
// Touch End added in Lesson 2
override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {
for touch in touches {
let xJoystickCoordinate = touch.location(in: joystick!).x
let xLimit: CGFloat = 200.0
if xJoystickCoordinate > -xLimit && xJoystickCoordinate > xLimit {
resetKnobPosition()
}
}
}
}
// MARKS: Action added in Lesson 2
extension GameScene {
func resetKnobPosition() {
let initialPoint = CGPoint(x:0, y:0)
let moveBack = SKAction.move(to: initialPoint, duration:0.1)
moveBack.timingMode = .linear
joystickKnob?.run(moveBack)
joystickAction = false
}
}
// MARKS: Game Loop added in Lesson 2
extension GameScene {
override func update( _ currentTime: TimeInterval) {
let deltaTime = currentTime - previousTimeInterval
previousTimeInterval = currentTime
// player movement
guard let joystickKnob = joystickKnob else { return }
let xPosition = Double(joystickKnob.position.x)
let yPosition = Double(joystickKnob.position.y)
let displacement = CGVector(dx:deltaTime * xPosition * playerSpeed, dy:deltaTime * yPosition * playerSpeed)
let move = SKAction.move(by: displacement, duration:0)
player?.run(move)
}
}
struct SKContentView: View {
var scene: SKScene {
let scene = GameScene()
scene.size = CGSize(width: 300, height: 400)
scene.scaleMode = .fill
return scene
}
var body: some View {
SpriteView(scene: scene)
.frame(width: 300, height: 400)
.edgesIgnoringSafeArea(.all)
}
}
struct SKContentView_Previews: PreviewProvider {
static var previews: some View {
SKContentView()
}
}
</code></dl>
<br /><br /><br /><br />
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-JT6WYZheuDA/X5PiihHdJdI/AAAAAAAA22Y/Ics43TRFKhwfTY0bqJu_VP5AxtWymnJ4QCLcBGAsYHQ/s1064/player.gif" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="320" data-original-height="1064" data-original-width="710" src="https://1.bp.blogspot.com/-JT6WYZheuDA/X5PiihHdJdI/AAAAAAAA22Y/Ics43TRFKhwfTY0bqJu_VP5AxtWymnJ4QCLcBGAsYHQ/s320/player.gif"/></a></div>
<br/><br/>
player/1<br/>
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-qB90hVlRafc/X5PgQ3z4kJI/AAAAAAAA22E/k2ipq6CLHlcg6-iOIi3JS81Nva4qvrhiwCLcBGAsYHQ/s49/1.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="320" data-original-height="49" data-original-width="36" src="https://1.bp.blogspot.com/-qB90hVlRafc/X5PgQ3z4kJI/AAAAAAAA22E/k2ipq6CLHlcg6-iOIi3JS81Nva4qvrhiwCLcBGAsYHQ/s320/1.png"/></a></div>
arrow<br/>
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-gpF3g1EBO7g/X5PgQ5uJKUI/AAAAAAAA22I/7RRnV92lGyozJLG_z5wE_GSXmSvUCtDPQCLcBGAsYHQ/s140/arrow.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="320" data-original-height="140" data-original-width="140" src="https://1.bp.blogspot.com/-gpF3g1EBO7g/X5PgQ5uJKUI/AAAAAAAA22I/7RRnV92lGyozJLG_z5wE_GSXmSvUCtDPQCLcBGAsYHQ/s320/arrow.png"/></a></div>
knob<br/>
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-4WlP5lj3JkY/X5PgQyyXS6I/AAAAAAAA22A/fWZdL8aR9hgAlWjipI_I_YSoiJaX8SPNgCLcBGAsYHQ/s89/knob.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="320" data-original-height="89" data-original-width="87" src="https://1.bp.blogspot.com/-4WlP5lj3JkY/X5PgQyyXS6I/AAAAAAAA22A/fWZdL8aR9hgAlWjipI_I_YSoiJaX8SPNgCLcBGAsYHQ/s320/knob.png"/></a></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-62018897625477778992020-10-17T17:34:00.037+08:002021-06-10T09:34:45.900+08:00How to download WWDC2020 videos and subtitlesThe scripts for previous WWDC are here
https://iphonesdkdev.blogspot.com/2017/07/how-to-fetch-wwdc-2017-video-subtitle.html
<br /><br />
<span style="font-size: large;">Create and Run this script wwdc2020_fetch_srt.sh to fetch WWDC2020 subtitle</span><br />
<br />
<br />
<dl class="codebox">
<dt>wwdc2020_fetch_en.srt.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/bin/bash
# @Last Modified by: javacom
# @Last Modified time: 2020-10-17
WWDC_YEAR=2020;
#
WWDC_SESSION_PREFIX=https://developer.apple.com/videos/play/wwdc$WWDC_YEAR;
WWDC_LOCAL_DIR=$(basename $WWDC_SESSION_PREFIX);
detect_video_m3u8 () {
local session_url=$WWDC_SESSION_PREFIX/$SESSION_ID/;
local session_html=$(curl -s $session_url);
local video_url=$(echo "$session_html" | grep .m3u8 | grep $SESSION_ID | head -n1 | sed "s#.*\"\(https://.*m3u8\)\".*#\1#");
echo "$session_html" | grep .mp4 | grep $SESSION_ID | sed "s#.*\"\(https://.*mp4\).*\".*#\1#" | while read mp4_url; do
local mp4_filename=$(basename $mp4_url | cut -d. -f1);
local srt_filename=$mp4_filename.en.srt;
# local srt_filename=$mp4_filename.zh.srt;
echo "> Subtitle local: $WWDC_LOCAL_DIR/$srt_filename" >&2;
> $WWDC_LOCAL_DIR/$srt_filename;
done
echo "$video_url";
echo "> Video: $video_url" >&2;
}
detect_subtitle_m3u8 () {
local video_url=$1;
# en subtitle
local subtitle_uri=$(curl -s $video_url | grep "LANGUAGE=\"en\",URI" | sed "s#.*URI=\"\(.*\)\"#\1#");
# zh subtitle
#local subtitle_uri=$(curl -s $video_url | grep "LANGUAGE=\"zh\"" | sed "s#.*URI=\"\(.*\)\"#\1#");
local subtitle_url=$subtitle_uri;
[[ "$subtitle_uri" != http* ]] && {
subtitle_url=$(dirname $video_url)/$subtitle_uri;
}
echo "$subtitle_url";
echo "> Subtitle: $subtitle_url" >&2;
}
download_subtitle_contents () {
local subtitle_url=$1;
echo "> Downloading... "
local subtitle_base_url=$(dirname $subtitle_url);
curl -s $subtitle_url | grep "webvtt" | while read webvtt; do
local subtitle_webvtt=$subtitle_base_url/$webvtt;
#echo "- get $subtitle_webvtt";
local subtitle_content=$(curl -s $subtitle_webvtt);
# en subtitle
ls $WWDC_LOCAL_DIR/"wwdc$WWDC_YEAR"_"$SESSION_ID"*.en.srt | while read srt_file; do
# zh subtitle
# ls $WWDC_LOCAL_DIR/"wwdc$WWDC_YEAR"_"$SESSION_ID"*.zh.srt | while read srt_file; do
echo "$subtitle_content" >> $srt_file;
done
done
}
main () {
[ ! -d $WWDC_LOCAL_DIR ] && {
mkdir $WWDC_LOCAL_DIR;
}
#Year 2020 change {3\} to {3,5\}
curl -s $WWDC_SESSION_PREFIX | grep /videos/play/wwdc$WWDC_YEAR | sed "s#.*/videos/play/wwdc$WWDC_YEAR/\([0-9]\{3,5\}\).*#\1#" | sort | uniq | while read SESSION_ID; do
#echo "SESSION_ID is" $SESSION_ID
local video_url=$(detect_video_m3u8 $SESSION_ID);
local subtitle_url=$(detect_subtitle_m3u8 $video_url);
download_subtitle_contents $subtitle_url;
done
}
main;
</code></dl>
<br /><br /><br /><br />
<span style="font-size: large;">Run this shell script to format as SRT subtitle</span><br />
<br />
<dl class="codebox">
<dt>shellscript.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>WWDC_YEAR=2020;
#
cd wwdc$WWDC_YEAR
mkdir -p sd
mkdir -p hd
for i in *_sd.??.srt; do sed -e '/WEBVTT/d;/X-TIMESTAMP/d;s/align.middle line.*$//;' $i | awk '/^[0-9]{2}:[0-9]{2}:/ {seen[$0]++; skipduplicated=0} {if (seen[$0]>1) skipduplicated=1; if (!skipduplicated) print $0}' | awk -v RS="" '{gsub("\n", "-Z"); print}' | awk '$0 !~/^WEB/ {print $0}' | uniq | awk '{printf "\n%s-Z%s", NR,$0 }' | awk -v ORS="\n\n" '{gsub("-Z", "\n"); print}' | sed -e 's/.A:middle$//g;s/&gt;/>/g;s/&lt;/</g;1,2d;' > sd/$i; done
for i in *_hd.??.srt; do sed -e '/WEBVTT/d;/X-TIMESTAMP/d;s/align.middle line.*$//;' $i | awk '/^[0-9]{2}:[0-9]{2}:/ {seen[$0]++; skipduplicated=0} {if (seen[$0]>1) skipduplicated=1; if (!skipduplicated) print $0}' | awk -v RS="" '{gsub("\n", "-Z"); print}' | awk '$0 !~/^WEB/ {print $0}' | uniq | awk '{printf "\n%s-Z%s", NR,$0 }' | awk -v ORS="\n\n" '{gsub("-Z", "\n"); print}' | sed -e 's/.A:middle$//g;s/&gt;/>/g;s/&lt;/</g;1,2d;' > hd/$i; done
</code></dl>
<br /><br /><br /><br />
<span style="font-size: large;">Run this script wwdc2020_fetch_mp4.sh to download all mp4 (HD and SD) videos (also works for 2021)</span><br />
<br />
<dl class="codebox">
<dt>wwdc2020_fetch_mp4.sh  <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/bin/bash
# @Last Modified by: javacom
# @Last Modified time: 2020-10-17
WWDC_YEAR=2020; # change to 2021 also works for WWDC2021
#
WWDC_SESSION_PREFIX=https://developer.apple.com/videos/play/wwdc$WWDC_YEAR;
WWDC_LOCAL_DIR=$(basename $WWDC_SESSION_PREFIX);
download_mp4_video () {
local session_url=$WWDC_SESSION_PREFIX/$SESSION_ID/;
local session_html=$(curl -s $session_url);
local video_url=$(echo "$session_html" | grep .m3u8 | grep $SESSION_ID | head -n1 | sed "s#.*\"\(https://.*m3u8\)\".*#\1#");
echo "$session_html" | grep .mp4 | grep $SESSION_ID | sed "s#.*\"\(https://.*mp4\).*\".*#\1#" | while read mp4_url; do
local mp4_filename=$(basename $mp4_url);
if [ -e $WWDC_LOCAL_DIR/$mp4_filename ]
then
echo "> MP4 already existed : $WWDC_LOCAL_DIR/$mp4_filename" >&2;
echo "> To resume broken download use curl -C - --connect-timeout 1200 -o $WWDC_LOCAL_DIR/$mp4_filename $mp4_url" >&2;
echo " " >&2;
else
echo "> MP4 Downloading... : $mp4_url" >&2;
curl --connect-timeout 120 -o $WWDC_LOCAL_DIR/$mp4_filename $mp4_url
fi
done
}
main () {
[ ! -d $WWDC_LOCAL_DIR ] && {
mkdir $WWDC_LOCAL_DIR;
}
#Year 2020 change {3\} to {3,5\}
curl -s $WWDC_SESSION_PREFIX | grep /videos/play/wwdc$WWDC_YEAR | sed "s#.*/videos/play/wwdc$WWDC_YEAR/\([0-9]\{3,5\}\).*#\1#" | sort | uniq | while read SESSION_ID; do
download_mp4_video $SESSION_ID;
done
}
main;
</code></dl><br /><br />
<br /><br /><br /><br />
<span style="font-size: large;">Run this script to rename the video or subtitles to proper title (HD & SD) videos</span><br />
<br />
<dl class="codebox">
<dt>wwdc2020_rename_title.sh  <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/bin/sh
# @Last Modified by: javacom
# @Last Modified time: 2020-10-17
#sd video
curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_sd.mp4" \] \&\& mv "wwdc2020_\1_sd.mp4" "wwdc2020_\1_sd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.mp4"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash
#hd video
curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_hd.mp4" \] \&\& mv "wwdc2020_\1_hd.mp4" "wwdc2020_\1_hd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.mp4"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash
#sd video en.srt
curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_sd.en.srt" \] \&\& mv "wwdc2020_\1_sd.en.srt" "wwdc2020_\1_sd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.en.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash
#hd video en.srt
curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_hd.en.srt" \] \&\& mv "wwdc2020_\1_hd.en.srt" "wwdc2020_\1_hd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.en.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash
#sd video zh.srt
curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_sd.zh.srt" \] \&\& mv "wwdc2020_\1_sd.zh.srt" "wwdc2020_\1_sd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.zh.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash
#hd video zh.srt
curl -s https://developer.apple.com/videos/wwdc2020/ | grep -B1 "video-title" | sed -e 's#[[:space:]]\{10,\}<a href="/videos/play/wwdc2020/\([0-9]\{3,5\}\).*#\[ -f "wwdc2020_\1_hd.zh.srt" \] \&\& mv "wwdc2020_\1_hd.zh.srt" "wwdc2020_\1_hd_#' -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.zh.srt"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | /bin/bash
# For WWDC2021
WWDC_YEAR=2021; VIDEO=sd; curl -s https://developer.apple.com/videos/wwdc${WWDC_YEAR}/ | grep -B1 "video-title" | sed -e "s#[[:space:]]\{10,\}<a href=./videos/play/wwdc202./\([0-9]\{3,5\}\).*#\[ -f \"wwdc${WWDC_YEAR}-\1_${VIDEO}.mp4\" \] \&\& mv \"wwdc${WWDC_YEAR}-\1_${VIDEO}.mp4\" \"wwdc${WWDC_YEAR}-\1_sd_#" -e "s/[\@:’\'\,?]//g" -e "s/\"\(The.*\)\"/\1/" -e 's#.*video-title..\(.*\)\(</h4>\)#\1.mp4"#' -e '/--/d' | sed '/^\[ -f/{N;s/\n//;}' | sed "s#[\/@:’\'\,]#_#g" | /bin/bash
</code></dl><br /><br />
<br /><br /><br /><br />
# Example nohup commnad for Linux Download<br/><br/>
nohup bash -c 'cd $HOME/Downloads/WWDC/; ./wwdc2020_fetch_mp4.sh' &> nohup.wwdc2020.mp4.out &<br/><br/>
nohup bash -c 'cd $HOME/Downloads/WWDC/; ./wwdc2020_fetch_en.srt.sh' &> nohup.wwdc2020.en.srt.out &<br/><br/>
# or use screen utility to download<br/>
screen -S wwdcdownloadmp4<br/>
bash wwdc2020_fetch_mp4.sh<br/><br/>
<br/><br/>
If want to remove special characters in filename "brew install rename" and use this command to rename<br/>
rename "s/[\@:’\'\,]//g" *.mp4 *.srt<br/>
<br/><br/>
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-k8REvAEBt44/X4uFEHdLlsI/AAAAAAAA2u8/Ax2emC6AobAx_Qa4qA80Dg6PkhHq72bYwCPcBGAYYCw/s1920/Screenshot_20201018-074903.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="640" data-original-height="1080" data-original-width="1920" src="https://1.bp.blogspot.com/-k8REvAEBt44/X4uFEHdLlsI/AAAAAAAA2u8/Ax2emC6AobAx_Qa4qA80Dg6PkhHq72bYwCPcBGAYYCw/s320/Screenshot_20201018-074903.png"/></a></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-2428259418895237852019-08-07T19:58:00.003+08:002019-08-08T22:15:30.423+08:00Xcode 11 beta 5 SwiftUI changes <a href="https://1.bp.blogspot.com/-4GUSvXrns8o/XUq7cTMS7rI/AAAAAAAAxOU/_KgMV4e3z7U9Fs5ghHwjuXTbULo-9PP0wCLcBGAs/s1600/beta5%2Bchanges.png" imageanchor="1"><img border="0" data-original-height="694" data-original-width="1600" height="174" src="https://1.bp.blogspot.com/-4GUSvXrns8o/XUq7cTMS7rI/AAAAAAAAxOU/_KgMV4e3z7U9Fs5ghHwjuXTbULo-9PP0wCLcBGAs/s400/beta5%2Bchanges.png" width="640" height="170" /></a>
<br />
<dl class="codebox">
<dt>NavigationButton becomes NavigationLink <a href="https://www.blogger.com/blogger.g?blogID=6245963027152906987#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
<span style="background-color: magenta;color: white;">NavigationButton</span>(
destination: LandmarkDetail(landmark: landmark)) {
LandmarkRow(landmark: landmark)
}
</code></dl>
<br />
<dl class="codebox">
<dt>NavigationButton becomes NavigationLink <a href="https://www.blogger.com/blogger.g?blogID=6245963027152906987#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
<span style="color: red;">NavigationLink</span>(
destination: LandmarkDetail(landmark: landmark)
.environmentObject(self.userData)
) {
LandmarkRow(landmark: landmark)
}
</code></dl>
<br />
<br />
<dl class="codebox">
<dt>ForEach .identified by: <a href="https://www.blogger.com/blogger.g?blogID=6245963027152906987#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
<span style="background-color: magenta;color: white;">ForEach(["iPhone SE", "iPhone XS Max"].identified(by: \.self))</span> { deviceName in
LandmarkList()
.previewDevice(PreviewDevice(rawValue: deviceName))
.previewDisplayName(deviceName)
}
.environmentObject(UserData())
</code></dl>
<br />
<dl class="codebox">
<dt>ForEach id: <a href="https://www.blogger.com/blogger.g?blogID=6245963027152906987#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
<span style="color: red;">ForEach(["iPhone SE", "iPhone XS Max"], id: \.self)</span> { deviceName in
LandmarkList()
.previewDevice(PreviewDevice(rawValue: deviceName))
.previewDisplayName(deviceName)
}
.environmentObject(UserData())
</code></dl>
<br />
<br />
<a href="https://1.bp.blogspot.com/-olM5VmtU6oM/XUq_87Hj4bI/AAAAAAAAxOg/MAuuFZcfjY418Tj4MUZydaAo1fCQmgt2wCLcBGAs/s1600/beta4%2Bchnages.png" imageanchor="1" ><img border="0" src="https://1.bp.blogspot.com/-olM5VmtU6oM/XUq_87Hj4bI/AAAAAAAAxOg/MAuuFZcfjY418Tj4MUZydaAo1fCQmgt2wCLcBGAs/s640/beta4%2Bchnages.png" width="640" height="170" data-original-width="1600" data-original-height="424" /></a>
<br />
<dl class="codebox">
<dt>BindableObject becomes ObservableObject <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
final class UserData: <span style="background-color: magenta;color: white;">BindableObject</span> {
<span style="background-color: magenta;color: white;">let didChange = PassthroughSubject<UserData, Never>()</span>
var showFavoritesOnly = false <span style="background-color: magenta;color: white;">{
didSet {
didChange.send(self)
}
}</span>
var landmarks = landmarkData <span style="background-color: magenta;color: white;">{
didSet {
didChange.send(self)
}
}</span>
}
</code></dl>
<br />
<dl class="codebox">
<dt>BindableObject becomes ObservableObject <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
final class UserData: <span style="color: red;">ObservableObject</span> {
<span style="color: red;">@Published</span> var showFavoritesOnly = false
<span style="color: red;">@Published</span> var landmarks = landmarkData
}
</code></dl>
<br />
<br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-20139248331966877372019-06-08T18:40:00.000+08:002019-06-27T19:38:23.424+08:00How to use SwiftUI previews for existing project and storyboardHere is an example code to create a new swift file for an existing project
and use the storyboard Main.storyboard and preview the MainViewController instance in the new iOS 13 SwiftUI framework under Xcode 11.<br />
<br />
If you add TabbedView to NavigationView in this example, please use Xcode 11 Beta 2.<br />
<br />
<dl class="codebox">
<dt>MainViewPreviews.swift <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>// MainViewPreviews.swift
// FishTagCreator
//
// Created by javacom on 8 Jun 2019.
//
import SwiftUI
#if DEBUG
struct MainViewControllerPreviews : PreviewProvider, UIViewControllerRepresentable {
// MARK: PreviewProvider
static var previews: some View {
MainViewControllerPreviews()
}
// MARK: UIViewControllerRepresentable
typealias UIViewControllerType = MainViewController
func makeUIViewController(
context: Context
) -> MainViewController {
let mainStoryboard: UIStoryboard = UIStoryboard(name: "Main", bundle: nil)
let mainViewController: MainViewController = mainStoryboard.instantiateViewController(withIdentifier: "MainController") as! MainViewController
return mainViewController
}
func updateUIViewController(
_ uiViewController : MainViewController, context: Context) {
}
}
#endif
</code></dl>
<br />
<br />
It is important to assign a storyboard ID say "MainController" to your MainViewController in Main.storyboard's identity inspector.<br />
<a href="https://3.bp.blogspot.com/-XqJCuxghV1U/XPuRz8AIOnI/AAAAAAAAwdo/7MRzvcoScpoy7gtAd8JefjU38wpyJ1wDACLcBGAs/s1600/MainController.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-XqJCuxghV1U/XPuRz8AIOnI/AAAAAAAAwdo/7MRzvcoScpoy7gtAd8JefjU38wpyJ1wDACLcBGAs/s400/MainController.png" width="400" height="281" data-original-width="516" data-original-height="362" /></a>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-77752996763143091752018-07-07T09:43:00.002+08:002018-07-08T05:35:18.111+08:00How to make a bootable macOS installer on USB in macOS MojaveTo create a bootable installer of the macOS on USB drive is useful to repair filesystem when another mac is not bootable.<br/><br/>
Step 1: A 12GB Flash Drive (at least!) and formated with Mac OS Extended (Journaled) and Choose GUID Partition Map as the Scheme.<br/>The name of the USB Flash Drive will be named as Untitled as default
<br/><br/>
Step 2: Go to Mac App Store and from the past purchase history, see if there is any Sierra Developer Beta in purchase history and then download it. For unknown reasons, the previous versions of macOS cannot be downloaded in the Mac App Store of Mojave.<br/>
There is one more rule : "A Mac can boot NO version OLDER than the version it shipped with". So choose the newer version.
<br/><br/>
<a href="https://3.bp.blogspot.com/-FFQZs-yut9c/W0Ae3SdHuGI/AAAAAAAAqx0/lRUO8q7QxPkWKJUjUIxQc8ImfHbb3-7NQCLcBGAs/s1600/macOS_beta.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-FFQZs-yut9c/W0Ae3SdHuGI/AAAAAAAAqx0/lRUO8q7QxPkWKJUjUIxQc8ImfHbb3-7NQCLcBGAs/s320/macOS_beta.png" width="320" height="169" data-original-width="520" data-original-height="274" /></a><br/>
Install_macOS_Sierra_Developer_Beta.rar (4.40GB)<br/>
https://mega.nz/#!yk4lSSRQ!WoOSpLf5BSlRR4if3RrbHVHQptG0Tfmw0Bnx4BCrHlA<br/>
Install_macOS_High_Sierra.rar (4.86GB)<br/>
https://mega.nz/#!WopVXYqQ!LlfKompmLDag20CE6UrsYQmL6e9mKoEgW08bLAvcnbs<br/>
Install_macOS_Mojave_Beta.rar (5.22GB)<br/>
https://mega.nz/#!qtwxkS7T!7_lG6VhwQLL1Zyc_-s_T5jjVu06vnnvHJTsSTa7fNiI
<br/><br/>
Step 3: Go to Terminal and type command<br />
<dl class="codebox">
<dt>shellscript <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># for Sierra Developer beta the command is
sudo /Applications/Install\ macOS\ Sierra\ Developer\ Beta.app/Contents/Resources/createinstallmedia --volume /Volumes/Untitled --applicationpath /Applications/Install\ macOS\ Sierra\ Developer\ Beta.app
# for High Sierra the command is
sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Untitled --applicationpath /Applications/Install\ macOS\ High\ Sierra.app
# for Mojave beta the command is
sudo /Applications/Install\ macOS\ Mojave\ Beta.app/Contents/Resources/createinstallmedia --volume /Volumes/Untitled --nointeraction --downloadassets
</code></dl>
<br/><br/>
Step 4: Use the bootable macOS installer USB in a mac and press Option key when boot and use terminal to repair disk or filesystem. The reason to use High Sierra or above is that it can mount the new Apple File System (APFS).
<br /><br /> Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-65511754333817926522018-06-29T11:24:00.000+08:002018-06-29T18:02:16.406+08:00How to install CocoaPods for macOS 10.14 beta<br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># update gem
sudo gem update --system
# Operation not permitted error
# yes do it twice
sudo gem update --system
# install cocoapods
sudo gem install -n /usr/local/bin cocoapods
# install dependencies for project
cd ~/MyProject
pod install
# if re-clone CocoaPods repo spec
cd ~/.cocoapods/repos/
rm -fr master/
git clone --depth 1 https://github.com/CocoaPods/Specs.git master
</code></dl>
<br />
<br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-44442178127097242642018-06-17T19:55:00.001+08:002018-06-29T11:35:38.760+08:00How to install turicreate on macOS 10.14 betaInstall turicreate on macOS 10.14 beta 1
<br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># upgrade pip
# curl https://bootstrap.pypa.io/get-pip.py | sudo python
curl https://bootstrap.pypa.io/get-pip.py | python
# install packages
sudo pip install requests==2.18.4 turicreate==5.0b1
</code></dl>
<br />
<br />
(1) Test turicreate example - Image Classifier
<br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>mkdir -p $HOME/MLClassifier
cd $HOME/MLClassifier
# download dataset and cleanup
curl -L -o dataset.zip https://drive.google.com/uc?id=1ZLigrn7YcETalcj2qK6UqXceDdOV3244&export=download
unzip dataset.zip
rm -fr __MACOSX; rm dataset/.DS_Store dataset/*/.DS_Store
# create python script
cat > classifier.py << 'EOF'
import turicreate as turi
# load images from dataset folder
url = "dataset/"
data = turi.image_analysis.load_images(url)
# define image categories
data["foodType"] = data["path"].apply(lambda path: "Rice" if "rice" in path else "Soup")
# create sframe
data.save("rice_or_soup.sframe")
# preview dataset
data.explore()
# load sframe
dataBuffer = turi.SFrame("rice_or_soup.sframe")
# create training data using 90% of dataset
trainingBuffers, testingBuffers = dataBuffer.random_split(0.9)
# create model
model = turi.image_classifier.create(trainingBuffers, target="foodType", model="squeezenet_v1.1", max_iterations=100)
# Alternate model use ResNet-50
# model = turi.image_classifier.create(trainingBuffers, target="foodType", model="resnet-50")
# evaluate model
evaluations = model.evaluate(testingBuffers)
print evaluations["accuracy"]
# save model
model.save("rice_or_soup.model")
model.export_coreml("RiceSoupClassifier.mlmodel")
EOF
#run script
python classifier.py
</code></dl>
<br />
<br />
(2) Test turicreate example - Logistic Regression
<br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>mkdir -p $HOME/LGClassifier
cd $HOME/LGClassifier
# create python script
cat > classifier.py << 'EOF'
import turicreate as turi
data = turi.SFrame('http://static.turi.com/datasets/regression/yelp-data.csv')
data['is_good'] = data['stars'] >= 3
# create sframe
data.save("yelp.sframe")
# preview dataset
#data.show()
# load sframe
dataBuffer = turi.SFrame("yelp.sframe")
# create training data using 80% of dataset
train_data, test_data = dataBuffer.random_split(0.8)
# create model
model=turi.logistic_classifier.create(train_data, target='is_good',
features = ['user_avg_stars',
'business_avg_stars',
'user_review_count',
'business_review_count',
'city',
'categories_dict'],
max_iterations=200)
print model
# save predictions
predictions = model.classify(test_data)
print predictions
# evaluate model
evaluations = model.evaluate(test_data)
print "Accuracy : %s" % evaluations["accuracy"]
print "Confusion Matrix : \n%s" % evaluations["confusion_matrix"]
EOF
#run script
python classifier.py
</code></dl>
<br />
<br />
(3) Some data manipulation tips when preparing training data
<br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># remove the quotes (replace the number with the quotes with the number without them) in csv file, typically "save as CSV" from excel file.
# for example, "222,267.87","455,365.44",... convert to 222267.87,455365.44,...
#In shell script
cat exceldata.csv | perl -p -e 's/,(?=[\d,.]*\d")//g and s/"(\d[\d,.]*)"/\1/g' > dataset.csv
# use map, lambda and zip functions when convert and compute numeric data from 2 data columns
#In python script
import math
data['rate'] = map(lambda (x,y): 0 if x is None or y is None else (0 if math.isnan(x) or math.isnan(y) or math.isinf(y) or x==0 else (999999 if math.isinf(x) or y==0 else 999999 if x/y > 999999 else x/y)) , zip(data['OS'], data['Total Amount']))
# replace training data when values are <b>inf</b>(infinity) or <b>nan</b>(Not A Number) in 'amount' column
#In python script
import math
train_data['amount'] = train_data['amount'].apply(lambda x: 0 if math.isnan(x) else x)
train_data['amount'] = train_data['amount'].apply(lambda x: 999 if math.isinf(x) else x)
# or use nested if else
#In python script
import math
train_data['amount'] = train_data['amount'].apply(lambda x: 0 if math.isnan(x) else (999 if math.isinf(x) else x ))
print train_data['amount'].summary()
# remove rows in training data with <b>inf</b>(infinity) or <b>nan</b>(Not A Number) values in 'amount' column
#In python script
import math
train_data = train_data[train_data['amount'].apply(lambda x: 0 if math.isinf(x) or math.isnan(x) else 1)]
# SFrame methods but beware, some of the methods are not working
https://apple.github.io/turicreate/docs/api/generated/turicreate.SFrame.html
# Other SFrame data manipulation examples
https://github.com/apple/turicreate/blob/master/userguide/sframe/data-manipulation.md
</code></dl>
<br />
<br />
(4) Some data examination tips
<br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># summary
print train_data['amount'].summary()
# crosstab
import pandas as pd
pd.crosstab(data["Rating"], data["is_bad"], margins=True)
# custom frequency count for 'amount' column
import pandas as pd
pd.crosstab(train_data['amount'].apply(lambda x: " 0-10" if x <=10 else ("10-20" if x <=20 else ("20-30" if x <=30 else ("30-40" if x <=30 else ("40-50" if x <=50 else ">50"))))), "Count")
</code></dl>
<br />
<br />Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-5539080117393505382018-06-09T18:53:00.000+08:002018-06-10T11:42:55.872+08:00Playground examples for XCode 10 Beta 1<span style="font-size: large;">Playground Support for iOS12 and Swift 4.2</span><br />
<dl class="codebox">
<dt>iOS.playground <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>import UIKit
import PlaygroundSupport
//: **Markup**
//: ### Define UIView
class MyView : UIView {
@objc public func changeTitle(_ sender:UIButton!) {
sender.setTitle("Welcome to WWDC2018", for: [])
}
}
let myView = MyView(frame: CGRect(x:0, y:0, width:500, height:500))
var button = UIButton(type: .system)
button.frame = CGRect(x:100, y:100, width:300, height:200)
button.setTitle("Hi Press me!", for: [])
button.tintColor = .blue
button.setTitleColor(.orange, for: [])
button.addTarget(myView, action: #selector(MyView.changeTitle(_:)), for: .touchUpInside)
myView.addSubview(button)
PlaygroundPage.current.liveView = myView
</code></dl>
<br /><br /><br /><br />
<span style="font-size: large;">Playground Icon Drawings in iOS and Swift 4.2</span><br />
<dl class="codebox">
<dt>macOS.playground <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>import UIKit
//: Define IconView
class IconView: UIView {
override func draw(_ rect: CGRect) {
drawRawBackgroundWithBaseColor(strokeColor: UIColor.orange, backgroundRectangle: self.bounds)
let textAttributes: [NSAttributedString.Key : Any] = [
NSAttributedString.Key.foregroundColor: UIColor.red,
NSAttributedString.Key.font: UIFont.systemFont(ofSize: 32.0)]
let FString: String = "Hello World"
let distanceX: CGFloat = -12.0
let distanceY: CGFloat = 0.0
let centerX = self.bounds.midX
let centerY = self.bounds.midY
FString.draw(at: CGPoint(x:centerX+distanceX, y:centerY+distanceY), withAttributes: textAttributes)
}
}
func drawRawBackgroundWithBaseColor(strokeColor: UIColor, backgroundRectangle:CGRect) {
let lineWidth = backgroundRectangle.width/36.0
let cornerRadius = backgroundRectangle.width/16.0
let tileRectangle = backgroundRectangle.insetBy(dx: lineWidth/2.0, dy: lineWidth/2.0)
// Stroke Drawing
let strokePath = UIBezierPath(roundedRect:tileRectangle, cornerRadius:cornerRadius)
strokeColor.setStroke()
strokePath.lineWidth = lineWidth
strokePath.stroke()
// Draw an ellipse
let ovalPath = UIBezierPath(ovalIn: backgroundRectangle.insetBy(dx: lineWidth*1.5, dy: lineWidth*1.5))
UIColor.blue.setStroke()
ovalPath.lineWidth = lineWidth
ovalPath.stroke()
let context:CGContext = UIGraphicsGetCurrentContext()!
context.setFillColor(UIColor.green.cgColor)
context.addRect(CGRect(x: 100.0, y: 100.0, width: 60.0, height: 60.0))
context.fillPath()
}
//: Instantiate the UIView
let rect = CGRect(x: 0.0, y: 0.0, width: 420.0, height: 320.0)
let icon = IconView(frame: rect)
icon.backgroundColor = UIColor.clear
</code></dl>
<br /><br /><br /><br />
<span style="font-size: large;">CreateML for macOS 10.14 Beta 1 (requires macOS 10.14 Mojave)</span><br />
<dl class="codebox">
<dt>macOS.playground <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>import Cocoa
import CreateML
//: Specify Data
/*
Input as CSV
mycsv.csv:
beds,baths,squareFt, price
2,2,2000,400000
4,3,2500,500000
3,2,1800,450000
3,2,1500,300000
let houseData = try MLDataTable(contentsOf: URL(fileURLWithPath: "mycsv.csv"))
*/
//: Input as dictionary
let mydata : [String: MLDataValueConvertible] = [
"beds": [2,4,3,3],
"baths": [2,3,2,2],
"squareFt": [2000,2500,1800,1500],
"price": [400000,500000,450000,300000]
]
let houseData = try MLDataTable(dictionary:mydata)
let (trainingData, testData) = houseData.randomSplit(by: 0.8, seed: 0)
//: Create Model
let pricer = try MLRegressor(trainingData: houseData, targetColumn: "price")
//: Evaluate Model
let evalator = pricer.evaluation(on: testData)
print(pricer)
//: Save Model
try pricer.write(to:URL(fileURLWithPath: "HousePricer.mlmodel"))
</code></dl>
<br /><br /><br /><br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-14919686527629547192017-07-04T18:31:00.002+08:002019-06-07T15:59:13.685+08:00How to fetch WWDC 2017/2018/2019 Video Subtitle to SRT format<span style="font-size: large;">Create and Run this script wwdc_fetch_srt.sh to fetch WWDC2019 subtitle</span><br />
Reference : https://github.com/wsvn53/wwdc2016-subtitles<br />
<br />
<dl class="codebox">
<dt>wwdc_fetch_srt.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/bin/sh
# @Author: Ethan
# @Date: 2016-06-22 14:10:53
# @Last Modified by: javacom
# @Last Modified time: 2019-06-06
WWDC_YEAR=2019;
# change to 2017/2018 and also works for WWDC2017 or WWDC2018
WWDC_SESSION_PREFIX=https://developer.apple.com/videos/play/wwdc$WWDC_YEAR;
WWDC_LOCAL_DIR=$(basename $WWDC_SESSION_PREFIX);
detect_video_m3u8 () {
local session_url=$WWDC_SESSION_PREFIX/$SESSION_ID/;
local session_html=$(curl -s $session_url);
local video_url=$(echo "$session_html" | grep .m3u8 | grep $SESSION_ID | head -n1 | sed "s#.*\"\(https://.*m3u8\)\".*#\1#");
echo "$session_html" | grep .mp4 | grep $SESSION_ID | sed "s#.*\"\(https://.*mp4\).*\".*#\1#" | while read mp4_url; do
local mp4_filename=$(basename $mp4_url | cut -d. -f1);
local srt_filename=$mp4_filename.srt;
echo "> Subtitle local: $WWDC_LOCAL_DIR/$srt_filename" >&2;
> $WWDC_LOCAL_DIR/$srt_filename;
done
echo "$video_url";
echo "> Video: $video_url" >&2;
}
detect_subtitle_m3u8 () {
local video_url=$1;
local subtitle_uri=$(curl -s $video_url | grep "LANGUAGE=\"eng\"" | sed "s#.*URI=\"\(.*\)\"#\1#");
local subtitle_url=$subtitle_uri;
[[ "$subtitle_uri" != http* ]] && {
subtitle_url=$(dirname $video_url)/$subtitle_uri;
}
echo "$subtitle_url";
echo "> Subtitle: $subtitle_url" >&2;
}
download_subtitle_contents () {
local subtitle_url=$1;
echo "> Downloading... "
local subtitle_base_url=$(dirname $subtitle_url);
curl -s $subtitle_url | grep "webvtt" | while read webvtt; do
local subtitle_webvtt=$subtitle_base_url/$webvtt;
#echo "- get $subtitle_webvtt";
local subtitle_content=$(curl -s $subtitle_webvtt);
ls $WWDC_LOCAL_DIR/"$SESSION_ID"_* | while read srt_file; do
echo "$subtitle_content" >> $srt_file;
done
done
}
main () {
[ ! -d $WWDC_LOCAL_DIR ] && {
mkdir $WWDC_LOCAL_DIR;
}
curl -s $WWDC_SESSION_PREFIX | grep /videos/play/wwdc$WWDC_YEAR | sed "s#.*/videos/play/wwdc$WWDC_YEAR/\([0-9]\{3\}\).*#\1#" | sort | uniq | while read SESSION_ID; do
#echo "SESSION_ID is" $SESSION_ID
local video_url=$(detect_video_m3u8 $SESSION_ID);
local subtitle_url=$(detect_subtitle_m3u8 $video_url);
download_subtitle_contents $subtitle_url;
done
}
main;
</code></dl>
<br /><br /><br /><br />
<span style="font-size: large;">Run this shell script to format as SRT subtitle</span><br />
<br />
<dl class="codebox">
<dt>shellscript.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>WWDC_YEAR=2019;
# change to 2017/2018 and also works for WWDC2017 or WWDC2018
cd wwdc$WWDC_YEAR
mkdir -p sd
mkdir -p hd
for i in ???_sd_*.srt; do sed -e '/WEBVTT/d;/X-TIMESTAMP/d;' $i | awk '/^[0-9]{2}:[0-9]{2}:/ {seen[$0]++; skipduplicated=0} {if (seen[$0]>1) skipduplicated=1; if (!skipduplicated) print $0}' | awk -v RS="" '{gsub("\n", "-Z"); print}' | awk '$0 !~/^WEB/ {print $0}' | uniq | awk '{printf "\n%s-Z%s", NR,$0 }' | awk -v ORS="\n\n" '{gsub("-Z", "\n"); print}' | sed -e 's/.A:middle$//g;s/&gt;/>/g;s/&lt;/</g;1,2d;' > sd/$i; done
for i in ???_hd_*.srt; do sed -e '/WEBVTT/d;/X-TIMESTAMP/d;' $i | awk '/^[0-9]{2}:[0-9]{2}:/ {seen[$0]++; skipduplicated=0} {if (seen[$0]>1) skipduplicated=1; if (!skipduplicated) print $0}' | awk -v RS="" '{gsub("\n", "-Z"); print}' | awk '$0 !~/^WEB/ {print $0}' | uniq | awk '{printf "\n%s-Z%s", NR,$0 }' | awk -v ORS="\n\n" '{gsub("-Z", "\n"); print}' | sed -e 's/.A:middle$//g;s/&gt;/>/g;s/&lt;/</g;1,2d;' > hd/$i; done
</code></dl>
<br /><br /><br /><br />
<span style="font-size: large;">Run this script wwdc_fetch_mp4.sh to download all mp4 (HD and SD) videos</span><br />
<br />
<dl class="codebox">
<dt>wwdc_fetch_mp4.sh  <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/bin/sh
# @Last Modified by: javacom
# @Last Modified time: 2019-06-06
WWDC_YEAR=2019;
# change to 2017/2018 and also works for WWDC2017 or WWDC2018
WWDC_SESSION_PREFIX=https://developer.apple.com/videos/play/wwdc$WWDC_YEAR;
WWDC_LOCAL_DIR=$(basename $WWDC_SESSION_PREFIX);
download_mp4_video () {
local session_url=$WWDC_SESSION_PREFIX/$SESSION_ID/;
local session_html=$(curl -s $session_url);
local video_url=$(echo "$session_html" | grep .m3u8 | grep $SESSION_ID | head -n1 | sed "s#.*\"\(https://.*m3u8\)\".*#\1#");
echo "$session_html" | grep .mp4 | grep $SESSION_ID | sed "s#.*\"\(https://.*mp4\).*\".*#\1#" | while read mp4_url; do
local mp4_filename=$(basename $mp4_url);
if [ -e $WWDC_LOCAL_DIR/$mp4_filename ]
then
echo "> MP4 already existed : $WWDC_LOCAL_DIR/$mp4_filename" >&2;
echo "> To resume broken download use curl -C - --connect-timeout 1200 -o $WWDC_LOCAL_DIR/$mp4_filename $mp4_url" >&2;
echo " " >&2;
else
echo "> MP4 Downloading... : $mp4_url" >&2;
curl --connect-timeout 120 -o $WWDC_LOCAL_DIR/$mp4_filename $mp4_url
fi
done
}
main () {
[ ! -d $WWDC_LOCAL_DIR ] && {
mkdir $WWDC_LOCAL_DIR;
}
curl -s $WWDC_SESSION_PREFIX | grep /videos/play/wwdc$WWDC_YEAR | sed "s#.*/videos/play/wwdc$WWDC_YEAR/\([0-9]\{3\}\).*#\1#" | sort | uniq | while read SESSION_ID; do
download_mp4_video $SESSION_ID;
done
}
main;
</code></dl><br /><br />
<span style="font-size: large;">One liner version wwdc2019_fetch_mp4.sh to download all mp4 videos</span><br />
<br />
<dl class="codebox">
<dt>wwdc2019_fetch_mp4.sh  <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># one liner for hd videos download
# change to 2017/2018 and also works for WWDC2017 or WWDC2018
WWDCYEAR="wwdc2019"; for i in `curl -s https://developer.apple.com/videos/$WWDCYEAR/ | grep -o '<a href="/videos/play/'"$WWDCYEAR"'/[0-9]*' | cut -d '"' -f2 | sort | uniq`; do video_url=$(curl -s https://developer.apple.com${i} | grep -o 'http.*_hd_.*.mp4'); if [ ! -z "$video_url" ]; then mp4_filename=$(basename $video_url); if [ -e $mp4_filename ]; then echo "skipping $mp4_filename"; else echo "Downloading ... $mp4_filename";curl --connect-timeout 120 -O $video_url; fi; fi; done
# one liner for sd videos download
WWDCYEAR="wwdc2019"; for i in `curl -s https://developer.apple.com/videos/$WWDCYEAR/ | grep -o '<a href="/videos/play/'"$WWDCYEAR"'/[0-9]*' | cut -d '"' -f2 | sort | uniq`; do video_url=$(curl -s https://developer.apple.com${i} | grep -o 'http.*_sd_.*.mp4'); if [ ! -z "$video_url" ]; then mp4_filename=$(basename $video_url); if [ -e $mp4_filename ]; then echo "skipping $mp4_filename"; else echo "Downloading ... $mp4_filename";curl -O $video_url; fi; fi; done
</code></dl>
<br /><br /><br /><br />
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-76247505985933492652017-06-07T18:59:00.003+08:002017-07-31T13:42:19.643+08:00How to train dataset in python and convert to CoreML model for iOS11Reference http://machinelearningmastery.com/machine-learning-in-python-step-by-step/<br />
<br />
Environment : macOS 10.12.4<br />
matplotlib==2.0.0<br />
numpy==1.12.1<br />
pandas==0.19.2<br />
scikit-learn==0.18.1<br />
scipy==0.19.0<br />
six==1.10.0<br />
sklearn==0.18.1<br />
coremltools==0.3.0<br />
protobuf==3.3.0<br />
<br />
<span style="font-size: large;">Upgrade the pip and install the following python packages</span><br />
<dl class="codebox">
<dt>shellscript.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>pip install --upgrade pip
sudo -H pip install numpy scipy matplotlib pandas sklearn coremltools protobuf
</code></dl>
<br />
<br /><a href="https://3.bp.blogspot.com/-XArCn4xLk0Q/WTjFybLzE6I/AAAAAAAAVts/a-MH-mBAn9QbLES5f5mwFw3OggLzmpitACLcB/s1600/convert_to_ml.png" imageanchor="1" ><img border="0" src="https://3.bp.blogspot.com/-XArCn4xLk0Q/WTjFybLzE6I/AAAAAAAAVts/a-MH-mBAn9QbLES5f5mwFw3OggLzmpitACLcB/s400/convert_to_ml.png" width="400" height="208" data-original-width="1600" data-original-height="832" /></a><br />
<span style="font-size: large;">Convert to Core ML</span> Run the following python code to show machine learning in python step by step and finally generate iris_lr.mlmodel
<br />
<dl class="codebox">
<dt>iris_learn.py <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/usr/bin/env python
# Check the versions of libraries
# Python version
import sys
print('Python: {}'.format(sys.version))
# scipy
import scipy
print('scipy: {}'.format(scipy.__version__))
# numpy
import numpy
print('numpy: {}'.format(numpy.__version__))
# matplotlib
import matplotlib
print('matplotlib: {}'.format(matplotlib.__version__))
# pandas
import pandas
print('pandas: {}'.format(pandas.__version__))
# scikit-learn
import sklearn
print('sklearn: {}'.format(sklearn.__version__))
# Load libraries
import pandas
from pandas.tools.plotting import scatter_matrix
import matplotlib.pyplot as plt
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
# Load dataset
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"
names = ['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'class']
dataset = pandas.read_csv(url, names=names)
# shape
print(dataset.shape)
# head
print(dataset.head(20))
# descriptions
print(dataset.describe())
# class distribution
print(dataset.groupby('class').size())
# box and whisker plots
dataset.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False)
plt.suptitle("Box and Whisker Plots for inputs")
plt.show()
# histograms
dataset.hist()
plt.suptitle('Histograms for inputs')
plt.show()
# scatter plot matrix
scatter_matrix(dataset)
plt.suptitle('Scatter Plot Matrix for inputs')
plt.show()
# Split-out validation dataset
array = dataset.values
X = array[:,0:4]
Y = array[:,4]
validation_size = 0.20
seed = 7
X_train, X_validation, Y_train, Y_validation = model_selection.train_test_split(X, Y, test_size=validation_size, random_state=seed)
# Test options and evaluation metric
seed = 7
scoring = 'accuracy'
# Spot Check Algorithms
models = []
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))
# evaluate each model in turn
results = []
names = []
for name, model in models:
kfold = model_selection.KFold(n_splits=10, random_state=seed)
cv_results = model_selection.cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring)
results.append(cv_results)
names.append(name)
msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
print(msg)
# Compare Algorithms
fig = plt.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()
# Make predictions on validation dataset
knn = KNeighborsClassifier()
knn.fit(X_train, Y_train)
predictions = knn.predict(X_validation)
print(accuracy_score(Y_validation, predictions))
print(confusion_matrix(Y_validation, predictions))
print(classification_report(Y_validation, predictions))
print("Make predictions on LogisticRegression Model")
model = LogisticRegression()
model.fit(X_train, Y_train)
predictions = model.predict(X_validation)
print(accuracy_score(Y_validation, predictions))
print(confusion_matrix(Y_validation, predictions))
print(classification_report(Y_validation, predictions))
# print prediction results on test data
for i, prediction in enumerate(predictions):
print 'Predicted: %s, Target: %s %s' % (prediction, Y_validation[i], '' if prediction==Y_validation[i] else '(WRONG!!!)')
<span style="color: red;">#convert and save scikit.learn model</span>
#support LogisticRegression of scikit.learn
print("Convert LogisticRegression Model to coreml model")
import coremltools
coreml_model = coremltools.converters.sklearn.convert(model,
["sepal-length", "sepal-width", "petal-length", "petal-width"],
"class")
#set model metadata
coreml_model.author = 'Author'
coreml_model.license = 'BSD'
coreml_model.short_description = 'LogisticRegression on Iris flower data set'
#set features description manually
coreml_model.input_description['sepal-length'] = 'Sepal Length in centimetres'
coreml_model.input_description['sepal-width'] = 'Sepal Width in centimetres'
coreml_model.input_description['petal-length'] = 'Petal Length in centimetres'
coreml_model.input_description['petal-width'] = 'Petal Width in centimetres'
#set the ouput description
coreml_model.output_description['class'] = 'Distinguish the species'
#save the model
coreml_model.save('iris_lr.mlmodel')
from coremltools.models import MLModel
model = MLModel('iris_lr.mlmodel')
#get the spec of the model
print(model.get_spec())
</code></dl>
<br />
<br />
<span style="font-size: large;">Download Xcode 9 beta and the sample code from Apple</span><br />
<br />
https://docs-assets.developer.apple.com/published/51ff0c1668/IntegratingaCoreMLModelintoYourApp.zip<br />
Modify it and add the model to the xcode project<br /><a href="https://4.bp.blogspot.com/-8CsWcouY9nA/WTjlln1d2dI/AAAAAAAAVt8/xqpoFCfvxBkayzqLscKF0vwfVYhAoC2WACLcB/s1600/irismodel.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/-8CsWcouY9nA/WTjlln1d2dI/AAAAAAAAVt8/xqpoFCfvxBkayzqLscKF0vwfVYhAoC2WACLcB/s400/irismodel.png" width="400" height="307" data-original-width="1388" data-original-height="1064" /></a>
<br />
<br />
Try the new refactoring tool in Xcode 9. It is amazing.<br />
<a href="https://4.bp.blogspot.com/--iwkg68yGks/WTn7jbgclcI/AAAAAAAAVuM/bu8BOREndhEHPBn0i6lhHHM1igCHFG15gCLcB/s1600/refactoring.png" imageanchor="1" ><img border="0" src="https://4.bp.blogspot.com/--iwkg68yGks/WTn7jbgclcI/AAAAAAAAVuM/bu8BOREndhEHPBn0i6lhHHM1igCHFG15gCLcB/s400/refactoring.png" width="400" height="313" data-original-width="1550" data-original-height="1212" /></a>
<br /><br />
<span style="font-size: large;">Train data using Neural Network Model Keras</span><br />
Reference : http://machinelearningmastery.com/5-step-life-cycle-neural-network-models-keras/<br />
<br />
<dl class="codebox">
<dt>shellscript.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
# download training data
curl -O http://archive.ics.uci.edu/ml/machine-learning-databases/pima-indians-diabetes/pima-indians-diabetes.data
# install and activate virtual environment and install necessary python packages
# use deactivate to stop the python virtual env
sudo -H pip install --upgrade virtualenv
virtualenv --system-site-packages ~/tensorflow
source ~/tensorflow/bin/activate
# macOS, CPU only non-optimised, Python 2.7:
# https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.1.0-py2-none-any.whl
# macOS, GPU enabled, Python 2.7:
# https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-1.1.0-py2-none-any.whl
# or find optimised wheel files from the community https://github.com/yaroslavvb/tensorflow-community-wheels/issues
# this optimised one (SSE4.1,SSE4.2,AVX,AVX2,FMA) works for Python 2.7 macOS 10.12 Tensoflow 1.1.0 CPU https://github.com/fdalvi/tensorflow-builds
# this one works for GeForce GT 650M GPU and CPU (SSE4.2, AVX) and CUDA 8.0, and cuDNN v5.1 https://github.com/bodak/tensorflow-wheels/releases/tag/v1.1.0_27
# instruction to build your own python package https://ctmakro.github.io/site/on_learning/tf1c.html
# suppose, install the official non-optimised wheel file as below
pip install --upgrade https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.1.0-py2-none-any.whl
pip install coremltools protobuf
pip install keras==1.2.2 h5py
</code></dl>
<br />
<span style="font-size: large;">Convert to Core ML</span> Run the following python code in virtual environment (tensorflow) to generate pima_keras.mlmodel
<br />
<dl class="codebox">
<dt>keras_learn.py <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#!/usr/bin/env python
from keras.models import Sequential
from keras.layers import Dense
import numpy
# fix random seed for reproducibility
numpy.random.seed(7)
# load pima indians dataset
#dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
dataset = numpy.loadtxt("pima-indians-diabetes.data", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]
# create model
model = Sequential()
model.add(Dense(12, input_dim=8, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Fit the model
#model.fit(X, Y, epochs=150, batch_size=10)
model.fit(X, Y, 10, 150) # parameters change to keras 1.2.2
# evaluate the model
scores = model.evaluate(X, Y)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
<span style="color: red;">#convert and save keras model</span>
model.save('pima.h5')
print("Convert Model to coreml model")
import coremltools
coreml_model = coremltools.converters.keras.convert('pima.h5')
#set model metadata
coreml_model.author = 'Author'
coreml_model.license = 'BSD'
coreml_model.short_description = 'pima-indians-diabetes'
#save the model
coreml_model.save('pima_keras.mlmodel')
from coremltools.models import MLModel
mlmodel = MLModel('pima_keras.mlmodel')
#get the spec of the model
print(mlmodel.get_spec())
</code></dl>
<br/><br/>
Note: <a href="https://pypi.python.org/pypi/coremltools/0.3.0">coremltools</a> require python 2.7 (not for 3.x) and supports keras==1.2.2 with Tenorflow (1.0.x, 1.1.x) only. <a href="https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-1.1.0-py2-none-any.whl">Tenorflow_gpu</a> requires <a href="https://developer.nvidia.com/cuda-downloads">Nvidia Cuda 8.0</a> and <a href="https://developer.nvidia.com/rdp/cudnn-download">cuDNN v5.1</a> (which also requires macOS 10.11/10.12) but recent models of Mac are all bundled AMD GPUs. Unless you could get an old Mac Pro with upgraded Nvidia GPU with at least 4 GB of video RAM, it is better to stay with Mac CPU i7 or get a Linux machine for data training purpose only.<br /><br />
Hardware reference for Linux : https://www.oreilly.com/learning/build-a-super-fast-deep-learning-machine-for-under-1000<br /><br />
For Windows PC, tensorflow/tensorflow_gpu is only available for Python 3.5 and 64 bits only as below. As current coremltools keras convertors are not compatible with python 3.5, so direct conversion is not available in PC yet.<br />
https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.1.0-cp35-cp35m-win_amd64.whl<br />
https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-1.1.0-cp35-cp35m-win_amd64.whl<br />
<br/><br/>
<br />
<span style="font-size: large;">keras-inception-test</span> Run the following python code in virtual environment (tensorflow) to test Keras Inceptionv3 model. This will download the trained Inception V3 weights from https://github.com/fchollet/deep-learning-models/releases/download/v0.2/inception_v3_weights_tf_dim_ordering_tf_kernels.h5
<br />
<dl class="codebox">
<dt>shellscript.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>git clone git://github.com/vml-ffleschner/coremltools-keras-inception-test
cd coremltools-keras-inception-test/
# based on the created virtualenv in ~/tensorflow as above
source ~/tensorflow/bin/activate
# additional installation of packages
pip install olefile pillow
#Add
coreml_model.author = 'Author'
coreml_model.license = 'BSD'
coreml_model.short_description = 'Image InceptionV3 model'
coreml_model.save('Inceptionv3.mlmodel')
print("CoreML model file Created")
#After
#print("CoreML Converted")
#in playground.py
# note : coreml_model.predict requires macOS 10.13 High Sierra
python playground.py
</code></dl>
<br/><br/>
Install tensorflow 1.1.0 library for Java is here <br/>
<dl class="codebox"><dt>shellscript.sh <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>curl -O https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-1.1.0.jar
curl -O https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow_jni-cpu-darwin-x86_64-1.1.0.tar.gz<br/>
# install
tar xzvf libtensorflow_jni-cpu-darwin-x86_64-1.1.0.tar.gz -C ./jni
# compile and run HelloTF
javac -cp libtensorflow-1.1.0.jar HelloTF.java
java -cp libtensorflow-1.1.0.jar:. -Djava.library.path=./jni HelloTF
</code></dl><br/><br/>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-6245963027152906987.post-50949064451927163342016-11-28T02:36:00.005+08:002021-06-16T18:21:32.119+08:00How to build c++ libraries (boost & QuantLib) for iOS XCode 8/9It is possible to skip the armv7, armv7s and i386 architectures if these binaries are not required. Just modify the relevant part of the scripts below<br /><br />
<br /><br />
Updated Note<br />
The macOS 10.14 SDK no longer contains support for compiling 32-bit applications. If developers need to compile for i386, Xcode 9.4 or earlier is required.<br />
<br /><br />
<b>(1) Download boost 1.62.0 and extract source</b><br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># download boost source
cd $HOME/Downloads
curl -O https://nchc.dl.sourceforge.net/project/boost/boost/1.62.0/boost_1_62_0.tar.bz2
tar xjvf boost_1_62_0.tar.bz2
cd boost_1_62_0
# create the following user-config.jam at ~/Downloads/boost_1_62_0/tools/build/src/
cat > tools/build/src/user-config.jam << 'EOF'
# ------------------
# darwin iphone configuration.
# ------------------
local XCODE_ROOT = /Applications/Xcode.app/Contents/Developer ;
# xcrun --sdk iphoneos --show-sdk-path
local IPHONESDK = iPhoneOS.sdk ;
local MACOSXSDK = MacOSX.sdk ;
# xcrun --show-sdk-path
# xcrun --sdk iphonesimulator --show-sdk-path
local IPHONESIMULATORSDK = iPhoneSimulator.sdk ;
using darwin : armv7
:
$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
:
<striper>$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip
<root>$(XCODE_ROOT)/Platforms/iPhoneOS.platform/Developer/SDKs/$(IPHONESDK)
<compileflags>"-isysroot $(XCODE_ROOT)/Platforms/iPhoneOS.platform/Developer/SDKs/$(IPHONESDK)"
<compileflags>"-arch armv7"
<compileflags>-fexceptions
<compileflags>-frtti
<compileflags>-fpic
<compileflags>-DBOOST_SP_USE_SPINLOCK
<compileflags>-std=c++11
<compileflags>-stdlib=libc++
<compileflags>-miphoneos-version-min=7.0
<compileflags>-fvisibility=hidden
<compileflags>-fvisibility-inlines-hidden
<compileflags>-O2
<compileflags>-g
<compileflags>-Os
<compileflags>-Wno-long-long
<compileflags>-Wno-missing-field-initializers
<compileflags>-Wno-unused-variable
<linkflags>-stdlib=libc++
<architecture>arm
<address-model>32
<target-os>iphone
;
using darwin : armv7s
:
$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
:
<striper>$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip
<root>$(XCODE_ROOT)/Platforms/iPhoneOS.platform/Developer/SDKs/$(IPHONESDK)
<compileflags>"-isysroot $(XCODE_ROOT)/Platforms/iPhoneOS.platform/Developer/SDKs/$(IPHONESDK)"
<compileflags>"-arch armv7s"
<compileflags>-fexceptions
<compileflags>-frtti
<compileflags>-fpic
<compileflags>-DBOOST_SP_USE_SPINLOCK
<compileflags>-std=c++11
<compileflags>-stdlib=libc++
<compileflags>-miphoneos-version-min=7.0
<compileflags>-fvisibility=hidden
<compileflags>-fvisibility-inlines-hidden
<compileflags>-O2
<compileflags>-g
<compileflags>-Os
<compileflags>-Wno-long-long
<compileflags>-Wno-missing-field-initializers
<compileflags>-Wno-unused-variable
<linkflags>-stdlib=libc++
<architecture>arm
<address-model>32
<target-os>iphone
;
using darwin : arm64
:
$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
:
<striper>$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip
<root>$(XCODE_ROOT)/Platforms/iPhoneOS.platform/Developer/SDKs/$(IPHONESDK)
<compileflags>"-isysroot $(XCODE_ROOT)/Platforms/iPhoneOS.platform/Developer/SDKs/$(IPHONESDK)"
#lt;root>$(XCODE_ROOT)/Platforms/MacOSX.platform/Developer/SDKs/$(MACOSXSDK)
#lt;compileflags>"-isysroot $(XCODE_ROOT)/Platforms/MacOSX.platform/Developer/SDKs/$(MACOSXSDK)"
<compileflags>"-arch arm64"
<compileflags>-fexceptions
<compileflags>-frtti
<compileflags>-fpic
<compileflags>-DBOOST_SP_USE_SPINLOCK
<compileflags>-std=c++11
<compileflags>-stdlib=libc++
<compileflags>-miphoneos-version-min=7.0
#<compileflags>-mmacosx-version-min=10.9
<compileflags>-fvisibility=hidden
<compileflags>-fvisibility-inlines-hidden
<compileflags>-O2
<compileflags>-g
<compileflags>-Os
<compileflags>-Wno-long-long
<compileflags>-Wno-missing-field-initializers
<compileflags>-Wno-unused-variable
<linkflags>-stdlib=libc++
<architecture>arm
<address-model>64
<target-os>iphone
;
using darwin : i386
:
$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
:
<striper>$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip
<root>$(XCODE_ROOT)/Platforms/iPhoneSimulator.platform/Developer/SDKs/$(IPHONESIMULATORSDK)
<compileflags>"-arch i386"
<compileflags>"-isysroot $(XCODE_ROOT)/Platforms/iPhoneSimulator.platform/Developer/SDKs/$(IPHONESIMULATORSDK)"
<compileflags>-fexceptions
<compileflags>-frtti
<compileflags>-fpic
<compileflags>-DBOOST_SP_USE_SPINLOCK
<compileflags>-std=c++11
<compileflags>-stdlib=libc++
<compileflags>-miphoneos-version-min=7.0
<compileflags>-fvisibility=hidden
<compileflags>-fvisibility-inlines-hidden
<compileflags>-O2
<compileflags>-g
<compileflags>-Os
<compileflags>-Wno-long-long
<compileflags>-Wno-missing-field-initializers
<compileflags>-Wno-unused-variable
<linkflags>-stdlib=libc++
<architecture>x86
<address-model>32
<target-os>iphone
;
using darwin : x86_64
:
$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
:
<striper>$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip
<root>$(XCODE_ROOT)/Platforms/iPhoneSimulator.platform/Developer/SDKs/$(IPHONESIMULATORSDK)
<compileflags>"-isysroot $(XCODE_ROOT)/Platforms/iPhoneSimulator.platform/Developer/SDKs/$(IPHONESIMULATORSDK)"
#lt;root>$(XCODE_ROOT)/Platforms/MacOSX.platform/Developer/SDKs/$(MACOSXSDK)
#lt;compileflags>"-isysroot $(XCODE_ROOT)/Platforms/MacOSX.platform/Developer/SDKs/$(MACOSXSDK)"
<compileflags>"-arch x86_64"
<compileflags>-fexceptions
<compileflags>-frtti
<compileflags>-fpic
<compileflags>-DBOOST_SP_USE_SPINLOCK
<compileflags>-std=c++11
<compileflags>-stdlib=libc++
<compileflags>-miphoneos-version-min=7.0
#<compileflags>-mmacosx-version-min=10.9
<compileflags>-fvisibility=hidden
<compileflags>-fvisibility-inlines-hidden
<compileflags>-O2
<compileflags>-g
<compileflags>-Os
<compileflags>-Wno-long-long
<compileflags>-Wno-missing-field-initializers
<compileflags>-Wno-unused-variable
<linkflags>-stdlib=libc++
<architecture>x86
<address-model>64
<target-os>iphone
;
EOF
</code></dl>
<br /><br />
<b>(2) patch </b>boost_1_62_0/tools/build/src/tools/darwin.jam<br />
<dl class="codebox">
<dt>~/Downloads/boost_1_62_0/tools/build/src/tools/darwin.jam <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
#change from
case arm :
{
if $(instruction-set) {
<span style="color: blue;">options = -arch$(_)$(instruction-set) ;</span>
} else {
<span style="color: blue;">options = -arch arm ;</span>
}
}
#to
case arm :
{
if $(instruction-set) {
<span style="color: blue;">options = -arch$(_)$(instruction-set) ;</span>
<span style="color: red;">} else if $(address-model) = 64 {
options = -arch arm64 ;</span>
} else {
<span style="color: blue;">options = -arch arm ;</span>
}
}
</code></dl>
<br /><br />
<b>(3) build boost libraries for iOS</b><br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
# bootstrap
cd $HOME/Downloads/boost_1_62_0
# apply this patch if bootstrap error: pushd tools/build; wget https://github.com/boostorg/build/commit/48e9017139dd94446633480661e5447c7e0d8b1b.patch; patch -p1 < 48e9017139dd94446633480661e5447c7e0d8b1b.patch; popd
./bootstrap.sh --with-libraries=atomic,chrono,date_time,exception,filesystem,graph,iostreams,math,program_options,random,regex,serialization,signals,system,test,thread,wave
# clean and build for iOS, j option is num of cores x 1.5
rm -fr ios-build
./b2 -j6 --build-dir=ios-build --stagedir=ios-build/armv7 toolset=darwin-armv7 architecture=arm instruction-set=armv7 address-model=32 target-os=iphone threading=multi link=static stage
./b2 -j6 --build-dir=ios-build --stagedir=ios-build/armv7s toolset=darwin-armv7s architecture=arm instruction-set=armv7s address-model=32 target-os=iphone threading=multi link=static stage
./b2 -j6 --build-dir=ios-build --stagedir=ios-build/arm64 toolset=darwin-arm64 architecture=arm address-model=64 target-os=iphone threading=multi link=static stage
./b2 -j6 --build-dir=ios-build --stagedir=ios-build/i386 toolset=darwin-i386 architecture=x86 address-model=32 target-os=iphone threading=multi link=static stage
./b2 -j6 --build-dir=ios-build --stagedir=ios-build/x86_64 toolset=darwin-x86_64 architecture=x86 address-model=64 target-os=iphone threading=multi link=static stage
# create libboost.a archive for each architecture
cd $HOME/Downloads/boost_1_62_0
cd ios-build
xcrun ar crus armv7/libboost.a boost/bin.v2/libs/*/build/darwin-armv7/release/instruction-set-armv7/link-static/target-os-iphone/threading-multi/*/*.o
xcrun ar crus armv7s/libboost.a boost/bin.v2/libs/*/build/darwin-armv7s/release/instruction-set-armv7s/link-static/target-os-iphone/threading-multi/*/*.o
xcrun ar crus arm64/libboost.a boost/bin.v2/libs/*/build/darwin-arm64/release/link-static/target-os-iphone/threading-multi/*/*.o
xcrun ar crus i386/libboost.a boost/bin.v2/libs/*/build/darwin-i386/release/link-static/target-os-iphone/threading-multi/*/*.o
xcrun ar crus x86_64/libboost.a boost/bin.v2/libs/*/build/darwin-x86_64/release/link-static/target-os-iphone/threading-multi/*/*.o
# create FAT libboost.a archive
mkdir -p lib
xcrun lipo \
-arch armv7 "armv7/libboost.a" \
-arch armv7s "armv7s/libboost.a" \
-arch arm64 "arm64/libboost.a" \
-arch i386 "i386/libboost.a" \
-arch x86_64 "x86_64/libboost.a" \
-output "lib/libboost.a" \
-create
# create link for include folder
mkdir include
cd include
ln -s ../../boost .
</code></dl>
<br />
<br />
<b>(4) Create boost Framework for iOS</b><br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
cd $HOME/Downloads/boost_1_62_0
cd ios-build
#Create the following shell script for build-boostframework.sh
cat > build-boostframework.sh << 'boostFrameworkEOF'
#!/bin/bash
# build-boostframework.sh
VERSION_TYPE=Alpha
FRAMEWORK_NAME=boost
FRAMEWORK_VERSION=A
FRAMEWORKDIR=.
FRAMEWORK_CURRENT_VERSION=1.62.0
FRAMEWORK_COMPATIBILITY_VERSION=1.62.0
FRAMEWORK_BUNDLE=${FRAMEWORKDIR}/${FRAMEWORK_NAME}.framework
rm -rf ${FRAMEWORK_BUNDLE}
mkdir -p ${FRAMEWORK_BUNDLE}
mkdir -p ${FRAMEWORK_BUNDLE}/Versions
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Resources
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Headers
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Documentation
ln -s $FRAMEWORK_VERSION ${FRAMEWORK_BUNDLE}/Versions/Current
ln -s Versions/Current/Headers ${FRAMEWORK_BUNDLE}/Headers
ln -s Versions/Current/Resources ${FRAMEWORK_BUNDLE}/Resources
ln -s Versions/Current/Documentation ${FRAMEWORK_BUNDLE}/Documentation
ln -s Versions/Current/$FRAMEWORK_NAME ${FRAMEWORK_BUNDLE}/$FRAMEWORK_NAME
FRAMEWORK_INSTALL_NAME=${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/${FRAMEWORK_NAME}
echo "Framework: Lipoing library into ${FRAMEWORK_INSTALL_NAME}"
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo \
-arch armv7 "armv7/libboost.a" \
-arch armv7s "armv7s/libboost.a" \
-arch arm64 "arm64/libboost.a" \
-arch i386 "i386/libboost.a" \
-arch x86_64 "x86_64/libboost.a" \
-output "${FRAMEWORK_INSTALL_NAME}" \
-create
echo "Framework: Copying includes..."
cp -r include/${FRAMEWORK_NAME}/* ${FRAMEWORK_BUNDLE}/Headers/
cat > ${FRAMEWORK_BUNDLE}/Resources/Info.plist <<InfoplistEOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>English</string>
<key>CFBundleExecutable</key>
<string>${FRAMEWORK_NAME}</string>
<key>CFBundleIdentifier</key>
<string>${FRAMEWORK_NAME}.org</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundlePackageType</key>
<string>FMWK</string>
<key>CFBundleSignature</key>
<string>????</string>
<key>CFBundleVersion</key>
<string>${FRAMEWORK_CURRENT_VERSION}</string>
</dict>
</plist>
InfoplistEOF
echo "Done: ${FRAMEWORK_BUNDLE}"
boostFrameworkEOF
# build framework
sh build-boostframework.sh
</code></dl>
<br />
<br />
<b>(5) Download and build QuantLib 1.12 for iOS</b><br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code># download quantlib source
cd $HOME/Downloads
# latest is https://jaist.dl.sourceforge.net/project/quantlib/QuantLib/1.12/QuantLib-1.12.tar.gz (February 1st, 2018)
export QL_VERSION=1.12
curl -O https://jaist.dl.sourceforge.net/project/quantlib/QuantLib/${QL_VERSION}/QuantLib-${QL_VERSION}.tar.gz
# curl -O https://jaist.dl.sourceforge.net/project/quantlib/QuantLib/1.9/QuantLib-1.9.tar.gz
tar xzvf QuantLib-${QL_VERSION}.tar.gz
cd QuantLib-${QL_VERSION}
#Create the following shell script for armv7
cat > ~/Downloads/QuantLib-${QL_VERSION}/build-armv7.sh << 'armv7EOF'
#!/bin/bash
# build-armv7.sh
BUILD_DIR=ios-build
mkdir -p ${BUILD_DIR}
INSTALL_DIR="`pwd`/${BUILD_DIR}"
YOUR_PATH_TO_INSTALL_DIR="`cd "${INSTALL_DIR}";pwd`"
# XCODE_VERSION=Xcode-9.0
# XCODE_ROOT=/Applications/${XCODE_VERSION}.app/Contents/Developer
XCODE_ROOT=/Applications/Xcode.app/Contents/Developer
YOUR_TOOLCHAIN=${XCODE_ROOT}/Toolchains/XcodeDefault.xctoolchain/usr/bin
# xcrun --sdk iphoneos --show-sdk-path
# sysroot=${XCODE_ROOT}/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk
sysroot="`xcrun --sdk iphoneos --show-sdk-path`"
export CC=${YOUR_TOOLCHAIN}/clang
export CXX=${YOUR_TOOLCHAIN}/clang++
export CPP=${YOUR_TOOLCHAIN}/cpp
export AR=${YOUR_TOOLCHAIN}/ar
export RANLIB=${YOUR_TOOLCHAIN}/ranlib
export LD=${YOUR_TOOLCHAIN}/ld
export CFLAGS="-arch armv7 -std=c99 -miphoneos-version-min=7.0 -isysroot ${sysroot}"
export CXXFLAGS="-arch armv7 -fexceptions -frtti -fpic -std=c++11 -stdlib=libc++ -miphoneos-version-min=7.0 -O2 -g -fvisibility=hidden -fvisibility-inlines-hidden -Os -isysroot ${sysroot}"
export LDFLAGS="-stdlib=libc++"
boost_include="`cd "../boost_1_62_0/${BUILD_DIR}/include";pwd`"
boost_lib="`cd "../boost_1_62_0/${BUILD_DIR}/armv7/lib";pwd`"
make distclean
./configure --prefix=/armv7 --host=arm-apple-darwin10 --with-sysroot=${sysroot} --with-boost-include=${boost_include} --with-boost-lib=${boost_lib} --enable-static --enable-shared=no --disable-examples
make -j6
make install DESTDIR=${YOUR_PATH_TO_INSTALL_DIR}
armv7EOF
#Create the following shell script for armv7s
cat > ~/Downloads/QuantLib-${QL_VERSION}/build-armv7s.sh << 'armv7sEOF'
#!/bin/bash
# build-armv7s.sh
BUILD_DIR=ios-build
mkdir -p ${BUILD_DIR}
INSTALL_DIR="`pwd`/${BUILD_DIR}"
YOUR_PATH_TO_INSTALL_DIR="`cd "${INSTALL_DIR}";pwd`"
# XCODE_VERSION=Xcode-9.0
# XCODE_ROOT=/Applications/${XCODE_VERSION}.app/Contents/Developer
XCODE_ROOT=/Applications/Xcode.app/Contents/Developer
YOUR_TOOLCHAIN=${XCODE_ROOT}/Toolchains/XcodeDefault.xctoolchain/usr/bin
# xcrun --sdk iphoneos --show-sdk-path
# sysroot=${XCODE_ROOT}/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk
sysroot="`xcrun --sdk iphoneos --show-sdk-path`"
export CC=${YOUR_TOOLCHAIN}/clang
export CXX=${YOUR_TOOLCHAIN}/clang++
export CPP=${YOUR_TOOLCHAIN}/cpp
export AR=${YOUR_TOOLCHAIN}/ar
export RANLIB=${YOUR_TOOLCHAIN}/ranlib
export LD=${YOUR_TOOLCHAIN}/ld
export CFLAGS="-arch armv7s -std=c99 -miphoneos-version-min=7.0 -isysroot ${sysroot}"
export CXXFLAGS="-arch armv7s -fexceptions -frtti -fpic -std=c++11 -stdlib=libc++ -miphoneos-version-min=7.0 -O2 -g -fvisibility=hidden -fvisibility-inlines-hidden -Os -isysroot ${sysroot}"
export LDFLAGS="-stdlib=libc++"
boost_include="`cd "../boost_1_62_0/${BUILD_DIR}/include";pwd`"
boost_lib="`cd "../boost_1_62_0/${BUILD_DIR}/armv7s/lib";pwd`"
make distclean
./configure --prefix=/armv7s --host=arm-apple-darwin10 --with-sysroot=${sysroot} --with-boost-include=${boost_include} --with-boost-lib=${boost_lib} --enable-static --enable-shared=no --disable-examples
make -j6
make install DESTDIR=${YOUR_PATH_TO_INSTALL_DIR}
armv7sEOF
#Create the following shell script for arm64
cat > ~/Downloads/QuantLib-${QL_VERSION}/build-arm64.sh << 'arm64EOF'
#!/bin/bash
# build-arm64.sh
ARCH=arm64
BOOST_FOLDER=boost_1_62_0
BUILD_DIR=ios-build
mkdir -p ${BUILD_DIR}
INSTALL_DIR="`pwd`/${BUILD_DIR}"
YOUR_PATH_TO_INSTALL_DIR="`cd "${INSTALL_DIR}";pwd`"
XCODE_ROOT=/Applications/Xcode.app/Contents/Developer
YOUR_TOOLCHAIN=${XCODE_ROOT}/Toolchains/XcodeDefault.xctoolchain/usr/bin
# xcrun --sdk iphoneos --show-sdk-path
# sysroot=${XCODE_ROOT}/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk
# sysroot="`xcrun --show-sdk-path`"
sysroot="`xcrun --sdk iphoneos --show-sdk-path`"
export CC=${YOUR_TOOLCHAIN}/clang
export CXX=${YOUR_TOOLCHAIN}/clang++
export CPP=${YOUR_TOOLCHAIN}/cpp
export AR=${YOUR_TOOLCHAIN}/ar
export RANLIB=${YOUR_TOOLCHAIN}/ranlib
export LD=${YOUR_TOOLCHAIN}/ld
export CFLAGS="-arch ${ARCH} -std=c99 -miphoneos-version-min=7.0 -isysroot ${sysroot}"
export CXXFLAGS="-arch ${ARCH} -fexceptions -frtti -fpic -std=c++11 -stdlib=libc++ -miphoneos-version-min=7.0 -O2 -g -fvisibility=hidden -fvisibility-inlines-hidden -Os -isysroot ${sysroot}"
export LDFLAGS="-stdlib=libc++"
boost_include="`cd "../${BOOST_FOLDER}/${BUILD_DIR}/include";pwd`"
boost_lib="`cd "../${BOOST_FOLDER}/${BUILD_DIR}/arm64/lib";pwd`"
make distclean
./configure --prefix=/${ARCH} --host=${ARCH}-apple-darwin10 --with-sysroot=${sysroot} --with-boost-include=${boost_include} --with-boost-lib=${boost_lib} --enable-static --enable-shared=no --disable-examples
make -j6
make install DESTDIR=${YOUR_PATH_TO_INSTALL_DIR}
arm64EOF
#Create the following shell script for i386
cat > ~/Downloads/QuantLib-${QL_VERSION}/build-i386.sh << 'i386EOF'
#!/bin/bash
# build-i386.sh
BUILD_DIR=ios-build
mkdir -p ${BUILD_DIR}
INSTALL_DIR="`pwd`/${BUILD_DIR}"
YOUR_PATH_TO_INSTALL_DIR="`cd "${INSTALL_DIR}";pwd`"
# XCODE_VERSION=Xcode-9.0
# XCODE_ROOT=/Applications/${XCODE_VERSION}.app/Contents/Developer
XCODE_ROOT=/Applications/Xcode.app/Contents/Developer
YOUR_TOOLCHAIN=${XCODE_ROOT}/Toolchains/XcodeDefault.xctoolchain/usr/bin
# xcrun --sdk iphonesimulator --show-sdk-path
# sysroot=${XCODE_ROOT}/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk
sysroot="`xcrun --sdk iphonesimulator --show-sdk-path`"
export CC=${YOUR_TOOLCHAIN}/clang
export CXX=${YOUR_TOOLCHAIN}/clang++
export CPP=${YOUR_TOOLCHAIN}/cpp
export AR=${YOUR_TOOLCHAIN}/ar
export RANLIB=${YOUR_TOOLCHAIN}/ranlib
export LD=${YOUR_TOOLCHAIN}/ld
export CFLAGS="-arch i386 -std=c99 -miphoneos-version-min=7.0 -isysroot ${sysroot}"
export CXXFLAGS="-arch i386 -fexceptions -frtti -fpic -std=c++11 -stdlib=libc++ -miphoneos-version-min=7.0 -O2 -g -fvisibility=hidden -fvisibility-inlines-hidden -Os -isysroot ${sysroot}"
export LDFLAGS="-stdlib=libc++"
boost_include="`cd "../boost_1_62_0/${BUILD_DIR}/include";pwd`"
boost_lib="`cd "../boost_1_62_0/${BUILD_DIR}/i386/lib";pwd`"
make distclean
./configure --prefix=/i386 --host=i386-apple-darwin10 --with-sysroot=${sysroot} --with-boost-include=${boost_include} --with-boost-lib=${boost_lib} --enable-static --enable-shared=no --disable-examples
make -j6
make install DESTDIR=${YOUR_PATH_TO_INSTALL_DIR}
i386EOF
#Create the following shell script for x86_64
cat > ~/Downloads/QuantLib-${QL_VERSION}/build-x86_64.sh << 'x86_64EOF'
#!/bin/bash
# build-x86_64.sh
ARCH=x86_64
BOOST_FOLDER=boost_1_62_0
BUILD_DIR=ios-build
mkdir -p ${BUILD_DIR}
INSTALL_DIR="`pwd`/${BUILD_DIR}"
YOUR_PATH_TO_INSTALL_DIR="`cd "${INSTALL_DIR}";pwd`"
XCODE_ROOT=/Applications/Xcode.app/Contents/Developer
YOUR_TOOLCHAIN=${XCODE_ROOT}/Toolchains/XcodeDefault.xctoolchain/usr/bin
# xcrun --sdk iphonesimulator --show-sdk-path
# sysroot=${XCODE_ROOT}/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk
# sysroot="`xcrun --show-sdk-path`"
sysroot="`xcrun --sdk iphonesimulator --show-sdk-path`"
export CC=${YOUR_TOOLCHAIN}/clang
export CXX=${YOUR_TOOLCHAIN}/clang++
export CPP=${YOUR_TOOLCHAIN}/cpp
export AR=${YOUR_TOOLCHAIN}/ar
export RANLIB=${YOUR_TOOLCHAIN}/ranlib
export LD=${YOUR_TOOLCHAIN}/ld
export CFLAGS="-arch ${ARCH} -std=c99 -miphoneos-version-min=7.0 -isysroot ${sysroot}"
export CXXFLAGS="-arch ${ARCH} -fexceptions -frtti -fpic -std=c++11 -stdlib=libc++ -miphoneos-version-min=7.0 -O2 -g -fvisibility=hidden -fvisibility-inlines-hidden -Os -isysroot ${sysroot}"
export LDFLAGS="-stdlib=libc++"
boost_include="`cd "../${BOOST_FOLDER}/${BUILD_DIR}/include";pwd`"
boost_lib="`cd "../${BOOST_FOLDER}/${BUILD_DIR}/x86_64/lib";pwd`"
make distclean
./configure --prefix=/${ARCH} --host=${ARCH}-apple-darwin10 --with-sysroot=${sysroot} --with-boost-include=${boost_include} --with-boost-lib=${boost_lib} --enable-static --enable-shared=no --disable-examples
make -j6
make install DESTDIR=${YOUR_PATH_TO_INSTALL_DIR}
x86_64EOF
#run shell scripts to clean and build for iOS
cd $HOME/Downloads
cd QuantLib-${QL_VERSION}
rm -fr ios-build
sh build-armv7.sh
sh build-armv7s.sh
sh build-arm64.sh
sh build-i386.sh
sh build-x86_64.sh
# create FAT libQuantLib.a archive
cd $HOME/Downloads/QuantLib-${QL_VERSION}
cd ios-build
mkdir -p lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo \
-arch armv7 "armv7/lib/libQuantLib.a" \
-arch armv7s "armv7s/lib/libQuantLib.a" \
-arch arm64 "arm64/lib/libQuantLib.a" \
-arch i386 "i386/lib/libQuantLib.a" \
-arch x86_64 "x86_64/lib/libQuantLib.a" \
-output "lib/libQuantLib.a" \
-create
# create link for include folder
mkdir include
cd include
ln -s ../arm64/include/ql .
</code></dl>
<br />
<br />
<b>(6) Create QuantLib Framework for iOS</b><br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>export QL_VERSION=1.12
cd $HOME/Downloads/QuantLib-${QL_VERSION}
cd ios-build
#Create the following shell script for build-qlframework.sh
cat > build-qlframework.sh << 'qlFrameworkEOF'
#!/bin/bash
# build-qlframework.sh
VERSION_TYPE=Alpha
FRAMEWORK_NAME=ql
FRAMEWORK_VERSION=A
FRAMEWORKDIR=.
FRAMEWORK_CURRENT_VERSION=${QL_VERSION}
FRAMEWORK_COMPATIBILITY_VERSION=${QL_VERSION}
FRAMEWORK_BUNDLE=${FRAMEWORKDIR}/${FRAMEWORK_NAME}.framework
rm -rf ${FRAMEWORK_BUNDLE}
mkdir -p ${FRAMEWORK_BUNDLE}
mkdir -p ${FRAMEWORK_BUNDLE}/Versions
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Resources
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Headers
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Documentation
ln -s $FRAMEWORK_VERSION ${FRAMEWORK_BUNDLE}/Versions/Current
ln -s Versions/Current/Headers ${FRAMEWORK_BUNDLE}/Headers
ln -s Versions/Current/Resources ${FRAMEWORK_BUNDLE}/Resources
ln -s Versions/Current/Documentation ${FRAMEWORK_BUNDLE}/Documentation
ln -s Versions/Current/$FRAMEWORK_NAME ${FRAMEWORK_BUNDLE}/$FRAMEWORK_NAME
FRAMEWORK_INSTALL_NAME=${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/${FRAMEWORK_NAME}
echo "Framework: Lipoing library into ${FRAMEWORK_INSTALL_NAME}"
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo \
-arch armv7 "armv7/lib/libQuantLib.a" \
-arch armv7s "armv7s/lib/libQuantLib.a" \
-arch arm64 "arm64/lib/libQuantLib.a" \
-arch i386 "i386/lib/libQuantLib.a" \
-arch x86_64 "x86_64/lib/libQuantLib.a" \
-output "${FRAMEWORK_INSTALL_NAME}" \
-create
echo "Framework: Copying includes..."
cp -r include/${FRAMEWORK_NAME}/* ${FRAMEWORK_BUNDLE}/Headers/
cat > ${FRAMEWORK_BUNDLE}/Resources/Info.plist <<InfoplistEOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>English</string>
<key>CFBundleExecutable</key>
<string>${FRAMEWORK_NAME}</string>
<key>CFBundleIdentifier</key>
<string>quantlib.org</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundlePackageType</key>
<string>FMWK</string>
<key>CFBundleSignature</key>
<string>????</string>
<key>CFBundleVersion</key>
<string>${FRAMEWORK_CURRENT_VERSION}</string>
</dict>
</plist>
InfoplistEOF
echo "Done: ${FRAMEWORK_BUNDLE}"
qlFrameworkEOF
# build framework
sh build-qlframework.sh
</code></dl>
<br />
<br />
<b>(7) Test build x86_64 for iOS frameworks with</b><br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>#Create testql.cpp
cd $HOME/Downloads
cat > testql.cpp << 'testqlEOF'
#include <ql/quantlib.hpp>
int main() {
std::cout << "BOOST version is " << BOOST_VERSION << std::endl;
std::cout << "QL version is " << QL_VERSION << std::endl;
#if __x86_64__ || __WORDSIZE == 64
std::cout << "This is 64 bits" << std::endl;
#elif __i386__ || __WORDSIZE == 32
std::cout << "This is 32 bits" << std::endl;
#else
std::cout << "This is something else" << std::endl;
#endif
return 0;
}
testqlEOF
# test build example using iOS frameworks
export QL_VERSION=1.12
xcrun clang++ -std=c++11 -stdlib=libc++ -F $HOME/Downloads/QuantLib-${QL_VERSION}/ios-build -F $HOME/Downloads/boost_1_62_0/ios-build -framework ql -isysroot `xcrun --sdk iphonesimulator --show-sdk-path` -arch x86_64 -miphoneos-version-min=7.0 -fvisibility=hidden -o testql testql.cpp
./testql
</code></dl>
<br />
<br />
<b>(8) Build frameworks for macosx and build RQuantLib package from source</b><br />
<dl class="codebox">
<dt>shell script <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>cd $HOME/Downloads/boost_1_62_0
# Append the following user-config.jam at ~/Downloads/boost_1_62_0/tools/build/src/
cat >> tools/build/src/user-config.jam << 'EOF'
# ------------------
# darwin macosx configuration.
# ------------------
local XCODE_ROOT = /Applications/Xcode.app/Contents/Developer ;
# xcrun --sdk macosx --show-sdk-path
local MACOSXSDK = MacOSX.sdk ;
using darwin : i386~macosx
:
$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
:
<striper>$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip
<root>$(XCODE_ROOT)/Platforms/MacOSX.platform/Developer/SDKs/$(MACOSXSDK)
<compileflags>"-arch i386"
<compileflags>"-isysroot $(XCODE_ROOT)/Platforms/MacOSX.platform/Developer/SDKs/$(MACOSXSDK)"
<compileflags>-fexceptions
<compileflags>-frtti
<compileflags>-fpic
<compileflags>-DBOOST_SP_USE_SPINLOCK
<compileflags>-std=c++11
<compileflags>-stdlib=libc++
<compileflags>-mmacosx-version-min=10.9
<compileflags>-O2
<compileflags>-g
<compileflags>-Os
<compileflags>-Wno-long-long
<compileflags>-Wno-missing-field-initializers
<compileflags>-Wno-unused-variable
<linkflags>-stdlib=libc++
<linkflags>-mmacosx-version-min=10.9
<architecture>x86
<address-model>32
<target-os>darwin
;
using darwin : x86_64~macosx
:
$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
:
<striper>$(XCODE_ROOT)/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip
<root>$(XCODE_ROOT)/Platforms/MacOSX.platform/Developer/SDKs/$(MACOSXSDK)
<compileflags>"-arch x86_64"
<compileflags>"-isysroot $(XCODE_ROOT)/Platforms/MacOSX.platform/Developer/SDKs/$(MACOSXSDK)"
<compileflags>-fexceptions
<compileflags>-frtti
<compileflags>-fpic
<compileflags>-DBOOST_SP_USE_SPINLOCK
<compileflags>-std=c++11
<compileflags>-stdlib=libc++
<compileflags>-mmacosx-version-min=10.9
<compileflags>-O2
<compileflags>-g
<compileflags>-Os
<compileflags>-Wno-long-long
<compileflags>-Wno-missing-field-initializers
<compileflags>-Wno-unused-variable
<linkflags>-stdlib=libc++
<linkflags>-mmacosx-version-min=10.9
<architecture>x86
<address-model>64
<target-os>darwin
;
EOF
# build boost libraries for macosx
./bootstrap.sh --with-libraries=atomic,chrono,date_time,exception,filesystem,graph,iostreams,math,program_options,random,regex,serialization,signals,system,test,thread,wave
# clean and build for macosx, j option is num of cores x 1.5
rm -fr macosx-build
./b2 -j6 --build-dir=macosx-build --stagedir=macosx-build/i386 toolset=darwin-i386~macosx architecture=x86 address-model=32 target-os=darwin threading=multi link=static stage
./b2 -j6 --build-dir=macosx-build --stagedir=macosx-build/x86_64 toolset=darwin-x86_64~macosx architecture=x86 address-model=64 target-os=darwin threading=multi link=static stage
# create libboost.a archive for each architecture
cd $HOME/Downloads/boost_1_62_0/macosx-build
xcrun --sdk macosx ar crus i386/libboost.a boost/bin.v2/libs/*/build/darwin-i386~macosx/release/link-static/threading-multi/*/*.o
xcrun --sdk macosx ar crus x86_64/libboost.a boost/bin.v2/libs/*/build/darwin-x86_64~macosx/release/link-static/threading-multi/*/*.o
# create FAT libboost.a archive
mkdir -p lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo \
-arch i386 "i386/libboost.a" \
-arch x86_64 "x86_64/libboost.a" \
-output "lib/libboost.a" \
-create
# create link for include folder
mkdir include
cd include
ln -s ../../boost .
# build Boost Framework for macosx
#Create the following shell script for build-boostframework.sh
cd $HOME/Downloads/boost_1_62_0/macosx-build
cat > build-boostframework.sh << 'boostFrameworkEOF'
#!/bin/bash
# build-boostframework.sh
VERSION_TYPE=Alpha
FRAMEWORK_NAME=boost
FRAMEWORK_VERSION=A
FRAMEWORKDIR=.
FRAMEWORK_CURRENT_VERSION=1.62.0
FRAMEWORK_COMPATIBILITY_VERSION=1.62.0
FRAMEWORK_BUNDLE=${FRAMEWORKDIR}/${FRAMEWORK_NAME}.framework
rm -rf ${FRAMEWORK_BUNDLE}
mkdir -p ${FRAMEWORK_BUNDLE}
mkdir -p ${FRAMEWORK_BUNDLE}/Versions
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Resources
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Headers
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Documentation
ln -s $FRAMEWORK_VERSION ${FRAMEWORK_BUNDLE}/Versions/Current
ln -s Versions/Current/Headers ${FRAMEWORK_BUNDLE}/Headers
ln -s Versions/Current/Resources ${FRAMEWORK_BUNDLE}/Resources
ln -s Versions/Current/Documentation ${FRAMEWORK_BUNDLE}/Documentation
ln -s Versions/Current/$FRAMEWORK_NAME ${FRAMEWORK_BUNDLE}/$FRAMEWORK_NAME
FRAMEWORK_INSTALL_NAME=${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/${FRAMEWORK_NAME}
echo "Framework: Lipoing library into ${FRAMEWORK_INSTALL_NAME}"
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo \
-arch i386 "i386/libboost.a" \
-arch x86_64 "x86_64/libboost.a" \
-output "${FRAMEWORK_INSTALL_NAME}" \
-create
echo "Framework: Copying includes..."
cp -r include/${FRAMEWORK_NAME}/* ${FRAMEWORK_BUNDLE}/Headers/
cat > ${FRAMEWORK_BUNDLE}/Resources/Info.plist <<InfoplistEOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>English</string>
<key>CFBundleExecutable</key>
<string>${FRAMEWORK_NAME}</string>
<key>CFBundleIdentifier</key>
<string>${FRAMEWORK_NAME}.org</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundlePackageType</key>
<string>FMWK</string>
<key>CFBundleSignature</key>
<string>????</string>
<key>CFBundleVersion</key>
<string>${FRAMEWORK_CURRENT_VERSION}</string>
</dict>
</plist>
InfoplistEOF
echo "Done: ${FRAMEWORK_BUNDLE}"
boostFrameworkEOF
# build framework
sh build-boostframework.sh
# build QuantLib libraries for macosx
export QL_VERSION=1.12
cd $HOME/Downloads/QuantLib-${QL_VERSION}
#Create the following shell script for i386
cat > ~/Downloads/QuantLib-${QL_VERSION}/build-i386-macosx.sh << 'i386EOF'
#!/bin/bash
# build-i386-macosx.sh
BUILD_DIR=macosx-build
mkdir -p ${BUILD_DIR}
INSTALL_DIR="`pwd`/${BUILD_DIR}"
YOUR_PATH_TO_INSTALL_DIR="`cd "${INSTALL_DIR}";pwd`"
# XCODE_VERSION=Xcode-9.0
# XCODE_ROOT=/Applications/${XCODE_VERSION}.app/Contents/Developer
XCODE_ROOT=/Applications/Xcode.app/Contents/Developer
YOUR_TOOLCHAIN=${XCODE_ROOT}/Toolchains/XcodeDefault.xctoolchain/usr/bin
# xcrun --sdk macosx --show-sdk-path
# sysroot=${XCODE_ROOT}/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
sysroot="`xcrun --sdk macosx --show-sdk-path`"
export CC=${YOUR_TOOLCHAIN}/clang
export CXX=${YOUR_TOOLCHAIN}/clang++
export CPP=${YOUR_TOOLCHAIN}/cpp
export AR=${YOUR_TOOLCHAIN}/ar
export RANLIB=${YOUR_TOOLCHAIN}/ranlib
export LD=${YOUR_TOOLCHAIN}/ld
export CFLAGS="-arch i386 -std=c99 -mmacosx-version-min=10.7 -isysroot ${sysroot}"
export CXXFLAGS="-arch i386 -fexceptions -frtti -fpic -std=c++11 -stdlib=libc++ -mmacosx-version-min=10.9 -O2 -g -Os -isysroot ${sysroot}"
export LDFLAGS="-stdlib=libc++ -mmacosx-version-min=10.9"
boost_include="`cd "../boost_1_62_0/${BUILD_DIR}/include";pwd`"
boost_lib="`cd "../boost_1_62_0/${BUILD_DIR}/i386/lib";pwd`"
make distclean
./configure --prefix=/i386 --host=i386-apple-darwin10 --with-sysroot=${sysroot} --with-boost-include=${boost_include} --with-boost-lib=${boost_lib} --enable-intraday --enable-static --enable-shared=no --disable-examples
make -j6
make install DESTDIR=${YOUR_PATH_TO_INSTALL_DIR}
i386EOF
#Create the following shell script for x86_64
cat > ~/Downloads/QuantLib-${QL_VERSION}/build-x86_64-macosx.sh << 'x86_64EOF'
#!/bin/bash
# build-x86_64-macosx.sh
BUILD_DIR=macosx-build
mkdir -p ${BUILD_DIR}
INSTALL_DIR="`pwd`/${BUILD_DIR}"
YOUR_PATH_TO_INSTALL_DIR="`cd "${INSTALL_DIR}";pwd`"
XCODE_ROOT=/Applications/Xcode.app/Contents/Developer
YOUR_TOOLCHAIN=${XCODE_ROOT}/Toolchains/XcodeDefault.xctoolchain/usr/bin
# xcrun --sdk macosx --show-sdk-path
# sysroot=${XCODE_ROOT}/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
sysroot="`xcrun --sdk macosx --show-sdk-path`"
export CC=${YOUR_TOOLCHAIN}/clang
export CXX=${YOUR_TOOLCHAIN}/clang++
export CPP=${YOUR_TOOLCHAIN}/cpp
export AR=${YOUR_TOOLCHAIN}/ar
export RANLIB=${YOUR_TOOLCHAIN}/ranlib
export LD=${YOUR_TOOLCHAIN}/ld
export CFLAGS="-arch x86_64 -std=c99 -mmacosx-version-min=10.7 -isysroot ${sysroot}"
export CXXFLAGS="-arch x86_64 -fexceptions -frtti -fpic -std=c++11 -stdlib=libc++ -mmacosx-version-min=10.9 -O2 -g -Os -isysroot ${sysroot}"
export LDFLAGS="-stdlib=libc++ -mmacosx-version-min=10.9"
boost_include="`cd "../boost_1_62_0/${BUILD_DIR}/include";pwd`"
boost_lib="`cd "../boost_1_62_0/${BUILD_DIR}/x86_64/lib";pwd`"
make distclean
./configure --prefix=/x86_64 --host=x86_64-apple-darwin10 --with-sysroot=${sysroot} --with-boost-include=${boost_include} --with-boost-lib=${boost_lib} --enable-intraday --enable-static --enable-shared=no --disable-examples
make -j6
make install DESTDIR=${YOUR_PATH_TO_INSTALL_DIR}
x86_64EOF
#run shell scripts to clean and build for macosx
cd $HOME/Downloads/QuantLib-${QL_VERSION}
rm -fr macosx-build
sh build-i386-macosx.sh
sh build-x86_64-macosx.sh
# create FAT libQuantLib.a archive
cd $HOME/Downloads/QuantLib-${QL_VERSION}
cd macosx-build
mkdir -p lib
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo \
-arch i386 "i386/lib/libQuantLib.a" \
-arch x86_64 "x86_64/lib/libQuantLib.a" \
-output "lib/libQuantLib.a" \
-create
# create link for include folder
mkdir include
cd include
ln -s ../x86_64/include/ql .
cd $HOME/Downloads/QuantLib-${QL_VERSION}/macosx-build
# build QuantLib Framework for macosx
#Create the following shell script for build-qlframework.sh
cat > build-qlframework.sh << 'qlFrameworkEOF'
#!/bin/bash
# build-qlframework.sh
VERSION_TYPE=Alpha
FRAMEWORK_NAME=ql
FRAMEWORK_VERSION=A
FRAMEWORKDIR=.
FRAMEWORK_CURRENT_VERSION=${QL_VERSION}
FRAMEWORK_COMPATIBILITY_VERSION=${QL_VERSION}
FRAMEWORK_BUNDLE=${FRAMEWORKDIR}/${FRAMEWORK_NAME}.framework
rm -rf ${FRAMEWORK_BUNDLE}
mkdir -p ${FRAMEWORK_BUNDLE}
mkdir -p ${FRAMEWORK_BUNDLE}/Versions
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Resources
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Headers
mkdir -p ${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/Documentation
ln -s $FRAMEWORK_VERSION ${FRAMEWORK_BUNDLE}/Versions/Current
ln -s Versions/Current/Headers ${FRAMEWORK_BUNDLE}/Headers
ln -s Versions/Current/Resources ${FRAMEWORK_BUNDLE}/Resources
ln -s Versions/Current/Documentation ${FRAMEWORK_BUNDLE}/Documentation
ln -s Versions/Current/$FRAMEWORK_NAME ${FRAMEWORK_BUNDLE}/$FRAMEWORK_NAME
FRAMEWORK_INSTALL_NAME=${FRAMEWORK_BUNDLE}/Versions/${FRAMEWORK_VERSION}/${FRAMEWORK_NAME}
echo "Framework: Lipoing library into ${FRAMEWORK_INSTALL_NAME}"
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo \
-arch i386 "i386/lib/libQuantLib.a" \
-arch x86_64 "x86_64/lib/libQuantLib.a" \
-output "${FRAMEWORK_INSTALL_NAME}" \
-create
echo "Framework: Copying includes..."
cp -r include/${FRAMEWORK_NAME}/* ${FRAMEWORK_BUNDLE}/Headers/
cat > ${FRAMEWORK_BUNDLE}/Resources/Info.plist <<InfoplistEOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleDevelopmentRegion</key>
<string>English</string>
<key>CFBundleExecutable</key>
<string>${FRAMEWORK_NAME}</string>
<key>CFBundleIdentifier</key>
<string>quantlib.org</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundlePackageType</key>
<string>FMWK</string>
<key>CFBundleSignature</key>
<string>????</string>
<key>CFBundleVersion</key>
<string>${FRAMEWORK_CURRENT_VERSION}</string>
</dict>
</plist>
InfoplistEOF
echo "Done: ${FRAMEWORK_BUNDLE}"
qlFrameworkEOF
# build framework
sh build-qlframework.sh
# Test build example using macosx frameworks
export QL_VERSION=1.12
clang++ -std=c++11 -stdlib=libc++ -F $HOME/Downloads/QuantLib-${QL_VERSION}/macosx-build -F $HOME/Downloads/boost_1_62_0/macosx-build -framework ql -arch x86_64 -o testql testql.cpp
./testql
# Build RQuantLib for Mac OSX 10.12.6
(1) Build the above macosx x86_64 libraries for QuantLib and Boost. For Quantlib, it should be version >=1.8 and with --enable-intraday option. For Boost and QuantLib, should remove -fvisibility=hidden -fvisibility-inlines-hidden
(2) Install R 3.4.4 from https://cran.r-project.org/bin/macosx/ and optionally install RStudio
(3) Install clang 4.0 from http://r.research.att.com/libs/
sudo tar xzvf ${HOME}/Downloads/clang-4.0.0-darwin15.6-Release.tar.gz -C /
(4) Create R/Makevars
mkdir -p ${HOME}/.R
cat > ${HOME}/.R/Makevars << 'EOF'
CC=/usr/local/clang4/bin/clang
CXX=/usr/local/clang4/bin/clang++
CXXFLAGS=-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -Wno-nonportable-include-path -Wno-unused-variable -I${HOME}/Downloads/QuantLib-1.12/macosx-build/x86_64/include
LDFLAGS=-L/usr/local/clang4/lib -L${HOME}/Downloads/QuantLib-1.12/macosx-build/x86_64/lib
EOF
# if compile using clang6 under the latest R 3.5.0, should change path to /usr/local/clang6 accordingly.
(5) Install QuantLib 1.12 to /usr/local
export QL_VERSION=1.12
cd ${HOME}/Downloads/QuantLib-${QL_VERSION}/macosx-build/x86_64; tar czvf ../ql-${QL_VERSION}.tgz bin include lib share
cd ${HOME}/Downloads/QuantLib-${QL_VERSION}/macosx-build
sudo tar xzvf ql-${QL_VERSION}.tgz -C /usr/local
(6) Build RQuantLib with R terminal session from source and install
R
install.packages("RQuantLib", configure.args = "--with-boost-include=${HOME}/Downloads/boost_1_62_0/macosx-build/include --with-boost-lib=${HOME}/Downloads/boost_1_62_0/macosx-build/x86_64/lib", type = "source")
# installing to /Library/Frameworks/R.framework/Versions/3.4/Resources/library/RQuantLib/libs
# build binary package (should install dependencies Rcpp and zoo first) using the following
# curl -O https://cran.cnr.berkeley.edu/src/contrib/RQuantLib_0.4.4.tar.gz
# R CMD INSTALL --build --configure-args="--with-boost-include=${HOME}/Downloads/boost_1_62_0/macosx-build/include --with-boost-lib=${HOME}/Downloads/boost_1_62_0/macosx-build/x86_64/lib" RQuantLib_0.4.4.tar.gz
(7) Optionally, install https://www.xquartz.org/ if want to remove warning on : unable to load shared object '/Library/Frameworks/R.framework/Resources/modules//R_X11.so'
# Install dependencies for testing RQuantLib
R
install.packages(c("Rcpp","zoo"))
install.packages(c("rgl","RUnit"))
</code></dl>
<br />
<br />
<b>(9) Test RQuantLib binary package for R version 3.4.4 on Mac OSX 10.12.6</b>
<dl class="codebox">
<dt>bond1.R <a href="#" onclick="selectCode(this); return false;">Select all</a></dt>
<code>
#Simple call with a flat curve
require(RQuantLib)
bond <- list(faceAmount=100,
settlementDays=1,
issueDate=as.Date("2014-11-30"),
redemption=100)
<br/>
scheduleparams <- list(effectiveDate=as.Date("2014-11-30"),
maturityDate=as.Date("2018-11-30"),
period=2,
calendar="UnitedStates",
businessDayConvention = 4,
terminationDateConvention=4,
dateGeneration=1,
endOfMonth=1,
dayCounter = 'Thirty360')
<br/>
coupon.rate <- c(0.02875)
<br/>
params <- list(tradeDate=as.Date('2014-2-15'),
settleDate=as.Date('2014-2-19'),
dt=.25,
interpWhat="discount",
interpHow="loglinear")
<br/>
setEvaluationDate(as.Date("2014-11-22"))
<br/>
discountCurve.flat <- DiscountCurve(params, list(flat=0.05))
<br/>
FixedRateBond(bond=bond, rates=coupon.rate, schedule=scheduleparams, calc=list(dayCounter='ActualActual.ISMA',compounding='Compounded', freq='Annual', durationType='Modified'),discountCurve=discountCurve.flat)
<br/><br/>
FixedRateBondPriceByYield(
settlementDays=1,
yield=0.05127,
faceAmount=100,
effectiveDate=as.Date("2014-11-30"),
maturityDate=as.Date("2018-11-30"),
period=2,
calendar="UnitedStates",
rates=c(0.05),
dayCounter=2,
businessDayConvention=4,
compound=0,
redemption=100,
issueDate=as.Date("2014-11-30"))
<br/><br/>
FixedRateBondYield(
settlementDays=1,
price=99.54593,
faceAmount=100,
effectiveDate=as.Date("2014-11-30"),
maturityDate=as.Date("2018-11-30"),
period=2,
calendar="UnitedStates",
rates=c(0.05),
dayCounter=2,
businessDayConvention=4,
compound=0,
redemption=100,
issueDate=as.Date("2014-11-30"))
</code></dl><br/><br/>
<br />
<br />
<br />Unknownnoreply@blogger.com0